Stage 4a: a platform STUN and TURN relay

Date: 2026-09-07, revision 6 Status: rulings taken. Paul ruled on the six decisions in §9 on 2026-09-07, each marked (ruling); the roadmap's stage 4a text and gate are amended so the two documents agree. Reviewed five times by GPT-6 via Codex: revision 1's 20 findings are in CODEX-FINDINGS-RELAY.md, revision 2's 12 in CODEX-FINDINGS-RELAY-R2.md, revision 3's 8 in CODEX-FINDINGS-RELAY-R3.md, revision 4's 3 in CODEX-FINDINGS-RELAY-R4.md, revision 5's 1 in CODEX-FINDINGS-RELAY-R5.md, unedited; every one is taken, marked (codex R1-N) through (codex R5-N). Round 6 returned accept with no findings (CODEX-FINDINGS-RELAY-R6.md). Stage 4a of the media roadmap (2026-09-07-media-platform-roadmap-design.md), built under rulings 8, 10 and 11 there.

Why now. Oficina's Meet is peer-to-peer WebRTC and cannot name a third-party STUN server, because suite ruling 6 forbids egress that is a property of the product. Without STUN, browsers gather only LAN candidates and calls connect on the same network only. A platform STUN listener is the smallest thing that makes Meet work across the internet; TURN on the same listener is what makes the one call in ten that cannot hole-punch connect anyway. Meet is blocked on nothing else (Meet session, 2026-09-07), and it has reviewed the app-facing contract in §2.

1. What is built

One new platform unit, carlos-relay.service, on every region box the operator arms, running carlos relay serve from the platform binary, built on pion/turn v4.1.4, pure Go, which fetches and cross-compiles clean for the CI shards (checked 2026-09-07). It answers STUN binding requests to anyone and TURN allocations to holders of a credential an enabled app minted. It learns which apps are enabled from a record the console writes, binds every allocation to the identity that made it, ends allocations when their credential, app, region, master or its own authority lapses, meters what it forwards per app, and refuses peers on private networks.

Three console doors and a CLI verb, carlos relay enable|status|disable, that turn the relay on for an app and deliver its keys as sealed config, plus an operator verb carlos relay rotate. An operator-side master key per region, staged in a file on the region's boxes and, as a console secret, in the console. A config-adoption acknowledgement the host's config sync writes, so status can say "adopted" truthfully. A security-group change opening the relay's ports. A DNS name per region relay.

Not built, by ruling: TURN over TLS, an inter-region byte meter. Not built, by scope: an SFU, recording, an issuance API, client-side relay selection, commercial quotas. §8 lists them with reasons.

2. The app-facing contract

This is what Meet codes against. It is fixed by this spec; the rest of the design may change without touching it.

Environment. One variable reaches the instance through the sealed config.json path, the way store credentials do. Because the config renderer refuses quotes, backslashes and newlines in any value (internal/appconfig/materialize.go:23), the value is the standard base64 encoding, with padding, of a JSON document; the app decodes the base64, then parses the JSON. (codex R1-1)

• CARLOS_RELAY: base64 of a JSON array, one element per region the app may use, in the console's order (§5). Absent or empty means no relay: the app shows its floor.

[{"region":"eu-west-1", "urls":["stun:relay.eu-west-1.oncarlos.com:3478", "turn:relay.eu-west-1.oncarlos.com:3478?transport=udp", "turn:relay.eu-west-1.oncarlos.com:3478?transport=tcp"], "key":"<standard base64, with padding, of 32 raw bytes>", "username_suffix":"abc:meet:0123456789abcdef:k1"}]

region is the AWS region name, which is also the vocabulary of the relay hostnames. username_suffix is <account>:<app>:<incarnation>: <kid>, supplied by the platform so the app never has to know its own canonical account sqid or app name. (codex R1-11, R1-12) There is no separate incarnation variable; the suffix carries it. The app builds its ICE server list from whatever urls are present and assumes nothing about their number or schemes.

Minting, per call. For the region it wants, the app computes:

key = base64decode( element.key ) // 32 raw bytes username = "<expiry>:" + element.usernamesuffix // ASCII credential = base64_std( HMAC-SHA256( key, username ) )

where expiry is a decimal Unix time in seconds, at most four hours ahead (ruling 1). The username has exactly five colon-separated fields: expiry ([0-9]+), account ([a-z0-9]+), app (^[a-z][a-z0-9-]{0,31}$), incarnation ([0-9a-f]{16}) and kid ([a-z0-9]{1,8}). No field can contain a colon. The app hands the browser an RTCIceServer list: stun: entries bare, turn: entries carrying username and credential. STUN needs no credential.

Expiry. The relay checks the expiry on every authenticated request the browser's ICE agent sends (allocate, refresh, permission, channel bind) and ends the allocation at the expiry regardless of what the client sends (§3, "termination"), so a credential's deadline is a hard deadline on relayed traffic. An app that needs a call to outlive its credential re-mints and performs an ICE restart before the deadline. Four hours is the maximum, not a promise about meeting lengths. (codex R1-4, R1-20)

What changes the incarnation. Only carlos relay disable followed by carlos relay enable. A repeated enable, a region change, a deploy, a restart and a master rotation all keep it. (codex R1-13)

What changes the key. A master rotation in a region changes that region's key and kid; a new incarnation changes every region's key. Either re-renders the app's config within about a minute, as any config change does today, and what happens next depends on the route's shape (internal/appconfig/sync.go:30, syncexec.go:1): a unit-backed route and a socket-backed exec route are restarted onto the new file; a TCP-address exec route whose bundle opts into file delivery receives the file and is not restarted, because its process belongs to an external supervisor CARLOS cannot bounce, so the app must re-read it and then tell the platform it has (carlos relay adopted, §5); a TCP-address route without file delivery receives nothing, and an app with such a route cannot enable the relay until it opts that route into file delivery or moves it to a unit. carlos relay status names which of these each route is. (codex R1-10, R1-14, R2-6, R3-4, R5-1)

A credential keeps working for its whole life provided the master that signed its key is not force-retired before its expiry. The rotation procedure in §5 guarantees that when followed: the old master is retired no earlier than four hours after the last instance stopped minting from it, as observed by the console, and no earlier than 28 hours after rotation regardless. (codex R2-4)

Test vector. Master is SHA-256("carlos-relay-test-master"), account abc, app meet, incarnation 0123456789abcdef, kid k1, expiry 1900000000:

master 841d374df509340d6ab18a139a27ed0d9c5d61d0bdf306e4124ff913a7a92fb5 (hex) key uMhwW4P/9O5dQUIs91t2CCHGhwJbeiM0xQMQ2G0CJLU= username 1900000000:abc:meet:0123456789abcdef:k1 credential jRU479Y3GO45nrRHqxBf3BMeBHE6Fip1cwAccNv28MQ=

The key derivation is in §4; an app never needs it, only the key it is given. The vector will be checked into internal/relay/vector_test.go by the plan, and Meet's minting test uses the same numbers.

Logging. The app logs which region it used and nothing else about the credential. The relay logs the account, app and outcome of an authentication, never the credential or a key.

3. The relay unit

carlos-relay.service is a media service in the roadmap's §3 terms: a network-facing platform unit like the edge, not a codec helper.

• Process. carlos relay serve, from the platform binary, so it rides the same release and promote path as the edge. A new file in internal/host/units/, embedded and written by bootstrap like every unit, with ConditionPathExists=/etc/carlos/relay-masters so a box without a master file never listens. Added to enabledUnits and to the list of units the systemctl plan starts, not only enables, and to the adopt restart path beside restartEdge (internal/host/systemapply.go:568), so a platform update restarts the relay onto the new binary and a fresh box runs it without a reboot. The staging bootstrap plan prints the same steps without running them, as it does for the edge. (codex R1-16)
• Hardening and memory. From carlos-edge.service and then cut down: no CAP_NET_BIND_SERVICE (3478 is unprivileged), User=carlos, ProtectSystem=strict, ReadWritePaths=/data/relay /run/carlos, RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX. Memory: MemoryMax=256M, MemoryHigh=160M, OOMScoreAdjust=1000. What that guarantees, stated exactly: the relay can never take more than 256 MiB of the box; above 160 MiB it refuses new allocations and ends its oldest until it is under; and it is the kernel's preferred victim among processes of similar footprint, since 1000 is the largest adjustment there is. What it does not guarantee: that the relay dies before a tenant under every host-wide pressure. The kernel's score counts swap and page tables as well as RSS, tenants may swap up to their memory cap (internal/cgroup/cgroup.go:313), and a kill inside the edge's own 80% cgroup considers only the edge's tree. So the relay does not wait for the kernel: it watches the box's memory pressure (/proc/pressure/memory, the some ten-second average) and sheds allocations, oldest first, whenever it exceeds 20%, whatever caused the pressure, resuming admission when it falls under 10%. The gate in §7 tests that shedding, not a kill order. Nice=5 and CPUWeight=50 for CPU. (codex R1-19, R2-8, R3-7)
• Listeners. UDP 3478 and TCP 3478 on the box's primary address. No TLS listener (ruling 4). The relay allocation range is UDP 49152 to 65535, bound through the relay's own address generator (below) with the box's Elastic IP as the advertised relay address, learned as the edge learns it, from IMDS (readPublicIP, internal/host/agent.go:1808), with CARLOS_RELAY_PUBLIC_IP as the override for a box without IMDS.
• STUN. Binding requests are answered on every listener without authentication, which is pion's default and the reason this stage exists.
• The relay owns every socket pion uses. pion has no public call to end an allocation; its allocation manager is internal and Server exposes only AllocationCount and Close. What it does expose is the RelayAddressGenerator that creates each allocation's relay socket (server_config.go:20, returning a net.PacketConn), the listener sockets the relay passes in, and an OnAllocationCreated callback that supplies the relay address before pion starts reading from it (internal/allocation/allocation_manager.go:140). pion's read loop deletes the allocation when the relay socket returns an error (internal/allocation/allocation.go:301). So the relay implements the generator and wraps both kinds of socket: every relay socket is recorded against its allocation at OnAllocationCreated and can be closed by the relay at any time, which ends that allocation inside pion within one read; and both relay and listener sockets count what they forward and apply the token buckets below. That one mechanism is how every deadline is enforced, how metering is attributed, and how bandwidth is bounded. (codex R1-4, R1-17)
• Owner binding. pion locates a refresh, permission or channel bind by the client's five-tuple and does not compare the authenticating username to the one that created the allocation (internal/server/turn.go:34); it also tags every allocation's tuple as UDP even for TCP clients, so the event's protocol string cannot distinguish transports. The relay keys its binding on the typed source address: the Go type of the net.Addr it receives (*net.UDPAddr or *net.TCPAddr) plus its string, which pion passes unchanged to AuthHandler and to the events. On OnAllocationCreated it records that key to the identity fields (account, app, incarnation); its AuthHandler refuses a username whose identity differs from the one bound to a live allocation from that key. Cleanup belongs to the wrapped relay socket, not to OnAllocationDeleted, which carries no relay address (internal/allocation/event_handler.go:20): each wrapper knows the binding it belongs to, and on Close, whether pion closed it or the relay did, it removes that binding only if the map still points at itself, so a replacement allocation bound while the old one was closing (pion removes from its manager before closing, allocation_manager.go:150) is never erased. (codex R1-3, R2-3, R3-3)
• Termination. An allocation is ended, by closing its relay socket, at the earliest of: its credential's expiry (a timer set at creation from the username's expiry field); the app being disabled, its incarnation changing, or this region leaving its list (on the next authority read); the master named by its kid reaching retire_at (on the next masters-file read or timer); the relay's authority lease expiring (below); the app or account crossing a byte budget (below); the relay shedding under memory pressure. Ending one allocation ends its permissions and channels with it. pion's own maximum requested lifetime is one hour; the relay does not rely on it. (codex R1-4, R2-4)
• Authentication. A custom AuthHandler (§4), called by pion on every authenticated request, which is where the username's expiry and the authority record are checked; pion verifies the message integrity against the key the handler returns.
• Peer policy. A PermissionHandler refuses CreatePermission and ChannelBind to loopback, link-local (including the IMDS address), RFC 1918 and unique-local ranges, the box's own private address and VPC CIDR (from CARLOS_RELAY_DENY_CIDRS, set by the operator from tofu output), and any other relay's private address. The box's own Elastic IP is allowed by the permission handler, because two clients that both fall back to this relay hold relay candidates on that address and must reach each other, but only live relay allocation ports are reachable there: the permission handler sees only an IP, so the wrapped relay socket enforces the port on WriteTo, dropping any packet to the EIP whose destination port is not a currently allocated relay port. Nothing else on the box is reachable through the relay, whatever the security group says about source addresses. Refusals and drops are counted per app. (codex R1-8, R2-9)
• Limits (ruling 6), all admitted under one mutex across every listener so counts cannot race, with counters reset on process start and the rolling-hour byte windows rebuilt on start from every hourly metrics object that intersects the window, normally the previous hour's and the current one (codex R3-6): - per app per box: 64 live allocations, 30 new a minute, 20 GB a rolling hour; - per account per box: 256 live, 60 new a minute, 60 GB a rolling hour, whatever the app count; - per box: 2048 live, 500 new a minute; - instantaneous: a token bucket of 8 Mbps per allocation and 200 Mbps per box applied in the wrapped sockets on both forwarding legs, dropping packets beyond the bucket, so a burst is bounded before it reaches the edge's share of the NIC; - crossing a byte budget refuses new allocations with 486 for the app or account and ends its live allocations, because a budget that only stops new ones bounds nothing. Failed allocations count against the per-minute rates. These are operational limits with those defaults, tunable by env; commercial quotas are out of scope. (codex R1-9, R2-7)
• Metering. Two meters per app and one gauge, measured where pion forwards and only for what it forwards: (codex R1-17, R1-18, R2-5; ruling 5) - in: payload bytes the relay accepted from the client and forwarded to a peer, counted at the relay socket's WriteTo; - out: payload bytes the relay forwarded from a peer to the client, counted at the listener socket's WriteTo to the client's address, attributed through the owner binding; - allocs: live allocations at sample time. Both meters are host egress and are what the bill is made of. Peer bytes that arrive at a relay socket and are dropped for lack of permission are counted separately as rejected and never against a budget. Every sixty seconds the relay appends one sample {at, apps: {"<account>/<app>": {in, out, allocs, rejected}}}, with in, out and rejected as deltas since the previous sample, to its own hourly object metrics/relay/<box>/<hour>.json, which it alone writes: read, append, CAS-write, retry on conflict. On restart it reads the current hour's object back, so earlier samples survive and its rolling-hour budgets resume from them. The raw readers and rollups learn the new key in the plan. An inter-region meter is not built (§8).
• Authority record and lease. The relay reads control/relay.json (§5) from the deployment bucket every sixty seconds, and additionally whenever control/serial changes as seen by its own poll of that object (the host agent's fastConvergeLoop, internal/host/agent.go:1437, is a separate process; the relay runs the same shape of poll itself). The lease clock is the time of the last successful read; a failed read renews nothing. If that time is more than fifteen minutes ago the relay refuses new allocations and ends the ones it carries, per the roadmap's serving lease, and resumes on the next successful read. A relay with no record yet serves STUN only. Each read also writes a heartbeat, control/relay-ack/<box>.json, carrying the record's written_at, the read time, which listeners are up, the kids currently loaded from the masters file, and the last masters-file reload error if any. (codex R1-5, R1-14, R2-4)

4. Keys and verification

Region master. Thirty-two random bytes per region, generated by the operator with carlos relay keygen, which prints a kid and the key. On each of the region's boxes they live in /etc/carlos/relay-masters, root-owned, mode 0640, group carlos, one per line:

<kid> <base64 key> <retire_at RFC 3339 or "-">

The relay reads the file at start and re-reads it when it changes, so a rotation never restarts the process. A master past its retire_at is ignored for new requests and its live allocations are ended (§3). (codex R1-10, R2-4)

Where the console's copy lives, and what that exposes (ruling 3). The console must hold every region's masters to derive app keys at enable. It holds them as console secrets, CARLOS_RELAY_MASTERS_<REGION> (region name uppercased with hyphens as underscores, since variable names refuse hyphens, internal/appconfig/appconfig.go:103), one or more kid:key pairs per region, of which the one named by the issuer pointer control/relay-issuer.json is the one keys are derived from (§5, rotation); and console secrets are sealed config like any app's. Sealed config is decryptable on every host (roadmap §1). So the honest statement is: a compromised host can read every region's master, not only the derived keys the roadmap already conceded, and with a master it can mint credentials for any app in that region, present and future, until the operator rotates. Per-region masters still mean a relay box itself holds only its own region's, and rotation of one region recovers that region. The operator docs say this in those words. A design where only a region's boxes ever hold its master, the region's relay deriving app keys and writing them into config itself, is filed as the tightening and is not version one, because it makes the relay a writer of app config with all the CAS and ordering that implies. (codex R1-7, R2-10)

Derived key. For an app, region and incarnation:

key = HKDF-SHA256( ikm = master, salt = "carlos-relay-v1", info = account || 0x00 || app || 0x00 || incarnation, length = 32 )

Verification, in the relay's AuthHandler(username, realm, src), which pion calls once per authenticated request and which returns one key, so the master is chosen from the username, not by trial: (codex R1-2)

1. Split the username on : into exactly five fields; anything else is refused.
2. Parse expiry; refuse if in the past or more than four hours ahead.
3. If the typed source address has a live allocation bound to a different identity (§3, owner binding), refuse.
4. Look up account/app in the authority record; refuse if absent, if the incarnation differs, or if this relay's region is not in the app's list.
5. Look up kid in the masters file; refuse if absent or retired.
6. Derive the key, compute the expected credential, and return turn.GenerateAuthKey(username, realm, credential). pion verifies MESSAGE-INTEGRITY against that key, so a wrong credential fails inside pion; the handler never sees what the client sent.
7. Count the refusal reason per app: bad shape, expired, bound elsewhere, unknown app, wrong incarnation, wrong region, unknown kid.

The MD5 inside GenerateAuthKey is the long-term credential key of RFC 8489's compatibility mechanism, which pion implements; RFC 8489 also defines a SHA-256 variant that pion does not. The HMAC-SHA256 above is ours. (codex R1-20)

5. Enable, disable, rotate

Records. Two, both additive. (codex R1-6, R2-1, R2-2)

• apps/<acct>/<app>/relay.json, CAS-guarded like store.json, is the source of truth for the app: {enabled, incarnation, regions, op, phase, issued_digest, issued_suffixes, cutovers, enabled_at, disabled_at}. issued_digest is the sha256 of the sealed envelope that was delivered, which is what Record.IssuedDigest records and what reconcile compares; an earlier draft of this spec called it config_generation and no implementation ever carried that name. issued_suffixes (region to the username suffix last issued) and cutovers (the per-route member acknowledgements) are the other two fields relay.Record has always carried and this listing has always omitted. (discussion #120) op is a counter bumped by every operation; phase is one of config, aggregate, done and records how far the operation got. The incarnation is sixteen hex characters drawn at random at first enable and at each re-enable after a disable, never a counter, so reuse across delete and name reuse is not a question.
• control/relay.json, the authority record the relays read: {apps: [{account, app, incarnation, regions}], written_at}. It is derived, never patched. Publishing it is one fixed sequence: read the aggregate's current etag; list every relay.json; build the aggregate from records whose enabled is true, whatever their phase; write with a CAS on the etag read first; on conflict start the whole sequence again. Because the etag is read before the listing, a disable that lands during the listing makes the write fail rather than restore the app. Followed by a control/serial bump so relays notice within seconds; the bump is a latency hint, not the lease.

Every operation is a walk the reconcile can finish, and the reconcile converges everything. An operation writes its intent to relay.json first (op bumped, phase set), then does the config write, then the aggregate publish, advancing phase after each step with a CAS that carries the op. One writer, enforced. Every relay operation, handler or reconcile, runs only in a console process that holds the relay-writer lease, control/relay-writer.json: a CAS-written record naming the holder with a sixty-second expiry, renewed every twenty seconds, taken over by another process only after it lapses. A process that fails a renewal stops every relay write in flight before its next bucket write and abandons the operation, which the next holder's reconcile finishes. Inside the holder, operations on one app are serialised behind a per-app lock. So two console processes, during a deploy overlap or otherwise, never both write relay config, and a superseded key selection cannot reach config after a migration: the writer that selected it either holds the lease, in which case it is the only writer and no migration happened under it, or lost it and stopped. (codex R3-2, R4-1) On top of that fence, the reconcile does not only finish unfinished records, it converges both derived artefacts from every record, every five minutes and at console start: for each enabled record it derives the expected CARLOS_RELAY value from the record and the active masters and re-applies it if the config bundle holds anything else; for each disabled record it removes the entry if present; and it always rebuilds and publishes the aggregate whether or not any record is unfinished. So a stale config write that lands after a disable is undone within five minutes, an interrupted enable delivers its config within five minutes, an interrupted disable removes it within five minutes, a repeated enable while a previous one is unfinished resumes it rather than skipping the config write, and a deletion that removed the record but died before publishing is repaired by the next unconditional rebuild. (codex R2-2, R3-1, R3-2)

When a disable is complete. The app is out of every relay's authority when the aggregate has been published without it and every armed relay's heartbeat carries a written_at at or after that publish. carlos relay status shows "disable pending on <regions>" until then, and "revoked" after. Bounds: the publish is immediate on the happy path and within five minutes after a crash; each relay reads within sixty seconds of the publish; a relay that cannot read stops serving on its own fifteen-minute lease. (codex R2-1)

State transitions for relay.json: (codex R1-13)

| From | Operation | Incarnation | Keys re-derived | Allocations | |---|---|---|---|---| | none | enable | new | yes | none | | enabled, done | enable (same regions) | kept | no, no config write | untouched | | enabled, not done | enable | kept | resumes the unfinished operation | untouched | | enabled | enable --regions (change) | kept | only for added regions; removed regions' entries dropped | ended in removed regions on next authority read | | enabled | disable | kept in record, app removed from aggregate | entry removed | ended everywhere on next authority read | | disabled | enable | new | yes, all regions | none | | any | app deleted | record removed with the app prefix; the deletion walk publishes the aggregate | | ended on next authority read | | any | rotate --region r | kept | region r only, new kid | untouched; old kid accepted until retirement |

carlos relay enable [--regions r1,r2] (member, API-mode only, on the store model in store.go and internal/console/apistore.go; ruling 2):

1. apiAccountApp gate; unknown and foreign are one byte-identical 404.
2. Refuse if the deployment has no relay regions (CARLOS_RELAY_REGIONS in console config is empty) with a message naming the operator. Refuse if any of the app's routes has no config delivery (a TCP-address route whose bundle does not opt into file delivery), naming the route, because such a route could never adopt a rotated key and would block retirement forever. (codex R5-1)
3. Regions are every region in CARLOS_RELAY_REGIONS, in the console's order, or --regions narrows them keeping that order. There is no placement-first promise: placement names (ie, fleet/x) are not AWS regions and an app may have several, so the order is one fixed list for the app and every browser tries every listed relay. (codex R1-12, R2-11)
4. Apply the transition above to relay.json (CAS, op bumped, phase: config).
5. For each region needing keys, derive from that region's current master and its kid, and write CARLOS_RELAY as one kind:"secret" entry through appconfig.Apply (the writeEnv shape in internal/objstore/walk.go:602). Apply touches the app's config generation, a nonce plus timestamp (internal/appconfig/ generation.go:40); record that generation string in relay.json and advance to phase: aggregate. (codex R2-6, R2-12)
6. Publish control/relay.json, bump control/serial, advance to phase: done.
7. Answer with the record. The CLI prints regions, incarnation, and "config generation <g> written; delivery is reported by carlos relay status", never "delivered".

Config adoption, observed per route. The host's config sync, after it renders a route's env file and does whatever that route's shape calls for, writes an additive acknowledgement control/config-ack/<box>/<route host>.json: the route's shape (unit, socket exec, TCP exec with file delivery), the time, whether a restart happened and succeeded, and the relay key identities it rendered: the username_suffix per region parsed out of the CARLOS_RELAY value it just wrote (account, app, incarnation and kid together), or none. A kid alone would not do: a disable and re-enable between two host syncs changes the incarnation and the key while keeping the kid, and an ack naming only kids would report the old credentials as adopted. (codex R4-2) The ack is keyed on the route, not the app, because an app can have several routes on one box and one can adopt while another does not; and it names identities rather than a generation string, because the app-wide generation object is a change notification that can coalesce or fail after the manifest write (internal/appconfig/generation.go:28) and is not an identity of what was rendered. Nothing reads the ack today but carlos relay status; it is the first per-route statement of "this is what the process has" the platform has, and the plan adds it to internal/appconfig/sync.go as its own small step. (codex R1-14, R2-6, R3-4)

carlos relay status reports, separately: desired state from relay.json with its op and phase; the identities (username suffixes) the console last issued per region; delivery per route, "adopted at <time>, restarted" when a config-ack names exactly the issued identities and a successful restart, "written, awaiting delivery" when no ack names them yet, "file delivered, awaiting carlos relay adopted" for a TCP exec route with file delivery until a member acknowledges it, "adopted by member at <time>" after, "restart failed" when the ack says so, and "hibernating, delivered on wake" when the route is asleep (a route added without delivery after enable shows "no config delivery: relay will not reach this route" and blocks retirement until fixed); disable progress as above; and per region, from control/relay-ack/<box>.json, the heartbeat age, whether the record it holds is current, whether its lease is live, which listeners are up, and which kids it has loaded. A heartbeat older than five minutes is shown as "stale", so a dead relay cannot look current.

carlos relay disable applies the transition (phase: config), removes the CARLOS_RELAY entry through appconfig.Apply, publishes the aggregate, marks done.

Rotation (carlos relay rotate --region r, operator): (codex R1-10, R2-4)

1. Operator runs carlos relay keygen, which prints a new kid and key.
2. Operator adds the line to /etc/carlos/relay-masters on each of the region's boxes with retire_at -. The relay re-reads the file; no restart, no allocation touched. Its next heartbeat lists the new kid.
3. Operator adds the new kid:key to the console secret for the region, which holds every master the console may need, CARLOS_RELAY_MASTERS_<REGION>="<kid>:<key>[,<kid>:<key>]". Adding it issues nothing: which master the console issues from is a separate, unsealed pointer, control/relay-issuer.json {"<region>": "<kid>"}, and every path that derives a key for an app (enable, region change, re-enable, reconcile) reads that pointer and uses the master it names. A staged master is inert until the pointer moves. (codex R3-5)
4. carlos relay rotate --region r refuses until every armed relay in the region has heartbeated the new kid, then moves the issuer pointer to it, re-derives and re-delivers keys for every app enabled in that region as an operation on each record, and prints the earliest permitted retirement time for the old kid: 28 hours from now.
5. carlos relay status --region r shows, per route of every enabled app, which identities its config-ack names and when. Each route's cutover is established one of two ways, by route shape. A unit-backed or socket-exec route: the config sync's ack naming the issued identities, since the platform restarted it. A TCP-address exec route with file delivery: its process belongs to an external supervisor the platform cannot restart, so the cutover is a member acknowledgement, carlos relay adopted --host <route>, given after the member has restarted the process; the console records it in relay.json against that route together with the identities issued at that moment, so an acknowledgement given before a later rotation does not count for it. A route in neither category cannot exist under an enabled relay (enable refuses it), and a route added later without delivery blocks retirement until it is fixed. The old master's retire_at may be set no earlier than max(rotation + 28h, latest cutover + 4h) across every route of every enabled app in the region, and status prints that time once every route has a cutover, or names the routes still without one. Setting it earlier is refused by nothing but the docs, and ends any allocation still on the old kid, which is the force-retirement case §2 excludes from its guarantee. (codex R3-4, R4-3, R5-1)
6. After retire_at the operator removes the line from the boxes and the console secret at leisure.

App deletion removes relay.json with the rest of the app prefix, and the deletion walk publishes the aggregate so the app is gone from authority at once. If the walk dies between the two, the reconcile's unconditional rebuild, which lists records rather than looking for unfinished ones, publishes without the app within five minutes. (codex R3-1)

6. Infrastructure

• infra/modules/region-host/main.tf: a relay variable, default off, that adds ingress for UDP 3478, TCP 3478 and UDP 49152 to 65535 from 0.0.0.0/0, and outputs the VPC CIDR for CARLOS_RELAY_DENY_CIDRS. Flagship turns it on per region.
• DNS: an A record relay.<aws region>.<platform domain> to the region's Elastic IP, next to the existing aws_eip.host resource.
• Each armed box gains /etc/carlos/relay-masters (staged by the operator, like host.env) and CARLOS_RELAY_DENY_CIDRS and optionally CARLOS_RELAY_PUBLIC_IP in host.env.
• Console config gains CARLOS_RELAY_REGIONS and one CARLOS_RELAY_MASTERS_<REGION> secret per region.
• Egress cost: every forwarded byte leaves the box once. The unit-economics spec gets a line for relay egress per region, and the two meters in §3 are what fill it.

7. Testing

• internal/relay: derivation and credential against the §2 vector; the AuthHandler against a fixture authority record and masters file covering every refusal reason in §4, including a valid credential for app B presented from a typed source bound to app A, and a UDP and a TCP client with the same address string holding separate bindings; the permission handler against each denied range and the allowed EIP; the relay socket dropping a packet to the EIP on a non-allocated port; limits against a fake clock, including an account spreading allocations across apps, an allocation ended by the byte budget, and a token bucket dropping a burst; the meter against wrapped in-memory sockets, checking that rejected peer ingress is not counted as in and that both forwarding legs are counted; the hourly object surviving a restart, including a restart across an hour boundary that rebuilds the rolling window from both objects; a wrapper's close removing only its own binding while a replacement binding for the same typed source survives; shedding triggered by a fake pressure reading.
• End to end, on loopback with a fixture master: a pion TURN client allocates, refreshes, and keeps relaying until the credential's expiry, at which point traffic stops with no further client request; the same after a disable written to the fixture authority record; two clients on one relay reaching each other's relay candidates; a rotation, by rewriting the masters file, during which an allocation on the old kid carries traffic uninterrupted, then a retire_at ending it; a lease lapse ending allocations and a successful read restoring service.
• internal/host: the unit is embedded, written and started by the bootstrap plan when the staging root carries a masters file, and not started when it does not; the adopt restart path names the relay; the config sync writes a config-ack.
• internal/console: the doors through rawReq and a relayURL fixture: the byte-identical 404, the no-regions refusal, each row of the transition table, the config entry written and removed, the aggregate after concurrent enable and disable of two apps with the etag-first ordering, an enable interrupted after phase: config finished by the reconcile, a superseded writer refused by its CAS, a stale config write landing after a disable and undone by the next reconcile, a deletion that removed the record without publishing repaired by the next rebuild, an enable during a staged-but-unmoved rotation issuing from the old kid, a process that lost the writer lease making no further relay write, a delayed old-key publisher refused after a migration, a disable and re-enable between two host syncs not reported as adopted by the old ack, and rotate refusing until the fixture heartbeats list the kid; the config sync writing a per-route ack naming the identities it rendered; the retirement time computed with a manual cutover after hour 28; enable refused for an app with a route lacking config delivery; a carlos relay adopted given before a later rotation not counting as that rotation's cutover; an externally supervised TCP route with file delivery blocking retirement until acknowledged.
• docs/site/cli.md gains ### carlos relay with a fenced example, or the docsite gate fails.
• Flagship gate, as amended in the roadmap: a Meet call between two symmetric-NAT networks connecting through the Ireland relay; both browsers forced to relay and connecting through it on the real EIP, and a probe to a non-allocated port on the EIP dropped; a credential minted for one app refused for another, including on an existing allocation's tuple; a credential from before a disable refused after re-enable and its allocation ended; traffic stopping at a credential's expiry with the browser doing nothing; a relay with a lease older than fifteen minutes refusing new allocations and ending carried ones; a permission to a private address refused; the relay meter object showing the call's bytes on both legs; a rotation during a call with the call unaffected; the relay shedding allocations under host memory pressure caused by a tenant, with the edge still serving.

8. Not built

• TURN over TLS (5349 or 443) (ruling 4). Certificate issuance for a non-HTTP listener needs the relay hostname registered with the deployment's ACME issuer, which is elected and may be another box (internal/edge/certsync.go), and its warmer excludes such names (internal/edge/certwarm.go); a listener that cannot start until a certificate exists cannot trigger issuance by being visited. That is its own small design, to be written when a network that needs it is reported. 443 additionally needs a second address per box. The roadmap's stage 4a text is amended to match. (codex R1-15, R2-10)
• An inter-region byte meter (ruling 5). Classifying a peer address by AWS region needs the published IP ranges on the box and a rule for non-AWS peers; version one meters what is forwarded in each direction, which is what the bill is made of. Filed for stage 5 with the usage view; the roadmap's meters and gate are amended. (codex R1-18, R2-10)
• An issuance API: apps mint offline from delivered keys.
• Client-aware relay selection: the list is in console order; ICE picks.
• Commercial quotas and an app-page usage view: the meters exist, the page is stage 5.
• An SFU: stage 4b, after this ships.
• Any change to how apps signal: SSE plus POST stays Meet's.

9. Rulings

Paul ruled on each of these on 2026-09-07. (codex R2-10)

1. (ruling) Maximum credential life is four hours. The roadmap's one-hour figure is amended.
2. (ruling) Enable is member-level; the operator's consent is the region list and the limits.
3. (ruling) The console holds every region's master as sealed config; the exposure is stated in §4 and the operator docs; the relay-derives-and-writes design is filed as the tightening.
4. (ruling) No TURN over TLS in version one; the roadmap's stage 4a ingress text is amended.
5. (ruling) Two meters in version one, in and out; the inter-region meter and the usage line move to stage 5; the roadmap's metering text and gate are amended.
6. (ruling) The limits in §3 with their stated defaults, including the token buckets.