| 1 | # Media on CARLOS: a staged roadmap |
| 2 | |
| 3 | **Date:** 2026-09-07, revision 6 |
| 4 | **Status:** rulings taken. Paul ruled on all eleven decisions in §7 on |
| 5 | 2026-09-07; each is marked **(ruling)**. Everything not so marked is a |
| 6 | **(decision)** that a stage spec may overturn cheaply. |
| 7 | |
| 8 | Brainstormed with Claude from discussions #107, #112 and #113 and the |
| 9 | requirements the Meet session sent on 2026-09-07. Reviewed four times |
| 10 | by GPT-6 via Codex, independently of the author: revision 1's 27 |
| 11 | findings are in `CODEX-FINDINGS-MEDIA.md`, revision 2's 14 in |
| 12 | `CODEX-FINDINGS-MEDIA-R2.md`, revision 3's 9 in |
| 13 | `CODEX-FINDINGS-MEDIA-R3.md`, revision 4's 8 in |
| 14 | `CODEX-FINDINGS-MEDIA-R4.md`, and revision 5's 2 in |
| 15 | `CODEX-FINDINGS-MEDIA-R5.md`, all unedited. **Round 5 returned accept: |
| 16 | no P1 or P2 remained, and its two wording corrections are in this |
| 17 | revision.** Each place a finding changed the text is marked |
| 18 | **(codex N)**, **(codex R2-N)**, **(codex R3-N)** or **(codex R4-N)**. |
| 19 | §10 names the two parts declined. |
| 20 | |
| 21 | This is a roadmap, not an implementation spec. Each stage below gets its |
| 22 | own design and plan when it is picked up. The roadmap exists so the first |
| 23 | stage is built with the later ones in view, so nobody builds a stage out |
| 24 | of order, and so the security properties are named before any URL shape |
| 25 | is. |
| 26 | |
| 27 | ## 0. The goal in one sentence |
| 28 | |
| 29 | Any app on CARLOS can serve, transform, stream and relay media at the |
| 30 | performance people expect from a CDN or a media API, with nothing to |
| 31 | install and no third-party service in the path, and every object can be |
| 32 | public, secret or private. |
| 33 | |
| 34 | "No third party" is not taste. Suite ruling 6 forbids egress that is a |
| 35 | property of the product, and a media CDN, an image API, a TURN service |
| 36 | or a hosted SFU that every app dials by default is exactly that. The |
| 37 | platform is the one recipient a team has already chosen, so media |
| 38 | handled by the platform is not egress at all. That distinction is what |
| 39 | makes this a platform feature rather than a per-app settings field, and |
| 40 | the plan keeps that language throughout. |
| 41 | |
| 42 | ## 1. Where things stand, measured on `main` |
| 43 | |
| 44 | Discussion #107 recorded most of this on 2026-08-31. Re-checked today, |
| 45 | and corrected where the reviews found the citations wrong. |
| 46 | |
| 47 | - **Apps already have object storage**, bucket-per-app, credentials |
| 48 | delivered sealed through `config.json` (object-storage spec §5). Reads |
| 49 | and writes go app-to-S3 directly. The spec says at line 27 that a |
| 50 | byte-proxy in the data path is "permanently out of scope". Reads must |
| 51 | cross that line for a CDN to exist. Writes never will. |
| 52 | - **Sealed config is decryptable on every host.** The deployment's |
| 53 | decryption key sits in `host.env` on every box (config-secrets spec |
| 54 | §4) and `internal/appconfig/materialize.go:30` renders secrets to env |
| 55 | at adopt time. A compromised host therefore already exposes every |
| 56 | app's store keys and secrets, not only its own residents'. Nothing in |
| 57 | this plan makes that better or worse, and no claim below may pretend |
| 58 | a per-region secret changes it. **(codex R2-2)** |
| 59 | - **The store already scans.** `internal/objstore` has a ClamAV scanner, |
| 60 | a resumable sweep that tags every object with a verdict, a scan queue |
| 61 | of one object per item under the app's own prefix, and, when a store's |
| 62 | scan block has enforcement on, a `Deny` on `GetObject` for objects |
| 63 | tagged infected, written into the store user's policy |
| 64 | (`internal/objstore/aws.go:211`, `objstore.go:82`). There is no upload |
| 65 | hook by design; the sweep counts what it has not seen. Any new reader |
| 66 | of the bucket must carry the same deny whenever the store user does, |
| 67 | or it is a bypass. **(codex 4, 26, R4-8)** |
| 68 | - **The edge cannot read a store bucket.** The region host role is scoped |
| 69 | to the deployment bucket and its own region's data bucket |
| 70 | (`infra/modules/region-host/main.tf:106`, residency argument in the |
| 71 | header at line 6). A new grant is most of the serving design. |
| 72 | - **The shared response cache is unsafe for anything authorised.** The |
| 73 | key is host, method, path and query; a hit returns before any handler |
| 74 | runs (`internal/edge/edge.go:1300`). `kind:status` already opts out |
| 75 | and caches for itself because an unbounded keyspace in a |
| 76 | process-global LRU lets one tenant evict every other. Media has that |
| 77 | problem and an authorisation one. |
| 78 | - **Purge is local to one box.** The purge listener is a unix socket on |
| 79 | the host it runs on, and the `Carlos-Purge` response header acts on |
| 80 | the edge that saw the response. Nothing distributes an invalidation |
| 81 | across regions today. **(codex 9)** |
| 82 | - **An edge keeps serving from last-converged state through a control |
| 83 | bucket outage** (multi-region ingress spec, line 123). Any authority |
| 84 | media adds to the route record inherits that: a disconnected edge |
| 85 | will honour a revoked key until told otherwise. **(codex R2-6)** |
| 86 | - **The tick path is local.** `activatorDeliverer` reaches an instance |
| 87 | through the activator and its unix socket on the same box, and refuses |
| 88 | routes without one (`internal/host/schedule_wiring.go:220`). Once-timers |
| 89 | are filed on an instance's control socket, bound to that host |
| 90 | (`internal/schedule/control.go:28`). Neither crosses a box. |
| 91 | **(codex R2-4, R2-8)** |
| 92 | - **Range and conditional handling come from `http.ServeFile`** for |
| 93 | `kind:static`, and only there. A streaming S3 handler has to build its |
| 94 | own HTTP mapping, and S3 `GetObject` serves one range per request. |
| 95 | **(codex 7)** |
| 96 | - **The edge is identity-free by design** (`internal/edge/edge.go:1`). |
| 97 | It never reads a cookie. Suite identity is a home-minted Ed25519 |
| 98 | assertion, audience-bound, single-use, two minutes long, delivered by |
| 99 | auto-POST; it cannot be replayed per image. Suite doctrine ruling 3 |
| 100 | says authorisation is never cached at any scope, and leak item 11 |
| 101 | names a cacheable personalised response at the platform edge as a |
| 102 | cross-person leak. |
| 103 | - **The doctrine's visibilities are rungs plus an orthogonal link.** |
| 104 | Ruling 15: an item is `private`, `granted` or `team`, and may |
| 105 | additionally carry a read-only link. `team` is not public. One item |
| 106 | can be reachable by its roster and by a link at the same time, so the |
| 107 | storage model must let both doors reach one object. **(codex 11, R2-3)** |
| 108 | - **Doctrine §8 says the recipient of any egress is the app's choice, |
| 109 | never the team's**, disclosed in the README's first paragraph and on |
| 110 | every artefact produced. A team-pasted relay URL is not admissible. |
| 111 | **(codex 12, R2-12)** |
| 112 | - **Members already control placement.** `carlos apps place` and |
| 113 | `carlos instances create --region` exist, and the instances API |
| 114 | reports which boxes and regions serve a host |
| 115 | (`internal/console/instancesplacement.go:21`). What does not exist is |
| 116 | a runtime signal an app can read about its host's transport |
| 117 | capabilities. **(codex R2-13)** |
| 118 | - **Wildcard certificates cover exactly two labels** under the apps |
| 119 | domain (`internal/edge/wildcard.go:186`), and app names are |
| 120 | `^[a-z][a-z0-9-]{0,31}$` (`internal/release/release.go:239`), so a |
| 121 | one-label media name for any app of up to 26 characters is also a |
| 122 | valid app name, and an app name may end in a hyphen, which a DNS |
| 123 | label may not (`internal/provision/provision.go:223`). Hostnames in |
| 124 | the account namespace are also claimed by the instances door |
| 125 | (`internal/console/apiinstances.go:571`), which lets an app take any |
| 126 | unclaimed label, not only its own name. **(codex 24, R2-10, R4-4, R4-5)** |
| 127 | - **The fleet is small ARM.** Four production regions on Graviton |
| 128 | `t4g.small`, one `t4g.medium`, burstable at a 20% per-vCPU baseline |
| 129 | with the same two vCPUs on both sizes; the medium adds memory, not |
| 130 | CPU. No GPU, no ffmpeg, no package installs at bootstrap. Remote hosts |
| 131 | are Raspberry Pis. **(codex 20)** |
| 132 | - **The Go tree is pure and CI keeps it so** (`.amadan/ci.d/20-cross@5` |
| 133 | sets `CGO_ENABLED=0` deliberately). That rule exists so the CLI |
| 134 | cross-compiles for the Macs people run it on. It constrains the tree, |
| 135 | not the box, and §3 sets the rule for native helpers. |
| 136 | - **There is no generic media job queue.** The scan queue is the one |
| 137 | bucket-backed queue in the tree and its header explains where jobs |
| 138 | must live so app deletion finds them. Stage 3 reuses that shape rather |
| 139 | than inventing a second one. **(codex 19)** |
| 140 | - **UDP is a stated non-goal of the capabilities spec** (line 931). A |
| 141 | relay is therefore a platform service next to the edge, not an app |
| 142 | capability. |
| 143 | - **A tunnel-backed host cannot carry `Connection: Upgrade`** |
| 144 | (remote-host-pi design, line 63). |
| 145 | - **Real demand exists.** Docs ships an "insert image" menu item disabled. |
| 146 | Meet is being designed now and needs STUN for calls to connect across |
| 147 | most home and mobile networks. Woodstar runs its own media service as |
| 148 | a tenant app, the shape this plan makes unnecessary. |
| 149 | - **Apps already hold immutable keys.** rastrillo's blob store is |
| 150 | content-addressed and presigns locally. A content-addressed key can be |
| 151 | cached forever, and a sealed (E2EE) blob is opaque bytes the platform |
| 152 | can serve but never transform. |
| 153 | |
| 154 | ## 2. Public, secret, private |
| 155 | |
| 156 | Three words, three delivery doors, two storage namespaces. The doors are |
| 157 | how bytes reach a browser, not a new authorisation model: the app still |
| 158 | decides who may see what, from its own grants, and the doctrine's rungs |
| 159 | map onto the doors as the last row of the table says. |
| 160 | |
| 161 | | Door | Who can fetch | What the edge checks | Bytes cached | Reads from | |
| 162 | |---|---|---|---|---| |
| 163 | | **public** `/p/` | anyone with the URL | nothing | by object key, long-lived, in every allowed region | `public/` only | |
| 164 | | **secret** `/s/<cap>/` | anyone holding a valid capability | the capability's signature, scope, namespace and expiry | by object key, once per object, in every allowed region | `protected/` | |
| 165 | | **private** `/_carlos/media/` on the app's origin | a person the app admits, now | asks the app on every request | by object key, on the app's home edge, after the app says yes | `protected/` | |
| 166 | | *doctrine* | `private`, `granted` and `team` rungs deliver through the **private** door; a read-only link on any of them is a **secret** capability naming the same object; nothing is **public** unless the app's own rule makes an item world-readable | | | | |
| 167 | |
| 168 | **Two namespaces, not three.** The media prefix has `public/` and |
| 169 | `protected/`. The public door reads `public/` and nothing else, so |
| 170 | rewriting a URL from one door to another cannot reach a protected |
| 171 | object: `/p/x` reads `public/x`, which does not exist for a protected |
| 172 | key. The secret and private doors both read `protected/`, which is what |
| 173 | lets one object be reachable by its roster through the app's check and |
| 174 | by a link through a capability at the same time, as the doctrine's |
| 175 | orthogonal link requires. Derivatives and renditions live under the same |
| 176 | namespace as their source. **(codex 1, R2-3)** |
| 177 | |
| 178 | **Changing visibility is a lifecycle, not a copy.** Between the two |
| 179 | namespaces it is: copy to the new namespace, delete the old key and its |
| 180 | derivatives, append a purge covering them, fence any transform or job in |
| 181 | flight against the old key (§4, stage 1, "publication fence"), and |
| 182 | report completion only when every box that could hold a copy, in any |
| 183 | region the app has ever been allowed, has either acknowledged the purge |
| 184 | or let its serving lease expire (§4, stage 1, "purge"). Bytes a browser |
| 185 | already holds cannot be withdrawn, and the docs say so. **(codex R3-1)** Within `protected/`, secret and private are not a move at all: |
| 186 | the app mints a capability or stops minting them, and an existing |
| 187 | capability dies at its expiry or when its key id is revoked. Stage 1's |
| 188 | gate tests public-to-protected and link-to-roster transitions on cold |
| 189 | and warm caches. **(codex R2-1)** |
| 190 | |
| 191 | **Public** is a plain URL on the media host. Content-addressed keys get |
| 192 | immutable caching. Mutable keys get whatever the app says with |
| 193 | `Cache-Control` on the object, defaulting to a short lifetime, and the |
| 194 | app can purge. |
| 195 | |
| 196 | **Secret** is the doctrine's read-only link and what S3 presign, Mux and |
| 197 | Cloudinary do: a bearer capability, signed by the app, naming an object |
| 198 | or a prefix in `protected/`, expiring. Anyone holding it can fetch; the |
| 199 | bytes are the same for every holder, so a popular file is cached once |
| 200 | and revocation lags by the expiry. **The capability is a path segment, |
| 201 | not a query string**, so a playlist's relative references to its |
| 202 | segments resolve under the same capability and every child request is |
| 203 | authorised without rewriting. **(codex 6)** What the signature covers is |
| 204 | defined in stage 1, not left to the implementer. **(codex 5)** |
| 205 | |
| 206 | **Private** is identity-bound, and the only way to honour ruling 3 is to |
| 207 | not decide at the edge. On every request the edge asks the app: "may |
| 208 | this request read this key?" The app answers from its own grants and |
| 209 | session, in a millisecond, and the edge serves from its byte cache or |
| 210 | fetches the object. This is the shape of nginx `auth_request`, with |
| 211 | these properties the reviews forced into the open: |
| 212 | |
| 213 | - **Private media is served on the app's own origin**, under a reserved |
| 214 | path the edge intercepts ahead of the shared-cache lookup and ahead of |
| 215 | any proxy branch, because that is the only place the app's session |
| 216 | cookie reaches. A separate media host never sees a host-only cookie, a |
| 217 | custom domain cannot set one for the apps domain, and cross-site |
| 218 | cookies cannot be relied on across supported browsers and privacy |
| 219 | settings. The cost is user content on the app's origin, and the |
| 220 | mitigations in stage 1 (nosniff, a CSP sandbox on every response, |
| 221 | attachment disposition for anything not image, audio, video, font or |
| 222 | PDF, and no inline SVG or HTML ever) are what make that acceptable. |
| 223 | **(codex 2, R2-14)** |
| 224 | - **The check travels over the route's own backend transport**, never |
| 225 | through a cache: the activator and unix socket for an instance on the |
| 226 | same box, the route's TCP dial for a service route, the tunnel for a |
| 227 | remote host. A private request for an app whose instance lives on |
| 228 | another box reaches the home edge by the existing backhaul, as every |
| 229 | other request for that route does, and the home edge's media cache is |
| 230 | the one that serves it. Private bytes are therefore cached on the |
| 231 | app's home edge only; public and secret bytes are cached in every |
| 232 | allowed region. The check fails closed on timeout, redirect, or |
| 233 | anything but a 200, and runs before `HEAD`, before a 304, before a |
| 234 | range is answered and before a cache hit is admitted. **(codex 3, R2-4)** |
| 235 | - **Private responses carry `Cache-Control: private, no-store`.** Secret |
| 236 | responses carry `Cache-Control: private, max-age=<remaining life>, |
| 237 | must-revalidate`, generated per request from the capability presented |
| 238 | and overriding any directive on the object, so no HTTP cache may serve |
| 239 | a stale copy past the capability's expiry. The edge's own byte cache is |
| 240 | internal and never a substitute for either. **(codex 3, R2-5)** |
| 241 | |
| 242 | The costs: the app's instance is in the hot path for private media, one |
| 243 | small request per fetch, and a hibernating instance wakes for it; the |
| 244 | app must expose the check endpoint, which rastrillo provides once for |
| 245 | every app; the edge stays identity-free, forwarding a cookie it never |
| 246 | reads. The edge never caches the app's answer, for any tenant, under |
| 247 | any knob. Ruling 1 says never; a proposal to cache a positive answer |
| 248 | comes back to §7 for a changed ruling, not to a stage spec as an option. |
| 249 | **(ruling 1 in §7; codex R3-7)** |
| 250 | |
| 251 | Media publication is never evidence for anything else. A sibling app |
| 252 | seeing an object under `public/` learns nothing about whether it may |
| 253 | ingest the source item; custody keys on the app's grants, as the |
| 254 | doctrine says. **(codex 11)** |
| 255 | |
| 256 | ## 3. Native helpers and media services: the rule |
| 257 | |
| 258 | Images and video are the two places where pure Go is not fast enough on |
| 259 | a `t4g.small`. The answer is not a cgo dependency. Two classes, with |
| 260 | different rules. **(codex 16, 22; ruling 10)** |
| 261 | |
| 262 | **Codec helpers** transform bytes and nothing else. |
| 263 | |
| 264 | - The Go tree stays `CGO_ENABLED=0`. A helper is a separate program in |
| 265 | whatever language wins the benchmark: Rust, C, C++, Go with wazero, |
| 266 | or an existing tool such as `vips` or `ffmpeg`. Every candidate, |
| 267 | including a Go one, runs as its own process. Nothing decodes inside |
| 268 | the edge. |
| 269 | - A helper is built and shipped as its own artifact, linux/arm64 only, |
| 270 | with an authenticated manifest naming its version, hash and the |
| 271 | libraries and codecs it carries. A box with the role that needs it |
| 272 | fetches it at bootstrap and at each promotion, and can roll back the |
| 273 | way it rolls back the platform binary. It never enters the CLI build. |
| 274 | - A platform unit spawns it under a hardened systemd scope with no |
| 275 | network, a read-only filesystem apart from scratch, and memory, |
| 276 | CPU, wall-time, scratch-space and process limits. It talks over |
| 277 | stdin/stdout or a unix socket, and the Go side owns the protocol, so |
| 278 | `vips` today and a Rust helper tomorrow answer the same calls. |
| 279 | |
| 280 | **Media services** are network-facing long-running units: the relay in |
| 281 | stage 4a, LiveKit in 4b, a live packager. They are platform units like |
| 282 | the edge, with their own units, ports and hardening, and the no-network |
| 283 | rule above does not apply to them. |
| 284 | |
| 285 | This is the answer to "could we build some dependencies in Rust or C for |
| 286 | throughput": yes, as codec helpers, and the spikes in stages 2 and 3 |
| 287 | decide whether the first ones are ours or borrowed. |
| 288 | |
| 289 | ## 4. The stages and how they depend on each other |
| 290 | |
| 291 | Six pieces of work. The graph replaces a single sequence; anything with |
| 292 | no arrow into it can start now. Stage 1 carries two things later stages |
| 293 | share, the derivative-write grant and the publication fence, so that |
| 294 | whichever of 2 or 3 ships first finds them ready. **(codex 27, R2-11; |
| 295 | ruling 11)** |
| 296 | |
| 297 | stage 1 serving class ─┬─▶ stage 2 image transforms |
| 298 | ├─▶ stage 3 audio/video on demand ─▶ recording (in 4b) |
| 299 | └─▶ stage 5 app-facing helpers |
| 300 | stage 4a STUN + TURN ────▶ stage 4b LiveKit SFU, live streaming |
| 301 | |
| 302 | | Stage | Delivers | Needs from the platform | Size | |
| 303 | |---|---|---|---| |
| 304 | | 1 | `kind:media`: serving class with the three doors | edge read grant with the scan deny, signed capabilities, streaming handler, isolated cache, distributed purge, hostname claim, derivative-write grant, publication fence, CLI | one spec, one plan | |
| 305 | | 2 | image transforms behind preset URLs | a codec helper, limits, derivative identity | one spec, one spike first | |
| 306 | | 3 | audio and video on demand: single rendition, then ladders, posters, waveforms | a media worker, a job queue under the app prefix, ffmpeg helper | one spec, one infra change | |
| 307 | | 4a | STUN and TURN | UDP ingress, a relay unit, per-app per-region credentials as config, limits | one spec, one infra change; **starts now** | |
| 308 | | 4b | LiveKit as a media service, recording, live streaming | a `c7g.large`, LiveKit's server unit, a live serving extension | one spec | |
| 309 | | 5 | uploads, rastrillo helpers, accounting, residency disclosure | small pieces spread across 1 to 4 | folded into each stage's plan | |
| 310 | |
| 311 | ### Stage 1: `kind:media`, the serving class |
| 312 | |
| 313 | A new serving class, the fourth app-mintable route kind beside |
| 314 | `instance`, `service` and `static` (the edge also dispatches the |
| 315 | platform's own `status`), that serves an app's store bucket through the |
| 316 | edge, cached, with the three doors of §2. This is piece A of #107 with |
| 317 | its open questions answered. **(codex R4-8)** |
| 318 | |
| 319 | **What an app sees.** `carlos media enable` writes an additive `media` |
| 320 | block into `store.json` (prefix, allowed regions, key ids, hostname) |
| 321 | and mints `media-<app>.<sqid>.<apps domain>`, one label so the account |
| 322 | wildcard already covers it. **(codex 24)** Because `media-<app>` can |
| 323 | also be a valid app name, the hostname is a **media-host claim** of its |
| 324 | own, owned by the source app and recorded in the account, not an app |
| 325 | row. It is one entry in the account's hostname namespace, and every |
| 326 | door that acquires a hostname there consults it: new app names, |
| 327 | renames, and the instances door that lets an app take an unclaimed |
| 328 | label. Enable is refused, with the error saying so, when `media-<x>` |
| 329 | is already an app's name or an existing route or instance host, and |
| 330 | those doors refuse a label a media claim holds; the acquisitions are |
| 331 | serialised on the claim so two racing requests cannot both win. An app |
| 332 | whose name ends in a hyphen cannot form a DNS label with `media-` |
| 333 | prefixed and is refused enable before any claim or grant is written, |
| 334 | with the error saying to rename. The generated label can be up to 38 |
| 335 | characters, inside a DNS label; the shared `release.ValidApp` grammar |
| 336 | is not narrowed, because existing manifests and channel pointers parse |
| 337 | through it. **(codex R2-10, R3-4, R4-4, R4-5)** The app writes objects |
| 338 | under `<prefix>/public/` or |
| 339 | `<prefix>/protected/` with the store credentials it already has, and |
| 340 | forms URLs: |
| 341 | |
| 342 | https://media-docs.abc.oncarlos.com/p/<key> public |
| 343 | https://media-docs.abc.oncarlos.com/s/<capability>/<key or rest> secret |
| 344 | https://docs.abc.oncarlos.com/_carlos/media/<key> private, on the app's origin |
| 345 | |
| 346 | The capability is one URL-safe base64 path segment encoding the signed |
| 347 | claims and the signature. Everything after it is resolved against the |
| 348 | scope the claims name, so `.../s/<cap>/videos/abc/index.m3u8` and the |
| 349 | `720p/seg001.m4s` it references are both authorised by the same |
| 350 | capability. The cache key strips the capability and keys on the object. |
| 351 | |
| 352 | **What a capability signs.** Domain separator (`carlos-media-v1`), the |
| 353 | app's store identity, the key id, the namespace (`protected/`), the |
| 354 | scope as either an exact key or a prefix that is matched only at a `/` |
| 355 | boundary after percent-decoding and normalising the path with no `..`, |
| 356 | the expiry, and in stage 2 the transform presets allowed. Maximum life |
| 357 | is seven days **(ruling 3)**. Rotation is a second key id with an |
| 358 | overlap; emergency revocation removes a key id from the route record. |
| 359 | The capability never appears in edge logs, and every media response |
| 360 | carries `Referrer-Policy: no-referrer`. **(codex 5)** |
| 361 | |
| 362 | **How the edge is granted read.** The provisioner gains a bucket-policy |
| 363 | operation. At enable it attaches statements on that one bucket letting |
| 364 | each allowed region's host role `GetObject` on both namespaces and |
| 365 | `PutObject` and `DeleteObject` under `derived/` within them, **with the |
| 366 | same `s3:ExistingObjectTag` deny the store user carries**, so an object |
| 367 | the scanner has tagged infected is unreadable through media exactly as |
| 368 | it is through the app's credentials. Enable and disable are a |
| 369 | reconciled walk like the store grant, with partial completion, store |
| 370 | revocation and route replacement handled, and disable also retires |
| 371 | cached bytes. Adopted and non-AWS stores are refused in version one. |
| 372 | **(codex 4, 25, R2-11; ruling 2)** |
| 373 | |
| 374 | **Signing keys.** An Ed25519 pair per app, minted at enable. The public |
| 375 | half goes in the route record, which every edge already syncs. The |
| 376 | private half reaches the app as `CARLOS_MEDIA_SIGNING_KEY` through the |
| 377 | sealed `config.json` path that already carries store keys. The edge |
| 378 | holds no secret. **(ruling 3)** |
| 379 | |
| 380 | **The handler.** `mediaHandlerFor` streams: one S3 `GetObject` per |
| 381 | request with a single normalised range, the body copied through, never |
| 382 | buffered. It defines its own HTTP mapping rather than borrowing |
| 383 | `http.ServeFile`'s: single-range only, multi-range answered with the |
| 384 | whole object, `If-Range` against the ETag, `If-None-Match` before |
| 385 | bytes, `HEAD` answered from metadata, 416 on unsatisfiable. Every cache |
| 386 | entry is bound to the object's ETag and published only after a complete |
| 387 | fill, so a range from one version is never stitched to metadata from |
| 388 | another. **(codex 7)** Response headers set on every media response, |
| 389 | whatever the object carries: `X-Content-Type-Options: nosniff`, |
| 390 | `Content-Security-Policy: sandbox`, `Referrer-Policy: no-referrer`, a |
| 391 | `Content-Type` from the object's metadata with a small allow-list |
| 392 | fallback, `Content-Disposition: attachment` for anything that is not an |
| 393 | image, audio, video, font or PDF, the door's `Cache-Control` from §2, |
| 394 | and a real CORS policy for the app's own hostnames with `Vary: Origin`, |
| 395 | because `Cross-Origin-Resource-Policy` is not CORS and an HLS player in |
| 396 | JavaScript needs the latter. **(codex 23)** |
| 397 | |
| 398 | **Its own cache.** `kind:media` is exempt from the shared LRU in both |
| 399 | directions and keeps a disk cache under `/data/media-cache/` that stores |
| 400 | objects as fixed-size blocks (1 MiB), so overlapping ranges share |
| 401 | storage. Isolation is per box: a total disk budget with filesystem |
| 402 | headroom, a fair share per route within it, eviction that takes first |
| 403 | from whichever route is furthest over its share, an account-level cap |
| 404 | across routes, and a bound on concurrent fills. The fleet-wide figure |
| 405 | is accounting, not a limit any one box can enforce. **(codex 8)** |
| 406 | |
| 407 | **Purge, distributed, with a serving gate.** A purge is a record |
| 408 | appended to a log in the platform bucket, sequence-numbered per app. |
| 409 | Every edge tails the log and applies it; entries carry the generation |
| 410 | they were filled under, so an in-flight fill cannot repopulate an entry |
| 411 | a purge already covers. A returning edge reconciles its purge cursor |
| 412 | and its route authority **before** admitting any hit from the |
| 413 | persistent cache, and if the log history it needs is gone it drops its |
| 414 | media cache rather than guess. **Serving media is a lease.** An edge |
| 415 | may serve any media door, public included, only while its media |
| 416 | authority (the route record's media block and its purge cursor) is |
| 417 | younger than fifteen minutes; past that every media door on that edge |
| 418 | fails closed, because a public object that was moved to `protected/`, |
| 419 | a disabled route, a removed region or an infected verdict must not |
| 420 | outlive an outage on a box that clients can still reach. Retirement of |
| 421 | a key, a route or a region is complete when every box that could hold |
| 422 | a copy has acknowledged it or its lease has expired, and the API and |
| 423 | CLI report completion per box on exactly that condition; the local |
| 424 | `Carlos-Purge` header and socket stay as fast paths and never count as |
| 425 | completion. A scan verdict turning infected files a purge for the key |
| 426 | and its derivatives. **(codex 4, 9, R2-6, R3-1)** |
| 427 | |
| 428 | **The publication fence.** Anything that writes derived output, a stage |
| 429 | 2 transform or a stage 3 job, writes to an attempt-specific key under |
| 430 | `derived/_attempts/` and has no authority to publish it. **The console |
| 431 | is the publisher.** A commit is a console door, `POST |
| 432 | /a/<sqid>/apps/<app>/media/publish`, carrying the attempt, the lease |
| 433 | and the source ETag; the console checks the lease is current in its |
| 434 | registry, the source's purge generation and the app's incarnation match |
| 435 | what it holds, and only then writes the per-source manifest itself. |
| 436 | That is one serialiser with the state to refuse, which a conditional |
| 437 | write against the manifest is not: an expired worker with no successor |
| 438 | would find the manifest's ETag unchanged and its `If-Match` would |
| 439 | succeed. The serving path reads derived output only through a committed |
| 440 | manifest, **with no exception**: a synchronous transform's first |
| 441 | requester waits for the publish door's answer and receives the bytes |
| 442 | only if it was 200, because an attempt that stalled past its lease or |
| 443 | across an infected verdict must not reach even the request that caused |
| 444 | it. One console round trip on a first miss is the cost. Purge, an |
| 445 | infected verdict, cancellation and app deletion each bump the |
| 446 | generation in the registry, so a late commit is refused by the door. |
| 447 | Attempt keys are swept. This lives in stage 1 because both later stages |
| 448 | need it and neither may reinvent it. **(codex 17, R2-7, R3-2, R4-3)** |
| 449 | |
| 450 | **Residency, enforced.** The `media` block names the regions allowed to |
| 451 | serve and cache the app's media. The bucket policy grants only those |
| 452 | regions' host roles, the media host's DNS records exist only in those |
| 453 | regions, and the handler on any other edge answers 421 with the right |
| 454 | host as a backstop, so a client addressing a foreign edge directly |
| 455 | cannot fill a cache there. Default at enable: the regions the app's |
| 456 | routes are already placed in, shown back by `carlos media status` and |
| 457 | on the media docs page. **(codex 10, R2-9; ruling 4)** |
| 458 | |
| 459 | **Store spec amendment.** Line 27 of the object-storage spec gains one |
| 460 | sentence: reads may cross the data plane through `kind:media`; writes |
| 461 | never do. Uploads stay app-to-S3 on the app's credentials. |
| 462 | |
| 463 | **What stage 1 must prove** (§9) is the attacks in §2 and this section |
| 464 | failing, not only playback succeeding. **(codex 27)** |
| 465 | |
| 466 | **Not in stage 1:** transforms, transcoding, uploads through the edge, |
| 467 | per-viewer analytics, signed cookies, bandwidth quotas, dashboards |
| 468 | beyond a bytes-served counter, multipart upload helpers. |
| 469 | |
| 470 | ### Stage 2: image transforms |
| 471 | |
| 472 | `/p/<key>?t=thumb`, `/s/<cap>/<key>?t=thumb` and the private form, where |
| 473 | `thumb` is a preset the app declared. A transform is synchronous, pure |
| 474 | CPU in a codec helper, and its result is written back to the app's |
| 475 | bucket through the publication fence so the second region never redoes |
| 476 | the first's work. |
| 477 | |
| 478 | **Abuse surface first.** A public transform URL is CPU amplification on |
| 479 | the box that terminates TLS for every tenant. From the first commit, |
| 480 | transform parameters are a **named preset** the app declared, or for |
| 481 | secret objects the presets the capability names, or for private objects |
| 482 | the presets the app's check endpoint approves. Free-form parameters are |
| 483 | refused everywhere. **(codex 5)** On top of that: single-flight per |
| 484 | derived identity, a per-box concurrency cap with per-tenant fair |
| 485 | admission so one tenant cannot hold every slot, a queue that turns into |
| 486 | 503 after a short wait, a source pixel-count cap, output dimension caps, |
| 487 | and a per-route derivative byte budget. **(codex 17)** |
| 488 | |
| 489 | **Derivative identity.** A derivative is named by the source key, the |
| 490 | source ETag, the normalised preset, the resolved output format and the |
| 491 | helper version, and committed through the fence. `f_auto` picks AVIF, |
| 492 | WebP or JPEG from `Accept`, and the response says `Vary: Accept`. |
| 493 | **(codex 17)** |
| 494 | |
| 495 | **The engine, decided by a spike.** Three candidates, all run as separate |
| 496 | processes under the §3 rule. First, a Go helper embedding wazero-hosted |
| 497 | `libwebp`, `libavif` and `jpegli` (the `gen2brain/*` packages), which |
| 498 | keeps the build pure Go end to end. Second, `vips` as a fetched helper. |
| 499 | Third, our own Rust helper using the `image` and `ravif` crates or |
| 500 | linking libvips, persistent over a unix socket. The spike measures a |
| 501 | 4000×3000 JPEG to an 800-wide WebP and AVIF on a `t4g.small` that is |
| 502 | also serving: thumbnail latency, peak memory, sustained throughput |
| 503 | under eight concurrent requests, behaviour on malformed input, and the |
| 504 | serving latency of the box while it runs. Under a second for WebP with |
| 505 | serving latency flat is the bar, and the candidate that clears it with |
| 506 | the least to operate wins. The bar is not a ceiling on the fleet: if |
| 507 | nothing clears it on a `t4g.small`, the choices are a bigger edge box, |
| 508 | a dedicated transform box the fleet shares, or telling apps what to |
| 509 | expect, and the spike's numbers decide which. **(codex 16; ruling 5)** |
| 510 | |
| 511 | **What a transform does.** Resize, crop modes, format, quality, and |
| 512 | always strips metadata. EXIF GPS in a public photo is a privacy leak the |
| 513 | platform should not require every app to remember. Animated GIF and |
| 514 | WebP are passed through untransformed in version one. Sealed blobs are |
| 515 | refused with 415. |
| 516 | |
| 517 | **Derivatives** land in the tenant's bucket under |
| 518 | `<namespace>/derived/<source key>/<identity>.<fmt>`, written by the |
| 519 | region host role under stage 1's `derived/` statement, and count |
| 520 | against the tenant's storage. Purging a source key removes its |
| 521 | derivatives. **(ruling 6)** |
| 522 | |
| 523 | **Not in stage 2:** face detection, smart crop, watermarks, uploads, |
| 524 | SVG rasterisation, anything needing a model. |
| 525 | |
| 526 | ### Stage 3: audio and video on demand |
| 527 | |
| 528 | Stage 1 already plays a well-formed MP4 or MP3 progressively: `Range` |
| 529 | passes through and the browser's `<video>` seeks. For a short clip or a |
| 530 | voice note that is the whole feature and it ships in stage 1. Stage 3 is |
| 531 | for what progressive cannot do, and it has two products the review |
| 532 | insisted be kept apart. **(codex 21)** |
| 533 | |
| 534 | - **A single rendition, fast.** Probe the input; if it is H.264 and AAC |
| 535 | in MP4 with sane keyframe spacing, remux to CMAF and one HLS playlist |
| 536 | in seconds. If it is WebM from a browser recording, re-encode one |
| 537 | rendition at the source's size. This is what a phone upload in Docs |
| 538 | or a recorded call needs first, and it follows stage 1 directly. |
| 539 | - **An adaptive ladder, slower.** Re-encode to three rungs (1080p, 720p, |
| 540 | 360p, capped at the source's height), CMAF, one multivariant playlist. |
| 541 | Remuxing cannot make lower rungs; this always encodes. AV1 and DASH |
| 542 | are one flag away and off. This follows the worker measurements. |
| 543 | |
| 544 | Both products also write posters at three timestamps for video, and a |
| 545 | waveform JSON and duration for audio. Everything is served by stage 1 |
| 546 | under one prefix-scoped capability or one private door, which is why |
| 547 | the capability is a path segment. |
| 548 | |
| 549 | **The media worker.** A platform unit, `carlos-media.service`, on one box |
| 550 | per region that has media-enabled buckets. Jobs live where the scan |
| 551 | queue lives and for the same reason: under the app's own prefix, |
| 552 | `apps/<acct>/<app>/media-jobs/<id>.json`, one object per job, so app |
| 553 | deletion finds them and the name-claim gate sees them. **(codex 19)** |
| 554 | |
| 555 | - **Submission** is a console door, `POST |
| 556 | /a/<sqid>/apps/<app>/media/jobs`, reached from the app through a |
| 557 | rastrillo helper with the app's route-scoped credential, and from |
| 558 | `carlos media transcode <key>`. The door derives the source and |
| 559 | destination scope itself; the app names a key, never a path. Delivered |
| 560 | store credentials never gain the right to write job records. |
| 561 | **(codex 18)** |
| 562 | - **Eligibility, decided at submission.** The processing region is the |
| 563 | bucket's region, and it must be in the app's allowed list and have a |
| 564 | worker. Version one runs one worker, in Ireland, so a job for a bucket |
| 565 | elsewhere is refused at the door with an error naming the reason, not |
| 566 | queued to wait for a worker that does not exist. **(codex R2-9)** |
| 567 | - **Authority.** The worker reads the source through the media bucket |
| 568 | policy and writes renditions under `derived/` through stage 1's |
| 569 | statement, on buckets in its own region only. |
| 570 | - **Leases and the fence.** A worker takes a lease with an expiry; its |
| 571 | output goes to attempt keys and is committed through the stage 1 |
| 572 | publication fence, which refuses a commit from an expired lease, a |
| 573 | purged source or a deleted app. Bounded retries, cancellation, a |
| 574 | terminal failure state. **(codex 18, R2-7)** |
| 575 | - **Completion** is a result record the app polls in version one. |
| 576 | Callbacks come later as a durable completion outbox that the |
| 577 | destination host converges into a local once-timer, because nothing |
| 578 | files a timer across boxes today. **(codex R2-8)** |
| 579 | - **Limits from the first job**: admitted source size and duration, |
| 580 | concurrent jobs per box, memory and scratch caps on the ffmpeg helper, |
| 581 | and measurement of serving latency, I/O and CPU credit consumption |
| 582 | under sustained encoding before any promise about co-location is |
| 583 | made. **(codex 20)** |
| 584 | |
| 585 | **Which box.** A `t4g.medium` has the same two burstable vCPUs as a |
| 586 | `t4g.small`; it is not a transcoding box. Version one runs the worker |
| 587 | on the Ireland medium behind the limits above, and the honest |
| 588 | expectation is that the first tenant wanting ladders moves it to a |
| 589 | dedicated `c7g.large`. Encode throughput is the first sustained |
| 590 | bottleneck; host memory and scratch pressure fail earlier if left |
| 591 | unbounded, which is why the limits come first. **(codex 20; ruling 7)** |
| 592 | |
| 593 | **Not in stage 3:** caption generation, DRM, per-viewer watermarking, |
| 594 | server-side ad insertion, live packaging. Caption files the app supplies |
| 595 | are just objects. |
| 596 | |
| 597 | ### Stage 4: real time |
| 598 | |
| 599 | Meet's requirements, in the order Meet needs them. Stage 4a depends on |
| 600 | nothing above and starts now. **(codex 27; ruling 11)** |
| 601 | |
| 602 | **4a. STUN and TURN.** A STUN listener the platform runs is what lets a |
| 603 | WebRTC app connect across most home and mobile networks without an |
| 604 | egress exception; TURN is the same listener relaying, and it is what |
| 605 | turns "one call in ten cannot connect" into "calls connect". Both are |
| 606 | UDP. The capabilities spec's non-goal stands: UDP is not something an |
| 607 | app binds. The platform runs `carlos-relay.service` on every region |
| 608 | box, built on `pion/turn`, pure Go, in the same binary as the edge, a |
| 609 | §3 media service. |
| 610 | |
| 611 | - **Ingress.** UDP 3478 and a relay allocation range (49152 to 65535) |
| 612 | opened in the region security group and configured as the allocator's |
| 613 | only range; TCP 3478 for networks that block UDP. The relay advertises |
| 614 | the box's public Elastic IP. **No TURN over TLS in version one** |
| 615 | (Paul, 2026-09-07, stage 4a spec ruling 4): 5349 needs certificate |
| 616 | issuance for a non-HTTP listener, which is its own small design, and |
| 617 | 443 would need a second address on the box because 443 is the edge's. |
| 618 | The docs say a network that blocks 3478 is the remaining case that |
| 619 | fails. **(codex 15; amended by the stage 4a spec)** |
| 620 | - **Credentials, scoped per app and per region.** Each region has a |
| 621 | relay master key. At `carlos relay enable` the instance receives, |
| 622 | sealed through `config.json`, one derived key per allowed region: |
| 623 | `HKDF(region master, account, app, incarnation)`, as |
| 624 | `CARLOS_RELAY_KEYS` alongside `CARLOS_RELAY_URLS`. The app mints TURN |
| 625 | REST credentials per call, `username = |
| 626 | expiry:account:app:incarnation`, `credential = HMAC(derived key, |
| 627 | username)`, with a four-hour maximum (Paul, 2026-09-07, stage 4a |
| 628 | spec ruling 1; the relay ends the allocation at expiry, so one hour |
| 629 | would end any longer call). The exact wire shape, a base64-wrapped |
| 630 | JSON `CARLOS_RELAY` with a platform-supplied username suffix and a |
| 631 | key id, is the stage 4a spec's §2. The relay holds a **relay |
| 632 | authority record**, `control/relay.json`, rebuilt by the console from |
| 633 | per-app relay records and read by every relay every minute: for each |
| 634 | enabled app, its regions and current incarnation. On a request it |
| 635 | parses the username, refuses it unless the app is enabled and the |
| 636 | incarnation matches the record, re-derives the key from its own |
| 637 | region master and the username's fields, and verifies. Verification |
| 638 | is local and per-app metering is real. Disable removes the app from |
| 639 | the record, which ends existing allocations and makes every |
| 640 | outstanding credential fail; re-enable draws a new random |
| 641 | incarnation, so nothing minted before a disable ever works again and |
| 642 | nothing is reused across deletion or name reuse. A relay whose authority |
| 643 | record is older than fifteen minutes refuses new allocations **and |
| 644 | terminates the ones it is carrying**, the same lease rule the edge |
| 645 | applies to media: a disable the relay never heard about must not |
| 646 | outlive the outage by the length of a credential. Rotation is a new |
| 647 | region master and a re-render of every enabled app's config. |
| 648 | **(codex 13, R2-2, R3-3, R4-6; ruling 8)** |
| 649 | - **What that does and does not protect.** Per-region masters mean a |
| 650 | relay box holds no other region's master. They do not mean a |
| 651 | compromised host learns nothing about other regions: sealed config is |
| 652 | decryptable on every host (§1), and the console must hold every |
| 653 | region's master as sealed config to derive app keys, so **a |
| 654 | compromised host can read every region's master** and mint for any |
| 655 | app in any region, present and future, until the operator rotates |
| 656 | (Paul, 2026-09-07, stage 4a spec ruling 3). That is wider than the |
| 657 | derived-key exposure alone, and it is stated in those words in the |
| 658 | operator docs. Rotation of one region recovers that region. A design |
| 659 | where only a region's boxes ever hold its master, the relay deriving |
| 660 | and writing app keys itself, is filed as the tightening. **(codex |
| 661 | R2-2; amended by the stage 4a spec)** |
| 662 | - **Policy and limits.** Peer permissions refuse loopback, link-local, |
| 663 | RFC 1918, the VPC's ranges and the box's private addresses; the box's |
| 664 | own Elastic IP is reachable, but only on ports holding a live relay |
| 665 | allocation, so two clients on one relay can reach each other and |
| 666 | nothing else on the box is exposed. Allocations, |
| 667 | allocation rate, bandwidth and lifetimes are bounded per app, per |
| 668 | account and per box. Established allocations end when their |
| 669 | credential expires or the app is disabled. These are operational |
| 670 | limits in 4a; commercial quotas wait for measurement. **(codex 14)** |
| 671 | - **Placement.** The app's `CARLOS_RELAY` lists its regions in the |
| 672 | console's order, one list for the whole app, since placement names |
| 673 | are not regions and an app may have several placements (stage 4a |
| 674 | spec). The browser gathers candidates from all of them |
| 675 | and ICE nominates a working pair by its own procedures; that is not a |
| 676 | lowest-latency guarantee and the roadmap does not claim one. |
| 677 | Client-aware relay selection is a separate later decision. |
| 678 | **(codex 15)** |
| 679 | - **Metering.** Bytes forwarded from the client and bytes forwarded to |
| 680 | the client are two meters, per app, written by the relay to its own |
| 681 | metrics objects; the inter-region meter the first draft named, and |
| 682 | the account usage line, move to stage 5 with the usage view (Paul, |
| 683 | 2026-09-07, stage 4a spec ruling 5). The unit-economics spec gets the |
| 684 | definitions. **(codex 26; amended by the stage 4a spec)** |
| 685 | - **Custody language.** A platform relay is not egress under ruling 6: |
| 686 | the recipient is the operator the team already trusts with its pages. |
| 687 | A third-party relay is egress, and under doctrine §8 which third |
| 688 | party is the app's decision, never the team's. |
| 689 | |
| 690 | **4b. LiveKit as a media service, recording, live streaming.** An SFU |
| 691 | lifts the four-participant cap, changes the privacy story from |
| 692 | peer-to-peer to server-in-the-path, and is where recording becomes |
| 693 | possible. **Paul ruled to host LiveKit's open-source server** rather |
| 694 | than build on `pion/webrtc`, as a §3 media service, `carlos-sfu.service`, |
| 695 | fronted by the platform's own API. **(ruling 9)** LiveKit Cloud is not an |
| 696 | option for suite apps: a third party in every call's media path is |
| 697 | egress that is a property of the product. It is also not cheaper past |
| 698 | about sixty room-hours a month, since both are dominated by bandwidth. |
| 699 | |
| 700 | - **Box.** The server is one Go binary on pion and needs no Redis as a |
| 701 | single node. A `c7g.large` in Ireland (2 vCPU, 4 GB, about sixty |
| 702 | dollars a month) carries a handful of concurrent eight-person video |
| 703 | rooms; a `c7g.xlarge` gives headroom for tens. CPU is spent per |
| 704 | forwarded track, and LiveKit's own figures put thousands of tracks on |
| 705 | sixteen vCPUs, so the box is not the limit at this scale. |
| 706 | - **Cost is egress.** Each participant downloads everyone else's stream. |
| 707 | An eight-person video room at about 1 Mbps per stream is about 25 GB |
| 708 | an hour, roughly two dollars at AWS rates. That line goes in the |
| 709 | unit-economics amendment beside the relay meters. |
| 710 | - **Ports and hostname.** LiveKit's clients speak WebSocket to the |
| 711 | server's `/rtc` endpoint on 7880, so the SFU has a platform hostname |
| 712 | of its own, `sfu.<region>.<platform domain>`, with TLS terminated by |
| 713 | the edge on that box and the upgrade proxied to LiveKit. The browser |
| 714 | reaches it directly, never through the app's route, so an app on a |
| 715 | tunnel-backed host still joins SFU rooms. Port 7880 is reachable only |
| 716 | from that proxy. TCP 7881 and UDP 50000 to 60000 are opened directly, |
| 717 | the same shape as the relay. **(codex R3-5)** |
| 718 | - **Tenant boundary, enforced at admission.** The platform holds |
| 719 | LiveKit's API key and secret; no app ever does. Rooms are named |
| 720 | `<account>/<app>/<incarnation>/<room>` and only the platform mints |
| 721 | tokens: the app's backend calls `POST |
| 722 | /a/<sqid>/apps/<app>/media/sfu/token` with its route credential, |
| 723 | naming a room, a participant identity and grants, and the door |
| 724 | prefixes the room, caps the grants to that app's namespace, signs a |
| 725 | ten-minute token, and returns it for the app to hand to its browser. |
| 726 | **The proxy in front of 7880 is the admission gate**: on every join |
| 727 | and reconnect it verifies the token, checks the room's revocation |
| 728 | list, and only then opens the upstream connection, serialised against |
| 729 | removals so a rejoin cannot race a removal. LiveKit's own token |
| 730 | refresh for connected clients does not matter, because admission is |
| 731 | keyed on identity and room against platform state, not on a token |
| 732 | being unexpired. Removal is an app-to-platform door that records the |
| 733 | revocation, calls LiveKit's remove-participant, which is what ends the |
| 734 | active media session, and has the proxy close that participant's |
| 735 | signalling connection and refuse its next admission, none of it |
| 736 | waiting for anything the client does. Webhooks are for |
| 737 | observation and reconciliation, never enforcement. Management |
| 738 | operations are the door's, scoped to the app's rooms. **(codex R3-6, |
| 739 | R4-1, R4-2)** |
| 740 | - **No third-party defaults.** Stock LiveKit hands participants Google's |
| 741 | and Twilio's STUN servers when none are configured and uses them for |
| 742 | its own external-IP discovery. The platform's LiveKit is configured |
| 743 | with the stage 4a relay as its only STUN and TURN, for the server's |
| 744 | discovery and the client ICE configuration alike, and the 4b gate |
| 745 | runs a call with all third-party network access blocked to prove no |
| 746 | fallback remains. This is why 4b follows 4a. **(codex R4-7)** |
| 747 | - **Recording and ingest are separate LiveKit services** and are |
| 748 | deferred until asked for, with their prerequisites named now so the |
| 749 | launch box is not mistaken for them: LiveKit Egress needs Redis to |
| 750 | talk to the server, its own worker of at least four vCPUs and 4 GB, |
| 751 | and Chrome's sandbox requirements for composite recording; WHIP and |
| 752 | RTMP ingest is a third service with the same Redis dependency. |
| 753 | Recording writes per-track or composite files into the app's bucket |
| 754 | and submits a stage 3 single-rendition job, so a recorded call |
| 755 | becomes a playable asset with no new serving code. Recording depends |
| 756 | on stage 3, not the other way round. **(codex R2-11, R3-8)** |
| 757 | - **Live streaming to many viewers** is a distinct long-running workload: |
| 758 | ingest feeds a live packager, and LL-HLS needs partial segments, |
| 759 | blocking playlist reloads and delivery directives that an object |
| 760 | reader does not supply, so it is a named serving extension with its |
| 761 | own latency contract, not a reuse of stage 1 unchanged. **(codex 22)** |
| 762 | |
| 763 | **Signalling.** In peer-to-peer mode, with or without the stage 4a relay, |
| 764 | offers, answers and candidates ride the app's own HTTP, and §8 answers |
| 765 | #113 about that. In SFU mode the LiveKit client SDK carries its own |
| 766 | signalling over the WebSocket to the SFU hostname above; the app's HTTP |
| 767 | is then for room coordination and token refresh only. **(codex R3-5)** |
| 768 | |
| 769 | ### Stage 5: what makes it usable from an app |
| 770 | |
| 771 | Each of these is small and belongs in the plan of the stage that first |
| 772 | needs it. None of them is in stage 1's gate. **(codex 27)** |
| 773 | |
| 774 | - **Direct browser uploads.** Apps already presign PUTs, but a browser |
| 775 | cannot PUT to the bucket until the bucket has a CORS rule, and S3 CORS |
| 776 | is bucket-wide with no prefix selector. `carlos media enable` |
| 777 | therefore owns the bucket's CORS configuration for the app's |
| 778 | hostnames, merging with nothing (a store with CORS already set is |
| 779 | refused until the operator clears it), exposing `ETag` and the |
| 780 | multipart headers, and removing it at disable. Multipart upload for |
| 781 | anything over 100 MB is a rastrillo helper. **(codex 23)** |
| 782 | - **rastrillo helpers.** One package, `rastrillo/media`: a URL builder |
| 783 | for the three doors, a capability signer that reads |
| 784 | `CARLOS_MEDIA_SIGNING_KEY`, the private-check handler that answers the |
| 785 | edge's subrequest from the app's grants, a visibility-move helper |
| 786 | that drives the lifecycle in §2, a presigned-upload helper with |
| 787 | CORS-correct headers, a job filer, and a TURN credential minter. |
| 788 | Docs' disabled "insert image" is the first consumer. |
| 789 | - **Accounting.** Bytes served and cached in stage 1 as one counter; |
| 790 | transform seconds, transcode minutes, the three relay meters and SFU |
| 791 | egress when their stages land; on the console's app page and in |
| 792 | `carlos media status`. Quotas come after measurement, not before. |
| 793 | - **Scanning.** Nothing to hook: the sweep already tags what it finds, |
| 794 | and the media grant carries the deny. The one addition is the verdict |
| 795 | purge in stage 1. |
| 796 | - **Residency disclosure.** One paragraph on the media docs page and in |
| 797 | `carlos media status` naming the allowed regions and, for private |
| 798 | media, the home edge. |
| 799 | - **Docs page.** `docs/site/media.md`, and `docs/site/cli.md` headings for |
| 800 | every new verb, because the docsite gate fails otherwise. |
| 801 | |
| 802 | ## 5. What apps might want that is deliberately not here |
| 803 | |
| 804 | Named so nobody wonders whether it was forgotten. |
| 805 | |
| 806 | - **Signed cookies** instead of path capabilities. Not needed while |
| 807 | capabilities are prefix-scoped; revisit if a tenant needs a whole site |
| 808 | of secret media. |
| 809 | - **A media-session cookie on the media host** for private media. Cannot |
| 810 | be relied on across browsers and privacy settings on custom domains; |
| 811 | the app-origin door in §2 is the answer. |
| 812 | - **Per-viewer watermarking or DRM.** Different product, different buyer. |
| 813 | - **AI on media**: captions, transcription, moderation, smart crop. Each |
| 814 | is a model-provider egress under ruling 6 and a per-app opt-in there, |
| 815 | not a platform primitive. |
| 816 | - **A platform upload door** through the edge. Writes never cross the |
| 817 | data plane; uploads stay app-to-S3. |
| 818 | - **Peer-to-peer file transfer** over the relay. TURN would carry it, but |
| 819 | it is an app protocol, not a platform one. |
| 820 | - **A GPU tier.** Nothing in stages 1 to 4 needs one on a fleet this size. |
| 821 | - **LiveKit Cloud, or any hosted SFU.** Egress that is a property of the |
| 822 | product. |
| 823 | |
| 824 | ## 6. What must be true before an app can build on this |
| 825 | |
| 826 | The review's last question was what an app hits in its first month. The |
| 827 | answers are all inside stages 1 and 5, and are the reason stage 1's gate |
| 828 | is a security gate, but they are worth listing plainly: |
| 829 | |
| 830 | - a way to put a private image on a page that works on a custom domain |
| 831 | in Safari (the app-origin door); |
| 832 | - a way to hand a video player one URL and have every segment load (the |
| 833 | path capability); |
| 834 | - a way to share a roster-visible video by link without moving it (the |
| 835 | shared `protected/` namespace); |
| 836 | - a way to replace an object and not serve the old bytes (ETag-bound |
| 837 | cache, distributed purge, the publication fence); |
| 838 | - a way to know which regions hold copies (the media block); |
| 839 | - an upload that does not fail at the last byte of a phone video (CORS |
| 840 | ownership, multipart helper); |
| 841 | - a helper package so none of the above is hand-rolled per app. |
| 842 | |
| 843 | ## 7. Rulings |
| 844 | |
| 845 | Paul ruled on each of these on 2026-09-07. They are the roadmap's fixed |
| 846 | points; a stage spec that wants to move one comes back here first. |
| 847 | |
| 848 | 1. **(ruling)** Private means ask-the-app, per request, never cached, |
| 849 | served on the app's own origin under a reserved path. |
| 850 | 2. **(ruling)** Edge read is granted by bucket policy on the two media |
| 851 | namespaces, per allowed region host role, carrying the infected deny, |
| 852 | attached and removed by a reconciled walk. |
| 853 | 3. **(ruling)** Ed25519 per app for capabilities, public half in the |
| 854 | route record, private half as env through sealed `config.json`, key |
| 855 | id in the capability for rotation, seven-day maximum life. |
| 856 | 4. **(ruling)** Allowed regions are explicit in the media block and |
| 857 | enforced in the bucket policy, DNS and the handler; default is the |
| 858 | regions the app's routes are placed in. |
| 859 | 5. **(ruling)** The transform engine is chosen by a measured spike among |
| 860 | a Go wazero helper, `vips` and a Rust helper, all as separate |
| 861 | processes, benchmarked on a `t4g.small`, with a bigger or dedicated |
| 862 | box on the table if the numbers say so. |
| 863 | 6. **(ruling)** Derivatives and renditions land in the tenant's bucket |
| 864 | under `derived/` in the source's namespace, counted as the tenant's |
| 865 | storage, removed with their source. |
| 866 | 7. **(ruling)** The media worker starts on the Ireland `t4g.medium` |
| 867 | behind hard admission limits, with a dedicated `c7g.large` when |
| 868 | ladders are wanted. |
| 869 | 8. **(ruling)** Relay credentials are per-region HKDF-derived per app and |
| 870 | incarnation, verified locally; the exposure through sealed config is |
| 871 | stated, not hidden. |
| 872 | 9. **(ruling)** The SFU is LiveKit's open-source server hosted as a |
| 873 | platform media service on a `c7g.large`, recording and ingest |
| 874 | services deferred until asked for. |
| 875 | 10. **(ruling)** Native helpers are allowed as separate processes only, |
| 876 | with codec helpers and media services as distinct classes. |
| 877 | 11. **(ruling)** Stage 1 and stage 4a start now, independently. Stage 2 |
| 878 | and the stage 3 single-rendition product follow stage 1. The ladder |
| 879 | follows the worker measurements; 4b follows 4a. |
| 880 | |
| 881 | ## 8. Answers to #112 and #113 |
| 882 | |
| 883 | **#112.** Direction A: the platform runs the relay. STUN and TURN are |
| 884 | stage 4a and start now; LiveKit as a hosted media service is 4b. UDP |
| 885 | ingress is something the platform will carry, as a platform unit on the |
| 886 | region boxes, never as an app-bindable port. A platform relay is not |
| 887 | egress; a third-party one is. |
| 888 | |
| 889 | One correction to Meet's interim plan: under doctrine §8 the recipient |
| 890 | of an egress is the app's decision, never the team's, so a settings |
| 891 | field where a team pastes any TURN URL is not admissible. What is |
| 892 | admissible until 4a ships is either no relay, with the disclosed floor |
| 893 | of calls that connect without one, or an app-named provider that a team |
| 894 | can switch on, disclosed in the README's first paragraph **and on every |
| 895 | artefact the app produces while it is on**, which for Meet means any |
| 896 | recording or export. Meet should pick one of those two, not the |
| 897 | free-text field. **(codex 12, R2-12)** |
| 898 | |
| 899 | **#113.** Confirmed as a standing constraint: no `Connection: Upgrade` |
| 900 | over a tunnel-backed host. The placement picture is more nuanced than |
| 901 | the question assumed: a member can place an app with `carlos apps place` |
| 902 | and `carlos instances create --region`, and the instances API reports |
| 903 | which boxes serve a host, so a team can keep Meet off tunnel-backed |
| 904 | hosts by choice. What does not exist, and is not planned before extended |
| 905 | CONNECT, is a runtime signal the app can read saying "this host carries |
| 906 | upgrades", or a manifest field that refuses placement without it. |
| 907 | **(codex R2-13)** So SSE plus POST remains the right default: it works |
| 908 | everywhere, its volume is small, and a socket can be adopted later on |
| 909 | hosts the team has placed deliberately, with SSE as the fallback Meet |
| 910 | would need anyway. |
| 911 | |
| 912 | ## 9. Acceptance gates |
| 913 | |
| 914 | Each stage has its own gate; what a gate unlocks is whatever §4's graph |
| 915 | hangs off it, not "the next stage" in this list. The gates are attacks |
| 916 | failing, then playback working. **(codex 27, R3-9)** |
| 917 | |
| 918 | - **Stage 1**, on flagship: `/p/` of a protected key is 404 cold and |
| 919 | warm; a capability with the expiry edited, the scope widened past a |
| 920 | `/` boundary, the namespace changed, or the key id revoked is refused; |
| 921 | a private fetch is refused the request after the app revokes the |
| 922 | grant, on a warm cache; a public-to-protected move leaves nothing |
| 923 | readable through `/p/` in any region once it reports complete; an |
| 924 | object tagged infected is unreadable through media and its cached |
| 925 | copy is gone within a purge round; a purge issued while a fill is in |
| 926 | flight leaves no stale entry; an edge reconnecting after an outage |
| 927 | serves nothing from disk until its cursor is current; a revoked key, |
| 928 | and a public object moved to `protected/`, both stop being served by |
| 929 | a disconnected edge within fifteen minutes; an edge outside the |
| 930 | allowed regions refuses to fill; a late commit from a worker whose |
| 931 | lease expired with no successor is refused by the publish door, and |
| 932 | the request that caused a stalled synchronous transform receives no |
| 933 | bytes; an app named `media-<x>` created before enable keeps working |
| 934 | and enable for `<x>` is refused with the right error; an instance |
| 935 | host `media-<x>` taken by another app before enable, and one |
| 936 | attempted after, are each refused in the right direction; an app |
| 937 | name ending in a hyphen is refused enable. Then: a public hit served from |
| 938 | a second region's cache, one object reached both by the app's check |
| 939 | and by a capability, an HLS playlist and its segments loading under |
| 940 | one capability, and a 2 GB MP4 seeking in a browser through a |
| 941 | `t4g.small` without the box paging. |
| 942 | - **Stage 2**: the spike's numbers, then 9,999 distinct transform URLs |
| 943 | against a public object producing zero decodes, and one tenant's |
| 944 | burst leaving another's thumbnails under a second. |
| 945 | - **Stage 3**: a 1080p H.264 phone recording of two minutes uploaded |
| 946 | through Docs and playing in Safari as a single HLS rendition within |
| 947 | thirty seconds, with serving latency on the same box flat; a WebM |
| 948 | browser recording of the same length playable within three minutes; a |
| 949 | job for a non-Ireland bucket refused at the door; the ladder measured, |
| 950 | not promised. |
| 951 | - **Stage 4a**: a Meet call between two symmetric-NAT networks connecting |
| 952 | through the Ireland relay with a credential that expires in four |
| 953 | hours, and both browsers forced to relay connecting through it; a |
| 954 | credential minted for one app refused for another; a credential |
| 955 | minted before a disable refused after re-enable, and the allocation |
| 956 | it held ended; a relay with an authority record older than fifteen |
| 957 | minutes refusing a new allocation and ending one already carrying |
| 958 | traffic; a relay permission to a private |
| 959 | address refused; the two meters moving in the relay's metrics object; |
| 960 | a network that blocks 3478 failing the way the docs say it will. The |
| 961 | full gate is the stage 4a spec's §7. |
| 962 | - **Stage 4b**: a token minted for one app refused for another app's |
| 963 | room; a participant removed by the app dropped within a second and |
| 964 | unable to rejoin with its old token or a LiveKit-refreshed one, with |
| 965 | the webhook delayed and then lost; a browser on an app served from a |
| 966 | tunnel-backed host joining an SFU room; a call completing with all |
| 967 | third-party network access blocked from the SFU box; an eight-person |
| 968 | room's egress appearing on the account's usage line. |
| 969 | |
| 970 | ## 10. Findings not accepted |
| 971 | |
| 972 | - **Codex 12, the part asking for a doctrine amendment** to allow |
| 973 | team-selected external relays. Not requested; the roadmap instead |
| 974 | corrects Meet's interim plan to what §8 already allows, now including |
| 975 | artefact disclosure. |
| 976 | - **Codex 15, TURN over TLS on 443.** Acknowledged as a real gap and |
| 977 | documented as such; a second address per box is deferred until a |
| 978 | tenant reports a 443-only network, because it changes the fleet's |
| 979 | networking for every box to serve a case nobody has hit yet. |
| 980 | |
| 981 | Every finding from revisions 2 to 5 was accepted. |
| 982 | |
| 983 | ## 11. Amendments to existing specs |
| 984 | |
| 985 | - `2026-08-10-object-storage-primitive-design.md:27`: reads may cross the |
| 986 | data plane through `kind:media`; writes never do. |
| 987 | - `2026-08-03-app-capabilities-design.md:931`: UDP remains a non-goal |
| 988 | for apps; the platform's own relay and SFU units are the exception and |
| 989 | are specified in stage 4. |
| 990 | - `2026-08-11-unit-economics-design.md`: a media section, at stage 1 |
| 991 | for served bytes, at 4a for the two relay meters (bytes forwarded in |
| 992 | each direction), at stage 5 for the inter-region meter and the usage |
| 993 | line, at 4b for SFU egress. |
| 994 | - `infra/modules/region-host/main.tf:6` header: residency by |
| 995 | construction now has a named exception, the per-region media bucket |
| 996 | policy, and the exception is itself region-scoped. |
| 997 | - The console's name-claim gate (`internal/console/server.go:2904`) and |
| 998 | the instances door (`internal/console/apiinstances.go:571`): new app |
| 999 | names, renames and instance hosts consult the media-host claim table. |
| 1000 | The `release.ValidApp` grammar is unchanged. **(codex R3-4, R4-4)** |
| 1001 | |
amadan