amadan
spec: edge response compression (gzip)
One gzip ResponseWriter wrapper at the client-write boundary, so cache hits, proxied misses, static trees and the status host all compress through a single site. The recorder stays above it (cache keeps storing identity bodies) and the usage meter's counter below it (compressed bytes are what a tenant is billed for). The load-bearing detail is a private header map: this package's writer wrappers share one header map all the way down, and maybeStore caches rec.Header().Clone() — so setting Content-Encoding on the shared map would store an identity body labelled gzip and serve it to every later visitor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoqzkUVahW5ykubsBQitxc
1 file changed,
+184
−0
docs/superpowers/specs/2026-08-18-edge-response-compression-design.md+184 −0
diff --git a/docs/superpowers/specs/2026-08-18-edge-response-compression-design.md b/docs/superpowers/specs/2026-08-18-edge-response-compression-design.md| new file mode 100644 |
| index 000000000..c8b58224e |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-18-edge-response-compression-design.md |
| @@ -0,0 +1,184 @@ |
| +# Edge response compression (gzip) — design |
| + |
| +Status: approved 2026-08-18. Implements the transport half of the promise |
| +rastrillo's assets spec makes to apps ("ship identity bytes; the platform edge |
| +handles transport concerns"). |
| + |
| +## Problem |
| + |
| +Nothing the fleet serves is compressed on the wire. The only gzip in |
| +`internal/edge` today is `static.go`'s artifact *storage* format (`site.tar.gz`, |
| +extracted to disk before serving) and `cache.go`'s `Vary: Accept-Encoding` |
| +machinery, which anticipates encoding variants that no code path emits. |
| + |
| +Apps are identity-only by design. rastrillo's `tokens.css` is 21.5 KB on the |
| +wire where 5.6 KB would do. Compressed bytes are also what the usage meter |
| +should count — a tenant billed for `BytesOut` is billed for bytes the platform |
| +chose not to compress. |
| + |
| +## Shape |
| + |
| +One wrapper, applied once at the client-write boundary in `Edge.ServeHTTP`, so |
| +every serving class passes through a single site: cache hits (`writeCached`), |
| +proxied misses, static trees, and the status host. |
| + |
| +Writer stack, real connection outward to handler: |
| + |
| +``` |
| +real conn → countingWriter → gzipWriter → recordingResponseWriter → proxy/static/status |
| +``` |
| + |
| +- The recorder sits **above** the gzip writer, so it captures identity bytes: |
| + the response cache keeps storing identity bodies and one variant, and the |
| + compression happens after retrieval, on every delivery. |
| +- The counter sits **below**, so the usage meter counts what actually crossed |
| + the wire. |
| +- `defer gw.finish()` is registered **after** the meter's defer, so LIFO closes |
| + the gzip stream (writing its trailer through the counter) before `m.Record` |
| + reads the byte count. |
| + |
| +### Placement |
| + |
| +The wrap goes **after** the `isUpgradeRequest` branch and **before** the cache |
| +check in `Edge.ServeHTTP`. |
| + |
| +After the upgrade branch: an upgrade keeps today's exact writer stack, byte for |
| +byte. `purge_header.go`'s contract records that this package has already paid |
| +for wrapping the raw ResponseWriter on the upgrade path, and this design does |
| +not re-open that class. |
| + |
| +Before the cache check: a cache hit is a delivery like any other and must |
| +compress on replay. |
| + |
| +The tunnel hub and the fleet store door return ahead of this point (they are |
| +not registry routes, and the hub ends its handshake in a hijack). They stay |
| +uncompressed. Out of scope. |
| + |
| +## The private header map |
| + |
| +`http.ResponseWriter` wrappers in this package embed the writer below, so |
| +`Header()` returns one shared map all the way down. `maybeStore` caches |
| +`rec.Header().Clone()` (edge.go). If the gzip wrapper set `Content-Encoding: |
| +gzip` on that shared map, the cache would store an **identity body labelled |
| +gzip** and serve it to every later visitor as a broken response. |
| + |
| +So `gzipWriter` owns a private `http.Header`: |
| + |
| +- `Header()` returns the private map. Everything above — the recorder, the |
| + proxy, `http.ServeFile` — reads and writes only that. |
| +- At final `WriteHeader`, the wrapper clears the real writer's map, copies the |
| + private map into it, then applies its edits **to the real map only**: |
| + set `Content-Encoding: gzip`, delete `Content-Length`, add |
| + `Vary: Accept-Encoding`, weaken a strong `ETag`. |
| + |
| +Consequences, each pinned by a test: |
| + |
| +- **1xx**: a `103 Early Hints` block copies-and-forwards without latching a |
| + status, mirroring `recordingResponseWriter.WriteHeader`. Without the copy the |
| + hints would be lost, since they live in the private map. |
| +- **Silent handler**: if the handler returns having never called `WriteHeader` |
| + or `Write`, `finish()` syncs the private map down. Otherwise the stdlib emits |
| + a 200 with no headers at all. |
| +- **Vary placement**: `Vary: Accept-Encoding` is added below the recorder, so it |
| + reaches clients and downstream CDNs but never enters the cache key. That is |
| + correct: the stored variant is identity, keyed by whatever the app itself |
| + varied on. `cache.go`'s existing Vary machinery is unaffected. |
| + |
| +## Eligibility |
| + |
| +Decided once, at final `WriteHeader`, from headers alone. No buffering, no |
| +sniffing, no interference with streaming. |
| + |
| +Compress when **all** hold: |
| + |
| +1. The request accepts gzip. `Accept-Encoding` is parsed as a q-valued token |
| + list: `gzip;q=0` and `*;q=0` refuse; a bare `*` with non-zero q accepts. |
| +2. Method is not `HEAD`. |
| +3. Status is not 204, 205, 206 or 304, and no `Content-Range` is present. |
| +4. No `Content-Encoding` already on the response. |
| +5. No `Cache-Control: no-transform` (RFC 9110 §5.5 — the origin forbidding |
| + exactly this). |
| +6. `Content-Type` is present and compressible: `text/*` except |
| + `text/event-stream`; plus `application/json`, `application/javascript`, |
| + `application/x-javascript`, `application/xml`, `application/xhtml+xml`, |
| + `application/manifest+json`, `application/rss+xml`, `application/atom+xml`, |
| + `image/svg+xml`, `application/wasm`; plus any type with a `+json` or `+xml` |
| + suffix. |
| +7. `Content-Length` is absent, or present and ≥ 1024 bytes. |
| + |
| +**A missing `Content-Type` means identity, always.** Content sniffing belongs to |
| +the stdlib, and it runs on the bytes we would have already compressed — it would |
| +sniff gzip magic and label the response `application/x-gzip`. `http.ServeFile` |
| +and real apps set the type; the ones that don't, don't compress. |
| + |
| +`HEAD` is skipped rather than compressed-with-no-body: a HEAD's truthful |
| +`Content-Length` is the entire value of the response. The accepted cost is that |
| +a HEAD advertises the identity length while a GET returns gzip with no length. |
| + |
| +## Behaviour at the edges |
| + |
| +- **ETag**: a strong `ETag` is weakened to `W/"…"` on the real header only, so |
| + the cached entry keeps the strong tag for its identity body. `ServeFile`'s |
| + `If-None-Match` check compares weakly, so a conditional request still 304s. |
| +- **Flush**: flush the gzip stream, then the writer below. SSE never compresses |
| + (content type), so its flushes pass straight through. |
| +- **Hijack**: forwarded to the writer below, matching `countingWriter` and |
| + `recordingResponseWriter`. Not expected on this path — upgrades return |
| + earlier. |
| +- **Level**: `gzip.DefaultCompression`, writers from a `sync.Pool`. We compress |
| + on every delivery including cache hits, so allocation matters more than the |
| + last few percent of ratio. |
| +- **Kill switch**: `CARLOS_EDGE_GZIP=off` (also `false`/`0`), read once in |
| + `New`, matching the `CARLOS_*` idiom in `certcache.go` and `dns.go`. When off, |
| + the wrapper is never constructed and behaviour is byte-identical to today. |
| + This is the rollback that needs no redeploy. |
| + |
| +## Accepted cost: BREACH |
| + |
| +Compressing HTML that carries a secret (a CSRF token) alongside |
| +attacker-influenced reflected text is the BREACH precondition. Every CDN and |
| +reverse proxy in production makes this trade; the mitigation is token masking in |
| +the application, not identity transport. Named here so it is a decision on the |
| +record rather than a discovery later. |
| + |
| +## Rejected alternatives |
| + |
| +- **Precompress in `PackDir` at ship time.** Changes an artifact shape (the |
| + wire-shape rule: everything written before the change must still parse) and |
| + helps only static routes — proxied apps, the status host, and cache hits stay |
| + uncompressed. |
| +- **Compress above the recorder, cache compressed bodies.** Compresses once per |
| + cached response instead of once per delivery, but stores two variants per URL |
| + against the shared LRU's budget and makes every cache-correctness question a |
| + question about encodings too. The CPU saved is not worth widening that |
| + surface. |
| +- **brotli / zstd.** New dependency for a marginal ratio gain. Stdlib gzip only. |
| + |
| +## Testing |
| + |
| +In-package tests, TDD. Load-bearing: |
| + |
| +- Cache stores identity: after a compressible cacheable response, the stored |
| + entry's headers carry no `Content-Encoding` and its body is raw; both the miss |
| + and the hit deliver valid gzip; a client with no `Accept-Encoding` gets |
| + identity from that same entry. |
| +- Usage meter records compressed byte counts (strictly less than identity). |
| +- A WebSocket handshake with `Accept-Encoding: gzip` set still returns 101. |
| +- The existing 103 Early Hints pass-through still holds. |
| +- A handler that writes nothing still delivers its headers. |
| +- An already-gzipped upstream response is passed through, not double-compressed. |
| +- 206, HEAD, `text/event-stream`, missing `Content-Type`, and sub-1024 |
| + `Content-Length` all stay identity. |
| +- Strong ETag weakened; conditional request still 304s. |
| +- `CARLOS_EDGE_GZIP=off` reproduces today's bytes exactly. |
| + |
| +Existing tests asserting exact `BytesOut` or `Content-Length` on compressible |
| +bodies ≥ 1024 bytes will need updating; the threshold spares the rest. |
| + |
| +## Files |
| + |
| +- `internal/edge/compress.go` — new. The wrapper, the eligibility predicate, the |
| + `Accept-Encoding` parser, the pool. |
| +- `internal/edge/compress_test.go` — new. |
| +- `internal/edge/edge.go` — the wrap site in `ServeHTTP`, the `Edge` field, the |
| + env read in `New`. |