amadan
plan: edge response compression (gzip)
Five tasks, TDD throughout: Accept-Encoding negotiation, the eligibility predicate, the gzipWriter with its private header map, the ServeHTTP wrap site plus the cache-identity and metered-bytes invariants, then the regression battery and the gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoqzkUVahW5ykubsBQitxc
1 file changed,
+1456
−0
docs/superpowers/plans/2026-08-18-edge-response-compression.md+1456 −0
diff --git a/docs/superpowers/plans/2026-08-18-edge-response-compression.md b/docs/superpowers/plans/2026-08-18-edge-response-compression.md| new file mode 100644 |
| index 000000000..06dda8858 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-18-edge-response-compression.md |
| @@ -0,0 +1,1456 @@ |
| +# Edge Response Compression (gzip) Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Compress eligible responses at the CARLOS edge with stdlib gzip, at a single client-write boundary that covers cache hits, proxied misses, static trees and the status host. |
| + |
| +**Architecture:** One `gzipWriter` wrapper inserted in `Edge.ServeHTTP` after the upgrade branch and before the cache check, giving the writer stack `real conn → countingWriter → gzipWriter → recordingResponseWriter → handler`. The recorder above it keeps recording identity bytes (the response cache keeps storing identity bodies, one variant); the usage meter's counter below it counts compressed bytes. The wrapper owns a **private header map** so the compression headers it adds never reach the recorder — that is what stops the cache storing an identity body labelled `Content-Encoding: gzip`. |
| + |
| +**Tech Stack:** Go, stdlib only (`compress/gzip`, `sync.Pool`). No brotli, no zstd, no new module dependency. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-08-18-edge-response-compression-design.md` |
| + |
| +## Global Constraints |
| + |
| +- **Stdlib only.** `compress/gzip` at `gzip.DefaultCompression`, writers from a `sync.Pool`. No new dependency in `go.mod`. |
| +- **Placement is load-bearing.** The wrap site is in `Edge.ServeHTTP`, strictly **after** the `isUpgradeRequest` branch and strictly **before** the `cacheable` computation. Upgrades must keep today's exact writer stack (see `purge_header.go:222` on the WebSocket-breakage class this package has already paid for). |
| +- **The wrapper owns a private `http.Header`.** Every other writer wrapper in this package embeds the writer below and shares one header map; `maybeStore` caches `rec.Header().Clone()` (edge.go:958). Compression headers go on the **real** writer's map only. |
| +- **Defer order.** `defer gw.finish()` is registered *after* the usage meter's `defer m.Record(...)`, so LIFO closes the gzip stream before the byte count is read. |
| +- **Threshold:** `minCompressSize = 1024`. A response whose `Content-Length` is present and below it is never compressed. |
| +- **Kill switch:** `CARLOS_EDGE_GZIP` — the values `off`, `false`, `0`, `no` (case-insensitive, trimmed) disable compression. Anything else, including unset, leaves it on. Read once in `New`. |
| +- **Missing `Content-Type` means identity, always.** Never sniff. |
| +- **Build gate before any commit that touches Go:** `go build ./...`, `go vet ./...`, `go test ./... -count=1`, `gofmt -l` clean for touched files. Sandbox env: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`. |
| +- **Every change lands through a PR.** Never merge to `main` locally. |
| + |
| +--- |
| + |
| +### Task 1: Accept-Encoding negotiation |
| + |
| +The request-side half of eligibility: does this client actually want gzip? Pure functions, no writer involved, so they can be tested exhaustively without a server. |
| + |
| +**Files:** |
| +- Create: `internal/edge/compress.go` |
| +- Test: `internal/edge/compress_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `joinHeader(h http.Header, name string) string` — already in `edge.go`, returns every line of a field comma-joined (a client may legally split `Accept-Encoding` across lines). |
| +- Produces: |
| + - `func acceptsGzip(h http.Header) bool` |
| + - `func qValue(params string) float64` |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +Create `internal/edge/compress_test.go`: |
| + |
| +```go |
| +package edge |
| + |
| +import ( |
| + "net/http" |
| + "testing" |
| +) |
| + |
| +// TestAcceptsGzip pins the q-value rules. The two that matter in production |
| +// are `gzip;q=0` (a client explicitly refusing gzip — some corporate proxies |
| +// and old curl invocations send it) and a bare `*`, which admits gzip only |
| +// when its own q is non-zero. A specific token always outranks the wildcard, |
| +// so `gzip;q=0, *` is a refusal however generous the wildcard is. |
| +func TestAcceptsGzip(t *testing.T) { |
| + for _, tc := range []struct { |
| + ae string |
| + want bool |
| + }{ |
| + {"", false}, |
| + {"gzip", true}, |
| + {"GZIP", true}, |
| + {"x-gzip", true}, |
| + {"gzip, deflate, br", true}, |
| + {" gzip ; q=0.5 ", true}, |
| + {"gzip;q=0", false}, |
| + {"gzip;q=0.0", false}, |
| + {"gzip;q=0, *", false}, |
| + {"identity", false}, |
| + {"deflate, br", false}, |
| + {"*", true}, |
| + {"*;q=0", false}, |
| + {"deflate, *;q=0.1", true}, |
| + } { |
| + h := http.Header{} |
| + if tc.ae != "" { |
| + h.Set("Accept-Encoding", tc.ae) |
| + } |
| + if got := acceptsGzip(h); got != tc.want { |
| + t.Errorf("acceptsGzip(%q) = %v, want %v", tc.ae, got, tc.want) |
| + } |
| + } |
| +} |
| + |
| +// TestAcceptsGzipReadsEveryLine: Accept-Encoding may arrive on several lines |
| +// and that is identical in meaning to one comma-joined line (RFC 9110 §5.3). |
| +// Reading it with Header.Get would see only the first, which is the exact bug |
| +// joinHeader exists to prevent for Vary and Cache-Control. |
| +func TestAcceptsGzipReadsEveryLine(t *testing.T) { |
| + h := http.Header{} |
| + h.Add("Accept-Encoding", "deflate") |
| + h.Add("Accept-Encoding", "gzip") |
| + if !acceptsGzip(h) { |
| + t.Fatal("gzip on a second Accept-Encoding line was invisible — read every line via joinHeader, not Get") |
| + } |
| +} |
| + |
| +func TestQValue(t *testing.T) { |
| + for _, tc := range []struct { |
| + params string |
| + want float64 |
| + }{ |
| + {"", 1}, |
| + {"q=1", 1}, |
| + {"q=0", 0}, |
| + {"q=0.5", 0.5}, |
| + {" Q = 0.25 ", 0.25}, |
| + {"q=bogus", 1}, // malformed q is ignored, not treated as a refusal |
| + {"charset=utf-8", 1}, |
| + } { |
| + if got := qValue(tc.params); got != tc.want { |
| + t.Errorf("qValue(%q) = %v, want %v", tc.params, got, tc.want) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run test to verify it fails** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/edge/ -run 'TestAcceptsGzip|TestQValue' -count=1` |
| +Expected: FAIL — `undefined: acceptsGzip`, `undefined: qValue`. |
| + |
| +- [ ] **Step 3: Write minimal implementation** |
| + |
| +Create `internal/edge/compress.go`: |
| + |
| +```go |
| +// Edge response compression: one gzip ResponseWriter wrapper applied at the |
| +// client-write boundary in Edge.ServeHTTP, so every serving class — cache |
| +// hits, proxied misses, static trees, the status host — compresses through a |
| +// single site. Apps ship identity bytes; transport is the platform's problem. |
| +// |
| +// The wrapper sits BELOW recordingResponseWriter and ABOVE countingWriter: |
| +// the cache keeps storing identity bodies (one variant, compressed after |
| +// retrieval on every delivery) and the usage meter counts what actually |
| +// crossed the wire. |
| +package edge |
| + |
| +import ( |
| + "net/http" |
| + "strconv" |
| + "strings" |
| +) |
| + |
| +// acceptsGzip reports whether r's Accept-Encoding admits gzip. Parsed as the |
| +// q-valued token list RFC 9110 §12.5.3 defines rather than a substring search: |
| +// "gzip;q=0" is a client REFUSING gzip, and answering it with a compressed |
| +// body is a broken response, not a missed optimization. |
| +// |
| +// A specific token outranks the wildcard, so the first gzip token decides — |
| +// "gzip;q=0, *" is a refusal. The wildcard only speaks when gzip is unnamed. |
| +// |
| +// Read through joinHeader, not Header.Get: a field may legally arrive on |
| +// several lines, and Get returns only the first (the same class of bug |
| +// joinHeader's doc comment describes for Vary and Cache-Control). |
| +func acceptsGzip(h http.Header) bool { |
| + var wildcard, wildcardOK bool |
| + for _, part := range strings.Split(joinHeader(h, "Accept-Encoding"), ",") { |
| + token, params, _ := strings.Cut(strings.TrimSpace(part), ";") |
| + switch strings.ToLower(strings.TrimSpace(token)) { |
| + case "gzip", "x-gzip": |
| + return qValue(params) > 0 |
| + case "*": |
| + wildcard, wildcardOK = true, qValue(params) > 0 |
| + } |
| + } |
| + if wildcard { |
| + return wildcardOK |
| + } |
| + return false |
| +} |
| + |
| +// qValue reads the q parameter out of one Accept-Encoding element's parameter |
| +// string ("q=0.5", "q=0"). Absent or malformed means q=1: a client whose |
| +// header we cannot fully parse is treated as accepting, which is the same |
| +// default the stdlib's own negotiation takes. |
| +func qValue(params string) float64 { |
| + for _, p := range strings.Split(params, ";") { |
| + name, val, ok := strings.Cut(strings.TrimSpace(p), "=") |
| + if !ok || !strings.EqualFold(strings.TrimSpace(name), "q") { |
| + continue |
| + } |
| + f, err := strconv.ParseFloat(strings.TrimSpace(val), 64) |
| + if err != nil { |
| + return 1 |
| + } |
| + return f |
| + } |
| + return 1 |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run test to verify it passes** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/edge/ -run 'TestAcceptsGzip|TestQValue' -count=1` |
| +Expected: PASS (`ok github.com/carlosframework/platform/internal/edge`) |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +gofmt -l internal/edge/compress.go internal/edge/compress_test.go |
| +git add internal/edge/compress.go internal/edge/compress_test.go |
| +git commit -m "edge: Accept-Encoding gzip negotiation |
| + |
| +q-valued token list, not a substring search: gzip;q=0 is a client |
| +refusing gzip, and a specific token outranks the wildcard." |
| +``` |
| + |
| +--- |
| + |
| +### Task 2: The eligibility predicate |
| + |
| +The response-side half: given the headers a handler produced, is this response worth compressing and safe to compress? Also pure — the request-side questions (Accept-Encoding, HEAD) are settled once at wrapper construction in Task 3, so this function reads headers and status only. |
| + |
| +**Files:** |
| +- Modify: `internal/edge/compress.go` |
| +- Test: `internal/edge/compress_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `hasCCDirective(cc, want string) bool` and `joinHeader` from `edge.go`. |
| +- Produces: |
| + - `const minCompressSize = 1024` |
| + - `func compressible(h http.Header, status int) bool` |
| + - `func compressibleType(ct string) bool` |
| + - `func weakenETag(h http.Header)` |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +Append to `internal/edge/compress_test.go`: |
| + |
| +```go |
| +// TestCompressibleEligibility walks the response-side gate. Each false case |
| +// is a response that would be WRONG or pointless to compress, not merely |
| +// unprofitable — a 206 range body, an already-encoded body, an origin that |
| +// said no-transform. |
| +func TestCompressibleEligibility(t *testing.T) { |
| + base := func() http.Header { |
| + h := http.Header{} |
| + h.Set("Content-Type", "text/css") |
| + return h |
| + } |
| + for _, tc := range []struct { |
| + name string |
| + mutate func(http.Header) |
| + status int |
| + want bool |
| + }{ |
| + {"plain css", func(http.Header) {}, 200, true}, |
| + {"html error page", func(h http.Header) { h.Set("Content-Type", "text/html; charset=utf-8") }, 404, true}, |
| + {"json api", func(h http.Header) { h.Set("Content-Type", "application/json") }, 200, true}, |
| + {"svg", func(h http.Header) { h.Set("Content-Type", "image/svg+xml") }, 200, true}, |
| + {"long body", func(h http.Header) { h.Set("Content-Length", "4096") }, 200, true}, |
| + {"unknown length", func(h http.Header) { h.Del("Content-Length") }, 200, true}, |
| + |
| + {"no content type", func(h http.Header) { h.Del("Content-Type") }, 200, false}, |
| + {"png", func(h http.Header) { h.Set("Content-Type", "image/png") }, 200, false}, |
| + {"event stream", func(h http.Header) { h.Set("Content-Type", "text/event-stream") }, 200, false}, |
| + {"already encoded", func(h http.Header) { h.Set("Content-Encoding", "gzip") }, 200, false}, |
| + {"no-transform", func(h http.Header) { h.Set("Cache-Control", "max-age=60, no-transform") }, 200, false}, |
| + {"short body", func(h http.Header) { h.Set("Content-Length", "512") }, 200, false}, |
| + {"exactly at threshold", func(h http.Header) { h.Set("Content-Length", "1024") }, 200, true}, |
| + {"partial content", func(h http.Header) { h.Set("Content-Range", "bytes 0-99/4096") }, 206, false}, |
| + {"no content", func(http.Header) {}, 204, false}, |
| + {"reset content", func(http.Header) {}, 205, false}, |
| + {"not modified", func(http.Header) {}, 304, false}, |
| + } { |
| + h := base() |
| + tc.mutate(h) |
| + if got := compressible(h, tc.status); got != tc.want { |
| + t.Errorf("%s: compressible(%v, %d) = %v, want %v", tc.name, h, tc.status, got, tc.want) |
| + } |
| + } |
| +} |
| + |
| +// TestCompressibleTypeSuffixes: a media type we have never heard of still |
| +// compresses when its structured suffix says it is text underneath. |
| +func TestCompressibleTypeSuffixes(t *testing.T) { |
| + for _, ct := range []string{ |
| + "application/manifest+json", "application/atom+xml", "application/ld+json", |
| + "application/vnd.api+json", "text/plain", "application/javascript", |
| + "application/wasm", "application/xml", "text/css;charset=UTF-8", |
| + } { |
| + if !compressibleType(ct) { |
| + t.Errorf("compressibleType(%q) = false, want true", ct) |
| + } |
| + } |
| + for _, ct := range []string{ |
| + "", "image/png", "video/mp4", "application/octet-stream", |
| + "application/zip", "font/woff2", "text/event-stream", |
| + } { |
| + if compressibleType(ct) { |
| + t.Errorf("compressibleType(%q) = true, want false", ct) |
| + } |
| + } |
| +} |
| + |
| +// TestWeakenETag: a gzipped body is a different representation, so its |
| +// validator must not claim byte-equality with the identity one. Weakening |
| +// keeps conditional requests working — the stdlib's If-None-Match check |
| +// compares weakly for GET — while dropping the strong-equality claim a |
| +// Range request would rely on. |
| +func TestWeakenETag(t *testing.T) { |
| + for _, tc := range []struct{ in, want string }{ |
| + {`"abc123"`, `W/"abc123"`}, |
| + {`W/"abc123"`, `W/"abc123"`}, // already weak: untouched |
| + {"", ""}, // absent: nothing to weaken |
| + } { |
| + h := http.Header{} |
| + if tc.in != "" { |
| + h.Set("ETag", tc.in) |
| + } |
| + weakenETag(h) |
| + if got := h.Get("ETag"); got != tc.want { |
| + t.Errorf("weakenETag(%q) -> %q, want %q", tc.in, got, tc.want) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run test to verify it fails** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/edge/ -run 'TestCompressible|TestWeakenETag' -count=1` |
| +Expected: FAIL — `undefined: compressible`, `undefined: compressibleType`, `undefined: weakenETag`. |
| + |
| +- [ ] **Step 3: Write minimal implementation** |
| + |
| +Append to `internal/edge/compress.go`: |
| + |
| +```go |
| +// minCompressSize is the Content-Length below which compressing is not worth |
| +// the CPU: gzip's own header and trailer are 18 bytes, and a sub-kilobyte |
| +// response often lands in one packet either way. A response whose length is |
| +// UNKNOWN (streamed, chunked) is compressed regardless — refusing those would |
| +// exempt most proxied HTML, which is the bulk of what there is to save. |
| +const minCompressSize = 1024 |
| + |
| +// compressible reports whether a response with these headers and this status |
| +// should be gzipped. The request-side questions — does the client accept |
| +// gzip, is this a HEAD — are settled once at wrapper construction |
| +// (newGzipWriter), so this reads the response alone. |
| +// |
| +// Every false below is a response that would be WRONG or pointless to |
| +// compress, not merely unprofitable: |
| +// |
| +// - 204/205 have no body; 304 has none and must keep the validators of the |
| +// representation it refers to; 206 is a byte range of a larger body, and |
| +// compressing a range makes the offsets it names meaningless. |
| +// - An existing Content-Encoding means the origin already encoded it. |
| +// Compressing again produces a body no client can decode without knowing |
| +// we did it. |
| +// - Cache-Control: no-transform is the origin forbidding exactly this |
| +// (RFC 9110 §5.5). |
| +// |
| +// A MISSING Content-Type is refused rather than sniffed. Sniffing is the |
| +// stdlib's job and it runs on the bytes we would have already compressed — |
| +// it would read gzip's magic number and label the response application/x-gzip. |
| +func compressible(h http.Header, status int) bool { |
| + switch status { |
| + case http.StatusNoContent, http.StatusResetContent, |
| + http.StatusPartialContent, http.StatusNotModified: |
| + return false |
| + } |
| + if h.Get("Content-Range") != "" { |
| + return false |
| + } |
| + if h.Get("Content-Encoding") != "" { |
| + return false |
| + } |
| + if hasCCDirective(joinHeader(h, "Cache-Control"), "no-transform") { |
| + return false |
| + } |
| + if n, err := strconv.Atoi(h.Get("Content-Length")); err == nil && n < minCompressSize { |
| + return false |
| + } |
| + return compressibleType(h.Get("Content-Type")) |
| +} |
| + |
| +// compressibleType reports whether a media type is text underneath. text/* is |
| +// the bulk of it; the structured suffixes (+json, +xml) catch types nobody has |
| +// enumerated yet, which is why they are here rather than a longer switch. |
| +// |
| +// text/event-stream is the one text type excluded: it is an open stream a |
| +// client consumes event by event, and buffering it inside a compressor is a |
| +// hang, not a saving. |
| +func compressibleType(ct string) bool { |
| + mt, _, _ := strings.Cut(ct, ";") |
| + mt = strings.ToLower(strings.TrimSpace(mt)) |
| + if mt == "" || mt == "text/event-stream" { |
| + return false |
| + } |
| + if strings.HasPrefix(mt, "text/") || |
| + strings.HasSuffix(mt, "+json") || strings.HasSuffix(mt, "+xml") { |
| + return true |
| + } |
| + switch mt { |
| + case "application/json", "application/javascript", "application/x-javascript", |
| + "application/xml", "application/wasm": |
| + return true |
| + } |
| + return false |
| +} |
| + |
| +// weakenETag downgrades a strong validator to a weak one, in place. A gzipped |
| +// body is a different representation of the same resource, so an ETag that |
| +// claims byte-for-byte equality with the identity body is a lie — and the one |
| +// thing that lie breaks is a Range request served against the wrong bytes. |
| +// Conditional requests keep working: If-None-Match compares weakly for GET, |
| +// so a client returning W/"abc" still matches the origin's "abc" and gets its |
| +// 304. |
| +func weakenETag(h http.Header) { |
| + if et := h.Get("ETag"); strings.HasPrefix(et, `"`) { |
| + h.Set("ETag", `W/`+et) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run test to verify it passes** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/edge/ -run 'TestCompressible|TestWeakenETag|TestAcceptsGzip|TestQValue' -count=1` |
| +Expected: PASS |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +gofmt -l internal/edge/compress.go internal/edge/compress_test.go |
| +git add internal/edge/compress.go internal/edge/compress_test.go |
| +git commit -m "edge: gzip eligibility predicate |
| + |
| +Header-only, decided once. A missing Content-Type is refused rather |
| +than sniffed: sniffing runs on bytes we would have already compressed |
| +and would label the response application/x-gzip." |
| +``` |
| + |
| +--- |
| + |
| +### Task 3: The gzipWriter wrapper |
| + |
| +The wrapper itself, tested directly against `httptest.NewRecorder()` — no Edge, no proxy, so each invariant fails for exactly one reason. The private header map is the whole point of this task. |
| + |
| +**Files:** |
| +- Modify: `internal/edge/compress.go` |
| +- Test: `internal/edge/compress_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `acceptsGzip`, `compressible`, `weakenETag` from Tasks 1-2. |
| +- Produces: |
| + - `type gzipWriter struct` — embeds `http.ResponseWriter`, implements `Header`, `WriteHeader`, `Write`, `Flush`, `Hijack`, and `finish()`. |
| + - `func newGzipWriter(w http.ResponseWriter, r *http.Request) *gzipWriter` — returns **nil** when this request can never be compressed. |
| + - `func (g *gzipWriter) finish()` — flushes and returns the pooled writer; the caller defers it. |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +Append to `internal/edge/compress_test.go` (add `bufio`, `compress/gzip`, `io`, `net`, `net/http/httptest`, `strings` to its imports): |
| + |
| +```go |
| +// hijackableRecorder is a recorder that also implements http.Hijacker, so the |
| +// forwarding test has something to forward TO. httptest.ResponseRecorder |
| +// deliberately does not implement it. |
| +type hijackableRecorder struct { |
| + *httptest.ResponseRecorder |
| + hijacked bool |
| +} |
| + |
| +func (h *hijackableRecorder) Hijack() (net.Conn, *bufio.ReadWriter, error) { |
| + h.hijacked = true |
| + return nil, nil, nil |
| +} |
| + |
| +// gzipReq builds a request that accepts gzip. |
| +func gzipReq(method, url string) *http.Request { |
| + r := httptest.NewRequest(method, url, nil) |
| + r.Header.Set("Accept-Encoding", "gzip") |
| + return r |
| +} |
| + |
| +// TestGzipWriterKeepsCompressionHeadersOffTheCallerMap is THE test this design |
| +// exists for. Every writer wrapper in this package embeds the writer below, so |
| +// Header() returns one shared map all the way down — and maybeStore caches |
| +// rec.Header().Clone(). If the compression headers landed on that shared map, |
| +// the cache would store an IDENTITY body labelled Content-Encoding: gzip and |
| +// serve it to every later visitor as an undecodable response. |
| +func TestGzipWriterKeepsCompressionHeadersOffTheCallerMap(t *testing.T) { |
| + rec := httptest.NewRecorder() |
| + g := newGzipWriter(rec, gzipReq("GET", "http://app.test/tokens.css")) |
| + if g == nil { |
| + t.Fatal("newGzipWriter returned nil for a gzip-accepting GET") |
| + } |
| + g.Header().Set("Content-Type", "text/css") |
| + g.Header().Set("ETag", `"abc123"`) |
| + g.Header().Set("Content-Length", "4000") |
| + g.WriteHeader(http.StatusOK) |
| + body := strings.Repeat("a{color:red}\n", 400) |
| + io.WriteString(g, body) |
| + g.finish() |
| + |
| + if got := g.Header().Get("Content-Encoding"); got != "" { |
| + t.Errorf("caller-visible Content-Encoding = %q, want empty — the recorder above would cache an identity body labelled gzip", got) |
| + } |
| + if got := g.Header().Get("ETag"); got != `"abc123"` { |
| + t.Errorf("caller-visible ETag = %q, want the strong tag untouched (the cached body IS the identity one)", got) |
| + } |
| + if got := g.Header().Get("Content-Length"); got != "4000" { |
| + t.Errorf("caller-visible Content-Length = %q, want it untouched", got) |
| + } |
| + |
| + if got := rec.Header().Get("Content-Encoding"); got != "gzip" { |
| + t.Errorf("client Content-Encoding = %q, want gzip", got) |
| + } |
| + if got := rec.Header().Get("Content-Length"); got != "" { |
| + t.Errorf("client Content-Length = %q, want it dropped — it described the identity body", got) |
| + } |
| + if got := rec.Header().Get("Vary"); got != "Accept-Encoding" { |
| + t.Errorf("client Vary = %q, want Accept-Encoding", got) |
| + } |
| + if got := rec.Header().Get("ETag"); got != `W/"abc123"` { |
| + t.Errorf("client ETag = %q, want the weakened form", got) |
| + } |
| + |
| + zr, err := gzip.NewReader(rec.Body) |
| + if err != nil { |
| + t.Fatalf("client body is not gzip: %v", err) |
| + } |
| + got, err := io.ReadAll(zr) |
| + if err != nil { |
| + t.Fatalf("read gzip body: %v", err) |
| + } |
| + if string(got) != body { |
| + t.Fatalf("decompressed body differs from what the handler wrote") |
| + } |
| + if rec.Body.Len() >= len(body) { |
| + t.Fatalf("compressed size %d >= identity size %d", rec.Body.Len(), len(body)) |
| + } |
| +} |
| + |
| +// TestGzipWriterNilForRequestsThatCanNeverCompress: construction is where the |
| +// request-side questions are settled, once. A nil wrapper means ServeHTTP |
| +// leaves the writer stack exactly as it is today. |
| +func TestGzipWriterNilForRequestsThatCanNeverCompress(t *testing.T) { |
| + if g := newGzipWriter(httptest.NewRecorder(), httptest.NewRequest("GET", "http://app.test/", nil)); g != nil { |
| + t.Error("no Accept-Encoding must yield a nil wrapper") |
| + } |
| + if g := newGzipWriter(httptest.NewRecorder(), gzipReq("HEAD", "http://app.test/big.css")); g != nil { |
| + t.Error("HEAD must yield a nil wrapper — there is no body to compress, and its truthful Content-Length is the whole value of the response") |
| + } |
| +} |
| + |
| +// TestGzipWriterPassesThroughIneligibleResponses: an ineligible response must |
| +// be byte-identical to the unwrapped one, headers included. |
| +func TestGzipWriterPassesThroughIneligibleResponses(t *testing.T) { |
| + rec := httptest.NewRecorder() |
| + g := newGzipWriter(rec, gzipReq("GET", "http://app.test/logo.png")) |
| + g.Header().Set("Content-Type", "image/png") |
| + g.Header().Set("Content-Length", "9") |
| + g.WriteHeader(http.StatusOK) |
| + io.WriteString(g, "PNG-BYTES") |
| + g.finish() |
| + |
| + if got := rec.Header().Get("Content-Encoding"); got != "" { |
| + t.Errorf("Content-Encoding = %q on an image, want empty", got) |
| + } |
| + if got := rec.Header().Get("Content-Length"); got != "9" { |
| + t.Errorf("Content-Length = %q, want it preserved on a pass-through", got) |
| + } |
| + if got := rec.Body.String(); got != "PNG-BYTES" { |
| + t.Errorf("body = %q, want the identity bytes", got) |
| + } |
| +} |
| + |
| +// TestGzipWriterSilentHandlerStillDeliversHeaders: the handler set headers and |
| +// returned without ever writing. Those headers live in the PRIVATE map, so |
| +// without a sync in finish() the stdlib would emit a bare 200 with none of |
| +// them — X-Carlos-Version included. |
| +func TestGzipWriterSilentHandlerStillDeliversHeaders(t *testing.T) { |
| + rec := httptest.NewRecorder() |
| + g := newGzipWriter(rec, gzipReq("GET", "http://app.test/")) |
| + g.Header().Set("X-Carlos-Version", "abc123") |
| + g.finish() |
| + |
| + if got := rec.Header().Get("X-Carlos-Version"); got != "abc123" { |
| + t.Fatalf("X-Carlos-Version = %q, want abc123 — a handler that wrote nothing still lost every header it set", got) |
| + } |
| +} |
| + |
| +// TestGzipWriter1xxDoesNotLatch mirrors recordingResponseWriter.WriteHeader: a |
| +// 103 Early Hints block precedes the final status. Latching it would freeze |
| +// the compression decision on a status that has no body, and the real response |
| +// would then be delivered uncompressed — or worse, with the hints lost, since |
| +// they live in the private map. |
| +func TestGzipWriter1xxDoesNotLatch(t *testing.T) { |
| + rec := httptest.NewRecorder() |
| + g := newGzipWriter(rec, gzipReq("GET", "http://app.test/")) |
| + g.Header().Set("Link", "</tokens.css>; rel=preload") |
| + g.WriteHeader(http.StatusEarlyHints) |
| + |
| + if got := rec.Header().Get("Link"); got != "</tokens.css>; rel=preload" { |
| + t.Errorf("early-hints Link = %q, want it forwarded — it was set on the private map", got) |
| + } |
| + |
| + g.Header().Del("Link") |
| + g.Header().Set("Content-Type", "text/html") |
| + g.WriteHeader(http.StatusOK) |
| + io.WriteString(g, strings.Repeat("<p>hi</p>", 200)) |
| + g.finish() |
| + |
| + if got := rec.Header().Get("Content-Encoding"); got != "gzip" { |
| + t.Errorf("Content-Encoding = %q after a 103, want gzip — the 1xx latched the decision", got) |
| + } |
| + if got := rec.Header().Get("Link"); got != "" { |
| + t.Errorf("Link = %q, want empty — the final header block replaces the informational one", got) |
| + } |
| +} |
| + |
| +// TestGzipWriterFlushReachesTheClient: a flushed response must actually leave |
| +// the compressor, not sit in its buffer until the handler returns. SSE never |
| +// compresses (content type), but a streamed HTML page does. |
| +func TestGzipWriterFlushReachesTheClient(t *testing.T) { |
| + rec := httptest.NewRecorder() |
| + g := newGzipWriter(rec, gzipReq("GET", "http://app.test/stream")) |
| + g.Header().Set("Content-Type", "text/html") |
| + g.WriteHeader(http.StatusOK) |
| + io.WriteString(g, strings.Repeat("<p>chunk</p>", 100)) |
| + g.Flush() |
| + |
| + if rec.Body.Len() == 0 { |
| + t.Fatal("Flush left the bytes inside the compressor — a streaming response would hang until the handler returned") |
| + } |
| + if !rec.Flushed { |
| + t.Fatal("Flush did not reach the writer below") |
| + } |
| + g.finish() |
| +} |
| + |
| +// TestGzipWriterForwardsHijack: the upgrade path returns before this wrapper |
| +// is ever built, so this is not a hot path — it exists so the wrapper is never |
| +// itself the reason a hijack fails, matching the convention countingWriter and |
| +// recordingResponseWriter already follow. |
| +func TestGzipWriterForwardsHijack(t *testing.T) { |
| + rec := &hijackableRecorder{ResponseRecorder: httptest.NewRecorder()} |
| + g := newGzipWriter(rec, gzipReq("GET", "http://app.test/")) |
| + if _, _, err := g.Hijack(); err != nil { |
| + t.Fatalf("Hijack: %v", err) |
| + } |
| + if !rec.hijacked { |
| + t.Fatal("Hijack did not reach the writer below") |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run test to verify it fails** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/edge/ -run TestGzipWriter -count=1` |
| +Expected: FAIL — `undefined: newGzipWriter`. |
| + |
| +- [ ] **Step 3: Write minimal implementation** |
| + |
| +Append to `internal/edge/compress.go`, and extend its imports to |
| +`bufio`, `compress/gzip`, `fmt`, `io`, `net`, `net/http`, `strconv`, `strings`, `sync`: |
| + |
| +```go |
| +// gzipPool recycles compressors. We compress on every delivery, cache hits |
| +// included, so a fresh gzip.Writer per response would be ~1.2 MB/s of garbage |
| +// on a busy edge for no reason. |
| +var gzipPool = sync.Pool{New: func() any { |
| + w, _ := gzip.NewWriterLevel(io.Discard, gzip.DefaultCompression) |
| + return w |
| +}} |
| + |
| +// gzipWriter compresses an eligible response on its way to the client. |
| +// |
| +// It owns a PRIVATE header map. Every other ResponseWriter wrapper in this |
| +// package embeds the writer below, so Header() returns one shared map from the |
| +// handler all the way down to the connection — and maybeStore caches |
| +// rec.Header().Clone(). Were the compression headers set on that shared map, |
| +// the cache would store an identity body labelled Content-Encoding: gzip and |
| +// hand it to every later visitor. So callers above read and write hdr, and the |
| +// edits that describe the COMPRESSED representation go on the real writer's |
| +// map alone. |
| +// |
| +// Deliberately does NOT implement io.ReaderFrom: io.Copy (and so |
| +// http.ServeFile) falls back to Write, which is what compresses. |
| +type gzipWriter struct { |
| + http.ResponseWriter |
| + hdr http.Header |
| + gz *gzip.Writer // nil unless this response is being compressed |
| + wroteHdr bool |
| +} |
| + |
| +// newGzipWriter returns a wrapper for r, or nil when this request can never |
| +// produce a compressed response — no wrapper at all, so the writer stack is |
| +// exactly what it is today. The request-side questions are settled here, once: |
| +// everything after depends only on the response. |
| +// |
| +// HEAD is excluded rather than compressed-with-no-body: there is nothing to |
| +// compress, and a HEAD's truthful Content-Length is the entire value of the |
| +// response. The accepted cost is that a HEAD advertises the identity length |
| +// while a GET of the same URL returns gzip with no length at all. |
| +func newGzipWriter(w http.ResponseWriter, r *http.Request) *gzipWriter { |
| + if r.Method == http.MethodHead || !acceptsGzip(r.Header) { |
| + return nil |
| + } |
| + return &gzipWriter{ResponseWriter: w, hdr: http.Header{}} |
| +} |
| + |
| +// Header returns the PRIVATE map. See the type's doc comment: this is what |
| +// keeps the response cache storing identity bodies. |
| +func (g *gzipWriter) Header() http.Header { return g.hdr } |
| + |
| +// syncHeader replaces the real writer's header block with the private one. |
| +// Wholesale, not a merge: a handler that DELETED a header between blocks (the |
| +// shape the purge-on-103 test exercises) must see that deletion reach the |
| +// client, which a merge would silently undo. |
| +func (g *gzipWriter) syncHeader() { |
| + real := g.ResponseWriter.Header() |
| + for k := range real { |
| + delete(real, k) |
| + } |
| + for k, v := range g.hdr { |
| + real[k] = v |
| + } |
| +} |
| + |
| +func (g *gzipWriter) WriteHeader(status int) { |
| + // 1xx blocks (103 Early Hints, forwarded by httputil.ReverseProxy) precede |
| + // the final status. Sync and forward without latching — the same shape |
| + // recordingResponseWriter.WriteHeader uses, for the same reason: latching |
| + // one would decide compression on a status that has no body. |
| + if status < 200 { |
| + g.syncHeader() |
| + g.ResponseWriter.WriteHeader(status) |
| + return |
| + } |
| + if g.wroteHdr { |
| + g.ResponseWriter.WriteHeader(status) // superfluous; let the stdlib log it |
| + return |
| + } |
| + g.wroteHdr = true |
| + g.syncHeader() |
| + if compressible(g.hdr, status) { |
| + h := g.ResponseWriter.Header() |
| + h.Set("Content-Encoding", "gzip") |
| + h.Del("Content-Length") // it described the identity body |
| + h.Add("Vary", "Accept-Encoding") |
| + weakenETag(h) |
| + g.gz = gzipPool.Get().(*gzip.Writer) |
| + g.gz.Reset(g.ResponseWriter) |
| + } |
| + g.ResponseWriter.WriteHeader(status) |
| +} |
| + |
| +func (g *gzipWriter) Write(p []byte) (int, error) { |
| + if !g.wroteHdr { |
| + g.WriteHeader(http.StatusOK) |
| + } |
| + if g.gz != nil { |
| + return g.gz.Write(p) |
| + } |
| + return g.ResponseWriter.Write(p) |
| +} |
| + |
| +// finish closes the compressor and returns it to the pool. ServeHTTP defers |
| +// it, AFTER the usage meter's defer, so LIFO runs this one first and the |
| +// metered byte count includes gzip's trailer. |
| +// |
| +// The no-header case is not a no-op: a handler that returned without ever |
| +// writing set its headers on the private map, and nothing has copied them |
| +// down. Without this sync the stdlib emits a bare 200 with none of them. |
| +func (g *gzipWriter) finish() { |
| + if !g.wroteHdr { |
| + g.syncHeader() |
| + return |
| + } |
| + if g.gz == nil { |
| + return |
| + } |
| + g.gz.Close() |
| + g.gz.Reset(io.Discard) // don't pin this request's writer inside the pool |
| + gzipPool.Put(g.gz) |
| + g.gz = nil |
| +} |
| + |
| +// Flush pushes the compressor's buffered bytes out and then flushes the writer |
| +// below — needed so a streamed response reaches the client incrementally |
| +// instead of buffering until the handler returns. |
| +func (g *gzipWriter) Flush() { |
| + if g.gz != nil { |
| + g.gz.Flush() |
| + } |
| + if f, ok := g.ResponseWriter.(http.Flusher); ok { |
| + f.Flush() |
| + } |
| +} |
| + |
| +// Hijack forwards to the writer below, when it supports it — the same |
| +// optional-interface pattern countingWriter and recordingResponseWriter use. |
| +// ServeHTTP returns on the upgrade path before this wrapper is constructed, so |
| +// this is not expected on the hot path; it exists so the wrapper is never |
| +// itself the reason a hijack fails. |
| +func (g *gzipWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) { |
| + hj, ok := g.ResponseWriter.(http.Hijacker) |
| + if !ok { |
| + return nil, nil, fmt.Errorf("edge: underlying ResponseWriter does not support http.Hijacker") |
| + } |
| + return hj.Hijack() |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run test to verify it passes** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/edge/ -run 'TestGzip|TestCompressible|TestAcceptsGzip|TestQValue|TestWeakenETag' -count=1` |
| +Expected: PASS |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +gofmt -l internal/edge/compress.go internal/edge/compress_test.go |
| +git add internal/edge/compress.go internal/edge/compress_test.go |
| +git commit -m "edge: gzip ResponseWriter with a private header map |
| + |
| +The wrapper owns its own http.Header: every other wrapper here embeds |
| +the writer below, so Header() is one shared map and maybeStore caches |
| +rec.Header().Clone() — setting Content-Encoding on it would store an |
| +identity body labelled gzip and serve it to every later visitor." |
| +``` |
| + |
| +--- |
| + |
| +### Task 4: Wire it into Edge.ServeHTTP |
| + |
| +The wrap site, the kill switch, and the two invariants that dictated the ordering: the cache stores identity, the meter counts compressed. |
| + |
| +**Files:** |
| +- Modify: `internal/edge/edge.go` — the `Edge` struct (near line 138), `New` (line 181), `ServeHTTP` (after the `isUpgradeRequest` block, ~line 806) |
| +- Modify: `internal/edge/compress.go` — add `gzipEnabled()` |
| +- Test: `internal/edge/compress_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `newGzipWriter`, `(*gzipWriter).finish` from Task 3; `EnableCache(*Cache)`, `NewCache(maxEntries, maxBodyBytes int) *Cache`, `EnableUsage(*UsageMeter)`, `(*Cache).get(CacheKey)`, `openReg(t)` from `edge_test.go`. |
| +- Produces: |
| + - `Edge.gzipOn bool` field |
| + - `func gzipEnabled() bool` |
| + - `func edgeWithBackend(t *testing.T, h http.Handler) *Edge` (test helper, used again in Task 5) |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +Append to `internal/edge/compress_test.go` (imports gain `github.com/carlosframework/platform/internal/registry`): |
| + |
| +```go |
| +// edgeWithBackend builds an Edge with one proxied route, "app.test", pointed |
| +// at h. Driven with httptest.NewRecorder + httptest.NewRequest rather than a |
| +// real client: http.Transport adds its own Accept-Encoding and transparently |
| +// decompresses what comes back, which would hide exactly what these tests are |
| +// checking. |
| +func edgeWithBackend(t *testing.T, h http.Handler) *Edge { |
| + t.Helper() |
| + backend := httptest.NewServer(h) |
| + t.Cleanup(backend.Close) |
| + reg := openReg(t) |
| + if err := reg.Upsert(registry.Route{ |
| + Host: "app.test", Kind: "service", Status: "ready", |
| + Addr: backend.Listener.Addr().String(), |
| + }); err != nil { |
| + t.Fatalf("upsert: %v", err) |
| + } |
| + e, err := New(reg, nil) |
| + if err != nil { |
| + t.Fatalf("New: %v", err) |
| + } |
| + return e |
| +} |
| + |
| +// cssBackend answers every request with a cacheable stylesheet of n bytes. |
| +func cssBackend(n int) http.Handler { |
| + body := strings.Repeat("a{color:red}\n", n/13+1)[:n] |
| + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + w.Header().Set("Content-Type", "text/css") |
| + w.Header().Set("Cache-Control", "max-age=600") |
| + io.WriteString(w, body) |
| + }) |
| +} |
| + |
| +// TestEdgeCompressesProxiedAssets is the payoff: an app ships identity bytes |
| +// and the edge puts them on the wire compressed. |
| +func TestEdgeCompressesProxiedAssets(t *testing.T) { |
| + e := edgeWithBackend(t, cssBackend(4096)) |
| + rec := httptest.NewRecorder() |
| + e.ServeHTTP(rec, gzipReq("GET", "http://app.test/tokens.css")) |
| + |
| + if got := rec.Header().Get("Content-Encoding"); got != "gzip" { |
| + t.Fatalf("Content-Encoding = %q, want gzip", got) |
| + } |
| + if got := rec.Header().Get("Vary"); !strings.Contains(got, "Accept-Encoding") { |
| + t.Errorf("Vary = %q, want it to name Accept-Encoding", got) |
| + } |
| + zr, err := gzip.NewReader(rec.Body) |
| + if err != nil { |
| + t.Fatalf("body is not gzip: %v", err) |
| + } |
| + got, err := io.ReadAll(zr) |
| + if err != nil { |
| + t.Fatalf("read gzip body: %v", err) |
| + } |
| + if len(got) != 4096 { |
| + t.Fatalf("decompressed %d bytes, want 4096", len(got)) |
| + } |
| +} |
| + |
| +// TestEdgeCacheStoresIdentityBodies is the invariant the whole writer ordering |
| +// exists to hold. The recorder sits ABOVE the compressor, so what lands in the |
| +// cache is the identity body with the app's own headers — one variant, not one |
| +// per encoding — and a client that cannot decode gzip is still served correctly |
| +// from that same entry. |
| +func TestEdgeCacheStoresIdentityBodies(t *testing.T) { |
| + e := edgeWithBackend(t, cssBackend(4096)) |
| + e.EnableCache(NewCache(10, 1<<20)) |
| + |
| + rec := httptest.NewRecorder() |
| + e.ServeHTTP(rec, gzipReq("GET", "http://app.test/tokens.css")) // miss |
| + if got := rec.Header().Get("Content-Encoding"); got != "gzip" { |
| + t.Fatalf("miss: Content-Encoding = %q, want gzip", got) |
| + } |
| + |
| + stored, ok := e.cache.get(CacheKey{Host: "app.test", Method: "GET", Path: "/tokens.css"}) |
| + if !ok { |
| + t.Fatal("the response was not cached — this test cannot say anything about what was stored") |
| + } |
| + if ce := stored.Header.Get("Content-Encoding"); ce != "" { |
| + t.Errorf("cached entry carries Content-Encoding %q: an identity body labelled gzip, served to every later visitor as an undecodable response", ce) |
| + } |
| + if v := stored.Header.Get("Vary"); strings.Contains(v, "Accept-Encoding") { |
| + t.Errorf("cached entry carries Vary %q: the edge's own Vary belongs below the recorder, not in the cache key", v) |
| + } |
| + if len(stored.Body) != 4096 { |
| + t.Errorf("cached body is %d bytes, want the 4096-byte identity body", len(stored.Body)) |
| + } |
| + |
| + hit := httptest.NewRecorder() |
| + e.ServeHTTP(hit, gzipReq("GET", "http://app.test/tokens.css")) |
| + if got := hit.Header().Get("Content-Encoding"); got != "gzip" { |
| + t.Fatalf("cache hit: Content-Encoding = %q, want gzip — a hit is a delivery like any other", got) |
| + } |
| + zr, err := gzip.NewReader(hit.Body) |
| + if err != nil { |
| + t.Fatalf("cache hit body is not gzip: %v", err) |
| + } |
| + if b, err := io.ReadAll(zr); err != nil || len(b) != 4096 { |
| + t.Fatalf("cache hit decompressed to %d bytes (err %v), want 4096", len(b), err) |
| + } |
| + |
| + plain := httptest.NewRecorder() |
| + e.ServeHTTP(plain, httptest.NewRequest("GET", "http://app.test/tokens.css", nil)) |
| + if got := plain.Header().Get("Content-Encoding"); got != "" { |
| + t.Errorf("identity client got Content-Encoding %q from the shared entry", got) |
| + } |
| + if plain.Body.Len() != 4096 { |
| + t.Errorf("identity client got %d bytes, want the 4096-byte identity body", plain.Body.Len()) |
| + } |
| +} |
| + |
| +// TestEdgeMetersCompressedBytes: a tenant is billed for what crossed the wire. |
| +// The counter sits BELOW the compressor and finish() runs before Record — the |
| +// equality below is what pins that defer order, since gzip buffers until Close |
| +// and a reversed order would meter a fraction of the bytes. |
| +func TestEdgeMetersCompressedBytes(t *testing.T) { |
| + e := edgeWithBackend(t, cssBackend(8192)) |
| + m := NewUsageMeter() |
| + e.EnableUsage(m) |
| + |
| + rec := httptest.NewRecorder() |
| + e.ServeHTTP(rec, gzipReq("GET", "http://app.test/tokens.css")) |
| + |
| + row, ok := m.Drain()["app.test"] |
| + if !ok { |
| + t.Fatal("no usage row for app.test") |
| + } |
| + if row.BytesOut != int64(rec.Body.Len()) { |
| + t.Fatalf("metered %d bytes, client received %d — finish() must run before Record (defer order) or the trailer is missed", row.BytesOut, rec.Body.Len()) |
| + } |
| + if row.BytesOut >= 8192 { |
| + t.Fatalf("metered %d bytes for an 8192-byte identity body — the meter is counting bytes the edge did not send", row.BytesOut) |
| + } |
| +} |
| + |
| +// TestEdgeGzipKillSwitch: CARLOS_EDGE_GZIP=off is the rollback that needs no |
| +// redeploy, so it must produce today's bytes exactly. |
| +func TestEdgeGzipKillSwitch(t *testing.T) { |
| + t.Setenv("CARLOS_EDGE_GZIP", "off") |
| + e := edgeWithBackend(t, cssBackend(4096)) |
| + rec := httptest.NewRecorder() |
| + e.ServeHTTP(rec, gzipReq("GET", "http://app.test/tokens.css")) |
| + |
| + if got := rec.Header().Get("Content-Encoding"); got != "" { |
| + t.Errorf("Content-Encoding = %q with the kill switch off, want empty", got) |
| + } |
| + if got := rec.Header().Get("Vary"); strings.Contains(got, "Accept-Encoding") { |
| + t.Errorf("Vary = %q with the kill switch off — nothing may hint at a variant that is never produced", got) |
| + } |
| + if rec.Body.Len() != 4096 { |
| + t.Errorf("body = %d bytes, want the 4096-byte identity body", rec.Body.Len()) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run test to verify it fails** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/edge/ -run TestEdge -count=1` |
| +Expected: FAIL — `TestEdgeCompressesProxiedAssets` reports `Content-Encoding = "" want gzip` (nothing is wired yet). |
| + |
| +- [ ] **Step 3: Write minimal implementation** |
| + |
| +3a. Append to `internal/edge/compress.go` (imports gain `os`): |
| + |
| +```go |
| +// gzipEnabled reads the CARLOS_EDGE_GZIP kill switch. Compression is ON unless |
| +// the variable names a false value — the rollback path is an env change and a |
| +// restart, not a platform binary roll. Matches the CARLOS_* env idiom in |
| +// certcache.go and dns.go. |
| +func gzipEnabled() bool { |
| + switch strings.ToLower(strings.TrimSpace(os.Getenv("CARLOS_EDGE_GZIP"))) { |
| + case "off", "false", "0", "no": |
| + return false |
| + } |
| + return true |
| +} |
| +``` |
| + |
| +3b. In `internal/edge/edge.go`, add to the `Edge` struct (beside the `cache` field, ~line 138): |
| + |
| +```go |
| + // gzipOn is the CARLOS_EDGE_GZIP kill switch, read once in New. When |
| + // false, ServeHTTP never constructs a gzipWriter and the writer stack is |
| + // byte-for-byte what it was before compression existed (compress.go). |
| + gzipOn bool |
| +``` |
| + |
| +3c. In `New` (edge.go:181), set it: |
| + |
| +```go |
| + e := &Edge{reg: reg, act: act, probers: newProberSet(), gzipOn: gzipEnabled()} |
| +``` |
| + |
| +3d. In `ServeHTTP`, immediately after the `if isUpgradeRequest(r) { ... }` block and before the `cacheable := ...` line: |
| + |
| +```go |
| + // Response compression (compress.go). Placed AFTER the upgrade branch, so |
| + // an upgrading request keeps the exact writer stack it has always had — |
| + // purge_header.go's contract records what wrapping the raw ResponseWriter |
| + // on that path has already cost this package — and BEFORE the cache check, |
| + // so a cache hit compresses on replay like any other delivery. |
| + // |
| + // The recorders constructed below wrap THIS writer, so they record |
| + // identity bytes and the cache keeps storing identity bodies; the usage |
| + // meter's countingWriter is below it, so the metered count is the |
| + // compressed one. This defer is registered after the meter's, so LIFO |
| + // closes the gzip stream — trailer included — before Record reads it. |
| + if e.gzipOn { |
| + if gw := newGzipWriter(w, r); gw != nil { |
| + w = gw |
| + defer gw.finish() |
| + } |
| + } |
| +``` |
| + |
| +- [ ] **Step 4: Run test to verify it passes** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/edge/ -run TestEdge -count=1` |
| +Expected: PASS |
| + |
| +Then the whole package: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/edge/ -count=1` |
| +Expected: PASS. If an existing test fails asserting an exact `Content-Length` or byte count on a compressible body of 1024+ bytes, that is the feature working — update the test to decompress or to assert the identity length via a non-gzip request, and say so in the commit message. Do **not** weaken the feature to keep an assertion. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +gofmt -l internal/edge/compress.go internal/edge/edge.go internal/edge/compress_test.go |
| +git add internal/edge/compress.go internal/edge/edge.go internal/edge/compress_test.go |
| +git commit -m "edge: compress responses at the client-write boundary |
| + |
| +One wrap site in ServeHTTP after the upgrade branch and before the |
| +cache check, so cache hits, proxied misses, static trees and the status |
| +host all pass through it. The cache keeps storing identity bodies and |
| +the usage meter counts the compressed byte count. |
| + |
| +CARLOS_EDGE_GZIP=off disables it without a binary roll." |
| +``` |
| + |
| +--- |
| + |
| +### Task 5: Regression battery, full gate, PR |
| + |
| +Everything the wrap site could plausibly have broken, plus the serving classes that reach it through a different door. Then the gate and the PR. |
| + |
| +**Files:** |
| +- Test: `internal/edge/compress_test.go` |
| +- Modify: `CLAUDE.md` (one bullet under "Rules this repo has paid for") |
| + |
| +**Interfaces:** |
| +- Consumes: `edgeWithBackend`, `gzipReq`, `cssBackend` from Task 4; `upgradeEchoBackend(t, tag) string`, `openReg(t)` and `testEdgeWithStore(t, rt)` from `edge_test.go`; `testKeys(t)` and `shipSite(t, store, app, version, files)` from `static_test.go`. |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +Append to `internal/edge/compress_test.go` (imports gain `bufio`, `context`, `fmt`, `net`, `path/filepath`, `time`, and `github.com/carlosframework/platform/internal/release`): |
| + |
| +```go |
| +// TestUpgradeIsUntouchedByCompression: the wrap site is after the upgrade |
| +// branch precisely so a handshake keeps the writer stack it has always had. |
| +// A gzip-accepting client (every browser) must still get its 101, and bytes |
| +// must still flow both ways after the switch. |
| +func TestUpgradeIsUntouchedByCompression(t *testing.T) { |
| + addr := upgradeEchoBackend(t, "ws") |
| + reg := openReg(t) |
| + if err := reg.Upsert(registry.Route{ |
| + Host: "ws.test", Kind: "service", Status: "ready", Addr: addr, |
| + }); err != nil { |
| + t.Fatalf("upsert: %v", err) |
| + } |
| + e, err := New(reg, nil) |
| + if err != nil { |
| + t.Fatalf("New: %v", err) |
| + } |
| + srv := httptest.NewServer(e) |
| + defer srv.Close() |
| + |
| + c, err := net.Dial("tcp", strings.TrimPrefix(srv.URL, "http://")) |
| + if err != nil { |
| + t.Fatalf("dial edge: %v", err) |
| + } |
| + defer c.Close() |
| + if err := c.SetDeadline(time.Now().Add(10 * time.Second)); err != nil { |
| + t.Fatalf("set deadline: %v", err) |
| + } |
| + if _, err := fmt.Fprint(c, "GET /socket HTTP/1.1\r\nHost: ws.test\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nAccept-Encoding: gzip\r\n\r\n"); err != nil { |
| + t.Fatalf("write handshake: %v", err) |
| + } |
| + br := bufio.NewReader(c) |
| + resp, err := http.ReadResponse(br, nil) |
| + if err != nil { |
| + t.Fatalf("read handshake response: %v", err) |
| + } |
| + if resp.StatusCode != http.StatusSwitchingProtocols { |
| + t.Fatalf("handshake status = %d, want 101 — an Accept-Encoding header must not divert an upgrade through the compressor", resp.StatusCode) |
| + } |
| + if _, err := fmt.Fprint(c, "ping\n"); err != nil { |
| + t.Fatalf("write post-upgrade frame: %v", err) |
| + } |
| + line, err := br.ReadString('\n') |
| + if err != nil { |
| + t.Fatalf("read post-upgrade frame: %v", err) |
| + } |
| + if line != "ws:ping\n" { |
| + t.Fatalf("post-upgrade echo = %q, want %q — bytes must flow both ways after the switch", line, "ws:ping\n") |
| + } |
| +} |
| + |
| +// TestAlreadyEncodedUpstreamPassesThrough: an app that compresses for itself |
| +// must not be compressed again into a body nobody can decode. |
| +func TestAlreadyEncodedUpstreamPassesThrough(t *testing.T) { |
| + var raw []byte |
| + e := edgeWithBackend(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + var buf strings.Builder |
| + zw := gzip.NewWriter(&buf) |
| + io.WriteString(zw, strings.Repeat("already compressed\n", 200)) |
| + zw.Close() |
| + raw = []byte(buf.String()) |
| + w.Header().Set("Content-Type", "text/plain") |
| + w.Header().Set("Content-Encoding", "gzip") |
| + w.Write(raw) |
| + })) |
| + rec := httptest.NewRecorder() |
| + e.ServeHTTP(rec, gzipReq("GET", "http://app.test/pre.txt")) |
| + |
| + if got := rec.Header().Values("Content-Encoding"); len(got) != 1 || got[0] != "gzip" { |
| + t.Fatalf("Content-Encoding = %v, want exactly one gzip", got) |
| + } |
| + zr, err := gzip.NewReader(rec.Body) |
| + if err != nil { |
| + t.Fatalf("body did not survive as single-encoded gzip: %v", err) |
| + } |
| + body, err := io.ReadAll(zr) |
| + if err != nil { |
| + t.Fatalf("read body: %v", err) |
| + } |
| + if !strings.HasPrefix(string(body), "already compressed") { |
| + t.Fatalf("body = %.40q, want the upstream's own gzip decoded in ONE pass", body) |
| + } |
| +} |
| + |
| +// TestEventStreamIsNotCompressed: SSE is an open stream consumed event by |
| +// event. Buffering it inside a compressor is a hang, not a saving. |
| +func TestEventStreamIsNotCompressed(t *testing.T) { |
| + e := edgeWithBackend(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + w.Header().Set("Content-Type", "text/event-stream") |
| + w.WriteHeader(http.StatusOK) |
| + for i := 0; i < 3; i++ { |
| + fmt.Fprintf(w, "data: %d\n\n", i) |
| + if f, ok := w.(http.Flusher); ok { |
| + f.Flush() |
| + } |
| + } |
| + })) |
| + rec := httptest.NewRecorder() |
| + e.ServeHTTP(rec, gzipReq("GET", "http://app.test/events")) |
| + |
| + if got := rec.Header().Get("Content-Encoding"); got != "" { |
| + t.Fatalf("Content-Encoding = %q on an event stream, want empty", got) |
| + } |
| + if !strings.Contains(rec.Body.String(), "data: 0") { |
| + t.Fatalf("event stream body = %q, want the plain events", rec.Body.String()) |
| + } |
| +} |
| + |
| +// TestRangeAndConditionalRequestsSurvive: a 206 must keep its byte offsets |
| +// meaningful, and a conditional request must still 304 against the weakened |
| +// validator the client was given. |
| +func TestRangeAndConditionalRequestsSurvive(t *testing.T) { |
| + body := strings.Repeat("a{color:red}\n", 400) |
| + e := edgeWithBackend(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + w.Header().Set("Content-Type", "text/css") |
| + http.ServeContent(w, r, "tokens.css", time.Unix(1, 0), strings.NewReader(body)) |
| + })) |
| + |
| + rng := gzipReq("GET", "http://app.test/tokens.css") |
| + rng.Header.Set("Range", "bytes=0-9") |
| + rec := httptest.NewRecorder() |
| + e.ServeHTTP(rec, rng) |
| + if rec.Code != http.StatusPartialContent { |
| + t.Fatalf("range status = %d, want 206", rec.Code) |
| + } |
| + if got := rec.Header().Get("Content-Encoding"); got != "" { |
| + t.Errorf("206 carries Content-Encoding %q — compressing a byte range makes the offsets it names meaningless", got) |
| + } |
| + if rec.Body.Len() != 10 { |
| + t.Errorf("range body = %d bytes, want 10", rec.Body.Len()) |
| + } |
| + |
| + full := httptest.NewRecorder() |
| + e.ServeHTTP(full, gzipReq("GET", "http://app.test/tokens.css")) |
| + etag := full.Header().Get("ETag") |
| + if etag == "" { |
| + t.Skip("ServeContent emitted no ETag; nothing to condition on") |
| + } |
| + if !strings.HasPrefix(etag, `W/`) { |
| + t.Errorf("ETag = %q on a compressed body, want the weakened form", etag) |
| + } |
| + cond := gzipReq("GET", "http://app.test/tokens.css") |
| + cond.Header.Set("If-None-Match", etag) |
| + nm := httptest.NewRecorder() |
| + e.ServeHTTP(nm, cond) |
| + if nm.Code != http.StatusNotModified { |
| + t.Fatalf("conditional request = %d, want 304 — weakening the validator broke revalidation", nm.Code) |
| + } |
| + if got := nm.Header().Get("Content-Encoding"); got != "" { |
| + t.Errorf("304 carries Content-Encoding %q; a 304 has no body to encode", got) |
| + } |
| +} |
| + |
| +// TestStatusHostCompresses: the status class reaches the wrap site through a |
| +// different door (its own handler, exempt from the shared cache and the usage |
| +// meter) and must compress like everything else. |
| +func TestStatusHostCompresses(t *testing.T) { |
| + s, h := testEdgeWithStore(t, registry.Route{Host: "status.test", Kind: "status", Status: "ready"}) |
| + s.Put(context.Background(), "status/fleet.json", []byte(`{"boxes":[`+strings.Repeat(`{"id":"box"},`, 300)+`{}]}`), "") |
| + rec := httptest.NewRecorder() |
| + h.ServeHTTP(rec, gzipReq("GET", "http://status.test/fleet.json")) |
| + |
| + if got := rec.Header().Get("Content-Encoding"); got != "gzip" { |
| + t.Fatalf("Content-Encoding = %q on the status host, want gzip", got) |
| + } |
| + zr, err := gzip.NewReader(rec.Body) |
| + if err != nil { |
| + t.Fatalf("body is not gzip: %v", err) |
| + } |
| + got, err := io.ReadAll(zr) |
| + if err != nil || !strings.Contains(string(got), `"boxes"`) { |
| + t.Fatalf("decompressed status body = %.40q (err %v)", got, err) |
| + } |
| +} |
| + |
| +// TestStaticSiteCompresses: the static class is the payoff case — a shipped |
| +// site tree whose CSS is served straight off disk by http.ServeFile. It |
| +// reaches the wrap site through Edge.ServeHTTP like every other class, so |
| +// this pins that the class nobody proxies is covered too. |
| +func TestStaticSiteCompresses(t *testing.T) { |
| + store := release.NewDirStore(filepath.Join(t.TempDir(), "store")) |
| + priv, pub := testKeys(t) |
| + css := strings.Repeat("a{color:red}\n", 400) // 5200 bytes, well over the threshold |
| + shipSite(t, store, "site", "v1", map[string]string{ |
| + "index.html": "home", |
| + "tokens.css": css, |
| + }) |
| + if err := release.Promote(context.Background(), store, priv, release.OpsAccount, "site", "canary/x", "v1"); err != nil { |
| + t.Fatalf("Promote: %v", err) |
| + } |
| + |
| + reg := openReg(t) |
| + if err := reg.Upsert(registry.Route{ |
| + Host: "site.test", Kind: "static", App: "site", Channel: "canary/x", Status: "ready", |
| + }); err != nil { |
| + t.Fatalf("upsert: %v", err) |
| + } |
| + e, err := New(reg, nil) |
| + if err != nil { |
| + t.Fatalf("New: %v", err) |
| + } |
| + e.EnableStatic(store, pub, filepath.Join(t.TempDir(), "cache")) |
| + if err := e.Reload(); err != nil { |
| + t.Fatalf("Reload: %v", err) |
| + } |
| + |
| + rec := httptest.NewRecorder() |
| + e.ServeHTTP(rec, gzipReq("GET", "http://site.test/tokens.css")) |
| + if rec.Code != http.StatusOK { |
| + t.Fatalf("status = %d, want 200", rec.Code) |
| + } |
| + if got := rec.Header().Get("Content-Encoding"); got != "gzip" { |
| + t.Fatalf("Content-Encoding = %q on a static asset, want gzip", got) |
| + } |
| + if got := rec.Header().Get("X-Carlos-Version"); got != "v1" { |
| + t.Errorf("X-Carlos-Version = %q, want v1 — compression must not disturb the static class's per-request version stamp", got) |
| + } |
| + zr, err := gzip.NewReader(rec.Body) |
| + if err != nil { |
| + t.Fatalf("body is not gzip: %v", err) |
| + } |
| + got, err := io.ReadAll(zr) |
| + if err != nil { |
| + t.Fatalf("read gzip body: %v", err) |
| + } |
| + if string(got) != css { |
| + t.Fatalf("decompressed static asset differs from the shipped file") |
| + } |
| + if rec.Body.Len() >= len(css) { |
| + t.Fatalf("compressed %d bytes >= identity %d", rec.Body.Len(), len(css)) |
| + } |
| +} |
| + |
| +// TestHeadKeepsItsContentLength: a HEAD has no body to compress, and its |
| +// truthful Content-Length is the whole value of the response. The backend |
| +// sets the length explicitly — Go only infers one when the whole body fits |
| +// its 2048-byte buffer, so a 4096-byte handler would otherwise chunk and this |
| +// test would be asserting the stdlib's buffering, not the edge's behaviour. |
| +func TestHeadKeepsItsContentLength(t *testing.T) { |
| + e := edgeWithBackend(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + w.Header().Set("Content-Type", "text/css") |
| + w.Header().Set("Content-Length", "4096") |
| + w.WriteHeader(http.StatusOK) |
| + if r.Method != http.MethodHead { |
| + io.WriteString(w, strings.Repeat("x", 4096)) |
| + } |
| + })) |
| + rec := httptest.NewRecorder() |
| + e.ServeHTTP(rec, gzipReq("HEAD", "http://app.test/tokens.css")) |
| + |
| + if got := rec.Header().Get("Content-Encoding"); got != "" { |
| + t.Errorf("HEAD Content-Encoding = %q, want empty", got) |
| + } |
| + if got := rec.Header().Get("Content-Length"); got != "4096" { |
| + t.Errorf("HEAD Content-Length = %q, want 4096", got) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run test to verify it fails (or reveals a real gap)** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/edge/ -run 'TestUpgradeIsUntouched|TestAlreadyEncoded|TestEventStream|TestRangeAndConditional|TestStatusHost|TestStaticSiteCompresses|TestHeadKeeps' -count=1 -v` |
| +Expected: these are regression tests over Task 4's implementation, so most should PASS on first run. **Any failure here is a real defect, not a missing feature** — fix `compress.go`, never the assertion. The likely one is the conditional-request case: if `http.ServeContent` compares `If-None-Match` strongly against its own tag, the weakening in `weakenETag` needs the 304 path to strip `W/` on the way back in. Diagnose before changing anything. |
| + |
| +- [ ] **Step 3: Fix any defect the battery found** |
| + |
| +If everything passed, this step is a no-op — record that in the commit message rather than inventing work. |
| + |
| +- [ ] **Step 4: Run the full gate** |
| + |
| +```bash |
| +export GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off |
| +go build ./... && go vet ./... && go test ./... -count=1 |
| +GOOS=darwin go build ./... |
| +gofmt -l internal/edge/compress.go internal/edge/compress_test.go internal/edge/edge.go |
| +``` |
| +Expected: every package passes, the darwin cross-compile succeeds, `gofmt -l` prints nothing. |
| + |
| +- [ ] **Step 5: Record the rule and open the PR** |
| + |
| +Add to `CLAUDE.md` under "Rules this repo has paid for": |
| + |
| +```markdown |
| +- **A ResponseWriter wrapper that edits headers needs its own header map.** |
| + Wrappers here embed the writer below, so `Header()` is ONE shared map from |
| + handler to connection — and `maybeStore` caches `rec.Header().Clone()`. The |
| + gzip wrapper (`internal/edge/compress.go`) keeps a private map and writes |
| + `Content-Encoding` only to the real writer's; on the shared map it would |
| + have stored an identity body labelled gzip and served it to every later |
| + visitor. Ordering is load-bearing too: recorder ABOVE the compressor (cache |
| + stores identity), usage counter BELOW it (tenants are billed for wire |
| + bytes), and the wrap site AFTER the upgrade branch. |
| +``` |
| + |
| +```bash |
| +git add CLAUDE.md internal/edge/compress_test.go internal/edge/compress.go |
| +git commit -m "edge: gzip regression battery + record the wrapper rule |
| + |
| +Upgrades still 101 with Accept-Encoding set, an already-encoded |
| +upstream is not double-compressed, SSE and 206 stay identity, the |
| +status host compresses, and a weakened ETag still revalidates." |
| +git push -u origin gzip-assets |
| +gh pr create --title "Edge response compression (gzip)" --body "$(cat <<'BODY' |
| +Compresses eligible responses at the edge with stdlib gzip, at one |
| +client-write boundary covering cache hits, proxied misses, static trees |
| +and the status host. Apps stay identity-only; transport is the |
| +platform's problem. |
| + |
| +Writer stack: `real conn -> countingWriter -> gzipWriter -> recordingResponseWriter -> handler`. |
| +The recorder above keeps the response cache storing identity bodies |
| +(one variant, compressed after retrieval); the counter below means |
| +tenants are metered for what actually crossed the wire. |
| + |
| +The wrapper owns a private header map — wrappers here embed the writer |
| +below, so `Header()` is one shared map and `maybeStore` caches |
| +`rec.Header().Clone()`. Setting `Content-Encoding` there would have |
| +cached an identity body labelled gzip. |
| + |
| +Eligibility is header-only, decided once at WriteHeader: compressible |
| +Content-Type, no existing encoding, no `no-transform`, not 204/205/206/304, |
| +`Content-Length` absent or >= 1024. A missing Content-Type is never |
| +sniffed. Upgrades return before the wrap site. |
| + |
| +Rollback: `CARLOS_EDGE_GZIP=off`, no binary roll. |
| + |
| +Spec: `docs/superpowers/specs/2026-08-18-edge-response-compression-design.md` |
| +Plan: `docs/superpowers/plans/2026-08-18-edge-response-compression.md` |
| + |
| +🤖 Generated with [Claude Code](https://claude.com/claude-code) |
| + |
| +https://claude.ai/code/session_01NoqzkUVahW5ykubsBQitxc |
| +BODY |
| +)" |
| +``` |
| + |
| +--- |
| + |
| +## Deployment note (not a task) |
| + |
| +This ships in the platform binary and takes effect at the edge process restart |
| +that a platform roll performs. Verify on a real host with the header, against |
| +the box, not a cached answer: |
| + |
| +```bash |
| +curl -sI -H 'Accept-Encoding: gzip' --resolve <host>:443:<box-ip> https://<host>/<asset.css> \ |
| + | grep -iE 'content-encoding|vary|x-carlos-version' |
| +``` |
| + |
| +Expect `content-encoding: gzip`, a `vary` naming `Accept-Encoding`, and the |
| +`X-Carlos-Version` of the build you just rolled. A request without the |
| +`Accept-Encoding` header must come back identity from the same URL. |