amadan
spec: request references — one ID a customer can read out, and we can find
A customer who hits an error sees a reference like R-0K7M-3QXA-9F2C; support types `carlos errors --app <a> --ref <R-…>` and finds the group, the occurrence and the log lines around it. The edge mints it once at the first CARLOS edge (30 bits of seconds, 30 random, Crockford base32), keeps it across verified hops and the tunnel only if it parses as a reference, carries it to the app as X-Request-Id, returns it on every response (set in countingWriter.WriteHeader so gzip and 103 handling cannot lose it), keeps it out of the cache, shows it on every edge-written error page, stores up to 20 per coalesced error record, and looks it up by decoding its time. Approved by Bill with four rulings (§8); reviewed by Fable, all amendments taken (§10). Also corrects the errors spec's five-minute flush claim: the ship tick is 10 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+383
−0
docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md+3 −0docs/superpowers/specs/2026-09-24-request-reference-design.md+380 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md b/docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md| index 12f9dc8b3..c4fda4d7b 100644 |
| --- a/docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md |
| +++ b/docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md |
| @@ -203,6 +203,9 @@ occurrences, labelled as such on the tab. A record belongs to a read |
| window when `[first, last]` overlaps it and is counted whole; the flush |
| interval is five minutes, so a window edge is wrong by at most one |
| interval, and the tab's window labels say "to the nearest flush". |
| +(Correction, 2026-09-24: the ship loop as built flushes every 10 s, |
| +`errshipTick` in `internal/host/errship.go`, so records coalesce over |
| +10 s, not five minutes. Found reviewing the request-reference spec.) |
| **Storage (decision, amended per Fable 6).** `errors/` is a sibling family |
| of `logs/`: `apps/<sqid>/<app>/errors/<box>/<hour>/<seq>-<size>.jsonl.gz`. |
diff --git a/docs/superpowers/specs/2026-09-24-request-reference-design.md b/docs/superpowers/specs/2026-09-24-request-reference-design.md| new file mode 100644 |
| index 000000000..bf2b07e4d |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-09-24-request-reference-design.md |
| @@ -0,0 +1,380 @@ |
| +# Request references: one ID a customer can read out, and we can find |
| + |
| +**Date:** 2026-09-24 |
| +**Status:** approved (Bill, 2026-09-24). Nothing here is built. Every |
| +choice is marked **(decision)** and can be overturned cheaply; the four in |
| +§8 are Bill's **(ruling)**s of 2026-09-24. Builds on |
| +`2026-09-07-errors-and-analytics-design.md` (§3.2 and §11, Availability). |
| +Line numbers cite `main` at `38639c1e`. Reviewed by Fable |
| +(approve-with-amendments); §10 records what it changed. |
| + |
| +## 1. The idea |
| + |
| +When something goes wrong for a customer, the support call today starts |
| +with "what time was it, roughly, and what were you doing?" The Errors and |
| +Availability tabs group failures well, but nothing links one customer's |
| +failed request to one row. |
| + |
| +The fix most services converge on is a request reference: a short ID |
| +minted where the request enters, handed to the app, shown on every error |
| +page, and stored with the error. The customer reads it out, support types |
| +`carlos errors --app titogo --ref R-0K7M-3QXA-9F2C`, and the answer is the |
| +group, the occurrence, and the log lines around it. |
| + |
| +CARLOS is well placed to do this, because the edge is where every request |
| +enters. The app does not need to mint anything. It only reads a header and |
| +prints it on its own error page. |
| + |
| +## 2. What is already right, and must stay right |
| + |
| +- **The edge's wrapper order is load-bearing** (AGENTS.md). Minting a |
| + reference is a header write near the top of `ServeHTTP`. It moves no |
| + wrapper. |
| +- **Nothing the platform does for itself may keep an instance awake.** |
| + Minting is CPU only, on a path every request already takes. |
| +- **Hop trust is proven, never assumed.** `acceptHop` |
| + (`internal/edge/homemap.go:767`) verifies a backhaul hop before the |
| + receiving edge believes anything it carries, and `stripForwardedHeaders` |
| + discards the claims of an unproven one. A reference arriving from outside |
| + is a claim like any other. |
| +- **Coalescing keeps one sample per group.** The errors sink folds every |
| + occurrence of `(fp, host, release)` between two ship ticks into one |
| + record carrying the *first* occurrence's details (§3.2 of the errors |
| + spec). The tick is 10 s (`errshipTick`, `internal/host/errship.go:22`); |
| + the errors spec's "five minutes" describes a design the ship loop never |
| + used. That is what keeps an error storm cheap. It also means a |
| + reference cannot simply be "the occurrence's": most occurrences have no |
| + record of their own. §3.4 is the answer. |
| +- **Cached responses replay stored headers.** `writeCached` |
| + (`internal/edge/edge.go:1897`) copies the stored header map onto the |
| + response wholesale. A reference captured into the cache would be |
| + replayed to every later visitor, so the reference must never be stored |
| + there (§3.2). |
| +- **Response headers set early do not survive.** The gzip writer keeps a |
| + private header map and, at its final `WriteHeader`, deletes every key on |
| + the real writer's map before copying its own down |
| + (`internal/edge/compress.go:211-245`). `httputil.ReverseProxy` clears |
| + the handler-visible map after a 103 Early Hints, and copies upstream |
| + headers with `Add`, not `Set`. A response header set at the top of |
| + `ServeHTTP` is therefore wiped on most responses, or duplicated when the |
| + app echoes its own (§3.2). |
| + |
| +## 3. The change |
| + |
| +### 3.1 The reference |
| + |
| +**Format (decision, amended in review).** `R-` followed by twelve Crockford |
| +base32 characters, shown in three groups of four: `R-0K7M-3QXA-9F2C`. The |
| +60 bits are: |
| + |
| +- 30 bits: seconds since 2026-01-01T00:00:00Z, which lasts until 2060; |
| +- 30 bits: random. |
| + |
| +Crockford base32 has no I, L, O or U, and decoding is case-insensitive |
| +and maps those look-alikes to 1 and 0. That makes the reference safe to |
| +read aloud, type from a screenshot, or copy with the wrong case. |
| +Uniqueness only has to hold within one second, and the honest measure is |
| +the birthday bound, not the size of the space. With n requests in one |
| +second the chance of any collision is about n²/2^31. At 100 requests a |
| +second that is about one pair every two or three days. At 1,000 a second |
| +it is about 40 a day. The first draft's 28 random bits gave four times that. A collision |
| +only matters when both requests fail and one is looked up; lookup then |
| +shows both occurrences rather than guessing (§3.5). The random half comes |
| +from `math/rand/v2`. The reference is an identifier, not a secret, and |
| +nothing authorises on it. |
| + |
| +**Why the time is inside it (decision).** Lookup must not need an index. |
| +The reference decodes to the second it was minted, so the console reads |
| +one narrow window of chunks and never scans the day (§3.5). The reference |
| +does not encode the box or the app. The window across every box of one |
| +app is small, and a box label in a customer-facing string would be |
| +infrastructure detail leaking into support conversations. |
| + |
| +### 3.2 Where it is minted, and who may carry one in |
| + |
| +**Minted once, at the first CARLOS edge the request reaches (decision).** |
| +In `ServeHTTP`, immediately after `acceptHop` and before the route |
| +lookup, so the backhaul and tunnel branches below carry it on: |
| + |
| +- a request on a **verified hop** (`hopVerified`) keeps the reference the |
| + sending edge minted; |
| +- a request that arrived **over this box's own tunnel client** keeps the |
| + front box's reference. Today nothing marks such a request: the tunnel |
| + client serves the edge as a plain handler |
| + (`internal/host/tunnel_client.go:53`). The flag is set by a wrapper that |
| + lives in `internal/edge` (an exported `TunnelArrived(h)`), so the edge's |
| + own tunnel e2e test drives the same handler production runs. The |
| + channel is one this box dialled with TLS verification |
| + (`internal/tunnelclient/tunnelclient.go:207`), so the hub is the only |
| + possible sender. The two trusted marks never coexist: the front box |
| + deletes hop headers before it proxies over the tunnel; |
| +- **on both trusted paths the inbound value is kept only if it parses as |
| + a reference** (`R-` and twelve Crockford characters); otherwise a new |
| + one is minted. During a rollout the front box may be on an older build |
| + that neither mints nor strips, and without this check a client's own |
| + `X-Request-Id` would pass through it and be trusted by the new box |
| + behind it; |
| +- **every other request gets a new reference**, and any `X-Request-Id` the |
| + client sent is deleted first. A client-supplied value would let anyone |
| + plant a reference that collides with a real one or points support at the |
| + wrong occurrence. |
| + |
| +**Carried as `X-Request-Id` on the inbound request (decision).** The mint |
| +site sets it on `r.Header`, and that one value is the carrier everywhere: |
| +`httputil.ReverseProxy` clones inbound headers to the outbound request, |
| +so edges and the app receive it with no Rewrite change; the activator's |
| +wake 503s, the static handler and the pending page read it from the same |
| +request. It is the name Rails' `ActionDispatch::RequestId`, most Go |
| +middleware, and most log shippers already read, so a Rails app gets it |
| +into its tagged logs and into `request.request_id` with no change. |
| + |
| +**Returned on every response (ruling, §8.1).** The edge sets |
| +`X-Request-Id` on the response too, so it shows in browser dev tools and |
| +in any `curl -I`. The value is the edge's reference, replacing anything |
| +the upstream set. An app echoing its own ID back does not get to change |
| +the one support will search for. |
| + |
| +Where it is set matters (§2). It is set with `Header().Set` in |
| +`countingWriter.WriteHeader` (`internal/edge/usagemeter.go:80`) for any |
| +status of 200 or more. The counting writer is the bottom wrapper, so this |
| +runs after the gzip writer has synced its private map and after |
| +`ReverseProxy` has added the upstream's headers: the value replaces |
| +rather than duplicates. **No new wrapper is added** and none moves. Two |
| +cases fall outside it: |
| + |
| +- **WebSocket upgrades.** `ReverseProxy` writes a 101's headers straight |
| + to the hijacked connection and never calls `WriteHeader`. For an |
| + upgrade request the edge sets the header on `w.Header()` just before |
| + proxying. If the app also sets one, the 101 carries both; this is |
| + accepted and documented rather than worked around. |
| +- **Paths that answer before the counting writer exists**: the steering |
| + probe, the tunnel hub and store door, the stale-hop refusal, the |
| + backhaul sender's own 502/503, the unknown-host answer and the pending |
| + page. No gzip writer wraps these, so each sets the header directly. |
| + |
| +**Never cached.** `maybeStore` deletes `X-Request-Id` from the header map |
| +it clones (`internal/edge/edge.go:1720`). That covers both shapes: with |
| +gzip on the clone is the gzip writer's private map, with it off the real |
| +one. `writeCached` needs no change, because the counting writer's |
| +`WriteHeader` sets the current request's reference after the replay. A |
| +test pins that two requests served from one cache entry carry different |
| +references. |
| + |
| +### 3.3 Shown on every error page the edge writes |
| + |
| +Every status the edge produces itself says the reference: |
| + |
| +- the 502s written by the proxy error handlers; |
| +- the activator's wake-path 503s; |
| +- the fleet-dark and no-healthy-upstream 503s; |
| +- the static-site 502; |
| +- the pending-host retry page (`internal/edge/pending.go`). |
| + |
| +The plain-text bodies gain one line, `Reference: R-0K7M-3QXA-9F2C`. The |
| +HTML retry page shows it in small type under its message. That page is |
| +rendered once today (`e.pendingBody`, `pending.go:137,186`) and becomes |
| +per-request; the plan checks `pending_script_test.go` for anything that |
| +pins its bytes or its script hash. The format is |
| +fixed so a support person can recognise it on a screenshot. |
| + |
| +The app's own error pages are the app's. The platform's part is the |
| +header. The getting-started skill and `docs/site` get a paragraph on |
| +showing `X-Request-Id` on a 500 page, with Rails and Go examples. |
| + |
| +### 3.4 Stored with the error |
| + |
| +**`refs` on the errors record (decision).** `apperr.Record` gains |
| +`Refs []string` (`omitempty`, additive). The sink appends the reference of |
| +each occurrence it folds into a record, up to `MaxRefs = 20` per record |
| +per ship tick. `Refs` holds the first twenty, in order. `Count` already |
| +says how many there were. The sink clears `Refs` on the `_overflow` row |
| +it builds from the first excess record (`sink.go:104-110` copies the |
| +whole record), so the overflow row never carries any. |
| + |
| +**A ref names the group, not the occurrence's page.** A folded occurrence |
| +only increments `Count` and moves `Last` (`sink.go:96-100`). Host and |
| +release are exact, because they are part of the group's key. Path, |
| +status and fields are the group's *first* sample. On Availability, where |
| +groups are fingerprinted by cause and not by path, the fifteenth ref's |
| +page may differ from the sample's. Lookup says so (§3.5). Storing a path |
| +beside each ref is an additive later step if support finds it wants one. |
| + |
| +Twenty is enough for the support case, a person who hit an error and |
| +called. It keeps the record's size bounded in an error storm: 20 × 16 |
| +bytes, against a stack that is already capped at 8 KB. A storm is not |
| +the case where one reference matters: a thousand customers hit the same |
| +thing, and the group itself is the answer. |
| + |
| +**A reference past the cap is still findable, less precisely.** Lookup |
| +(§3.5) falls back to what the reference's time says: the groups on that |
| +app with an occurrence overlapping that second, labelled as such. |
| + |
| +**Every source carries it.** |
| + |
| +- `edge`: the observing defer already has the request, so it reads the |
| + reference from the request. |
| +- `log`: an app that logs its `X-Request-Id` has the reference in the line |
| + itself. Lookup greps the window for it. |
| +- `server` (rollout step 4 of the errors spec, not built): the verb gains a |
| + `ref` field; the app passes the header's value. |
| +- `browser` (step 3, not built): out of scope here. A page cannot read its |
| + own response headers. The snippet would need the reference rendered |
| + into the page, which is the app's to do. |
| + |
| +### 3.5 Looking one up |
| + |
| +**`carlos errors --app <a> --ref <R-…>` (decision)**, and a Reference box |
| +on both the Errors and Availability tabs, answered by one function |
| +through the API as `?ref=` on both paths: |
| + |
| +1. Decode the reference to its second `t`. A malformed reference is a 400 |
| + naming the expected shape. A reference from the future, or from before |
| + the app existed, says so rather than searching nothing. |
| +2. Read both families over `[t − 1 min, t + 30 min]`. The failure is |
| + filed when the request *ends*, not when it was minted: a dead socket is |
| + redialled for 13 s, a backhaul waits up to 30 s for headers, and an |
| + abandoned wait on a wedged instance can run for minutes. The listing |
| + prunes by hour (`internal/applog/read.go`), so a wide window inside |
| + one hour costs nothing extra. |
| +3. **Exact:** records whose `Refs` contains the reference. The answer is |
| + the group, with that occurrence's host and release, and the group's |
| + first sample's path and status, labelled as the sample (§3.4). It says |
| + which tab the group lives on. Two exact matches (a collision, §3.1) |
| + are both shown. |
| +4. **Near:** when nothing matched exactly, the groups with an occurrence |
| + overlapping `[t, t + 60 s]` on that app, plainly labelled "this |
| + reference was not stored individually; these failed within a minute |
| + of it". The minute covers the dial retry and the abandoned grace. |
| +5. **Logs:** lines in `[t − 5 s, t + 60 s]` whose message or any field |
| + value contains the reference. `applog.Filter.Grep` matches the message |
| + only, and `Filter.Fields` is exact equality on a named key, so this |
| + needs a new any-field substring match; JSON log lines put |
| + `request_id` in fields, tagged plain-text lines put it in the message. |
| + The read decodes every chunk in the hour, so this costs what |
| + `carlos logs --since 1h` costs for the app. |
| +6. **Not shipped yet:** when `t` is within about a minute of now, or the |
| + app's newest chunk is older than `t`, the answer is "this reference is |
| + too recent to have been shipped; try again in a minute". Boxes ship |
| + errors every 10 s and hold a failed upload for the next tick. |
| +7. **Nothing:** "no error or log line carries this reference". This is |
| + also the honest answer for a request that succeeded but that the |
| + customer thought was slow, and support should hear it plainly. |
| + |
| +App-scoped in v1: support knows which app the customer was using. A |
| +cross-app lookup needs an account-level door with its own gate (§6). |
| + |
| +## 4. Nothing here wakes an instance |
| + |
| +Minting and header writes happen on paths every request already takes. |
| +Lookup is a console read over chunks the agent already ships. Nothing |
| +new touches the activator ahead of the cache check. The tunnel |
| +flag is one context value. |
| + |
| +## 5. Wire shapes |
| + |
| +- `X-Request-Id` on requests to apps is new traffic, not a changed shape. |
| + An app that already sets its own ID from a missing header keeps doing |
| + so; an app that trusted a client-supplied `X-Request-Id` now sees the |
| + edge's instead. |
| +- `refs` on the errors record is `omitempty` and additive. |
| +- A box on an older build mints nothing and stores no refs. Lookup falls |
| + back to "near" for its occurrences, and the answer says so. |
| + |
| +## 6. What is not built |
| + |
| +- **A cross-app lookup.** An account-level `carlos errors --ref` with no |
| + `--app` means reading every app's window, and needs its own gate. |
| +- **Honouring a customer's own proxy's ID** (for example a CDN's ray ID). |
| + Ruled out for now (§8.2). |
| +- **An edge access log.** `applog.StreamEdge` is reserved and unbuilt. With |
| + it, every request, successful or not, would be findable by reference. |
| + That is the natural next step, and it is a different cost decision |
| + (storage per request, not per failure). |
| + |
| +## 7. Rollout |
| + |
| +1. **Mint and carry.** The edge mints, strips, carries across hops and the |
| + tunnel, sets the request and response headers, and keeps the reference |
| + out of the cache. Its own error pages show it. Useful on its own: |
| + support can already ask for the reference and grep logs for it. |
| +2. **Store and look up.** `Refs` on the record, `?ref=` on both APIs, |
| + `--ref` on `carlos errors` and `carlos availability`, the Reference box |
| + on both tabs. Docs. |
| +3. **Apps show it.** titogo prints `X-Request-Id` on its 500 page. That |
| + change is in titogo, not here; the docs paragraph in step 2 is the |
| + recipe. |
| + |
| +Each step is its own branch. Step 1 touches the edge serving path and gets |
| +the Opus-then-Fable review AGENTS.md asks for. |
| + |
| +## 8. Rulings (Bill, 2026-09-24) |
| + |
| +Four questions were open in the draft; Bill answered all four. |
| + |
| +1. **The reference is returned on every response (ruling)**, not only on |
| + errors. It costs one header, and it answers "which request was the slow |
| + one" from dev tools, including for a request that succeeded. |
| +2. **A client-supplied `X-Request-Id` is ignored (ruling).** It is deleted |
| + before the edge mints its own, and not kept. Keeping it as a |
| + `client_id` field, so a customer behind their own CDN can correlate the |
| + two IDs, stays an additive later step (§6). |
| +3. **`MaxRefs = 20` (ruling).** |
| +4. **The `R-` prefix and the 4-4-4 grouping stand (ruling).** |
| + |
| +## 9. Testing |
| + |
| +- **Format:** round-trip encode and decode; case-insensitive decode; |
| + look-alike characters (I→1, L→1, O→0); rejection of wrong lengths and |
| + prefixes; the time half survives the round trip to the second. |
| +- **Minting:** a client-supplied `X-Request-Id` never reaches the app; a |
| + verified hop's reference survives to the home box; an unverified hop's |
| + does not; a tunnel-arrived request keeps the front box's; every other |
| + request gets a fresh one. |
| +- **Cache:** two hits on one entry carry different references, and the |
| + stored entry holds none. |
| +- **Error pages:** every edge-written 502/503/504 body names the reference |
| + the response header carries. |
| +- **Storage:** twenty-one occurrences in one tick store twenty refs and a |
| + count of twenty-one; the overflow row carries none. |
| +- **Response header:** present and single-valued with gzip on and off, |
| + after a 103 Early Hints, when the app echoes its own `X-Request-Id`, |
| + and on each early-return path; on a 101 upgrade, present. |
| +- **Trust:** a non-reference value on a verified hop or over the tunnel is |
| + replaced, not kept. |
| +- **Lookup:** exact (including two matches), near, logs, not-shipped-yet |
| + and nothing, each against seeded chunks; a failure filed 13 s and 10 |
| + minutes after its reference's second is still an exact match; the |
| + malformed and future cases. |
| + |
| +## 10. Review round 1 (Fable, 2026-09-24) |
| + |
| +Approve-with-amendments. Every amendment was taken: |
| + |
| +- **Response header site.** The first draft set it at the mint site. The |
| + gzip writer, 103 handling and `Add` copying would have lost or |
| + duplicated it on most responses. It now lives in the counting writer's |
| + `WriteHeader`, with the upgrade and early-return cases named (§3.2). |
| +- **One carrier.** `X-Request-Id` on the inbound request, not a |
| + `setForwarded` change (§3.2). |
| +- **Mixed-version trust.** Inbound values on trusted paths must parse as |
| + references (§3.2). |
| +- **Tunnel flag placement** in `internal/edge`, so the e2e test runs the |
| + production handler (§3.2). |
| +- **Birthday bound.** Stated honestly, and the split moved from 32/28 to |
| + 30/30. This is within Bill's format ruling, which fixed the prefix and |
| + grouping, not the bit split. Lookup shows collisions rather than |
| + guessing (§3.1, §3.5). |
| +- **The ship tick is 10 s, not five minutes.** The window reasoning was |
| + rebuilt on request duration instead, which is what actually moves a |
| + record past its reference's second (§2, §3.5). |
| +- **A ref names the group, not the page**, and lookup labels the sample |
| + as a sample (§3.4, §3.5). |
| +- **Overflow row clears refs** (§3.4). |
| +- **"Not shipped yet"** before "nothing" (§3.5). |
| +- **Log matching** needs an any-field substring match, and its cost is |
| + stated (§3.5). |
| +- **The cache drop** lives in `maybeStore` (§3.2); **the pending page** |
| + becomes per-request (§3.3). |
| + |
| +Declined: nothing. |
| + |