amadan
Merge branch 'refs-design' into main
Merge branch 'refs-design': spec: request references
2 files changed,
+383
−0
docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md+3 −0docs/superpowers/specs/2026-09-24-request-reference-design.md+380 −0
CI cancelled — superseded by a newer push — run details
diff --git a/docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md b/docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md| index 12f9dc8b3..c4fda4d7b 100644 |
| --- a/docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md |
| +++ b/docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md |
| @@ -203,6 +203,9 @@ occurrences, labelled as such on the tab. A record belongs to a read |
| window when `[first, last]` overlaps it and is counted whole; the flush |
| interval is five minutes, so a window edge is wrong by at most one |
| interval, and the tab's window labels say "to the nearest flush". |
| +(Correction, 2026-09-24: the ship loop as built flushes every 10 s, |
| +`errshipTick` in `internal/host/errship.go`, so records coalesce over |
| +10 s, not five minutes. Found reviewing the request-reference spec.) |
| **Storage (decision, amended per Fable 6).** `errors/` is a sibling family |
| of `logs/`: `apps/<sqid>/<app>/errors/<box>/<hour>/<seq>-<size>.jsonl.gz`. |
diff --git a/docs/superpowers/specs/2026-09-24-request-reference-design.md b/docs/superpowers/specs/2026-09-24-request-reference-design.md| new file mode 100644 |
| index 000000000..bf2b07e4d |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-09-24-request-reference-design.md |
| @@ -0,0 +1,380 @@ |
| +# Request references: one ID a customer can read out, and we can find |
| + |
| +**Date:** 2026-09-24 |
| +**Status:** approved (Bill, 2026-09-24). Nothing here is built. Every |
| +choice is marked **(decision)** and can be overturned cheaply; the four in |
| +§8 are Bill's **(ruling)**s of 2026-09-24. Builds on |
| +`2026-09-07-errors-and-analytics-design.md` (§3.2 and §11, Availability). |
| +Line numbers cite `main` at `38639c1e`. Reviewed by Fable |
| +(approve-with-amendments); §10 records what it changed. |
| + |
| +## 1. The idea |
| + |
| +When something goes wrong for a customer, the support call today starts |
| +with "what time was it, roughly, and what were you doing?" The Errors and |
| +Availability tabs group failures well, but nothing links one customer's |
| +failed request to one row. |
| + |
| +The fix most services converge on is a request reference: a short ID |
| +minted where the request enters, handed to the app, shown on every error |
| +page, and stored with the error. The customer reads it out, support types |
| +`carlos errors --app titogo --ref R-0K7M-3QXA-9F2C`, and the answer is the |
| +group, the occurrence, and the log lines around it. |
| + |
| +CARLOS is well placed to do this, because the edge is where every request |
| +enters. The app does not need to mint anything. It only reads a header and |
| +prints it on its own error page. |
| + |
| +## 2. What is already right, and must stay right |
| + |
| +- **The edge's wrapper order is load-bearing** (AGENTS.md). Minting a |
| + reference is a header write near the top of `ServeHTTP`. It moves no |
| + wrapper. |
| +- **Nothing the platform does for itself may keep an instance awake.** |
| + Minting is CPU only, on a path every request already takes. |
| +- **Hop trust is proven, never assumed.** `acceptHop` |
| + (`internal/edge/homemap.go:767`) verifies a backhaul hop before the |
| + receiving edge believes anything it carries, and `stripForwardedHeaders` |
| + discards the claims of an unproven one. A reference arriving from outside |
| + is a claim like any other. |
| +- **Coalescing keeps one sample per group.** The errors sink folds every |
| + occurrence of `(fp, host, release)` between two ship ticks into one |
| + record carrying the *first* occurrence's details (§3.2 of the errors |
| + spec). The tick is 10 s (`errshipTick`, `internal/host/errship.go:22`); |
| + the errors spec's "five minutes" describes a design the ship loop never |
| + used. That is what keeps an error storm cheap. It also means a |
| + reference cannot simply be "the occurrence's": most occurrences have no |
| + record of their own. §3.4 is the answer. |
| +- **Cached responses replay stored headers.** `writeCached` |
| + (`internal/edge/edge.go:1897`) copies the stored header map onto the |
| + response wholesale. A reference captured into the cache would be |
| + replayed to every later visitor, so the reference must never be stored |
| + there (§3.2). |
| +- **Response headers set early do not survive.** The gzip writer keeps a |
| + private header map and, at its final `WriteHeader`, deletes every key on |
| + the real writer's map before copying its own down |
| + (`internal/edge/compress.go:211-245`). `httputil.ReverseProxy` clears |
| + the handler-visible map after a 103 Early Hints, and copies upstream |
| + headers with `Add`, not `Set`. A response header set at the top of |
| + `ServeHTTP` is therefore wiped on most responses, or duplicated when the |
| + app echoes its own (§3.2). |
| + |
| +## 3. The change |
| + |
| +### 3.1 The reference |
| + |
| +**Format (decision, amended in review).** `R-` followed by twelve Crockford |
| +base32 characters, shown in three groups of four: `R-0K7M-3QXA-9F2C`. The |
| +60 bits are: |
| + |
| +- 30 bits: seconds since 2026-01-01T00:00:00Z, which lasts until 2060; |
| +- 30 bits: random. |
| + |
| +Crockford base32 has no I, L, O or U, and decoding is case-insensitive |
| +and maps those look-alikes to 1 and 0. That makes the reference safe to |
| +read aloud, type from a screenshot, or copy with the wrong case. |
| +Uniqueness only has to hold within one second, and the honest measure is |
| +the birthday bound, not the size of the space. With n requests in one |
| +second the chance of any collision is about n²/2^31. At 100 requests a |
| +second that is about one pair every two or three days. At 1,000 a second |
| +it is about 40 a day. The first draft's 28 random bits gave four times that. A collision |
| +only matters when both requests fail and one is looked up; lookup then |
| +shows both occurrences rather than guessing (§3.5). The random half comes |
| +from `math/rand/v2`. The reference is an identifier, not a secret, and |
| +nothing authorises on it. |
| + |
| +**Why the time is inside it (decision).** Lookup must not need an index. |
| +The reference decodes to the second it was minted, so the console reads |
| +one narrow window of chunks and never scans the day (§3.5). The reference |
| +does not encode the box or the app. The window across every box of one |
| +app is small, and a box label in a customer-facing string would be |
| +infrastructure detail leaking into support conversations. |
| + |
| +### 3.2 Where it is minted, and who may carry one in |
| + |
| +**Minted once, at the first CARLOS edge the request reaches (decision).** |
| +In `ServeHTTP`, immediately after `acceptHop` and before the route |
| +lookup, so the backhaul and tunnel branches below carry it on: |
| + |
| +- a request on a **verified hop** (`hopVerified`) keeps the reference the |
| + sending edge minted; |
| +- a request that arrived **over this box's own tunnel client** keeps the |
| + front box's reference. Today nothing marks such a request: the tunnel |
| + client serves the edge as a plain handler |
| + (`internal/host/tunnel_client.go:53`). The flag is set by a wrapper that |
| + lives in `internal/edge` (an exported `TunnelArrived(h)`), so the edge's |
| + own tunnel e2e test drives the same handler production runs. The |
| + channel is one this box dialled with TLS verification |
| + (`internal/tunnelclient/tunnelclient.go:207`), so the hub is the only |
| + possible sender. The two trusted marks never coexist: the front box |
| + deletes hop headers before it proxies over the tunnel; |
| +- **on both trusted paths the inbound value is kept only if it parses as |
| + a reference** (`R-` and twelve Crockford characters); otherwise a new |
| + one is minted. During a rollout the front box may be on an older build |
| + that neither mints nor strips, and without this check a client's own |
| + `X-Request-Id` would pass through it and be trusted by the new box |
| + behind it; |
| +- **every other request gets a new reference**, and any `X-Request-Id` the |
| + client sent is deleted first. A client-supplied value would let anyone |
| + plant a reference that collides with a real one or points support at the |
| + wrong occurrence. |
| + |
| +**Carried as `X-Request-Id` on the inbound request (decision).** The mint |
| +site sets it on `r.Header`, and that one value is the carrier everywhere: |
| +`httputil.ReverseProxy` clones inbound headers to the outbound request, |
| +so edges and the app receive it with no Rewrite change; the activator's |
| +wake 503s, the static handler and the pending page read it from the same |
| +request. It is the name Rails' `ActionDispatch::RequestId`, most Go |
| +middleware, and most log shippers already read, so a Rails app gets it |
| +into its tagged logs and into `request.request_id` with no change. |
| + |
| +**Returned on every response (ruling, §8.1).** The edge sets |
| +`X-Request-Id` on the response too, so it shows in browser dev tools and |
| +in any `curl -I`. The value is the edge's reference, replacing anything |
| +the upstream set. An app echoing its own ID back does not get to change |
| +the one support will search for. |
| + |
| +Where it is set matters (§2). It is set with `Header().Set` in |
| +`countingWriter.WriteHeader` (`internal/edge/usagemeter.go:80`) for any |
| +status of 200 or more. The counting writer is the bottom wrapper, so this |
| +runs after the gzip writer has synced its private map and after |
| +`ReverseProxy` has added the upstream's headers: the value replaces |
| +rather than duplicates. **No new wrapper is added** and none moves. Two |
| +cases fall outside it: |
| + |
| +- **WebSocket upgrades.** `ReverseProxy` writes a 101's headers straight |
| + to the hijacked connection and never calls `WriteHeader`. For an |
| + upgrade request the edge sets the header on `w.Header()` just before |
| + proxying. If the app also sets one, the 101 carries both; this is |
| + accepted and documented rather than worked around. |
| +- **Paths that answer before the counting writer exists**: the steering |
| + probe, the tunnel hub and store door, the stale-hop refusal, the |
| + backhaul sender's own 502/503, the unknown-host answer and the pending |
| + page. No gzip writer wraps these, so each sets the header directly. |
| + |
| +**Never cached.** `maybeStore` deletes `X-Request-Id` from the header map |
| +it clones (`internal/edge/edge.go:1720`). That covers both shapes: with |
| +gzip on the clone is the gzip writer's private map, with it off the real |
| +one. `writeCached` needs no change, because the counting writer's |
| +`WriteHeader` sets the current request's reference after the replay. A |
| +test pins that two requests served from one cache entry carry different |
| +references. |
| + |
| +### 3.3 Shown on every error page the edge writes |
| + |
| +Every status the edge produces itself says the reference: |
| + |
| +- the 502s written by the proxy error handlers; |
| +- the activator's wake-path 503s; |
| +- the fleet-dark and no-healthy-upstream 503s; |
| +- the static-site 502; |
| +- the pending-host retry page (`internal/edge/pending.go`). |
| + |
| +The plain-text bodies gain one line, `Reference: R-0K7M-3QXA-9F2C`. The |
| +HTML retry page shows it in small type under its message. That page is |
| +rendered once today (`e.pendingBody`, `pending.go:137,186`) and becomes |
| +per-request; the plan checks `pending_script_test.go` for anything that |
| +pins its bytes or its script hash. The format is |
| +fixed so a support person can recognise it on a screenshot. |
| + |
| +The app's own error pages are the app's. The platform's part is the |
| +header. The getting-started skill and `docs/site` get a paragraph on |
| +showing `X-Request-Id` on a 500 page, with Rails and Go examples. |
| + |
| +### 3.4 Stored with the error |
| + |
| +**`refs` on the errors record (decision).** `apperr.Record` gains |
| +`Refs []string` (`omitempty`, additive). The sink appends the reference of |
| +each occurrence it folds into a record, up to `MaxRefs = 20` per record |
| +per ship tick. `Refs` holds the first twenty, in order. `Count` already |
| +says how many there were. The sink clears `Refs` on the `_overflow` row |
| +it builds from the first excess record (`sink.go:104-110` copies the |
| +whole record), so the overflow row never carries any. |
| + |
| +**A ref names the group, not the occurrence's page.** A folded occurrence |
| +only increments `Count` and moves `Last` (`sink.go:96-100`). Host and |
| +release are exact, because they are part of the group's key. Path, |
| +status and fields are the group's *first* sample. On Availability, where |
| +groups are fingerprinted by cause and not by path, the fifteenth ref's |
| +page may differ from the sample's. Lookup says so (§3.5). Storing a path |
| +beside each ref is an additive later step if support finds it wants one. |
| + |
| +Twenty is enough for the support case, a person who hit an error and |
| +called. It keeps the record's size bounded in an error storm: 20 × 16 |
| +bytes, against a stack that is already capped at 8 KB. A storm is not |
| +the case where one reference matters: a thousand customers hit the same |
| +thing, and the group itself is the answer. |
| + |
| +**A reference past the cap is still findable, less precisely.** Lookup |
| +(§3.5) falls back to what the reference's time says: the groups on that |
| +app with an occurrence overlapping that second, labelled as such. |
| + |
| +**Every source carries it.** |
| + |
| +- `edge`: the observing defer already has the request, so it reads the |
| + reference from the request. |
| +- `log`: an app that logs its `X-Request-Id` has the reference in the line |
| + itself. Lookup greps the window for it. |
| +- `server` (rollout step 4 of the errors spec, not built): the verb gains a |
| + `ref` field; the app passes the header's value. |
| +- `browser` (step 3, not built): out of scope here. A page cannot read its |
| + own response headers. The snippet would need the reference rendered |
| + into the page, which is the app's to do. |
| + |
| +### 3.5 Looking one up |
| + |
| +**`carlos errors --app <a> --ref <R-…>` (decision)**, and a Reference box |
| +on both the Errors and Availability tabs, answered by one function |
| +through the API as `?ref=` on both paths: |
| + |
| +1. Decode the reference to its second `t`. A malformed reference is a 400 |
| + naming the expected shape. A reference from the future, or from before |
| + the app existed, says so rather than searching nothing. |
| +2. Read both families over `[t − 1 min, t + 30 min]`. The failure is |
| + filed when the request *ends*, not when it was minted: a dead socket is |
| + redialled for 13 s, a backhaul waits up to 30 s for headers, and an |
| + abandoned wait on a wedged instance can run for minutes. The listing |
| + prunes by hour (`internal/applog/read.go`), so a wide window inside |
| + one hour costs nothing extra. |
| +3. **Exact:** records whose `Refs` contains the reference. The answer is |
| + the group, with that occurrence's host and release, and the group's |
| + first sample's path and status, labelled as the sample (§3.4). It says |
| + which tab the group lives on. Two exact matches (a collision, §3.1) |
| + are both shown. |
| +4. **Near:** when nothing matched exactly, the groups with an occurrence |
| + overlapping `[t, t + 60 s]` on that app, plainly labelled "this |
| + reference was not stored individually; these failed within a minute |
| + of it". The minute covers the dial retry and the abandoned grace. |
| +5. **Logs:** lines in `[t − 5 s, t + 60 s]` whose message or any field |
| + value contains the reference. `applog.Filter.Grep` matches the message |
| + only, and `Filter.Fields` is exact equality on a named key, so this |
| + needs a new any-field substring match; JSON log lines put |
| + `request_id` in fields, tagged plain-text lines put it in the message. |
| + The read decodes every chunk in the hour, so this costs what |
| + `carlos logs --since 1h` costs for the app. |
| +6. **Not shipped yet:** when `t` is within about a minute of now, or the |
| + app's newest chunk is older than `t`, the answer is "this reference is |
| + too recent to have been shipped; try again in a minute". Boxes ship |
| + errors every 10 s and hold a failed upload for the next tick. |
| +7. **Nothing:** "no error or log line carries this reference". This is |
| + also the honest answer for a request that succeeded but that the |
| + customer thought was slow, and support should hear it plainly. |
| + |
| +App-scoped in v1: support knows which app the customer was using. A |
| +cross-app lookup needs an account-level door with its own gate (§6). |
| + |
| +## 4. Nothing here wakes an instance |
| + |
| +Minting and header writes happen on paths every request already takes. |
| +Lookup is a console read over chunks the agent already ships. Nothing |
| +new touches the activator ahead of the cache check. The tunnel |
| +flag is one context value. |
| + |
| +## 5. Wire shapes |
| + |
| +- `X-Request-Id` on requests to apps is new traffic, not a changed shape. |
| + An app that already sets its own ID from a missing header keeps doing |
| + so; an app that trusted a client-supplied `X-Request-Id` now sees the |
| + edge's instead. |
| +- `refs` on the errors record is `omitempty` and additive. |
| +- A box on an older build mints nothing and stores no refs. Lookup falls |
| + back to "near" for its occurrences, and the answer says so. |
| + |
| +## 6. What is not built |
| + |
| +- **A cross-app lookup.** An account-level `carlos errors --ref` with no |
| + `--app` means reading every app's window, and needs its own gate. |
| +- **Honouring a customer's own proxy's ID** (for example a CDN's ray ID). |
| + Ruled out for now (§8.2). |
| +- **An edge access log.** `applog.StreamEdge` is reserved and unbuilt. With |
| + it, every request, successful or not, would be findable by reference. |
| + That is the natural next step, and it is a different cost decision |
| + (storage per request, not per failure). |
| + |
| +## 7. Rollout |
| + |
| +1. **Mint and carry.** The edge mints, strips, carries across hops and the |
| + tunnel, sets the request and response headers, and keeps the reference |
| + out of the cache. Its own error pages show it. Useful on its own: |
| + support can already ask for the reference and grep logs for it. |
| +2. **Store and look up.** `Refs` on the record, `?ref=` on both APIs, |
| + `--ref` on `carlos errors` and `carlos availability`, the Reference box |
| + on both tabs. Docs. |
| +3. **Apps show it.** titogo prints `X-Request-Id` on its 500 page. That |
| + change is in titogo, not here; the docs paragraph in step 2 is the |
| + recipe. |
| + |
| +Each step is its own branch. Step 1 touches the edge serving path and gets |
| +the Opus-then-Fable review AGENTS.md asks for. |
| + |
| +## 8. Rulings (Bill, 2026-09-24) |
| + |
| +Four questions were open in the draft; Bill answered all four. |
| + |
| +1. **The reference is returned on every response (ruling)**, not only on |
| + errors. It costs one header, and it answers "which request was the slow |
| + one" from dev tools, including for a request that succeeded. |
| +2. **A client-supplied `X-Request-Id` is ignored (ruling).** It is deleted |
| + before the edge mints its own, and not kept. Keeping it as a |
| + `client_id` field, so a customer behind their own CDN can correlate the |
| + two IDs, stays an additive later step (§6). |
| +3. **`MaxRefs = 20` (ruling).** |
| +4. **The `R-` prefix and the 4-4-4 grouping stand (ruling).** |
| + |
| +## 9. Testing |
| + |
| +- **Format:** round-trip encode and decode; case-insensitive decode; |
| + look-alike characters (I→1, L→1, O→0); rejection of wrong lengths and |
| + prefixes; the time half survives the round trip to the second. |
| +- **Minting:** a client-supplied `X-Request-Id` never reaches the app; a |
| + verified hop's reference survives to the home box; an unverified hop's |
| + does not; a tunnel-arrived request keeps the front box's; every other |
| + request gets a fresh one. |
| +- **Cache:** two hits on one entry carry different references, and the |
| + stored entry holds none. |
| +- **Error pages:** every edge-written 502/503/504 body names the reference |
| + the response header carries. |
| +- **Storage:** twenty-one occurrences in one tick store twenty refs and a |
| + count of twenty-one; the overflow row carries none. |
| +- **Response header:** present and single-valued with gzip on and off, |
| + after a 103 Early Hints, when the app echoes its own `X-Request-Id`, |
| + and on each early-return path; on a 101 upgrade, present. |
| +- **Trust:** a non-reference value on a verified hop or over the tunnel is |
| + replaced, not kept. |
| +- **Lookup:** exact (including two matches), near, logs, not-shipped-yet |
| + and nothing, each against seeded chunks; a failure filed 13 s and 10 |
| + minutes after its reference's second is still an exact match; the |
| + malformed and future cases. |
| + |
| +## 10. Review round 1 (Fable, 2026-09-24) |
| + |
| +Approve-with-amendments. Every amendment was taken: |
| + |
| +- **Response header site.** The first draft set it at the mint site. The |
| + gzip writer, 103 handling and `Add` copying would have lost or |
| + duplicated it on most responses. It now lives in the counting writer's |
| + `WriteHeader`, with the upgrade and early-return cases named (§3.2). |
| +- **One carrier.** `X-Request-Id` on the inbound request, not a |
| + `setForwarded` change (§3.2). |
| +- **Mixed-version trust.** Inbound values on trusted paths must parse as |
| + references (§3.2). |
| +- **Tunnel flag placement** in `internal/edge`, so the e2e test runs the |
| + production handler (§3.2). |
| +- **Birthday bound.** Stated honestly, and the split moved from 32/28 to |
| + 30/30. This is within Bill's format ruling, which fixed the prefix and |
| + grouping, not the bit split. Lookup shows collisions rather than |
| + guessing (§3.1, §3.5). |
| +- **The ship tick is 10 s, not five minutes.** The window reasoning was |
| + rebuilt on request duration instead, which is what actually moves a |
| + record past its reference's second (§2, §3.5). |
| +- **A ref names the group, not the page**, and lookup labels the sample |
| + as a sample (§3.4, §3.5). |
| +- **Overflow row clears refs** (§3.4). |
| +- **"Not shipped yet"** before "nothing" (§3.5). |
| +- **Log matching** needs an any-field substring match, and its cost is |
| + stated (§3.5). |
| +- **The cache drop** lives in `maybeStore` (§3.2); **the pending page** |
| + becomes per-request (§3.3). |
| + |
| +Declined: nothing. |
| + |