Request references: one ID a customer can read out, and we can find

Date: 2026-09-24 Status: approved (Bill, 2026-09-24). Nothing here is built. Every choice is marked (decision) and can be overturned cheaply; the four in §8 are Bill's (ruling)s of 2026-09-24. Builds on 2026-09-07-errors-and-analytics-design.md (§3.2 and §11, Availability). Line numbers cite main at 38639c1e. Reviewed by Fable (approve-with-amendments); §10 records what it changed.

1. The idea

When something goes wrong for a customer, the support call today starts with "what time was it, roughly, and what were you doing?" The Errors and Availability tabs group failures well, but nothing links one customer's failed request to one row.

The fix most services converge on is a request reference: a short ID minted where the request enters, handed to the app, shown on every error page, and stored with the error. The customer reads it out, support types carlos errors --app titogo --ref R-0K7M-3QXA-9F2C, and the answer is the group, the occurrence, and the log lines around it.

CARLOS is well placed to do this, because the edge is where every request enters. The app does not need to mint anything. It only reads a header and prints it on its own error page.

2. What is already right, and must stay right

• The edge's wrapper order is load-bearing (AGENTS.md). Minting a reference is a header write near the top of ServeHTTP. It moves no wrapper.
• Nothing the platform does for itself may keep an instance awake. Minting is CPU only, on a path every request already takes.
• Hop trust is proven, never assumed. acceptHop (internal/edge/homemap.go:767) verifies a backhaul hop before the receiving edge believes anything it carries, and stripForwardedHeaders discards the claims of an unproven one. A reference arriving from outside is a claim like any other.
• Coalescing keeps one sample per group. The errors sink folds every occurrence of (fp, host, release) between two ship ticks into one record carrying the first occurrence's details (§3.2 of the errors spec). The tick is 10 s (errshipTick, internal/host/errship.go:22); the errors spec's "five minutes" describes a design the ship loop never used. That is what keeps an error storm cheap. It also means a reference cannot simply be "the occurrence's": most occurrences have no record of their own. §3.4 is the answer.
• Cached responses replay stored headers. writeCached (internal/edge/edge.go:1897) copies the stored header map onto the response wholesale. A reference captured into the cache would be replayed to every later visitor, so the reference must never be stored there (§3.2).
• Response headers set early do not survive. The gzip writer keeps a private header map and, at its final WriteHeader, deletes every key on the real writer's map before copying its own down (internal/edge/compress.go:211-245). httputil.ReverseProxy clears the handler-visible map after a 103 Early Hints, and copies upstream headers with Add, not Set. A response header set at the top of ServeHTTP is therefore wiped on most responses, or duplicated when the app echoes its own (§3.2).

3. The change

3.1 The reference

Format (decision, amended in review). R- followed by twelve Crockford base32 characters, shown in three groups of four: R-0K7M-3QXA-9F2C. The 60 bits are:

• 30 bits: seconds since 2026-01-01T00:00:00Z, which lasts until 2060;
• 30 bits: random.

Crockford base32 has no I, L, O or U, and decoding is case-insensitive and maps those look-alikes to 1 and 0. That makes the reference safe to read aloud, type from a screenshot, or copy with the wrong case. Uniqueness only has to hold within one second, and the honest measure is the birthday bound, not the size of the space. With n requests in one second the chance of any collision is about n²/2^31. At 100 requests a second that is about one pair every two or three days. At 1,000 a second it is about 40 a day. The first draft's 28 random bits gave four times that. A collision only matters when both requests fail and one is looked up; lookup then shows both occurrences rather than guessing (§3.5). The random half comes from math/rand/v2. The reference is an identifier, not a secret, and nothing authorises on it.

Why the time is inside it (decision). Lookup must not need an index. The reference decodes to the second it was minted, so the console reads one narrow window of chunks and never scans the day (§3.5). The reference does not encode the box or the app. The window across every box of one app is small, and a box label in a customer-facing string would be infrastructure detail leaking into support conversations.

3.2 Where it is minted, and who may carry one in

Minted once, at the first CARLOS edge the request reaches (decision). In ServeHTTP, immediately after acceptHop and before the route lookup, so the backhaul and tunnel branches below carry it on:

• a request on a verified hop (hopVerified) keeps the reference the sending edge minted;
• a request that arrived over this box's own tunnel client keeps the front box's reference. Today nothing marks such a request: the tunnel client serves the edge as a plain handler (internal/host/tunnel_client.go:53). The flag is set by a wrapper that lives in internal/edge (an exported TunnelArrived(h)), so the edge's own tunnel e2e test drives the same handler production runs. The channel is one this box dialled with TLS verification (internal/tunnelclient/tunnelclient.go:207), so the hub is the only possible sender. The two trusted marks never coexist: the front box deletes hop headers before it proxies over the tunnel;
• on both trusted paths the inbound value is kept only if it parses as a reference (R- and twelve Crockford characters); otherwise a new one is minted. During a rollout the front box may be on an older build that neither mints nor strips, and without this check a client's own X-Request-Id would pass through it and be trusted by the new box behind it;
• every other request gets a new reference, and any X-Request-Id the client sent is deleted first. A client-supplied value would let anyone plant a reference that collides with a real one or points support at the wrong occurrence.

Carried as X-Request-Id on the inbound request (decision). The mint site sets it on r.Header, and that one value is the carrier everywhere: httputil.ReverseProxy clones inbound headers to the outbound request, so edges and the app receive it with no Rewrite change; the activator's wake 503s, the static handler and the pending page read it from the same request. It is the name Rails' ActionDispatch::RequestId, most Go middleware, and most log shippers already read, so a Rails app gets it into its tagged logs and into request.request_id with no change.

Returned on every response (ruling, §8.1). The edge sets X-Request-Id on the response too, so it shows in browser dev tools and in any curl -I. The value is the edge's reference, replacing anything the upstream set. An app echoing its own ID back does not get to change the one support will search for.

Where it is set matters (§2). It is set with Header().Set in countingWriter.WriteHeader (internal/edge/usagemeter.go:80) for any status of 200 or more. The counting writer is the bottom wrapper, so this runs after the gzip writer has synced its private map and after ReverseProxy has added the upstream's headers: the value replaces rather than duplicates. No new wrapper is added and none moves. Two cases fall outside it:

• WebSocket upgrades. ReverseProxy writes a 101's headers straight to the hijacked connection and never calls WriteHeader. For an upgrade request the edge sets the header on w.Header() just before proxying. If the app also sets one, the 101 carries both; this is accepted and documented rather than worked around.
• Paths that answer before the counting writer exists: the steering probe, the tunnel hub and store door, the stale-hop refusal, the backhaul sender's own 502/503, the unknown-host answer and the pending page. No gzip writer wraps these, so each sets the header directly.

Never cached. maybeStore deletes X-Request-Id from the header map it clones (internal/edge/edge.go:1720). That covers both shapes: with gzip on the clone is the gzip writer's private map, with it off the real one. writeCached needs no change, because the counting writer's WriteHeader sets the current request's reference after the replay. A test pins that two requests served from one cache entry carry different references.

3.3 Shown on every error page the edge writes

Every status the edge produces itself says the reference:

• the 502s written by the proxy error handlers;
• the activator's wake-path 503s;
• the fleet-dark and no-healthy-upstream 503s;
• the static-site 502;
• the pending-host retry page (internal/edge/pending.go).

The plain-text bodies gain one line, Reference: R-0K7M-3QXA-9F2C. The HTML retry page shows it in small type under its message. That page is rendered once today (e.pendingBody, pending.go:137,186) and becomes per-request; the plan checks pending_script_test.go for anything that pins its bytes or its script hash. The format is fixed so a support person can recognise it on a screenshot.

The app's own error pages are the app's. The platform's part is the header. The getting-started skill and docs/site get a paragraph on showing X-Request-Id on a 500 page, with Rails and Go examples.

3.4 Stored with the error

refs on the errors record (decision). apperr.Record gains Refs []string (omitempty, additive). The sink appends the reference of each occurrence it folds into a record, up to MaxRefs = 20 per record per ship tick. Refs holds the first twenty, in order. Count already says how many there were. The sink clears Refs on the _overflow row it builds from the first excess record (sink.go:104-110 copies the whole record), so the overflow row never carries any.

A ref names the group, not the occurrence's page. A folded occurrence only increments Count and moves Last (sink.go:96-100). Host and release are exact, because they are part of the group's key. Path, status and fields are the group's first sample. On Availability, where groups are fingerprinted by cause and not by path, the fifteenth ref's page may differ from the sample's. Lookup says so (§3.5). Storing a path beside each ref is an additive later step if support finds it wants one.

Twenty is enough for the support case, a person who hit an error and called. It keeps the record's size bounded in an error storm: 20 × 16 bytes, against a stack that is already capped at 8 KB. A storm is not the case where one reference matters: a thousand customers hit the same thing, and the group itself is the answer.

A reference past the cap is still findable, less precisely. Lookup (§3.5) falls back to what the reference's time says: the groups on that app with an occurrence overlapping that second, labelled as such.

Every source carries it.

• edge: the observing defer already has the request, so it reads the reference from the request.
• log: an app that logs its X-Request-Id has the reference in the line itself. Lookup greps the window for it.
• server (rollout step 4 of the errors spec, not built): the verb gains a ref field; the app passes the header's value.
• browser (step 3, not built): out of scope here. A page cannot read its own response headers. The snippet would need the reference rendered into the page, which is the app's to do.

3.5 Looking one up

carlos errors --app <a> --ref <R-…> (decision), and a Reference box on both the Errors and Availability tabs, answered by one function through the API as ?ref= on both paths:

1. Decode the reference to its second t. A malformed reference is a 400 naming the expected shape. A reference from the future, or from before the app existed, says so rather than searching nothing.
2. Read both families over [t − 1 min, t + 30 min]. The failure is filed when the request ends, not when it was minted: a dead socket is redialled for 13 s, a backhaul waits up to 30 s for headers, and an abandoned wait on a wedged instance can run for minutes. The listing prunes by hour (internal/applog/read.go), so a wide window inside one hour costs nothing extra.
3. Exact: records whose Refs contains the reference. The answer is the group, with that occurrence's host and release, and the group's first sample's path and status, labelled as the sample (§3.4). It says which tab the group lives on. Two exact matches (a collision, §3.1) are both shown.
4. Near: when nothing matched exactly, the groups with an occurrence overlapping [t, t + 60 s] on that app, plainly labelled "this reference was not stored individually; these failed within a minute of it". The minute covers the dial retry and the abandoned grace.
5. Logs: lines in [t − 5 s, t + 60 s] whose message or any field value contains the reference. applog.Filter.Grep matches the message only, and Filter.Fields is exact equality on a named key, so this needs a new any-field substring match; JSON log lines put request_id in fields, tagged plain-text lines put it in the message. The read decodes every chunk in the hour, so this costs what carlos logs --since 1h costs for the app.
6. Not shipped yet: when t is within about a minute of now, or the app's newest chunk is older than t, the answer is "this reference is too recent to have been shipped; try again in a minute". Boxes ship errors every 10 s and hold a failed upload for the next tick.
7. Nothing: "no error or log line carries this reference". This is also the honest answer for a request that succeeded but that the customer thought was slow, and support should hear it plainly.

App-scoped in v1: support knows which app the customer was using. A cross-app lookup needs an account-level door with its own gate (§6).

4. Nothing here wakes an instance

Minting and header writes happen on paths every request already takes. Lookup is a console read over chunks the agent already ships. Nothing new touches the activator ahead of the cache check. The tunnel flag is one context value.

5. Wire shapes

• X-Request-Id on requests to apps is new traffic, not a changed shape. An app that already sets its own ID from a missing header keeps doing so; an app that trusted a client-supplied X-Request-Id now sees the edge's instead.
• refs on the errors record is omitempty and additive.
• A box on an older build mints nothing and stores no refs. Lookup falls back to "near" for its occurrences, and the answer says so.

6. What is not built

• A cross-app lookup. An account-level carlos errors --ref with no --app means reading every app's window, and needs its own gate.
• Honouring a customer's own proxy's ID (for example a CDN's ray ID). Ruled out for now (§8.2).
• An edge access log. applog.StreamEdge is reserved and unbuilt. With it, every request, successful or not, would be findable by reference. That is the natural next step, and it is a different cost decision (storage per request, not per failure).

7. Rollout

1. Mint and carry. The edge mints, strips, carries across hops and the tunnel, sets the request and response headers, and keeps the reference out of the cache. Its own error pages show it. Useful on its own: support can already ask for the reference and grep logs for it.
2. Store and look up. Refs on the record, ?ref= on both APIs, --ref on carlos errors and carlos availability, the Reference box on both tabs. Docs.
3. Apps show it. titogo prints X-Request-Id on its 500 page. That change is in titogo, not here; the docs paragraph in step 2 is the recipe.

Each step is its own branch. Step 1 touches the edge serving path and gets the Opus-then-Fable review AGENTS.md asks for.

8. Rulings (Bill, 2026-09-24)

Four questions were open in the draft; Bill answered all four.

1. The reference is returned on every response (ruling), not only on errors. It costs one header, and it answers "which request was the slow one" from dev tools, including for a request that succeeded.
2. A client-supplied X-Request-Id is ignored (ruling). It is deleted before the edge mints its own, and not kept. Keeping it as a client_id field, so a customer behind their own CDN can correlate the two IDs, stays an additive later step (§6).
3. MaxRefs = 20 (ruling).
4. The R- prefix and the 4-4-4 grouping stand (ruling).

9. Testing

• Format: round-trip encode and decode; case-insensitive decode; look-alike characters (I→1, L→1, O→0); rejection of wrong lengths and prefixes; the time half survives the round trip to the second.
• Minting: a client-supplied X-Request-Id never reaches the app; a verified hop's reference survives to the home box; an unverified hop's does not; a tunnel-arrived request keeps the front box's; every other request gets a fresh one.
• Cache: two hits on one entry carry different references, and the stored entry holds none.
• Error pages: every edge-written 502/503/504 body names the reference the response header carries.
• Storage: twenty-one occurrences in one tick store twenty refs and a count of twenty-one; the overflow row carries none.
• Response header: present and single-valued with gzip on and off, after a 103 Early Hints, when the app echoes its own X-Request-Id, and on each early-return path; on a 101 upgrade, present.
• Trust: a non-reference value on a verified hop or over the tunnel is replaced, not kept.
• Lookup: exact (including two matches), near, logs, not-shipped-yet and nothing, each against seeded chunks; a failure filed 13 s and 10 minutes after its reference's second is still an exact match; the malformed and future cases.

10. Review round 1 (Fable, 2026-09-24)

Approve-with-amendments. Every amendment was taken:

• Response header site. The first draft set it at the mint site. The gzip writer, 103 handling and Add copying would have lost or duplicated it on most responses. It now lives in the counting writer's WriteHeader, with the upgrade and early-return cases named (§3.2).
• One carrier. X-Request-Id on the inbound request, not a setForwarded change (§3.2).
• Mixed-version trust. Inbound values on trusted paths must parse as references (§3.2).
• Tunnel flag placement in internal/edge, so the e2e test runs the production handler (§3.2).
• Birthday bound. Stated honestly, and the split moved from 32/28 to 30/30. This is within Bill's format ruling, which fixed the prefix and grouping, not the bit split. Lookup shows collisions rather than guessing (§3.1, §3.5).
• The ship tick is 10 s, not five minutes. The window reasoning was rebuilt on request duration instead, which is what actually moves a record past its reference's second (§2, §3.5).
• A ref names the group, not the page, and lookup labels the sample as a sample (§3.4, §3.5).
• Overflow row clears refs (§3.4).
• "Not shipped yet" before "nothing" (§3.5).
• Log matching needs an any-field substring match, and its cost is stated (§3.5).
• The cache drop lives in maybeStore (§3.2); the pending page becomes per-request (§3.3).

Declined: nothing.