carlos / platform Public

branch:

Clone

git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1	# Request references: one ID a customer can read out, and we can find
2
3	Date: 2026-09-24
4	Status: approved (Bill, 2026-09-24). Nothing here is built. Every
5	choice is marked (decision) and can be overturned cheaply; the four in
6	§8 are Bill's (ruling)s of 2026-09-24. Builds on
7	`2026-09-07-errors-and-analytics-design.md` (§3.2 and §11, Availability).
8	Line numbers cite `main` at `38639c1e`. Reviewed by Fable
9	(approve-with-amendments); §10 records what it changed.
10
11	## 1. The idea
12
13	When something goes wrong for a customer, the support call today starts
14	with "what time was it, roughly, and what were you doing?" The Errors and
15	Availability tabs group failures well, but nothing links one customer's
16	failed request to one row.
17
18	The fix most services converge on is a request reference: a short ID
19	minted where the request enters, handed to the app, shown on every error
20	page, and stored with the error. The customer reads it out, support types
21	`carlos errors --app titogo --ref R-0K7M-3QXA-9F2C`, and the answer is the
22	group, the occurrence, and the log lines around it.
23
24	CARLOS is well placed to do this, because the edge is where every request
25	enters. The app does not need to mint anything. It only reads a header and
26	prints it on its own error page.
27
28	## 2. What is already right, and must stay right
29
30	- The edge's wrapper order is load-bearing (AGENTS.md). Minting a
31	reference is a header write near the top of `ServeHTTP`. It moves no
32	wrapper.
33	- Nothing the platform does for itself may keep an instance awake.
34	Minting is CPU only, on a path every request already takes.
35	- Hop trust is proven, never assumed. `acceptHop`
36	(`internal/edge/homemap.go:767`) verifies a backhaul hop before the
37	receiving edge believes anything it carries, and `stripForwardedHeaders`
38	discards the claims of an unproven one. A reference arriving from outside
39	is a claim like any other.
40	- Coalescing keeps one sample per group. The errors sink folds every
41	occurrence of `(fp, host, release)` between two ship ticks into one
42	record carrying the first occurrence's details (§3.2 of the errors
43	spec). The tick is 10 s (`errshipTick`, `internal/host/errship.go:22`);
44	the errors spec's "five minutes" describes a design the ship loop never
45	used. That is what keeps an error storm cheap. It also means a
46	reference cannot simply be "the occurrence's": most occurrences have no
47	record of their own. §3.4 is the answer.
48	- Cached responses replay stored headers. `writeCached`
49	(`internal/edge/edge.go:1897`) copies the stored header map onto the
50	response wholesale. A reference captured into the cache would be
51	replayed to every later visitor, so the reference must never be stored
52	there (§3.2).
53	- Response headers set early do not survive. The gzip writer keeps a
54	private header map and, at its final `WriteHeader`, deletes every key on
55	the real writer's map before copying its own down
56	(`internal/edge/compress.go:211-245`). `httputil.ReverseProxy` clears
57	the handler-visible map after a 103 Early Hints, and copies upstream
58	headers with `Add`, not `Set`. A response header set at the top of
59	`ServeHTTP` is therefore wiped on most responses, or duplicated when the
60	app echoes its own (§3.2).
61
62	## 3. The change
63
64	### 3.1 The reference
65
66	Format (decision, amended in review). `R-` followed by twelve Crockford
67	base32 characters, shown in three groups of four: `R-0K7M-3QXA-9F2C`. The
68	60 bits are:
69
70	- 30 bits: seconds since 2026-01-01T00:00:00Z, which lasts until 2060;
71	- 30 bits: random.
72
73	Crockford base32 has no I, L, O or U, and decoding is case-insensitive
74	and maps those look-alikes to 1 and 0. That makes the reference safe to
75	read aloud, type from a screenshot, or copy with the wrong case.
76	Uniqueness only has to hold within one second, and the honest measure is
77	the birthday bound, not the size of the space. With n requests in one
78	second the chance of any collision is about n²/2^31. At 100 requests a
79	second that is about one pair every two or three days. At 1,000 a second
80	it is about 40 a day. The first draft's 28 random bits gave four times that. A collision
81	only matters when both requests fail and one is looked up; lookup then
82	shows both occurrences rather than guessing (§3.5). The random half comes
83	from `math/rand/v2`. The reference is an identifier, not a secret, and
84	nothing authorises on it.
85
86	Why the time is inside it (decision). Lookup must not need an index.
87	The reference decodes to the second it was minted, so the console reads
88	one narrow window of chunks and never scans the day (§3.5). The reference
89	does not encode the box or the app. The window across every box of one
90	app is small, and a box label in a customer-facing string would be
91	infrastructure detail leaking into support conversations.
92
93	### 3.2 Where it is minted, and who may carry one in
94
95	Minted once, at the first CARLOS edge the request reaches (decision).
96	In `ServeHTTP`, immediately after `acceptHop` and before the route
97	lookup, so the backhaul and tunnel branches below carry it on:
98
99	- a request on a verified hop (`hopVerified`) keeps the reference the
100	sending edge minted;
101	- a request that arrived over this box's own tunnel client keeps the
102	front box's reference. Today nothing marks such a request: the tunnel
103	client serves the edge as a plain handler
104	(`internal/host/tunnel_client.go:53`). The flag is set by a wrapper that
105	lives in `internal/edge` (an exported `TunnelArrived(h)`), so the edge's
106	own tunnel e2e test drives the same handler production runs. The
107	channel is one this box dialled with TLS verification
108	(`internal/tunnelclient/tunnelclient.go:207`), so the hub is the only
109	possible sender. The two trusted marks never coexist: the front box
110	deletes hop headers before it proxies over the tunnel;
111	- **on both trusted paths the inbound value is kept only if it parses as
112	a reference** (`R-` and twelve Crockford characters); otherwise a new
113	one is minted. During a rollout the front box may be on an older build
114	that neither mints nor strips, and without this check a client's own
115	`X-Request-Id` would pass through it and be trusted by the new box
116	behind it;
117	- every other request gets a new reference, and any `X-Request-Id` the
118	client sent is deleted first. A client-supplied value would let anyone
119	plant a reference that collides with a real one or points support at the
120	wrong occurrence.
121
122	Carried as `X-Request-Id` on the inbound request (decision). The mint
123	site sets it on `r.Header`, and that one value is the carrier everywhere:
124	`httputil.ReverseProxy` clones inbound headers to the outbound request,
125	so edges and the app receive it with no Rewrite change; the activator's
126	wake 503s, the static handler and the pending page read it from the same
127	request. It is the name Rails' `ActionDispatch::RequestId`, most Go
128	middleware, and most log shippers already read, so a Rails app gets it
129	into its tagged logs and into `request.request_id` with no change.
130
131	Returned on every response (ruling, §8.1). The edge sets
132	`X-Request-Id` on the response too, so it shows in browser dev tools and
133	in any `curl -I`. The value is the edge's reference, replacing anything
134	the upstream set. An app echoing its own ID back does not get to change
135	the one support will search for.
136
137	Where it is set matters (§2). It is set with `Header().Set` in
138	`countingWriter.WriteHeader` (`internal/edge/usagemeter.go:80`) for any
139	status of 200 or more. The counting writer is the bottom wrapper, so this
140	runs after the gzip writer has synced its private map and after
141	`ReverseProxy` has added the upstream's headers: the value replaces
142	rather than duplicates. No new wrapper is added and none moves. Two
143	cases fall outside it:
144
145	- WebSocket upgrades. `ReverseProxy` writes a 101's headers straight
146	to the hijacked connection and never calls `WriteHeader`. For an
147	upgrade request the edge sets the header on `w.Header()` just before
148	proxying. If the app also sets one, the 101 carries both; this is
149	accepted and documented rather than worked around.
150	- Paths that answer before the counting writer exists: the steering
151	probe, the tunnel hub and store door, the stale-hop refusal, the
152	backhaul sender's own 502/503, the unknown-host answer and the pending
153	page. No gzip writer wraps these, so each sets the header directly.
154
155	Never cached. `maybeStore` deletes `X-Request-Id` from the header map
156	it clones (`internal/edge/edge.go:1720`). That covers both shapes: with
157	gzip on the clone is the gzip writer's private map, with it off the real
158	one. `writeCached` needs no change, because the counting writer's
159	`WriteHeader` sets the current request's reference after the replay. A
160	test pins that two requests served from one cache entry carry different
161	references.
162
163	### 3.3 Shown on every error page the edge writes
164
165	Every status the edge produces itself says the reference:
166
167	- the 502s written by the proxy error handlers;
168	- the activator's wake-path 503s;
169	- the fleet-dark and no-healthy-upstream 503s;
170	- the static-site 502;
171	- the pending-host retry page (`internal/edge/pending.go`).
172
173	The plain-text bodies gain one line, `Reference: R-0K7M-3QXA-9F2C`. The
174	HTML retry page shows it in small type under its message. That page is
175	rendered once today (`e.pendingBody`, `pending.go:137,186`) and becomes
176	per-request; the plan checks `pending_script_test.go` for anything that
177	pins its bytes or its script hash. The format is
178	fixed so a support person can recognise it on a screenshot.
179
180	The app's own error pages are the app's. The platform's part is the
181	header. The getting-started skill and `docs/site` get a paragraph on
182	showing `X-Request-Id` on a 500 page, with Rails and Go examples.
183
184	### 3.4 Stored with the error
185
186	`refs` on the errors record (decision). `apperr.Record` gains
187	`Refs []string` (`omitempty`, additive). The sink appends the reference of
188	each occurrence it folds into a record, up to `MaxRefs = 20` per record
189	per ship tick. `Refs` holds the first twenty, in order. `Count` already
190	says how many there were. The sink clears `Refs` on the `_overflow` row
191	it builds from the first excess record (`sink.go:104-110` copies the
192	whole record), so the overflow row never carries any.
193
194	A ref names the group, not the occurrence's page. A folded occurrence
195	only increments `Count` and moves `Last` (`sink.go:96-100`). Host and
196	release are exact, because they are part of the group's key. Path,
197	status and fields are the group's first sample. On Availability, where
198	groups are fingerprinted by cause and not by path, the fifteenth ref's
199	page may differ from the sample's. Lookup says so (§3.5). Storing a path
200	beside each ref is an additive later step if support finds it wants one.
201
202	Twenty is enough for the support case, a person who hit an error and
203	called. It keeps the record's size bounded in an error storm: 20 × 16
204	bytes, against a stack that is already capped at 8 KB. A storm is not
205	the case where one reference matters: a thousand customers hit the same
206	thing, and the group itself is the answer.
207
208	A reference past the cap is still findable, less precisely. Lookup
209	(§3.5) falls back to what the reference's time says: the groups on that
210	app with an occurrence overlapping that second, labelled as such.
211
212	Every source carries it.
213
214	- `edge`: the observing defer already has the request, so it reads the
215	reference from the request.
216	- `log`: an app that logs its `X-Request-Id` has the reference in the line
217	itself. Lookup greps the window for it.
218	- `server` (rollout step 4 of the errors spec, not built): the verb gains a
219	`ref` field; the app passes the header's value.
220	- `browser` (step 3, not built): out of scope here. A page cannot read its
221	own response headers. The snippet would need the reference rendered
222	into the page, which is the app's to do.
223
224	### 3.5 Looking one up
225
226	`carlos errors --app <a> --ref <R-…>` (decision), and a Reference box
227	on both the Errors and Availability tabs, answered by one function
228	through the API as `?ref=` on both paths:
229
230	1. Decode the reference to its second `t`. A malformed reference is a 400
231	naming the expected shape. A reference from the future, or from before
232	the app existed, says so rather than searching nothing.
233	2. Read both families over `[t − 1 min, t + 30 min]`. The failure is
234	filed when the request ends, not when it was minted: a dead socket is
235	redialled for 13 s, a backhaul waits up to 30 s for headers, and an
236	abandoned wait on a wedged instance can run for minutes. The listing
237	prunes by hour (`internal/applog/read.go`), so a wide window inside
238	one hour costs nothing extra.
239	3. Exact: records whose `Refs` contains the reference. The answer is
240	the group, with that occurrence's host and release, and the group's
241	first sample's path and status, labelled as the sample (§3.4). It says
242	which tab the group lives on. Two exact matches (a collision, §3.1)
243	are both shown.
244	4. Near: when nothing matched exactly, the groups with an occurrence
245	overlapping `[t, t + 60 s]` on that app, plainly labelled "this
246	reference was not stored individually; these failed within a minute
247	of it". The minute covers the dial retry and the abandoned grace.
248	5. Logs: lines in `[t − 5 s, t + 60 s]` whose message or any field
249	value contains the reference. `applog.Filter.Grep` matches the message
250	only, and `Filter.Fields` is exact equality on a named key, so this
251	needs a new any-field substring match; JSON log lines put
252	`request_id` in fields, tagged plain-text lines put it in the message.
253	The read decodes every chunk in the hour, so this costs what
254	`carlos logs --since 1h` costs for the app.
255	6. Not shipped yet: when `t` is within about a minute of now, or the
256	app's newest chunk is older than `t`, the answer is "this reference is
257	too recent to have been shipped; try again in a minute". Boxes ship
258	errors every 10 s and hold a failed upload for the next tick.
259	7. Nothing: "no error or log line carries this reference". This is
260	also the honest answer for a request that succeeded but that the
261	customer thought was slow, and support should hear it plainly.
262
263	App-scoped in v1: support knows which app the customer was using. A
264	cross-app lookup needs an account-level door with its own gate (§6).
265
266	## 4. Nothing here wakes an instance
267
268	Minting and header writes happen on paths every request already takes.
269	Lookup is a console read over chunks the agent already ships. Nothing
270	new touches the activator ahead of the cache check. The tunnel
271	flag is one context value.
272
273	## 5. Wire shapes
274
275	- `X-Request-Id` on requests to apps is new traffic, not a changed shape.
276	An app that already sets its own ID from a missing header keeps doing
277	so; an app that trusted a client-supplied `X-Request-Id` now sees the
278	edge's instead.
279	- `refs` on the errors record is `omitempty` and additive.
280	- A box on an older build mints nothing and stores no refs. Lookup falls
281	back to "near" for its occurrences, and the answer says so.
282
283	## 6. What is not built
284
285	- A cross-app lookup. An account-level `carlos errors --ref` with no
286	`--app` means reading every app's window, and needs its own gate.
287	- Honouring a customer's own proxy's ID (for example a CDN's ray ID).
288	Ruled out for now (§8.2).
289	- An edge access log. `applog.StreamEdge` is reserved and unbuilt. With
290	it, every request, successful or not, would be findable by reference.
291	That is the natural next step, and it is a different cost decision
292	(storage per request, not per failure).
293
294	## 7. Rollout
295
296	1. Mint and carry. The edge mints, strips, carries across hops and the
297	tunnel, sets the request and response headers, and keeps the reference
298	out of the cache. Its own error pages show it. Useful on its own:
299	support can already ask for the reference and grep logs for it.
300	2. Store and look up. `Refs` on the record, `?ref=` on both APIs,
301	`--ref` on `carlos errors` and `carlos availability`, the Reference box
302	on both tabs. Docs.
303	3. Apps show it. titogo prints `X-Request-Id` on its 500 page. That
304	change is in titogo, not here; the docs paragraph in step 2 is the
305	recipe.
306
307	Each step is its own branch. Step 1 touches the edge serving path and gets
308	the Opus-then-Fable review AGENTS.md asks for.
309
310	## 8. Rulings (Bill, 2026-09-24)
311
312	Four questions were open in the draft; Bill answered all four.
313
314	1. The reference is returned on every response (ruling), not only on
315	errors. It costs one header, and it answers "which request was the slow
316	one" from dev tools, including for a request that succeeded.
317	2. A client-supplied `X-Request-Id` is ignored (ruling). It is deleted
318	before the edge mints its own, and not kept. Keeping it as a
319	`client_id` field, so a customer behind their own CDN can correlate the
320	two IDs, stays an additive later step (§6).
321	3. `MaxRefs = 20` (ruling).
322	4. The `R-` prefix and the 4-4-4 grouping stand (ruling).
323
324	## 9. Testing
325
326	- Format: round-trip encode and decode; case-insensitive decode;
327	look-alike characters (I→1, L→1, O→0); rejection of wrong lengths and
328	prefixes; the time half survives the round trip to the second.
329	- Minting: a client-supplied `X-Request-Id` never reaches the app; a
330	verified hop's reference survives to the home box; an unverified hop's
331	does not; a tunnel-arrived request keeps the front box's; every other
332	request gets a fresh one.
333	- Cache: two hits on one entry carry different references, and the
334	stored entry holds none.
335	- Error pages: every edge-written 502/503/504 body names the reference
336	the response header carries.
337	- Storage: twenty-one occurrences in one tick store twenty refs and a
338	count of twenty-one; the overflow row carries none.
339	- Response header: present and single-valued with gzip on and off,
340	after a 103 Early Hints, when the app echoes its own `X-Request-Id`,
341	and on each early-return path; on a 101 upgrade, present.
342	- Trust: a non-reference value on a verified hop or over the tunnel is
343	replaced, not kept.
344	- Lookup: exact (including two matches), near, logs, not-shipped-yet
345	and nothing, each against seeded chunks; a failure filed 13 s and 10
346	minutes after its reference's second is still an exact match; the
347	malformed and future cases.
348
349	## 10. Review round 1 (Fable, 2026-09-24)
350
351	Approve-with-amendments. Every amendment was taken:
352
353	- Response header site. The first draft set it at the mint site. The
354	gzip writer, 103 handling and `Add` copying would have lost or
355	duplicated it on most responses. It now lives in the counting writer's
356	`WriteHeader`, with the upgrade and early-return cases named (§3.2).
357	- One carrier. `X-Request-Id` on the inbound request, not a
358	`setForwarded` change (§3.2).
359	- Mixed-version trust. Inbound values on trusted paths must parse as
360	references (§3.2).
361	- Tunnel flag placement in `internal/edge`, so the e2e test runs the
362	production handler (§3.2).
363	- Birthday bound. Stated honestly, and the split moved from 32/28 to
364	30/30. This is within Bill's format ruling, which fixed the prefix and
365	grouping, not the bit split. Lookup shows collisions rather than
366	guessing (§3.1, §3.5).
367	- The ship tick is 10 s, not five minutes. The window reasoning was
368	rebuilt on request duration instead, which is what actually moves a
369	record past its reference's second (§2, §3.5).
370	- A ref names the group, not the page, and lookup labels the sample
371	as a sample (§3.4, §3.5).
372	- Overflow row clears refs (§3.4).
373	- "Not shipped yet" before "nothing" (§3.5).
374	- Log matching needs an any-field substring match, and its cost is
375	stated (§3.5).
376	- The cache drop lives in `maybeStore` (§3.2); the pending page
377	becomes per-request (§3.3).
378
379	Declined: nothing.
380
381