amadan
spec: Errors and Analytics tabs for every app (DRAFT for review)
Design for two app-dashboard tabs fed by what the platform already sees: log-level errors and edge 5xx/wake failures for Errors; edge-counted pageviews and salted-hash visitors (HLL, no cookies) for Analytics. Adds a browser door on /_carlos/ answered between route lookup and the meter wrap so a hibernating instance never wakes, and a server door on the control socket beside schedule.Mount. Two review rounds from a second Fable context and Codex are folded in and recorded in sections 9 and 10.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017PvHDGSqVe2EbPJgqYMHDa
1 file changed,
+775
−0
docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md+775 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md b/docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md| new file mode 100644 |
| index 000000000..a6594b56c |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md |
| @@ -0,0 +1,775 @@ |
| +# Errors and Analytics: two tabs every app gets for free |
| + |
| +**Date:** 2026-09-07 |
| +**Status:** DRAFT for Paul's review. Nothing here has been approved. Every |
| +choice is marked **(decision)** and can be overturned cheaply; none is a |
| +ruling. Drafted by Claude Fable, then reviewed by a second Fable context |
| +and by Codex; both returned needs-rework on the first draft and §9 records |
| +what they changed. Line numbers cite `origin/main` at `09a22d6`. |
| + |
| +## 1. The idea, and why it fits this platform |
| + |
| +A member who ships an app to CARLOS today gets a URL, a certificate, logs, |
| +schedules, and a bill. What they do not get is the two questions every |
| +operator asks within a day of launch: *is it broken* and *is anyone using |
| +it*. The answers live in AppSignal, Datadog, Sentry, PostHog, Plausible: a |
| +second account, a second bill, an SDK to wire in, and for the analytics |
| +half a cookie banner to justify. |
| + |
| +The platform is unusually well placed to answer both questions itself, |
| +because it already stands in the two places the data passes through: |
| + |
| +- **The edge sees every request** to every app, including the ones its |
| + cache answers without waking the instance (`writeCached`, |
| + `internal/edge/edge.go:1311`). Method, path, status, referer and user |
| + agent are in hand at the point where `countingWriter` is installed to |
| + bill the bytes (`edge.go:1241`). Server-side page-view counting there |
| + costs the app nothing and needs no JavaScript and no cookie. It is the |
| + Plausible model, done where Plausible cannot stand. |
| +- **The agent already captures every line the app writes** and ships it |
| + as hour chunks to the bucket (`internal/applog`, |
| + `internal/host/logship.go`). A line the app tagged `level=error` is |
| + already an error event; it is just filed under Logs, undifferentiated |
| + from a million info lines. And when the app dies, the edge answers a |
| + 502 or reports a wake failure: the crash is already observed. |
| + |
| +So the first version of both tabs is mostly *reading what is already |
| +flowing past*. The hooks add the cases the platform cannot see on its own: |
| +a public endpoint on the app's own hostname for browser errors and named |
| +events, and a loopback verb for server-side errors and events. The member |
| +who wants AppSignal can still rent it; the event shape is deliberately |
| +close to what those services accept, so a "forward to" setting later is a |
| +translator, not a redesign (§5). |
| + |
| +The proposal is two tabs on the app dashboard, **Errors** and |
| +**Analytics**, between Logs and Activity in `appnav.html`, each with a |
| +console API and a `carlos` verb behind it (AGENTS.md: every command must |
| +complete the job for a user with no infra access). |
| + |
| +## 2. What is already right, and must stay right |
| + |
| +**Logs are the pattern for events.** `applog.Record` |
| +(`internal/applog/record.go:72`) is buffered on the box under a token |
| +budget (`sink.go:95`), flushed as gzip'd JSONL hour chunks under |
| +`apps/<sqid>/<app>/logs/<box>/<hour>/`, read back by `applog.ReadWindow` |
| +merging chunks across boxes, pruned by `logshipPrune` |
| +(`logship.go:161`) on age and byte cap, and served through |
| +`GET /api/cli/apps/{account}/{app}/logs` (`authcli.go:595`) with a cursor. |
| +Errors that the platform must store are events too; they reuse this |
| +machinery rather than growing a second one (§3.2). |
| + |
| +**Usage is the pattern for counters.** `usage.EdgeDay` |
| +(`internal/usage/edge.go`) is a CAS-merged day file, hour-bucketed, one |
| +per box, flushed every five minutes by `usageLoop` |
| +(`internal/host/usage_loop.go:27`) from `UsageMeter.Drain()`. Page views |
| +and visitors are counters; they reuse this shape, with two durability |
| +fixes the reviewers asked for (§3.3). |
| + |
| +**The control socket is the pattern for a loopback API.** `ListenControl` |
| +(`internal/edge/purge.go:108`) mounts verbs on a 0600 unix socket the app |
| +finds in `CARLOS_CONTROL_SOCKET`; `schedule.Mount` |
| +(`internal/schedule/control.go:41`) shows the bearer check against |
| +`CARLOS_ADMIN_TOKEN`. Server-side ingestion mounts beside it (§3.5). The |
| +socket is not the activator: a call on it never reaches `beginRequest`. |
| + |
| +**Nothing the platform does for itself may keep an instance awake.** The |
| +scheduled-ticks spec (2026-08-30) exists because one line of accounting |
| +cost a member fifteen minutes of billed residency per millisecond of |
| +work. Every ingest path here is handled in the edge or the agent and none |
| +enters `activator.Service.ServeHTTP`; a beacon from a browser to a |
| +hibernating app is answered by the edge and the app stays asleep (§4). |
| + |
| +**The edge's wrapper order is load-bearing** (AGENTS.md): recorder above |
| +the compressor, usage counter below it, wrap site after the upgrade |
| +branch. This design changes what the usage counter *observes*, never |
| +where it sits (§3.3). |
| + |
| +**Fleet boxes write through the store door**, whose write grant is |
| +segment-exact `apps/<sqid>/<app>/logs/**` (`aclWriteAppLogs`, |
| +`internal/edge/storedoor.go:503`), and usage is rebased under |
| +`fleets/<name>/` (`usage_loop.go:47`). Two new families need the door to |
| +know about them, or a fleet box's writes are refused (§3.2). |
| + |
| +## 3. The change |
| + |
| +### 3.1 One event shape, four sources |
| + |
| +Both tabs are fed by events in one JSON shape. **(decision)** Fields: |
| + |
| +``` |
| +ts RFC3339 nanoseconds, stamped by the platform, never taken from input |
| +source "log" | "edge" | "server" | "browser" |
| +kind errors: exception class or "http"; analytics: "pageview" | "event" |
| +name errors: the message; analytics: the event name |
| +host route host that received it (how owners name instances) |
| +box the box that recorded it |
| +release X-Carlos-Version of the serving build, when known |
| +path request path, query and fragment removed, token segments collapsed (§3.3) |
| +status HTTP status, when the event came from a request |
| +referer host-only, never the full URL |
| +ua browser family only, never the raw string |
| +stack errors only, capped at 8 KB |
| +fields map[string]string of caller-supplied scalars, 32 keys, 256 B each |
| +fp errors only: fingerprint (§3.2) |
| +fpv errors only: fingerprint algorithm version, 1 |
| +``` |
| + |
| +Sources and where they enter: |
| + |
| +| source | errors | analytics | |
| +|---------|----------------------------------------------------------|---------------------------------------------| |
| +| log | applog record with level `error` or `fatal`, read-derived | none | |
| +| edge | upstream 5xx, gateway 502/504, wake failure | every HTML 200 GET becomes a pageview | |
| +| server | `POST /errors` on the control socket | `POST /events` on the control socket | |
| +| browser | `POST /_carlos/e` on the app's own host | `POST /_carlos/a` on the app's own host | |
| + |
| +`log` and `edge` need no integration at all. They are why the tabs are not |
| +empty on day one for every app in the fleet. |
| + |
| +**Sanitisation before buffering (decision, Codex).** Errors carry |
| +messages, stacks, paths and caller fields, and those carry secrets in |
| +practice. Before any record is buffered, on every source: query strings |
| +and fragments are removed from `path`; in `name`, `stack` and `fields`, |
| +email addresses become `<email>`, `Bearer`/`Basic` credentials and |
| +`token=`/`key=`/`secret=`/`password=` values become `<redacted>`, and |
| +hex or base64 runs of 32+ characters become `<hex>`. The browser snippet |
| +applies the same rules before sending, so nothing leaves the page that |
| +the platform would strip. This is a floor, not a guarantee; the Errors |
| +tab says so in one line. |
| + |
| +### 3.2 Errors: sources, fingerprints, coalescing, storage |
| + |
| +**The `log` source is derived on read, not stored twice (decision, Fable |
| +review 7).** The first draft mirrored `level=error` lines into a second |
| +family. `ReadWindow` with `Filter.Level="error"` (`filter.go:22`) over |
| +`logs/` yields the same rows with the same retention, and a mirror would |
| +have had to choose a side of the token budget at `sink.go:95`. So the |
| +Errors tab reads two families and merges: `logs/` filtered to |
| +`error`/`fatal`, and `errors/` for the other three sources. A JSON line |
| +whose `level` is unrecognised is not an error; `ParseAppLine` never |
| +guesses, and neither does this. A plain-text `panic:` on stderr carries no |
| +level and is not a `log` error either: the crash reaches the tab through |
| +the `edge` source when the next request meets a dead instance, which is |
| +the honest signal. |
| + |
| +**Edge errors (decision).** Where the observing writer (§3.3) sees a |
| +final status of 500–599 from the upstream, or the edge itself answers |
| +502/504 because the instance did not come up, it emits `source: edge, |
| +kind: http, name: "<status> <method> <path>"`. Wake failures already reach |
| +the member through `emitEvent` in `internal/activator/logsink.go`; the |
| +same call site sends one record to the errors sink. To keep that exactly |
| +once, the activator branch marks the request context when it answered |
| +the failure itself, and the observing writer skips a 502/504 that |
| +carries the mark; the writer's own emission covers only statuses the |
| +upstream returned or the proxy produced without the activator. Client 4xx are not errors. |
| + |
| +**Fingerprint (decision, amended per both reviews).** `fp` groups |
| +occurrences into one row. A caller may supply one, capped at 64 bytes. |
| +Otherwise the platform computes `sha256(len-delimited(source, kind, |
| +norm(name), norm(top frame)))[:16]` with `fpv: 1`. `norm` splits on whitespace |
| +and punctuation; a token that is a digit run of 4+ characters, a hex run |
| +of 8+, a UUID, or a quoted string becomes a placeholder; tokens of 1–3 |
| +digits and mixed tokens like `v2` or `E404` are kept, so `HTTP 503`, |
| +`v2` and `retry 3` still separate groups while `user 4123` and |
| +`user 9911` do not. Small ids under 1,000 will split groups; that is the |
| +stated trade-off, and `fp` overrides it. A frame is normalised to file and function with line, |
| +column and bundle hash removed, so a redeploy does not split every |
| +browser group. The rule is versioned so it can change without merging |
| +old groups into new ones. It is wrong sometimes; a caller who knows |
| +better sets `fp`. |
| + |
| +**Coalescing at the box (decision, amended per Codex 8).** An error storm |
| +is when the tab must work and when a naive stream fails. The box sink |
| +keeps, per `(app, fp, host, release)` per flush interval, one record |
| +carrying `count`, `first`, `last`, and the first sample's `stack`, `path`, |
| +`status` and `fields`. Ten thousand identical panics in a minute become |
| +one record with `count: 10000`. Coalesced records are capped per app per |
| +hour at 1,000, counting `(fp, host, release)` combinations, not distinct |
| +fingerprints, so the memory bound is real (per app, not per box: one |
| +tenant's storm must not suppress a neighbour's, Codex 9); beyond that, |
| +the overflow folds into a single `_overflow` row whose `count` is |
| +occurrences, labelled as such on the tab. A record belongs to a read |
| +window when `[first, last]` overlaps it and is counted whole; the flush |
| +interval is five minutes, so a window edge is wrong by at most one |
| +interval, and the tab's window labels say "to the nearest flush". |
| + |
| +**Storage (decision, amended per Fable 6).** `errors/` is a sibling family |
| +of `logs/`: `apps/<sqid>/<app>/errors/<box>/<hour>/<seq>-<size>.jsonl.gz`. |
| +The chunk key and hour layout are shared, so `ChunkKey`, `Prefix`, |
| +`BoxPrefix`, `HourPrefix`, `ChunkKeySize`, `hourNameOfDir`, `hourNameOf`, |
| +`logshipPrune` and `logshipLoop` take a family argument. The record is |
| +not shared: `applog.Record` has none of `fp`, `count`, `first`, `last`, |
| +`stack`, `source`, `kind`, `status`, `path`, `release`, and `EncodeChunk`, |
| +`DecodeChunk`, `ReadWindow`, `Filter.Match` and `Cursor` all take |
| +`Record`. The codec becomes generic over the record type, or `Record` is |
| +widened with `omitempty` fields; the implementer picks whichever leaves |
| +`logs/` chunks byte-identical, and the existing applog tests are the |
| +proof. The consumer list for the key shape, per AGENTS.md, is: |
| +`ChunkKeySize` (`chunk.go:65`), `hourNameOfDir` (`read.go:91`), |
| +`hourNameOf` (`logship.go:216`), `aclWriteAppLogs` (`storedoor.go:503`), |
| +`logshipPrune` (`logship.go:161`). Wire shape is additive: nothing under |
| +`logs/` changes. |
| + |
| +**Store door (decision, Fable 3).** `aclWriteAppLogs` becomes a family |
| +allow-list: `logs`, `errors`, `analytics`, each with the same |
| +"at least one component below" rule, with door tests for every family |
| +and for `errors` as an app name. Without this a fleet box's writes are |
| +refused and the tabs are empty for exactly the members on fleets. |
| + |
| +### 3.3 Analytics: counting at the edge |
| + |
| +**Where the edge sees it (decision, Fable 2).** The first draft assumed a |
| +tap on the recorder. The recorder is built only on the hibernating and |
| +cacheable branches (`edge.go:1322, 1333`); a cache hit goes through |
| +`writeCached` (`:1311`) and the plain path (`:1339`), every POST included, |
| +has no recorder at all. The one wrapper installed at a single site for |
| +every response shape is `countingWriter` (`usagemeter.go:64`), which today |
| +has no `WriteHeader` and is installed only when a meter is armed |
| +(`edge.go:1241`). It becomes an *observing* writer: installed |
| +unconditionally with a nil-safe meter, latching final status and |
| +`Content-Type` at the first `WriteHeader` of 200 or above (1xx are |
| +forwarded and ignored), an implicit 200 at first `Write`, and a 200 with |
| +no `Content-Type` when the handler returns having written nothing, which |
| +is never a pageview. `Flush` and `Hijack` forward as they do today. The |
| +meter wrap today skips `Kind == "status"` routes (`edge.go:1241`); the |
| +observing writer is installed there too, but the tap ignores status |
| +routes, so the status host never enters any app's analytics. It |
| +stays below the compressor; `Content-Type` is visible there because |
| +`gzipWriter.syncHeader` copies its private map down before the underlying |
| +`WriteHeader` (`compress.go:236`). HEAD skips gzip (`compress.go:197`), so |
| +the tap excludes by method itself. The upgrade branch (`:1257`) is |
| +untouched; a 101 is never a pageview. The order rule in AGENTS.md is |
| +preserved: nothing moves, one wrapper learns to look. |
| + |
| +**What is counted (decision, amended per Codex 3).** A request is a |
| +*pageview* when the method is GET, the final status is 200, and the |
| +response media type parses as `text/html`. Redirects are not views (the |
| +destination will be); 304s are not views (no media type, and a |
| +revalidated page is an undercount the tab states); 204/205 are not; |
| +requests with `Sec-Purpose: prefetch` or `Purpose: prefetch` are not; |
| +`Sec-Fetch-Dest`, when present, must be `document` (an `empty` dest is |
| +`fetch()`, which is how htmx and SPAs load HTML fragments, and a fragment |
| +is not a page; an absent header is an old UA or a bot, and the bot list |
| +handles the latter). Requests under |
| +`/_carlos/` are never counted. A request whose user agent matches the |
| +vendored bot list is counted into `bots` and nowhere else; the list is |
| +incomplete by nature and the tab shows `bots` so the member can see what |
| +was excluded. |
| + |
| +**Client IP (decision, Fable 5).** There is no client-IP value in the |
| +edge today. `clientIP(r)` is defined as: the last element of |
| +`X-Forwarded-For` (all header lines joined, as `setForwarded` does at |
| +`homemap.go:915`) when `hopVerified(ctx)` (`homemap.go:767` accepts a |
| +verified chain verbatim; `:800` strips an unverified one), otherwise the |
| +host of `RemoteAddr`. A header the client can set never reaches it. |
| + |
| +**Visitor (decision, amended per Codex 1 and Fable 10).** A visitor is |
| +`sha256(day-salt ‖ route host ‖ clientIP ‖ user agent)[:8]`. The salt is |
| +per box per UTC day, generated at first use, written 0600 to the box's state |
| +directory beside the registry (the agent already owns it) so a restart |
| +mid-day does not re-salt and double-count, and deleted on rotation; it is never written to the bucket. |
| +No cookie is set, no IP is stored, and the platform holds nothing that |
| +joins a hash across days. Because a route is served by its home box (a |
| +box without the row backhauls, `edge.go:1216`), a visitor is normally |
| +seen by one box; during an instance move they are counted twice for a |
| +day. Multi-day ranges on the tab are labelled "daily visitors, summed", |
| +because that is what they are; Plausible's are too. |
| + |
| +**Cardinality (decision, amended per Codex 10).** Unique visitors per day |
| +per box are a HyperLogLog sketch with 2¹¹ registers (2 KB, standard error |
| +about 2.3%), one per day, encoded as base64 with a one-byte version |
| +prefix, merged across boxes by register max. Pageviews per hour are |
| +integers. Top pages, top referers and named events are Misra–Gries |
| +summaries with 500 counters each: bounded, mergeable, and correct for any |
| +app with fewer than 500 distinct keys a day, which is most of them. The |
| +reader re-folds after merging so the tab never shows more than 500 rows |
| +plus `_other`. Below 500 keys the counts are exact; above, a counter is a |
| +lower bound within the summary's error and the table is labelled |
| +"approximate: more than 500 distinct keys today". Keys are capped at 256 |
| +bytes. |
| + |
| +**Page paths (decision, Fable 10).** `pages` keys are the sanitised |
| +`path` of §3.1: query and fragment removed, and any segment that is a |
| +UUID, a hex run of 16+ characters, a base64-looking run of 20+, or an |
| +email replaced by `:token`. `/reset/8f3a…` never reaches the bucket; |
| +`/blog/2024-review` does. |
| + |
| +**Storage (decision, amended per Codex 13).** |
| +`apps/<sqid>/<app>/analytics/<box>/days/<date>.json`, written by a |
| +sibling of `UpsertEdgeDay` with the same read-etag-merge-conditional-put |
| +loop, and two differences from usage. First, buckets are keyed by *event |
| +time*, not flush time, so a flush delayed across an hour boundary lands |
| +in the right hour. Second, the drain returns deltas, and a failed flush |
| +re-queues its deltas into the next drain rather than losing them |
| +(`usageLoop` documents that it loses the window; modelling grade is fine |
| +for billing estimates, not for a number a member reads as truth). One |
| +window survives a failure; a second consecutive failure drops the older |
| +one and increments `dropped.flush`. A conditional put whose outcome is |
| +unknown (timeout after send) must not double-count on retry: every delta |
| +batch carries a box-monotonic `seq`, the day file records `seq` per box, |
| +and a merge whose `seq` is not greater than the recorded one is a no-op. A process restart loses the in-memory |
| +window, stated on the tab as "counts since the last flush may be |
| +missing" only when a restart happened that day. Shape: |
| + |
| +```json |
| +{ |
| + "v": 1, |
| + "date": "2026-09-07", |
| + "hours": {"09": {"pv": 412, "bots": 30, "bytes": 1843201}}, |
| + "visitors": "AQ…", |
| + "pages": {"/": 300, "/pricing": 80, "_other": 32}, |
| + "referers": {"news.ycombinator.com": 200, "direct": 180}, |
| + "events": {"signup": 12, "checkout": 3}, |
| + "seq": {"box-a": 1187}, |
| + "dropped": {"e": {"rate": 0, "size": 0, "origin": 0, "malformed": 0}, |
| + "a": {"rate": 0, "size": 0, "origin": 0, "malformed": 0}, |
| + "flush": 0}, |
| + "countries": {} |
| +} |
| +``` |
| + |
| +`countries` is present and empty in v1 so a GeoIP pass is additive. |
| +`dropped` is what the member sees when a cap or a failure bit: the tab |
| +shows it, so a number that is not the whole truth says so. |
| + |
| +**Named events.** `carlos.track("signup")` from the browser, or |
| +`POST /events {"name":"signup"}` from the server, increments |
| +`events[name]`. Names are lowercased, 64 bytes of `[a-z0-9_.-]`, and |
| +bounded by the same 500-counter summary. Event properties are accepted in |
| +`fields` and *not stored* in v1 (§5). |
| + |
| +**Accumulation.** The edge holds deltas in memory; the agent's |
| +five-minute `usageLoop` cadence flushes them in the same tick as usage, |
| +as a second CAS write (two objects, two writes; the first draft's "one |
| +round-trip" was wrong). |
| + |
| +**The switch (decision, Fable 9).** `features.json` is console-only and |
| +no box reads it. The switches are one bucket object |
| +`apps/<sqid>/<app>/analytics.json` |
| +`{"count": "edge" | "off", "door": "on" | "off"}`, written by the console |
| +with a serial bump and converged by boxes on the next pass like |
| +`instances/*.json`. Absent means `edge` and `on`. `count: off` stops |
| +counting at the observing writer and makes both doors drop analytics |
| +events (`/_carlos/a` and the socket's `/events`) with `202`; existing day |
| +files stay until retention. `door: off` closes the browser door |
| +entirely, errors included, for an owner who wants no public ingest |
| +surface at all. `"count": "browser"` is reserved for a later mode (§5) |
| +so the shape is additive. |
| + |
| +### 3.4 The browser door |
| + |
| +**Where in the edge (decision, Fable 1, Codex 6).** The first draft said |
| +"before any routing". That is the wrong spot: the app is known only from |
| +`up.route`, which exists after `lookupUpstreamPath` (`edge.go:1197`), and |
| +a box that has no row for the host must backhaul (`:1216`) so the home |
| +box answers. The intercept sits between the lookup (`:1197`) and the |
| +meter wrap (`:1241`): after routing, before upgrade (`:1257`), gzip |
| +(`:1275`), cache (`:1311`) and the activator (`:1322`). At that spot |
| +beacons are unbilled like `status` routes and `beginRequest` is never |
| +reached, which is what §4 promises. |
| + |
| +**Attribution on a shared hostname.** `lookupUpstreamPath` can route |
| +different path prefixes on one hostname to different apps (`:1008`). A |
| +beacon carries no app id. The snippet puts `location.pathname` in every |
| +beacon's `path`, errors and events alike, and the beacon is attributed |
| +to the upstream the lookup returns for that *raw* path on that hostname |
| +(the match runs before §3.1 sanitisation, which is applied to what is |
| +stored), and to the hostname's root app when `path` is absent or matches |
| +nothing. Because |
| +`Origin` must equal that hostname, the sender is already a page served |
| +from it: same origin, same trust boundary the browser already grants |
| +those siblings. An unknown hostname takes the existing `!ok` branch and |
| +is never answered by this code. The lookup must never accept `_carlos` as |
| +a prefix key; a test pins that. |
| + |
| +**Endpoint (decision).** |
| + |
| +``` |
| +POST /_carlos/e one error {kind, name, stack, path, fields, fp} |
| +POST /_carlos/a one event {name, fields} | {"name":"pageview","path":…} |
| +GET /_carlos/carlos.js loader, Cache-Control: max-age=300, must-revalidate |
| +GET /_carlos/carlos.<hash>.js the snippet, immutable |
| +``` |
| + |
| +Both POSTs answer `202` with an empty body in every case: accepted, |
| +rate-limited, rejected, malformed. The endpoint reveals nothing, and the |
| +browser never sees a failure worth retrying. Every drop increments the |
| +app's `dropped.rate` (or `dropped.origin`, `dropped.size`), which the |
| +tab shows, so silence outward is not silence to the owner (Codex 11). |
| +`/_carlos/` is reserved on all routes and never proxied upstream from |
| +this change on. No app in this repo serves it; §8 asks about the |
| +deployment roots. |
| + |
| +**Who may post (decision, amended per Codex 11, Fable 12).** The request |
| +must carry an `Origin` whose scheme is `https` and whose host and port |
| +resolve, through the edge's own map (`lookupIn(e.current(), originHost)`, |
| +the same lookup the request itself took), to an entry whose `route.Host` |
| +equals `up.route.Host`; aliases share the primary's route verbatim, so |
| +this covers them with no registry call on the hot path. A bad |
| +`Origin` is rejected even if `Referer` passes; `Referer` is consulted |
| +only when `Origin` is absent. `Sec-Fetch-Site: cross-site` is rejected |
| +outright. This is identification, not authentication: a direct client |
| +can forge any of it, as it can against every public beacon on the |
| +internet, and a per-app public key would add revocation but not truth. |
| +What bounds the damage is that a forged post can only put rows on the |
| +owner's own tab, under the owner's own caps, and the owner can turn the |
| +door off. Body is capped at 16 KB; `stack` at 8 KB. |
| + |
| +**Rate (decision, amended per Codex 9).** Token buckets per *app* of |
| +600/min for errors and 600/min for events, separately, and per visitor |
| +hash of 60/min, all in edge memory, all dropping silently and counting |
| +`dropped.rate`. Visitor-bucket state is bounded to 10,000 entries per app |
| +with LRU eviction. Errors are then coalesced by fingerprint at the box as |
| +in §3.2, so a page throwing in a loop on ten thousand screens is one row |
| +with a large count and, if it exceeds the bucket, an honest `dropped`. |
| + |
| +**The snippet (decision).** `carlos.js` is vendored into the edge binary |
| +the way Lucide icons are vendored into the console: in-code, never a |
| +CDN. It installs `window.onerror` and `unhandledrejection` handlers that |
| +post to `/_carlos/e` after §3.1 sanitisation, and exposes |
| +`carlos.track(name, fields)` which posts to `/_carlos/a` via |
| +`navigator.sendBeacon`. **It does not send pageviews in edge mode.** The |
| +edge already counted the page when it served the HTML; a JS pageview |
| +would double every number. What edge mode misses, stated on the tab: SPA |
| +route changes, HTML served by a CDN in front of CARLOS, and back/forward |
| +cache restores. The reserved `"count": "browser"` mode (§5) is the |
| +answer for those apps, not a same-second dedup heuristic. Until then an |
| +SPA may call `carlos.track("route", {path})` and see it as a named |
| +event. Members include the snippet with one line: |
| + |
| +```html |
| +<script src="/_carlos/carlos.js" defer></script> |
| +``` |
| + |
| +### 3.5 The server door |
| + |
| +**Verbs (decision).** `errors.Mount` and `analytics.Mount` follow |
| +`schedule.Mount` exactly: same bearer check, same 0600 socket, same |
| +`host` binding. |
| + |
| +``` |
| +POST /errors {kind, name, stack, path, status, fields, fp} |
| +POST /events {name, fields} |
| +``` |
| + |
| +The app reads `CARLOS_CONTROL_SOCKET` and `CARLOS_ADMIN_TOKEN` from its |
| +environment as it already does for schedules. A Rails app wires it in an |
| +`ActionDispatch` error subscriber; a Go app in an `http.Handler` |
| +middleware. The confused-deputy discussion in `purge.go` applies |
| +unchanged: a same-uid sibling that can read the socket can post a fake |
| +error, and the worst it can do is put a row on the owner's own tab. A |
| +call on the control socket does not stamp `lastActive`: `ListenControl` |
| +serves its own mux (wired at `backend_exec.go:624`) and never enters |
| +`Service.serve`. The same 16 KB body cap, 8 KB stack cap and 600/min |
| +per-app buckets as the browser door apply, counted into the same |
| +`dropped` keys; the socket answers `202` on a drop and `400` only on a |
| +body it cannot parse, since the caller here is the owner's own code and |
| +can act on it. |
| + |
| +Unit-backed and DynamicUser tenants have no control socket today and so |
| +have no server door; they have `log` and `edge` sources and the browser |
| +door like everyone else. This is the same gap `purge.go` records for |
| +schedules, and it closes when that one does. |
| + |
| +### 3.6 The tabs, the API, the CLI |
| + |
| +**Errors tab** (`GET /a/{sqid}/apps/{app}/errors`, `handleAppErrors` |
| +through `appPage(w, r, "errors")`, template `app_errors.html`, |
| +`ActiveTab: "errors"`). Default window 24 h, selectable 1 h / 24 h / 7 d. |
| +One row per fingerprint: source badge, kind, name, count in window, first |
| +seen, last seen, hosts affected, releases seen. Rows sort by last seen. A |
| +row expands in place to the stored sample: stack, path, status, fields, |
| +and a link to Logs at that timestamp. Filters: source, host, release. |
| +Grouping happens on read over the window. `ReadWindow` fetches the whole |
| +window and discards beyond `logsCollectCap` (50,000), which bounds |
| +memory, not bucket reads and not totals (Codex 14): when the cap is hit |
| +the tab says "showing the most recent 50,000 of a larger window" and the |
| +counts are labelled as counts within what was read. A day rollup is not |
| +built until a real app shows the read is too slow (§5). |
| + |
| +**Analytics tab** (`/apps/{app}/analytics`). Range today / 7 d / 30 d. |
| +Headline: pageviews, visitors (daily, summed for ranges), bytes served, |
| +bots excluded, dropped. One chart, per hour for today and per day |
| +otherwise, drawn the way `templates/metrics.html:29` draws sparklines: |
| +points computed in Go, an inline `<svg>` in the template, no library. |
| +Three tables: top pages, top referers, named events. |
| + |
| +**API (decision, Fable 8).** Two `cliRoute` paths beside the logs read: |
| + |
| +``` |
| +GET /api/cli/apps/{account}/{app}/errors svcrole.Allow(Instance, Operate) |
| +GET /api/cli/apps/{account}/{app}/analytics svcrole.Allow(Instance, Operate) |
| +``` |
| + |
| +Bearer-authenticated through `apiAccountApp` like `/logs`; an app token |
| +may read its own app's errors and analytics. The session → CSRF → owner |
| +(byte-identical 404) → fresh-auth ordering applies to the two *pages*, |
| +which go through `appPage`; the API paths have no session and no CSRF by |
| +construction. The tabs render from the same functions the API answers |
| +with. |
| + |
| +**CLI.** `carlos errors -app X [-since 24h] [-source browser] [-host h] |
| +[-release v] [-fp <id>]` lists groups, `-fp` expands one; |
| +`carlos analytics -app X [-range 7d]` prints the headline, chart as a |
| +text sparkline, and the three tables. Both get headings in |
| +`docs/site/cli.md` in the same branch, or `internal/docsite` fails the |
| +gate. |
| + |
| +**Settings.** `analytics: on|off` on the Settings tab and via |
| +`carlos apps`, writing the object in §3.3. Errors have no switch: an app |
| +that is failing is an operational fact the owner needs. A member may |
| +delete an error group from the tab or the API (`DELETE |
| +…/errors/{fp}`, owner on the page, `Operate` token on the API). This |
| +writes a tombstone `apps/<sqid>/<app>/errors/tombstones/<fp>.json` |
| +carrying the deletion time; the reader hides records of that `fp` with |
| +`last` before that time, from both `errors/` and the log-derived rows, |
| +until retention removes the chunks. A later occurrence reappears: this |
| +is delete, not mute (§5). Logs themselves are untouched; deleting log |
| +lines is a Logs feature and not part of this design. |
| + |
| +**Retention (decision).** Error chunks follow log retention, both age and |
| +byte cap, in `logshipPrune`. Analytics day files are kept 400 days, |
| +enough for year-over-year, and are small enough (tens of KB per app-day) |
| +that no byte cap is needed; the same prune loop deletes past the age. |
| + |
| +**Coverage line.** Each tab carries one line saying which sources are |
| +live for this app: "log, edge" for an app with nothing wired, "+browser" |
| +once a beacon has arrived, "+server" once the socket verb has. A member |
| +should never wonder whether silence means no errors or no wiring. |
| + |
| +## 4. The consequence to be explicit about: nothing here wakes an instance |
| + |
| +Every path in §3 is checked against the scheduled-ticks rule: |
| + |
| +- `/_carlos/*` is answered between route lookup and the meter wrap, before |
| + the cache and the activator. A beacon to a hibernating app returns `202` |
| + from the edge; `beginRequest` is never called and the idle clock is not |
| + stamped. The test reads `a.entries[host].lastActive` directly, as |
| + `activator_test.go:1378` does. |
| +- Analytics counting is a tap on responses the edge is already sending. |
| +- The `log` source is a read over chunks the agent already ships. |
| +- The server door is a control-socket verb, which does not stamp. |
| +- Flushes ride the existing `usageLoop` and `logshipLoop` tickers in the |
| + agent. |
| + |
| +The one new cost is bucket writes: one analytics day-file CAS per box per |
| +five minutes per app with traffic, and error chunks at the cadence of log |
| +chunks. Both are bounded by the caps in §3.2–3.3 regardless of traffic. |
| +Memory on the edge is bounded per app: 500-counter summaries, one 2 KB |
| +sketch, a 10,000-entry visitor bucket table, and the coalescing map |
| +capped at 1,000 fingerprints. |
| + |
| +## 5. What is not built |
| + |
| +- **Resolve and mute.** An owner will want to mark a group resolved or |
| + mute a known one. It is the first triage feature to add and comes |
| + before alerts; the tombstone in §3.6 is its seed. |
| +- **Alerts.** "Email me when a new fingerprint appears" belongs with the |
| + existing `alertwatch` machinery, not here. |
| +- **Browser-counted mode** (`"count": "browser"`): the snippet sends |
| + pageviews with SPA route support and prerender activation, and the edge |
| + stops counting HTML. For apps behind a CDN or built as SPAs. |
| +- **Event properties.** `fields` on analytics events are accepted and |
| + dropped; property breakdowns change the day-file shape additively. |
| +- **Error day rollups.** Grouping reads the window every time. Fine to |
| + 50k records; a heavy app will tell us when it is not. |
| +- **Forwarding.** A `forward: sentry|posthog|…` setting that relays |
| + events to a rented service. The shape in §3.1 makes this a translator. |
| +- **Source maps, breadcrumbs, session replay, funnels, GeoIP, per-country |
| + tables.** Rent those. |
| +- **Fleet-wide views.** Errors across a member's apps, or across the |
| + fleet for ops, are a different page with different gates. |
| + |
| +## 6. Testing |
| + |
| +- **Sanitisation:** table test over messages, stacks, fields and paths |
| + containing emails, bearer tokens, `password=`, 32-char hex and base64 |
| + runs; each is replaced and nothing else is touched. |
| +- **Fingerprint:** table test of `norm`: digits, hex, UUIDs and quoted |
| + strings are placeholders; `v2`, `E404`, `HTTP 503` survive; frames lose |
| + line, column and bundle hash; a caller-supplied `fp` is taken verbatim |
| + and capped; `fpv` is stamped. |
| +- **Coalescing:** 10,000 identical errors in one interval produce one |
| + record with `count: 10000` and `first < last`; the same `fp` on two |
| + hosts produces two records; 1,001 distinct fingerprints for one app |
| + produce 1,000 records and one `_overflow`; a second app on the same box |
| + is unaffected. |
| +- **Log-derived rows:** a `level=error` JSON line appears on the Errors |
| + read and in Logs; `level=info`, an unparseable line, and a plain |
| + `panic:` line do not appear as `log` errors. |
| +- **Edge errors:** an upstream 500 on a POST (the plain path), a 502 from |
| + a failed wake, and a 500 on a cacheable GET each produce exactly one |
| + edge error; a 404 produces none. |
| +- **Chunk family:** `logs/` chunks written and read after the refactor |
| + are byte-identical to before; every existing applog test passes |
| + unchanged; `errors/` round-trips the new record. |
| +- **Store door:** a fleet box may write `apps/<sqid>/<app>/errors/x` and |
| + `…/analytics/x` and may not write `apps/<sqid>/errors/x`, |
| + `apps/<sqid>/<app>/errors`, or `apps/<sqid>/<app>/other/x`. |
| +- **Observing writer:** status and `Content-Type` latch at `WriteHeader` |
| + and at implicit 200; visible below gzip; installed with a nil meter; |
| + byte counts unchanged for every existing usage test. |
| +- **Pageview definition:** `httptest` through `Edge.ServeHTTP` with a fake |
| + upstream: HTML 200 GET counts; JSON 200, HTML 404, HTML 302, 304, HEAD, |
| + `Sec-Purpose: prefetch`, `/_carlos/carlos.js` do not; a cache hit |
| + counts; `Sec-Fetch-Dest: empty` does not; a known bot UA counts only |
| + into `bots`; a `status` route counts nothing; `count: off` counts |
| + nothing. |
| +- **Client IP:** verified hop takes the last XFF element; unverified hop |
| + and direct take `RemoteAddr`; a client-set XFF on a direct request is |
| + ignored. |
| +- **Visitor hash:** same IP+UA on the same day is one visitor; a |
| + different day is different; the salt survives a restart and never |
| + appears in the day file; the salt file is removed on rotation. |
| +- **HLL:** across ten random datasets of 1k, 10k and 100k, the estimate is |
| + within 3 standard errors of truth; merge is commutative and idempotent; |
| + the encoding round-trips with its version byte. |
| +- **Misra–Gries:** with fewer than 500 keys the summary is exact; two |
| + summaries merge and re-fold to 500 plus `_other` with the total |
| + preserved. |
| +- **Day-file CAS:** event-time bucketing places a flush at 10:00:02 for |
| + events at 09:59 into `09`; a failed flush re-queues and the next |
| + succeeds with the sum; two failures drop the older window and increment |
| + `dropped.flush`; two boxes writing the same app-day converge to the sum |
| + of pageviews and the union of sketches. |
| +- **Browser door:** correct `Origin` counts; missing `Origin` with matching |
| + `Referer` counts; foreign `Origin` with matching `Referer` does not; |
| + `Sec-Fetch-Site: cross-site` does not; 17 KB body does not; the 601st |
| + post in a minute does not; each drop increments the right `dropped` |
| + key; every case answers `202`. A hibernating route receives a beacon |
| + and `lastActive` is unchanged. On a path-routed hostname the beacon |
| + lands on the app owning its `path` prefix. `_carlos` is refused as a |
| + prefix-map key. |
| +- **`/_carlos/` reservation:** a route whose upstream serves `/_carlos/x` |
| + never sees the request; the loader revalidates and the hashed asset is |
| + immutable. |
| +- **Server door:** a bad bearer is refused; a good one lands a record; |
| + `lastActive` is unchanged after the call; a 17 KB body answers `202` |
| + and increments `dropped.e.size`; `count: off` drops `/events` and |
| + still accepts `/errors`. |
| +- **Seq idempotency:** re-merging a batch with an already-recorded `seq` |
| + leaves the day file unchanged. |
| +- **Tombstone:** after `DELETE`, the group is absent from the read and |
| + the API; a new occurrence after the deletion time reappears; Logs |
| + still shows the underlying lines. |
| +- **Wake failure once:** a failed wake produces one edge error, not two. |
| +- **Console gates:** both pages give a non-member the byte-identical 404 |
| + the Logs tab gives; both API paths refuse a token without `Instance` or |
| + `Operate`. |
| +- **Switch convergence:** writing `analytics.json` with `count: off` and |
| + bumping the serial stops counting on the box within the next pass; the |
| + browser door drops `/_carlos/a` and still accepts `/_carlos/e`; |
| + `door: off` drops both. |
| +- **Docsite:** the gate passes with both new CLI headings present and |
| + fails with either removed. |
| + |
| +## 7. Rollout |
| + |
| +Each step is a separate branch that ships value on its own and needs no |
| +member action. Each is canaried on the flagship for a day against edge |
| +latency, edge memory, bucket write count and the hibernation invariant |
| +before promotion, in the ladder the fleet already runs. |
| + |
| +1. **Errors from `log` and `edge` sources, tab, API, CLI.** Every app in |
| + the fleet gets a populated Errors tab. Requires the chunk-family |
| + refactor, the store-door allow-list, the observing writer (for edge |
| + errors), and the coalescing sink. |
| +2. **Analytics counting at the edge, tab, API, CLI, the switch.** Every |
| + app gets pageviews and visitors with no JavaScript. Shares the |
| + observing writer with step 1, so it lands second. |
| +3. **Browser door and `carlos.js`.** Members opt in with one script tag; |
| + JS errors and named events appear. |
| +4. **Server door on the control socket.** Exec-backed apps post |
| + structured errors and events. |
| + |
| +Wire shapes are additive at every step: `logs/` is untouched, `errors/` |
| +and `analytics/` are new prefixes, `analytics.json` absent means on, and |
| +a box on an older build writes none of them. A mixed-version fleet shows |
| +the coverage line per host so a missing source is visible rather than |
| +puzzling. |
| + |
| +Binaries roll as usual: promote → adopt → restart for the console; edge |
| +and agent pick up on adoption. |
| + |
| +## 8. Open questions for Paul |
| + |
| +1. **Default for analytics: on or off?** On, above, because the feature is |
| + invisible if nobody turns it on. The visitor design is built to be |
| + consent-free under the reading Plausible defends (no cookie, no stored |
| + identifier, no cross-day join), but that is a reading, not a ruling, |
| + and Codex is right that it is the deployment's assessment to make, not |
| + the spec's. If the deployment serves EU members who would rather opt |
| + in, off-by-default costs one setting. |
| +2. **Does `/_carlos/` collide with anything in `platform-infrastructure`?** |
| + Nothing in this repo serves it; the deployment roots are not here. |
| +3. **Should the `log` source include `warn`?** No, above; a warn-level |
| + Errors tab is noise on day one. |
| +4. **Salt persistence location.** Beside the registry DB is the |
| + suggestion; if the box state directory has a better home for a |
| + day-scoped secret, use it. |
| + |
| +## 9. Review round 1 |
| + |
| +Both reviewers read the first draft against `main` and returned |
| +needs-rework. What each changed: |
| + |
| +**Fable (second context).** Moved the beacon intercept from "before |
| +routing" to between `lookupUpstreamPath` and the meter wrap, with the |
| +backhaul argument (§3.4). Replaced the recorder tap with the observing |
| +`countingWriter`, citing the three response paths that bypass the |
| +recorder (§3.3). Found that fleet boxes' writes would be refused by |
| +`aclWriteAppLogs` (§3.2). Corrected the §1 citations, which pointed at |
| +ACME and attribution comments. Defined `clientIP` from the verified-hop |
| +chain (§3.3). Listed the real refactor scope and the key-shape consumer |
| +list, and corrected `retentionPass` to `logshipPrune` (§3.2, §3.6). |
| +Argued the log mirror away in favour of a read-side filter (§3.2). Named |
| +the `cliRoute` API pattern and the gate-ordering scope (§3.6). Found that |
| +no box reads `features.json`, so the switch became a bucket object |
| +(§3.3). Sanitised page paths (§3.3). Corrected the chart precedent, the |
| +Logs-tab grep, and the non-existent test hook (§3.6, §4). Noted the |
| +500-key merge overflow (§3.3). |
| + |
| +**Codex.** Made the visitor salt restart-stable and the multi-day |
| +visitor figure honest (§3.3). Reframed the privacy claim as the |
| +deployment's assessment (§8). Tightened the pageview definition to 200 |
| +only, excluding redirects, 304s, HEAD and prefetch (§3.3). Reserved a |
| +browser-counted mode instead of a same-second dedup (§3.4, §5). Named the |
| +path-routed attribution problem independently of Fable (§3.4). Made |
| +fingerprints token-boundary, versioned, and frame-normalised (§3.2). Made |
| +the coalesced record carry `first`, `last`, host and release (§3.2). Made |
| +caps per app and exposed `dropped` everywhere something is dropped (§3.2, |
| +§3.3, §3.4). Replaced first-admitted maps with Misra–Gries and stated the |
| +HLL error correctly (§3.3). Tightened Origin matching and kept the |
| +uniform 202 only alongside owner-visible drop counts (§3.4). Added |
| +sanitisation before buffering and owner-triggered deletion (§3.1, §3.6). |
| +Made day-file buckets event-time and flushes re-queue on failure (§3.3). |
| +Added the collect-cap disclosure and the coverage line (§3.6). Split the |
| +snippet into a revalidating loader and a hashed immutable asset (§3.4). |
| +Put resolve/mute ahead of alerts in §5. Noted that a plain `panic:` line |
| +is not a parsed error, which §3.2 now answers through the `edge` source. |
| + |
| +Not taken: a per-app public key for the browser door (identification |
| +without truth, and the caps already bound the damage; §3.4 says why); |
| +storing the raw pageview path (the token-collapsing rule keeps real page |
| +names and drops secrets); a mirror of `log` errors into `errors/` (read |
| +derivation is cheaper and loses nothing). |
| + |
| +## 10. Review round 2 |
| + |
| +Both reviewers re-read the rewrite. Fable: approve-with-amendments, all |
| +three blockers confirmed resolved against the code; amendments taken: |
| +the Origin check is a `lookupIn` on the edge's map, not an alias list |
| +(`upstream.alias` is a bool); `Sec-Fetch-Dest: empty` is a fetch, not a |
| +page; status routes are excluded from the tap; joined `X-Forwarded-For` |
| +lines; salt file 0600. Codex: needs-rework on under-specification; |
| +taken: 1–3 digit tokens survive normalisation so `HTTP 503` does; the |
| +snippet stamps `path` on every beacon so path-routed events attribute; |
| +the coalescing cap counts records and the window rule is stated; a |
| +`seq` per box makes CAS retries idempotent; a `door` switch closes the |
| +browser door independently of counting and `count: off` also drops the |
| +socket's `/events`; the server door gets the same size and rate limits; |
| +Misra–Gries tables are labelled approximate above 500 keys; the |
| +`dropped` schema lists every key it uses, split by errors and events; |
| +wake failures are suppressed at the writer when the activator already |
| +emitted; tombstones have a scope, an authorisation, and a stated |
| +non-effect on Logs. Nothing from round 2 was declined. |