Errors and Analytics: two tabs every app gets for free

Date: 2026-09-07 Status: approved (Paul, 2026-09-07: "go ahead with that and plan, build review cycle overnight"). Rollout step 1 (§7) is merged (forge main 1a99060; plan docs/superpowers/plans/2026-09-07-errors-step1-log-and-edge.md). Step 2 is implemented on branch analytics-step2 (plan docs/superpowers/plans/2026-09-07-analytics-step2-edge-counting.md). Every choice below is marked (decision) and can be overturned cheaply; the four in §8 are Paul's (ruling)s of 2026-09-07 and are marked where they land. Drafted by Claude Fable, then reviewed by a second Fable context and by Codex; both returned needs-rework on the first draft and §9 records what they changed. Line numbers cite origin/main at 09a22d6.

1. The idea, and why it fits this platform

A member who ships an app to CARLOS today gets a URL, a certificate, logs, schedules, and a bill. What they do not get is the two questions every operator asks within a day of launch: is it broken and is anyone using it. The answers live in AppSignal, Datadog, Sentry, PostHog, Plausible: a second account, a second bill, an SDK to wire in, and for the analytics half a cookie banner to justify.

The platform is unusually well placed to answer both questions itself, because it already stands in the two places the data passes through:

• The edge sees every request to every app, including the ones its cache answers without waking the instance (writeCached, internal/edge/edge.go:1311). Method, path, status, referer and user agent are in hand at the point where countingWriter is installed to bill the bytes (edge.go:1241). Server-side page-view counting there costs the app nothing and needs no JavaScript and no cookie. It is the Plausible model, done where Plausible cannot stand.
• The agent already captures every line the app writes and ships it as hour chunks to the bucket (internal/applog, internal/host/logship.go). A line the app tagged level=error is already an error event; it is just filed under Logs, undifferentiated from a million info lines. And when the app dies, the edge answers a 502 or reports a wake failure: the crash is already observed.

So the first version of both tabs is mostly reading what is already flowing past. The hooks add the cases the platform cannot see on its own: a public endpoint on the app's own hostname for browser errors and named events, and a loopback verb for server-side errors and events. The member who wants AppSignal can still rent it; the event shape is deliberately close to what those services accept, so a "forward to" setting later is a translator, not a redesign (§5).

The proposal is two tabs on the app dashboard, Errors and Analytics, between Logs and Activity in appnav.html, each with a console API and a carlos verb behind it (AGENTS.md: every command must complete the job for a user with no infra access).

2. What is already right, and must stay right

Logs are the pattern for events. applog.Record (internal/applog/record.go:72) is buffered on the box under a token budget (sink.go:95), flushed as gzip'd JSONL hour chunks under apps/<sqid>/<app>/logs/<box>/<hour>/, read back by applog.ReadWindow merging chunks across boxes, pruned by logshipPrune (logship.go:161) on age and byte cap, and served through GET /api/cli/apps/{account}/{app}/logs (authcli.go:595) with a cursor. Errors that the platform must store are events too; they reuse this machinery rather than growing a second one (§3.2).

Usage is the pattern for counters. usage.EdgeDay (internal/usage/edge.go) is a CAS-merged day file, hour-bucketed, one per box, flushed every five minutes by usageLoop (internal/host/usage_loop.go:27) from UsageMeter.Drain(). Page views and visitors are counters; they reuse this shape, with two durability fixes the reviewers asked for (§3.3).

The control socket is the pattern for a loopback API. ListenControl (internal/edge/purge.go:108) mounts verbs on a 0600 unix socket the app finds in CARLOS_CONTROL_SOCKET; schedule.Mount (internal/schedule/control.go:41) shows the bearer check against CARLOS_ADMIN_TOKEN. Server-side ingestion mounts beside it (§3.5). The socket is not the activator: a call on it never reaches beginRequest.

Nothing the platform does for itself may keep an instance awake. The scheduled-ticks spec (2026-08-30) exists because one line of accounting cost a member fifteen minutes of billed residency per millisecond of work. Every ingest path here is handled in the edge or the agent and none enters activator.Service.ServeHTTP; a beacon from a browser to a hibernating app is answered by the edge and the app stays asleep (§4).

The edge's wrapper order is load-bearing (AGENTS.md): recorder above the compressor, usage counter below it, wrap site after the upgrade branch. This design changes what the usage counter observes, never where it sits (§3.3).

Fleet boxes write through the store door, whose write grant is segment-exact apps/<sqid>/<app>/logs/** (aclWriteAppLogs, internal/edge/storedoor.go:503), and usage is rebased under fleets/<name>/ (usage_loop.go:47). Two new families need the door to know about them, or a fleet box's writes are refused (§3.2).

3. The change

3.1 One event shape, four sources

Both tabs are fed by events in one JSON shape. (decision) Fields:

ts        RFC3339 nanoseconds, stamped by the platform, never taken from input
source    "log" | "edge" | "server" | "browser"
kind      errors: exception class or "http"; analytics: "pageview" | "event"
name      errors: the message; analytics: the event name
host      route host that received it (how owners name instances)
box       the box that recorded it
release   X-Carlos-Version of the serving build, when known
path      request path, query and fragment removed, token segments collapsed (§3.3)
status    HTTP status, when the event came from a request
referer   host-only, never the full URL
ua        browser family only, never the raw string
stack     errors only, capped at 8 KB
fields    map[string]string of caller-supplied scalars, 32 keys, 256 B each
fp        errors only: fingerprint (§3.2)
fpv       errors only: fingerprint algorithm version, 1

Sources and where they enter:

| source | errors | analytics | |---------|----------------------------------------------------------|---------------------------------------------| | log | applog record at or above errors.log_level, read-derived | none | | edge | upstream 5xx, gateway 502/504, wake failure | every HTML 200 GET becomes a pageview | | server | POST /errors on the control socket | POST /events on the control socket | | browser | POST /__carlos/e on the app's own host | POST /__carlos/a on the app's own host |

log and edge need no integration at all. They are why the Errors tab is not empty on day one for any app in the fleet, and why Analytics starts counting the moment an account turns it on (§3.3).

Sanitisation before buffering (decision, Codex). Errors carry messages, stacks, paths and caller fields, and those carry secrets in practice. Before any record is buffered, on every source: query strings and fragments are removed from path; in name, stack and fields, email addresses become <email>, Bearer/Basic credentials and token=/key=/secret=/password= values become <redacted>, and hex or base64 runs of 32+ characters become <hex>. The browser snippet applies the same rules before sending, so nothing leaves the page that the platform would strip. This is a floor, not a guarantee; the Errors tab says so in one line.

3.2 Errors: sources, fingerprints, coalescing, storage

The log source is derived on read, not stored twice (decision, Fable review 7). The first draft mirrored level=error lines into a second family. ReadWindow with Filter.Level="error" (filter.go:22) over logs/ yields the same rows with the same retention, and a mirror would have had to choose a side of the token budget at sink.go:95. So the Errors tab reads two families and merges: logs/ filtered to the levels the errors.log_level setting names (§3.3; error means error+fatal, warn adds warn; a setting, per Paul's ruling of 2026-09-07), and errors/ for the other three sources. Whatever the setting, every row carries its level and the tab, API and CLI filter on it, so a member on warn can look at errors only and a member on error can widen one read to warn without changing the setting. A JSON line whose level is unrecognised is not an error; ParseAppLine never guesses, and neither does this. A plain-text panic: on stderr carries no level and is not a log error either: the crash reaches the tab through the edge source when the next request meets a dead instance, which is the honest signal.

Edge errors (decision). Where the observing writer (§3.3) sees a final status of 500–599 from the upstream, or the edge itself answers 502/504 because the instance did not come up, it emits source: edge, kind: http, name: "<status> <method> <path>". Wake failures already reach the member through emitEvent in internal/activator/logsink.go; the same call site sends one record to the errors sink. To keep that exactly once, the activator branch marks the request context when it answered the failure itself, and the observing writer skips a 502/504 that carries the mark; the writer's own emission covers only statuses the upstream returned or the proxy produced without the activator. Client 4xx are not errors.

Fingerprint (decision, amended per both reviews). fp groups occurrences into one row. A caller may supply one, capped at 64 bytes. Otherwise the platform computes sha256(len-delimited(source, kind, norm(name), norm(top frame)))[:16] with fpv: 1. norm splits on whitespace and punctuation; a token that is a digit run of 4+ characters, a hex run of 8+, a UUID, or a quoted string becomes a placeholder; tokens of 1–3 digits and mixed tokens like v2 or E404 are kept, so HTTP 503, v2 and retry 3 still separate groups while user 4123 and user 9911 do not. Small ids under 1,000 will split groups; that is the stated trade-off, and fp overrides it. A frame is normalised to file and function with line, column and bundle hash removed, so a redeploy does not split every browser group. The rule is versioned so it can change without merging old groups into new ones. It is wrong sometimes; a caller who knows better sets fp.

Coalescing at the box (decision, amended per Codex 8). An error storm is when the tab must work and when a naive stream fails. The box sink keeps, per (app, fp, host, release) per flush interval, one record carrying count, first, last, and the first sample's stack, path, status and fields. Ten thousand identical panics in a minute become one record with count: 10000. Coalesced records are capped per app per hour at 1,000, counting (fp, host, release) combinations, not distinct fingerprints, so the memory bound is real (per app, not per box: one tenant's storm must not suppress a neighbour's, Codex 9); beyond that, the overflow folds into a single _overflow row whose count is occurrences, labelled as such on the tab. A record belongs to a read window when [first, last] overlaps it and is counted whole; the flush interval is five minutes, so a window edge is wrong by at most one interval, and the tab's window labels say "to the nearest flush". (Correction, 2026-09-24: the ship loop as built flushes every 10 s, errshipTick in internal/host/errship.go, so records coalesce over 10 s, not five minutes. Found reviewing the request-reference spec.)

Storage (decision, amended per Fable 6). errors/ is a sibling family of logs/: apps/<sqid>/<app>/errors/<box>/<hour>/<seq>-<size>.jsonl.gz. The chunk key and hour layout are shared, so ChunkKey, Prefix, BoxPrefix, HourPrefix, ChunkKeySize, hourNameOfDir, hourNameOf, logshipPrune and logshipLoop take a family argument. The record is not shared: applog.Record has none of fp, count, first, last, stack, source, kind, status, path, release, and EncodeChunk, DecodeChunk, ReadWindow, Filter.Match and Cursor all take Record. The codec becomes generic over the record type, or Record is widened with omitempty fields; the implementer picks whichever leaves logs/ chunks byte-identical, and the existing applog tests are the proof. The consumer list for the key shape, per AGENTS.md, is: ChunkKeySize (chunk.go:65), hourNameOfDir (read.go:91), hourNameOf (logship.go:216), aclWriteAppLogs (storedoor.go:503), logshipPrune (logship.go:161). Wire shape is additive: nothing under logs/ changes.

Store door (decision, Fable 3). aclWriteAppLogs becomes a family allow-list: logs, errors, analytics, each with the same "at least one component below" rule, with door tests for every family and for errors as an app name. Without this a fleet box's writes are refused and the tabs are empty for exactly the members on fleets.

3.3 Analytics: counting at the edge

Where the edge sees it (decision, Fable 2). The first draft assumed a tap on the recorder. The recorder is built only on the hibernating and cacheable branches (edge.go:1322, 1333); a cache hit goes through writeCached (:1311) and the plain path (:1339), every POST included, has no recorder at all. The one wrapper installed at a single site for every response shape is countingWriter (usagemeter.go:64), which today has no WriteHeader and is installed only when a meter is armed (edge.go:1241). It becomes an observing writer: installed unconditionally with a nil-safe meter, latching final status and Content-Type at the first WriteHeader of 200 or above (1xx are forwarded and ignored), an implicit 200 at first Write, and a 200 with no Content-Type when the handler returns having written nothing, which is never a pageview. Flush and Hijack forward as they do today. The meter wrap today skips Kind == "status" routes (edge.go:1241); the observing writer is installed there too, but the tap ignores status routes, so the status host never enters any app's analytics. It stays below the compressor; Content-Type is visible there because gzipWriter.syncHeader copies its private map down before the underlying WriteHeader (compress.go:236). HEAD skips gzip (compress.go:197), so the tap excludes by method itself. The upgrade branch (:1257) is untouched; a 101 is never a pageview. The order rule in AGENTS.md is preserved: nothing moves, one wrapper learns to look.

What is counted (decision, amended per Codex 3). A request is a pageview when the method is GET, the final status is 200, and the response media type parses as text/html. Redirects are not views (the destination will be); 304s are not views (no media type, and a revalidated page is an undercount the tab states); 204/205 are not; requests with Sec-Purpose: prefetch or Purpose: prefetch are not; Sec-Fetch-Dest, when present, must be document (an empty dest is fetch(), which is how htmx and SPAs load HTML fragments, and a fragment is not a page; an absent header is an old UA or a bot, and the bot list handles the latter). Requests under /__carlos/ are never counted. A request whose user agent matches the vendored bot list is counted into bots and nowhere else; the list is incomplete by nature and the tab shows bots so the member can see what was excluded.

Client IP (decision, Fable 5). There is no client-IP value in the edge today. clientIP(r) is defined as: the last element of X-Forwarded-For (all header lines joined, as setForwarded does at homemap.go:915) when hopVerified(ctx) (homemap.go:767 accepts a verified chain verbatim; :800 strips an unverified one), otherwise the host of RemoteAddr. A header the client can set never reaches it.

Visitor (decision, amended per Codex 1 and Fable 10). A visitor is sha256(day-salt ‖ route host ‖ clientIP ‖ user agent)[:8]. The salt is per box per UTC day, generated at first use, written 0600 to the box's state directory beside the registry DB (/data/, which the agent already owns) so a restart mid-day does not re-salt and double-count, and deleted on rotation; it is never written to the bucket. That location is Fable's recommendation, confirmed against who owns the directory; Codex's ask was for a restart-stable, app-scoped salt shared by every box serving the app, which the persisted file gives for the first half and which the home-box rule makes unnecessary for the second. No cookie is set, no IP is stored, and the platform holds nothing that joins a hash across days. Because a route is served by its home box (a box without the row backhauls, edge.go:1216), a visitor is normally seen by one box; during an instance move they are counted twice for a day. Multi-day ranges on the tab are labelled "daily visitors, summed", because that is what they are; Plausible's are too.

Cardinality (decision, amended per Codex 10). Unique visitors per day per box are a HyperLogLog sketch with 2¹¹ registers (2 KB, standard error about 2.3%), one per day, encoded as base64 with a one-byte version prefix, merged across boxes by register max. Pageviews per hour are integers. Top pages, top referers and named events are Misra–Gries summaries with 500 counters each: bounded, mergeable, and correct for any app with fewer than 500 distinct keys a day, which is most of them. The reader re-folds after merging so the tab never shows more than 500 rows plus _other. Below 500 keys the counts are exact; above, a counter is a lower bound within the summary's error and the table is labelled "approximate: more than 500 distinct keys today". Keys are capped at 256 bytes.

Page paths (decision, Fable 10). pages keys are the sanitised path of §3.1: query and fragment removed, and any segment that is a UUID, a hex run of 16+ characters, a base64-looking run of 20+, or an email replaced by :token. /reset/8f3a… never reaches the bucket; /blog/2024-review does.

Storage (decision, amended per Codex 13). apps/<sqid>/<app>/analytics/<box>/days/<date>.json, written by a sibling of UpsertEdgeDay with the same read-etag-merge-conditional-put loop, and two differences from usage. First, buckets are keyed by event time, not flush time, so a flush delayed across an hour boundary lands in the right hour. Second, the drain returns deltas, and a failed flush re-queues its deltas into the next drain rather than losing them (usageLoop documents that it loses the window; modelling grade is fine for billing estimates, not for a number a member reads as truth). One window survives a failure; a second consecutive failure drops the older one and increments dropped.flush. A conditional put whose outcome is unknown (timeout after send) must not double-count on retry: every delta batch carries a box-monotonic seq, the day file records seq per box, and a merge whose seq is not greater than the recorded one is a no-op. A process restart loses the in-memory window, stated on the tab as "counts since the last flush may be missing" only when a restart happened that day. Shape:

{
  "v": 1,
  "date": "2026-09-07",
  "hours":    {"09": {"pv": 412, "bots": 30, "bytes": 1843201}},
  "visitors": "AQ…",
  "pages":    {"/": 300, "/pricing": 80, "_other": 32},
  "referers": {"news.ycombinator.com": 200, "direct": 180},
  "events":   {"signup": 12, "checkout": 3},
  "seq":      {"box-a": 1187},
  "dropped":  {"e": {"rate": 0, "size": 0, "origin": 0, "malformed": 0},
               "a": {"rate": 0, "size": 0, "origin": 0, "malformed": 0},
               "flush": 0},
  "countries": {}
}

countries is present and empty in v1 so a GeoIP pass is additive. dropped is what the member sees when a cap or a failure bit: the tab shows it, so a number that is not the whole truth says so.

Named events. carlos.track("signup") from the browser, or POST /events {"name":"signup"} from the server, increments events[name]. Names are lowercased, 64 bytes of [a-z0-9_.-], and bounded by the same 500-counter summary. Event properties are accepted in fields and not stored in v1 (§5).

Accumulation. The edge holds deltas in memory; the agent's five-minute usageLoop cadence flushes them in the same tick as usage, as a second CAS write (two objects, two writes; the first draft's "one round-trip" was wrong).

The settings object (decision, Fable 9; default is a ruling). features.json is console-only and no box reads it. The switches are one bucket object at two levels with the same shape:

accounts/<sqid>/telemetry.json           account default
apps/<sqid>/<app>/telemetry.json          per-app override
{"analytics": {"count": "edge" | "off", "door": "on" | "off"},
 "errors":    {"log_level": "error" | "warn"}}

Each key resolves app → account → built-in default, so an account can turn analytics on once for every app it has and a single app can still differ. The console writes either object with a serial bump and boxes converge on the next pass like instances/*.json; the account object lives beside account.json, which stays identity-only. Built-in defaults: count: off (ruling, Paul, 2026-09-07), door: on, log_level: error. Analytics is therefore opt-in: the visitor design is built to be defensible without consent, but whether it is consent-free is the deployment's assessment and the member's, not the platform's, so the platform does not switch it on for anyone. count: off stops counting at the observing writer and makes both doors drop analytics events (/__carlos/a and the socket's /events) with 202; existing day files stay until retention. door: off closes the browser door entirely, errors included, for an owner who wants no public ingest surface at all. "count": "browser" is reserved for a later mode (§5) so the shape is additive.

Because the default is off, the tab is not empty when it is off: it shows the switch, what turning it on records, and the one-line privacy reading, so a member decides from the tab rather than finding a setting by accident. The getting-started skill that carlos skills fetches from carlosframework.com (published from the website repo, not here) gains one question in the same step: whether to turn analytics on for the account, with the same one-line reading. That is where an agent setting up a member's first app will ask it.

3.4 The browser door

Where in the edge (decision, Fable 1, Codex 6). The first draft said "before any routing". That is the wrong spot: the app is known only from up.route, which exists after lookupUpstreamPath (edge.go:1197), and a box that has no row for the host must backhaul (:1216) so the home box answers. The intercept sits between the lookup (:1197) and the meter wrap (:1241): after routing, before upgrade (:1257), gzip (:1275), cache (:1311) and the activator (:1322). At that spot beacons are unbilled like status routes and beginRequest is never reached, which is what §4 promises.

Attribution on a shared hostname. lookupUpstreamPath can route different path prefixes on one hostname to different apps (:1008). A beacon carries no app id. The snippet puts location.pathname in every beacon's path, errors and events alike, and the beacon is attributed to the upstream the lookup returns for that raw path on that hostname (the match runs before §3.1 sanitisation, which is applied to what is stored), and to the hostname's root app when path is absent or matches nothing. Because Origin must equal that hostname, the sender is already a page served from it: same origin, same trust boundary the browser already grants those siblings. An unknown hostname takes the existing !ok branch and is never answered by this code. The lookup must never accept __carlos as a prefix key; a test pins that.

Endpoint (decision).

POST /__carlos/e               one error   {kind, name, stack, path, fields, fp}
POST /__carlos/a               one event   {name, fields} | {"name":"pageview","path":…}
GET  /__carlos/carlos.js       loader, Cache-Control: max-age=300, must-revalidate
GET  /__carlos/carlos.<hash>.js  the snippet, immutable

Both POSTs answer 202 with an empty body in every case: accepted, rate-limited, rejected, malformed. The endpoint reveals nothing, and the browser never sees a failure worth retrying. Every drop increments the app's dropped.rate (or dropped.origin, dropped.size), which the tab shows, so silence outward is not silence to the owner (Codex 11). /__carlos/ is reserved on all routes and never proxied upstream from this change on. No app in this repo serves it; §8 asks about the deployment roots.

Who may post (decision, amended per Codex 11, Fable 12). The request must carry an Origin whose scheme is https and whose host and port resolve, through the edge's own map (lookupIn(e.current(), originHost), the same lookup the request itself took), to an entry whose route.Host equals up.route.Host; aliases share the primary's route verbatim, so this covers them with no registry call on the hot path. A bad Origin is rejected even if Referer passes; Referer is consulted only when Origin is absent. Sec-Fetch-Site: cross-site is rejected outright. This is identification, not authentication: a direct client can forge any of it, as it can against every public beacon on the internet, and a per-app public key would add revocation but not truth. What bounds the damage is that a forged post can only put rows on the owner's own tab, under the owner's own caps, and the owner can turn the door off. Body is capped at 16 KB; stack at 8 KB.

Rate (decision, amended per Codex 9). Token buckets per app of 600/min for errors and 600/min for events, separately, and per visitor hash of 60/min, all in edge memory, all dropping silently and counting dropped.rate. Visitor-bucket state is bounded to 10,000 entries per app with LRU eviction. Errors are then coalesced by fingerprint at the box as in §3.2, so a page throwing in a loop on ten thousand screens is one row with a large count and, if it exceeds the bucket, an honest dropped.

The snippet (decision). carlos.js is vendored into the edge binary the way Lucide icons are vendored into the console: in-code, never a CDN. It installs window.onerror and unhandledrejection handlers that post to /__carlos/e after §3.1 sanitisation, and exposes carlos.track(name, fields) which posts to /__carlos/a via navigator.sendBeacon. It does not send pageviews in edge mode. The edge already counted the page when it served the HTML; a JS pageview would double every number. What edge mode misses, stated on the tab: SPA route changes, HTML served by a CDN in front of CARLOS, and back/forward cache restores. The reserved "count": "browser" mode (§5) is the answer for those apps, not a same-second dedup heuristic. Until then an SPA may call carlos.track("route", {path}) and see it as a named event. Members include the snippet with one line:

<script src="/__carlos/carlos.js" defer></script>

3.5 The server door

Verbs (decision). errors.Mount and analytics.Mount follow schedule.Mount exactly: same bearer check, same 0600 socket, same host binding.

POST /errors   {kind, name, stack, path, status, fields, fp}
POST /events   {name, fields}

The app reads CARLOS_CONTROL_SOCKET and CARLOS_ADMIN_TOKEN from its environment as it already does for schedules. A Rails app wires it in an ActionDispatch error subscriber; a Go app in an http.Handler middleware. The confused-deputy discussion in purge.go applies unchanged: a same-uid sibling that can read the socket can post a fake error, and the worst it can do is put a row on the owner's own tab. A call on the control socket does not stamp lastActive: ListenControl serves its own mux (wired at backend_exec.go:624) and never enters Service.serve. The same 16 KB body cap, 8 KB stack cap and 600/min per-app buckets as the browser door apply, counted into the same dropped keys; the socket answers 202 on a drop and 400 only on a body it cannot parse, since the caller here is the owner's own code and can act on it.

Unit-backed and DynamicUser tenants have no control socket today and so have no server door; they have log and edge sources and the browser door like everyone else. This is the same gap purge.go records for schedules, and it closes when that one does.

3.6 The tabs, the API, the CLI

Errors tab (GET /a/{sqid}/apps/{app}/errors, handleAppErrors through appPage(w, r, "errors"), template app_errors.html, ActiveTab: "errors"). Default window 24 h, selectable 1 h / 24 h / 7 d. One row per fingerprint: source badge, kind, name, count in window, first seen, last seen, hosts affected, releases seen. Rows sort by last seen. A row expands in place to the stored sample: stack, path, status, fields, and a link to Logs at that timestamp. Filters: source, level, host, release. Grouping happens on read over the window. ReadWindow fetches the whole window and discards beyond logsCollectCap (50,000), which bounds memory, not bucket reads and not totals (Codex 14): when the cap is hit the tab says "showing the most recent 50,000 of a larger window" and the counts are labelled as counts within what was read. A day rollup is not built until a real app shows the read is too slow (§5).

Analytics tab (/apps/{app}/analytics). Range today / 7 d / 30 d. Headline: pageviews, visitors (daily, summed for ranges), bytes served, bots excluded, dropped. One chart, per hour for today and per day otherwise, drawn the way templates/metrics.html:29 draws sparklines: points computed in Go, an inline <svg> in the template, no library. Three tables: top pages, top referers, named events.

API (decision, Fable 8). Two cliRoute paths beside the logs read:

GET /api/cli/apps/{account}/{app}/errors      svcrole.Allow(Instance, Operate)
GET /api/cli/apps/{account}/{app}/analytics   svcrole.Allow(Instance, Operate)

Bearer-authenticated through apiAccountApp like /logs; an app token may read its own app's errors and analytics. The session → CSRF → owner (byte-identical 404) → fresh-auth ordering applies to the two pages, which go through appPage; the API paths have no session and no CSRF by construction. The tabs render from the same functions the API answers with.

CLI. carlos errors -app X [-since 24h] [-source browser] [-level warn] [-host h] [-release v] [-fp <id>] lists groups, -fp expands one; carlos analytics -app X [-range 7d] prints the headline, chart as a text sparkline, and the three tables. Both get headings in docs/site/cli.md in the same branch, or internal/docsite fails the gate.

Settings. The three keys of §3.3 on the app Settings tab and on the account settings page, and via carlos apps and carlos accounts, writing the two objects. Error capture has no switch: an app that is failing is an operational fact the owner needs; log_level only widens what the log source counts. A member may delete an error group from the tab or the API (DELETE …/errors/{fp}, owner on the page, Operate token on the API). This writes a tombstone apps/<sqid>/<app>/errors/tombstones/<fp>.json carrying the deletion time; the reader hides records of that fp with last before that time, from both errors/ and the log-derived rows, until retention removes the chunks. A later occurrence reappears: this is delete, not mute (§5). Logs themselves are untouched; deleting log lines is a Logs feature and not part of this design.

Retention (decision). Error chunks follow log retention, both age and byte cap, in logshipPrune. Analytics day files are kept 400 days, enough for year-over-year, and are small enough (tens of KB per app-day) that no byte cap is needed; the same prune loop deletes past the age.

Coverage line. Each tab carries one line saying which sources are live for this app: "log, edge" for an app with nothing wired, "+browser" once a beacon has arrived, "+server" once the socket verb has. A member should never wonder whether silence means no errors or no wiring.

4. The consequence to be explicit about: nothing here wakes an instance

Every path in §3 is checked against the scheduled-ticks rule:

• /__carlos/* is answered between route lookup and the meter wrap, before the cache and the activator. A beacon to a hibernating app returns 202 from the edge; beginRequest is never called and the idle clock is not stamped. The test reads a.entries[host].lastActive directly, as activator_test.go:1378 does.
• Analytics counting is a tap on responses the edge is already sending.
• The log source is a read over chunks the agent already ships.
• The server door is a control-socket verb, which does not stamp.
• Flushes ride the existing usageLoop and logshipLoop tickers in the agent.

The one new cost is bucket writes: one analytics day-file CAS per box per five minutes per app with traffic, and error chunks at the cadence of log chunks. Both are bounded by the caps in §3.2–3.3 regardless of traffic. Memory on the edge is bounded per app: 500-counter summaries, one 2 KB sketch, a 10,000-entry visitor bucket table, and the coalescing map capped at 1,000 fingerprints.

5. What is not built

• Resolve and mute. An owner will want to mark a group resolved or mute a known one. It is the first triage feature to add and comes before alerts; the tombstone in §3.6 is its seed.
• Alerts. "Email me when a new fingerprint appears" belongs with the existing alertwatch machinery, not here.
• Browser-counted mode ("count": "browser"): the snippet sends pageviews with SPA route support and prerender activation, and the edge stops counting HTML. For apps behind a CDN or built as SPAs.
• Event properties. fields on analytics events are accepted and dropped; property breakdowns change the day-file shape additively.
• Error day rollups. Grouping reads the window every time. Fine to 50k records; a heavy app will tell us when it is not.
• Forwarding. A forward: sentry|posthog|… setting that relays events to a rented service. The shape in §3.1 makes this a translator.
• Source maps, breadcrumbs, session replay, funnels, GeoIP, per-country tables. Rent those.
• Fleet-wide views. Errors across a member's apps, or across the fleet for ops, are a different page with different gates.

6. Testing

• Sanitisation: table test over messages, stacks, fields and paths containing emails, bearer tokens, password=, 32-char hex and base64 runs; each is replaced and nothing else is touched.
• Fingerprint: table test of norm: digits, hex, UUIDs and quoted strings are placeholders; v2, E404, HTTP 503 survive; frames lose line, column and bundle hash; a caller-supplied fp is taken verbatim and capped; fpv is stamped.
• Coalescing: 10,000 identical errors in one interval produce one record with count: 10000 and first < last; the same fp on two hosts produces two records; 1,001 distinct fingerprints for one app produce 1,000 records and one _overflow; a second app on the same box is unaffected.
• Log-derived rows: a level=error JSON line appears on the Errors read and in Logs; level=info, an unparseable line, and a plain panic: line do not appear as log errors.
• Edge errors: an upstream 500 on a POST (the plain path), a 502 from a failed wake, and a 500 on a cacheable GET each produce exactly one edge error; a 404 produces none.
• Chunk family: logs/ chunks written and read after the refactor are byte-identical to before; every existing applog test passes unchanged; errors/ round-trips the new record.
• Store door: a fleet box may write apps/<sqid>/<app>/errors/x and …/analytics/x and may not write apps/<sqid>/errors/x, apps/<sqid>/<app>/errors, or apps/<sqid>/<app>/other/x.
• Observing writer: status and Content-Type latch at WriteHeader and at implicit 200; visible below gzip; installed with a nil meter; byte counts unchanged for every existing usage test.
• Pageview definition: httptest through Edge.ServeHTTP with a fake upstream: HTML 200 GET counts; JSON 200, HTML 404, HTML 302, 304, HEAD, Sec-Purpose: prefetch, /__carlos/carlos.js do not; a cache hit counts; Sec-Fetch-Dest: empty does not; a known bot UA counts only into bots; a status route counts nothing; count: off counts nothing.
• Client IP: verified hop takes the last XFF element; unverified hop and direct take RemoteAddr; a client-set XFF on a direct request is ignored.
• Visitor hash: same IP+UA on the same day is one visitor; a different day is different; the salt survives a restart and never appears in the day file; the salt file is removed on rotation.
• HLL: across ten random datasets of 1k, 10k and 100k, the estimate is within 3 standard errors of truth; merge is commutative and idempotent; the encoding round-trips with its version byte.
• Misra–Gries: with fewer than 500 keys the summary is exact; two summaries merge and re-fold to 500 plus _other with the total preserved.
• Day-file CAS: event-time bucketing places a flush at 10:00:02 for events at 09:59 into 09; a failed flush re-queues and the next succeeds with the sum; two failures drop the older window and increment dropped.flush; two boxes writing the same app-day converge to the sum of pageviews and the union of sketches.
• Browser door: correct Origin counts; missing Origin with matching Referer counts; foreign Origin with matching Referer does not; Sec-Fetch-Site: cross-site does not; 17 KB body does not; the 601st post in a minute does not; each drop increments the right dropped key; every case answers 202. A hibernating route receives a beacon and lastActive is unchanged. On a path-routed hostname the beacon lands on the app owning its path prefix. __carlos is refused as a prefix-map key.
• /__carlos/ reservation: a route whose upstream serves /__carlos/x never sees the request; the loader revalidates and the hashed asset is immutable.
• Server door: a bad bearer is refused; a good one lands a record; lastActive is unchanged after the call; a 17 KB body answers 202 and increments dropped.e.size; count: off drops /events and still accepts /errors.
• Seq idempotency: re-merging a batch with an already-recorded seq leaves the day file unchanged.
• Tombstone: after DELETE, the group is absent from the read and the API; a new occurrence after the deletion time reappears; Logs still shows the underlying lines.
• Wake failure once: a failed wake produces one edge error, not two.
• Console gates: both pages give a non-member the byte-identical 404 the Logs tab gives; both API paths refuse a token without Instance or Operate.
• Settings resolution: with no object at either level nothing is counted and log_level is error; an account object with count: edge turns counting on for every app; an app object with count: off under that account turns it off for that app alone; writing either and bumping the serial converges on the box within the next pass; the browser door drops /__carlos/a and still accepts /__carlos/e when off; door: off drops both; log_level: warn brings warn lines into the Errors read and -level error filters them back out.
• Docsite: the gate passes with both new CLI headings present and fails with either removed.

7. Rollout

Each step is a separate branch that ships value on its own and needs no member action. Each is canaried on the flagship for a day against edge latency, edge memory, bucket write count and the hibernation invariant before promotion, in the ladder the fleet already runs.

1. Errors from log and edge sources, tab, API, CLI. Every app in the fleet gets a populated Errors tab. Requires the chunk-family refactor, the store-door allow-list, the observing writer (for edge errors), and the coalescing sink.
2. Analytics counting at the edge, tab, API, CLI, the settings objects at both levels, and the skill question. An account that turns it on gets pageviews and visitors for every app with no JavaScript. Shares the observing writer with step 1, so it lands second. The getting-started skill change is a companion branch in the website repo, landed the same day.
3. Browser door and carlos.js. Members opt in with one script tag; JS errors and named events appear.
4. Server door on the control socket. Exec-backed apps post structured errors and events.

Wire shapes are additive at every step: logs/ is untouched, errors/ and analytics/ are new prefixes, telemetry.json absent means off, and a box on an older build writes none of them. A mixed-version fleet shows the coverage line per host so a missing source is visible rather than puzzling.

Binaries roll as usual: promote → adopt → restart for the console; edge and agent pick up on adoption.

8. Rulings (Paul, 2026-09-07)

Four questions were open in the draft; Paul answered all four.

1. Analytics is off by default (ruling). Settable at the account level so one switch covers every app, overridable per app, and asked about in the getting-started skill. §3.3 carries the objects and the resolution order; §7 step 2 carries the skill change.
2. The reserved prefix is /__carlos/ (ruling). Paul asked whether a random per-app identifier would be better for unpredictability. The recommendation is no, and the spec stays static: the path is not a secret and cannot be one, because the snippet URL is in every page's source and the beacon endpoint is public by construction (§3.4). What bounds abuse is the Origin check, the caps, dropped, and the door switch, none of which a random path would strengthen. A random path would also cost the one-line snippet its one line and would make the reservation rule ("this prefix is never proxied upstream") per-app state instead of a constant. The double underscore buys the collision safety Paul wanted: no app in this repo serves it, and nothing conventional does.
3. Warn level is a setting, and level is always filterable (ruling). errors.log_level in §3.3 at either level, error by default; the tab, API and CLI filter on level regardless (§3.2, §3.6).
4. The salt lives beside the registry DB, 0600 (Fable's recommendation, taken). §3.3 records both reviewers' positions: Fable named the location and confirmed the agent owns it; Codex asked for restart-stability and cross-box sharing, and the persisted file gives the first while the home-box rule makes the second unnecessary.

The platform-infrastructure collision check for /__carlos/ is still worth one grep before step 3 ships; the double underscore makes a hit unlikely enough that it is a checklist line, not an open question.

9. Review round 1

Both reviewers read the first draft against main and returned needs-rework. What each changed:

Fable (second context). Moved the beacon intercept from "before routing" to between lookupUpstreamPath and the meter wrap, with the backhaul argument (§3.4). Replaced the recorder tap with the observing countingWriter, citing the three response paths that bypass the recorder (§3.3). Found that fleet boxes' writes would be refused by aclWriteAppLogs (§3.2). Corrected the §1 citations, which pointed at ACME and attribution comments. Defined clientIP from the verified-hop chain (§3.3). Listed the real refactor scope and the key-shape consumer list, and corrected retentionPass to logshipPrune (§3.2, §3.6). Argued the log mirror away in favour of a read-side filter (§3.2). Named the cliRoute API pattern and the gate-ordering scope (§3.6). Found that no box reads features.json, so the switch became a bucket object (§3.3). Sanitised page paths (§3.3). Corrected the chart precedent, the Logs-tab grep, and the non-existent test hook (§3.6, §4). Noted the 500-key merge overflow (§3.3).

Codex. Made the visitor salt restart-stable and the multi-day visitor figure honest (§3.3). Reframed the privacy claim as the deployment's assessment (§8). Tightened the pageview definition to 200 only, excluding redirects, 304s, HEAD and prefetch (§3.3). Reserved a browser-counted mode instead of a same-second dedup (§3.4, §5). Named the path-routed attribution problem independently of Fable (§3.4). Made fingerprints token-boundary, versioned, and frame-normalised (§3.2). Made the coalesced record carry first, last, host and release (§3.2). Made caps per app and exposed dropped everywhere something is dropped (§3.2, §3.3, §3.4). Replaced first-admitted maps with Misra–Gries and stated the HLL error correctly (§3.3). Tightened Origin matching and kept the uniform 202 only alongside owner-visible drop counts (§3.4). Added sanitisation before buffering and owner-triggered deletion (§3.1, §3.6). Made day-file buckets event-time and flushes re-queue on failure (§3.3). Added the collect-cap disclosure and the coverage line (§3.6). Split the snippet into a revalidating loader and a hashed immutable asset (§3.4). Put resolve/mute ahead of alerts in §5. Noted that a plain panic: line is not a parsed error, which §3.2 now answers through the edge source.

Not taken: a per-app public key for the browser door (identification without truth, and the caps already bound the damage; §3.4 says why); storing the raw pageview path (the token-collapsing rule keeps real page names and drops secrets); a mirror of log errors into errors/ (read derivation is cheaper and loses nothing).

10. Review round 2

Both reviewers re-read the rewrite. Fable: approve-with-amendments, all three blockers confirmed resolved against the code; amendments taken: the Origin check is a lookupIn on the edge's map, not an alias list (upstream.alias is a bool); Sec-Fetch-Dest: empty is a fetch, not a page; status routes are excluded from the tap; joined X-Forwarded-For lines; salt file 0600. Codex: needs-rework on under-specification; taken: 1–3 digit tokens survive normalisation so HTTP 503 does; the snippet stamps path on every beacon so path-routed events attribute; the coalescing cap counts records and the window rule is stated; a seq per box makes CAS retries idempotent; a door switch closes the browser door independently of counting and count: off also drops the socket's /events; the server door gets the same size and rate limits; Misra–Gries tables are labelled approximate above 500 keys; the dropped schema lists every key it uses, split by errors and events; wake failures are suppressed at the writer when the activator already emitted; tombstones have a scope, an authorisation, and a stated non-effect on Logs. Nothing from round 2 was declined.

11. Amendment 2026-09-24: Availability

Why. A day of the flagship's Errors tab (titogo, 2026-09-23) held 29 groups: 25 edge 502s, 3 edge 503s and one 500. Only the 500 was a failure of what a request asked for. The rest said the instance was down, restarting or unreachable when some page was asked for, and §3.2's "<status> <method> <path>" fingerprint split each outage into one row per page that happened to hit it. 17 of the 29 rows were one scanner probing /.env, /phpinfo.php and so on against a dead canary. The record also could not say why the edge had no answer: the proxies logged the error and threw it away.

Availability is the Errors tab's sibling (decision, Bill 2026-09-24). A sub-tab between Errors and Analytics (GET /a/{sqid}/apps/{app}/availability), an API path beside /errors with the same gate (GET /api/cli/apps/{account}/{app}/availability), and carlos availability. It holds:

• every edge-observed 502, 503 and 504, whoever wrote it; and
• the platform stream's events at warn and above (failed wakes, exits, a sidecar that will not restart), whatever errors.log_level says, since an exit on a signal is filed at warn and is exactly the event an outage row needs beside it. The synthetic rate-cap line (applog.IsDropNotice) is about the logging budget and is left out.

Errors keeps the rest: the app's own 500, 501 and 505+, and the log source's app-stream lines. Nothing is in both.

Grouped by cause, never by path (decision). An edge 502/503/504 is filed as kind: unavailable with cause and name: "<status> · <cause in the member's terms>", and so fingerprinted by status and cause alone; the path stays on the record as the occurrence's sample. The cause comes from the ErrorHandler that wrote the status, through a per-request outcome on the context (edge/answered.go, which already carried the activator's answered flag and now rides every observed request, not only hibernating ones):

| cause | where | |---------------|--------------------------------------------------------------| | refused | ECONNREFUSED / ENOENT on the dial: nothing listening | | reset | EOF, unexpected EOF, ECONNRESET, EPIPE: hung up mid-request | | timeout | a deadline or a net timeout | | abandoned | the client left after waiting 5 s or more (below) | | no_upstream | multiProxy: every upstream failing its probe | | fleet_dark | fleetProxy: no box of the fleet connected | | tunnel | any other failure through a fleet box's tunnel | | static | a static site's files could not be resolved | | proxy | any other proxy failure | | upstream | no handler noted one: the app wrote the status itself (direct routes only; below) |

unixProxy, tcpProxy and tunnelProxy had no ErrorHandler; they get one that is httputil.ReverseProxy's default (same log line, same bare 502) plus the note.

A visitor who left quickly is not an outage; one kept waiting is (decision, amended in review). When the client's own context is done by the time a proxy fails, ReverseProxy still writes a 502, to nobody, and the error is the cancellation whatever the instance was doing. Within 5 s of the request (clientGoneGrace) the edge notes client_gone and files nothing: a visitor navigating away mid-load is not the instance's fault, and a single-count 502 on a font file is the typical shape of it. Past 5 s it files abandoned. The review caught why that half matters: a dead unix socket is redialled for 13 s and a wedged instance is never timed out at all, so a monitor or visitor giving up first is how those outages end — skipping them would hide exactly the worst ones.

Behind a tunnel, the front box files only the hop (decision, Fable review; corrected in its follow-up). When a box proxies a request over a reverse channel, the upstream is not the app but another box's edge — the agent's, the only tunnel client — which files what it sees serving the app, and deliberately files nothing for a wake 503 its own activator answered. A 5xx that merely passed back through the front box was therefore filed twice, and under this amendment would have been labelled "the app answered it itself" for an app that never saw the request. The front box now files only failures it noted itself (tunnel, fleet_dark, abandoned); everything else is the serving box's record. This also ends the double filing of tunnelled 500s, which predates the amendment.

The rule keys on how THIS box served the request (upstream.viaTunnel, the build-time decision), never on the route row: the fleet box that serves the app carries the same fleet/<name> placement on its own row, has no hub, and serves from the socket — it must file everything. (The first cut keyed on the row, and a fleet-placed app's 500s were filed nowhere; TestEdgeServingFleetBoxFilesItsOwn pins the far end.) Two edges remain: a far agent whose status store failed to open has no error sink and files nothing, which the front box no longer covers; and a client that leaves past the grace can be filed abandoned on both boxes, since the stream reset reaches the far box's request too.

The overflow row is per family (decision, from review). The sink's hourly _overflow row copied its source, kind and status from the first record that overflowed, so one family's excess could land on the other tab. It is now one row per family (IsAvailability), and AsAvailability leaves an _overflow row alone.

History regroups on read (decision). Records filed before this carry kind: http, no cause, and the per-path fingerprint. The read recognises them by status and regroups them as "<status> · cause not recorded" (apperr.AsAvailability); the FP filter applies after the regroup so the row the tab shows expands. The bucket is not rewritten.

Not changed. The wire shape is additive (cause and a read-only stream on apperr.Record, both omitempty); nothing under logs/ or errors/ moves. The sending half of a backhaul still files nothing when its hop fails: it has no route row and so no app to file under, the same reason it bills nothing. Correlating a row with the nearest deploy or restart on that host, and alerting on a new cause, are the natural next steps and are not built here.