| 1 | # Errors and Analytics: two tabs every app gets for free |
| 2 | |
| 3 | **Date:** 2026-09-07 |
| 4 | **Status:** approved (Paul, 2026-09-07: "go ahead with that and plan, build |
| 5 | review cycle overnight"). Rollout step 1 (§7) is merged (forge main 1a99060; plan |
| 6 | `docs/superpowers/plans/2026-09-07-errors-step1-log-and-edge.md`). Step 2 is |
| 7 | implemented on branch `analytics-step2` (plan |
| 8 | `docs/superpowers/plans/2026-09-07-analytics-step2-edge-counting.md`). Every |
| 9 | choice below is marked **(decision)** and can be overturned cheaply; the |
| 10 | four in §8 are Paul's **(ruling)**s of 2026-09-07 and are marked where |
| 11 | they land. Drafted by Claude Fable, |
| 12 | then reviewed by a second Fable context and by Codex; both returned |
| 13 | needs-rework on the first draft and §9 records what they changed. Line numbers cite `origin/main` at `09a22d6`. |
| 14 | |
| 15 | ## 1. The idea, and why it fits this platform |
| 16 | |
| 17 | A member who ships an app to CARLOS today gets a URL, a certificate, logs, |
| 18 | schedules, and a bill. What they do not get is the two questions every |
| 19 | operator asks within a day of launch: *is it broken* and *is anyone using |
| 20 | it*. The answers live in AppSignal, Datadog, Sentry, PostHog, Plausible: a |
| 21 | second account, a second bill, an SDK to wire in, and for the analytics |
| 22 | half a cookie banner to justify. |
| 23 | |
| 24 | The platform is unusually well placed to answer both questions itself, |
| 25 | because it already stands in the two places the data passes through: |
| 26 | |
| 27 | - **The edge sees every request** to every app, including the ones its |
| 28 | cache answers without waking the instance (`writeCached`, |
| 29 | `internal/edge/edge.go:1311`). Method, path, status, referer and user |
| 30 | agent are in hand at the point where `countingWriter` is installed to |
| 31 | bill the bytes (`edge.go:1241`). Server-side page-view counting there |
| 32 | costs the app nothing and needs no JavaScript and no cookie. It is the |
| 33 | Plausible model, done where Plausible cannot stand. |
| 34 | - **The agent already captures every line the app writes** and ships it |
| 35 | as hour chunks to the bucket (`internal/applog`, |
| 36 | `internal/host/logship.go`). A line the app tagged `level=error` is |
| 37 | already an error event; it is just filed under Logs, undifferentiated |
| 38 | from a million info lines. And when the app dies, the edge answers a |
| 39 | 502 or reports a wake failure: the crash is already observed. |
| 40 | |
| 41 | So the first version of both tabs is mostly *reading what is already |
| 42 | flowing past*. The hooks add the cases the platform cannot see on its own: |
| 43 | a public endpoint on the app's own hostname for browser errors and named |
| 44 | events, and a loopback verb for server-side errors and events. The member |
| 45 | who wants AppSignal can still rent it; the event shape is deliberately |
| 46 | close to what those services accept, so a "forward to" setting later is a |
| 47 | translator, not a redesign (§5). |
| 48 | |
| 49 | The proposal is two tabs on the app dashboard, **Errors** and |
| 50 | **Analytics**, between Logs and Activity in `appnav.html`, each with a |
| 51 | console API and a `carlos` verb behind it (AGENTS.md: every command must |
| 52 | complete the job for a user with no infra access). |
| 53 | |
| 54 | ## 2. What is already right, and must stay right |
| 55 | |
| 56 | **Logs are the pattern for events.** `applog.Record` |
| 57 | (`internal/applog/record.go:72`) is buffered on the box under a token |
| 58 | budget (`sink.go:95`), flushed as gzip'd JSONL hour chunks under |
| 59 | `apps/<sqid>/<app>/logs/<box>/<hour>/`, read back by `applog.ReadWindow` |
| 60 | merging chunks across boxes, pruned by `logshipPrune` |
| 61 | (`logship.go:161`) on age and byte cap, and served through |
| 62 | `GET /api/cli/apps/{account}/{app}/logs` (`authcli.go:595`) with a cursor. |
| 63 | Errors that the platform must store are events too; they reuse this |
| 64 | machinery rather than growing a second one (§3.2). |
| 65 | |
| 66 | **Usage is the pattern for counters.** `usage.EdgeDay` |
| 67 | (`internal/usage/edge.go`) is a CAS-merged day file, hour-bucketed, one |
| 68 | per box, flushed every five minutes by `usageLoop` |
| 69 | (`internal/host/usage_loop.go:27`) from `UsageMeter.Drain()`. Page views |
| 70 | and visitors are counters; they reuse this shape, with two durability |
| 71 | fixes the reviewers asked for (§3.3). |
| 72 | |
| 73 | **The control socket is the pattern for a loopback API.** `ListenControl` |
| 74 | (`internal/edge/purge.go:108`) mounts verbs on a 0600 unix socket the app |
| 75 | finds in `CARLOS_CONTROL_SOCKET`; `schedule.Mount` |
| 76 | (`internal/schedule/control.go:41`) shows the bearer check against |
| 77 | `CARLOS_ADMIN_TOKEN`. Server-side ingestion mounts beside it (§3.5). The |
| 78 | socket is not the activator: a call on it never reaches `beginRequest`. |
| 79 | |
| 80 | **Nothing the platform does for itself may keep an instance awake.** The |
| 81 | scheduled-ticks spec (2026-08-30) exists because one line of accounting |
| 82 | cost a member fifteen minutes of billed residency per millisecond of |
| 83 | work. Every ingest path here is handled in the edge or the agent and none |
| 84 | enters `activator.Service.ServeHTTP`; a beacon from a browser to a |
| 85 | hibernating app is answered by the edge and the app stays asleep (§4). |
| 86 | |
| 87 | **The edge's wrapper order is load-bearing** (AGENTS.md): recorder above |
| 88 | the compressor, usage counter below it, wrap site after the upgrade |
| 89 | branch. This design changes what the usage counter *observes*, never |
| 90 | where it sits (§3.3). |
| 91 | |
| 92 | **Fleet boxes write through the store door**, whose write grant is |
| 93 | segment-exact `apps/<sqid>/<app>/logs/**` (`aclWriteAppLogs`, |
| 94 | `internal/edge/storedoor.go:503`), and usage is rebased under |
| 95 | `fleets/<name>/` (`usage_loop.go:47`). Two new families need the door to |
| 96 | know about them, or a fleet box's writes are refused (§3.2). |
| 97 | |
| 98 | ## 3. The change |
| 99 | |
| 100 | ### 3.1 One event shape, four sources |
| 101 | |
| 102 | Both tabs are fed by events in one JSON shape. **(decision)** Fields: |
| 103 | |
| 104 | ``` |
| 105 | ts RFC3339 nanoseconds, stamped by the platform, never taken from input |
| 106 | source "log" | "edge" | "server" | "browser" |
| 107 | kind errors: exception class or "http"; analytics: "pageview" | "event" |
| 108 | name errors: the message; analytics: the event name |
| 109 | host route host that received it (how owners name instances) |
| 110 | box the box that recorded it |
| 111 | release X-Carlos-Version of the serving build, when known |
| 112 | path request path, query and fragment removed, token segments collapsed (§3.3) |
| 113 | status HTTP status, when the event came from a request |
| 114 | referer host-only, never the full URL |
| 115 | ua browser family only, never the raw string |
| 116 | stack errors only, capped at 8 KB |
| 117 | fields map[string]string of caller-supplied scalars, 32 keys, 256 B each |
| 118 | fp errors only: fingerprint (§3.2) |
| 119 | fpv errors only: fingerprint algorithm version, 1 |
| 120 | ``` |
| 121 | |
| 122 | Sources and where they enter: |
| 123 | |
| 124 | | source | errors | analytics | |
| 125 | |---------|----------------------------------------------------------|---------------------------------------------| |
| 126 | | log | applog record at or above `errors.log_level`, read-derived | none | |
| 127 | | edge | upstream 5xx, gateway 502/504, wake failure | every HTML 200 GET becomes a pageview | |
| 128 | | server | `POST /errors` on the control socket | `POST /events` on the control socket | |
| 129 | | browser | `POST /__carlos/e` on the app's own host | `POST /__carlos/a` on the app's own host | |
| 130 | |
| 131 | `log` and `edge` need no integration at all. They are why the Errors tab |
| 132 | is not empty on day one for any app in the fleet, and why Analytics |
| 133 | starts counting the moment an account turns it on (§3.3). |
| 134 | |
| 135 | **Sanitisation before buffering (decision, Codex).** Errors carry |
| 136 | messages, stacks, paths and caller fields, and those carry secrets in |
| 137 | practice. Before any record is buffered, on every source: query strings |
| 138 | and fragments are removed from `path`; in `name`, `stack` and `fields`, |
| 139 | email addresses become `<email>`, `Bearer`/`Basic` credentials and |
| 140 | `token=`/`key=`/`secret=`/`password=` values become `<redacted>`, and |
| 141 | hex or base64 runs of 32+ characters become `<hex>`. The browser snippet |
| 142 | applies the same rules before sending, so nothing leaves the page that |
| 143 | the platform would strip. This is a floor, not a guarantee; the Errors |
| 144 | tab says so in one line. |
| 145 | |
| 146 | ### 3.2 Errors: sources, fingerprints, coalescing, storage |
| 147 | |
| 148 | **The `log` source is derived on read, not stored twice (decision, Fable |
| 149 | review 7).** The first draft mirrored `level=error` lines into a second |
| 150 | family. `ReadWindow` with `Filter.Level="error"` (`filter.go:22`) over |
| 151 | `logs/` yields the same rows with the same retention, and a mirror would |
| 152 | have had to choose a side of the token budget at `sink.go:95`. So the |
| 153 | Errors tab reads two families and merges: `logs/` filtered to the |
| 154 | levels the `errors.log_level` setting names (§3.3; `error` means |
| 155 | `error`+`fatal`, `warn` adds `warn`; **a setting, per Paul's ruling of |
| 156 | 2026-09-07**), and `errors/` for the other three sources. Whatever the |
| 157 | setting, every row carries its level and the tab, API and CLI filter on |
| 158 | it, so a member on `warn` can look at errors only and a member on |
| 159 | `error` can widen one read to `warn` without changing the setting. A |
| 160 | JSON line whose `level` is unrecognised is not an error; `ParseAppLine` never |
| 161 | guesses, and neither does this. A plain-text `panic:` on stderr carries no |
| 162 | level and is not a `log` error either: the crash reaches the tab through |
| 163 | the `edge` source when the next request meets a dead instance, which is |
| 164 | the honest signal. |
| 165 | |
| 166 | **Edge errors (decision).** Where the observing writer (§3.3) sees a |
| 167 | final status of 500–599 from the upstream, or the edge itself answers |
| 168 | 502/504 because the instance did not come up, it emits `source: edge, |
| 169 | kind: http, name: "<status> <method> <path>"`. Wake failures already reach |
| 170 | the member through `emitEvent` in `internal/activator/logsink.go`; the |
| 171 | same call site sends one record to the errors sink. To keep that exactly |
| 172 | once, the activator branch marks the request context when it answered |
| 173 | the failure itself, and the observing writer skips a 502/504 that |
| 174 | carries the mark; the writer's own emission covers only statuses the |
| 175 | upstream returned or the proxy produced without the activator. Client 4xx are not errors. |
| 176 | |
| 177 | **Fingerprint (decision, amended per both reviews).** `fp` groups |
| 178 | occurrences into one row. A caller may supply one, capped at 64 bytes. |
| 179 | Otherwise the platform computes `sha256(len-delimited(source, kind, |
| 180 | norm(name), norm(top frame)))[:16]` with `fpv: 1`. `norm` splits on whitespace |
| 181 | and punctuation; a token that is a digit run of 4+ characters, a hex run |
| 182 | of 8+, a UUID, or a quoted string becomes a placeholder; tokens of 1–3 |
| 183 | digits and mixed tokens like `v2` or `E404` are kept, so `HTTP 503`, |
| 184 | `v2` and `retry 3` still separate groups while `user 4123` and |
| 185 | `user 9911` do not. Small ids under 1,000 will split groups; that is the |
| 186 | stated trade-off, and `fp` overrides it. A frame is normalised to file and function with line, |
| 187 | column and bundle hash removed, so a redeploy does not split every |
| 188 | browser group. The rule is versioned so it can change without merging |
| 189 | old groups into new ones. It is wrong sometimes; a caller who knows |
| 190 | better sets `fp`. |
| 191 | |
| 192 | **Coalescing at the box (decision, amended per Codex 8).** An error storm |
| 193 | is when the tab must work and when a naive stream fails. The box sink |
| 194 | keeps, per `(app, fp, host, release)` per flush interval, one record |
| 195 | carrying `count`, `first`, `last`, and the first sample's `stack`, `path`, |
| 196 | `status` and `fields`. Ten thousand identical panics in a minute become |
| 197 | one record with `count: 10000`. Coalesced records are capped per app per |
| 198 | hour at 1,000, counting `(fp, host, release)` combinations, not distinct |
| 199 | fingerprints, so the memory bound is real (per app, not per box: one |
| 200 | tenant's storm must not suppress a neighbour's, Codex 9); beyond that, |
| 201 | the overflow folds into a single `_overflow` row whose `count` is |
| 202 | occurrences, labelled as such on the tab. A record belongs to a read |
| 203 | window when `[first, last]` overlaps it and is counted whole; the flush |
| 204 | interval is five minutes, so a window edge is wrong by at most one |
| 205 | interval, and the tab's window labels say "to the nearest flush". |
| 206 | (Correction, 2026-09-24: the ship loop as built flushes every 10 s, |
| 207 | `errshipTick` in `internal/host/errship.go`, so records coalesce over |
| 208 | 10 s, not five minutes. Found reviewing the request-reference spec.) |
| 209 | |
| 210 | **Storage (decision, amended per Fable 6).** `errors/` is a sibling family |
| 211 | of `logs/`: `apps/<sqid>/<app>/errors/<box>/<hour>/<seq>-<size>.jsonl.gz`. |
| 212 | The chunk key and hour layout are shared, so `ChunkKey`, `Prefix`, |
| 213 | `BoxPrefix`, `HourPrefix`, `ChunkKeySize`, `hourNameOfDir`, `hourNameOf`, |
| 214 | `logshipPrune` and `logshipLoop` take a family argument. The record is |
| 215 | not shared: `applog.Record` has none of `fp`, `count`, `first`, `last`, |
| 216 | `stack`, `source`, `kind`, `status`, `path`, `release`, and `EncodeChunk`, |
| 217 | `DecodeChunk`, `ReadWindow`, `Filter.Match` and `Cursor` all take |
| 218 | `Record`. The codec becomes generic over the record type, or `Record` is |
| 219 | widened with `omitempty` fields; the implementer picks whichever leaves |
| 220 | `logs/` chunks byte-identical, and the existing applog tests are the |
| 221 | proof. The consumer list for the key shape, per AGENTS.md, is: |
| 222 | `ChunkKeySize` (`chunk.go:65`), `hourNameOfDir` (`read.go:91`), |
| 223 | `hourNameOf` (`logship.go:216`), `aclWriteAppLogs` (`storedoor.go:503`), |
| 224 | `logshipPrune` (`logship.go:161`). Wire shape is additive: nothing under |
| 225 | `logs/` changes. |
| 226 | |
| 227 | **Store door (decision, Fable 3).** `aclWriteAppLogs` becomes a family |
| 228 | allow-list: `logs`, `errors`, `analytics`, each with the same |
| 229 | "at least one component below" rule, with door tests for every family |
| 230 | and for `errors` as an app name. Without this a fleet box's writes are |
| 231 | refused and the tabs are empty for exactly the members on fleets. |
| 232 | |
| 233 | ### 3.3 Analytics: counting at the edge |
| 234 | |
| 235 | **Where the edge sees it (decision, Fable 2).** The first draft assumed a |
| 236 | tap on the recorder. The recorder is built only on the hibernating and |
| 237 | cacheable branches (`edge.go:1322, 1333`); a cache hit goes through |
| 238 | `writeCached` (`:1311`) and the plain path (`:1339`), every POST included, |
| 239 | has no recorder at all. The one wrapper installed at a single site for |
| 240 | every response shape is `countingWriter` (`usagemeter.go:64`), which today |
| 241 | has no `WriteHeader` and is installed only when a meter is armed |
| 242 | (`edge.go:1241`). It becomes an *observing* writer: installed |
| 243 | unconditionally with a nil-safe meter, latching final status and |
| 244 | `Content-Type` at the first `WriteHeader` of 200 or above (1xx are |
| 245 | forwarded and ignored), an implicit 200 at first `Write`, and a 200 with |
| 246 | no `Content-Type` when the handler returns having written nothing, which |
| 247 | is never a pageview. `Flush` and `Hijack` forward as they do today. The |
| 248 | meter wrap today skips `Kind == "status"` routes (`edge.go:1241`); the |
| 249 | observing writer is installed there too, but the tap ignores status |
| 250 | routes, so the status host never enters any app's analytics. It |
| 251 | stays below the compressor; `Content-Type` is visible there because |
| 252 | `gzipWriter.syncHeader` copies its private map down before the underlying |
| 253 | `WriteHeader` (`compress.go:236`). HEAD skips gzip (`compress.go:197`), so |
| 254 | the tap excludes by method itself. The upgrade branch (`:1257`) is |
| 255 | untouched; a 101 is never a pageview. The order rule in AGENTS.md is |
| 256 | preserved: nothing moves, one wrapper learns to look. |
| 257 | |
| 258 | **What is counted (decision, amended per Codex 3).** A request is a |
| 259 | *pageview* when the method is GET, the final status is 200, and the |
| 260 | response media type parses as `text/html`. Redirects are not views (the |
| 261 | destination will be); 304s are not views (no media type, and a |
| 262 | revalidated page is an undercount the tab states); 204/205 are not; |
| 263 | requests with `Sec-Purpose: prefetch` or `Purpose: prefetch` are not; |
| 264 | `Sec-Fetch-Dest`, when present, must be `document` (an `empty` dest is |
| 265 | `fetch()`, which is how htmx and SPAs load HTML fragments, and a fragment |
| 266 | is not a page; an absent header is an old UA or a bot, and the bot list |
| 267 | handles the latter). Requests under |
| 268 | `/__carlos/` are never counted. A request whose user agent matches the |
| 269 | vendored bot list is counted into `bots` and nowhere else; the list is |
| 270 | incomplete by nature and the tab shows `bots` so the member can see what |
| 271 | was excluded. |
| 272 | |
| 273 | **Client IP (decision, Fable 5).** There is no client-IP value in the |
| 274 | edge today. `clientIP(r)` is defined as: the last element of |
| 275 | `X-Forwarded-For` (all header lines joined, as `setForwarded` does at |
| 276 | `homemap.go:915`) when `hopVerified(ctx)` (`homemap.go:767` accepts a |
| 277 | verified chain verbatim; `:800` strips an unverified one), otherwise the |
| 278 | host of `RemoteAddr`. A header the client can set never reaches it. |
| 279 | |
| 280 | **Visitor (decision, amended per Codex 1 and Fable 10).** A visitor is |
| 281 | `sha256(day-salt ‖ route host ‖ clientIP ‖ user agent)[:8]`. The salt is |
| 282 | per box per UTC day, generated at first use, written 0600 to the box's |
| 283 | state directory beside the registry DB (`/data/`, which the agent |
| 284 | already owns) so a restart mid-day does not re-salt and double-count, |
| 285 | and deleted on rotation; it is never written to the bucket. That |
| 286 | location is Fable's recommendation, confirmed against who owns the |
| 287 | directory; Codex's ask was for a restart-stable, app-scoped salt shared |
| 288 | by every box serving the app, which the persisted file gives for the |
| 289 | first half and which the home-box rule makes unnecessary for the |
| 290 | second. |
| 291 | No cookie is set, no IP is stored, and the platform holds nothing that |
| 292 | joins a hash across days. Because a route is served by its home box (a |
| 293 | box without the row backhauls, `edge.go:1216`), a visitor is normally |
| 294 | seen by one box; during an instance move they are counted twice for a |
| 295 | day. Multi-day ranges on the tab are labelled "daily visitors, summed", |
| 296 | because that is what they are; Plausible's are too. |
| 297 | |
| 298 | **Cardinality (decision, amended per Codex 10).** Unique visitors per day |
| 299 | per box are a HyperLogLog sketch with 2¹¹ registers (2 KB, standard error |
| 300 | about 2.3%), one per day, encoded as base64 with a one-byte version |
| 301 | prefix, merged across boxes by register max. Pageviews per hour are |
| 302 | integers. Top pages, top referers and named events are Misra–Gries |
| 303 | summaries with 500 counters each: bounded, mergeable, and correct for any |
| 304 | app with fewer than 500 distinct keys a day, which is most of them. The |
| 305 | reader re-folds after merging so the tab never shows more than 500 rows |
| 306 | plus `_other`. Below 500 keys the counts are exact; above, a counter is a |
| 307 | lower bound within the summary's error and the table is labelled |
| 308 | "approximate: more than 500 distinct keys today". Keys are capped at 256 |
| 309 | bytes. |
| 310 | |
| 311 | **Page paths (decision, Fable 10).** `pages` keys are the sanitised |
| 312 | `path` of §3.1: query and fragment removed, and any segment that is a |
| 313 | UUID, a hex run of 16+ characters, a base64-looking run of 20+, or an |
| 314 | email replaced by `:token`. `/reset/8f3a…` never reaches the bucket; |
| 315 | `/blog/2024-review` does. |
| 316 | |
| 317 | **Storage (decision, amended per Codex 13).** |
| 318 | `apps/<sqid>/<app>/analytics/<box>/days/<date>.json`, written by a |
| 319 | sibling of `UpsertEdgeDay` with the same read-etag-merge-conditional-put |
| 320 | loop, and two differences from usage. First, buckets are keyed by *event |
| 321 | time*, not flush time, so a flush delayed across an hour boundary lands |
| 322 | in the right hour. Second, the drain returns deltas, and a failed flush |
| 323 | re-queues its deltas into the next drain rather than losing them |
| 324 | (`usageLoop` documents that it loses the window; modelling grade is fine |
| 325 | for billing estimates, not for a number a member reads as truth). One |
| 326 | window survives a failure; a second consecutive failure drops the older |
| 327 | one and increments `dropped.flush`. A conditional put whose outcome is |
| 328 | unknown (timeout after send) must not double-count on retry: every delta |
| 329 | batch carries a box-monotonic `seq`, the day file records `seq` per box, |
| 330 | and a merge whose `seq` is not greater than the recorded one is a no-op. A process restart loses the in-memory |
| 331 | window, stated on the tab as "counts since the last flush may be |
| 332 | missing" only when a restart happened that day. Shape: |
| 333 | |
| 334 | ```json |
| 335 | { |
| 336 | "v": 1, |
| 337 | "date": "2026-09-07", |
| 338 | "hours": {"09": {"pv": 412, "bots": 30, "bytes": 1843201}}, |
| 339 | "visitors": "AQ…", |
| 340 | "pages": {"/": 300, "/pricing": 80, "_other": 32}, |
| 341 | "referers": {"news.ycombinator.com": 200, "direct": 180}, |
| 342 | "events": {"signup": 12, "checkout": 3}, |
| 343 | "seq": {"box-a": 1187}, |
| 344 | "dropped": {"e": {"rate": 0, "size": 0, "origin": 0, "malformed": 0}, |
| 345 | "a": {"rate": 0, "size": 0, "origin": 0, "malformed": 0}, |
| 346 | "flush": 0}, |
| 347 | "countries": {} |
| 348 | } |
| 349 | ``` |
| 350 | |
| 351 | `countries` is present and empty in v1 so a GeoIP pass is additive. |
| 352 | `dropped` is what the member sees when a cap or a failure bit: the tab |
| 353 | shows it, so a number that is not the whole truth says so. |
| 354 | |
| 355 | **Named events.** `carlos.track("signup")` from the browser, or |
| 356 | `POST /events {"name":"signup"}` from the server, increments |
| 357 | `events[name]`. Names are lowercased, 64 bytes of `[a-z0-9_.-]`, and |
| 358 | bounded by the same 500-counter summary. Event properties are accepted in |
| 359 | `fields` and *not stored* in v1 (§5). |
| 360 | |
| 361 | **Accumulation.** The edge holds deltas in memory; the agent's |
| 362 | five-minute `usageLoop` cadence flushes them in the same tick as usage, |
| 363 | as a second CAS write (two objects, two writes; the first draft's "one |
| 364 | round-trip" was wrong). |
| 365 | |
| 366 | **The settings object (decision, Fable 9; default is a ruling).** |
| 367 | `features.json` is console-only and no box reads it. The switches are one |
| 368 | bucket object at two levels with the same shape: |
| 369 | |
| 370 | ``` |
| 371 | accounts/<sqid>/telemetry.json account default |
| 372 | apps/<sqid>/<app>/telemetry.json per-app override |
| 373 | {"analytics": {"count": "edge" | "off", "door": "on" | "off"}, |
| 374 | "errors": {"log_level": "error" | "warn"}} |
| 375 | ``` |
| 376 | |
| 377 | Each key resolves app → account → built-in default, so an account can |
| 378 | turn analytics on once for every app it has and a single app can still |
| 379 | differ. The console writes either object with a serial bump and boxes |
| 380 | converge on the next pass like `instances/*.json`; the account object |
| 381 | lives beside `account.json`, which stays identity-only. Built-in |
| 382 | defaults: **`count: off` (ruling, Paul, 2026-09-07)**, `door: on`, |
| 383 | `log_level: error`. Analytics is therefore opt-in: the visitor design is |
| 384 | built to be defensible without consent, but whether it is consent-free |
| 385 | is the deployment's assessment and the member's, not the platform's, so |
| 386 | the platform does not switch it on for anyone. `count: off` stops |
| 387 | counting at the observing writer and makes both doors drop analytics |
| 388 | events (`/__carlos/a` and the socket's `/events`) with `202`; existing |
| 389 | day files stay until retention. `door: off` closes the browser door |
| 390 | entirely, errors included, for an owner who wants no public ingest |
| 391 | surface at all. `"count": "browser"` is reserved for a later mode (§5) |
| 392 | so the shape is additive. |
| 393 | |
| 394 | Because the default is off, the tab is not empty when it is off: it |
| 395 | shows the switch, what turning it on records, and the one-line privacy |
| 396 | reading, so a member decides from the tab rather than finding a setting |
| 397 | by accident. The `getting-started` skill that `carlos skills` fetches |
| 398 | from carlosframework.com (published from the website repo, not here) |
| 399 | gains one question in the same step: whether to turn analytics on for |
| 400 | the account, with the same one-line reading. That is where an agent |
| 401 | setting up a member's first app will ask it. |
| 402 | |
| 403 | ### 3.4 The browser door |
| 404 | |
| 405 | **Where in the edge (decision, Fable 1, Codex 6).** The first draft said |
| 406 | "before any routing". That is the wrong spot: the app is known only from |
| 407 | `up.route`, which exists after `lookupUpstreamPath` (`edge.go:1197`), and |
| 408 | a box that has no row for the host must backhaul (`:1216`) so the home |
| 409 | box answers. The intercept sits between the lookup (`:1197`) and the |
| 410 | meter wrap (`:1241`): after routing, before upgrade (`:1257`), gzip |
| 411 | (`:1275`), cache (`:1311`) and the activator (`:1322`). At that spot |
| 412 | beacons are unbilled like `status` routes and `beginRequest` is never |
| 413 | reached, which is what §4 promises. |
| 414 | |
| 415 | **Attribution on a shared hostname.** `lookupUpstreamPath` can route |
| 416 | different path prefixes on one hostname to different apps (`:1008`). A |
| 417 | beacon carries no app id. The snippet puts `location.pathname` in every |
| 418 | beacon's `path`, errors and events alike, and the beacon is attributed |
| 419 | to the upstream the lookup returns for that *raw* path on that hostname |
| 420 | (the match runs before §3.1 sanitisation, which is applied to what is |
| 421 | stored), and to the hostname's root app when `path` is absent or matches |
| 422 | nothing. Because |
| 423 | `Origin` must equal that hostname, the sender is already a page served |
| 424 | from it: same origin, same trust boundary the browser already grants |
| 425 | those siblings. An unknown hostname takes the existing `!ok` branch and |
| 426 | is never answered by this code. The lookup must never accept `__carlos` as |
| 427 | a prefix key; a test pins that. |
| 428 | |
| 429 | **Endpoint (decision).** |
| 430 | |
| 431 | ``` |
| 432 | POST /__carlos/e one error {kind, name, stack, path, fields, fp} |
| 433 | POST /__carlos/a one event {name, fields} | {"name":"pageview","path":…} |
| 434 | GET /__carlos/carlos.js loader, Cache-Control: max-age=300, must-revalidate |
| 435 | GET /__carlos/carlos.<hash>.js the snippet, immutable |
| 436 | ``` |
| 437 | |
| 438 | Both POSTs answer `202` with an empty body in every case: accepted, |
| 439 | rate-limited, rejected, malformed. The endpoint reveals nothing, and the |
| 440 | browser never sees a failure worth retrying. Every drop increments the |
| 441 | app's `dropped.rate` (or `dropped.origin`, `dropped.size`), which the |
| 442 | tab shows, so silence outward is not silence to the owner (Codex 11). |
| 443 | `/__carlos/` is reserved on all routes and never proxied upstream from |
| 444 | this change on. No app in this repo serves it; §8 asks about the |
| 445 | deployment roots. |
| 446 | |
| 447 | **Who may post (decision, amended per Codex 11, Fable 12).** The request |
| 448 | must carry an `Origin` whose scheme is `https` and whose host and port |
| 449 | resolve, through the edge's own map (`lookupIn(e.current(), originHost)`, |
| 450 | the same lookup the request itself took), to an entry whose `route.Host` |
| 451 | equals `up.route.Host`; aliases share the primary's route verbatim, so |
| 452 | this covers them with no registry call on the hot path. A bad |
| 453 | `Origin` is rejected even if `Referer` passes; `Referer` is consulted |
| 454 | only when `Origin` is absent. `Sec-Fetch-Site: cross-site` is rejected |
| 455 | outright. This is identification, not authentication: a direct client |
| 456 | can forge any of it, as it can against every public beacon on the |
| 457 | internet, and a per-app public key would add revocation but not truth. |
| 458 | What bounds the damage is that a forged post can only put rows on the |
| 459 | owner's own tab, under the owner's own caps, and the owner can turn the |
| 460 | door off. Body is capped at 16 KB; `stack` at 8 KB. |
| 461 | |
| 462 | **Rate (decision, amended per Codex 9).** Token buckets per *app* of |
| 463 | 600/min for errors and 600/min for events, separately, and per visitor |
| 464 | hash of 60/min, all in edge memory, all dropping silently and counting |
| 465 | `dropped.rate`. Visitor-bucket state is bounded to 10,000 entries per app |
| 466 | with LRU eviction. Errors are then coalesced by fingerprint at the box as |
| 467 | in §3.2, so a page throwing in a loop on ten thousand screens is one row |
| 468 | with a large count and, if it exceeds the bucket, an honest `dropped`. |
| 469 | |
| 470 | **The snippet (decision).** `carlos.js` is vendored into the edge binary |
| 471 | the way Lucide icons are vendored into the console: in-code, never a |
| 472 | CDN. It installs `window.onerror` and `unhandledrejection` handlers that |
| 473 | post to `/__carlos/e` after §3.1 sanitisation, and exposes |
| 474 | `carlos.track(name, fields)` which posts to `/__carlos/a` via |
| 475 | `navigator.sendBeacon`. **It does not send pageviews in edge mode.** The |
| 476 | edge already counted the page when it served the HTML; a JS pageview |
| 477 | would double every number. What edge mode misses, stated on the tab: SPA |
| 478 | route changes, HTML served by a CDN in front of CARLOS, and back/forward |
| 479 | cache restores. The reserved `"count": "browser"` mode (§5) is the |
| 480 | answer for those apps, not a same-second dedup heuristic. Until then an |
| 481 | SPA may call `carlos.track("route", {path})` and see it as a named |
| 482 | event. Members include the snippet with one line: |
| 483 | |
| 484 | ```html |
| 485 | <script src="/__carlos/carlos.js" defer></script> |
| 486 | ``` |
| 487 | |
| 488 | ### 3.5 The server door |
| 489 | |
| 490 | **Verbs (decision).** `errors.Mount` and `analytics.Mount` follow |
| 491 | `schedule.Mount` exactly: same bearer check, same 0600 socket, same |
| 492 | `host` binding. |
| 493 | |
| 494 | ``` |
| 495 | POST /errors {kind, name, stack, path, status, fields, fp} |
| 496 | POST /events {name, fields} |
| 497 | ``` |
| 498 | |
| 499 | The app reads `CARLOS_CONTROL_SOCKET` and `CARLOS_ADMIN_TOKEN` from its |
| 500 | environment as it already does for schedules. A Rails app wires it in an |
| 501 | `ActionDispatch` error subscriber; a Go app in an `http.Handler` |
| 502 | middleware. The confused-deputy discussion in `purge.go` applies |
| 503 | unchanged: a same-uid sibling that can read the socket can post a fake |
| 504 | error, and the worst it can do is put a row on the owner's own tab. A |
| 505 | call on the control socket does not stamp `lastActive`: `ListenControl` |
| 506 | serves its own mux (wired at `backend_exec.go:624`) and never enters |
| 507 | `Service.serve`. The same 16 KB body cap, 8 KB stack cap and 600/min |
| 508 | per-app buckets as the browser door apply, counted into the same |
| 509 | `dropped` keys; the socket answers `202` on a drop and `400` only on a |
| 510 | body it cannot parse, since the caller here is the owner's own code and |
| 511 | can act on it. |
| 512 | |
| 513 | Unit-backed and DynamicUser tenants have no control socket today and so |
| 514 | have no server door; they have `log` and `edge` sources and the browser |
| 515 | door like everyone else. This is the same gap `purge.go` records for |
| 516 | schedules, and it closes when that one does. |
| 517 | |
| 518 | ### 3.6 The tabs, the API, the CLI |
| 519 | |
| 520 | **Errors tab** (`GET /a/{sqid}/apps/{app}/errors`, `handleAppErrors` |
| 521 | through `appPage(w, r, "errors")`, template `app_errors.html`, |
| 522 | `ActiveTab: "errors"`). Default window 24 h, selectable 1 h / 24 h / 7 d. |
| 523 | One row per fingerprint: source badge, kind, name, count in window, first |
| 524 | seen, last seen, hosts affected, releases seen. Rows sort by last seen. A |
| 525 | row expands in place to the stored sample: stack, path, status, fields, |
| 526 | and a link to Logs at that timestamp. Filters: source, level, host, |
| 527 | release. |
| 528 | Grouping happens on read over the window. `ReadWindow` fetches the whole |
| 529 | window and discards beyond `logsCollectCap` (50,000), which bounds |
| 530 | memory, not bucket reads and not totals (Codex 14): when the cap is hit |
| 531 | the tab says "showing the most recent 50,000 of a larger window" and the |
| 532 | counts are labelled as counts within what was read. A day rollup is not |
| 533 | built until a real app shows the read is too slow (§5). |
| 534 | |
| 535 | **Analytics tab** (`/apps/{app}/analytics`). Range today / 7 d / 30 d. |
| 536 | Headline: pageviews, visitors (daily, summed for ranges), bytes served, |
| 537 | bots excluded, dropped. One chart, per hour for today and per day |
| 538 | otherwise, drawn the way `templates/metrics.html:29` draws sparklines: |
| 539 | points computed in Go, an inline `<svg>` in the template, no library. |
| 540 | Three tables: top pages, top referers, named events. |
| 541 | |
| 542 | **API (decision, Fable 8).** Two `cliRoute` paths beside the logs read: |
| 543 | |
| 544 | ``` |
| 545 | GET /api/cli/apps/{account}/{app}/errors svcrole.Allow(Instance, Operate) |
| 546 | GET /api/cli/apps/{account}/{app}/analytics svcrole.Allow(Instance, Operate) |
| 547 | ``` |
| 548 | |
| 549 | Bearer-authenticated through `apiAccountApp` like `/logs`; an app token |
| 550 | may read its own app's errors and analytics. The session → CSRF → owner |
| 551 | (byte-identical 404) → fresh-auth ordering applies to the two *pages*, |
| 552 | which go through `appPage`; the API paths have no session and no CSRF by |
| 553 | construction. The tabs render from the same functions the API answers |
| 554 | with. |
| 555 | |
| 556 | **CLI.** `carlos errors -app X [-since 24h] [-source browser] [-level warn] |
| 557 | [-host h] [-release v] [-fp <id>]` lists groups, `-fp` expands one; |
| 558 | `carlos analytics -app X [-range 7d]` prints the headline, chart as a |
| 559 | text sparkline, and the three tables. Both get headings in |
| 560 | `docs/site/cli.md` in the same branch, or `internal/docsite` fails the |
| 561 | gate. |
| 562 | |
| 563 | **Settings.** The three keys of §3.3 on the app Settings tab and on the |
| 564 | account settings page, and via `carlos apps` and `carlos accounts`, |
| 565 | writing the two objects. Error *capture* has no switch: an app that is |
| 566 | failing is an operational fact the owner needs; `log_level` only widens |
| 567 | what the `log` source counts. A member may |
| 568 | delete an error group from the tab or the API (`DELETE |
| 569 | …/errors/{fp}`, owner on the page, `Operate` token on the API). This |
| 570 | writes a tombstone `apps/<sqid>/<app>/errors/tombstones/<fp>.json` |
| 571 | carrying the deletion time; the reader hides records of that `fp` with |
| 572 | `last` before that time, from both `errors/` and the log-derived rows, |
| 573 | until retention removes the chunks. A later occurrence reappears: this |
| 574 | is delete, not mute (§5). Logs themselves are untouched; deleting log |
| 575 | lines is a Logs feature and not part of this design. |
| 576 | |
| 577 | **Retention (decision).** Error chunks follow log retention, both age and |
| 578 | byte cap, in `logshipPrune`. Analytics day files are kept 400 days, |
| 579 | enough for year-over-year, and are small enough (tens of KB per app-day) |
| 580 | that no byte cap is needed; the same prune loop deletes past the age. |
| 581 | |
| 582 | **Coverage line.** Each tab carries one line saying which sources are |
| 583 | live for this app: "log, edge" for an app with nothing wired, "+browser" |
| 584 | once a beacon has arrived, "+server" once the socket verb has. A member |
| 585 | should never wonder whether silence means no errors or no wiring. |
| 586 | |
| 587 | ## 4. The consequence to be explicit about: nothing here wakes an instance |
| 588 | |
| 589 | Every path in §3 is checked against the scheduled-ticks rule: |
| 590 | |
| 591 | - `/__carlos/*` is answered between route lookup and the meter wrap, before |
| 592 | the cache and the activator. A beacon to a hibernating app returns `202` |
| 593 | from the edge; `beginRequest` is never called and the idle clock is not |
| 594 | stamped. The test reads `a.entries[host].lastActive` directly, as |
| 595 | `activator_test.go:1378` does. |
| 596 | - Analytics counting is a tap on responses the edge is already sending. |
| 597 | - The `log` source is a read over chunks the agent already ships. |
| 598 | - The server door is a control-socket verb, which does not stamp. |
| 599 | - Flushes ride the existing `usageLoop` and `logshipLoop` tickers in the |
| 600 | agent. |
| 601 | |
| 602 | The one new cost is bucket writes: one analytics day-file CAS per box per |
| 603 | five minutes per app with traffic, and error chunks at the cadence of log |
| 604 | chunks. Both are bounded by the caps in §3.2–3.3 regardless of traffic. |
| 605 | Memory on the edge is bounded per app: 500-counter summaries, one 2 KB |
| 606 | sketch, a 10,000-entry visitor bucket table, and the coalescing map |
| 607 | capped at 1,000 fingerprints. |
| 608 | |
| 609 | ## 5. What is not built |
| 610 | |
| 611 | - **Resolve and mute.** An owner will want to mark a group resolved or |
| 612 | mute a known one. It is the first triage feature to add and comes |
| 613 | before alerts; the tombstone in §3.6 is its seed. |
| 614 | - **Alerts.** "Email me when a new fingerprint appears" belongs with the |
| 615 | existing `alertwatch` machinery, not here. |
| 616 | - **Browser-counted mode** (`"count": "browser"`): the snippet sends |
| 617 | pageviews with SPA route support and prerender activation, and the edge |
| 618 | stops counting HTML. For apps behind a CDN or built as SPAs. |
| 619 | - **Event properties.** `fields` on analytics events are accepted and |
| 620 | dropped; property breakdowns change the day-file shape additively. |
| 621 | - **Error day rollups.** Grouping reads the window every time. Fine to |
| 622 | 50k records; a heavy app will tell us when it is not. |
| 623 | - **Forwarding.** A `forward: sentry|posthog|…` setting that relays |
| 624 | events to a rented service. The shape in §3.1 makes this a translator. |
| 625 | - **Source maps, breadcrumbs, session replay, funnels, GeoIP, per-country |
| 626 | tables.** Rent those. |
| 627 | - **Fleet-wide views.** Errors across a member's apps, or across the |
| 628 | fleet for ops, are a different page with different gates. |
| 629 | |
| 630 | ## 6. Testing |
| 631 | |
| 632 | - **Sanitisation:** table test over messages, stacks, fields and paths |
| 633 | containing emails, bearer tokens, `password=`, 32-char hex and base64 |
| 634 | runs; each is replaced and nothing else is touched. |
| 635 | - **Fingerprint:** table test of `norm`: digits, hex, UUIDs and quoted |
| 636 | strings are placeholders; `v2`, `E404`, `HTTP 503` survive; frames lose |
| 637 | line, column and bundle hash; a caller-supplied `fp` is taken verbatim |
| 638 | and capped; `fpv` is stamped. |
| 639 | - **Coalescing:** 10,000 identical errors in one interval produce one |
| 640 | record with `count: 10000` and `first < last`; the same `fp` on two |
| 641 | hosts produces two records; 1,001 distinct fingerprints for one app |
| 642 | produce 1,000 records and one `_overflow`; a second app on the same box |
| 643 | is unaffected. |
| 644 | - **Log-derived rows:** a `level=error` JSON line appears on the Errors |
| 645 | read and in Logs; `level=info`, an unparseable line, and a plain |
| 646 | `panic:` line do not appear as `log` errors. |
| 647 | - **Edge errors:** an upstream 500 on a POST (the plain path), a 502 from |
| 648 | a failed wake, and a 500 on a cacheable GET each produce exactly one |
| 649 | edge error; a 404 produces none. |
| 650 | - **Chunk family:** `logs/` chunks written and read after the refactor |
| 651 | are byte-identical to before; every existing applog test passes |
| 652 | unchanged; `errors/` round-trips the new record. |
| 653 | - **Store door:** a fleet box may write `apps/<sqid>/<app>/errors/x` and |
| 654 | `…/analytics/x` and may not write `apps/<sqid>/errors/x`, |
| 655 | `apps/<sqid>/<app>/errors`, or `apps/<sqid>/<app>/other/x`. |
| 656 | - **Observing writer:** status and `Content-Type` latch at `WriteHeader` |
| 657 | and at implicit 200; visible below gzip; installed with a nil meter; |
| 658 | byte counts unchanged for every existing usage test. |
| 659 | - **Pageview definition:** `httptest` through `Edge.ServeHTTP` with a fake |
| 660 | upstream: HTML 200 GET counts; JSON 200, HTML 404, HTML 302, 304, HEAD, |
| 661 | `Sec-Purpose: prefetch`, `/__carlos/carlos.js` do not; a cache hit |
| 662 | counts; `Sec-Fetch-Dest: empty` does not; a known bot UA counts only |
| 663 | into `bots`; a `status` route counts nothing; `count: off` counts |
| 664 | nothing. |
| 665 | - **Client IP:** verified hop takes the last XFF element; unverified hop |
| 666 | and direct take `RemoteAddr`; a client-set XFF on a direct request is |
| 667 | ignored. |
| 668 | - **Visitor hash:** same IP+UA on the same day is one visitor; a |
| 669 | different day is different; the salt survives a restart and never |
| 670 | appears in the day file; the salt file is removed on rotation. |
| 671 | - **HLL:** across ten random datasets of 1k, 10k and 100k, the estimate is |
| 672 | within 3 standard errors of truth; merge is commutative and idempotent; |
| 673 | the encoding round-trips with its version byte. |
| 674 | - **Misra–Gries:** with fewer than 500 keys the summary is exact; two |
| 675 | summaries merge and re-fold to 500 plus `_other` with the total |
| 676 | preserved. |
| 677 | - **Day-file CAS:** event-time bucketing places a flush at 10:00:02 for |
| 678 | events at 09:59 into `09`; a failed flush re-queues and the next |
| 679 | succeeds with the sum; two failures drop the older window and increment |
| 680 | `dropped.flush`; two boxes writing the same app-day converge to the sum |
| 681 | of pageviews and the union of sketches. |
| 682 | - **Browser door:** correct `Origin` counts; missing `Origin` with matching |
| 683 | `Referer` counts; foreign `Origin` with matching `Referer` does not; |
| 684 | `Sec-Fetch-Site: cross-site` does not; 17 KB body does not; the 601st |
| 685 | post in a minute does not; each drop increments the right `dropped` |
| 686 | key; every case answers `202`. A hibernating route receives a beacon |
| 687 | and `lastActive` is unchanged. On a path-routed hostname the beacon |
| 688 | lands on the app owning its `path` prefix. `__carlos` is refused as a |
| 689 | prefix-map key. |
| 690 | - **`/__carlos/` reservation:** a route whose upstream serves `/__carlos/x` |
| 691 | never sees the request; the loader revalidates and the hashed asset is |
| 692 | immutable. |
| 693 | - **Server door:** a bad bearer is refused; a good one lands a record; |
| 694 | `lastActive` is unchanged after the call; a 17 KB body answers `202` |
| 695 | and increments `dropped.e.size`; `count: off` drops `/events` and |
| 696 | still accepts `/errors`. |
| 697 | - **Seq idempotency:** re-merging a batch with an already-recorded `seq` |
| 698 | leaves the day file unchanged. |
| 699 | - **Tombstone:** after `DELETE`, the group is absent from the read and |
| 700 | the API; a new occurrence after the deletion time reappears; Logs |
| 701 | still shows the underlying lines. |
| 702 | - **Wake failure once:** a failed wake produces one edge error, not two. |
| 703 | - **Console gates:** both pages give a non-member the byte-identical 404 |
| 704 | the Logs tab gives; both API paths refuse a token without `Instance` or |
| 705 | `Operate`. |
| 706 | - **Settings resolution:** with no object at either level nothing is |
| 707 | counted and `log_level` is `error`; an account object with |
| 708 | `count: edge` turns counting on for every app; an app object with |
| 709 | `count: off` under that account turns it off for that app alone; |
| 710 | writing either and bumping the serial converges on the box within the |
| 711 | next pass; the browser door drops `/__carlos/a` and still accepts |
| 712 | `/__carlos/e` when off; `door: off` drops both; `log_level: warn` |
| 713 | brings warn lines into the Errors read and `-level error` filters |
| 714 | them back out. |
| 715 | - **Docsite:** the gate passes with both new CLI headings present and |
| 716 | fails with either removed. |
| 717 | |
| 718 | ## 7. Rollout |
| 719 | |
| 720 | Each step is a separate branch that ships value on its own and needs no |
| 721 | member action. Each is canaried on the flagship for a day against edge |
| 722 | latency, edge memory, bucket write count and the hibernation invariant |
| 723 | before promotion, in the ladder the fleet already runs. |
| 724 | |
| 725 | 1. **Errors from `log` and `edge` sources, tab, API, CLI.** Every app in |
| 726 | the fleet gets a populated Errors tab. Requires the chunk-family |
| 727 | refactor, the store-door allow-list, the observing writer (for edge |
| 728 | errors), and the coalescing sink. |
| 729 | 2. **Analytics counting at the edge, tab, API, CLI, the settings |
| 730 | objects at both levels, and the skill question.** An account that |
| 731 | turns it on gets pageviews and visitors for every app with no |
| 732 | JavaScript. Shares the observing writer with step 1, so it lands |
| 733 | second. The `getting-started` skill change is a companion branch in |
| 734 | the website repo, landed the same day. |
| 735 | 3. **Browser door and `carlos.js`.** Members opt in with one script tag; |
| 736 | JS errors and named events appear. |
| 737 | 4. **Server door on the control socket.** Exec-backed apps post |
| 738 | structured errors and events. |
| 739 | |
| 740 | Wire shapes are additive at every step: `logs/` is untouched, `errors/` |
| 741 | and `analytics/` are new prefixes, `telemetry.json` absent means off, and |
| 742 | a box on an older build writes none of them. A mixed-version fleet shows |
| 743 | the coverage line per host so a missing source is visible rather than |
| 744 | puzzling. |
| 745 | |
| 746 | Binaries roll as usual: promote → adopt → restart for the console; edge |
| 747 | and agent pick up on adoption. |
| 748 | |
| 749 | ## 8. Rulings (Paul, 2026-09-07) |
| 750 | |
| 751 | Four questions were open in the draft; Paul answered all four. |
| 752 | |
| 753 | 1. **Analytics is off by default (ruling).** Settable at the account |
| 754 | level so one switch covers every app, overridable per app, and asked |
| 755 | about in the `getting-started` skill. §3.3 carries the objects and |
| 756 | the resolution order; §7 step 2 carries the skill change. |
| 757 | 2. **The reserved prefix is `/__carlos/` (ruling).** Paul asked whether a |
| 758 | random per-app identifier would be better for unpredictability. The |
| 759 | recommendation is no, and the spec stays static: the path is not a |
| 760 | secret and cannot be one, because the snippet URL is in every page's |
| 761 | source and the beacon endpoint is public by construction (§3.4). |
| 762 | What bounds abuse is the Origin check, the caps, `dropped`, and the |
| 763 | `door` switch, none of which a random path would strengthen. A |
| 764 | random path would also cost the one-line snippet its one line and |
| 765 | would make the reservation rule ("this prefix is never proxied |
| 766 | upstream") per-app state instead of a constant. The double underscore |
| 767 | buys the collision safety Paul wanted: no app in this repo serves it, |
| 768 | and nothing conventional does. |
| 769 | 3. **Warn level is a setting, and level is always filterable (ruling).** |
| 770 | `errors.log_level` in §3.3 at either level, `error` by default; the |
| 771 | tab, API and CLI filter on level regardless (§3.2, §3.6). |
| 772 | 4. **The salt lives beside the registry DB, 0600 (Fable's |
| 773 | recommendation, taken).** §3.3 records both reviewers' positions: |
| 774 | Fable named the location and confirmed the agent owns it; Codex asked |
| 775 | for restart-stability and cross-box sharing, and the persisted file |
| 776 | gives the first while the home-box rule makes the second unnecessary. |
| 777 | |
| 778 | The `platform-infrastructure` collision check for `/__carlos/` is still |
| 779 | worth one grep before step 3 ships; the double underscore makes a hit |
| 780 | unlikely enough that it is a checklist line, not an open question. |
| 781 | |
| 782 | ## 9. Review round 1 |
| 783 | |
| 784 | Both reviewers read the first draft against `main` and returned |
| 785 | needs-rework. What each changed: |
| 786 | |
| 787 | **Fable (second context).** Moved the beacon intercept from "before |
| 788 | routing" to between `lookupUpstreamPath` and the meter wrap, with the |
| 789 | backhaul argument (§3.4). Replaced the recorder tap with the observing |
| 790 | `countingWriter`, citing the three response paths that bypass the |
| 791 | recorder (§3.3). Found that fleet boxes' writes would be refused by |
| 792 | `aclWriteAppLogs` (§3.2). Corrected the §1 citations, which pointed at |
| 793 | ACME and attribution comments. Defined `clientIP` from the verified-hop |
| 794 | chain (§3.3). Listed the real refactor scope and the key-shape consumer |
| 795 | list, and corrected `retentionPass` to `logshipPrune` (§3.2, §3.6). |
| 796 | Argued the log mirror away in favour of a read-side filter (§3.2). Named |
| 797 | the `cliRoute` API pattern and the gate-ordering scope (§3.6). Found that |
| 798 | no box reads `features.json`, so the switch became a bucket object |
| 799 | (§3.3). Sanitised page paths (§3.3). Corrected the chart precedent, the |
| 800 | Logs-tab grep, and the non-existent test hook (§3.6, §4). Noted the |
| 801 | 500-key merge overflow (§3.3). |
| 802 | |
| 803 | **Codex.** Made the visitor salt restart-stable and the multi-day |
| 804 | visitor figure honest (§3.3). Reframed the privacy claim as the |
| 805 | deployment's assessment (§8). Tightened the pageview definition to 200 |
| 806 | only, excluding redirects, 304s, HEAD and prefetch (§3.3). Reserved a |
| 807 | browser-counted mode instead of a same-second dedup (§3.4, §5). Named the |
| 808 | path-routed attribution problem independently of Fable (§3.4). Made |
| 809 | fingerprints token-boundary, versioned, and frame-normalised (§3.2). Made |
| 810 | the coalesced record carry `first`, `last`, host and release (§3.2). Made |
| 811 | caps per app and exposed `dropped` everywhere something is dropped (§3.2, |
| 812 | §3.3, §3.4). Replaced first-admitted maps with Misra–Gries and stated the |
| 813 | HLL error correctly (§3.3). Tightened Origin matching and kept the |
| 814 | uniform 202 only alongside owner-visible drop counts (§3.4). Added |
| 815 | sanitisation before buffering and owner-triggered deletion (§3.1, §3.6). |
| 816 | Made day-file buckets event-time and flushes re-queue on failure (§3.3). |
| 817 | Added the collect-cap disclosure and the coverage line (§3.6). Split the |
| 818 | snippet into a revalidating loader and a hashed immutable asset (§3.4). |
| 819 | Put resolve/mute ahead of alerts in §5. Noted that a plain `panic:` line |
| 820 | is not a parsed error, which §3.2 now answers through the `edge` source. |
| 821 | |
| 822 | Not taken: a per-app public key for the browser door (identification |
| 823 | without truth, and the caps already bound the damage; §3.4 says why); |
| 824 | storing the raw pageview path (the token-collapsing rule keeps real page |
| 825 | names and drops secrets); a mirror of `log` errors into `errors/` (read |
| 826 | derivation is cheaper and loses nothing). |
| 827 | |
| 828 | ## 10. Review round 2 |
| 829 | |
| 830 | Both reviewers re-read the rewrite. Fable: approve-with-amendments, all |
| 831 | three blockers confirmed resolved against the code; amendments taken: |
| 832 | the Origin check is a `lookupIn` on the edge's map, not an alias list |
| 833 | (`upstream.alias` is a bool); `Sec-Fetch-Dest: empty` is a fetch, not a |
| 834 | page; status routes are excluded from the tap; joined `X-Forwarded-For` |
| 835 | lines; salt file 0600. Codex: needs-rework on under-specification; |
| 836 | taken: 1–3 digit tokens survive normalisation so `HTTP 503` does; the |
| 837 | snippet stamps `path` on every beacon so path-routed events attribute; |
| 838 | the coalescing cap counts records and the window rule is stated; a |
| 839 | `seq` per box makes CAS retries idempotent; a `door` switch closes the |
| 840 | browser door independently of counting and `count: off` also drops the |
| 841 | socket's `/events`; the server door gets the same size and rate limits; |
| 842 | Misra–Gries tables are labelled approximate above 500 keys; the |
| 843 | `dropped` schema lists every key it uses, split by errors and events; |
| 844 | wake failures are suppressed at the writer when the activator already |
| 845 | emitted; tombstones have a scope, an authorisation, and a stated |
| 846 | non-effect on Logs. Nothing from round 2 was declined. |
| 847 | |
| 848 | ## 11. Amendment 2026-09-24: Availability |
| 849 | |
| 850 | **Why.** A day of the flagship's Errors tab (titogo, 2026-09-23) held 29 |
| 851 | groups: 25 edge 502s, 3 edge 503s and one 500. Only the 500 was a failure |
| 852 | of what a request asked for. The rest said the instance was down, |
| 853 | restarting or unreachable when some page was asked for, and §3.2's |
| 854 | `"<status> <method> <path>"` fingerprint split each outage into one row |
| 855 | per page that happened to hit it. 17 of the 29 rows were one scanner |
| 856 | probing `/.env`, `/phpinfo.php` and so on against a dead canary. The |
| 857 | record also could not say *why* the edge had no answer: the proxies |
| 858 | logged the error and threw it away. |
| 859 | |
| 860 | **Availability is the Errors tab's sibling (decision, Bill 2026-09-24).** |
| 861 | A sub-tab between Errors and Analytics (`GET /a/{sqid}/apps/{app}/availability`), |
| 862 | an API path beside `/errors` with the same gate |
| 863 | (`GET /api/cli/apps/{account}/{app}/availability`), and `carlos |
| 864 | availability`. It holds: |
| 865 | |
| 866 | - every edge-observed 502, 503 and 504, whoever wrote it; and |
| 867 | - the platform stream's events at `warn` and above (failed wakes, |
| 868 | exits, a sidecar that will not restart), whatever `errors.log_level` |
| 869 | says, since an exit on a signal is filed at `warn` and is exactly the |
| 870 | event an outage row needs beside it. The synthetic rate-cap line |
| 871 | (`applog.IsDropNotice`) is about the logging budget and is left out. |
| 872 | |
| 873 | Errors keeps the rest: the app's own 500, 501 and 505+, and the `log` |
| 874 | source's app-stream lines. Nothing is in both. |
| 875 | |
| 876 | **Grouped by cause, never by path (decision).** An edge 502/503/504 is |
| 877 | filed as `kind: unavailable` with `cause` and `name: "<status> · <cause |
| 878 | in the member's terms>"`, and so fingerprinted by status and cause alone; |
| 879 | the path stays on the record as the occurrence's sample. The cause comes |
| 880 | from the ErrorHandler that wrote the status, through a per-request |
| 881 | `outcome` on the context (`edge/answered.go`, which already carried the |
| 882 | activator's answered flag and now rides every observed request, not only |
| 883 | hibernating ones): |
| 884 | |
| 885 | | cause | where | |
| 886 | |---------------|--------------------------------------------------------------| |
| 887 | | `refused` | ECONNREFUSED / ENOENT on the dial: nothing listening | |
| 888 | | `reset` | EOF, unexpected EOF, ECONNRESET, EPIPE: hung up mid-request | |
| 889 | | `timeout` | a deadline or a net timeout | |
| 890 | | `abandoned` | the client left after waiting 5 s or more (below) | |
| 891 | | `no_upstream` | `multiProxy`: every upstream failing its probe | |
| 892 | | `fleet_dark` | `fleetProxy`: no box of the fleet connected | |
| 893 | | `tunnel` | any other failure through a fleet box's tunnel | |
| 894 | | `static` | a static site's files could not be resolved | |
| 895 | | `proxy` | any other proxy failure | |
| 896 | | `upstream` | no handler noted one: the app wrote the status itself (direct routes only; below) | |
| 897 | |
| 898 | `unixProxy`, `tcpProxy` and `tunnelProxy` had no ErrorHandler; they get |
| 899 | one that is `httputil.ReverseProxy`'s default (same log line, same bare |
| 900 | 502) plus the note. |
| 901 | |
| 902 | **A visitor who left quickly is not an outage; one kept waiting is |
| 903 | (decision, amended in review).** When the client's own context is done by |
| 904 | the time a proxy fails, `ReverseProxy` still writes a 502, to nobody, and |
| 905 | the error is the cancellation whatever the instance was doing. Within |
| 906 | 5 s of the request (`clientGoneGrace`) the edge notes `client_gone` and |
| 907 | files nothing: a visitor navigating away mid-load is not the instance's |
| 908 | fault, and a single-count 502 on a font file is the typical shape of it. |
| 909 | Past 5 s it files `abandoned`. The review caught why that half matters: |
| 910 | a dead unix socket is redialled for 13 s and a wedged instance is never |
| 911 | timed out at all, so a monitor or visitor giving up first is how those |
| 912 | outages end — skipping them would hide exactly the worst ones. |
| 913 | |
| 914 | **Behind a tunnel, the front box files only the hop (decision, Fable |
| 915 | review; corrected in its follow-up).** When a box proxies a request over |
| 916 | a reverse channel, the upstream is not the app but another box's edge — |
| 917 | the agent's, the only tunnel client — which files what it sees serving |
| 918 | the app, and deliberately files nothing for a wake 503 its own activator |
| 919 | answered. A 5xx that merely passed back through the front box was |
| 920 | therefore filed twice, and under this amendment would have been labelled |
| 921 | "the app answered it itself" for an app that never saw the request. The |
| 922 | front box now files only failures it noted itself (`tunnel`, |
| 923 | `fleet_dark`, `abandoned`); everything else is the serving box's record. |
| 924 | This also ends the double filing of tunnelled 500s, which predates the |
| 925 | amendment. |
| 926 | |
| 927 | The rule keys on how THIS box served the request (`upstream.viaTunnel`, |
| 928 | the build-time decision), never on the route row: the fleet box that |
| 929 | serves the app carries the same `fleet/<name>` placement on its own row, |
| 930 | has no hub, and serves from the socket — it must file everything. (The |
| 931 | first cut keyed on the row, and a fleet-placed app's 500s were filed |
| 932 | nowhere; `TestEdgeServingFleetBoxFilesItsOwn` pins the far end.) Two |
| 933 | edges remain: a far agent whose status store failed to open has no error |
| 934 | sink and files nothing, which the front box no longer covers; and a |
| 935 | client that leaves past the grace can be filed `abandoned` on both boxes, |
| 936 | since the stream reset reaches the far box's request too. |
| 937 | |
| 938 | **The overflow row is per family (decision, from review).** The sink's |
| 939 | hourly `_overflow` row copied its source, kind and status from the first |
| 940 | record that overflowed, so one family's excess could land on the other |
| 941 | tab. It is now one row per family (`IsAvailability`), and |
| 942 | `AsAvailability` leaves an `_overflow` row alone. |
| 943 | |
| 944 | **History regroups on read (decision).** Records filed before this carry |
| 945 | `kind: http`, no `cause`, and the per-path fingerprint. The read |
| 946 | recognises them by status and regroups them as `"<status> · cause not |
| 947 | recorded"` (`apperr.AsAvailability`); the FP filter applies after the |
| 948 | regroup so the row the tab shows expands. The bucket is not rewritten. |
| 949 | |
| 950 | **Not changed.** The wire shape is additive (`cause` and a read-only |
| 951 | `stream` on `apperr.Record`, both `omitempty`); nothing under `logs/` or |
| 952 | `errors/` moves. The sending half of a backhaul still files nothing when |
| 953 | its hop fails: it has no route row and so no app to file under, the |
| 954 | same reason it bills nothing. Correlating a row with the nearest deploy or restart |
| 955 | on that host, and alerting on a new cause, are the natural next steps |
| 956 | and are not built here. |
| 957 | |
amadan