plan: Errors step 1, log and edge sources
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+3287
−0
docs/superpowers/plans/2026-09-07-errors-step1-log-and-edge.md+3287 −0
diff --git a/docs/superpowers/plans/2026-09-07-errors-step1-log-and-edge.md b/docs/superpowers/plans/2026-09-07-errors-step1-log-and-edge.md| new file mode 100644 |
| index 000000000..c8ec902d9 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-09-07-errors-step1-log-and-edge.md |
| @@ -0,0 +1,3287 @@ |
| +# Errors tab, step 1: log and edge sources — Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Every app in the fleet gets a populated Errors tab, API and CLI with zero integration: error-level log lines and edge-observed 5xx responses, grouped by fingerprint and coalesced on the box. |
| + |
| +**Architecture:** A new leaf package `internal/apperr` owns the error record, sanitisation, fingerprinting, the coalescing box sink, grouping, and the bucket read. It stores chunks in a sibling family of `logs/` (`apps/<sqid>/<app>/errors/<box>/<hour>/…jsonl.gz`) through family-parameterised helpers added to `internal/applog`. The edge's `countingWriter` learns to observe status and emits an edge error for upstream 5xx; the agent ships the sink like it ships logs. The console reads two families (`errors/` plus `logs/` filtered to a level from a new `internal/telemetry` settings object) and serves a tab, a `cliRoute` API and `carlos errors`. |
| + |
| +**Tech Stack:** Go 1.22+ (`net/http` patterns, generics), `html/template`, sqlite registry, `release.Store` bucket, no JS framework. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md` — §3.1 (shape, sanitisation), §3.2 (log source read-derived, edge errors, fingerprint, coalescing, storage, store door), §3.3 (observing writer, settings object), §3.6 (tab, API, CLI, settings, retention), §4, §6, §7 step 1, §8 rulings. |
| + |
| +## Global Constraints |
| + |
| +- Wire shapes are additive: `logs/` chunks must be byte-identical after the refactor; existing `internal/applog` tests pass unchanged. |
| +- Edge wrapper order is load-bearing: recorder above compressor, counter below; nothing moves, one wrapper learns to look (`internal/edge/edge.go:1241`). |
| +- Nothing here may enter `activator.Service.ServeHTTP` or stamp `lastActive`. |
| +- Store door write grant becomes a family allow-list: `logs`, `errors`, `analytics` (spec §3.2), each segment-exact. |
| +- Coalescing cap: 1,000 records per app per UTC hour counting `(fp, host, release)`; overflow folds into one `_overflow` row. |
| +- Fingerprint: `sha256(len-delimited(source, kind, norm(name), norm(top frame)))[:16]` hex, `fpv: 1`; 1–3 digit tokens survive normalisation, 4+ digits, hex 8+, UUIDs and quoted strings become placeholders. |
| +- Sanitisation before buffering on every source: query and fragment off paths; emails → `<email>`; `Bearer`/`Basic` credentials and `token=`/`key=`/`secret=`/`password=` values → `<redacted>`; hex or base64 runs of 32+ → `<hex>`. |
| +- Defaults (ruling): `errors.log_level` = `error`. Settings object: `accounts/<sqid>/telemetry.json` and `apps/<sqid>/<app>/telemetry.json`, resolved app → account → default. |
| +- API: `GET /api/cli/apps/{account}/{app}/errors` with `svcrole.Allow(svcrole.Instance, svcrole.Operate)`; pages go through `appPage` (session → CSRF → owner → fresh auth). |
| +- New `carlos errors` needs a `### carlos errors` heading in `docs/site/cli.md` or `internal/docsite` fails the gate. |
| +- Go commands: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`; sandbox may need disabling on "read-only file system". Before push: `GOOS=darwin GOARCH=arm64 go build ./...` and `GOOS=windows GOARCH=amd64 go build ./...`. |
| +- Every commit: `go build ./... && go vet ./... && gofmt -l .` clean for the packages touched; full `go test ./... -count=1` before the final push. |
| +- Commit trailer on every commit: `Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>`. |
| +- Branch: `spec-errors-analytics` (already exists, carries the spec). Push to `origin` (the amadan forge). Never `gh`. |
| + |
| +## File structure |
| + |
| +| File | Responsibility | |
| +|---|---| |
| +| `internal/applog/chunk.go` | + `Family`, `FamilyPrefix/BoxPrefix/HourPrefix/ChunkKey`, generic `EncodeJSONL`/`DecodeJSONL`; existing funcs become thin wrappers | |
| +| `internal/applog/read.go` | + generic `ReadChunks[T]`; `ReadWindow` re-based on it | |
| +| `internal/edge/storedoor.go` | `aclWriteAppLogs` → `aclWriteAppFamily` allow-list | |
| +| `internal/apperr/record.go` | `Record`, source/kind constants, `FromLog` | |
| +| `internal/apperr/sanitize.go` | `Sanitize`, `SanitizePath` | |
| +| `internal/apperr/fingerprint.go` | `Normalize`, `TopFrame`, `Fingerprint`, `FPVersion` | |
| +| `internal/apperr/sink.go` | coalescing `Sink` | |
| +| `internal/apperr/read.go` | `Filter`, `ReadWindow`, chunk key helpers | |
| +| `internal/apperr/group.go` | `Group`, `GroupRecords` | |
| +| `internal/activator/answered.go` | `WithAnswerFlag`, `MarkAnswered` | |
| +| `internal/activator/activator.go` | mark the three wake-path 503s | |
| +| `internal/edge/usagemeter.go` | `countingWriter` observes status and content type | |
| +| `internal/edge/edge.go` | `ErrorSink`, `EnableErrors`, unconditional writer, edge error emission | |
| +| `internal/host/logship.go` | `logshipPrune` → `chunkPrune(fam, …)` | |
| +| `internal/host/errship.go` | error sink ship loop | |
| +| `internal/host/agent.go` | construct `apperr.Sink`, wire edge + loop | |
| +| `internal/telemetry/telemetry.go` | settings object, `Resolve`, `Update` | |
| +| `internal/console/errorsview.go` | window read merging both families; page fold | |
| +| `internal/console/apierrors.go` | `GET …/errors`, `PUT …/telemetry` | |
| +| `internal/console/server.go` | routes, page fields, `handleAppErrors`, settings handlers | |
| +| `internal/console/templates/app_errors.html`, `appnav.html`, `app_settings.html`, `account_settings.html` | tab, nav link, settings forms | |
| +| `errors_cmd.go`, `apimode.go`, `main.go` | `carlos errors` | |
| +| `docs/site/cli.md` | `### carlos errors` | |
| + |
| +--- |
| + |
| +### Task 1: applog chunk families and a generic codec |
| + |
| +**Files:** |
| +- Modify: `internal/applog/chunk.go` |
| +- Modify: `internal/applog/read.go` |
| +- Test: `internal/applog/chunk_test.go` (append), `internal/applog/read_test.go` (append) |
| + |
| +**Interfaces:** |
| +- Produces: `type Family string`; `FamilyLogs`, `FamilyErrors`, `FamilyAnalytics`; `FamilyPrefix(fam Family, acct, app string) string`; `FamilyBoxPrefix(fam, acct, app, box)`; `FamilyHourPrefix(fam, acct, app, box, hour)`; `FamilyChunkKey(fam, acct, app, box, hour string, seq uint64, size int) string`; `EncodeJSONL[T any](recs []T) ([]byte, error)`; `DecodeJSONL[T any](b []byte) ([]T, error)`; `ReadChunks[T any](ctx, s release.Store, prefix string, since, until time.Time, keep func(T) bool, collectCap int) (perBox [][]T, trimmed bool, err error)`; `HourNameOfDir(dir string) string` (exported rename of `hourNameOfDir`). |
| +- Existing `Prefix`, `BoxPrefix`, `HourPrefix`, `ChunkKey`, `EncodeChunk`, `DecodeChunk`, `ReadWindow` keep their signatures and behaviour. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +Append to `internal/applog/chunk_test.go`: |
| + |
| +```go |
| +func TestFamilyKeysMirrorLogs(t *testing.T) { |
| + if got, want := FamilyPrefix(FamilyLogs, "bab", "web"), Prefix("bab", "web"); got != want { |
| + t.Fatalf("FamilyPrefix(logs) = %q, Prefix = %q", got, want) |
| + } |
| + if got, want := FamilyChunkKey(FamilyLogs, "bab", "web", "box1", "2026-09-07-10", 42, 7), ChunkKey("bab", "web", "box1", "2026-09-07-10", 42, 7); got != want { |
| + t.Fatalf("FamilyChunkKey(logs) = %q, ChunkKey = %q", got, want) |
| + } |
| + got := FamilyChunkKey(FamilyErrors, "bab", "web", "box1", "2026-09-07-10", 42, 7) |
| + want := "apps/bab/web/errors/box1/2026-09-07-10/0000000000000042-7.jsonl.gz" |
| + if got != want { |
| + t.Fatalf("errors key = %q, want %q", got, want) |
| + } |
| + if n, ok := ChunkKeySize(got); !ok || n != 7 { |
| + t.Fatalf("ChunkKeySize on an errors key = %d,%v", n, ok) |
| + } |
| +} |
| + |
| +func TestEncodeJSONLRoundTripsAnyShape(t *testing.T) { |
| + type rec struct { |
| + A string `json:"a"` |
| + N int `json:"n,omitempty"` |
| + } |
| + b, err := EncodeJSONL([]rec{{A: "x", N: 1}, {A: "y"}}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + out, err := DecodeJSONL[rec](b) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(out) != 2 || out[0].A != "x" || out[0].N != 1 || out[1].A != "y" || out[1].N != 0 { |
| + t.Fatalf("round trip = %+v", out) |
| + } |
| + // EncodeChunk must be byte-identical to EncodeJSONL over the same |
| + // records: the logs family's wire shape does not move. |
| + recs := []Record{{TS: time.Date(2026, 9, 7, 10, 0, 0, 0, time.UTC), Msg: "hi", Seq: 1}} |
| + a, _ := EncodeChunk(recs) |
| + c, _ := EncodeJSONL(recs) |
| + da, _ := DecodeChunk(a) |
| + dc, _ := DecodeJSONL[Record](c) |
| + if len(da) != 1 || len(dc) != 1 || da[0].Msg != dc[0].Msg || !da[0].TS.Equal(dc[0].TS) { |
| + t.Fatalf("EncodeChunk and EncodeJSONL disagree: %+v vs %+v", da, dc) |
| + } |
| +} |
| +``` |
| + |
| +Append to `internal/applog/read_test.go` (use the same store construction the file already uses — `release.NewDirStore(t.TempDir())` — and its existing helper for writing a chunk if one exists; otherwise `s.Put(ctx, ChunkKey(...), body, "")`): |
| + |
| +```go |
| +func TestReadChunksWalksAnyFamily(t *testing.T) { |
| + ctx := context.Background() |
| + s := release.NewDirStore(t.TempDir()) |
| + type ev struct { |
| + TS time.Time `json:"ts"` |
| + Name string `json:"name"` |
| + } |
| + now := time.Date(2026, 9, 7, 10, 30, 0, 0, time.UTC) |
| + body, _ := EncodeJSONL([]ev{{TS: now, Name: "boom"}, {TS: now.Add(time.Minute), Name: "skip"}}) |
| + key := FamilyChunkKey(FamilyErrors, "bab", "web", "box1", HourOf(now), 1, len(body)) |
| + if _, err := s.Put(ctx, key, body, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + perBox, trimmed, err := ReadChunks[ev](ctx, s, FamilyPrefix(FamilyErrors, "bab", "web"), |
| + now.Add(-time.Hour), now.Add(time.Hour), func(e ev) bool { return e.Name != "skip" }, 100) |
| + if err != nil || trimmed { |
| + t.Fatalf("ReadChunks: %v trimmed=%v", err, trimmed) |
| + } |
| + if len(perBox) != 1 || len(perBox[0]) != 1 || perBox[0][0].Name != "boom" { |
| + t.Fatalf("perBox = %+v", perBox) |
| + } |
| + // An hour outside the window is never listed. |
| + perBox, _, err = ReadChunks[ev](ctx, s, FamilyPrefix(FamilyErrors, "bab", "web"), |
| + now.Add(2*time.Hour), time.Time{}, func(ev) bool { return true }, 100) |
| + if err != nil || len(perBox) != 1 || len(perBox[0]) != 0 { |
| + t.Fatalf("out-of-window read = %+v, %v", perBox, err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run tests to verify they fail** |
| + |
| +Run: `go test ./internal/applog/ -run 'TestFamilyKeys|TestEncodeJSONL|TestReadChunks' -count=1` |
| +Expected: FAIL to compile (`undefined: FamilyPrefix` etc.). |
| + |
| +- [ ] **Step 3: Implement families and the generic codec in `chunk.go`** |
| + |
| +Replace the `Prefix`/`BoxPrefix`/`HourPrefix`/`ChunkKey` block and the codec with: |
| + |
| +```go |
| +// Family names one chunk family under an app — the path segment after |
| +// apps/<sqid>/<app>/. logs/ is the original; errors/ and analytics/ are |
| +// the siblings the errors-and-analytics spec adds (§3.2). The key shape |
| +// below the family segment is identical for every family, which is what |
| +// lets one prune loop, one store-door grant and one reader serve all three. |
| +type Family string |
| + |
| +const ( |
| + FamilyLogs Family = "logs" |
| + FamilyErrors Family = "errors" |
| + FamilyAnalytics Family = "analytics" |
| +) |
| + |
| +// FamilyPrefix is the app's whole subtree for one family — under |
| +// release.AppPrefix on purpose (spec §3): it is the app's data, so the |
| +// owner gate maps onto the prefix and app trash/purge sweeps it for free. |
| +func FamilyPrefix(fam Family, acct, app string) string { |
| + return release.AppPrefix(acct, app) + string(fam) + "/" |
| +} |
| + |
| +// FamilyBoxPrefix is one box's subtree within a family — the unit of |
| +// write-side ownership: each box prunes only what it wrote. |
| +func FamilyBoxPrefix(fam Family, acct, app, box string) string { |
| + return FamilyPrefix(fam, acct, app) + box + "/" |
| +} |
| + |
| +// FamilyHourPrefix is one UTC hour within a box's subtree. |
| +func FamilyHourPrefix(fam Family, acct, app, box, hour string) string { |
| + return FamilyBoxPrefix(fam, acct, app, box) + hour + "/" |
| +} |
| + |
| +// FamilyChunkKey names one flushed chunk in any family. See ChunkKey for |
| +// why seq and size live in the key. |
| +func FamilyChunkKey(fam Family, acct, app, box, hour string, seq uint64, size int) string { |
| + return fmt.Sprintf("%s%016d-%d.jsonl.gz", FamilyHourPrefix(fam, acct, app, box, hour), seq, size) |
| +} |
| + |
| +// Prefix is the app's whole log subtree: FamilyPrefix for logs. |
| +func Prefix(acct, app string) string { return FamilyPrefix(FamilyLogs, acct, app) } |
| + |
| +// BoxPrefix is one box's subtree within an app's logs. |
| +func BoxPrefix(acct, app, box string) string { return FamilyBoxPrefix(FamilyLogs, acct, app, box) } |
| + |
| +// HourPrefix is one UTC hour within a box's log subtree. |
| +func HourPrefix(acct, app, box, hour string) string { |
| + return FamilyHourPrefix(FamilyLogs, acct, app, box, hour) |
| +} |
| + |
| +// ChunkKey names one flushed log chunk: seq is the first record's per-box |
| +// sequence number, zero-padded (16 digits — seq is milliseconds-based, see |
| +// NewSink) so lexical listing order is numeric order, and size is the |
| +// object's own byte length. Size lives IN the key on purpose: |
| +// release.Store's List returns keys only, and retention's byte cap needs |
| +// sizes — embedding them costs a suffix, extending the Store interface |
| +// costs every implementation. Chunks are written once and never appended |
| +// or rewritten (spec §3). |
| +func ChunkKey(acct, app, box, hour string, seq uint64, size int) string { |
| + return FamilyChunkKey(FamilyLogs, acct, app, box, hour, seq, size) |
| +} |
| +``` |
| + |
| +Keep `ChunkKeySize` as is. Replace `EncodeChunk`/`DecodeChunk` with: |
| + |
| +```go |
| +// EncodeJSONL renders any record shape as gzipped JSONL, one per line — |
| +// the chunk codec every family shares. |
| +func EncodeJSONL[T any](recs []T) ([]byte, error) { |
| + var buf bytes.Buffer |
| + zw := gzip.NewWriter(&buf) |
| + enc := json.NewEncoder(zw) |
| + for _, r := range recs { |
| + if err := enc.Encode(r); err != nil { |
| + return nil, err |
| + } |
| + } |
| + if err := zw.Close(); err != nil { |
| + return nil, err |
| + } |
| + return buf.Bytes(), nil |
| +} |
| + |
| +// DecodeJSONL parses a chunk, skipping malformed lines rather than failing |
| +// the whole object — one corrupt line must not cost the member the rest of |
| +// the hour. Corrupt gzip is an error: that is a broken object, not a line. |
| +func DecodeJSONL[T any](b []byte) ([]T, error) { |
| + zr, err := gzip.NewReader(bytes.NewReader(b)) |
| + if err != nil { |
| + return nil, err |
| + } |
| + defer zr.Close() |
| + var out []T |
| + sc := bufio.NewScanner(zr) |
| + sc.Buffer(nil, 1<<20) // lines are capped well below this at the tee |
| + for sc.Scan() { |
| + var r T |
| + if json.Unmarshal(sc.Bytes(), &r) != nil { |
| + continue |
| + } |
| + out = append(out, r) |
| + } |
| + if err := sc.Err(); err != nil && err != io.EOF { |
| + return out, err |
| + } |
| + return out, nil |
| +} |
| + |
| +// EncodeChunk renders log records: EncodeJSONL over Record. |
| +func EncodeChunk(recs []Record) ([]byte, error) { return EncodeJSONL(recs) } |
| + |
| +// DecodeChunk parses a log chunk: DecodeJSONL over Record. |
| +func DecodeChunk(b []byte) ([]Record, error) { return DecodeJSONL[Record](b) } |
| +``` |
| + |
| +- [ ] **Step 4: Implement `ReadChunks` in `read.go` and re-base `ReadWindow` on it** |
| + |
| +```go |
| +// ReadChunks walks every box's hour directories under prefix (a |
| +// FamilyPrefix) whose hours intersect [since, until], decodes each chunk as |
| +// T, keeps what passes keep, and returns one slice per box in listing |
| +// order. Hour pruning happens at the PREFIX level — an hour outside the |
| +// window is never listed, which is what makes a 1h read over 7d of history |
| +// cheap (spec §3). collectCap bounds memory per box: past it the OLDEST |
| +// records go and trimmed says so. Store errors fail the read — a partial |
| +// answer that looks whole is the one thing this must never produce. |
| +func ReadChunks[T any](ctx context.Context, s release.Store, prefix string, since, until time.Time, keep func(T) bool, collectCap int) ([][]T, bool, error) { |
| + boxDirs, err := s.ListDirs(ctx, prefix) |
| + if err != nil { |
| + return nil, false, err |
| + } |
| + trimmed := false |
| + perBox := make([][]T, 0, len(boxDirs)) |
| + for _, boxDir := range boxDirs { |
| + hourDirs, err := s.ListDirs(ctx, boxDir) |
| + if err != nil { |
| + return nil, false, err |
| + } |
| + var recs []T |
| + for _, hourDir := range hourDirs { // lexically sorted = chronological |
| + if !hourIntersects(HourNameOfDir(hourDir), since, until) { |
| + continue |
| + } |
| + keys, err := s.List(ctx, hourDir) |
| + if err != nil { |
| + return nil, false, err |
| + } |
| + for _, key := range keys { // lexically sorted = seq order |
| + body, _, err := s.Get(ctx, key) |
| + if err != nil { |
| + return nil, false, err |
| + } |
| + chunk, err := DecodeJSONL[T](body) |
| + if err != nil { |
| + return nil, false, err |
| + } |
| + for _, r := range chunk { |
| + if keep(r) { |
| + recs = append(recs, r) |
| + } |
| + } |
| + if over := len(recs) - collectCap; over > 0 { |
| + recs = recs[over:] |
| + trimmed = true |
| + } |
| + } |
| + } |
| + perBox = append(perBox, recs) |
| + } |
| + return perBox, trimmed, nil |
| +} |
| +``` |
| + |
| +Then `ReadWindow` becomes: |
| + |
| +```go |
| +func ReadWindow(ctx context.Context, s release.Store, acct, app string, f Filter, collectCap int) ([]Record, bool, error) { |
| + perBox, trimmed, err := ReadChunks[Record](ctx, s, Prefix(acct, app), f.Since, f.Until, f.Match, collectCap) |
| + if err != nil { |
| + return nil, false, err |
| + } |
| + merged := MergeRecords(perBox...) |
| + if over := len(merged) - collectCap; over > 0 { |
| + merged = merged[over:] |
| + trimmed = true |
| + } |
| + return merged, trimmed, nil |
| +} |
| +``` |
| + |
| +Rename `hourNameOfDir` to `HourNameOfDir` (exported; the host package's `hourNameOf` duplicate goes away in Task 6) and update its one caller. |
| + |
| +- [ ] **Step 5: Run the whole package** |
| + |
| +Run: `go test ./internal/applog/ -count=1` |
| +Expected: PASS, every pre-existing test included. |
| + |
| +- [ ] **Step 6: Commit** |
| + |
| +```bash |
| +git add internal/applog |
| +git commit -m "applog: chunk families and a generic JSONL codec, logs/ unchanged |
| + |
| +Prefix/ChunkKey/EncodeChunk/ReadWindow keep their signatures and bytes; |
| +FamilyPrefix, FamilyChunkKey, EncodeJSONL/DecodeJSONL and ReadChunks are |
| +the seams the errors/ family (spec 2026-09-07 §3.2) builds on. |
| + |
| +Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 2: Store door family allow-list |
| + |
| +**Files:** |
| +- Modify: `internal/edge/storedoor.go:503-518` (`aclWriteAppLogs`) |
| +- Test: `internal/edge/storedoor_test.go` (rows in `TestStoreDoorACLMatrix`) |
| + |
| +**Interfaces:** |
| +- Produces: `aclWriteAppFamily(sqid, key string) bool` replacing `aclWriteAppLogs`; `aclWrite` calls it. |
| + |
| +- [ ] **Step 1: Add matrix rows** |
| + |
| +In `TestStoreDoorACLMatrix`, after the existing logs rows (around line 138), add: |
| + |
| +```go |
| + // --- sibling chunk families (errors-and-analytics spec §3.2) --- |
| + {"apps/ab12/web/errors/box-a/2026090710/0000000000000001-42.jsonl.gz", opWrite, true, "own app's error chunk"}, |
| + {"apps/ab12/web/errors/box-a/2026090710/0000000000000001-42.jsonl.gz", opDelete, true, "errship prunes its own subtree hourly"}, |
| + {"apps/ab12/web/analytics/box-a/days/2026-09-07.json", opWrite, true, "own app's analytics day file"}, |
| + {"apps/zz99/web/errors/box-a/x.jsonl.gz", opWrite, false, "another account's error chunks"}, |
| + {"apps/ab12/errors/box-a/x.jsonl.gz", opWrite, false, "errors must sit under an app segment"}, |
| + {"apps/ab12/web/errors", opWrite, false, "the bare family segment is not a chunk"}, |
| + {"apps/ab12/web/errors/", opWrite, false, "nor with a slash"}, |
| + {"apps/ab12/web/other/box-a/x.jsonl.gz", opWrite, false, "the family list is exact: logs, errors, analytics"}, |
| + {"apps/ab12/web/channels/errors/stable.json", opWrite, false, "\"errors\" deeper in the key grants nothing"}, |
| + {"apps/ab12/web/errorsx/box-a/x.jsonl.gz", opWrite, false, "the segment is errors/, not a prefix of a longer name"}, |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify the new rows fail** |
| + |
| +Run: `go test ./internal/edge/ -run TestStoreDoorACLMatrix -count=1` |
| +Expected: FAIL on the `errors` and `analytics` write rows (refused today). |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +Replace `aclWriteAppLogs` with: |
| + |
| +```go |
| +// chunkFamilies is the allow-list of app subtrees a box may write: the |
| +// chunk and day-file families it produces about its own tenant's apps |
| +// (applog.Family). Exact segment names — "logs" is not "logsx". |
| +var chunkFamilies = map[string]bool{"logs": true, "errors": true, "analytics": true} |
| + |
| +// aclWriteAppFamily implements aclWrite's second scope: exactly |
| +// apps/<sqid>/<app>/<family>/<something>, with <app> a single non-empty |
| +// segment, <family> one of chunkFamilies, and at least one key component |
| +// below the family. Anything else — a shorter key, a second path shape, |
| +// a family name as the app segment — is refused, so the grant can never be |
| +// widened by a key that merely contains the word. |
| +func aclWriteAppFamily(sqid, key string) bool { |
| + if sqid == "" { |
| + return false |
| + } |
| + rest, ok := strings.CutPrefix(key, "apps/"+sqid+"/") |
| + if !ok { |
| + return false |
| + } |
| + app, tail, ok := strings.Cut(rest, "/") |
| + if !ok || app == "" { |
| + return false |
| + } |
| + fam, below, ok := strings.Cut(tail, "/") |
| + return ok && chunkFamilies[fam] && below != "" |
| +} |
| +``` |
| + |
| +Update `aclWrite` to call `aclWriteAppFamily`, and update the doc comment above `aclWrite` (the "logs scope is SEGMENT-EXACT" paragraph) to say "family scope" and name the three families. |
| + |
| +- [ ] **Step 4: Run the matrix and the package** |
| + |
| +Run: `go test ./internal/edge/ -run TestStoreDoor -count=1` |
| +Expected: PASS. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/edge/storedoor.go internal/edge/storedoor_test.go |
| +git commit -m "store door: the app write grant is a family allow-list (logs, errors, analytics) |
| + |
| +Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 3: `internal/apperr` record, sanitisation, fingerprint |
| + |
| +**Files:** |
| +- Create: `internal/apperr/record.go`, `internal/apperr/sanitize.go`, `internal/apperr/fingerprint.go` |
| +- Test: `internal/apperr/sanitize_test.go`, `internal/apperr/fingerprint_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + ```go |
| + const (SourceLog = "log"; SourceEdge = "edge"; SourceServer = "server"; SourceBrowser = "browser") |
| + const (KindHTTP = "http"; KindLog = "log") |
| + const FPVersion = 1 |
| + type Record struct { TS time.Time; Box, Host, Source, Kind, Name, Level, Path string; Status int; Release, Stack string; Fields map[string]string; FP string; FPV int; Count int; First, Last time.Time; Seq uint64 } |
| + func FromLog(r applog.Record) Record |
| + func Sanitize(s string) string |
| + func SanitizePath(p string) string |
| + func Normalize(s string) string |
| + func TopFrame(stack string) string |
| + func Fingerprint(source, kind, name, stack string) string |
| + ``` |
| + |
| +- [ ] **Step 1: Write failing tests** |
| + |
| +`internal/apperr/sanitize_test.go`: |
| + |
| +```go |
| +package apperr |
| + |
| +import "testing" |
| + |
| +func TestSanitize(t *testing.T) { |
| + cases := map[string]string{ |
| + "user paul@tito.io not found": "user <email> not found", |
| + "Authorization: Bearer abc.def-ghi": "Authorization: Bearer <redacted>", |
| + "basic dXNlcjpwYXNz failed": "basic <redacted> failed", |
| + "GET /x?token=abc123&y=1": "GET /x?token=<redacted>&y=1", |
| + `{"password": "hunter2", "n": 3}`: `{"password": "<redacted>", "n": 3}`, |
| + "api_key=sk-live-1234 ok": "api_key=<redacted> ok", |
| + "digest 0123456789abcdef0123456789abcdef done": "digest <hex> done", |
| + "b64 QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo0MTIzNA== ok": "b64 <hex> ok", |
| + "short 4123 stays": "short 4123 stays", |
| + "nothing here": "nothing here", |
| + } |
| + for in, want := range cases { |
| + if got := Sanitize(in); got != want { |
| + t.Errorf("Sanitize(%q) = %q, want %q", in, got, want) |
| + } |
| + } |
| +} |
| + |
| +func TestSanitizePath(t *testing.T) { |
| + cases := map[string]string{ |
| + "/reset/8f3a9c1e2b4d6f708a9b0c1d2e3f4a5b?x=1#frag": "/reset/:token", |
| + "/users/paul@tito.io/edit": "/users/:token/edit", |
| + "/items/550e8400-e29b-41d4-a716-446655440000": "/items/:token", |
| + "/s/QUJDREVGR0hJSktMTU5PUFFSU1RV": "/s/:token", |
| + "/blog/2024-review": "/blog/2024-review", |
| + "/api/v2/orders/42": "/api/v2/orders/42", |
| + "/": "/", |
| + "": "/", |
| + } |
| + for in, want := range cases { |
| + if got := SanitizePath(in); got != want { |
| + t.Errorf("SanitizePath(%q) = %q, want %q", in, got, want) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +`internal/apperr/fingerprint_test.go`: |
| + |
| +```go |
| +package apperr |
| + |
| +import "testing" |
| + |
| +func TestNormalize(t *testing.T) { |
| + cases := map[string]string{ |
| + "user 4123 not found": "user <n> not found", |
| + "user 9911 not found": "user <n> not found", |
| + "HTTP 503 from upstream": "HTTP 503 from upstream", |
| + "retry 3 failed": "retry 3 failed", |
| + "v2 handler panicked": "v2 handler panicked", |
| + "E404 for id 550e8400-e29b-41d4-a716-446655440000": "E404 for id <uuid>", |
| + `missing key "stripe_secret"`: `missing key "?"`, |
| + "trace 0123456789abcdef bad": "trace <hex> bad", |
| + "deadbeef cafe": "<hex> cafe", |
| + } |
| + for in, want := range cases { |
| + if got := Normalize(in); got != want { |
| + t.Errorf("Normalize(%q) = %q, want %q", in, got, want) |
| + } |
| + } |
| +} |
| + |
| +func TestTopFrame(t *testing.T) { |
| + cases := map[string]string{ |
| + "TypeError: x\n at render (https://app.example/assets/main.3f2a9c1b.js:12:34)\n at y": "at render (https://app.example/assets/main.js)", |
| + "\n app/models/user.rb:42:in `find'\n app/x.rb:1": "app/models/user.rb:in `find'", |
| + "": "", |
| + "single line only": "single line only", |
| + } |
| + for in, want := range cases { |
| + if got := TopFrame(in); got != want { |
| + t.Errorf("TopFrame(%q) = %q, want %q", in, got, want) |
| + } |
| + } |
| +} |
| + |
| +func TestFingerprint(t *testing.T) { |
| + a := Fingerprint(SourceLog, KindLog, "user 4123 not found", "") |
| + b := Fingerprint(SourceLog, KindLog, "user 9911 not found", "") |
| + c := Fingerprint(SourceLog, KindLog, "user gone", "") |
| + d := Fingerprint(SourceEdge, KindLog, "user 4123 not found", "") |
| + if a != b { |
| + t.Fatalf("digit runs must group: %s != %s", a, b) |
| + } |
| + if a == c || a == d { |
| + t.Fatalf("different names or sources must not group: %s %s %s", a, c, d) |
| + } |
| + if len(a) != 16 { |
| + t.Fatalf("fingerprint length %d, want 16 hex", len(a)) |
| + } |
| + // Length-delimited: ("ab","c") and ("a","bc") differ. |
| + if Fingerprint("ab", "c", "n", "") == Fingerprint("a", "bc", "n", "") { |
| + t.Fatal("components must be length-delimited") |
| + } |
| + // A redeploy's bundle hash and line numbers do not split a browser group. |
| + s1 := "TypeError: x\n at render (https://a/main.3f2a9c1b.js:12:34)" |
| + s2 := "TypeError: x\n at render (https://a/main.9b8c7d6e.js:99:1)" |
| + if Fingerprint(SourceBrowser, "TypeError", "x", s1) != Fingerprint(SourceBrowser, "TypeError", "x", s2) { |
| + t.Fatal("frame normalisation must drop bundle hash and line:col") |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** |
| + |
| +Run: `go test ./internal/apperr/ -count=1` |
| +Expected: FAIL to compile (package does not exist). |
| + |
| +- [ ] **Step 3: Write `record.go`** |
| + |
| +```go |
| +// Package apperr is the pure half of the Errors tab (spec |
| +// 2026-09-07-errors-and-analytics-design.md §3.1–3.2): the wire record, |
| +// sanitisation, fingerprinting, the box-side coalescing sink, the chunk |
| +// read and the grouping fold. No I/O and no clock of its own — the same |
| +// split internal/applog keeps: internal/host ships, internal/edge and the |
| +// activator produce, internal/console reads. |
| +package apperr |
| + |
| +import ( |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/applog" |
| +) |
| + |
| +// Sources an error can come from (spec §3.1). log is never stored in the |
| +// errors/ family — it is derived on read from logs/ (§3.2). |
| +const ( |
| + SourceLog = "log" |
| + SourceEdge = "edge" |
| + SourceServer = "server" |
| + SourceBrowser = "browser" |
| +) |
| + |
| +// Kinds the platform itself produces. A server or browser caller sets its |
| +// own kind (an exception class, typically). |
| +const ( |
| + KindHTTP = "http" // an edge-observed 5xx |
| + KindLog = "log" // a log line at or above the configured level |
| +) |
| + |
| +// FPVersion stamps the fingerprint algorithm so it can change without |
| +// merging old groups into new ones. |
| +const FPVersion = 1 |
| + |
| +// Record is one coalesced error on the wire — one JSONL line inside an |
| +// errors/ chunk. Additive shape (AGENTS.md rule): every field but TS, |
| +// Source, Name and FP is omitempty, unknown keys are ignored on read. |
| +// Count, First and Last are the coalescing state: one record stands for |
| +// Count occurrences of FP on Host under Release between First and Last; |
| +// the other fields are the first occurrence's sample. |
| +type Record struct { |
| + TS time.Time `json:"ts"` |
| + Box string `json:"box,omitempty"` |
| + Host string `json:"host,omitempty"` |
| + Source string `json:"source"` |
| + Kind string `json:"kind,omitempty"` |
| + Name string `json:"name"` |
| + Level string `json:"level,omitempty"` // log source: warn | error |
| + Path string `json:"path,omitempty"` |
| + Status int `json:"status,omitempty"` |
| + Release string `json:"release,omitempty"` |
| + Stack string `json:"stack,omitempty"` |
| + Fields map[string]string `json:"fields,omitempty"` |
| + FP string `json:"fp"` |
| + FPV int `json:"fpv,omitempty"` |
| + Count int `json:"count,omitempty"` |
| + First time.Time `json:"first,omitempty"` |
| + Last time.Time `json:"last,omitempty"` |
| + Seq uint64 `json:"seq,omitempty"` |
| +} |
| + |
| +// MaxStack bounds a stored stack (spec §3.1). |
| +const MaxStack = 8 << 10 |
| + |
| +// FromLog derives the log-source record for one applog line (spec §3.2: |
| +// read-derived, never stored twice). Fingerprinted on the normalised |
| +// message alone — a log line has no frame. |
| +func FromLog(r applog.Record) Record { |
| + name := Sanitize(r.Msg) |
| + return Record{ |
| + TS: r.TS, Box: r.Box, Host: r.Host, |
| + Source: SourceLog, Kind: KindLog, Name: name, Level: r.Level, |
| + Fields: r.Fields, |
| + FP: Fingerprint(SourceLog, KindLog, name, ""), FPV: FPVersion, |
| + Count: 1, First: r.TS, Last: r.TS, Seq: r.Seq, |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Write `sanitize.go`** |
| + |
| +```go |
| +package apperr |
| + |
| +import ( |
| + "regexp" |
| + "strings" |
| +) |
| + |
| +// Sanitisation is a floor, not a guarantee (spec §3.1): the patterns below |
| +// catch what secrets look like in practice — addresses, bearer tokens, |
| +// key=value credentials, long opaque runs — before anything is buffered. |
| +var ( |
| + reEmail = regexp.MustCompile(`[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}`) |
| + reAuth = regexp.MustCompile(`(?i)\b(bearer|basic)\s+[A-Za-z0-9._~+/=-]+`) |
| + // key=value and "key": "value" shapes, value up to the next delimiter. |
| + reKV = regexp.MustCompile(`(?i)\b(token|key|secret|password|passwd|pwd|api_key|apikey|access_token|auth)("?\s*[=:]\s*"?)[^\s&"',;]+`) |
| + // A hex or base64 run of 32+ characters: a digest, a session id, a key. |
| + reLong = regexp.MustCompile(`\b[A-Za-z0-9+/_-]{32,}={0,2}`) |
| +) |
| + |
| +// Sanitize redacts secrets in free text: messages, stacks, field values. |
| +func Sanitize(s string) string { |
| + s = reEmail.ReplaceAllString(s, "<email>") |
| + s = reAuth.ReplaceAllStringFunc(s, func(m string) string { |
| + word := m[:strings.IndexAny(m, " \t")] |
| + return word + " <redacted>" |
| + }) |
| + s = reKV.ReplaceAllString(s, "${1}${2}<redacted>") |
| + s = reLong.ReplaceAllString(s, "<hex>") |
| + return s |
| +} |
| + |
| +var ( |
| + reUUID = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`) |
| + reHex16 = regexp.MustCompile(`^[0-9a-fA-F]{16,}$`) |
| + reB64ish = regexp.MustCompile(`^[A-Za-z0-9+/_-]{20,}={0,2}$`) |
| + reEmailNW = regexp.MustCompile(`^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$`) |
| +) |
| + |
| +// SanitizePath strips query and fragment and collapses token-shaped |
| +// segments to ":token" (spec §3.3): a UUID, a hex run of 16+, a base64- |
| +// looking run of 20+, or an email. "/blog/2024-review" and "/orders/42" |
| +// survive; "/reset/8f3a…" does not. Empty is "/". |
| +func SanitizePath(p string) string { |
| + if i := strings.IndexAny(p, "?#"); i >= 0 { |
| + p = p[:i] |
| + } |
| + if p == "" { |
| + return "/" |
| + } |
| + segs := strings.Split(p, "/") |
| + for i, s := range segs { |
| + if s == "" { |
| + continue |
| + } |
| + if reUUID.MatchString(s) || reHex16.MatchString(s) || reEmailNW.MatchString(s) || |
| + (reB64ish.MatchString(s) && strings.ContainsAny(s, "0123456789") && strings.ContainsAny(s, "ABCDEFGHIJKLMNOPQRSTUVWXYZ")) { |
| + segs[i] = ":token" |
| + } |
| + } |
| + return strings.Join(segs, "/") |
| +} |
| +``` |
| + |
| +The base64 segment rule requires both a digit and an upper-case letter so that ordinary long slugs (`some-long-page-name-here`) survive; the tests pin `/s/QUJD…` as a token and `/blog/2024-review` as not. |
| + |
| +- [ ] **Step 5: Write `fingerprint.go`** |
| + |
| +```go |
| +package apperr |
| + |
| +import ( |
| + "crypto/sha256" |
| + "encoding/hex" |
| + "regexp" |
| + "strconv" |
| + "strings" |
| +) |
| + |
| +var ( |
| + reQuoted = regexp.MustCompile(`"[^"]*"|'[^']*'`) |
| + reUUIDAny = regexp.MustCompile(`\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b`) |
| + reHexRun = regexp.MustCompile(`\b[0-9a-fA-F]{8,}\b`) |
| + reDigits4 = regexp.MustCompile(`\b\d{4,}\b`) |
| + reLineCol = regexp.MustCompile(`:\d+(:\d+)?`) |
| + reBundleJS = regexp.MustCompile(`\.[0-9a-fA-F]{8,}\.js`) |
| +) |
| + |
| +// Normalize folds the parts of a message that vary per occurrence into |
| +// placeholders on token boundaries (spec §3.2): quoted strings, UUIDs, hex |
| +// runs of 8+, digit runs of 4+. Tokens of 1–3 digits and mixed tokens |
| +// (v2, E404) survive, so "HTTP 503" and "retry 3" still separate groups |
| +// while "user 4123" and "user 9911" do not. Small ids under 1,000 will |
| +// split groups; that is the stated trade-off, and a caller-supplied FP |
| +// overrides it. |
| +func Normalize(s string) string { |
| + s = reQuoted.ReplaceAllString(s, `"?"`) |
| + s = reUUIDAny.ReplaceAllString(s, "<uuid>") |
| + s = reHexRun.ReplaceAllStringFunc(s, func(m string) string { |
| + if _, err := strconv.ParseUint(m, 10, 64); err == nil { |
| + return "<n>" // all digits: a digit run, not a hex run |
| + } |
| + return "<hex>" |
| + }) |
| + s = reDigits4.ReplaceAllString(s, "<n>") |
| + return s |
| +} |
| + |
| +// TopFrame is the first non-empty line of a stack with its line, column |
| +// and bundle hash removed, so a redeploy does not split every group. |
| +func TopFrame(stack string) string { |
| + for _, line := range strings.Split(stack, "\n") { |
| + line = strings.TrimSpace(line) |
| + if line == "" { |
| + continue |
| + } |
| + line = reBundleJS.ReplaceAllString(line, ".js") |
| + line = reLineCol.ReplaceAllString(line, "") |
| + return line |
| + } |
| + return "" |
| +} |
| + |
| +// Fingerprint groups occurrences: sha256 over length-delimited (source, |
| +// kind, Normalize(name), Normalize(TopFrame(stack)))[:16] as hex. For a |
| +// stack with a message line first (JS "TypeError: x\n at …"), the top |
| +// frame is the second line; callers pass the message as name and the |
| +// whole stack here. |
| +func Fingerprint(source, kind, name, stack string) string { |
| + h := sha256.New() |
| + frame := TopFrame(stack) |
| + if strings.HasPrefix(frame, name) || strings.Contains(frame, ": "+name) { |
| + // The stack's first line restates the message; take the next. |
| + rest := strings.TrimSpace(stack) |
| + if i := strings.IndexByte(rest, '\n'); i >= 0 { |
| + frame = TopFrame(rest[i+1:]) |
| + } |
| + } |
| + for _, part := range []string{source, kind, Normalize(name), Normalize(frame)} { |
| + h.Write([]byte(strconv.Itoa(len(part)))) |
| + h.Write([]byte{':'}) |
| + h.Write([]byte(part)) |
| + } |
| + return hex.EncodeToString(h.Sum(nil))[:16] |
| +} |
| +``` |
| + |
| +Note for `TestTopFrame`'s first case: the expected value is the message line (`TypeError: x`) is NOT the frame — adjust the test's first case to expect `TypeError: x` from `TopFrame` (it is the first non-empty line), and rely on `TestFingerprint`'s last assertion for the message-skipping behaviour. Concretely, change that case's expected string to `"TypeError: x"`. |
| + |
| +- [ ] **Step 6: Run tests, iterate until green** |
| + |
| +Run: `go test ./internal/apperr/ -count=1 -v` |
| +Expected: PASS. If a regex case disagrees with a test, fix the regex, not the test, unless the test contradicts the spec's stated rule. |
| + |
| +- [ ] **Step 7: Commit** |
| + |
| +```bash |
| +git add internal/apperr |
| +git commit -m "apperr: error record, sanitisation and fingerprinting (spec §3.1–3.2) |
| + |
| +Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 4: `internal/apperr` coalescing sink and bucket read |
| + |
| +**Files:** |
| +- Create: `internal/apperr/sink.go`, `internal/apperr/read.go` |
| +- Test: `internal/apperr/sink_test.go`, `internal/apperr/read_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + ```go |
| + const MaxGroupsPerAppHour = 1000 |
| + const OverflowFP = "_overflow" |
| + func NewSink(box string, now func() time.Time) *Sink |
| + func (s *Sink) Add(acct, app string, r Record) // sanitises, fingerprints if FP == "", coalesces |
| + func (s *Sink) Drain() map[applog.AppKey][]Record |
| + func (s *Sink) Known() []applog.AppKey |
| + func Prefix(acct, app string) string // applog.FamilyPrefix(FamilyErrors, …) |
| + func ChunkKey(acct, app, box, hour string, seq uint64, size int) string |
| + func EncodeChunk(recs []Record) ([]byte, error) |
| + func DecodeChunk(b []byte) ([]Record, error) |
| + type Filter struct { Since, Until time.Time; Host, Source, Release, FP string } |
| + func (f Filter) Match(r Record) bool // [First,Last] overlaps [Since,Until] |
| + func ReadWindow(ctx, s release.Store, acct, app string, f Filter, collectCap int) ([]Record, bool, error) |
| + func MergeRecords(slices ...[]Record) []Record // by (Last, Box, Seq) |
| + ``` |
| + |
| +- [ ] **Step 1: Write failing tests** |
| + |
| +`internal/apperr/sink_test.go`: |
| + |
| +```go |
| +package apperr |
| + |
| +import ( |
| + "fmt" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/applog" |
| +) |
| + |
| +func TestSinkCoalescesByFingerprintHostRelease(t *testing.T) { |
| + now := time.Date(2026, 9, 7, 10, 0, 0, 0, time.UTC) |
| + clock := now |
| + s := NewSink("box1", func() time.Time { return clock }) |
| + for i := 0; i < 10000; i++ { |
| + clock = now.Add(time.Duration(i) * time.Millisecond) |
| + s.Add("bab", "web", Record{Host: "h1", Source: SourceEdge, Kind: KindHTTP, Name: "500 GET /x", Status: 500, Release: "v1"}) |
| + } |
| + s.Add("bab", "web", Record{Host: "h2", Source: SourceEdge, Kind: KindHTTP, Name: "500 GET /x", Status: 500, Release: "v1"}) |
| + out := s.Drain() |
| + recs := out[applog.AppKey{Acct: "bab", App: "web"}] |
| + if len(recs) != 2 { |
| + t.Fatalf("records = %d, want 2 (one per host)", len(recs)) |
| + } |
| + var h1 Record |
| + for _, r := range recs { |
| + if r.Host == "h1" { |
| + h1 = r |
| + } |
| + } |
| + if h1.Count != 10000 || !h1.First.Equal(now) || !h1.Last.Equal(now.Add(9999*time.Millisecond)) { |
| + t.Fatalf("h1 = count %d first %v last %v", h1.Count, h1.First, h1.Last) |
| + } |
| + if h1.FP == "" || h1.FPV != FPVersion || h1.Box != "box1" || h1.Seq == 0 { |
| + t.Fatalf("stamping: %+v", h1) |
| + } |
| + if len(s.Drain()) != 0 { |
| + t.Fatal("second drain must be empty") |
| + } |
| +} |
| + |
| +func TestSinkCapsGroupsPerAppPerHour(t *testing.T) { |
| + now := time.Date(2026, 9, 7, 10, 0, 0, 0, time.UTC) |
| + s := NewSink("box1", func() time.Time { return now }) |
| + for i := 0; i < MaxGroupsPerAppHour+5; i++ { |
| + s.Add("bab", "web", Record{Source: SourceEdge, Kind: KindHTTP, Name: fmt.Sprintf("500 GET /p%d-x", i), FP: fmt.Sprintf("fp%d", i)}) |
| + } |
| + s.Add("bab", "other", Record{Source: SourceEdge, Kind: KindHTTP, Name: "unaffected", FP: "z"}) |
| + out := s.Drain() |
| + web := out[applog.AppKey{Acct: "bab", App: "web"}] |
| + if len(web) != MaxGroupsPerAppHour+1 { |
| + t.Fatalf("web records = %d, want cap+1 (the overflow row)", len(web)) |
| + } |
| + var over *Record |
| + for i := range web { |
| + if web[i].FP == OverflowFP { |
| + over = &web[i] |
| + } |
| + } |
| + if over == nil || over.Count != 5 || over.Name != "500 GET /p1000-x" { |
| + t.Fatalf("overflow = %+v", over) |
| + } |
| + if len(out[applog.AppKey{Acct: "bab", App: "other"}]) != 1 { |
| + t.Fatal("a neighbour app must not be suppressed") |
| + } |
| + // The cap counts per hour: the next hour admits new groups again. |
| + now = now.Add(time.Hour) |
| + s.Add("bab", "web", Record{Source: SourceEdge, Kind: KindHTTP, Name: "fresh", FP: "fresh"}) |
| + if recs := s.Drain()[applog.AppKey{Acct: "bab", App: "web"}]; len(recs) != 1 || recs[0].FP != "fresh" { |
| + t.Fatalf("next hour = %+v", recs) |
| + } |
| +} |
| + |
| +func TestSinkSanitisesAndFingerprintsOnAdd(t *testing.T) { |
| + now := time.Date(2026, 9, 7, 10, 0, 0, 0, time.UTC) |
| + s := NewSink("box1", func() time.Time { return now }) |
| + long := make([]byte, MaxStack+100) |
| + for i := range long { |
| + long[i] = 'x' |
| + } |
| + s.Add("bab", "web", Record{Source: SourceServer, Kind: "RuntimeError", Name: "mail to paul@tito.io bounced", |
| + Path: "/reset/8f3a9c1e2b4d6f708a9b0c1d2e3f4a5b?t=1", Stack: string(long), |
| + Fields: map[string]string{"user": "paul@tito.io", "k": "v"}}) |
| + r := s.Drain()[applog.AppKey{Acct: "bab", App: "web"}][0] |
| + if r.Name != "mail to <email> bounced" || r.Path != "/reset/:token" || r.Fields["user"] != "<email>" || r.Fields["k"] != "v" { |
| + t.Fatalf("sanitised = %+v", r) |
| + } |
| + if len(r.Stack) != MaxStack { |
| + t.Fatalf("stack len %d, want %d", len(r.Stack), MaxStack) |
| + } |
| + if r.FP != Fingerprint(SourceServer, "RuntimeError", "mail to <email> bounced", r.Stack) || r.FPV != FPVersion { |
| + t.Fatalf("fp = %q", r.FP) |
| + } |
| + if r.Count != 1 || !r.First.Equal(now) || !r.Last.Equal(now) || !r.TS.Equal(now) { |
| + t.Fatalf("stamps = %+v", r) |
| + } |
| +} |
| +``` |
| + |
| +`internal/apperr/read_test.go`: |
| + |
| +```go |
| +package apperr |
| + |
| +import ( |
| + "context" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +func TestReadWindowMergesAndFilters(t *testing.T) { |
| + ctx := context.Background() |
| + s := release.NewDirStore(t.TempDir()) |
| + now := time.Date(2026, 9, 7, 10, 30, 0, 0, time.UTC) |
| + put := func(box string, recs []Record) { |
| + body, err := EncodeChunk(recs) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := s.Put(ctx, ChunkKey("bab", "web", box, HourOf(recs[0].TS), recs[0].Seq, len(body)), body, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + put("box1", []Record{ |
| + {TS: now, Box: "box1", Host: "h1", Source: SourceEdge, Name: "a", FP: "fa", Count: 3, First: now.Add(-4 * time.Minute), Last: now, Seq: 1}, |
| + {TS: now, Box: "box1", Host: "h1", Source: SourceServer, Name: "b", FP: "fb", Count: 1, First: now, Last: now, Seq: 2}, |
| + }) |
| + put("box2", []Record{ |
| + {TS: now.Add(-3 * time.Hour), Box: "box2", Host: "h2", Source: SourceEdge, Name: "old", FP: "fo", Count: 1, First: now.Add(-3 * time.Hour), Last: now.Add(-3 * time.Hour), Seq: 1}, |
| + }) |
| + recs, trimmed, err := ReadWindow(ctx, s, "bab", "web", Filter{Since: now.Add(-time.Hour)}, 100) |
| + if err != nil || trimmed { |
| + t.Fatalf("read: %v %v", err, trimmed) |
| + } |
| + if len(recs) != 2 || recs[0].FP != "fa" || recs[1].FP != "fb" { |
| + t.Fatalf("window = %+v", recs) |
| + } |
| + // A record whose [First, Last] overlaps the window is in, whole. |
| + recs, _, _ = ReadWindow(ctx, s, "bab", "web", Filter{Since: now.Add(-2 * time.Minute), Until: now.Add(-time.Minute)}, 100) |
| + if len(recs) != 1 || recs[0].FP != "fa" || recs[0].Count != 3 { |
| + t.Fatalf("overlap = %+v", recs) |
| + } |
| + recs, _, _ = ReadWindow(ctx, s, "bab", "web", Filter{Since: now.Add(-time.Hour), Source: SourceServer}, 100) |
| + if len(recs) != 1 || recs[0].FP != "fb" { |
| + t.Fatalf("source filter = %+v", recs) |
| + } |
| + recs, _, _ = ReadWindow(ctx, s, "bab", "web", Filter{Since: now.Add(-4 * time.Hour), Host: "h2"}, 100) |
| + if len(recs) != 1 || recs[0].FP != "fo" { |
| + t.Fatalf("host filter = %+v", recs) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** |
| + |
| +Run: `go test ./internal/apperr/ -count=1` |
| +Expected: FAIL to compile. |
| + |
| +- [ ] **Step 3: Write `sink.go`** |
| + |
| +```go |
| +package apperr |
| + |
| +import ( |
| + "sync" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/applog" |
| +) |
| + |
| +// MaxGroupsPerAppHour caps coalesced records per app per UTC hour, counting |
| +// (fp, host, release) combinations (spec §3.2). Per app, not per box: one |
| +// tenant's storm must not suppress a neighbour's. |
| +const MaxGroupsPerAppHour = 1000 |
| + |
| +// OverflowFP names the one row the excess beyond the cap folds into. Its |
| +// Count is occurrences, and the tab labels it as such. |
| +const OverflowFP = "_overflow" |
| + |
| +// Sink coalesces errors between ship-loop flushes: per app, one record per |
| +// (fp, host, release) carrying count/first/last and the first sample. An |
| +// error storm is the moment this must work — ten thousand identical panics |
| +// in a minute become one record. Safe for concurrent use; the edge, the |
| +// activator and the ship loop share one Sink per agent process. |
| +type Sink struct { |
| + box string |
| + now func() time.Time |
| + |
| + mu sync.Mutex |
| + apps map[applog.AppKey]*appBuf |
| + seq uint64 |
| +} |
| + |
| +type groupKey struct{ fp, host, release string } |
| + |
| +type appBuf struct { |
| + groups map[groupKey]*Record |
| + order []groupKey // insertion order, for a deterministic drain |
| + overflow *Record |
| + hour string // the UTC hour `admitted` counts for |
| + admitted int // distinct groups admitted this hour, cap counter |
| +} |
| + |
| +// NewSink returns a Sink stamping records with box and timestamps from now. |
| +// seq starts at UnixMilli for the reason applog.NewSink gives: chunk keys |
| +// embed the first record's seq, and a restart must sort after last boot. |
| +func NewSink(box string, now func() time.Time) *Sink { |
| + return &Sink{box: box, now: now, apps: make(map[applog.AppKey]*appBuf), seq: uint64(now().UnixMilli())} |
| +} |
| + |
| +// Add buffers one occurrence for (acct, app). r is sanitised here (§3.1), |
| +// fingerprinted when it carries no FP, and folded into its group. |
| +func (s *Sink) Add(acct, app string, r Record) { |
| + now := s.now().UTC() |
| + r.Name = Sanitize(r.Name) |
| + r.Path = SanitizePath(r.Path) |
| + if r.Path == "/" && r.Status == 0 && r.Source != SourceEdge { |
| + r.Path = "" |
| + } |
| + if len(r.Stack) > MaxStack { |
| + r.Stack = r.Stack[:MaxStack] |
| + } |
| + r.Stack = Sanitize(r.Stack) |
| + if len(r.Fields) > 0 { |
| + f := make(map[string]string, len(r.Fields)) |
| + for k, v := range r.Fields { |
| + f[k] = Sanitize(v) |
| + } |
| + r.Fields = f |
| + } |
| + if r.FP == "" { |
| + r.FP = Fingerprint(r.Source, r.Kind, r.Name, r.Stack) |
| + r.FPV = FPVersion |
| + } |
| + if len(r.FP) > 64 { |
| + r.FP = r.FP[:64] |
| + } |
| + |
| + s.mu.Lock() |
| + defer s.mu.Unlock() |
| + b := s.apps[applog.AppKey{Acct: acct, App: app}] |
| + if b == nil { |
| + b = &appBuf{groups: make(map[groupKey]*Record)} |
| + s.apps[applog.AppKey{Acct: acct, App: app}] = b |
| + } |
| + if h := applog.HourOf(now); h != b.hour { |
| + b.hour, b.admitted = h, 0 |
| + } |
| + k := groupKey{r.FP, r.Host, r.Release} |
| + if g := b.groups[k]; g != nil { |
| + g.Count++ |
| + g.Last = now |
| + return |
| + } |
| + if b.admitted >= MaxGroupsPerAppHour { |
| + if b.overflow == nil { |
| + o := r |
| + o.FP = OverflowFP |
| + o.TS, o.First, o.Last, o.Count = now, now, now, 0 |
| + o.Box = s.box |
| + s.seq++ |
| + o.Seq = s.seq |
| + b.overflow = &o |
| + } |
| + b.overflow.Count++ |
| + b.overflow.Last = now |
| + return |
| + } |
| + b.admitted++ |
| + s.seq++ |
| + r.TS, r.First, r.Last, r.Count = now, now, now, 1 |
| + r.Box = s.box |
| + r.Seq = s.seq |
| + b.groups[k] = &r |
| + b.order = append(b.order, k) |
| +} |
| + |
| +// Drain returns everything buffered, keyed by app, and resets the groups |
| +// (the per-hour cap counter and seq persist — they are the hour's and the |
| +// process's, not the batch's). Empty apps are omitted. |
| +func (s *Sink) Drain() map[applog.AppKey][]Record { |
| + s.mu.Lock() |
| + defer s.mu.Unlock() |
| + out := make(map[applog.AppKey][]Record) |
| + for k, b := range s.apps { |
| + if len(b.order) == 0 && b.overflow == nil { |
| + continue |
| + } |
| + recs := make([]Record, 0, len(b.order)+1) |
| + for _, gk := range b.order { |
| + recs = append(recs, *b.groups[gk]) |
| + } |
| + if b.overflow != nil { |
| + recs = append(recs, *b.overflow) |
| + } |
| + out[k] = recs |
| + b.groups = make(map[groupKey]*Record) |
| + b.order = nil |
| + b.overflow = nil |
| + } |
| + return out |
| +} |
| + |
| +// Known lists every app this sink has ever buffered for — the prune |
| +// worklist, as applog.Sink.Known is for logs. |
| +func (s *Sink) Known() []applog.AppKey { |
| + s.mu.Lock() |
| + defer s.mu.Unlock() |
| + out := make([]applog.AppKey, 0, len(s.apps)) |
| + for k := range s.apps { |
| + out = append(out, k) |
| + } |
| + return out |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Write `read.go`** |
| + |
| +```go |
| +package apperr |
| + |
| +import ( |
| + "context" |
| + "sort" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/applog" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// Prefix is the app's errors/ subtree (spec §3.2): a sibling family of |
| +// logs/ with the same key shape below it. |
| +func Prefix(acct, app string) string { return applog.FamilyPrefix(applog.FamilyErrors, acct, app) } |
| + |
| +// BoxPrefix is one box's subtree within an app's errors. |
| +func BoxPrefix(acct, app, box string) string { |
| + return applog.FamilyBoxPrefix(applog.FamilyErrors, acct, app, box) |
| +} |
| + |
| +// ChunkKey names one flushed errors chunk; see applog.ChunkKey. |
| +func ChunkKey(acct, app, box, hour string, seq uint64, size int) string { |
| + return applog.FamilyChunkKey(applog.FamilyErrors, acct, app, box, hour, seq, size) |
| +} |
| + |
| +// HourOf is applog.HourOf, re-exported so shippers need one import. |
| +func HourOf(t time.Time) string { return applog.HourOf(t) } |
| + |
| +// EncodeChunk renders records as gzipped JSONL. |
| +func EncodeChunk(recs []Record) ([]byte, error) { return applog.EncodeJSONL(recs) } |
| + |
| +// DecodeChunk parses a chunk, skipping malformed lines. |
| +func DecodeChunk(b []byte) ([]Record, error) { return applog.DecodeJSONL[Record](b) } |
| + |
| +// Filter is the read-side match. A coalesced record belongs to a window |
| +// when [First, Last] overlaps [Since, Until] and is counted whole (spec |
| +// §3.2: a window edge is wrong by at most one flush interval). |
| +type Filter struct { |
| + Since, Until time.Time |
| + Host string |
| + Source string |
| + Release string |
| + FP string |
| +} |
| + |
| +// Match reports whether r survives f. |
| +func (f Filter) Match(r Record) bool { |
| + last, first := r.Last, r.First |
| + if last.IsZero() { |
| + last = r.TS |
| + } |
| + if first.IsZero() { |
| + first = r.TS |
| + } |
| + if !f.Since.IsZero() && last.Before(f.Since) { |
| + return false |
| + } |
| + if !f.Until.IsZero() && first.After(f.Until) { |
| + return false |
| + } |
| + if f.Host != "" && r.Host != f.Host { |
| + return false |
| + } |
| + if f.Source != "" && r.Source != f.Source { |
| + return false |
| + } |
| + if f.Release != "" && r.Release != f.Release { |
| + return false |
| + } |
| + if f.FP != "" && r.FP != f.FP { |
| + return false |
| + } |
| + return true |
| +} |
| + |
| +// ReadWindow assembles one app's errors/ records across boxes: every hour |
| +// intersecting the window, filtered through f, merged by (Last, Box, Seq). |
| +// collectCap bounds memory; past it the oldest go and the bool says so. |
| +func ReadWindow(ctx context.Context, s release.Store, acct, app string, f Filter, collectCap int) ([]Record, bool, error) { |
| + perBox, trimmed, err := applog.ReadChunks[Record](ctx, s, Prefix(acct, app), f.Since, f.Until, f.Match, collectCap) |
| + if err != nil { |
| + return nil, false, err |
| + } |
| + merged := MergeRecords(perBox...) |
| + if over := len(merged) - collectCap; over > 0 { |
| + merged = merged[over:] |
| + trimmed = true |
| + } |
| + return merged, trimmed, nil |
| +} |
| + |
| +// MergeRecords folds per-source slices into one timeline ordered by |
| +// (Last, Box, Seq), oldest first. |
| +func MergeRecords(slices ...[]Record) []Record { |
| + var out []Record |
| + for _, s := range slices { |
| + out = append(out, s...) |
| + } |
| + sort.SliceStable(out, func(i, j int) bool { |
| + a, b := out[i], out[j] |
| + if !a.Last.Equal(b.Last) { |
| + return a.Last.Before(b.Last) |
| + } |
| + if a.Box != b.Box { |
| + return a.Box < b.Box |
| + } |
| + return a.Seq < b.Seq |
| + }) |
| + return out |
| +} |
| +``` |
| + |
| +Note on `ReadChunks` and the hour window: a record's chunk lives under the hour of its `TS` (the first occurrence); a window that starts after that hour but before `Last` will not list the hour. Since the flush interval is at most ten seconds plus the hour boundary, the reader widens `Since` by one hour when listing: in `ReadWindow`, pass `f.Since.Add(-time.Hour)` as the listing `since` (when non-zero) and keep `f.Match` as the exact test. Add that line and a comment saying why. |
| + |
| +- [ ] **Step 5: Run tests** |
| + |
| +Run: `go test ./internal/apperr/ -count=1 -v` |
| +Expected: PASS. |
| + |
| +- [ ] **Step 6: Commit** |
| + |
| +```bash |
| +git add internal/apperr |
| +git commit -m "apperr: coalescing sink and errors/ chunk read (spec §3.2) |
| + |
| +Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 5: `internal/apperr` grouping for the tab |
| + |
| +**Files:** |
| +- Create: `internal/apperr/group.go` |
| +- Test: `internal/apperr/group_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + ```go |
| + type Group struct { FP, Source, Kind, Name, Level string; Count int; First, Last time.Time; Hosts, Releases []string; Sample Record } |
| + func GroupRecords(recs []Record) []Group // newest Last first |
| + ``` |
| + |
| +- [ ] **Step 1: Failing test** |
| + |
| +```go |
| +package apperr |
| + |
| +import ( |
| + "testing" |
| + "time" |
| +) |
| + |
| +func TestGroupRecords(t *testing.T) { |
| + now := time.Date(2026, 9, 7, 10, 0, 0, 0, time.UTC) |
| + recs := []Record{ |
| + {FP: "a", Source: SourceEdge, Kind: KindHTTP, Name: "500 GET /x", Host: "h1", Release: "v1", Count: 3, First: now.Add(-10 * time.Minute), Last: now.Add(-5 * time.Minute), Stack: "s1"}, |
| + {FP: "a", Source: SourceEdge, Kind: KindHTTP, Name: "500 GET /x", Host: "h2", Release: "v2", Count: 2, First: now.Add(-20 * time.Minute), Last: now.Add(-1 * time.Minute)}, |
| + {FP: "b", Source: SourceLog, Kind: KindLog, Name: "boom", Level: "error", Host: "h1", Count: 1, First: now.Add(-30 * time.Minute), Last: now.Add(-30 * time.Minute)}, |
| + {FP: OverflowFP, Source: SourceEdge, Name: "500 GET /z", Host: "h1", Count: 40, First: now, Last: now}, |
| + } |
| + gs := GroupRecords(recs) |
| + if len(gs) != 3 { |
| + t.Fatalf("groups = %d", len(gs)) |
| + } |
| + if gs[0].FP != OverflowFP || gs[1].FP != "a" || gs[2].FP != "b" { |
| + t.Fatalf("order by Last desc: %s %s %s", gs[0].FP, gs[1].FP, gs[2].FP) |
| + } |
| + a := gs[1] |
| + if a.Count != 5 || !a.First.Equal(now.Add(-20*time.Minute)) || !a.Last.Equal(now.Add(-1*time.Minute)) { |
| + t.Fatalf("a = %+v", a) |
| + } |
| + if len(a.Hosts) != 2 || a.Hosts[0] != "h1" || a.Hosts[1] != "h2" || len(a.Releases) != 2 { |
| + t.Fatalf("a hosts/releases = %v %v", a.Hosts, a.Releases) |
| + } |
| + if a.Sample.Stack != "s1" { |
| + t.Fatal("sample must be the earliest record") |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run, expect compile failure** |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +```go |
| +package apperr |
| + |
| +import ( |
| + "sort" |
| + "time" |
| +) |
| + |
| +// Group is one row of the Errors tab: every record sharing a fingerprint |
| +// in the window, summed (spec §3.6). |
| +type Group struct { |
| + FP string `json:"fp"` |
| + Source string `json:"source"` |
| + Kind string `json:"kind,omitempty"` |
| + Name string `json:"name"` |
| + Level string `json:"level,omitempty"` |
| + Count int `json:"count"` |
| + First time.Time `json:"first"` |
| + Last time.Time `json:"last"` |
| + Hosts []string `json:"hosts"` |
| + Releases []string `json:"releases,omitempty"` |
| + Sample Record `json:"sample"` |
| +} |
| + |
| +// GroupRecords folds records into groups by FP, newest Last first. The |
| +// sample is the earliest record of the group (its stack and fields are |
| +// the first occurrence's). |
| +func GroupRecords(recs []Record) []Group { |
| + byFP := make(map[string]*Group) |
| + var order []string |
| + for _, r := range recs { |
| + first, last := r.First, r.Last |
| + if first.IsZero() { |
| + first = r.TS |
| + } |
| + if last.IsZero() { |
| + last = r.TS |
| + } |
| + count := r.Count |
| + if count == 0 { |
| + count = 1 |
| + } |
| + g := byFP[r.FP] |
| + if g == nil { |
| + g = &Group{FP: r.FP, Source: r.Source, Kind: r.Kind, Name: r.Name, Level: r.Level, First: first, Last: last, Sample: r} |
| + byFP[r.FP] = g |
| + order = append(order, r.FP) |
| + } |
| + g.Count += count |
| + if first.Before(g.First) { |
| + g.First = first |
| + g.Sample = r |
| + } |
| + if last.After(g.Last) { |
| + g.Last = last |
| + } |
| + g.Hosts = addUnique(g.Hosts, r.Host) |
| + g.Releases = addUnique(g.Releases, r.Release) |
| + } |
| + out := make([]Group, 0, len(order)) |
| + for _, fp := range order { |
| + g := byFP[fp] |
| + sort.Strings(g.Hosts) |
| + sort.Strings(g.Releases) |
| + out = append(out, *g) |
| + } |
| + sort.SliceStable(out, func(i, j int) bool { return out[i].Last.After(out[j].Last) }) |
| + return out |
| +} |
| + |
| +func addUnique(list []string, v string) []string { |
| + if v == "" { |
| + return list |
| + } |
| + for _, x := range list { |
| + if x == v { |
| + return list |
| + } |
| + } |
| + return append(list, v) |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run, expect PASS. Commit** |
| + |
| +```bash |
| +git add internal/apperr/group.go internal/apperr/group_test.go |
| +git commit -m "apperr: group records by fingerprint for the Errors tab |
| + |
| +Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 6: Ship the error sink from the agent; prune by family |
| + |
| +**Files:** |
| +- Modify: `internal/host/logship.go` (`logshipPrune` → `chunkPrune`; remove `hourNameOf`) |
| +- Create: `internal/host/errship.go` |
| +- Modify: `internal/host/agent.go:234-237, 384` |
| +- Test: `internal/host/logship_test.go` (update two callers), `internal/host/errship_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `apperr.Sink`, `apperr.EncodeChunk`, `apperr.ChunkKey`, `apperr.BoxPrefix`, `applog.FamilyBoxPrefix`, `applog.HourNameOfDir`. |
| +- Produces: `chunkPrune(ctx, s release.Store, fam applog.Family, box string, k applog.AppKey, now time.Time)`; `errshipLoop(sink *apperr.Sink, box string)`; `errshipFlush(ctx, s, pending map[applog.AppKey][]apperr.Record)`; package-level `errSink *apperr.Sink` is NOT used — the sink is constructed in `runAgent` beside `logSink` and passed to the edge in the `RunWithActivatorReady` callback (Task 7 adds `EnableErrors`). |
| + |
| +- [ ] **Step 1: Refactor prune to take a family** |
| + |
| +In `logship.go`, rename `logshipPrune` to `chunkPrune` with signature `(ctx context.Context, s release.Store, fam applog.Family, box string, k applog.AppKey, now time.Time)`; inside, `prefix := applog.FamilyBoxPrefix(fam, k.Acct, k.App, box)` and `hourName := applog.HourNameOfDir(h.dir)`; delete `hourNameOf`. Update the call in `logshipLoop` to `chunkPrune(ctx, s, applog.FamilyLogs, box, k, time.Now())` and the two test callers in `logship_test.go` likewise. Update the doc comment: "for ONE app's ONE family on THIS box's subtree". |
| + |
| +Run: `go test ./internal/host/ -run Logship -count=1` — PASS. |
| + |
| +- [ ] **Step 2: Write the failing errship test** |
| + |
| +`internal/host/errship_test.go`: |
| + |
| +```go |
| +package host |
| + |
| +import ( |
| + "context" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/apperr" |
| + "github.com/carlosframework/platform/internal/applog" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +func TestErrshipFlushWritesOneChunkPerApp(t *testing.T) { |
| + s := release.NewDirStore(t.TempDir()) |
| + now := time.Date(2026, 9, 7, 10, 0, 0, 0, time.UTC) |
| + sink := apperr.NewSink("box1", func() time.Time { return now }) |
| + sink.Add("bab", "web", apperr.Record{Source: apperr.SourceEdge, Kind: apperr.KindHTTP, Name: "500 GET /x", Host: "h1"}) |
| + sink.Add("bab", "web", apperr.Record{Source: apperr.SourceEdge, Kind: apperr.KindHTTP, Name: "500 GET /x", Host: "h1"}) |
| + pending := map[applog.AppKey][]apperr.Record{} |
| + errshipGather(sink, pending) |
| + errshipFlush(context.Background(), s, pending) |
| + if len(pending) != 0 { |
| + t.Fatalf("pending after flush = %d", len(pending)) |
| + } |
| + recs, _, err := apperr.ReadWindow(context.Background(), s, "bab", "web", apperr.Filter{Since: now.Add(-time.Minute)}, 100) |
| + if err != nil || len(recs) != 1 || recs[0].Count != 2 { |
| + t.Fatalf("read back = %+v, %v", recs, err) |
| + } |
| +} |
| + |
| +func TestErrshipFlushHoldsOnFailure(t *testing.T) { |
| + pending := map[applog.AppKey][]apperr.Record{{Acct: "bab", App: "web"}: {{TS: time.Now(), Source: apperr.SourceEdge, Name: "x", FP: "f", Box: "box1", Seq: 1}}} |
| + errshipFlush(context.Background(), failingStore{}, pending) |
| + if len(pending) != 1 { |
| + t.Fatal("a failed flush must keep its records for the next round") |
| + } |
| +} |
| +``` |
| + |
| +(`failingStore` already exists in `logship_test.go`.) |
| + |
| +- [ ] **Step 3: Write `errship.go`** |
| + |
| +```go |
| +// The error shipping loop — the box-side pump of the Errors tab (spec |
| +// 2026-09-07-errors-and-analytics-design.md §3.2), shaped exactly like |
| +// logshipLoop: the edge and the activator coalesce into one apperr.Sink; |
| +// every tick this drains it and writes one immutable chunk per app under |
| +// apps/<sqid>/<app>/errors/<box>/<hour>/; on each hour boundary it prunes |
| +// its own subtree with the log retention (§3.6: errors follow log |
| +// retention, age and byte cap). |
| +package host |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "log" |
| + "os" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/apperr" |
| + "github.com/carlosframework/platform/internal/applog" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +const errshipTick = 10 * time.Second |
| + |
| +func errshipLoop(sink *apperr.Sink, box string) { |
| + defer func() { |
| + if r := recover(); r != nil { |
| + log.Printf("agent: errship loop panic, error shipping stopped for this process: %v", r) |
| + } |
| + }() |
| + ctx := context.Background() |
| + s, target, err := openStatusStore(ctx, os.Getenv) |
| + if err != nil { |
| + if errors.Is(err, errNoStatusStore) { |
| + log.Printf("agent: errship loop disabled: neither CARLOS_STATUS_BUCKET nor CARLOS_DEPLOYMENT_BUCKET is set") |
| + } else { |
| + log.Printf("agent: errship loop disabled: %v", err) |
| + } |
| + return |
| + } |
| + log.Printf("agent: errship loop armed (10s ticks), publishing to %s", target) |
| + |
| + pending := make(map[applog.AppKey][]apperr.Record) |
| + pruned := "" |
| + ticker := time.NewTicker(errshipTick) |
| + defer ticker.Stop() |
| + for range ticker.C { |
| + errshipGather(sink, pending) |
| + errshipFlush(ctx, s, pending) |
| + if hour := applog.HourOf(time.Now()); hour != pruned { |
| + for _, k := range errshipKnownApps(pending, sink) { |
| + chunkPrune(ctx, s, applog.FamilyErrors, box, k, time.Now()) |
| + } |
| + pruned = hour |
| + } |
| + } |
| +} |
| + |
| +// errshipGather drains the sink into pending under the same per-app cap |
| +// logship uses — oldest first. |
| +func errshipGather(sink *apperr.Sink, pending map[applog.AppKey][]apperr.Record) { |
| + for k, recs := range sink.Drain() { |
| + merged := append(pending[k], recs...) |
| + if over := len(merged) - logshipMaxPending; over > 0 { |
| + merged = merged[over:] |
| + } |
| + pending[k] = merged |
| + } |
| +} |
| + |
| +// errshipFlush writes one create-only chunk per pending app and forgets |
| +// what landed; a failed app keeps its records for the next round. |
| +func errshipFlush(ctx context.Context, s release.Store, pending map[applog.AppKey][]apperr.Record) { |
| + for k, recs := range pending { |
| + if len(recs) == 0 { |
| + delete(pending, k) |
| + continue |
| + } |
| + body, err := apperr.EncodeChunk(recs) |
| + if err != nil { |
| + log.Printf("agent: errship encode %s/%s: %v (dropping %d records)", k.Acct, k.App, err, len(recs)) |
| + delete(pending, k) |
| + continue |
| + } |
| + key := apperr.ChunkKey(k.Acct, k.App, recs[0].Box, apperr.HourOf(recs[0].TS), recs[0].Seq, len(body)) |
| + if _, err := s.Put(ctx, key, body, ""); err != nil { |
| + log.Printf("agent: errship flush %s/%s: %v (%d records held)", k.Acct, k.App, err, len(recs)) |
| + continue |
| + } |
| + delete(pending, k) |
| + } |
| +} |
| + |
| +func errshipKnownApps(pending map[applog.AppKey][]apperr.Record, sink *apperr.Sink) []applog.AppKey { |
| + seen := make(map[applog.AppKey]bool) |
| + var out []applog.AppKey |
| + for k := range pending { |
| + if !seen[k] { |
| + seen[k] = true |
| + out = append(out, k) |
| + } |
| + } |
| + for _, k := range sink.Known() { |
| + if !seen[k] { |
| + seen[k] = true |
| + out = append(out, k) |
| + } |
| + } |
| + return out |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Wire in `agent.go`** |
| + |
| +Beside `logSink` (line 234-237): |
| + |
| +```go |
| + var logSink *applog.Sink |
| + var errSink *apperr.Sink |
| + if _, _, err := openStatusStore(context.Background(), os.Getenv); err == nil { |
| + logSink = applog.NewSink(status.Label(readMachineID()), time.Now) |
| + errSink = apperr.NewSink(status.Label(readMachineID()), time.Now) |
| + } |
| +``` |
| + |
| +Beside `go logshipLoop(...)` (line 384): `if errSink != nil { go errshipLoop(errSink, status.Label(readMachineID())) }` — match however `logshipLoop` is gated there (read the surrounding lines and mirror the condition exactly). |
| + |
| +In the `RunWithActivatorReady` callback after `e.EnableUsage(m)`: `if errSink != nil { e.EnableErrors(errSink) }` — `EnableErrors` arrives in Task 7; add this line in Task 7 instead if the build order matters. Order the tasks so Task 7 lands before this file compiles: do Task 7's edge change first if convenient, or add the `EnableErrors` call in Task 7. **Decision: add the `agent.go` callback line in Task 7**, and only the sink construction and loop here. |
| + |
| +- [ ] **Step 5: Test and commit** |
| + |
| +Run: `go build ./... && go test ./internal/host/ -count=1` |
| +Expected: PASS. |
| + |
| +```bash |
| +git add internal/host |
| +git commit -m "host: ship coalesced errors like logs; prune by chunk family |
| + |
| +Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 7: The edge observes status and emits edge errors |
| + |
| +**Files:** |
| +- Create: `internal/activator/answered.go` |
| +- Modify: `internal/activator/activator.go:1298-1330` (three `http.Error` sites in `serve`) |
| +- Modify: `internal/edge/usagemeter.go` (`countingWriter`) |
| +- Modify: `internal/edge/edge.go` (`Edge` fields, `EnableErrors`, `ServeHTTP` at 1241-1250 and the hibernating branch) |
| +- Modify: `internal/host/agent.go` (callback line) |
| +- Test: `internal/activator/answered_test.go`, `internal/edge/usagemeter_test.go`, `internal/edge/edgeerrors_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `activator.WithAnswerFlag(r *http.Request) (*http.Request, *bool)`; `activator.MarkAnswered(r *http.Request)`; `edge.ErrorSink interface{ Add(acct, app string, r apperr.Record) }`; `(*Edge).EnableErrors(ErrorSink)`; `countingWriter` gains `status int`, `ctype string`, `wroteHdr bool` and a `WriteHeader`. |
| + |
| +- [ ] **Step 1: Failing tests** |
| + |
| +`internal/activator/answered_test.go`: |
| + |
| +```go |
| +package activator |
| + |
| +import ( |
| + "net/http/httptest" |
| + "testing" |
| +) |
| + |
| +func TestAnswerFlagRoundTrip(t *testing.T) { |
| + r := httptest.NewRequest("GET", "http://h/", nil) |
| + MarkAnswered(r) // no flag: a no-op, never a panic |
| + r, flag := WithAnswerFlag(r) |
| + if *flag { |
| + t.Fatal("flag must start false") |
| + } |
| + MarkAnswered(r) |
| + if !*flag { |
| + t.Fatal("MarkAnswered must set the flag the edge holds") |
| + } |
| +} |
| +``` |
| + |
| +Append to `internal/edge/usagemeter_test.go`: |
| + |
| +```go |
| +func TestCountingWriterObservesStatusAndType(t *testing.T) { |
| + var n int64 |
| + rec := httptest.NewRecorder() |
| + w := &countingWriter{ResponseWriter: rec, n: &n} |
| + w.Header().Set("Content-Type", "text/html; charset=utf-8") |
| + w.WriteHeader(103) // informational: forwarded, not latched |
| + w.WriteHeader(http.StatusBadGateway) |
| + w.WriteHeader(http.StatusOK) // superfluous second final status: ignored |
| + if w.status != http.StatusBadGateway || w.ctype != "text/html; charset=utf-8" { |
| + t.Fatalf("latched %d %q", w.status, w.ctype) |
| + } |
| + // Implicit 200 at first Write. |
| + rec2 := httptest.NewRecorder() |
| + w2 := &countingWriter{ResponseWriter: rec2, n: &n} |
| + w2.Write([]byte("x")) |
| + if w2.status != http.StatusOK { |
| + t.Fatalf("implicit status = %d", w2.status) |
| + } |
| + // Nothing written at all: status stays 0 (the caller treats it as a |
| + // bodiless 200 that is never a pageview and never an error). |
| + w3 := &countingWriter{ResponseWriter: httptest.NewRecorder(), n: &n} |
| + if w3.status != 0 { |
| + t.Fatal("untouched writer must report 0") |
| + } |
| +} |
| +``` |
| + |
| +`internal/edge/edgeerrors_test.go`: |
| + |
| +```go |
| +package edge |
| + |
| +import ( |
| + "net/http" |
| + "net/http/httptest" |
| + "strings" |
| + "sync" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/activator" |
| + "github.com/carlosframework/platform/internal/apperr" |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +type memErrSink struct { |
| + mu sync.Mutex |
| + recs []apperr.Record |
| + apps []string |
| +} |
| + |
| +func (m *memErrSink) Add(acct, app string, r apperr.Record) { |
| + m.mu.Lock() |
| + defer m.mu.Unlock() |
| + m.recs = append(m.recs, r) |
| + m.apps = append(m.apps, acct+"/"+app) |
| +} |
| + |
| +func TestEdgeEmitsErrorOnUpstream5xx(t *testing.T) { |
| + backend := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + switch r.URL.Path { |
| + case "/boom": |
| + http.Error(w, "kaboom", http.StatusInternalServerError) |
| + case "/missing": |
| + http.NotFound(w, r) |
| + default: |
| + w.Header().Set("Content-Type", "text/html") |
| + w.Write([]byte("<p>ok</p>")) |
| + } |
| + })) |
| + t.Cleanup(backend.Close) |
| + reg := openReg(t) |
| + rt := registry.Route{Host: "app.example.com", Kind: "service", Addr: backend.Listener.Addr().String(), Status: "ready", App: "web", Version: "v7"} |
| + if err := reg.Upsert(rt); err != nil { |
| + t.Fatal(err) |
| + } |
| + e, err := New(reg, nil) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + sink := &memErrSink{} |
| + e.EnableErrors(sink) |
| + srv := httptest.NewServer(e) |
| + t.Cleanup(srv.Close) |
| + |
| + do := func(method, path string) int { |
| + req, _ := http.NewRequest(method, srv.URL+path, nil) |
| + req.Host = "app.example.com" |
| + resp, err := http.DefaultClient.Do(req) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + resp.Body.Close() |
| + return resp.StatusCode |
| + } |
| + if do("POST", "/boom?token=abc") != 500 || do("GET", "/missing") != 404 || do("GET", "/") != 200 { |
| + t.Fatal("upstream statuses must pass through unchanged") |
| + } |
| + if len(sink.recs) != 1 { |
| + t.Fatalf("errors emitted = %d, want 1 (the 500 only)", len(sink.recs)) |
| + } |
| + r := sink.recs[0] |
| + if r.Source != apperr.SourceEdge || r.Kind != apperr.KindHTTP || r.Status != 500 || r.Host != "app.example.com" || r.Release != "v7" { |
| + t.Fatalf("record = %+v", r) |
| + } |
| + if r.Name != "500 POST /boom" || r.Path != "/boom" || strings.Contains(r.Name, "token") { |
| + t.Fatalf("name/path = %q %q", r.Name, r.Path) |
| + } |
| +} |
| + |
| +// answering503 is an Activator that answers every request itself with the |
| +// wake-path 503 the real one writes, and marks it as the activator's own. |
| +type answering503 struct{} |
| + |
| +func (answering503) ServeHTTP(w http.ResponseWriter, r *http.Request, rt registry.Route, proxy http.Handler) bool { |
| + activator.MarkAnswered(r) |
| + http.Error(w, "instance failed to wake", http.StatusServiceUnavailable) |
| + return true |
| +} |
| +func (answering503) RetireExcept(map[string]bool) {} |
| + |
| +func TestEdgeSkipsActivatorAnswered503(t *testing.T) { |
| + reg := openReg(t) |
| + rt := registry.Route{Host: "sleepy.example.com", Kind: "instance", Socket: t.TempDir() + "/x.sock", Status: "ready", Hibernate: true, App: "web"} |
| + if err := reg.Upsert(rt); err != nil { |
| + t.Fatal(err) |
| + } |
| + e, err := New(reg, answering503{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + sink := &memErrSink{} |
| + e.EnableErrors(sink) |
| + rec := httptest.NewRecorder() |
| + req := httptest.NewRequest("GET", "http://sleepy.example.com/", nil) |
| + e.ServeHTTP(rec, req) |
| + if rec.Code != 503 { |
| + t.Fatalf("status = %d", rec.Code) |
| + } |
| + if len(sink.recs) != 0 { |
| + t.Fatalf("an activator-answered 503 is already a platform log error; got %d edge errors", len(sink.recs)) |
| + } |
| +} |
| +``` |
| + |
| +If `registry.Route` needs different fields for a hibernating instance route to load into the edge's map (check `TestHibernat*` tests in `edge_test.go` or `activator_e2e_test.go` for the minimal upsert they use), copy that shape. |
| + |
| +- [ ] **Step 2: Run to confirm failure** |
| + |
| +Run: `go test ./internal/activator/ -run TestAnswerFlag -count=1; go test ./internal/edge/ -run 'TestCountingWriterObserves|TestEdgeEmits|TestEdgeSkips' -count=1` |
| +Expected: compile failures. |
| + |
| +- [ ] **Step 3: `internal/activator/answered.go`** |
| + |
| +```go |
| +package activator |
| + |
| +import ( |
| + "context" |
| + "net/http" |
| +) |
| + |
| +// answeredKey carries the flag the edge attaches so it can tell a 503 the |
| +// activator wrote on the wake path from one the upstream returned. The |
| +// wake failure already reaches the member as a platform log error |
| +// (emitEvent, logsink.go); the edge must not also file it as an edge |
| +// error (spec 2026-09-07 §3.2, "exactly once"). |
| +type answeredKey struct{} |
| + |
| +// WithAnswerFlag returns r carrying a fresh flag and the flag itself. |
| +func WithAnswerFlag(r *http.Request) (*http.Request, *bool) { |
| + f := new(bool) |
| + return r.WithContext(context.WithValue(r.Context(), answeredKey{}, f)), f |
| +} |
| + |
| +// MarkAnswered sets the flag when r carries one; a no-op otherwise. |
| +func MarkAnswered(r *http.Request) { |
| + if f, ok := r.Context().Value(answeredKey{}).(*bool); ok { |
| + *f = true |
| + } |
| +} |
| +``` |
| + |
| +In `serve` (activator.go), call `MarkAnswered(r)` immediately before each of the four `http.Error(w, …, http.StatusServiceUnavailable)` calls (sidecar-asleep, waking elsewhere, waking, failed to wake). |
| + |
| +- [ ] **Step 4: `countingWriter` observes** |
| + |
| +In `usagemeter.go`: |
| + |
| +```go |
| +// countingWriter counts response body bytes and OBSERVES the final status |
| +// and Content-Type (spec 2026-09-07 §3.3): it is the one wrapper installed |
| +// at a single site for every response shape — cache hit, plain proxy, |
| +// hibernating, upgrade — so it is where the edge learns that an upstream |
| +// answered 5xx (an edge error) or served HTML (a pageview, step 2). It |
| +// sits below the compressor, so Content-Type is the synced value |
| +// gzipWriter.syncHeader copied down before calling this WriteHeader. |
| +// Deliberately does NOT implement io.ReaderFrom: io.Copy falls back to |
| +// Write, which counts. Hijack/Flush forward via the interfaces the wrapped |
| +// writer provides, mirroring recordingResponseWriter (edge.go). |
| +type countingWriter struct { |
| + http.ResponseWriter |
| + n *int64 |
| + status int // final status (>= 200); 0 until one is written |
| + ctype string // Content-Type at the final WriteHeader |
| + wroteHdr bool |
| +} |
| + |
| +// WriteHeader latches the first final status. 1xx are forwarded and never |
| +// latched (recordingResponseWriter.WriteHeader's reasoning). |
| +func (c *countingWriter) WriteHeader(status int) { |
| + if status < 200 { |
| + c.ResponseWriter.WriteHeader(status) |
| + return |
| + } |
| + if !c.wroteHdr { |
| + c.wroteHdr = true |
| + c.status = status |
| + c.ctype = c.ResponseWriter.Header().Get("Content-Type") |
| + } |
| + c.ResponseWriter.WriteHeader(status) |
| +} |
| + |
| +func (c *countingWriter) Write(p []byte) (int, error) { |
| + if !c.wroteHdr { |
| + c.wroteHdr = true |
| + c.status = http.StatusOK |
| + c.ctype = c.ResponseWriter.Header().Get("Content-Type") |
| + } |
| + n, err := c.ResponseWriter.Write(p) |
| + *c.n += int64(n) |
| + return n, err |
| +} |
| +``` |
| + |
| +Keep `Hijack` and `Flush` as they are. |
| + |
| +- [ ] **Step 5: Edge wiring** |
| + |
| +In `edge.go`: |
| + |
| +Add near the `usageMeter` field: |
| + |
| +```go |
| + // errSink, when set, receives edge-observed errors (an upstream 5xx) |
| + // — spec 2026-09-07 §3.2. Nil disables emission; nothing else changes. |
| + errSink atomic.Pointer[errorSinkRef] |
| +``` |
| + |
| +and: |
| + |
| +```go |
| +// ErrorSink is where the edge files an error it observed on the serving |
| +// path. apperr.Sink satisfies it; the interface lives here so this package |
| +// depends on the contract, not the buffer. |
| +type ErrorSink interface { |
| + Add(acct, app string, r apperr.Record) |
| +} |
| + |
| +type errorSinkRef struct{ ErrorSink } |
| + |
| +// EnableErrors wires the error sink in. Called once at startup by the |
| +// agent; nil is a valid no-op. |
| +func (e *Edge) EnableErrors(s ErrorSink) { |
| + if s == nil { |
| + e.errSink.Store(nil) |
| + return |
| + } |
| + e.errSink.Store(&errorSinkRef{s}) |
| +} |
| + |
| +// emitEdgeError files one edge-observed 5xx for rt. The path is sanitised |
| +// (query and token segments off) before it becomes the record's name. |
| +func (e *Edge) emitEdgeError(rt registry.Route, r *http.Request, status int) { |
| + ref := e.errSink.Load() |
| + if ref == nil { |
| + return |
| + } |
| + path := apperr.SanitizePath(r.URL.Path) |
| + ref.Add(release.RouteAccount(rt), rt.App, apperr.Record{ |
| + Host: rt.Host, |
| + Source: apperr.SourceEdge, |
| + Kind: apperr.KindHTTP, |
| + Name: fmt.Sprintf("%d %s %s", status, r.Method, path), |
| + Path: path, |
| + Status: status, |
| + Release: rt.Version, |
| + }) |
| +} |
| +``` |
| + |
| +Replace the meter block at `edge.go:1241-1245` with: |
| + |
| +```go |
| + // The observing writer is installed UNCONDITIONALLY now — the usage |
| + // meter is one of its two consumers, the errors sink the other — at |
| + // the same single site, below everything that follows, so all three |
| + // dispatch shapes plus the upgrade path pass through it. Meter and |
| + // sink are each nil-safe. Status routes are neither billed nor observed. |
| + // |
| + // (hopped attribution paragraph: keep the existing comment text here.) |
| + var bytes int64 |
| + ob := &countingWriter{ResponseWriter: w, n: &bytes} |
| + w = ob |
| + var answered *bool |
| + if up.hibernating { |
| + // The activator marks the wake-path 503s it writes itself; those |
| + // are already the member's platform log error and are not filed |
| + // again as edge errors (spec §3.2). |
| + r, answered = activator.WithAnswerFlag(r) |
| + } |
| + if up.route.Kind != "status" { |
| + defer func() { |
| + if m := e.usageMeter.Load(); m != nil { |
| + m.Record(up.route, 1, bytes, hopped) |
| + } |
| + if ob.status >= 500 && (answered == nil || !*answered) { |
| + e.emitEdgeError(up.route, r, ob.status) |
| + } |
| + }() |
| + } |
| +``` |
| + |
| +Add `"github.com/carlosframework/platform/internal/activator"` and `"github.com/carlosframework/platform/internal/apperr"` to the imports. Confirm no cycle: `go build ./...` (activator must not import edge; it does not today). |
| + |
| +Existing usage tests that construct the edge without a meter must still pass: `Record` is skipped when the meter is nil. |
| + |
| +- [ ] **Step 6: `agent.go` callback** |
| + |
| +After `e.EnableUsage(m)`: |
| + |
| +```go |
| + if errSink != nil { |
| + e.EnableErrors(errSink) |
| + } |
| +``` |
| + |
| +- [ ] **Step 7: Run edge, activator, host packages** |
| + |
| +Run: `go build ./... && go vet ./internal/edge/ ./internal/activator/ ./internal/host/ && go test ./internal/edge/ ./internal/activator/ ./internal/host/ -count=1` |
| +Expected: PASS, including every pre-existing usage, compress and cache test. |
| + |
| +- [ ] **Step 8: Commit** |
| + |
| +```bash |
| +git add internal/activator/answered.go internal/activator/activator.go internal/activator/answered_test.go internal/edge internal/host/agent.go |
| +git commit -m "edge: the counting writer observes status; upstream 5xx become edge errors |
| + |
| +Installed unconditionally at the one site every response shape passes; |
| +the activator marks its own wake-path 503s so they are filed once, as the |
| +platform log error they already are (spec 2026-09-07 §3.2–3.3). |
| + |
| +Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 8: `internal/telemetry` settings object |
| + |
| +**Files:** |
| +- Create: `internal/telemetry/telemetry.go` |
| +- Test: `internal/telemetry/telemetry_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + ```go |
| + const (LogLevelError = "error"; LogLevelWarn = "warn"; CountEdge = "edge"; CountOff = "off"; DoorOn = "on"; DoorOff = "off") |
| + type Analytics struct { Count string `json:"count,omitempty"`; Door string `json:"door,omitempty"` } |
| + type Errors struct { LogLevel string `json:"log_level,omitempty"` } |
| + type Settings struct { Analytics *Analytics `json:"analytics,omitempty"`; Errors *Errors `json:"errors,omitempty"` } |
| + func AccountKey(sqid string) string |
| + func AppKey(sqid, app string) string |
| + func Read(ctx, s release.Store, key string) (Settings, string, error) // etag; os.ErrNotExist → zero Settings, "", nil |
| + func Update(ctx, s release.Store, key string, fn func(*Settings)) error // read-modify-CAS-write, 3 attempts |
| + type Resolved struct { LogLevel, LogLevelFrom, Count, CountFrom, Door, DoorFrom string } |
| + func Resolve(ctx, s release.Store, sqid, app string) (Resolved, error) |
| + func ValidLogLevel(s string) bool |
| + ``` |
| + |
| +- [ ] **Step 1: Failing test** |
| + |
| +```go |
| +package telemetry |
| + |
| +import ( |
| + "context" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +func TestResolveAppThenAccountThenDefault(t *testing.T) { |
| + ctx := context.Background() |
| + s := release.NewDirStore(t.TempDir()) |
| + r, err := Resolve(ctx, s, "bab", "web") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if r.LogLevel != LogLevelError || r.LogLevelFrom != "default" || r.Count != CountOff || r.CountFrom != "default" || r.Door != DoorOn { |
| + t.Fatalf("defaults = %+v", r) |
| + } |
| + if err := Update(ctx, s, AccountKey("bab"), func(st *Settings) { |
| + st.Errors = &Errors{LogLevel: LogLevelWarn} |
| + st.Analytics = &Analytics{Count: CountEdge} |
| + }); err != nil { |
| + t.Fatal(err) |
| + } |
| + r, _ = Resolve(ctx, s, "bab", "web") |
| + if r.LogLevel != LogLevelWarn || r.LogLevelFrom != "account" || r.Count != CountEdge || r.CountFrom != "account" || r.Door != DoorOn || r.DoorFrom != "default" { |
| + t.Fatalf("account level = %+v", r) |
| + } |
| + if err := Update(ctx, s, AppKey("bab", "web"), func(st *Settings) { |
| + st.Analytics = &Analytics{Count: CountOff} |
| + }); err != nil { |
| + t.Fatal(err) |
| + } |
| + r, _ = Resolve(ctx, s, "bab", "web") |
| + if r.LogLevel != LogLevelWarn || r.Count != CountOff || r.CountFrom != "app" { |
| + t.Fatalf("app override = %+v", r) |
| + } |
| + // Update preserves the other key. |
| + if err := Update(ctx, s, AppKey("bab", "web"), func(st *Settings) { st.Errors = &Errors{LogLevel: LogLevelError} }); err != nil { |
| + t.Fatal(err) |
| + } |
| + got, _, _ := Read(ctx, s, AppKey("bab", "web")) |
| + if got.Analytics == nil || got.Analytics.Count != CountOff || got.Errors == nil || got.Errors.LogLevel != LogLevelError { |
| + t.Fatalf("read back = %+v", got) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run, expect compile failure** |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +```go |
| +// Package telemetry is the settings object behind the Errors and Analytics |
| +// tabs (spec 2026-09-07-errors-and-analytics-design.md §3.3): one JSON |
| +// shape at two levels — accounts/<sqid>/telemetry.json and |
| +// apps/<sqid>/<app>/telemetry.json — resolved app → account → built-in |
| +// default. Defaults are rulings (Paul, 2026-09-07): analytics off, the |
| +// browser door on, the log level error. |
| +package telemetry |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "errors" |
| + "fmt" |
| + "os" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +const ( |
| + LogLevelError = "error" |
| + LogLevelWarn = "warn" |
| + CountEdge = "edge" |
| + CountOff = "off" |
| + DoorOn = "on" |
| + DoorOff = "off" |
| +) |
| + |
| +// Analytics is the analytics half. Empty strings mean "not set here". |
| +type Analytics struct { |
| + Count string `json:"count,omitempty"` |
| + Door string `json:"door,omitempty"` |
| +} |
| + |
| +// Errors is the errors half. |
| +type Errors struct { |
| + LogLevel string `json:"log_level,omitempty"` |
| +} |
| + |
| +// Settings is the wire shape at either level. Additive: unknown keys are |
| +// ignored, absent halves inherit. |
| +type Settings struct { |
| + Analytics *Analytics `json:"analytics,omitempty"` |
| + Errors *Errors `json:"errors,omitempty"` |
| +} |
| + |
| +// AccountKey is the account-level object, beside account.json (which stays |
| +// identity-only). |
| +func AccountKey(sqid string) string { return "accounts/" + sqid + "/telemetry.json" } |
| + |
| +// AppKey is the per-app override. |
| +func AppKey(sqid, app string) string { return release.AppPrefix(sqid, app) + "telemetry.json" } |
| + |
| +// ValidLogLevel reports whether s is a level the setting accepts. |
| +func ValidLogLevel(s string) bool { return s == LogLevelError || s == LogLevelWarn } |
| + |
| +// Read returns the object at key and its etag; an absent object is the |
| +// zero Settings with an empty etag and no error. |
| +func Read(ctx context.Context, s release.Store, key string) (Settings, string, error) { |
| + body, etag, err := s.Get(ctx, key) |
| + if err != nil { |
| + if errors.Is(err, os.ErrNotExist) { |
| + return Settings{}, "", nil |
| + } |
| + return Settings{}, "", err |
| + } |
| + var st Settings |
| + if err := json.Unmarshal(body, &st); err != nil { |
| + return Settings{}, "", fmt.Errorf("telemetry: %s: %w", key, err) |
| + } |
| + return st, etag, nil |
| +} |
| + |
| +// Update applies fn to the object at key under compare-and-swap, three |
| +// attempts, so two owners editing different halves at once both land. |
| +func Update(ctx context.Context, s release.Store, key string, fn func(*Settings)) error { |
| + var last error |
| + for attempt := 0; attempt < 3; attempt++ { |
| + st, etag, err := Read(ctx, s, key) |
| + if err != nil { |
| + return err |
| + } |
| + fn(&st) |
| + body, err := json.Marshal(st) |
| + if err != nil { |
| + return err |
| + } |
| + if _, err := s.Put(ctx, key, body, etag); err != nil { |
| + if errors.Is(err, release.ErrCASConflict) { |
| + last = err |
| + continue |
| + } |
| + return err |
| + } |
| + return nil |
| + } |
| + return last |
| +} |
| + |
| +// Resolved is what a reader acts on, with each value's provenance so a |
| +// settings page can say "inherited from the account". |
| +type Resolved struct { |
| + LogLevel, LogLevelFrom string |
| + Count, CountFrom string |
| + Door, DoorFrom string |
| +} |
| + |
| +// Resolve reads both levels and folds them over the defaults. A store |
| +// error fails the resolve; an absent object at either level is fine. |
| +func Resolve(ctx context.Context, s release.Store, sqid, app string) (Resolved, error) { |
| + r := Resolved{LogLevel: LogLevelError, LogLevelFrom: "default", Count: CountOff, CountFrom: "default", Door: DoorOn, DoorFrom: "default"} |
| + if s == nil { |
| + return r, nil |
| + } |
| + acct, _, err := Read(ctx, s, AccountKey(sqid)) |
| + if err != nil { |
| + return r, err |
| + } |
| + appSt, _, err := Read(ctx, s, AppKey(sqid, app)) |
| + if err != nil { |
| + return r, err |
| + } |
| + for _, lvl := range []struct { |
| + from string |
| + st Settings |
| + }{{"account", acct}, {"app", appSt}} { |
| + if lvl.st.Errors != nil && ValidLogLevel(lvl.st.Errors.LogLevel) { |
| + r.LogLevel, r.LogLevelFrom = lvl.st.Errors.LogLevel, lvl.from |
| + } |
| + if lvl.st.Analytics != nil { |
| + if c := lvl.st.Analytics.Count; c == CountEdge || c == CountOff { |
| + r.Count, r.CountFrom = c, lvl.from |
| + } |
| + if d := lvl.st.Analytics.Door; d == DoorOn || d == DoorOff { |
| + r.Door, r.DoorFrom = d, lvl.from |
| + } |
| + } |
| + } |
| + return r, nil |
| +} |
| +``` |
| + |
| +Check how `DirStore.Put` treats the `ifMatch` argument (`""` = unconditional? or create-only?). Read `internal/release/store.go` around `func (d *DirStore) Put` and confirm that `Put(ctx, key, body, "")` on an existing key succeeds or is create-only; if create-only, `Update` must pass the etag read back (which it does) and for a first write the empty etag must mean create — verify with the test. |
| + |
| +- [ ] **Step 4: Run, PASS, commit** |
| + |
| +```bash |
| +git add internal/telemetry |
| +git commit -m "telemetry: the two-level settings object behind Errors and Analytics (spec §3.3) |
| + |
| +Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 9: Console — Errors API, tab, and the log-level setting |
| + |
| +**Files:** |
| +- Create: `internal/console/errorsview.go`, `internal/console/apierrors.go`, `internal/console/templates/app_errors.html` |
| +- Modify: `internal/console/server.go` (routes near 1050 and 1071; page struct near 1370; `handleAppErrors`), `internal/console/authcli.go:595` (two `cliRoute`s), `internal/console/templates/appnav.html`, `internal/console/templates/app_settings.html`, `internal/console/templates/account_settings.html` |
| +- Test: `internal/console/apierrors_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `apperr.*`, `applog.ReadWindow`, `telemetry.Resolve/Update`, `s.instanceGate`, `s.appPage`, `s.accountApp`, `s.account`, `s.audit`, `s.setFlash`, `logsCollectCap`, `instanceStoreTimeout`. |
| +- Produces: |
| + ```go |
| + type errorsWindow struct { Groups []apperr.Group; Records []apperr.Record; Truncated bool; LogLevel, LogLevelFrom string; Sources []string } |
| + func (s *Server) readErrors(ctx, sqid, app string, f apperr.Filter, level string) (errorsWindow, error) |
| + type errorsResponse struct { Groups []apperr.Group `json:"groups"`; Records []apperr.Record `json:"records,omitempty"`; Truncated bool `json:"truncated"`; LogLevel string `json:"log_level"`; Sources []string `json:"sources"` } |
| + ``` |
| + Routes: `GET /api/cli/apps/{account}/{app}/errors` (Instance, Operate); `PUT /api/cli/apps/{account}/{app}/telemetry` (Operate) body `{"errors":{"log_level":"warn"}}`; `GET /a/{sqid}/apps/{app}/errors`; `POST /a/{sqid}/apps/{app}/errors/level`; `POST /a/{sqid}/errors/level`. |
| + |
| +- [ ] **Step 1: Failing tests** — `internal/console/apierrors_test.go` |
| + |
| +```go |
| +package console |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "net/http" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/apperr" |
| + "github.com/carlosframework/platform/internal/applog" |
| + "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/telemetry" |
| +) |
| + |
| +func errorsURL(fx *consoleFixture, acct, app, query string) string { |
| + u := fx.ts.URL + "/api/cli/apps/" + acct + "/" + app + "/errors" |
| + if query != "" { |
| + u += "?" + query |
| + } |
| + return u |
| +} |
| + |
| +// seedErrors ships one coalesced edge error and, via seedLogChunks, one |
| +// error-level platform line and one warn-level app line. |
| +func seedErrors(t *testing.T, fx *consoleFixture, now time.Time) { |
| + t.Helper() |
| + seedLogChunks(t, fx, now) |
| + warn := []applog.Record{{TS: now.Add(-7 * time.Minute), Box: "box1", Host: "titogo.example.com", |
| + Stream: applog.StreamApp, Level: applog.LevelWarn, Msg: "slow query 4123ms", Seq: 150}} |
| + body, _ := applog.EncodeChunk(warn) |
| + if _, err := fx.rel.Put(context.Background(), applog.ChunkKey(release.OpsAccount, "titogo", "box1", applog.HourOf(warn[0].TS), 150, len(body)), body, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + e := []apperr.Record{{TS: now.Add(-3 * time.Minute), Box: "box1", Host: "titogo.example.com", |
| + Source: apperr.SourceEdge, Kind: apperr.KindHTTP, Name: "500 POST /checkout", Path: "/checkout", Status: 500, |
| + Release: "v3", FP: "edge500", FPV: 1, Count: 12, First: now.Add(-9 * time.Minute), Last: now.Add(-3 * time.Minute), Seq: 200}} |
| + eb, _ := apperr.EncodeChunk(e) |
| + if _, err := fx.rel.Put(context.Background(), apperr.ChunkKey(release.OpsAccount, "titogo", "box1", apperr.HourOf(e[0].TS), 200, len(eb)), eb, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| +} |
| + |
| +func TestAPIErrorsGroupsBothFamilies(t *testing.T) { |
| + fx, token := instanceFixture(t, "code-errors-1") |
| + now := time.Now().UTC() |
| + seedErrors(t, fx, now) |
| + get := func(query string) errorsResponse { |
| + t.Helper() |
| + status, _, body := rawReq(t, http.MethodGet, errorsURL(fx, fx.sqid, "titogo", query), token, "") |
| + if status != http.StatusOK { |
| + t.Fatalf("GET ?%s = %d: %s", query, status, body) |
| + } |
| + var resp errorsResponse |
| + if err := json.Unmarshal([]byte(body), &resp); err != nil { |
| + t.Fatal(err) |
| + } |
| + return resp |
| + } |
| + resp := get("") |
| + if len(resp.Groups) != 2 { |
| + t.Fatalf("groups = %+v", resp.Groups) |
| + } |
| + if resp.Groups[0].FP != "edge500" || resp.Groups[0].Count != 12 || resp.Groups[0].Releases[0] != "v3" { |
| + t.Fatalf("newest first, coalesced count: %+v", resp.Groups[0]) |
| + } |
| + if resp.Groups[1].Source != apperr.SourceLog || !strings.Contains(resp.Groups[1].Name, "wake failed") { |
| + t.Fatalf("log-derived group: %+v", resp.Groups[1]) |
| + } |
| + if resp.LogLevel != "error" || len(resp.Sources) != 2 { |
| + t.Fatalf("meta = %q %v", resp.LogLevel, resp.Sources) |
| + } |
| + // A warn read widens without changing the setting. |
| + if resp = get("level=warn"); len(resp.Groups) != 3 { |
| + t.Fatalf("level=warn groups = %d", len(resp.Groups)) |
| + } |
| + // Source and fp filters. |
| + if resp = get("source=edge"); len(resp.Groups) != 1 || resp.Groups[0].FP != "edge500" { |
| + t.Fatalf("source filter = %+v", resp.Groups) |
| + } |
| + if resp = get("fp=edge500"); len(resp.Records) != 1 || resp.Records[0].Status != 500 { |
| + t.Fatalf("fp expand = %+v", resp.Records) |
| + } |
| +} |
| + |
| +func TestAPITelemetryPutSetsLogLevel(t *testing.T) { |
| + fx, token := instanceFixture(t, "code-errors-2") |
| + now := time.Now().UTC() |
| + seedErrors(t, fx, now) |
| + // The instance token may read but not write settings: Operate only. |
| + status, _, body := rawReq(t, http.MethodPut, fx.ts.URL+"/api/cli/apps/"+fx.sqid+"/titogo/telemetry", token, `{"errors":{"log_level":"warn"}}`) |
| + if status != http.StatusForbidden && status != http.StatusNotFound { |
| + t.Fatalf("instance token PUT = %d %s", status, body) |
| + } |
| + op := operateToken(t, fx) // see note below |
| + status, _, body = rawReq(t, http.MethodPut, fx.ts.URL+"/api/cli/apps/"+fx.sqid+"/titogo/telemetry", op, `{"errors":{"log_level":"warn"}}`) |
| + if status != http.StatusOK { |
| + t.Fatalf("PUT = %d %s", status, body) |
| + } |
| + r, _ := telemetry.Resolve(context.Background(), fx.rel, fx.sqid, "titogo") |
| + if r.LogLevel != "warn" || r.LogLevelFrom != "app" { |
| + t.Fatalf("resolved = %+v", r) |
| + } |
| + status, _, body = rawReq(t, http.MethodGet, errorsURL(fx, fx.sqid, "titogo", ""), token, "") |
| + var resp errorsResponse |
| + json.Unmarshal([]byte(body), &resp) |
| + if status != 200 || resp.LogLevel != "warn" || len(resp.Groups) != 3 { |
| + t.Fatalf("after set: %d level=%q groups=%d", status, resp.LogLevel, len(resp.Groups)) |
| + } |
| + status, _, body = rawReq(t, http.MethodPut, fx.ts.URL+"/api/cli/apps/"+fx.sqid+"/titogo/telemetry", op, `{"errors":{"log_level":"loud"}}`) |
| + if status != http.StatusBadRequest { |
| + t.Fatalf("bad level = %d %s", status, body) |
| + } |
| +} |
| + |
| +func TestAppErrorsPage(t *testing.T) { |
| + fx, _ := instanceFixture(t, "code-errors-page") |
| + now := time.Now().UTC() |
| + seedErrors(t, fx, now) |
| + fx.setUser("paul@keymail.dev") |
| + c := newBrowser(t) |
| + signUp(t, fx, c, "code-errors-signin") |
| + status, _, body := get(t, c, fx.u("/apps/titogo/errors")) |
| + if status != http.StatusOK { |
| + t.Fatalf("errors page: %d %s", status, body) |
| + } |
| + for _, want := range []string{"500 POST /checkout", "wake failed", "carlos errors --app titogo", "Sources live: log, edge", "to the nearest flush"} { |
| + if !strings.Contains(body, want) { |
| + t.Errorf("page missing %q", want) |
| + } |
| + } |
| + if strings.Contains(body, "slow query") { |
| + t.Error("warn lines must not show at the default level") |
| + } |
| + status, _, body = get(t, c, fx.u("/apps/titogo/errors?level=warn")) |
| + if status != 200 || !strings.Contains(body, "slow query") { |
| + t.Fatalf("level=warn page: %d", status) |
| + } |
| + status, _, body = get(t, c, fx.u("/apps/titogo/errors?fp=edge500")) |
| + if status != 200 || !strings.Contains(body, "/checkout") || !strings.Contains(body, "v3") { |
| + t.Fatalf("expanded group: %d", status) |
| + } |
| + status, _, body = get(t, c, fx.u("/apps/titogo/errors?since=1ns")) |
| + if status != 200 || !strings.Contains(body, "No errors in this window") { |
| + t.Fatalf("empty window: %d", status) |
| + } |
| +} |
| +``` |
| + |
| +`operateToken(t, fx)`: find how existing tests mint a service token with the `Operate` role (grep `svcrole.Operate` in `internal/console/*_test.go`; there is a helper minting tokens with a role list for the apicli tests). Use that helper; if its name differs, use it directly in place of `operateToken`. If no such helper exists, mint through the same store call `instanceFixture` uses with `svcrole.Operate` substituted. |
| + |
| +- [ ] **Step 2: Run, expect compile failure** |
| + |
| +- [ ] **Step 3: `errorsview.go`** |
| + |
| +```go |
| +package console |
| + |
| +// The Errors tab's read (spec 2026-09-07 §3.2, §3.6): errors/ records |
| +// plus log lines at or above the resolved level, derived on read and |
| +// never stored twice, grouped by fingerprint. The tab and the CLI's API |
| +// share this function — the API is not an afterthought. |
| + |
| +import ( |
| + "context" |
| + "regexp" |
| + "sort" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/apperr" |
| + "github.com/carlosframework/platform/internal/applog" |
| + "github.com/carlosframework/platform/internal/telemetry" |
| +) |
| + |
| +// errorsWindow is one assembled read. |
| +type errorsWindow struct { |
| + Groups []apperr.Group |
| + Records []apperr.Record // when f.FP is set: that group's records, oldest first |
| + Truncated bool |
| + LogLevel string // resolved setting |
| + LogLevelFrom string |
| + Sources []string // sources with at least one record in the window |
| +} |
| + |
| +// readErrors assembles the window. level overrides the resolved setting |
| +// for THIS read only ("" = use the setting); it is what lets a member on |
| +// error widen one look to warn without changing anything (§8 ruling 3). |
| +func (s *Server) readErrors(ctx context.Context, sqid, app string, f apperr.Filter, level string) (errorsWindow, error) { |
| + var out errorsWindow |
| + res, err := telemetry.Resolve(ctx, s.cfg.Releases, sqid, app) |
| + if err != nil { |
| + return out, err |
| + } |
| + out.LogLevel, out.LogLevelFrom = res.LogLevel, res.LogLevelFrom |
| + if level == "" { |
| + level = res.LogLevel |
| + } |
| + stored, trimmed, err := apperr.ReadWindow(ctx, s.cfg.Releases, sqid, app, f, logsCollectCap) |
| + if err != nil { |
| + return out, err |
| + } |
| + out.Truncated = trimmed |
| + var derived []apperr.Record |
| + if f.Source == "" || f.Source == apperr.SourceLog { |
| + lf := applog.Filter{Since: f.Since, Until: f.Until, Host: f.Host, Level: level} |
| + lines, ltrim, err := applog.ReadWindow(ctx, s.cfg.Releases, sqid, app, lf, logsCollectCap) |
| + if err != nil { |
| + return out, err |
| + } |
| + out.Truncated = out.Truncated || ltrim |
| + for _, l := range lines { |
| + r := apperr.FromLog(l) |
| + if (f.FP == "" || r.FP == f.FP) && (f.Release == "" || r.Release == f.Release) { |
| + derived = append(derived, r) |
| + } |
| + } |
| + } |
| + all := apperr.MergeRecords(stored, derived) |
| + if f.FP != "" { |
| + out.Records = all |
| + } |
| + out.Groups = apperr.GroupRecords(all) |
| + seen := map[string]bool{} |
| + for _, r := range all { |
| + if !seen[r.Source] { |
| + seen[r.Source] = true |
| + out.Sources = append(out.Sources, r.Source) |
| + } |
| + } |
| + sort.Strings(out.Sources) |
| + return out, nil |
| +} |
| + |
| +// errorsFilterFromQuery reads the tab's and the API's shared params: |
| +// since/until/source/host/release/fp/level. The empty badParam means they |
| +// parsed. |
| +func errorsFilterFromQuery(q map[string][]string, now time.Time) (f apperr.Filter, level string, badParam string) { |
| + get := func(k string) string { |
| + if v := q[k]; len(v) > 0 { |
| + return v[0] |
| + } |
| + return "" |
| + } |
| + var err error |
| + if f.Since, err = applog.ParseTimeArg(get("since"), now); err != nil { |
| + return f, "", "since: " + err.Error() |
| + } |
| + if f.Until, err = applog.ParseTimeArg(get("until"), now); err != nil { |
| + return f, "", "until: " + err.Error() |
| + } |
| + if f.Since.IsZero() { |
| + f.Since = now.Add(-24 * time.Hour) |
| + } |
| + switch v := get("source"); v { |
| + case "", apperr.SourceLog, apperr.SourceEdge, apperr.SourceServer, apperr.SourceBrowser: |
| + f.Source = v |
| + default: |
| + return f, "", "source: want log, edge, server or browser" |
| + } |
| + f.Host, f.Release, f.FP = get("host"), get("release"), get("fp") |
| + if !validFP.MatchString(f.FP) { |
| + return f, "", "fp: want a fingerprint id" |
| + } |
| + if level = get("level"); level != "" && !telemetry.ValidLogLevel(level) { |
| + return f, "", "level: want error or warn" |
| + } |
| + return f, level, "" |
| +} |
| + |
| +var validFP = regexp.MustCompile(`^[A-Za-z0-9_.-]{0,64}$`) |
| +``` |
| + |
| +- [ ] **Step 4: `apierrors.go`** |
| + |
| +```go |
| +package console |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "log" |
| + "net/http" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/apperr" |
| + "github.com/carlosframework/platform/internal/telemetry" |
| +) |
| + |
| +// errorsResponse is the wire shape `carlos errors` and the tab read. |
| +type errorsResponse struct { |
| + Groups []apperr.Group `json:"groups"` |
| + Records []apperr.Record `json:"records,omitempty"` |
| + Truncated bool `json:"truncated"` |
| + LogLevel string `json:"log_level"` |
| + Sources []string `json:"sources"` |
| +} |
| + |
| +// handleAPIErrors answers `carlos errors`: the instance family's gate |
| +// (an app token may read its own errors), the shared filter, one read. |
| +func (s *Server) handleAPIErrors(w http.ResponseWriter, r *http.Request) { |
| + _, acct, app, ok := s.instanceGate(w, r) |
| + if !ok { |
| + return |
| + } |
| + f, level, bad := errorsFilterFromQuery(r.URL.Query(), time.Now()) |
| + if bad != "" { |
| + jsonError(w, http.StatusBadRequest, bad) |
| + return |
| + } |
| + ctx, cancel := context.WithTimeout(r.Context(), instanceStoreTimeout) |
| + defer cancel() |
| + win, err := s.readErrors(ctx, acct.Sqid, app, f, level) |
| + if err != nil { |
| + log.Printf("console: api errors %s/%s: %v", acct.Sqid, app, err) |
| + jsonError(w, http.StatusBadGateway, "could not read the errors") |
| + return |
| + } |
| + resp := errorsResponse{Groups: win.Groups, Records: win.Records, Truncated: win.Truncated, LogLevel: win.LogLevel, Sources: win.Sources} |
| + if resp.Groups == nil { |
| + resp.Groups = []apperr.Group{} |
| + } |
| + if resp.Sources == nil { |
| + resp.Sources = []string{} |
| + } |
| + writeJSON(w, http.StatusOK, resp) |
| +} |
| + |
| +// handleAPITelemetryPut sets the app-level telemetry object: today the |
| +// errors log level; the analytics keys arrive with step 2. Operate only — |
| +// an app's own token must not widen what its owner sees. |
| +func (s *Server) handleAPITelemetryPut(w http.ResponseWriter, r *http.Request) { |
| + principal, acct, app, ok := s.instanceGate(w, r) |
| + if !ok { |
| + return |
| + } |
| + var body telemetry.Settings |
| + if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&body); err != nil { |
| + jsonError(w, http.StatusBadRequest, "body: want {\"errors\":{\"log_level\":\"error|warn\"}}") |
| + return |
| + } |
| + if body.Errors == nil || !telemetry.ValidLogLevel(body.Errors.LogLevel) { |
| + jsonError(w, http.StatusBadRequest, "errors.log_level: want error or warn") |
| + return |
| + } |
| + if body.Analytics != nil { |
| + jsonError(w, http.StatusBadRequest, "analytics: not settable yet") |
| + return |
| + } |
| + ctx, cancel := context.WithTimeout(r.Context(), instanceStoreTimeout) |
| + defer cancel() |
| + if err := telemetry.Update(ctx, s.cfg.Releases, telemetry.AppKey(acct.Sqid, app), func(st *telemetry.Settings) { |
| + st.Errors = &telemetry.Errors{LogLevel: body.Errors.LogLevel} |
| + }); err != nil { |
| + log.Printf("console: api telemetry %s/%s: %v", acct.Sqid, app, err) |
| + jsonError(w, http.StatusBadGateway, "could not write the setting") |
| + return |
| + } |
| + s.audit(acct, app, principal, "errors-level", "set the Errors log level to "+body.Errors.LogLevel) |
| + writeJSON(w, http.StatusOK, map[string]string{"log_level": body.Errors.LogLevel}) |
| +} |
| +``` |
| + |
| +`principal` is the first return of `instanceGate` (a string naming the token's principal — check the type; if it is not a string suitable for `audit`'s `addr`, pass what the logs/instances handlers pass). |
| + |
| +Register in `authcli.go` beside the logs route: |
| + |
| +```go |
| + s.cliRoute("GET /api/cli/apps/{account}/{app}/errors", svcrole.Allow(svcrole.Instance, svcrole.Operate), s.handleAPIErrors) |
| + s.cliRoute("PUT /api/cli/apps/{account}/{app}/telemetry", svcrole.Allow(svcrole.Operate), s.handleAPITelemetryPut) |
| +``` |
| + |
| +- [ ] **Step 5: Page handler, fields, routes, templates** |
| + |
| +Page struct (near line 1370), add: |
| + |
| +```go |
| + // The Errors tab (errorsview.go). |
| + ErrWin errorsWindow |
| + ErrSince string // "1h" | "24h" | "168h" |
| + ErrSource string |
| + ErrLevel string // "" = the setting |
| + ErrHost string |
| + ErrFP string // expanded group |
| + ErrsErr string |
| +``` |
| + |
| +Routes (near 1050 and 1071): |
| + |
| +```go |
| + s.mux.HandleFunc("GET /a/{sqid}/apps/{app}/errors", s.handleAppErrors) |
| + s.mux.HandleFunc("POST /a/{sqid}/apps/{app}/errors/level", s.handleAppErrorLevel) |
| + s.mux.HandleFunc("POST /a/{sqid}/errors/level", s.handleAccountErrorLevel) |
| +``` |
| + |
| +Handlers (in `errorsview.go`): |
| + |
| +```go |
| +// handleAppErrors renders the Errors tab: the same read the API answers, |
| +// with the filters as GET params so a view is a URL. |
| +func (s *Server) handleAppErrors(w http.ResponseWriter, r *http.Request) { |
| + p, acct, ok := s.appPage(w, r, "errors") |
| + if !ok { |
| + return |
| + } |
| + q := r.URL.Query() |
| + p.ErrSince = q.Get("since") |
| + if p.ErrSince != "1h" && p.ErrSince != "168h" { |
| + p.ErrSince = "24h" |
| + } |
| + q.Set("since", p.ErrSince) |
| + f, level, bad := errorsFilterFromQuery(q, time.Now()) |
| + if bad != "" { |
| + f, level, _ = errorsFilterFromQuery(map[string][]string{"since": {p.ErrSince}}, time.Now()) |
| + } |
| + p.ErrSource, p.ErrLevel, p.ErrHost, p.ErrFP = f.Source, level, f.Host, f.FP |
| + if s.cfg.Releases != nil { |
| + ctx, cancel := context.WithTimeout(r.Context(), instanceStoreTimeout) |
| + defer cancel() |
| + win, err := s.readErrors(ctx, acct.Sqid, p.AppName, f, level) |
| + if err != nil { |
| + log.Printf("console: app errors %s/%s: %v", acct.Sqid, p.AppName, err) |
| + p.ErrsErr = "The errors could not be read just now — try again in a moment." |
| + } |
| + p.ErrWin = win |
| + } |
| + s.render(w, http.StatusOK, "app_errors", p) |
| +} |
| + |
| +// handleAppErrorLevel is the Settings tab's form: owner, CSRF, one key. |
| +func (s *Server) handleAppErrorLevel(w http.ResponseWriter, r *http.Request) { |
| + addr, raw, acct, role, app, ok := s.accountApp(w, r) |
| + if !ok { |
| + return |
| + } |
| + if subtle.ConstantTimeCompare([]byte(r.FormValue("csrf")), []byte(raw)) != 1 { |
| + s.renderMessage(w, r, http.StatusForbidden, "Refused", "Cross-site request blocked.") |
| + return |
| + } |
| + if role != RoleOwner { |
| + http.NotFound(w, r) |
| + return |
| + } |
| + level := r.FormValue("level") |
| + back := "/a/" + url.PathEscape(acct.Sqid) + "/apps/" + url.PathEscape(app) + "/settings" |
| + if !telemetry.ValidLogLevel(level) { |
| + s.setFlash(w, "err", "Choose error or warn.") |
| + http.Redirect(w, r, back, http.StatusSeeOther) |
| + return |
| + } |
| + if err := telemetry.Update(r.Context(), s.cfg.Releases, telemetry.AppKey(acct.Sqid, app), func(st *telemetry.Settings) { |
| + st.Errors = &telemetry.Errors{LogLevel: level} |
| + }); err != nil { |
| + log.Printf("console: errors level %s/%s: %v", acct.Sqid, app, err) |
| + s.setFlash(w, "err", "The setting could not be saved — try again in a moment.") |
| + http.Redirect(w, r, back, http.StatusSeeOther) |
| + return |
| + } |
| + s.audit(acct, app, addr, "errors-level", "set the Errors log level to "+level) |
| + s.setFlash(w, "ok", "Errors now include "+level+" lines and above for this app.") |
| + http.Redirect(w, r, back, http.StatusSeeOther) |
| +} |
| + |
| +// handleAccountErrorLevel is the account settings form: the default every |
| +// app without its own setting inherits. |
| +func (s *Server) handleAccountErrorLevel(w http.ResponseWriter, r *http.Request) { |
| + addr, raw, acct, role, ok := s.account(w, r) |
| + if !ok { |
| + return |
| + } |
| + if subtle.ConstantTimeCompare([]byte(r.FormValue("csrf")), []byte(raw)) != 1 { |
| + s.renderMessage(w, r, http.StatusForbidden, "Refused", "Cross-site request blocked.") |
| + return |
| + } |
| + if role != RoleOwner { |
| + http.NotFound(w, r) |
| + return |
| + } |
| + level := r.FormValue("level") |
| + back := "/a/" + url.PathEscape(acct.Sqid) + "/settings" |
| + if !telemetry.ValidLogLevel(level) { |
| + s.setFlash(w, "err", "Choose error or warn.") |
| + http.Redirect(w, r, back, http.StatusSeeOther) |
| + return |
| + } |
| + if err := telemetry.Update(r.Context(), s.cfg.Releases, telemetry.AccountKey(acct.Sqid), func(st *telemetry.Settings) { |
| + st.Errors = &telemetry.Errors{LogLevel: level} |
| + }); err != nil { |
| + log.Printf("console: account errors level %s: %v", acct.Sqid, err) |
| + s.setFlash(w, "err", "The setting could not be saved — try again in a moment.") |
| + http.Redirect(w, r, back, http.StatusSeeOther) |
| + return |
| + } |
| + s.audit(acct, "", addr, "errors-level", "set the account's Errors log level to "+level) |
| + s.setFlash(w, "ok", "Errors now include "+level+" lines and above for every app without its own setting.") |
| + http.Redirect(w, r, back, http.StatusSeeOther) |
| +} |
| +``` |
| + |
| +Confirm `s.account`'s return arity by reading `handleAccountSettings` (`addr, raw, acct, role, ok`). Add `p.ErrLevelSetting telemetry.Resolved` to the page struct and populate it in `handleAppSettings` (`p.ErrLevelSetting, _ = telemetry.Resolve(ctx, s.cfg.Releases, acct.Sqid, app)`) and in `handleAccountSettings` (`Resolve(ctx, s.cfg.Releases, acct.Sqid, "")` — with an empty app name `AppKey` reads a key that does not exist, which returns defaults; that is acceptable for the account page and should be noted in a comment). |
| + |
| +`appnav.html`: insert after the Logs link: |
| + |
| +```html |
| + <a href="{{.Nav}}/apps/{{.AppName}}/errors"{{if eq .ActiveTab "errors"}} aria-current="page"{{end}}>Errors</a> |
| +``` |
| + |
| +`app_errors.html`: |
| + |
| +```html |
| +{{define "app_errors"}}{{template "top" .}} |
| +{{if .Flash}}<p class="banner">{{.Flash}}</p>{{end}} |
| +{{if .FlashErr}}<p class="banner error">{{.FlashErr}}</p>{{end}} |
| +{{template "appnav" .}} |
| + |
| +<h2>Errors</h2> |
| +{{if .ErrsErr}}<p class="banner error">{{.ErrsErr}}</p>{{end}} |
| +<form method="get" class="small"> |
| + <label>from the last |
| + <select name="since"> |
| + <option value="1h"{{if eq .ErrSince "1h"}} selected{{end}}>hour</option> |
| + <option value="24h"{{if eq .ErrSince "24h"}} selected{{end}}>day</option> |
| + <option value="168h"{{if eq .ErrSince "168h"}} selected{{end}}>week</option> |
| + </select> |
| + </label> |
| + <label>source |
| + <select name="source"> |
| + <option value=""{{if eq .ErrSource ""}} selected{{end}}>all</option> |
| + <option value="log"{{if eq .ErrSource "log"}} selected{{end}}>log</option> |
| + <option value="edge"{{if eq .ErrSource "edge"}} selected{{end}}>edge</option> |
| + <option value="server"{{if eq .ErrSource "server"}} selected{{end}}>server</option> |
| + <option value="browser"{{if eq .ErrSource "browser"}} selected{{end}}>browser</option> |
| + </select> |
| + </label> |
| + <label>log level |
| + <select name="level"> |
| + <option value=""{{if eq .ErrLevel ""}} selected{{end}}>setting ({{.ErrWin.LogLevel}})</option> |
| + <option value="error"{{if eq .ErrLevel "error"}} selected{{end}}>error</option> |
| + <option value="warn"{{if eq .ErrLevel "warn"}} selected{{end}}>warn and above</option> |
| + </select> |
| + </label> |
| + <label>host <input name="host" value="{{.ErrHost}}" size="24"></label> |
| + <button type="submit">Update</button> |
| +</form> |
| +<p class="small">Sources live: {{if .ErrWin.Sources}}{{range $i, $s := .ErrWin.Sources}}{{if $i}}, {{end}}{{$s}}{{end}}{{else}}none yet{{end}}. |
| +Counts are to the nearest flush. Messages are redacted for addresses and credentials before they are stored; treat that as a floor, not a guarantee.</p> |
| + |
| +{{if .ErrFP}} |
| +{{with .ErrWin.Records}} |
| +<h3>Occurrences of <code>{{$.ErrFP}}</code></h3> |
| +<p class="small"><a href="?since={{$.ErrSince}}">All groups</a></p> |
| +{{range .}} |
| +<div class="tablewrap"><table> |
| + <tr><th>Last</th><td>{{.Last.UTC.Format "2006-01-02 15:04:05"}}</td></tr> |
| + <tr><th>Count</th><td>{{.Count}}</td></tr> |
| + <tr><th>Source</th><td>{{.Source}}{{if .Kind}} / {{.Kind}}{{end}}</td></tr> |
| + <tr><th>Host</th><td>{{.Host}}</td></tr> |
| + {{if .Release}}<tr><th>Release</th><td>{{.Release}}</td></tr>{{end}} |
| + {{if .Path}}<tr><th>Path</th><td><code>{{.Path}}</code>{{if .Status}} → {{.Status}}{{end}}</td></tr>{{end}} |
| + <tr><th>Message</th><td>{{.Name}}</td></tr> |
| + {{if .Stack}}<tr><th>Stack</th><td><pre>{{.Stack}}</pre></td></tr>{{end}} |
| + {{if .Fields}}<tr><th>Fields</th><td>{{range $k, $v := .Fields}}<code>{{$k}}={{$v}}</code> {{end}}</td></tr>{{end}} |
| + <tr><th>Logs</th><td><a href="{{$.Nav}}/apps/{{$.AppName}}/logs?since=1h&host={{.Host}}">around this time</a></td></tr> |
| +</table></div> |
| +{{end}} |
| +{{else}}<p>No occurrences of this group in the window.</p>{{end}} |
| +{{else if .ErrWin.Groups}} |
| +{{if .ErrWin.Truncated}}<p class="small">Showing the most recent 50,000 records of a larger window — counts are within what was read.</p>{{end}} |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Last seen</th><th>Count</th><th>Source</th><th>Error</th><th>Hosts</th><th>Releases</th></tr> |
| + {{range .ErrWin.Groups}}<tr> |
| + <td>{{.Last.UTC.Format "2006-01-02 15:04"}}</td> |
| + <td>{{.Count}}{{if eq .FP "_overflow"}} <span class="small" title="more kinds than the hourly cap; occurrences">(overflow)</span>{{end}}</td> |
| + <td>{{.Source}}{{if .Level}} <span class="small">{{.Level}}</span>{{end}}</td> |
| + <td><a href="?since={{$.ErrSince}}&fp={{.FP}}{{if $.ErrLevel}}&level={{$.ErrLevel}}{{end}}">{{.Name}}</a></td> |
| + <td>{{range $i, $h := .Hosts}}{{if $i}}, {{end}}{{$h}}{{end}}</td> |
| + <td>{{range $i, $r := .Releases}}{{if $i}}, {{end}}<code>{{$r}}</code>{{end}}</td> |
| + </tr>{{end}} |
| +</table> |
| +</div> |
| +<p class="small">Times are UTC. The same view from a terminal: <code>carlos errors --app {{.AppName}} --since {{.ErrSince}}</code>.</p> |
| +{{else if not .ErrsErr}} |
| +<p>No errors in this window.</p> |
| +<p class="small">Log lines at <code>{{.ErrWin.LogLevel}}</code> and above and responses of 500 or higher appear here on their own; a JavaScript or server-side error needs the hook (coming in a later step). The level is set on the Settings tab. The same view from a terminal: <code>carlos errors --app {{.AppName}}</code>.</p> |
| +{{end}} |
| +{{template "bottom" .}}{{end}} |
| +``` |
| + |
| +`app_settings.html`: before the Feature flags section add: |
| + |
| +```html |
| +<h2>Errors</h2> |
| +<form method="post" action="{{.Base}}/apps/{{.Detail.App}}/errors/level" class="settings-row"> |
| + <input type="hidden" name="csrf" value="{{.CSRF}}"> |
| + <label>Log lines counted as errors: |
| + <select name="level"> |
| + <option value="error"{{if eq .ErrLevelSetting.LogLevel "error"}} selected{{end}}>error and above</option> |
| + <option value="warn"{{if eq .ErrLevelSetting.LogLevel "warn"}} selected{{end}}>warn and above</option> |
| + </select> |
| + </label> |
| + <button type="submit">Save</button> |
| + <span class="small">currently {{.ErrLevelSetting.LogLevel}}, from the {{.ErrLevelSetting.LogLevelFrom}}</span> |
| +</form> |
| +<p class="small">The Errors tab always lets you widen one look to warn without changing this.</p> |
| +``` |
| + |
| +Check the CSRF field name and value the rename form uses (`{{.CSRF}}`, `{{.Raw}}` or similar) and match it exactly. |
| + |
| +`account_settings.html`: before "Delete this account" add the same form posting to `{{.Base}}/errors/level`, with the copy "for every app without its own setting". |
| + |
| +- [ ] **Step 6: Run console tests** |
| + |
| +Run: `go build ./... && go vet ./internal/console/ && go test ./internal/console/ -count=1` |
| +Expected: PASS (the whole package; template parse errors surface at server construction in every test). |
| + |
| +- [ ] **Step 7: Commit** |
| + |
| +```bash |
| +git add internal/console |
| +git commit -m "console: Errors tab, errors API, and the log-level setting at both levels (spec §3.6) |
| + |
| +Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 10: `carlos errors` and the docs heading |
| + |
| +**Files:** |
| +- Create: `errors_cmd.go` |
| +- Modify: `apimode.go` (beside `logsGet`), `main.go:80` (dispatch) and the help text near line 204 |
| +- Modify: `docs/site/cli.md` after `### carlos logs` |
| +- Test: `errors_cmd_test.go` (format only) and the docsite gate |
| + |
| +**Interfaces:** |
| +- Consumes: `requireAPISession`, `resolvedAccount`, `resolvedApp`, `appFlag`, `flagUsage`, `apiPath`, `getJSON`, `putJSON` (check its name; `apimode.go` has a JSON PUT/POST helper used by `features set`), `colorFor`, `mutedf`, `stdout`. |
| +- Produces: `runErrors(args []string)`; `(*apiSession).errorsGet(ctx, a apiAcct, app string, q url.Values) (apiErrorsResponse, error)`; `(*apiSession).telemetryPut(ctx, a apiAcct, app string, body any) error`; `printErrorGroups(w io.Writer, groups []apperr.Group)`; `printErrorRecords(w io.Writer, recs []apperr.Record)`. |
| + |
| +- [ ] **Step 1: Failing format test** — `errors_cmd_test.go` |
| + |
| +```go |
| +package main |
| + |
| +import ( |
| + "bytes" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/apperr" |
| +) |
| + |
| +func TestPrintErrorGroups(t *testing.T) { |
| + now := time.Date(2026, 9, 7, 10, 0, 0, 0, time.UTC) |
| + var buf bytes.Buffer |
| + printErrorGroups(&buf, []apperr.Group{ |
| + {FP: "abc123", Source: "edge", Name: "500 POST /checkout", Count: 12, Last: now, Hosts: []string{"a.example"}, Releases: []string{"v3"}}, |
| + {FP: "_overflow", Source: "edge", Name: "500 GET /z", Count: 40, Last: now}, |
| + }) |
| + out := buf.String() |
| + for _, want := range []string{"2026-09-07 10:00", "12", "edge", "500 POST /checkout", "abc123", "a.example", "v3", "overflow"} { |
| + if !strings.Contains(out, want) { |
| + t.Errorf("output missing %q:\n%s", want, out) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Implement `errors_cmd.go`** |
| + |
| +```go |
| +// `carlos errors` — an app's error groups with no box access (spec |
| +// 2026-09-07-errors-and-analytics-design.md §3.6): log lines at the |
| +// configured level and edge-observed 5xx, coalesced by fingerprint, read |
| +// through the console API. API mode only, like `carlos logs`. |
| +package main |
| + |
| +import ( |
| + "context" |
| + "flag" |
| + "fmt" |
| + "io" |
| + "log" |
| + "net/url" |
| + "sort" |
| + "strconv" |
| + |
| + "github.com/carlosframework/platform/internal/apperr" |
| +) |
| + |
| +const errorsUsage = `usage: carlos errors --app <a> [flags] |
| + |
| +Lists error groups, newest first: one row per fingerprint with its count in |
| +the window, the source it came from (log, edge, server, browser), the hosts |
| +and releases it was seen on. --fp <id> expands one group to its occurrences. |
| +--set-level error|warn changes which log lines count for this app. |
| +` |
| + |
| +func runErrors(args []string) { |
| + fs := flag.NewFlagSet("errors", flag.ExitOnError) |
| + app := appFlag(fs, "app whose errors to read") |
| + since := fs.String("since", "24h", "window start: duration back from now, or RFC3339") |
| + until := fs.String("until", "", "window end (default: now)") |
| + source := fs.String("source", "", "only one source: log | edge | server | browser") |
| + level := fs.String("level", "", "log lines at this level and above for this read: error | warn (default: the app's setting)") |
| + host := fs.String("host", "", "only one instance (route host)") |
| + rel := fs.String("release", "", "only one release") |
| + fp := fs.String("fp", "", "expand one group: its occurrences, oldest first") |
| + setLevel := fs.String("set-level", "", "set the app's log level for errors: error | warn") |
| + consoleFlag := fs.String("console", "", "console to read through (default: the only logged-in one)") |
| + acctRaw := fs.String("account", "", "account (sqid or name) the app belongs to (default: your only account, or the configured/chosen one)") |
| + acctIDRaw := fs.String("account-id", "", "account sqid, matched as a sqid ONLY — supersedes --account") |
| + fs.Usage = flagUsage(fs, errorsUsage) |
| + fs.Parse(args) |
| + |
| + sess := requireAPISession("errors", *consoleFlag) |
| + acct := resolvedAccount(sess, acctRaw, acctIDRaw) |
| + app2 := resolvedApp(sess, acct, *app, errorsUsage) |
| + ctx := context.Background() |
| + |
| + if *setLevel != "" { |
| + if *setLevel != "error" && *setLevel != "warn" { |
| + log.Fatalf("errors: --set-level wants error or warn") |
| + } |
| + if err := sess.telemetryPut(ctx, acct, app2, map[string]any{"errors": map[string]string{"log_level": *setLevel}}); err != nil { |
| + log.Fatalf("errors: %v", err) |
| + } |
| + fmt.Fprintf(stdout, "Errors for %s now include %s lines and above.\n", app2, *setLevel) |
| + return |
| + } |
| + |
| + q := url.Values{} |
| + for k, v := range map[string]string{"since": *since, "until": *until, "source": *source, "level": *level, "host": *host, "release": *rel, "fp": *fp} { |
| + if v != "" { |
| + q.Set(k, v) |
| + } |
| + } |
| + res, err := sess.errorsGet(ctx, acct, app2, q) |
| + if err != nil { |
| + log.Fatalf("errors: %v", err) |
| + } |
| + if *fp != "" { |
| + printErrorRecords(stdout, res.Records) |
| + if len(res.Records) == 0 { |
| + mutedf(stdout, "no occurrences of %s in this window", *fp) |
| + } |
| + return |
| + } |
| + printErrorGroups(stdout, res.Groups) |
| + if len(res.Groups) == 0 { |
| + mutedf(stdout, "no errors in this window (log level %s; sources live: %s) — try --since 168h or --level warn", res.LogLevel, joinOr(res.Sources, "none yet")) |
| + } else { |
| + mutedf(stdout, "log level %s; sources live: %s; counts are to the nearest flush", res.LogLevel, joinOr(res.Sources, "none")) |
| + } |
| + if res.Truncated { |
| + mutedf(stdout, "(the window held more than could be read; counts are within what was read — narrow it)") |
| + } |
| +} |
| + |
| +func joinOr(xs []string, empty string) string { |
| + if len(xs) == 0 { |
| + return empty |
| + } |
| + out := xs[0] |
| + for _, x := range xs[1:] { |
| + out += ", " + x |
| + } |
| + return out |
| +} |
| + |
| +// printErrorGroups renders one row per group: last seen, count, source, |
| +// message, then the fingerprint, hosts and releases dimmed. |
| +func printErrorGroups(w io.Writer, groups []apperr.Group) { |
| + c := colorFor(w) |
| + for _, g := range groups { |
| + count := strconv.Itoa(g.Count) |
| + if g.FP == apperr.OverflowFP { |
| + count += " (overflow)" |
| + } |
| + line := c.muted(g.Last.UTC().Format("2006-01-02 15:04")) + " " + c.head(fmt.Sprintf("%6s", count)) + " " + |
| + c.muted(fmt.Sprintf("%-7s", g.Source)) + " " + c.bad(g.Name) |
| + line += "\n" + " " + c.muted("fp="+g.FP) |
| + if len(g.Hosts) > 0 { |
| + line += c.muted(" hosts=" + joinOr(g.Hosts, "")) |
| + } |
| + if len(g.Releases) > 0 { |
| + line += c.muted(" releases=" + joinOr(g.Releases, "")) |
| + } |
| + fmt.Fprintln(w, line) |
| + } |
| +} |
| + |
| +// printErrorRecords renders one group's occurrences, oldest first. |
| +func printErrorRecords(w io.Writer, recs []apperr.Record) { |
| + c := colorFor(w) |
| + for _, r := range recs { |
| + fmt.Fprintf(w, "%s %s %s %s\n", c.muted(r.Last.UTC().Format("2006-01-02 15:04:05")), |
| + c.head(fmt.Sprintf("x%d", r.Count)), c.muted(r.Host), c.bad(r.Name)) |
| + if r.Path != "" { |
| + fmt.Fprintf(w, " %s\n", c.muted(fmt.Sprintf("%s → %d", r.Path, r.Status))) |
| + } |
| + if r.Release != "" { |
| + fmt.Fprintf(w, " %s\n", c.muted("release "+r.Release)) |
| + } |
| + if r.Stack != "" { |
| + fmt.Fprintf(w, " %s\n", r.Stack) |
| + } |
| + keys := make([]string, 0, len(r.Fields)) |
| + for k := range r.Fields { |
| + keys = append(keys, k) |
| + } |
| + sort.Strings(keys) |
| + for _, k := range keys { |
| + fmt.Fprintf(w, " %s\n", c.muted(k+"="+r.Fields[k])) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +In `apimode.go`, beside `logsGet`: |
| + |
| +```go |
| +type apiErrorsResponse struct { |
| + Groups []apperr.Group `json:"groups"` |
| + Records []apperr.Record `json:"records"` |
| + Truncated bool `json:"truncated"` |
| + LogLevel string `json:"log_level"` |
| + Sources []string `json:"sources"` |
| +} |
| + |
| +func (s *apiSession) errorsGet(ctx context.Context, a apiAcct, app string, query url.Values) (apiErrorsResponse, error) { |
| + path := apiPath(a, app, "/errors") |
| + if enc := query.Encode(); enc != "" { |
| + if strings.Contains(path, "?") { |
| + path += "&" + enc |
| + } else { |
| + path += "?" + enc |
| + } |
| + } |
| + var res apiErrorsResponse |
| + status, body, err := s.getJSON(ctx, path) |
| + if err != nil { |
| + return res, err |
| + } |
| + if status != http.StatusOK { |
| + return res, apiErrorFromBody(status, body) |
| + } |
| + if err := json.Unmarshal(body, &res); err != nil { |
| + return res, fmt.Errorf("errors: unreadable response: %w", err) |
| + } |
| + return res, nil |
| +} |
| + |
| +func (s *apiSession) telemetryPut(ctx context.Context, a apiAcct, app string, body any) error { |
| + status, resp, err := s.sendJSON(ctx, http.MethodPut, apiPath(a, app, "/telemetry"), body) // use apimode.go's existing JSON-body sender; name it as it is there |
| + if err != nil { |
| + return err |
| + } |
| + if status != http.StatusOK { |
| + return apiErrorFromBody(status, resp) |
| + } |
| + return nil |
| +} |
| +``` |
| + |
| +Find the existing JSON-body request helper in `apimode.go` (grep `func (s \*apiSession) .*JSON` — `postJSON`/`doJSON`/`sendJSON`) and use its real name and signature. |
| + |
| +`main.go`: add `case "errors": runErrors(args)` beside `logs`, and in the help block near line 204 a line: ` errors error groups for an app: log lines at the set level plus 5xx the edge saw`. |
| + |
| +`docs/site/cli.md` after the `### carlos logs` section: |
| + |
| +```markdown |
| +### carlos errors |
| + |
| +One row per distinct error, newest first, with how many times it happened |
| +in the window and where. Two sources need no wiring at all: log lines the |
| +app tagged `error` (or `warn`, if you set the level) and responses of 500 |
| +or above that the edge saw. Server-side and browser hooks come later. |
| + |
| +```sh |
| +carlos errors --app hello --since 24h |
| +carlos errors --app hello --fp 3f2a9c1b5e7d0a4c |
| +carlos errors --app hello --set-level warn |
| +``` |
| + |
| +`--fp` expands one group to its occurrences. `--level warn` widens a single |
| +read without changing the setting; `--set-level` changes it for the app. |
| +The same view is the Errors tab on the app's dashboard. |
| +``` |
| + |
| +- [ ] **Step 3: Run the CLI test, docsite gate, whole build** |
| + |
| +Run: `go build ./... && go test . -run TestPrintErrorGroups -count=1 && go test ./internal/docsite/ -count=1` |
| +Expected: PASS; the docsite gate passes with the new heading (and would fail without it — try removing the heading once to see the failure, then restore it). |
| + |
| +- [ ] **Step 4: Commit** |
| + |
| +```bash |
| +git add errors_cmd.go errors_cmd_test.go apimode.go main.go docs/site/cli.md |
| +git commit -m "carlos errors: the app's error groups from a terminal, and the log-level setting |
| + |
| +Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 11: Whole-tree gate, cross-compile, push |
| + |
| +**Files:** none new. |
| + |
| +- [ ] **Step 1: Full gate** |
| + |
| +Run: |
| +```bash |
| +go build ./... && go vet ./... && gofmt -l . && go test ./... -count=1 |
| +GOOS=darwin GOARCH=arm64 go build ./... && GOOS=windows GOARCH=amd64 go build ./... |
| +``` |
| +Expected: all clean; `gofmt -l .` prints nothing. |
| + |
| +- [ ] **Step 2: Spec status line** |
| + |
| +In `docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md`, change the `**Status:**` line to `approved (Paul, 2026-09-07: "go ahead with that and plan, build review cycle overnight"); step 1 implemented on branch spec-errors-analytics, plan docs/superpowers/plans/2026-09-07-errors-step1-log-and-edge.md.` Commit with the plan file: |
| + |
| +```bash |
| +git add docs/superpowers |
| +git commit -m "spec: approved; step 1 plan |
| + |
| +Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| +``` |
| + |
| +- [ ] **Step 3: Push and describe** |
| + |
| +```bash |
| +git push origin spec-errors-analytics |
| +amadan branch describe carlos/platform spec-errors-analytics -body - < <description file> |
| +amadan ci status carlos/platform -branch spec-errors-analytics |
| +``` |
| + |
| +Do not merge. The branch waits for Paul's review of the diff. |
| + |
| +## Self-review against the spec |
| + |
| +- §3.1 shape: Task 3 `Record` carries every field but `ua`/`referer` (browser/edge analytics fields; step 2/3). Sanitisation: Task 3/4. ✔ |
| +- §3.2 log source read-derived: Task 9 `readErrors`. Edge errors with activator suppression: Task 7. Fingerprint rules: Task 3. Coalescing cap per app-hour on `(fp, host, release)` with `_overflow`: Task 4. Storage family and consumer list (`ChunkKeySize`, `HourNameOfDir`, `chunkPrune`, `aclWriteAppFamily`): Tasks 1, 2, 6. ✔ |
| +- §3.3 observing writer, unconditional, status routes excluded, 1xx ignored, implicit 200, Content-Type latched: Task 7. Settings object and resolution: Task 8. ✔ |
| +- §3.6 tab (window 1h/24h/7d, source/level/host filters, expand, coverage line, "nearest flush", collect-cap disclosure), API paths and roles, CLI flags including `-level`, settings at both levels, retention via `chunkPrune`: Tasks 6, 9, 10. Tombstone `DELETE …/errors/{fp}` is deferred to a follow-up branch and is called out in the branch description. `-release` filter on the CLI: Task 10 (`--release`). ✔ |
| +- §4 hibernation: nothing here calls the activator; Task 7's flag path is the only touch and it is a context value. ✔ |
| +- §6 tests: sanitisation, fingerprint, coalescing, log-derived rows, edge errors on the plain path, chunk family byte-identity, store door, observing writer, settings resolution, console gates (page 404 for non-member is inherited from `appPage`; add one `get` as a non-member in `TestAppErrorsPage` if the logs page test has an equivalent — mirror it), docsite. ✔ |
| +- Type consistency: `apperr.Filter` fields (`Since, Until, Host, Source, Release, FP`) are used identically in Tasks 4 and 9; `errorsResponse` fields match `apiErrorsResponse`; `chunkPrune(ctx, s, fam, box, k, now)` is called with that order in Tasks 6 and its tests. ✔ |