amadan
spec: carlos logs — per-app logs with no box access (#131, #78)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+212
−0
docs/superpowers/specs/2026-08-10-carlos-logs-design.md+212 −0
diff --git a/docs/superpowers/specs/2026-08-10-carlos-logs-design.md b/docs/superpowers/specs/2026-08-10-carlos-logs-design.md| new file mode 100644 |
| index 000000000..4223853d7 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-10-carlos-logs-design.md |
| @@ -0,0 +1,212 @@ |
| +# `carlos logs` — per-app logs with no box access |
| + |
| +2026-08-10. Closes the box-local-only debugging gap (#131, #78): a member |
| +whose app is failing gets exactly one string from the edge ("instance failed |
| +to wake") while the reason sits in a journal they cannot read. Paul's rule |
| +(CLAUDE.md 2026-08-08) is the bar: the command alone must complete the job |
| +for a user with zero infra access. |
| + |
| +## 1. Jobs, staged |
| + |
| +Three jobs, one surface, built in stages: |
| + |
| +1. **Diagnose a broken app** (stage 1, this spec's build): recent app output |
| + plus platform-side reasons — wake errors, restart actions — merged into |
| + one timeline. |
| +2. **Watch a deploy land** (stage 1): `carlos logs -f` follows live during |
| + ship/restart. |
| +3. **Everyday observability** (stages 2–3): edge request lines, unit-tenant |
| + journals, release-history join, longer retention knobs. |
| + |
| +The full stream vocabulary is fixed now so later stages are additive: |
| + |
| +| stream | contents | stage | |
| +|------------|---------------------------------------------------|-------| |
| +| `app` | the app process's own stdout/stderr | 1 (exec-backed), 2 (unit tenants via journald read) | |
| +| `platform` | wake attempts/failures, restart bounces, adopt/stage decisions, config sync, hibernation | 1 (agent-owned events), 2 (root adopt pass events — separate process, needs its own writer identity) | |
| +| `edge` | per-request access lines (host, method, path, status, duration, bytes; **no client IP** in the member-facing record) | 2 — does not exist today; new logging with its own volume cap | |
| +| `release` | ship/promote/adopt history | 3 — a read-time join over existing bucket records (manifests, channel pointers, tags), not a shipped stream | |
| + |
| +Static/bake apps have no process; for them the surface is `platform` + |
| +(later) `edge` lines only, and the CLI says so rather than showing an |
| +empty screen. |
| + |
| +## 2. Architecture |
| + |
| +One new pure package, `internal/applog` (record, sink, chunk codec, |
| +filter/merge — no I/O, no clock of its own), with thin plumbing around it. |
| +Deliberately the `internal/metrics` / `internal/host/metrics_loop.go` split. |
| + |
| +``` |
| +exec child stdout/stderr ─┐ |
| +agent platform events ────┼→ applog.Sink ─→ ship loop ─→ apps/<acct>/<app>/logs/<box>/<hour>/<seq>.jsonl.gz |
| +(edge lines, stage 2) ────┘ (ring, budget, │ |
| + drop honesty) ▼ |
| + console: merge boxes, filter, page |
| + │ |
| + ▼ |
| + carlos logs / console app tab |
| +``` |
| + |
| +Boxes push; the console never reaches into a box — the invariant `carlos |
| +restart` was built around. Works identically for customer fleets (they |
| +already write into the deployment bucket). Survives box death and |
| +hibernation: whatever shipped before the box died is readable. |
| + |
| +Transport choice (over on-demand rendezvous): rendezvous needs a *live* box |
| +— no logs precisely when the box is the problem — and gives no history; the |
| +box would need a local ring anyway. Continuous shipping serves all three |
| +jobs and reuses a flush loop this repo has already debugged once. |
| + |
| +## 3. Storage layout |
| + |
| +`apps/<acct>/<app>/logs/<box>/<YYYY-MM-DDTHH>/<seq>.jsonl.gz` |
| + |
| +- **Per-app, not per-box** (metrics is per-box because the host is its |
| + subject). The subject here is the app and the reader is its owner, so the |
| + console's per-(account, app) owner gate maps onto the prefix — no |
| + fleet-wide scan per read. App trash/purge sweeps logs for free (they live |
| + under `release.AppPrefix`). |
| +- **Hour directories**: retention is a prefix delete; `-since` skips whole |
| + prefixes unlisted. |
| +- **One object per flush, never appended or rewritten.** No S3 append, and |
| + the agent must not read-modify-write keys another writer could touch. |
| +- `<box>` is `status.Label(readMachineID())` — the identity metrics already |
| + publishes under. `<seq>` is the first record's per-box sequence number, |
| + zero-padded to 12 digits so lexical order is numeric order. |
| +- Chunks are gzipped JSONL, one record per line. |
| + |
| +## 4. Record shape (wire) |
| + |
| +```json |
| +{"ts":"2026-08-10T14:03:11.482913Z","box":"carlos-tito-ie", |
| + "host":"lpay.bde.oncarlos.com","stream":"app","level":"error", |
| + "msg":"bind: address already in use","fields":{"request_id":"a1b2"}, |
| + "seq":41822} |
| +``` |
| + |
| +- `ts` — box clock, UTC RFC3339Nano. Cross-box order is best-effort by |
| + `ts` with `(box, seq)` tiebreak; the spec promises per-box order only. |
| +- `host` — the route host; how owners already name instances. |
| +- `stream` — `app|platform|edge|release` (vocabulary fixed above). |
| +- `level` — set for platform/edge lines and for app lines **only when the |
| + app's own JSON carried one**. Never guessed from text: a guessed level |
| + that drops the crash line under `-level error` is worse than no flag. |
| +- `fields` — present only when the app line parsed as a flat-ish JSON |
| + object; enables `-field k=v` from day one. Non-JSON output is opaque |
| + `msg`, untouched. |
| +- `seq` — monotonic per box process; makes the follow cursor exact and |
| + drops detectable. |
| +- Additive wire shape (repo rule): every field but `ts` and `msg` optional; |
| + unknown fields ignored on read; anything written before a change must |
| + still parse. |
| +- One synthetic record shape: when the sink drops lines (budget/rate), it |
| + emits `{"stream":"platform","level":"warn","msg":"dropped N app log |
| + line(s) (rate cap)"}` — never silent thinning. |
| + |
| +## 5. Box side (stage 1) |
| + |
| +**Producers.** |
| + |
| +- *Exec children and seeds* (`internal/activator/backend_exec.go`): today |
| + `cmd.Stdout/Stderr = os.Stdout/os.Stderr`. Becomes a tee — journald |
| + exactly as before, plus a line-splitting writer into the sink tagged |
| + `(host, stream "app")`, stderr and stdout alike. Lines capped (8 KiB; |
| + longer is truncated with a marker suffix). The tee is an `io.Writer`; |
| + the WaitDelay discipline around piped stderr already in place covers it. |
| +- *Platform events*: an injected `Events` callback on the activator (and |
| + the agent's restart dispatch) — **curated member-safe messages at each |
| + emit site**, per #131's sanitize note, never relayed raw journal lines. |
| + Stage-1 emit sites: wake start, wake failure (the member-safe reason: |
| + e.g. "seed timed out", "no adoptable head for channel X", "instance |
| + socket already in use"), instance exit, restart bounce, hibernation |
| + stop. The callback is nil-safe so tests and non-shipping deployments |
| + change nothing. |
| + |
| +**Sink** (`applog.Sink`): bounded ring per app (default 4 MiB), token-bucket |
| +rate cap per app (default 1 MiB/min sustained, 4 MiB burst). Over budget → |
| +oldest dropped first + the synthetic drop record. `Drain()` hands the ship |
| +loop everything and resets. |
| + |
| +**Ship loop** (`internal/host/logship.go`): 10s tick; flush when non-empty; |
| +platform events poke an immediate (2s-debounced) flush so `carlos deploy`'s |
| +timeout window can see the wake error. Same panic-recover + disabled-without- |
| +bucket shape as `metricsLoop`. On crossing an hour boundary it prunes **its |
| +own `<box>` subtree only** (clean ownership): hours older than 7 days, and |
| +oldest-first beyond the per-app-per-box byte cap (default 50 MiB). |
| + |
| +## 6. Console read path (stage 1) |
| + |
| +`GET /api/cli/apps/{account}/{app}/logs` — registered next to the instances |
| +family in `authcli.go`, behind the same gate shape (`instanceGate`'s order: |
| +principal → rate → bucket). |
| + |
| +Query params: `since`, `until` (RFC3339 or Go duration shorthand), |
| +`stream`, `host`, `level`, `grep` (RE2, applied console-side), `field` |
| +(`k=v`, repeatable), `limit` (default 1000, max 5000), `cursor`. |
| + |
| +Read: `ListDirs(logs/)` → boxes; per box, hour prefixes clipped to |
| +[since, until]; decode chunks newest-last; filter; k-way merge by |
| +`(ts, box, seq)`; cap by `limit` **and** a byte ceiling (2 MiB) with a |
| +`truncated` flag and an opaque `cursor` (base64 of last `(ts, box, seq)`) |
| +— truncation is stated, never implied. Response: |
| + |
| +```json |
| +{"records":[…],"truncated":false,"cursor":"…"} |
| +``` |
| + |
| +`-f` is the same endpoint polled at 2s with the returned cursor; no |
| +server-side streaming, no new connection discipline. |
| + |
| +## 7. CLI (stage 1) |
| + |
| +`carlos logs -app <app> [-account <acct>] [-since 1h] [-until …] |
| +[-stream app|platform|edge|release] [-host <instance>] [-level warn|error] |
| +[-grep <re>] [-field k=v] [-n <limit>] [-f]` |
| + |
| +Remote form only (console API; account/app resolution and config defaults |
| +exactly as `carlos instances list`). Default window: last 1h, last 1000 |
| +lines, honest "showing last N — use -since for more" when capped. Render: |
| +`15:04:05 [stream host] msg`, level colored via the existing color |
| +conventions; `fields` appended `k=v` when present. `-f` polls at 2s until |
| +interrupted. A static/bake app with no process prints the platform stream |
| +and says why there is no `app` stream. |
| + |
| +## 8. Retention & cost |
| + |
| +7 days, per-app-per-box 50 MiB cap, oldest-first, enforced by each box for |
| +its own subtree. Steady-state cost: one small PUT per box per 10s *only |
| +when something logged*; an idle box ships nothing. When either cap binds, |
| +the drop is recorded in-stream (synthetic record) and visible in the CLI. |
| + |
| +## 9. Security |
| + |
| +- Owner-gated end to end: the API resolves the account exactly like the |
| + instances family; a non-member gets the byte-identical 404. |
| +- Platform-event text is composed member-safe at the emit site (no box |
| + paths, no journal relay). App stdout is the owner's own output and ships |
| + verbatim (minus line-length caps). |
| +- Edge lines (stage 2) will carry no client IP in the member-facing record. |
| +- The bucket prefix lives under the app, so deletion policy and trash |
| + already govern it. |
| + |
| +## 10. Testing |
| + |
| +- `internal/applog`: record round-trip + additive-parse pinning, sink |
| + budget/drop honesty, chunk codec, filter matrix, merge order. |
| +- Activator: tee preserves journald bytes; child lines land tagged; wake |
| + failure emits exactly one member-safe platform event (fixture Events fn). |
| +- Ship loop: DirStore-backed flush/prune/no-empty-PUT. |
| +- Console: handler filter/paging/gate tests on a DirStore fixture. |
| +- CLI: e2e against a fixture console via `carlosTestEnv` + scratch |
| + `cmd.Dir` (the repo's minted-real-accounts rule). |
| +- The full repo gate: build/vet/test every package, `GOOS=darwin` compile. |
| + |
| +## 11. Out of scope (recorded so they stay decisions, not accidents) |
| + |
| +- Root adopt pass skip reasons in-stream (stage 2; the other half of #78 — |
| + it is a separate process needing its own writer identity/seq). |
| +- Unit-tenant journald ingestion (stage 2). |
| +- Edge request lines (stage 2) and the release join (stage 3). |
| +- Full-text index/search over all history; alerting on log content. |
| +- Console web UI tab (API + CLI first; the tab reads the same endpoint). |