amadan
spec: host & process monitoring — needles for operators, on-box alerting
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+139
−0
docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md+139 −0
diff --git a/docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md b/docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md| new file mode 100644 |
| index 000000000..0c074bbde |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md |
| @@ -0,0 +1,139 @@ |
| +# Host & process monitoring — design |
| + |
| +Operator-grade monitoring for CARLOS boxes: real CPU, memory, and disk |
| +numbers per host and per process, with history, alerting, and a console |
| +dashboard. This is a new capability, distinct from the fleet status page |
| +(`2026-08-01-fleet-status-page-design.md`): that page publishes band words |
| +and aggregates for a public audience; this publishes needles for operators. |
| +The two share probes but never a pipeline — the status page's "no needles |
| +ever" contract stays structurally intact because its writer never reads |
| +anything this spec introduces. |
| + |
| +Approach: CARLOS-native. Extend the existing box agent, store in the |
| +bucket, render in the console. No Prometheus/Grafana fleet to host, no |
| +CloudWatch coupling — the fleet is a handful of boxes and the platform's |
| +control plane is already the bucket. |
| + |
| +## 1. Collection — a metrics sampler in the box agent |
| + |
| +A new sampler loop in `internal/host`, ticking every **60 seconds**, |
| +independent of the 5-minute status snapshot tick (which is untouched). |
| + |
| +Host-level, each tick: |
| + |
| +- **CPU busy fraction** — delta between successive `/proc/stat` readings |
| + (busy = non-idle, non-iowait jiffies over total). The delta is why this |
| + is a stateful sampler rather than a stateless probe like the existing |
| + ones; the first tick after start has no delta and reports nil. |
| +- **Load averages** — 1/5/15 from `/proc/loadavg`. |
| +- **Memory fraction** — reuse `readMemFrac` (MemAvailable-based). |
| +- **Disk fraction** — reuse `readDiskFrac` (Statfs on the data dir). |
| + |
| +Per-process, each tick — where "process" means **systemd unit**, read from |
| +cgroup v2 files, never `ps` scraping: |
| + |
| +- Covered units: every `carlos-app@*.service` (tenant apps — this is the |
| + noisy-neighbour attribution) plus the platform's own units: the console, |
| + the agent itself, litestream, and the keymail relay. |
| +- Per unit: CPU fraction (`cpu.stat` usage_usec delta over wall clock), |
| + `memory.current`, `memory.peak`, and cumulative read/write bytes from |
| + `io.stat`. |
| +- A hibernating app has no live cgroup and is simply **absent** from that |
| + tick's sample — absence is the data; nothing is invented for it. |
| + |
| +Every reading keeps the probe convention already documented in |
| +`internal/host/probes.go`: **fail toward nil, never a guessed zero.** A |
| +box that can't answer is not a box that is fine. |
| + |
| +Samples accumulate in memory and flush to the bucket every 5 minutes. |
| + |
| +## 2. Storage — bucket, additive only |
| + |
| +All-new objects under a new prefix; no existing artifact changes shape |
| +(wire-shape rule: everything already in production buckets still parses). |
| + |
| +- **Raw**: `metrics/boxes/<box>/raw/<YYYY-MM-DD-HH>.json` — the current |
| + hour's minute-samples (host + per-unit), rewritten in full at each |
| + 5-minute flush (12 PUTs/hour/box). `<box>` is the same label the status |
| + snapshot uses. Retained **48 hours**; the agent prunes expired hours on |
| + its hourly boundary. |
| +- **Rollups**: `metrics/boxes/<box>/days/<YYYY-MM-DD>.json` — hourly |
| + min/avg/max per metric, host and per-unit, appended once an hour by |
| + rewriting the current day object. Retained **indefinitely** (they are |
| + tiny); this is the capacity/trend history. |
| +- Object bodies carry a `"v": 1` field so later shape changes can be |
| + additive and detectable. |
| + |
| +Privacy boundary: these are operator-plane needles, private by |
| +construction — readable only with bucket access or through the console's |
| +operator role. The public status page writer takes no input from the |
| +`metrics/` or `alerts/` prefixes. |
| + |
| +## 3. Alerting — evaluated on-box |
| + |
| +The box judges itself, so alerting works even when the console is down. |
| + |
| +**v1 rules**, defaults baked in, overridable through the existing |
| +app-config mechanism: |
| + |
| +| Rule | Threshold | Sustain | |
| +|---|---|---| |
| +| Disk fraction | > 90% | immediate — disk doesn't spike back down | |
| +| Memory fraction | > 90% | 10 minutes | |
| +| Host CPU | > 95% | 15 minutes | |
| +| Litestream lag | > 15 minutes | immediate | |
| + |
| +Per-unit alerts are deliberately out of v1: the host-level alert points at |
| +the box, the dashboard's per-unit table answers "which app did it." |
| + |
| +**Hysteresis and state**: fire at threshold, clear at threshold minus 5 |
| +percentage points (lag: minus 5 minutes). Alert state persists on-box |
| +(under the data dir) so an agent restart neither re-fires nor forgets an |
| +active alert, and is published to `alerts/boxes/<box>.json`. |
| + |
| +**Delivery**, on fire and on clear: |
| + |
| +- **Keymail** to configured operator addresses. |
| +- **Webhook** — POST of the alert JSON to a configured URL, if set. |
| +- **Console** — reads `alerts/boxes/*.json` and shows red state on the |
| + operator dashboard; this channel needs no configuration. |
| + |
| +Delivery failure never blocks sampling or state transitions; it is logged |
| +and retried on the next tick while the alert remains active. |
| + |
| +## 4. Console — operator dashboard |
| + |
| +A new console page behind the **existing operator role**, with the |
| +existing security gate ordering (session → CSRF → role; byte-identical |
| +404 for non-operators, `requireFreshAuth` last if used). |
| + |
| +- Per-box: current CPU / memory / disk as real numbers, 48-hour |
| + sparklines from the raw chunks, longer trends from the daily rollups. |
| +- Per-box, per-unit table sortable by CPU and memory — the |
| + noisy-neighbour view. Hibernating (absent) units render as absent, not |
| + as zero. |
| +- Active alerts render red at the top; a cleared alert disappears rather |
| + than lingering. |
| +- House rendering rules: vendored assets only (Lucide-style in-code SVG, |
| + no CDN); sparklines drawn from inline data. |
| + |
| +## 5. Testing |
| + |
| +- Samplers and cgroup readers are injectable fixtures at the same seam |
| + the existing `statusTick` tests use — tests never touch `/proc`, |
| + cgroups, or a live syscall. |
| +- CPU-delta computation, banding, rule evaluation, and hysteresis are |
| + pure functions with direct unit tests (including: first-tick nil, |
| + sustain windows, fire/clear boundaries, restart-with-persisted-state). |
| +- A contract test asserts the public status page output contains no |
| + numeric needles — guarding the privacy boundary against regression. |
| +- Bucket object shapes get round-trip tests, and a |
| + parses-yesterday's-shape test once `v` ever bumps. |
| + |
| +## Out of scope (v1) |
| + |
| +- Per-unit alert rules. |
| +- Network and per-request metrics (edge/router latency, request rates). |
| +- Public exposure of any of this — the public page keeps band words only. |
| +- Cross-box aggregate alerting ("the fleet is degraded") — each box |
| + alerts for itself. |