carlos / platform Public

docs: monitoring env vars, bucket prefixes, and implementation deviations

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Paul Campbell 8d74aa634bd4e6ba9b1e55571da2b47c52d58513 parent 29aac10
2 files changed, +67 −0
  • README.md +35 −0
  • docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md +32 −0
diff --git a/README.md b/README.md
index f7dbddf35..74178b004 100644
--- a/README.md
+++ b/README.md
@@ -343,3 +343,38 @@ reading raw JSON already expects to view its own headers).
A fully-private deployment (`STATUS_PUBLIC` unset) whose console is down has
no page at all — the `aws s3 cp` fallback above is the entire story there;
nothing papers over it.
+
+## Host & process monitoring
+
+Operator-grade monitoring layered on top of fleet status: real CPU, memory,
+and disk numbers per host and per process, with history, alerting, and a
+console dashboard — needles where the public status page shows only band
+words. See
+[`docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md`](docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md)
+for the full design.
+
+**Bucket layout:**
+
+- `metrics/boxes/<box>/raw/<YYYY-MM-DD-HH>.json` — 60-second host + per-unit
+ samples, one object per UTC hour, retained 48 hours.
+- `metrics/boxes/<box>/days/<YYYY-MM-DD>.json` — hourly min/avg/max rollups,
+ kept forever (the capacity/trend history).
+- `alerts/boxes/<box>.json` — on-box alert state (fire/clear, hysteresis).
+
+**Config surface** — read by the box agent from `/etc/carlos/host.env`:
+
+- `CARLOS_ALERT_WEBHOOK` — optional; the agent POSTs the alert JSON here on
+ fire and on clear. Unset simply skips the webhook channel (keymail
+ delivery and the console UI channel are unaffected).
+- `CARLOS_ALERT_DISK_PCT` / `CARLOS_ALERT_MEM_PCT` / `CARLOS_ALERT_CPU_PCT` —
+ optional threshold overrides for the built-in disk / memory / host-CPU
+ rules, as fractions (0-1).
+
+**Console surfaces:**
+
+- `GET /fleet/metrics` — operator-only dashboard: per-box needles and
+ 48-hour sparklines, a per-box per-unit table sortable by CPU/memory, and
+ active alerts at the top.
+- A "Resources" section on each app's own page — owner-facing, scoped to
+ that app's own instances only (current CPU/memory/io plus sparklines and
+ daily trends); never host-level totals or other tenants' units.
diff --git a/docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md b/docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md
index 2b1fbaa8b..fc9b15049 100644
--- a/docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md
+++ b/docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md
@@ -118,6 +118,14 @@ active alert, and is published to `alerts/boxes/<box>.json`.
- **Console UI** — the same `alerts/` read shows red state on the
operator dashboard; this channel needs no configuration.
+Implementation note: keymail delivery is retried **per recipient**, not
+all-or-nothing. The console watcher keys its `alert_seen` state by (box,
+rule, recipient address); a failed send to one operator leaves only that
+operator's row unrecorded and retried next minute — healthy operators are
+not re-mailed. When no operator has a mailable address, a sentinel row
+(empty address) records the transition so operators granted later are not
+retroactively paged for an alert that fired before they joined.
+
Delivery failure never blocks sampling or state transitions; it is logged
and retried on the next tick while the alert remains active.
@@ -139,6 +147,12 @@ where it has band words, per-unit where it has aggregates.
- Active alerts render red at the top; a cleared alert disappears rather
than lingering.
+Implementation note: the metrics sweep behind this dashboard treats a raw
+chunk that exists but fails to parse as skippable — never fatal to the
+sweep. A transient store read error, by contrast, does fail the sweep, and
+a failed sweep is never cached (the last good cache keeps serving). A
+persistently corrupt chunk must not blank the dashboard for 48 hours.
+
**Owner metrics** — app owners see their own apps' metrics on the app
pages they already have (owner gate unchanged): per app, per instance
(each `carlos-app@<host>` unit, on whichever box runs it), current CPU /
@@ -150,9 +164,27 @@ units, box labels beyond what their app pages already show, or anything
from `alerts/`. Host needles would leak cross-tenant information; they
are operator-only.
+Implementation note: app names are unique per account, not globally, so
+this view derives an app's units starting from its routes, filtered to
+routes whose stamped account matches the viewing account (fails closed)
+before deriving each instance's `carlos-app@<host>.service` unit name — a
+route stamped for a different account is invisible here even if the app
+name matches.
+
+Implementation note: a box whose newest sample is older than 5 minutes
+(`metricsFreshWindow`) is treated as not-reporting in this view — its
+instances render "asleep — no recent readings" rather than presenting
+stale numbers as current. The operator dashboard, by contrast, shows the
+sample's own timestamp and applies no such cutoff.
+
House rendering rules for both views: vendored assets only (Lucide-style
in-code SVG, no CDN); sparklines drawn from inline data.
+Implementation note: a series longer than the sparkline's 240px viewBox is
+downsampled inside `sparkPoints` to at most 240 points via bucket means of
+non-nil values; a bucket with no non-nil samples produces a gap, never a
+zero.
+
## 5. Testing
- Samplers and cgroup readers are injectable fixtures at the same seam