amadan
spec: monitoring reality fixes — keymail via console, host.env config, Store.Delete
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+31
−11
docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md+31 −11
diff --git a/docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md b/docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md| index 447e2996e..2b1fbaa8b 100644 |
| --- a/docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md |
| +++ b/docs/superpowers/specs/2026-08-02-host-process-monitoring-design.md |
| @@ -32,9 +32,12 @@ Host-level, each tick: |
| Per-process, each tick — where "process" means **systemd unit**, read from |
| cgroup v2 files, never `ps` scraping: |
| -- Covered units: every `carlos-app@*.service` (tenant apps — this is the |
| - noisy-neighbour attribution) plus the platform's own units: the console, |
| - the agent itself, litestream, and the keymail relay. |
| +- Covered units: every cgroup under `system.slice` matching |
| + `carlos-*.service` — which is exactly `carlos-app@<host>.service` |
| + tenant instances (the noisy-neighbour attribution; the console and |
| + keymail are themselves `carlos-app@` instances) plus the platform's |
| + fixed units: `carlos-edge.service` (the agent), `carlos-litestream`, |
| + `carlos-adopt`, `carlos-restore-verify`. |
| - Per unit: CPU fraction (`cpu.stat` usage_usec delta over wall clock), |
| `memory.current`, `memory.peak`, and cumulative read/write bytes from |
| `io.stat`. |
| @@ -56,7 +59,9 @@ All-new objects under a new prefix; no existing artifact changes shape |
| hour's minute-samples (host + per-unit), rewritten in full at each |
| 5-minute flush (12 PUTs/hour/box). `<box>` is the same label the status |
| snapshot uses. Retained **48 hours**; the agent prunes expired hours on |
| - its hourly boundary. |
| + its hourly boundary. Pruning needs deletion, which `release.Store` |
| + doesn't have today — an additive `Delete(ctx, key)` method lands on the |
| + interface and both implementations as part of this work. |
| - **Rollups**: `metrics/boxes/<box>/days/<YYYY-MM-DD>.json` — hourly |
| min/avg/max per metric, host and per-unit, appended once an hour by |
| rewriting the current day object. Retained **indefinitely** (they are |
| @@ -73,8 +78,10 @@ operator role. The public status page writer takes no input from the |
| The box judges itself, so alerting works even when the console is down. |
| -**v1 rules**, defaults baked in, overridable through the existing |
| -app-config mechanism: |
| +**v1 rules**, defaults baked in, overridable via host env vars in |
| +`/etc/carlos/host.env` — the platform-level config surface every agent |
| +loop already reads (app-config is app-scoped and doesn't fit host |
| +thresholds): |
| | Rule | Threshold | Sustain | |
| |---|---|---| |
| @@ -83,6 +90,11 @@ app-config mechanism: |
| | Host CPU | > 95% | 15 minutes | |
| | Litestream lag | > 15 minutes | immediate | |
| +The lag rule is defined but **inert in v1**: nothing on the box surfaces |
| +litestream's replication progress yet (`Probes.LagSecs` is always nil |
| +today), and a nil reading never fires a rule — no data is never treated |
| +as bad data. Wiring a lag source arms it with no rule change. |
| + |
| Per-unit alerts are deliberately out of v1: the host-level alert points at |
| the box, the dashboard's per-unit table answers "which app did it." |
| @@ -91,11 +103,19 @@ percentage points (lag: minus 5 minutes). Alert state persists on-box |
| (under the data dir) so an agent restart neither re-fires nor forgets an |
| active alert, and is published to `alerts/boxes/<box>.json`. |
| -**Delivery**, on fire and on clear: |
| - |
| -- **Keymail** to configured operator addresses. |
| -- **Webhook** — POST of the alert JSON to a configured URL, if set. |
| -- **Console** — reads `alerts/boxes/*.json` and shows red state on the |
| +**Delivery**, on fire and on clear — split by who holds the credential: |
| + |
| +- **Webhook** — the agent POSTs the alert JSON to `CARLOS_ALERT_WEBHOOK` |
| + if set. No credential needed, so the box delivers this itself. |
| +- **Keymail** — sent by the **console**, not the agent: the only keymail |
| + send path that exists is the console's per-user OAuth grant |
| + (`Keymail.SendApp`), and the agent has no keymail identity. A console |
| + watcher polls `alerts/boxes/*.json` on a minute cadence, diffs against |
| + the last state it has seen (persisted in the console DB), and on each |
| + fire/clear sends a keymail to every operator using that operator's own |
| + stored grant token. Operators are exactly the people who granted the |
| + console send access by signing in — no new credential plane. |
| +- **Console UI** — the same `alerts/` read shows red state on the |
| operator dashboard; this channel needs no configuration. |
| Delivery failure never blocks sampling or state transitions; it is logged |