amadan
specs: exec-backed instance metrics design (issue #118)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+133
−0
docs/superpowers/specs/2026-08-08-exec-instance-metrics-design.md+133 −0
diff --git a/docs/superpowers/specs/2026-08-08-exec-instance-metrics-design.md b/docs/superpowers/specs/2026-08-08-exec-instance-metrics-design.md| new file mode 100644 |
| index 000000000..96edb5903 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-08-exec-instance-metrics-design.md |
| @@ -0,0 +1,133 @@ |
| +# Exec-backed instance metrics (issue #118) |
| + |
| +**Date:** 2026-08-08 |
| +**Issue:** carlosframework/platform#118 — exec-backed instances report no |
| +CPU/memory: metrics sample unit cgroups only. |
| + |
| +## Problem |
| + |
| +The 2026-08-02 host-process-monitoring design defined "process" as |
| +**systemd unit**: the sampler walks `/sys/fs/cgroup/system.slice` for |
| +`carlos-*.service` cgroups. An exec-backed instance is a direct child of |
| +`carlos agent` inside the agent's own cgroup — no unit cgroup exists, so |
| +it is invisible to per-instance attribution. Its app page reads |
| +"asleep — no recent readings" even during long awake stretches, which is |
| +how the gap was found (owners can't tell "asleep" from "broken"). The |
| +monitoring design's owner-facing promise — a Resources section scoped to |
| +the app's own instances — can never be met for the backing most |
| +provisioned instances get by default. |
| + |
| +## Options considered |
| + |
| +1. **Pid-based sampling of activator children** (chosen). The activator |
| + already tracks every awake exec child (`actEntry.proc` holds the |
| + `*exec.Cmd`). Expose the awake host→pid map; the sampler reads |
| + `/proc/<pid>/{stat,status,io}` and publishes the readings under the |
| + same `carlos-app@<host>.service` key the console already matches. |
| +2. **Per-child cgroup at spawn** (`carlos.slice/instance-<host>` scope). |
| + Rejected for v1: needs `Delegate=yes` on the agent's unit fleet-wide |
| + (pinfra change + restart on every box), a cgroup-v2 |
| + no-internal-process workaround (the agent's own pid can't share a |
| + cgroup with controller-enabled children), and walker changes anyway — |
| + delegated children live under the agent's own unit cgroup, a level |
| + `listCarlosUnits` deliberately does not scan. Same data, far larger |
| + blast radius. |
| +3. **UI honesty only** ("not measured (exec-backed)"). Superseded: with |
| + option 1 the readings exist, so there is nothing to apologize for. |
| + Not implemented — old-agent boxes render the existing state-neutral |
| + strings until they take the new binary, a transitional state not |
| + worth a console change. |
| + |
| +## Design |
| + |
| +### Key decision: exec children publish under the unit-name key |
| + |
| +An exec-backed host has no `carlos-app@<host>.service` unit *by |
| +construction* (units are for the edge; instances are the agent's |
| +children), so the key can never collide with a real cgroup reading on |
| +the same box. Publishing under `activator.UnitName(host)` means: |
| + |
| +- **Wire shape unchanged** — `metrics.Sample.Units` is a string-keyed |
| + map; new entries are additive. Everything already in production |
| + buckets still parses, and old consoles render new-agent chunks as |
| + more unit rows. |
| +- **Console unchanged** — `OwnerAppMetrics` allowlists exactly |
| + `activator.UnitName(h)` per route host; exec readings match as-is. |
| + The operator dashboard's per-unit table picks them up the same way. |
| +- If a collision ever did occur, the cgroup reading wins (it accounts |
| + the whole subtree; the pid reading is one process). |
| + |
| +### Activator: exposing awake children |
| + |
| +- `childProc` gains `Pid() int` (the spawned `cmd.Process.Pid`). |
| +- `Service.AwakeExecPids() map[string]int` returns host→pid for entries |
| + that are awake, whose proc has not exited, and whose Proc implements |
| + `Pid() int`. Taken under `a.mu`, like every entry read. |
| +- `unitProc` (unit-backed tenants) implements no `Pid` and is naturally |
| + excluded — those instances already report via their unit cgroup. |
| +- Waking and stopping entries are excluded: no stable process to read. |
| + |
| +### Sampler: /proc readers |
| + |
| +New pure parsers in `internal/metrics` (fixture-tested, like the cgroup |
| +parsers), with thin `os.ReadFile` glue in `internal/host` (untested by |
| +the same convention as the other readers): |
| + |
| +- `/proc/<pid>/stat` → CPU microseconds: `(utime + stime)` clock ticks |
| + × 1e6 / USER_HZ. USER_HZ is a named constant 100 — the kernel fixes |
| + the /proc tick unit at 100 on every architecture CARLOS runs on, |
| + independent of CONFIG_HZ. Parsed after the last `)` (comm may contain |
| + spaces and parens). Feeds the existing `UnitCPUFrac` delta machinery |
| + unchanged; a respawned child's counter reset yields a nil fraction |
| + for that tick (`curUsec < prevUsec` guard, already present). |
| +- `/proc/<pid>/status` → `VmRSS` (memory), `VmHWM` (peak), kB → bytes. |
| + RSS, not `memory.current`: no page-cache attribution, which for "what |
| + your app uses" is the more honest number a single-process reading can |
| + give. |
| +- `/proc/<pid>/io` → `read_bytes` / `write_bytes` — cumulative since |
| + process start, same semantics as the cgroup `io.stat` counters. |
| + |
| +Every field fails toward nil independently, per the probe convention: a |
| +pid that vanished mid-read (raced its own exit) contributes nils, never |
| +zeros. |
| + |
| +### Wiring |
| + |
| +- `metricsLoop(children func() map[string]int)` — `agent.go` passes |
| + `svc.AwakeExecPids` (nil `svc` → nil func → cgroup-only, exactly |
| + today's behavior; `carlos edge` without hibernation is unaffected). |
| +- The units probe becomes a merge: cgroup readings (as today) plus one |
| + `unitRaw` per awake exec child keyed by `activator.UnitName(host)`. |
| + The merge is a pure function, unit-tested; cgroup wins collisions. |
| + |
| +### Known limits (documented, accepted) |
| + |
| +- **Grandchildren are not counted.** A pid reading covers one process; |
| + a cgroup covers a subtree. Carlos apps are single-process Go |
| + binaries; an app that forks workers under-reports. Option 2 remains |
| + the upgrade path if that ever matters. |
| +- **Sidecars are not counted** toward their host's reading (v1 scope: |
| + the instance child only). |
| +- **Pid-reuse race**: between `AwakeExecPids` and the /proc read, an |
| + exiting child's pid could in principle be recycled. The window is |
| + sub-millisecond within a 60s tick, the activator excludes |
| + already-exited procs, and the failure mode is one sample's nils or |
| + one stray reading — accepted. |
| + |
| +## Testing |
| + |
| +- `internal/metrics`: fixture tests for the three parsers (mangled |
| + input → nil; comm-with-spaces stat lines; kB scaling). |
| +- `internal/activator`: `AwakeExecPids` over the fake-backend state |
| + machine — awake pid-bearing procs listed; waking/stopping/exited/ |
| + pid-less excluded. |
| +- `internal/host`: merge-function test (cgroup+exec compose, collision |
| + precedence, both-empty → nil); loop-level test that a sample tick |
| + carries exec readings through to `Sample.Units`. |
| + |
| +## Deploy |
| + |
| +Ship the agent normally (promote → adopt → restart per deploy rules). |
| +No unit files, no pinfra, no console deploy required; readings appear |
| +for awake exec instances within one flush (≤5 min) of the restarted |
| +agent's first tick. |