amadan
spec: exec-child accounting — delegated cgroup subtree design
Approved 2026-08-07: exec-backed instances get their own cgroups under a Delegate= subtree of carlos-edge.service; sampler and console read them under carlos-exec@<host>. Closes the default-path blind spot behind the keymail instances-page diagnosis.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+170
−0
docs/superpowers/specs/2026-08-07-exec-child-accounting-design.md+170 −0
diff --git a/docs/superpowers/specs/2026-08-07-exec-child-accounting-design.md b/docs/superpowers/specs/2026-08-07-exec-child-accounting-design.md| new file mode 100644 |
| index 000000000..8397325a1 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-07-exec-child-accounting-design.md |
| @@ -0,0 +1,170 @@ |
| +# Exec-child accounting — design |
| + |
| +Per-instance resource readings for exec-backed routes. Today an instance |
| +route with `backing=''` runs as an exec child of the agent, inside |
| +`carlos-edge.service`'s own cgroup, and the metrics sampler |
| +(`2026-08-02-host-process-monitoring-design.md`) — which reads only |
| +`carlos-*.service` cgroups under `system.slice` — can never see it. The |
| +owner Instances page renders "no per-instance readings" for every such |
| +route, permanently. This is the default: a plain `carlos add` produces an |
| +exec-backed route, so the blind spot covers most tenants (diagnosed |
| +2026-08-06 on keymail's three flagship routes; `home.kass.training`, |
| +unit-backed, reports fine). |
| + |
| +Approach, chosen 2026-08-07: give each exec child its **own cgroup** under a |
| +subtree systemd delegates to the agent, and extend the existing sampler to |
| +read it. The monitoring spec's convention — a process means a cgroup, never |
| +process-table scraping — is kept, the numbers are the same |
| +`cpu.stat`/`memory.current`/`io.stat` figures unit-backed tenants get, and |
| +grandchildren are covered for free. `/proc` sampling of tracked PIDs was |
| +considered and rejected: it breaks that convention, RSS is not comparable to |
| +`memory.current`, and CPU spent by grandchildren reaped between ticks is |
| +simply lost. Migrating routes to `backing=unit` was rejected as the fix |
| +because it patches one app while leaving the default path blind. |
| + |
| +The governing constraint is that the agent is **not root**: `User=carlos`, |
| +`ProtectControlGroups=yes`, seccomp-filtered. That rules out asking PID 1 |
| +for transient scopes over D-Bus (unprivileged callers can't), and it means |
| +cgroup writability must come from systemd delegation, not privilege. |
| + |
| +## 1. Unit file — delegation |
| + |
| +`internal/host/units/carlos-edge.service` changes: |
| + |
| +- Add `Delegate=cpu memory io pids`. systemd chowns the unit's own cgroup |
| + subtree to `User=carlos` and stops managing below it — the agent owns |
| + that directory tree and nothing else. |
| +- Remove `ProtectControlGroups=yes`, with a comment recording the why: |
| + delegation requires the subtree writable, and ordinary DAC keeps the rest |
| + of `/sys/fs/cgroup` read-only to an unprivileged user — the hardening |
| + this line bought is retained by ownership rather than by mount flags. |
| + |
| +Everything else — seccomp filter, `NoNewPrivileges`, capability set, |
| +`TasksMax` — stays exactly as it is. A child still runs inside the service's |
| +subtree, so `systemctl restart carlos-edge` still kills every instance: |
| +today's lifecycle, unchanged. |
| + |
| +## 2. Agent startup — the delegation dance |
| + |
| +A new small package, `internal/cgroup`, wired up by the agent |
| +(`internal/host/agent.go`). On start: |
| + |
| +1. Read the agent's own cgroup path from `/proc/self/cgroup` and resolve it |
| + under `/sys/fs/cgroup`. |
| +2. Probe for live delegation: the subtree's `cgroup.procs` and |
| + `cgroup.subtree_control` must be writable. If not — old unit file, a |
| + test, darwin — the manager is **nil**, one line is logged, and every |
| + spawn behaves exactly as today. Fail toward absent readings, never |
| + toward a broken spawn path. |
| +3. Create a `supervise/` leaf and move the agent's own process into it. |
| + cgroup v2 forbids enabling controllers on a cgroup that has member |
| + processes ("no internal processes"); the move is what makes step 4 |
| + legal. |
| +4. Enable `+cpu +memory +io +pids` in the subtree root's |
| + `cgroup.subtree_control`, so per-child cgroups get `memory.current`, |
| + `io.stat`, and `pids` accounting (`cpu.stat` exists regardless). |
| +5. Sweep stale `carlos-exec@*` directories left by a previous agent |
| + (post-restart they are empty; rmdir, failures tolerated). |
| + |
| +Any failure mid-dance degrades to the nil manager — never a fatal error. |
| + |
| +## 3. Spawning into per-host cgroups |
| + |
| +`activator.ExecBackend` gains the manager (nil-tolerant). Instance, |
| +sidecar, and **seed** processes for a host all spawn into one cgroup, |
| +`<subtree>/carlos-exec@<host>`, created on the host's first spawn of a wake |
| +— a seed's first-boot cost and a sidecar's load are that instance's work, |
| +and attributing them elsewhere would be a lie in the other direction. |
| + |
| +- Primary mechanism: Go's `SysProcAttr{UseCgroupFD, CgroupFD}` — clone3 |
| + with `CLONE_INTO_CGROUP`, so the child never spends a cycle in the wrong |
| + cgroup. |
| +- Fallback: if clone3 is refused (a seccomp filter older than the syscall; |
| + `SystemCallErrorNumber=EPERM` makes this an EPERM at spawn), start the |
| + child normally and write its PID into the cgroup's `cgroup.procs`. The |
| + window between start and move charges the child's first moments to the |
| + agent — mis-accounting in the safe direction. The first clone3 refusal |
| + latches the fallback for the life of the process: a seccomp filter is |
| + static, so retrying the syscall on every spawn buys nothing. |
| +- Cleanup: after the instance exits, rmdir the (empty) cgroup dir; a |
| + failure is tolerated and the startup sweep (§2.5) catches leftovers. |
| + |
| +## 4. Sampler — a third scan root |
| + |
| +`listCarlosUnits` (`internal/host/metrics_probes.go`) currently scans |
| +`system.slice` flat plus one level inside each `system-*.slice`. It gains a |
| +third root, passed in by the agent: the delegated subtree, matching |
| +directories named `carlos-exec@<host>` (the `supervise` leaf is skipped — |
| +it is the agent itself, already reported as `carlos-edge.service`). |
| + |
| +Sample keys are the directory name, `carlos-exec@<host>`, verbatim. The |
| +name is deliberately **not** a fake `.service`: an operator who takes a |
| +dashboard row to `systemctl status` must not be told a unit exists that |
| +doesn't. `readUnitRaw` is reused unchanged. |
| + |
| +`carlos-edge.service`'s own row keeps its current meaning: the hierarchical |
| +total, children included — which is what it already reports today, since |
| +the children have always lived inside it. |
| + |
| +## 5. Console — two allowlist keys per host |
| + |
| +`OwnerAppMetrics` (`internal/console/metricsview.go`) builds its allowlist |
| +as `want[activator.UnitName(h)] = h`. It gains a second entry per host: |
| +`want[activator.ExecName(h)] = h`, where `ExecName(host)` returns |
| +`"carlos-exec@" + host` (defined next to `UnitName`, the same one-place |
| +naming discipline). Nothing else changes: exec-backed instances flow |
| +through the existing row assembly, `Reporting()` turns true, sparklines |
| +render, the freshest-box tie-break already handles a host moving between |
| +backings. The operator dashboard needs zero changes — rows appear by key. |
| + |
| +A host can never legally report under both names at once (one route, one |
| +backing), but if both ever appear in a sweep the existing freshest-box |
| +logic picks deterministically; no special case is added. |
| + |
| +## 6. Compatibility and rollout |
| + |
| +Wire shape is additive: `Sample.Units` gains new keys, and every chunk and |
| +manifest already in a production bucket still parses (wire-shape rule). |
| +An old console rendering a new box's chunk shows the new keys on the |
| +operator dashboard and ignores them in owner views — harmless both ways. |
| + |
| +Rollout order, per box: |
| + |
| +1. Ship the new binary fleet-wide. It tolerates the old unit file (nil |
| + manager) indefinitely. |
| +2. Update `carlos-edge.service`, `systemctl daemon-reload`, restart the |
| + unit. This is a **one-time operator step**: platform unit files are not |
| + auto-converged today (the root adopt pass converges tenant drop-ins |
| + only). Teaching the root adopt pass to converge platform units is a |
| + real gap this work exposes but does not build; it is filed as a |
| + follow-up, not smuggled in. |
| + |
| +Boxes: flagship (`annie-welding`), tito (`betty-fossicking`), the Pi fleet. |
| +Verify on the dev box first, against the binary actually built (stale-binary |
| +rule). |
| + |
| +## 7. Testing |
| + |
| +- The delegation dance and directory lifecycle run against temp-dir |
| + fixtures — the same seam `listCarlosUnits` already uses; tests never |
| + touch the live cgroup tree. |
| +- The clone3 fallback is behind an injectable spawn error, so both paths |
| + are exercised without a seccomp filter in the loop. |
| +- Sampler discovery: fixture gains the third root; asserts `supervise` is |
| + skipped and `carlos-exec@<host>` keys come out verbatim. |
| +- Console: allowlist test covering both key forms for one host, and the |
| + no-double-report property of §5. |
| +- End-to-end on the dev box before flagship/tito: spawn keymail-shaped |
| + exec routes, watch readings appear on the Instances page. |
| + |
| +## Out of scope |
| + |
| +- Bespoke units (`keymail-relay.service`): invisible for a different |
| + reason — the `carlos-` name filter — and wanting a different, |
| + naming-convention fix. |
| +- `days/` rollups (still a later task's view). |
| +- Per-instance resource **limits**: out of scope here, but this design |
| + unlocks it — with each child in its own cgroup the agent could set |
| + `memory.max` per instance, which the unit file's "No MemoryMax" comment |
| + today rules out because a cap would catch every child at once. |
| +- Auto-convergence of platform unit files (filed as follow-up, §6). |