plan: exec-child accounting — seven tasks from the approved spec
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+1530
−0
docs/superpowers/plans/2026-08-08-exec-child-accounting.md+1530 −0
diff --git a/docs/superpowers/plans/2026-08-08-exec-child-accounting.md b/docs/superpowers/plans/2026-08-08-exec-child-accounting.md| new file mode 100644 |
| index 000000000..52aec680a |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-08-exec-child-accounting.md |
| @@ -0,0 +1,1530 @@ |
| +# Exec-Child Accounting Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Per-instance resource readings for exec-backed routes — each exec |
| +child gets its own cgroup under a subtree systemd delegates to the agent, the |
| +sampler reads it, the owner Instances page shows it. |
| + |
| +**Architecture:** A new `internal/cgroup` package owns the delegated subtree |
| +(delegation dance at agent start, per-host `carlos-exec@<host>` cgroups, |
| +clone3-into-cgroup spawn with a latched write-PID fallback). The activator |
| +spawns instance/sidecar/seed through it; the host sampler gains the subtree |
| +as a third scan root; the console allowlist gains a second key per host. |
| +Everything degrades to today's behavior when delegation is absent (nil |
| +manager) — fail toward absent readings, never toward a broken spawn path. |
| + |
| +**Tech Stack:** Go 1.26 (stdlib only — `syscall.SysProcAttr{UseCgroupFD}` |
| +for clone3 `CLONE_INTO_CGROUP`; no new dependencies). Module |
| +`github.com/carlosframework/platform`. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-08-07-exec-child-accounting-design.md` |
| +— read it before starting a task. |
| + |
| +## Global Constraints |
| + |
| +- Build/test env (sandboxed shells): `GOCACHE=/tmp/claude-1001/gocache |
| + GOFLAGS=-mod=mod GOPROXY=off`. Tests in `internal/activator`, |
| + `internal/host`, `internal/edge` need the sandbox disabled (unix-socket |
| + permission errors are the tell — pre-existing, not yours). |
| +- Before every commit: `go build ./...`, `go vet ./...`, and the touched |
| + package's tests green; `gofmt -l` clean for files you touch. |
| +- Wire shape is additive only: `Sample.Units` gains new keys |
| + (`carlos-exec@<host>` — no `.service` suffix, deliberately); nothing |
| + already in a production bucket may stop parsing. |
| +- Probe convention (`internal/host/probes.go`): fail toward nil, never a |
| + guessed zero. Thin OS glue is untested by convention; directory WALKS and |
| + pure parsers are tested against fixtures. Tests never touch `/proc`, |
| + live cgroups, or a live syscall. |
| +- The agent is NOT root (`User=carlos`). Nothing here may assume privilege; |
| + cgroup writability comes from systemd delegation (`Delegate=`) only. |
| +- Every cgroup-manager failure degrades to a logged line + today's |
| + behavior. A spawn must never fail because accounting failed. |
| +- Naming discipline: the string `carlos-exec@` is defined ONCE, in |
| + `internal/cgroup` (`ExecName`/`IsExecName`); everyone else calls those. |
| +- Commit messages: repo style is `component: what changed` (lowercase), body |
| + optional, ending with `Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>`. |
| + |
| +--- |
| + |
| +### Task 1: `internal/cgroup` — names and the /proc/self/cgroup parser |
| + |
| +**Files:** |
| +- Create: `internal/cgroup/cgroup.go` |
| +- Create: `internal/cgroup/cgroup_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: nothing (new leaf package; stdlib only). |
| +- Produces: `ExecName(host string) string` (returns `"carlos-exec@" + host`), |
| + `IsExecName(name string) bool`, `ParseSelfCgroup(data string) (string, bool)` |
| + (relative v2 cgroup path, no leading slash). Tasks 2–5 and 7 use these. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +Create `internal/cgroup/cgroup_test.go`: |
| + |
| +```go |
| +package cgroup |
| + |
| +import "testing" |
| + |
| +func TestExecNameAndIsExecName(t *testing.T) { |
| + if got := ExecName("keymail.dev"); got != "carlos-exec@keymail.dev" { |
| + t.Fatalf("ExecName = %q, want %q", got, "carlos-exec@keymail.dev") |
| + } |
| + for name, want := range map[string]bool{ |
| + "carlos-exec@keymail.dev": true, |
| + "carlos-exec@": false, // no host — not one of ours |
| + "supervise": false, // the agent's own leaf |
| + "carlos-app@x.service": false, |
| + "cgroup.procs": false, |
| + } { |
| + if got := IsExecName(name); got != want { |
| + t.Errorf("IsExecName(%q) = %v, want %v", name, got, want) |
| + } |
| + } |
| +} |
| + |
| +// TestParseSelfCgroup pins the /proc/self/cgroup v2 line format: the "0::" |
| +// line carries the unified-hierarchy path. v1-only content (no 0:: line) |
| +// and the root cgroup ("0::/") both report not-ok — neither is a subtree |
| +// systemd could have delegated to us. |
| +func TestParseSelfCgroup(t *testing.T) { |
| + cases := []struct { |
| + name, data, want string |
| + ok bool |
| + }{ |
| + {"systemd service", "0::/system.slice/carlos-edge.service\n", "system.slice/carlos-edge.service", true}, |
| + {"hybrid box, v2 line among v1", "12:pids:/system.slice\n0::/system.slice/carlos-edge.service\n", "system.slice/carlos-edge.service", true}, |
| + {"v1 only", "12:pids:/system.slice\n11:cpu:/system.slice\n", "", false}, |
| + {"root cgroup", "0::/\n", "", false}, |
| + {"empty", "", "", false}, |
| + } |
| + for _, c := range cases { |
| + got, ok := ParseSelfCgroup(c.data) |
| + if got != c.want || ok != c.ok { |
| + t.Errorf("%s: ParseSelfCgroup = (%q, %v), want (%q, %v)", c.name, got, ok, c.want, c.ok) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run tests to verify they fail** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/cgroup/ -count=1` |
| +Expected: FAIL — package does not exist / `ExecName` undefined. |
| + |
| +- [ ] **Step 3: Write the implementation** |
| + |
| +Create `internal/cgroup/cgroup.go`: |
| + |
| +```go |
| +// Package cgroup gives each exec-backed child its own cgroup under the |
| +// subtree systemd delegates to carlos-edge.service (Delegate= in the unit |
| +// file), so the metrics sampler can report per-instance figures for routes |
| +// that run as exec children of the agent — today those live inside the |
| +// agent's own cgroup and can never show readings. Design: |
| +// docs/superpowers/specs/2026-08-07-exec-child-accounting-design.md. |
| +// |
| +// The agent is NOT root: everything here works through ownership of the |
| +// delegated subtree, never privilege. When delegation is absent (old unit |
| +// file, a test, darwin) Setup returns a nil *Manager and every method on it |
| +// degrades to today's behavior — fail toward absent readings, never toward |
| +// a broken spawn path. |
| +package cgroup |
| + |
| +import "strings" |
| + |
| +// execPrefix is the ONE definition of the per-host exec cgroup naming — |
| +// activator.ExecName and the sampler's directory match both come through |
| +// this package. |
| +const execPrefix = "carlos-exec@" |
| + |
| +// ExecName is the cgroup directory name — and therefore the metrics sample |
| +// key — for an exec-backed host's children. Deliberately NOT dressed up as |
| +// a fake ".service": an operator who takes a dashboard row to `systemctl |
| +// status` must not be told a unit exists that doesn't. |
| +func ExecName(host string) string { return execPrefix + host } |
| + |
| +// IsExecName reports whether a cgroup directory name is one of our per-host |
| +// exec cgroups: the carlos-exec@ prefix with a non-empty host after it. |
| +func IsExecName(name string) bool { |
| + return strings.HasPrefix(name, execPrefix) && len(name) > len(execPrefix) |
| +} |
| + |
| +// ParseSelfCgroup extracts the cgroup v2 path from /proc/self/cgroup |
| +// content — the "0::" unified-hierarchy line — relative (no leading slash), |
| +// ready to join under /sys/fs/cgroup. Not-ok when there is no v2 line (a |
| +// v1-only box) or the process sits in the root cgroup: neither is a subtree |
| +// systemd could have delegated to us. |
| +func ParseSelfCgroup(data string) (string, bool) { |
| + for _, line := range strings.Split(data, "\n") { |
| + rest, found := strings.CutPrefix(line, "0::") |
| + if !found { |
| + continue |
| + } |
| + p := strings.TrimPrefix(rest, "/") |
| + if p == "" { |
| + return "", false |
| + } |
| + return p, true |
| + } |
| + return "", false |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run tests to verify they pass** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/cgroup/ -count=1` |
| +Expected: PASS |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/cgroup/cgroup.go internal/cgroup/cgroup_test.go |
| +git commit -m "cgroup: the exec-child cgroup names and the /proc/self/cgroup parser |
| + |
| +Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 2: `internal/cgroup` — the delegation dance |
| + |
| +**Files:** |
| +- Modify: `internal/cgroup/cgroup.go` (append) |
| +- Modify: `internal/cgroup/cgroup_test.go` (append) |
| + |
| +**Interfaces:** |
| +- Consumes: `IsExecName` from Task 1. |
| +- Produces: `type Manager struct` (unexported fields `root string`, |
| + `logf func(string, ...any)`, `fallback atomic.Bool`), |
| + `SetupAt(root string, logf func(string, ...any)) (*Manager, error)` |
| + (the testable core), `Setup(logf func(string, ...any)) *Manager` |
| + (production entry, nil on any failure), and nil-safe |
| + `func (m *Manager) Root() string` ("" for nil). Tasks 3, 4, 6 use these. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +Append to `internal/cgroup/cgroup_test.go` (add imports `os`, |
| +`path/filepath`, `strconv` to the existing import block): |
| + |
| +```go |
| +// newDelegatedRoot builds the temp-dir shape of a delegated cgroup: the two |
| +// control files systemd chowns to the unit's user when Delegate= is set. |
| +// Everything the dance does is plain file I/O, so a temp dir is a faithful |
| +// fixture — the same seam listCarlosUnits uses in internal/host. |
| +func newDelegatedRoot(t *testing.T) string { |
| + t.Helper() |
| + root := t.TempDir() |
| + for _, f := range []string{"cgroup.procs", "cgroup.subtree_control"} { |
| + if err := os.WriteFile(filepath.Join(root, f), nil, 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + return root |
| +} |
| + |
| +// TestSetupAtRefusesUndelegatedRoot: without Delegate= the control files are |
| +// root-owned (here: absent) — SetupAt must error so Setup degrades to the |
| +// nil manager instead of half-arming. |
| +func TestSetupAtRefusesUndelegatedRoot(t *testing.T) { |
| + if _, err := SetupAt(t.TempDir(), t.Logf); err == nil { |
| + t.Fatal("SetupAt on an undelegated root succeeded, want error") |
| + } |
| +} |
| + |
| +// TestSetupAtDance pins the dance's observable results: the agent's own PID |
| +// moved into the supervise/ leaf (cgroup v2 forbids enabling controllers on |
| +// a cgroup with member processes), then the four controllers enabled in |
| +// subtree_control, and Root reporting the subtree. |
| +func TestSetupAtDance(t *testing.T) { |
| + root := newDelegatedRoot(t) |
| + m, err := SetupAt(root, t.Logf) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if m.Root() != root { |
| + t.Errorf("Root = %q, want %q", m.Root(), root) |
| + } |
| + pid, err := os.ReadFile(filepath.Join(root, "supervise", "cgroup.procs")) |
| + if err != nil { |
| + t.Fatalf("supervise leaf not populated: %v", err) |
| + } |
| + if string(pid) != strconv.Itoa(os.Getpid()) { |
| + t.Errorf("supervise/cgroup.procs = %q, want our own pid %d", pid, os.Getpid()) |
| + } |
| + ctrl, err := os.ReadFile(filepath.Join(root, "cgroup.subtree_control")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if string(ctrl) != "+cpu +memory +io +pids" { |
| + t.Errorf("subtree_control = %q, want %q", ctrl, "+cpu +memory +io +pids") |
| + } |
| +} |
| + |
| +// TestSetupAtSweepsStaleExecDirs: a previous agent's per-host dirs are empty |
| +// after its children died with the service; the dance removes them. A |
| +// non-empty one (something still alive in it — shouldn't happen, but rmdir |
| +// would fail anyway) and a foreign name are left alone. |
| +func TestSetupAtSweepsStaleExecDirs(t *testing.T) { |
| + root := newDelegatedRoot(t) |
| + stale := filepath.Join(root, "carlos-exec@old.example.com") |
| + if err := os.Mkdir(stale, 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + occupied := filepath.Join(root, "carlos-exec@busy.example.com") |
| + if err := os.MkdirAll(filepath.Join(occupied, "child"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + foreign := filepath.Join(root, "not-ours") |
| + if err := os.Mkdir(foreign, 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + |
| + if _, err := SetupAt(root, t.Logf); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := os.Stat(stale); !os.IsNotExist(err) { |
| + t.Errorf("stale exec dir survived the sweep: %v", err) |
| + } |
| + if _, err := os.Stat(occupied); err != nil { |
| + t.Errorf("non-empty exec dir must be tolerated, not force-removed: %v", err) |
| + } |
| + if _, err := os.Stat(foreign); err != nil { |
| + t.Errorf("foreign dir must be untouched: %v", err) |
| + } |
| +} |
| + |
| +func TestNilManagerRoot(t *testing.T) { |
| + var m *Manager |
| + if got := m.Root(); got != "" { |
| + t.Fatalf("nil Root = %q, want \"\"", got) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run tests to verify they fail** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/cgroup/ -count=1` |
| +Expected: FAIL — `SetupAt`, `Manager` undefined. |
| + |
| +- [ ] **Step 3: Write the implementation** |
| + |
| +Append to `internal/cgroup/cgroup.go` (extend the import block to |
| +`fmt`, `os`, `path/filepath`, `strconv`, `strings`, `sync/atomic`): |
| + |
| +```go |
| +// Manager owns the delegated subtree. A nil *Manager is the documented |
| +// "no delegation" state: every method is nil-safe and degrades to today's |
| +// behavior. |
| +type Manager struct { |
| + root string |
| + logf func(string, ...any) |
| + // fallback, once latched, means clone3-into-cgroup is refused here (a |
| + // seccomp filter older than the syscall, an old kernel) and children are |
| + // started normally then moved by PID. A filter is static for the life of |
| + // the process, so retrying the syscall on every spawn buys nothing. |
| + fallback atomic.Bool |
| +} |
| + |
| +// Root is the delegated subtree's path — the sampler's third scan root. |
| +// "" for a nil manager (no delegation): the sampler skips an empty root. |
| +func (m *Manager) Root() string { |
| + if m == nil { |
| + return "" |
| + } |
| + return m.root |
| +} |
| + |
| +// Setup arms exec-child accounting for the running agent: resolve our own |
| +// cgroup from /proc/self/cgroup and run the delegation dance there. Any |
| +// failure — v1-only box, old unit file without Delegate=, a test, darwin — |
| +// returns nil with one logged line, and every spawn behaves exactly as |
| +// today. Fail toward absent readings, never toward a broken spawn path. |
| +func Setup(logf func(string, ...any)) *Manager { |
| + b, err := os.ReadFile("/proc/self/cgroup") |
| + if err != nil { |
| + logf("agent: exec-child accounting disabled: %v", err) |
| + return nil |
| + } |
| + rel, ok := ParseSelfCgroup(string(b)) |
| + if !ok { |
| + logf("agent: exec-child accounting disabled: no cgroup v2 path in /proc/self/cgroup") |
| + return nil |
| + } |
| + m, err := SetupAt(filepath.Join("/sys/fs/cgroup", rel), logf) |
| + if err != nil { |
| + logf("agent: exec-child accounting disabled (no delegated subtree — unit file without Delegate=?): %v", err) |
| + return nil |
| + } |
| + logf("agent: exec-child accounting armed (delegated subtree %s)", m.Root()) |
| + return m |
| +} |
| + |
| +// SetupAt is the delegation dance against an explicit subtree root — the |
| +// testable core Setup wraps, and the constructor activator tests use over a |
| +// temp-dir fixture. Order matters: probe, move self to a leaf, THEN enable |
| +// controllers — cgroup v2 refuses subtree_control writes while the cgroup |
| +// itself still has member processes ("no internal processes"). |
| +func SetupAt(root string, logf func(string, ...any)) (*Manager, error) { |
| + // 1. Probe live delegation: both control files must be writable by us. |
| + // systemd chowns exactly these to User= when Delegate= is set; without |
| + // it this open is the cheap, side-effect-free "no" that keeps an |
| + // undelegated box on today's path. |
| + for _, name := range []string{"cgroup.procs", "cgroup.subtree_control"} { |
| + f, err := os.OpenFile(filepath.Join(root, name), os.O_WRONLY, 0) |
| + if err != nil { |
| + return nil, fmt.Errorf("probe delegation: %w", err) |
| + } |
| + f.Close() |
| + } |
| + // 2. Move ourselves into a supervise/ leaf, making step 3 legal. |
| + sup := filepath.Join(root, "supervise") |
| + if err := os.MkdirAll(sup, 0o755); err != nil { |
| + return nil, fmt.Errorf("create supervise leaf: %w", err) |
| + } |
| + if err := os.WriteFile(filepath.Join(sup, "cgroup.procs"), []byte(strconv.Itoa(os.Getpid())), 0o644); err != nil { |
| + return nil, fmt.Errorf("move self into supervise leaf: %w", err) |
| + } |
| + // 3. Enable the controllers per-child cgroups need for memory.current, |
| + // io.stat and pids accounting (cpu.stat exists regardless, but +cpu |
| + // keeps utilization attributable). One write, all four: the kernel |
| + // applies subtree_control writes atomically. |
| + if err := os.WriteFile(filepath.Join(root, "cgroup.subtree_control"), []byte("+cpu +memory +io +pids"), 0o644); err != nil { |
| + return nil, fmt.Errorf("enable controllers: %w", err) |
| + } |
| + m := &Manager{root: root, logf: logf} |
| + // 4. Sweep per-host dirs a previous agent left behind — empty after a |
| + // restart killed their members. rmdir refuses a non-empty dir, which is |
| + // exactly the tolerance we want. |
| + m.sweepStale() |
| + return m, nil |
| +} |
| + |
| +// sweepStale removes leftover carlos-exec@* dirs under the root. Failures |
| +// are logged and tolerated: a dir that won't rmdir has something alive in |
| +// it, and killing is the service manager's job, not ours. |
| +func (m *Manager) sweepStale() { |
| + entries, err := os.ReadDir(m.root) |
| + if err != nil { |
| + return |
| + } |
| + for _, e := range entries { |
| + if !e.IsDir() || !IsExecName(e.Name()) { |
| + continue |
| + } |
| + if err := os.Remove(filepath.Join(m.root, e.Name())); err != nil { |
| + m.logf("cgroup: stale %s not removed: %v", e.Name(), err) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run tests to verify they pass** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/cgroup/ -count=1` |
| +Expected: PASS |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/cgroup/cgroup.go internal/cgroup/cgroup_test.go |
| +git commit -m "cgroup: the delegation dance — probe, supervise leaf, controllers, sweep |
| + |
| +Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 3: `internal/cgroup` — spawning into per-host cgroups |
| + |
| +**Files:** |
| +- Modify: `internal/cgroup/cgroup.go` (append) |
| +- Create: `internal/cgroup/cgroup_linux.go` |
| +- Create: `internal/cgroup/cgroup_other.go` |
| +- Create: `internal/cgroup/cgroup_linux_test.go` |
| +- Modify: `internal/cgroup/cgroup_test.go` (append) |
| + |
| +**Interfaces:** |
| +- Consumes: `Manager`, `ExecName` from Tasks 1–2. |
| +- Produces: |
| + `func (m *Manager) Start(host string, build func() *exec.Cmd) (*exec.Cmd, error)` |
| + — nil-safe (nil manager = plain `build().Start()`); `build` must return a |
| + fully configured, UNSTARTED Cmd and may be called twice (a Cmd is not |
| + reusable after a failed Start). On error the returned Cmd may be nil — |
| + callers must not touch it. |
| + `func (m *Manager) CleanupHost(host string)` — nil-safe, rmdir-tolerant. |
| + Task 4 uses both. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +Append to `internal/cgroup/cgroup_test.go` (add `os/exec` and `errors` to |
| +imports): |
| + |
| +```go |
| +// fakeStart swaps the package's spawn seam for a recorder, restoring it on |
| +// cleanup. The fake never spawns anything: "success" fabricates cmd.Process |
| +// from our own PID so the PID-move path has a real number to write. |
| +func fakeStart(t *testing.T, fail func(cmd *exec.Cmd) error) *[]*exec.Cmd { |
| + t.Helper() |
| + var calls []*exec.Cmd |
| + orig := startCmd |
| + startCmd = func(cmd *exec.Cmd) error { |
| + calls = append(calls, cmd) |
| + if err := fail(cmd); err != nil { |
| + return err |
| + } |
| + cmd.Process, _ = os.FindProcess(os.Getpid()) |
| + return nil |
| + } |
| + t.Cleanup(func() { startCmd = orig }) |
| + return &calls |
| +} |
| + |
| +func buildTrue() *exec.Cmd { return exec.Command("/bin/true") } |
| + |
| +// TestStartNilManagerSpawnsPlainly: the nil manager is the no-delegation |
| +// state — one plain start, no SysProcAttr, no filesystem writes. |
| +func TestStartNilManagerSpawnsPlainly(t *testing.T) { |
| + calls := fakeStart(t, func(*exec.Cmd) error { return nil }) |
| + var m *Manager |
| + cmd, err := m.Start("h.example.com", buildTrue) |
| + if err != nil || cmd == nil { |
| + t.Fatalf("Start = (%v, %v), want started cmd", cmd, err) |
| + } |
| + if len(*calls) != 1 || (*calls)[0].SysProcAttr != nil { |
| + t.Fatalf("nil manager must start exactly once with no SysProcAttr: %d calls", len(*calls)) |
| + } |
| +} |
| + |
| +// TestStartGenuineSpawnFailureDoesNotLatch: when the retry-without-cgroup |
| +// ALSO fails, the cgroup was not the problem — the error propagates and the |
| +// fallback must not latch (the next spawn should still try clone3). |
| +func TestStartGenuineSpawnFailureDoesNotLatch(t *testing.T) { |
| + root := newDelegatedRoot(t) |
| + m, err := SetupAt(root, t.Logf) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + fakeStart(t, func(*exec.Cmd) error { return errors.New("no such binary") }) |
| + if _, err := m.Start("h.example.com", buildTrue); err == nil { |
| + t.Fatal("Start with a failing spawn succeeded, want error") |
| + } |
| + if m.fallback.Load() { |
| + t.Fatal("a genuine spawn failure latched the clone3 fallback") |
| + } |
| +} |
| + |
| +// TestCleanupHost: removes an empty per-host dir, tolerates a non-empty |
| +// one, and is nil-safe. |
| +func TestCleanupHost(t *testing.T) { |
| + root := newDelegatedRoot(t) |
| + m, err := SetupAt(root, t.Logf) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + empty := filepath.Join(root, ExecName("gone.example.com")) |
| + if err := os.Mkdir(empty, 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + m.CleanupHost("gone.example.com") |
| + if _, err := os.Stat(empty); !os.IsNotExist(err) { |
| + t.Errorf("empty host dir not removed: %v", err) |
| + } |
| + |
| + busy := filepath.Join(root, ExecName("busy.example.com")) |
| + if err := os.MkdirAll(filepath.Join(busy, "child"), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + m.CleanupHost("busy.example.com") // must not panic or force-remove |
| + if _, err := os.Stat(busy); err != nil { |
| + t.Errorf("non-empty host dir must survive: %v", err) |
| + } |
| + |
| + var nilM *Manager |
| + nilM.CleanupHost("h.example.com") // must not panic |
| +} |
| +``` |
| + |
| +Create `internal/cgroup/cgroup_linux_test.go` — the clone3 paths only |
| +exist where `sysProcAttrInto` builds a real attr: |
| + |
| +```go |
| +//go:build linux |
| + |
| +package cgroup |
| + |
| +import ( |
| + "os" |
| + "os/exec" |
| + "path/filepath" |
| + "syscall" |
| + "testing" |
| +) |
| + |
| +// TestStartUsesCgroupFDFirst: with delegation live and no latch, the child |
| +// starts via clone3-into-cgroup — SysProcAttr set, host dir created, and NO |
| +// cgroup.procs write (the kernel placed it, no PID move needed). |
| +func TestStartUsesCgroupFDFirst(t *testing.T) { |
| + root := newDelegatedRoot(t) |
| + m, err := SetupAt(root, t.Logf) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + calls := fakeStart(t, func(*exec.Cmd) error { return nil }) |
| + if _, err := m.Start("h.example.com", buildTrue); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(*calls) != 1 { |
| + t.Fatalf("started %d times, want 1", len(*calls)) |
| + } |
| + attr := (*calls)[0].SysProcAttr |
| + if attr == nil || !attr.UseCgroupFD { |
| + t.Fatalf("SysProcAttr = %+v, want UseCgroupFD", attr) |
| + } |
| + dir := filepath.Join(root, ExecName("h.example.com")) |
| + if _, err := os.Stat(dir); err != nil { |
| + t.Fatalf("host cgroup dir not created: %v", err) |
| + } |
| + if _, err := os.Stat(filepath.Join(dir, "cgroup.procs")); !os.IsNotExist(err) { |
| + t.Errorf("clone3 path must not write cgroup.procs (the kernel places the child): %v", err) |
| + } |
| +} |
| + |
| +// TestStartFallsBackAndLatches pins the refusal path: clone3 refused |
| +// (seccomp EPERM), the retry WITHOUT the cgroup fd succeeds — proving the |
| +// cgroup was the problem — the child's PID is moved by file write, and the |
| +// latch makes every later spawn skip clone3 entirely. |
| +func TestStartFallsBackAndLatches(t *testing.T) { |
| + root := newDelegatedRoot(t) |
| + m, err := SetupAt(root, t.Logf) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + calls := fakeStart(t, func(cmd *exec.Cmd) error { |
| + if cmd.SysProcAttr != nil { |
| + return syscall.EPERM // the filter refusing clone3 |
| + } |
| + return nil |
| + }) |
| + if _, err := m.Start("h.example.com", buildTrue); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(*calls) != 2 { |
| + t.Fatalf("first spawn: %d start calls, want 2 (clone3 attempt + plain retry)", len(*calls)) |
| + } |
| + procs := filepath.Join(root, ExecName("h.example.com"), "cgroup.procs") |
| + if _, err := os.Stat(procs); err != nil { |
| + t.Fatalf("fallback did not move the PID: %v", err) |
| + } |
| + if !m.fallback.Load() { |
| + t.Fatal("refusal did not latch the fallback") |
| + } |
| + |
| + if _, err := m.Start("h2.example.com", buildTrue); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(*calls) != 3 { |
| + t.Fatalf("latched spawn: %d total start calls, want 3 (no clone3 re-attempt)", len(*calls)) |
| + } |
| + if (*calls)[2].SysProcAttr != nil { |
| + t.Fatal("latched spawn still set SysProcAttr") |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run tests to verify they fail** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/cgroup/ -count=1` |
| +Expected: FAIL — `startCmd`, `Start`, `CleanupHost` undefined. |
| + |
| +- [ ] **Step 3: Write the implementation** |
| + |
| +Create `internal/cgroup/cgroup_linux.go`: |
| + |
| +```go |
| +//go:build linux |
| + |
| +package cgroup |
| + |
| +import "syscall" |
| + |
| +// intoCgroupSupported gates the clone3 CLONE_INTO_CGROUP attempt to |
| +// platforms whose SysProcAttr has the fields — everywhere else Start goes |
| +// straight to the PID-move fallback. |
| +const intoCgroupSupported = true |
| + |
| +func sysProcAttrInto(fd int) *syscall.SysProcAttr { |
| + return &syscall.SysProcAttr{UseCgroupFD: true, CgroupFD: fd} |
| +} |
| +``` |
| + |
| +Create `internal/cgroup/cgroup_other.go`: |
| + |
| +```go |
| +//go:build !linux |
| + |
| +package cgroup |
| + |
| +import "syscall" |
| + |
| +const intoCgroupSupported = false |
| + |
| +func sysProcAttrInto(fd int) *syscall.SysProcAttr { return nil } |
| +``` |
| + |
| +Append to `internal/cgroup/cgroup.go` (add `os/exec` to imports): |
| + |
| +```go |
| +// startCmd is Start's spawn seam — a package var so the clone3 refusal |
| +// path is testable without a seccomp filter in the loop. |
| +var startCmd = func(cmd *exec.Cmd) error { return cmd.Start() } |
| + |
| +// hostDir is the per-host cgroup: instance, sidecar and seed all share it — |
| +// a seed's first-boot cost and a sidecar's load ARE that instance's work. |
| +func (m *Manager) hostDir(host string) string { return filepath.Join(m.root, ExecName(host)) } |
| + |
| +// Start runs build() and starts the result inside host's cgroup. Nil-safe: |
| +// with no manager (or any accounting failure) the child starts exactly as |
| +// today — a spawn must never fail because accounting failed. build must |
| +// return a fully configured, unstarted Cmd; it may be called twice, because |
| +// a Cmd cannot be reused after a failed Start. On a non-nil error the |
| +// returned Cmd may be nil. |
| +// |
| +// Primary mechanism: clone3 with CLONE_INTO_CGROUP (SysProcAttr.UseCgroupFD) |
| +// — the child never spends a cycle in the wrong cgroup. If that start |
| +// fails, the cgroup fd may or may not be why (a seccomp filter older than |
| +// clone3 answers EPERM; so does a missing binary): retry WITHOUT the fd, |
| +// and only if the retry succeeds — proving the cgroup was the problem — |
| +// latch the fallback and move the child by PID. The window between start |
| +// and move charges the child's first moments to the agent: mis-accounting |
| +// in the safe direction. |
| +func (m *Manager) Start(host string, build func() *exec.Cmd) (*exec.Cmd, error) { |
| + if m == nil { |
| + cmd := build() |
| + return cmd, startCmd(cmd) |
| + } |
| + dir := m.hostDir(host) |
| + if err := os.MkdirAll(dir, 0o755); err != nil { |
| + m.logf("cgroup: %s: create cgroup: %v — spawning unaccounted", host, err) |
| + cmd := build() |
| + return cmd, startCmd(cmd) |
| + } |
| + if intoCgroupSupported && !m.fallback.Load() { |
| + f, err := os.Open(dir) |
| + if err != nil { |
| + m.logf("cgroup: %s: open cgroup dir: %v — using PID-move fallback for this spawn", host, err) |
| + return m.startAndMove(host, build) |
| + } |
| + cmd := build() |
| + cmd.SysProcAttr = sysProcAttrInto(int(f.Fd())) |
| + err = startCmd(cmd) |
| + f.Close() |
| + if err == nil { |
| + return cmd, nil |
| + } |
| + firstErr := err |
| + retried, retryErr := m.startAndMove(host, build) |
| + if retryErr != nil { |
| + return nil, retryErr // failed both ways: not the cgroup's fault |
| + } |
| + m.fallback.Store(true) |
| + m.logf("cgroup: clone3-into-cgroup refused (%v) — latching PID-move fallback for the life of this process", firstErr) |
| + return retried, nil |
| + } |
| + return m.startAndMove(host, build) |
| +} |
| + |
| +// startAndMove is the fallback spawn: plain start, then write the child's |
| +// PID into the host cgroup. A failed move is logged and tolerated — the |
| +// child runs (unaccounted), which beats not running. |
| +func (m *Manager) startAndMove(host string, build func() *exec.Cmd) (*exec.Cmd, error) { |
| + cmd := build() |
| + if err := startCmd(cmd); err != nil { |
| + return nil, err |
| + } |
| + procs := filepath.Join(m.hostDir(host), "cgroup.procs") |
| + if err := os.WriteFile(procs, []byte(strconv.Itoa(cmd.Process.Pid)), 0o644); err != nil { |
| + m.logf("cgroup: %s: move pid %d: %v — child stays unaccounted", host, cmd.Process.Pid, err) |
| + } |
| + return cmd, nil |
| +} |
| + |
| +// CleanupHost removes host's per-host cgroup dir after its children exit. |
| +// Nil-safe and rmdir-tolerant: while a sibling (instance vs sidecar) is |
| +// still alive the dir is non-empty and the remove fails — the LAST exiter's |
| +// call succeeds, and the next agent start's sweep catches anything missed. |
| +func (m *Manager) CleanupHost(host string) { |
| + if m == nil { |
| + return |
| + } |
| + _ = os.Remove(m.hostDir(host)) |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run tests to verify they pass** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/cgroup/ -count=1` |
| +Expected: PASS |
| + |
| +- [ ] **Step 5: gofmt + vet, then commit** |
| + |
| +Run: `gofmt -l internal/cgroup/ && GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go vet ./internal/cgroup/` |
| + |
| +```bash |
| +git add internal/cgroup/ |
| +git commit -m "cgroup: spawn into per-host cgroups — clone3 first, latched PID-move fallback |
| + |
| +Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 4: activator — spawn every exec child through the manager |
| + |
| +**Files:** |
| +- Modify: `internal/activator/backend_unit.go` (add `ExecName` next to `UnitName`, line ~66) |
| +- Modify: `internal/activator/backend_exec.go` (ExecBackend field + Start/StartSidecar/Seed paths) |
| + |
| +**Interfaces:** |
| +- Consumes: `cgroup.Manager.Start(host, build)`, `cgroup.Manager.CleanupHost(host)`, |
| + `cgroup.ExecName(host)` from Tasks 1–3. |
| +- Produces: `activator.ExecName(host string) string` (Task 7 uses it); |
| + `ExecBackend.Cgroups *cgroup.Manager` field (Task 6 wires it). |
| + |
| +- [ ] **Step 1: Add `ExecName` next to `UnitName`** |
| + |
| +In `internal/activator/backend_unit.go`, directly after the existing |
| +`UnitName` (line ~66), add (and import |
| +`github.com/carlosframework/platform/internal/cgroup`): |
| + |
| +```go |
| +// ExecName is the metrics sample key for an exec-backed tenant's children — |
| +// the delegated cgroup directory the activator spawns them into. Defined by |
| +// internal/cgroup (which creates the directories); surfaced here next to |
| +// UnitName so the console learns both of a host's legal metric keys from |
| +// the same place. |
| +func ExecName(host string) string { return cgroup.ExecName(host) } |
| +``` |
| + |
| +- [ ] **Step 2: Add the manager field to ExecBackend** |
| + |
| +In `internal/activator/backend_exec.go`, add to the `ExecBackend` struct |
| +(after the `Cache` field, line ~58), and add the `cgroup` import: |
| + |
| +```go |
| + // Cgroups, when non-nil, places every spawned child — instance, sidecar, |
| + // seed — in a per-host cgroup under the agent's delegated subtree so the |
| + // metrics sampler can account it (exec-child accounting spec). Nil — no |
| + // delegation: old unit file, tests, darwin — spawns exactly as before. |
| + Cgroups *cgroup.Manager |
| +``` |
| + |
| +- [ ] **Step 3: Route the instance spawn through the manager** |
| + |
| +In `ExecBackend.Start` (line ~440), replace this block: |
| + |
| +```go |
| + cmd := exec.Command(bin, "--socket", rt.Socket, "--db", dbPath) |
| + cmd.Env = childEnv(childVars) |
| + // The child logs through the agent's own stderr/stdout (journald under |
| + // systemd); there's no per-instance unit to own a log stream anymore. |
| + cmd.Stdout = os.Stdout |
| + cmd.Stderr = os.Stderr |
| + // Record the build we're about to exec — read the symlink immediately before |
| + // Start() so it names the version the kernel will resolve. Reading it here, |
| + // rather than back in wake() after the health wait, closes a real window: a |
| + // concurrent ship's repoint during the up-to-30s wait would otherwise make |
| + // us record the NEW target while this child runs the OLD binary, and the |
| + // sweep would then think it's already converged and never cycle it. The |
| + // residual sub-exec race (a repoint between this read and Start) is in the |
| + // safe direction — we'd record the older version, so the sweep cycles once |
| + // too often rather than never. Empty on a read error: the sweep won't cycle. |
| + spawnVer, _ := b.LiveVersion(host) |
| + if err := cmd.Start(); err != nil { |
| + return nil, "", err |
| + } |
| +``` |
| + |
| +with: |
| + |
| +```go |
| + // Record the build we're about to exec — read the symlink immediately before |
| + // the start so it names the version the kernel will resolve. Reading it here, |
| + // rather than back in wake() after the health wait, closes a real window: a |
| + // concurrent ship's repoint during the up-to-30s wait would otherwise make |
| + // us record the NEW target while this child runs the OLD binary, and the |
| + // sweep would then think it's already converged and never cycle it. The |
| + // residual sub-exec race (a repoint between this read and the start) is in |
| + // the safe direction — we'd record the older version, so the sweep cycles |
| + // once too often rather than never. Empty on a read error: the sweep won't |
| + // cycle. |
| + spawnVer, _ := b.LiveVersion(host) |
| + // Spawned through the cgroup manager (nil-tolerant: no delegation means |
| + // a plain start, exactly as before) so the child lands in |
| + // carlos-exec@<host> and the sampler can account it. build may run |
| + // twice — the manager's clone3-refusal fallback — so everything about |
| + // the Cmd is configured inside it. |
| + cmd, err := b.Cgroups.Start(host, func() *exec.Cmd { |
| + cmd := exec.Command(bin, "--socket", rt.Socket, "--db", dbPath) |
| + cmd.Env = childEnv(childVars) |
| + // The child logs through the agent's own stderr/stdout (journald |
| + // under systemd); there's no per-instance unit to own a log stream |
| + // anymore. |
| + cmd.Stdout = os.Stdout |
| + cmd.Stderr = os.Stderr |
| + return cmd |
| + }) |
| + if err != nil { |
| + return nil, "", err |
| + } |
| +``` |
| + |
| +Then in the wait goroutine a few lines below, add the cleanup: |
| + |
| +```go |
| + go func() { |
| + p.err = cmd.Wait() |
| + // Last exiter wins: rmdir refuses while the sidecar (same host |
| + // cgroup) is still alive, and that failure is tolerated. |
| + b.Cgroups.CleanupHost(host) |
| + close(p.done) |
| + }() |
| +``` |
| + |
| +- [ ] **Step 4: Route the sidecar spawn through the manager** |
| + |
| +In `StartSidecar` (line ~602), replace: |
| + |
| +```go |
| + cmd := exec.Command(b.liveLink(rt.Host), "sidecar", "run") |
| + cmd.Env = childEnv(fileEnv) |
| + // Like the instance child: logs through the agent's own streams (journald |
| + // under systemd) — no per-host unit owns a log stream anymore. |
| + cmd.Stdout = os.Stdout |
| + cmd.Stderr = os.Stderr |
| + if err := cmd.Start(); err != nil { |
| + return nil, err |
| + } |
| + p := &childProc{cmd: cmd, done: make(chan struct{})} |
| + go func() { |
| + p.err = cmd.Wait() |
| + close(p.done) |
| + }() |
| +``` |
| + |
| +with: |
| + |
| +```go |
| + // Same host cgroup as the instance: a sidecar's load is that instance's |
| + // work, and attributing it elsewhere would be a lie in the other |
| + // direction. |
| + cmd, err := b.Cgroups.Start(rt.Host, func() *exec.Cmd { |
| + cmd := exec.Command(b.liveLink(rt.Host), "sidecar", "run") |
| + cmd.Env = childEnv(fileEnv) |
| + // Like the instance child: logs through the agent's own streams |
| + // (journald under systemd) — no per-host unit owns a log stream |
| + // anymore. |
| + cmd.Stdout = os.Stdout |
| + cmd.Stderr = os.Stderr |
| + return cmd |
| + }) |
| + if err != nil { |
| + return nil, err |
| + } |
| + p := &childProc{cmd: cmd, done: make(chan struct{})} |
| + go func() { |
| + p.err = cmd.Wait() |
| + b.Cgroups.CleanupHost(rt.Host) |
| + close(p.done) |
| + }() |
| +``` |
| + |
| +- [ ] **Step 5: Route the seed through the manager** |
| + |
| +In `Seed` (line ~213), replace the block from `ctx, cancel := |
| +context.WithTimeout(ctx, seedTimeout)` through `if err := cmd.Run(); err != |
| +nil {` — keeping the whole error-handling body — with: |
| + |
| +```go |
| + ctx, cancel := context.WithTimeout(ctx, seedTimeout) |
| + defer cancel() |
| + argv := append(seedArgs, "--db", dbPath) |
| + var stderr capBuffer |
| + log.Printf("activator: seeding %s: %s %s", host, b.liveLink(host), strings.Join(argv, " ")) |
| + // The seed shares the instance's carlos-exec@<host> cgroup: first-boot |
| + // cost is that instance's work. No CleanupHost here — the instance spawn |
| + // reuses the dir moments later, and its exit path (or the next agent |
| + // start's sweep) removes it. |
| + cmd, err := b.Cgroups.Start(host, func() *exec.Cmd { |
| + cmd := exec.CommandContext(ctx, b.liveLink(host), argv...) |
| + // The purge-channel vars are agent-owned in BOTH directions here too |
| + // (Start's discipline, see withoutVars): fileEnv is tenant-supplied, so |
| + // without this a tenant's own env file could hand its seed a |
| + // CARLOS_CONTROL_SOCKET the agent never opened and a CARLOS_PURGE for a |
| + // channel that does not exist — and the seed has no serving cache to |
| + // purge in any case. This is what makes the contract above true rather |
| + // than merely intended. |
| + cmd.Env = childEnv(withoutVars(fileEnv, "CARLOS_CONTROL_SOCKET", "CARLOS_PURGE")) |
| + // Like the instance child, the seed logs through the agent's own streams |
| + // (journald under systemd) — and its stderr is ALSO teed into a bounded |
| + // buffer, because the wake error this returns is the whole diagnosis an |
| + // operator gets for a host that refuses to come up. |
| + cmd.Stdout = os.Stdout |
| + cmd.Stderr = io.MultiWriter(os.Stderr, &stderr) |
| + // WaitDelay is what makes seedTimeout real. Because Stderr is a plain |
| + // io.Writer (the tee), os/exec pipes it and Wait waits for that pipe to |
| + // close — and a seed that spawned its own grandchild holds the write end |
| + // open long after the timeout killed the seed itself. Without this, a |
| + // wedged seed would hold the lease until the wake's own 5m budget expired. |
| + cmd.WaitDelay = seedDrainGrace |
| + return cmd |
| + }) |
| + if err == nil { |
| + err = cmd.Wait() |
| + } |
| + if err != nil { |
| +``` |
| + |
| +Inside the retained error-handling body, the `ErrWaitDelay` success check |
| +must guard against a nil cmd (a spawn that never started has no |
| +ProcessState): |
| + |
| +```go |
| + if cmd != nil && errors.Is(err, exec.ErrWaitDelay) && cmd.ProcessState != nil && cmd.ProcessState.Success() { |
| +``` |
| + |
| +Everything else in the body (`DeadlineExceeded` wrap, `seedOwnsDB` cleanup, |
| +stderr tail) stays byte-identical. Note the original `log.Printf("activator: |
| +seeding %s: ...")` line moved ABOVE the spawn (it was between cmd |
| +construction and Run; keep exactly one copy). Also delete the now-orphaned |
| +comment lines that referred to `cmd.Run` if any remain outside the closure. |
| + |
| +- [ ] **Step 6: Build, run the activator suite** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go build ./... && go vet ./internal/activator/` |
| +Then (sandbox disabled — the suite binds unix sockets): |
| +`GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/activator/ -count=1` |
| +Expected: PASS — every existing test runs with `Cgroups` nil, pinning that |
| +the nil manager is behavior-identical to today. |
| + |
| +- [ ] **Step 7: gofmt, commit** |
| + |
| +Run: `gofmt -l internal/activator/` |
| + |
| +```bash |
| +git add internal/activator/backend_unit.go internal/activator/backend_exec.go |
| +git commit -m "activator: instance, sidecar and seed spawn into the host's exec cgroup |
| + |
| +Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 5: host sampler — the third scan root |
| + |
| +**Files:** |
| +- Modify: `internal/host/metrics_probes.go` (add `listExecCgroups`) |
| +- Modify: `internal/host/metrics_probes_test.go` (add discovery test) |
| +- Modify: `internal/host/metrics_loop.go` (`realUnitReadings`, |
| + `realSampleProbes`, `metricsLoop` gain the exec root) |
| +- Modify: `internal/host/agent.go:164` (call-site signature only; real wiring is Task 6) |
| + |
| +**Interfaces:** |
| +- Consumes: `cgroup.IsExecName` from Task 1. |
| +- Produces: `metricsLoop(execRoot string)` — Task 6 passes the manager's |
| + root. Sample keys `carlos-exec@<host>` flow into `metrics.Sample.Units` |
| + verbatim (additive wire change; Task 7 consumes them in the console). |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +Append to `internal/host/metrics_probes_test.go`: |
| + |
| +```go |
| +// TestListExecCgroupsFindsPerHostDirs pins the delegated subtree's scan: |
| +// carlos-exec@<host> dirs come out with their names VERBATIM as sample keys |
| +// (deliberately not dressed up as .service units — an operator who takes a |
| +// dashboard row to systemctl must not be told a unit exists that doesn't). |
| +// The agent's own supervise leaf and foreign names are invisible, and an |
| +// empty root — the no-delegation state — scans nothing. |
| +func TestListExecCgroupsFindsPerHostDirs(t *testing.T) { |
| + root := t.TempDir() |
| + mk := func(name string) string { |
| + t.Helper() |
| + dir := filepath.Join(root, name) |
| + if err := os.MkdirAll(dir, 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + return dir |
| + } |
| + execDir := mk("carlos-exec@keymail.dev") |
| + mk("supervise") |
| + mk("carlos-exec@") // no host — not one of ours |
| + mk("random-dir") |
| + |
| + got := listExecCgroups(root) |
| + if len(got) != 1 { |
| + t.Fatalf("discovered %d cgroups, want 1: %+v", len(got), got) |
| + } |
| + if got[0].name != "carlos-exec@keymail.dev" || got[0].dir != execDir { |
| + t.Fatalf("got %+v, want name carlos-exec@keymail.dev dir %s", got[0], execDir) |
| + } |
| + |
| + if out := listExecCgroups(""); out != nil { |
| + t.Fatalf("empty root must scan nothing, got %+v", out) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run test to verify it fails** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/host/ -run TestListExecCgroups -count=1` |
| +Expected: FAIL — `listExecCgroups` undefined. |
| + |
| +- [ ] **Step 3: Implement the scan** |
| + |
| +Append to `internal/host/metrics_probes.go` (add import |
| +`github.com/carlosframework/platform/internal/cgroup`): |
| + |
| +```go |
| +// listExecCgroups lists the per-host exec-child cgroups under the agent's |
| +// delegated subtree — the third scan root, alongside listCarlosUnits' two |
| +// levels. An empty root (no delegation armed) or a read failure -> nil. The |
| +// agent's own supervise leaf fails IsExecName and is skipped: that process |
| +// is already reported as carlos-edge.service. readUnitRaw reads these dirs |
| +// unchanged — same cgroup v2 files, same fail-toward-nil fields. |
| +func listExecCgroups(root string) []unitCgroup { |
| + if root == "" { |
| + return nil |
| + } |
| + entries, err := os.ReadDir(root) |
| + if err != nil { |
| + return nil |
| + } |
| + var units []unitCgroup |
| + for _, entry := range entries { |
| + if !entry.IsDir() || !cgroup.IsExecName(entry.Name()) { |
| + continue |
| + } |
| + units = append(units, unitCgroup{name: entry.Name(), dir: filepath.Join(root, entry.Name())}) |
| + } |
| + return units |
| +} |
| +``` |
| + |
| +- [ ] **Step 4: Run test to verify it passes** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/host/ -run TestListExecCgroups -count=1` |
| +Expected: PASS |
| + |
| +- [ ] **Step 5: Thread the exec root through the loop** |
| + |
| +In `internal/host/metrics_loop.go` (all three are thin glue, untested by |
| +the package's convention — the walk itself was tested in Step 1): |
| + |
| +`realSampleProbes` (line ~621) gains the root: |
| + |
| +```go |
| +func realSampleProbes(execRoot string) sampleProbes { |
| + return sampleProbes{ |
| + procStat: readProcStat, |
| + loadavg: readLoadAvg, |
| + memFrac: readMemFrac, |
| + diskFrac: readDiskFrac, |
| + units: func() map[string]unitRaw { return realUnitReadings(execRoot) }, |
| + now: time.Now, |
| + } |
| +} |
| +``` |
| + |
| +`realUnitReadings` (line ~637) merges both discovery walks — update its doc |
| +comment's last sentence and the body: |
| + |
| +```go |
| +// realUnitReadings lists the box's currently active CARLOS units and reads |
| +// each one's cgroup usage — the production units func sampleProbes.units |
| +// wraps. Keyed by UNIT NAME (what the console matches against), read from the |
| +// unit's own cgroup DIRECTORY (which for a tenant instance is a level down; |
| +// see listCarlosUnits) — plus, when delegation is armed, the per-host |
| +// exec-child cgroups under execRoot (keyed carlos-exec@<host>; see |
| +// listExecCgroups). |
| +func realUnitReadings(execRoot string) map[string]unitRaw { |
| + units := listCarlosUnits(cgroupSliceDir) |
| + units = append(units, listExecCgroups(execRoot)...) |
| + if len(units) == 0 { |
| + return nil |
| + } |
| + out := make(map[string]unitRaw, len(units)) |
| + for _, u := range units { |
| + out[u.name] = readUnitRaw(u.dir) |
| + } |
| + return out |
| +} |
| +``` |
| + |
| +`metricsLoop` (line ~702) becomes `func metricsLoop(execRoot string)` and |
| +its `probes := realSampleProbes()` line becomes |
| +`probes := realSampleProbes(execRoot)`. |
| + |
| +In `internal/host/agent.go` line 164, change `go metricsLoop()` to |
| +`go metricsLoop("")` — the real root arrives in Task 6. |
| + |
| +- [ ] **Step 6: Build + full host suite (sandbox disabled), commit** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go build ./... && go vet ./internal/host/ && go test ./internal/host/ -count=1` |
| +Expected: PASS. Then `gofmt -l internal/host/`, and: |
| + |
| +```bash |
| +git add internal/host/metrics_probes.go internal/host/metrics_probes_test.go internal/host/metrics_loop.go internal/host/agent.go |
| +git commit -m "host: the sampler scans the delegated exec subtree as a third root |
| + |
| +Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 6: agent wiring + the unit file's delegation |
| + |
| +**Files:** |
| +- Modify: `internal/host/agent.go` (RunAgent: Setup + wire manager) |
| +- Modify: `internal/host/units/carlos-edge.service` |
| + |
| +**Interfaces:** |
| +- Consumes: `cgroup.Setup`, `Manager.Root()` (Task 2), |
| + `ExecBackend.Cgroups` (Task 4), `metricsLoop(execRoot)` (Task 5). |
| +- Produces: a fully armed production path. No new API. |
| + |
| +- [ ] **Step 1: Wire the manager in RunAgent** |
| + |
| +In `internal/host/agent.go`, add the import |
| +`github.com/carlosframework/platform/internal/cgroup`, then in `RunAgent` |
| +insert after the `cache := edge.NewCache(...)` line (~132): |
| + |
| +```go |
| + // Exec-child accounting: claim the cgroup subtree the unit file |
| + // delegates to us (Delegate= in carlos-edge.service). Nil when there is |
| + // no delegation — an old unit file, a dev run outside systemd, darwin — |
| + // in which case children spawn exactly as before and simply have no |
| + // per-instance readings. |
| + cgm := cgroup.Setup(log.Printf) |
| +``` |
| + |
| +In the `activator.ExecBackend` literal (line ~146), add the field: |
| + |
| +```go |
| + Exec: &activator.ExecBackend{ |
| + Cfg: *cfg, |
| + OptDir: release.OptDir(), |
| + DataDir: dd, |
| + Cache: cache, |
| + Cgroups: cgm, |
| + }, |
| +``` |
| + |
| +And change the Task-5 placeholder `go metricsLoop("")` (line ~164) to: |
| + |
| +```go |
| + go metricsLoop(cgm.Root()) |
| +``` |
| + |
| +- [ ] **Step 2: Delegate the subtree in the unit file** |
| + |
| +In `internal/host/units/carlos-edge.service`, two edits. |
| + |
| +First, replace the line `ProtectControlGroups=yes` with: |
| + |
| +```ini |
| +# ProtectControlGroups is deliberately ABSENT: it would mount /sys/fs/cgroup |
| +# read-only, and Delegate= (below) needs this unit's own subtree writable. |
| +# The hardening that line bought is retained by ownership instead — plain |
| +# DAC keeps everything OUTSIDE the chowned subtree read-only to this |
| +# unprivileged user. |
| +``` |
| + |
| +Second, directly after the `TasksMax=1024` line, add: |
| + |
| +```ini |
| +# Exec-backed instances run as this unit's children. Delegation hands the |
| +# unit its own cgroup subtree — chowned to User=carlos, unmanaged by systemd |
| +# below this point — so the agent can give each child its own cgroup for |
| +# per-instance accounting (docs/superpowers/specs/ |
| +# 2026-08-07-exec-child-accounting-design.md). The subtree still dies with |
| +# the service: restart semantics are unchanged. |
| +Delegate=cpu memory io pids |
| +``` |
| + |
| +- [ ] **Step 3: Full gates** |
| + |
| +Run (sandbox disabled for the test step): |
| + |
| +```bash |
| +GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go build ./... && \ |
| +GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go vet ./... && \ |
| +GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/host/ ./internal/activator/ ./internal/cgroup/ -count=1 |
| +``` |
| + |
| +Expected: all PASS. (`cgroup.Setup` in a test/dev process finds |
| +`/sys/fs/cgroup/...` unwritable and returns nil — the degradation path is |
| +what makes this safe to run anywhere.) |
| + |
| +- [ ] **Step 4: Commit** |
| + |
| +```bash |
| +git add internal/host/agent.go internal/host/units/carlos-edge.service |
| +git commit -m "host: the agent claims its delegated subtree; the unit file delegates it |
| + |
| +Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +### Task 7: console — the second allowlist key |
| + |
| +**Files:** |
| +- Modify: `internal/console/metricsview.go:656-666` (OwnerAppMetrics allowlist + doc comment; also the stale comment at line ~630) |
| +- Modify: `internal/console/metricsview_test.go` (new test) |
| + |
| +**Interfaces:** |
| +- Consumes: `activator.ExecName(host)` from Task 4. |
| +- Produces: nothing new — exec-backed instances flow through the existing |
| + row assembly (`Reporting()` turns true, sparklines render). |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +Append to `internal/console/metricsview_test.go` (it already imports |
| +`metrics` and has `newMetricsFixture`, `p`, `pi64`): |
| + |
| +```go |
| +// TestOwnerAppMetricsSeesExecBackedInstances pins the second allowlist key: |
| +// an exec-backed host reports under carlos-exec@<host> (the delegated |
| +// cgroup's verbatim name, no .service), and its readings reach the owner |
| +// row exactly like a unit-backed tenant's. Another tenant's exec cgroup |
| +// stays invisible — the allowlist boundary holds for the new key form too. |
| +func TestOwnerAppMetricsSeesExecBackedInstances(t *testing.T) { |
| + srv, rel := newMetricsFixture(t) |
| + ctx := context.Background() |
| + at := time.Now().UTC().Add(-time.Minute) |
| + if err := metrics.WriteRaw(ctx, rel, metrics.RawChunk{ |
| + V: 1, Box: "box-a", Hour: metrics.HourOf(at), |
| + Samples: []metrics.Sample{{ |
| + At: at, |
| + Units: map[string]metrics.UnitSample{ |
| + "carlos-exec@keymail.example.com": {CPUFrac: p(0.22), MemBytes: pi64(2048)}, |
| + "carlos-exec@other.example.com": {CPUFrac: p(0.44), MemBytes: pi64(4096)}, |
| + }, |
| + }}, |
| + }); err != nil { |
| + t.Fatal(err) |
| + } |
| + |
| + rows, err := srv.OwnerAppMetrics(ctx, []string{"keymail.example.com"}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(rows) != 1 { |
| + t.Fatalf("rows = %+v, want exactly the one allowlisted host", rows) |
| + } |
| + got := rows[0] |
| + if got.Host != "keymail.example.com" || !got.Reporting() { |
| + t.Fatalf("exec-backed host not reporting: %+v", got) |
| + } |
| + if got.CPUFrac == nil || *got.CPUFrac != 0.22 { |
| + t.Fatalf("CPU = %v, want 0.22", got.CPUFrac) |
| + } |
| + if got.MemBytes == nil || *got.MemBytes != 2048 { |
| + t.Fatalf("mem = %v, want 2048", got.MemBytes) |
| + } |
| + if got.SparkCPU == "" || got.SparkMem == "" { |
| + t.Fatalf("sparklines empty: %+v", got) |
| + } |
| + if got.CPUFrac != nil && *got.CPUFrac == 0.44 { |
| + t.Fatalf("row carries another tenant's exec reading: %+v", got) |
| + } |
| +} |
| + |
| +// TestOwnerAppMetricsExecAndUnitKeysNeverDoubleReport pins the spec's |
| +// no-double-report property: a host can't legally report under both key |
| +// forms at once (one route, one backing), but if a sweep ever carries both |
| +// — a backing flip mid-window, two boxes — the existing freshest-box rule |
| +// picks ONE deterministically. No special case, still exactly one row. |
| +func TestOwnerAppMetricsExecAndUnitKeysNeverDoubleReport(t *testing.T) { |
| + srv, rel := newMetricsFixture(t) |
| + ctx := context.Background() |
| + fresher := time.Now().UTC().Add(-time.Minute) |
| + older := fresher.Add(-2 * time.Minute) |
| + // The older box reports the host as a systemd unit, the fresher box as |
| + // an exec cgroup — the flip-mid-window shape. |
| + if err := metrics.WriteRaw(ctx, rel, metrics.RawChunk{ |
| + V: 1, Box: "box-old", Hour: metrics.HourOf(older), |
| + Samples: []metrics.Sample{{ |
| + At: older, |
| + Units: map[string]metrics.UnitSample{ |
| + "carlos-app@flip.example.com.service": {CPUFrac: p(0.99), MemBytes: pi64(9999)}, |
| + }, |
| + }}, |
| + }); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := metrics.WriteRaw(ctx, rel, metrics.RawChunk{ |
| + V: 1, Box: "box-new", Hour: metrics.HourOf(fresher), |
| + Samples: []metrics.Sample{{ |
| + At: fresher, |
| + Units: map[string]metrics.UnitSample{ |
| + "carlos-exec@flip.example.com": {CPUFrac: p(0.11), MemBytes: pi64(1111)}, |
| + }, |
| + }}, |
| + }); err != nil { |
| + t.Fatal(err) |
| + } |
| + |
| + rows, err := srv.OwnerAppMetrics(ctx, []string{"flip.example.com"}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(rows) != 1 { |
| + t.Fatalf("rows = %+v, want exactly one row for the host, never two", rows) |
| + } |
| + got := rows[0] |
| + if got.Box != "box-new" { |
| + t.Fatalf("Box = %q, want the freshest box-new: %+v", got.Box, got) |
| + } |
| + if got.CPUFrac == nil || *got.CPUFrac != 0.11 { |
| + t.Fatalf("CPU = %v, want the freshest box's 0.11", got.CPUFrac) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run test to verify it fails** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/console/ -run TestOwnerAppMetricsSeesExecBacked -count=1` |
| +Expected: FAIL — the row is not Reporting (key unmatched). |
| + |
| +- [ ] **Step 3: Add the second key** |
| + |
| +In `internal/console/metricsview.go`, `OwnerAppMetrics` (line ~660), |
| +replace: |
| + |
| +```go |
| + want := make(map[string]string, len(hosts)) |
| + for _, h := range hosts { |
| + if h == "" { |
| + continue |
| + } |
| + want[activator.UnitName(h)] = h |
| + } |
| +``` |
| + |
| +with: |
| + |
| +```go |
| + want := make(map[string]string, 2*len(hosts)) |
| + for _, h := range hosts { |
| + if h == "" { |
| + continue |
| + } |
| + // Two legal keys per host: the systemd unit a unit-backed tenant |
| + // reports under, and the delegated exec cgroup an exec-backed one |
| + // does. One route, one backing — a host can't legally report under |
| + // both at once; if both ever appear in a sweep, the freshest-box |
| + // rule below picks deterministically, no special case. |
| + want[activator.UnitName(h)] = h |
| + want[activator.ExecName(h)] = h |
| + } |
| +``` |
| + |
| +In the same function's row assembly (line ~709), the loop currently |
| +iterates `want`'s VALUES — which was one-per-host when each host had one |
| +key, but with two keys per host would render every host twice. Replace: |
| + |
| +```go |
| + rows := make([]InstanceMetrics, 0, len(want)) |
| + for _, host := range want { |
| + if row, ok := found[host]; ok { |
| + rows = append(rows, row) |
| + } else { |
| + rows = append(rows, InstanceMetrics{Host: host}) |
| + } |
| + } |
| +``` |
| + |
| +with: |
| + |
| +```go |
| + // Iterate the callers' hosts, not want's values: want now carries TWO |
| + // keys per host, and "every allowlisted host gets exactly one row" is a |
| + // promise about hosts, not keys. seen collapses duplicate input hosts, |
| + // which the old unique-keyed map did implicitly. |
| + rows := make([]InstanceMetrics, 0, len(hosts)) |
| + seen := make(map[string]bool, len(hosts)) |
| + for _, h := range hosts { |
| + if h == "" || seen[h] { |
| + continue |
| + } |
| + seen[h] = true |
| + if row, ok := found[h]; ok { |
| + rows = append(rows, row) |
| + } else { |
| + rows = append(rows, InstanceMetrics{Host: h}) |
| + } |
| + } |
| +``` |
| + |
| +(the `sort.Slice` line after it stays unchanged). |
| + |
| +Update the function's doc comment first sentence (line ~646) from |
| +"...EXACTLY the units carlos-app@<host>.service for the given hosts" to: |
| + |
| +```go |
| +// OwnerAppMetrics filters the cached 48h sweep down to EXACTLY the given |
| +// hosts' own per-instance readings — carlos-app@<host>.service for a |
| +// unit-backed tenant, carlos-exec@<host> for an exec-backed one. It returns |
| +// unit-level rows |
| +``` |
| + |
| +(keep the rest of the comment unchanged). |
| + |
| +Also update the stale parenthetical in the `Hibernate` field comment (line |
| +~630): change "(a bespoke unit like keymail-relay.service, or an exec child |
| +of the agent)" to "(a bespoke unit like keymail-relay.service, or an exec |
| +child on a box whose agent has no delegated subtree yet)". |
| + |
| +- [ ] **Step 4: Run tests to verify they pass** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/console/ -count=1` |
| +Expected: PASS (the whole console suite — the existing allowlist and |
| +freshness tests must not regress). |
| + |
| +- [ ] **Step 5: gofmt + full repo gates, commit** |
| + |
| +```bash |
| +gofmt -l internal/console/ |
| +GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go build ./... |
| +GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go vet ./... |
| +# full suite, sandbox disabled: |
| +GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./... -count=1 |
| +git add internal/console/metricsview.go internal/console/metricsview_test.go |
| +git commit -m "console: owner metrics match exec-backed instances by their cgroup key |
| + |
| +Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>" |
| +``` |
| + |
| +--- |
| + |
| +## After the plan: rollout (operator steps, NOT part of task execution) |
| + |
| +Per spec §6 — binary first, unit file second, verify on the dev box before |
| +flagship (`annie-welding`) / tito (`betty-fossicking`) / the Pi fleet: |
| + |
| +1. Ship the new binary fleet-wide; it tolerates the old unit file (nil |
| + manager) indefinitely. |
| +2. Per box, one-time operator step (platform unit files are not |
| + auto-converged — the follow-up gap is filed in the spec, not built |
| + here): install the updated `carlos-edge.service`, `systemctl |
| + daemon-reload`, `systemctl restart carlos-edge`. |
| +3. Verify with the binary actually built (stale-binary rule): journal line |
| + "exec-child accounting armed", then exec-backed readings on the |
| + Instances page (keymail's three flagship routes are the acceptance |
| + case). Pi fleet note: a Pi kernel booted without `cgroup_enable=memory` |
| + will fail the controller-enable write and degrade to nil — check the |
| + journal line before concluding anything about a Pi. |