| 1 | # Zero-downtime flag readiness, round 3: placement under the hardened edge unit |
| 2 | |
| 3 | > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| 4 | |
| 5 | **Goal:** Make armed generation starts work on the shipped, hardened `carlos-edge.service` by replacing clone3 `CLONE_INTO_CGROUP` placement with an exec-shim, make arming prove placement before it arms, and fix the three smaller defects the second dev-box check found. |
| 6 | |
| 7 | **Architecture:** Every `cgroup.Manager` start (flat, generation, worker, legacy sidecar, seed) starts the carlos binary itself as `/proc/self/exe __cgexec`. The shim receives the tenant's exec triple over a pipe, writes its own PID into the target cgroup's `cgroup.procs` through an already-validated directory fd, checks `/proc/self/cgroup` and the directory's identity, then `execve`s the tenant. The parent learns the outcome over a close-on-exec status pipe before `Start` returns, so a shim that exits before exec is a placement failure (`cgroup.ErrLimitNotApplied` for generations) and the tenant never runs outside its cgroup. Arming does one real shim placement into a scratch cgroup and declines with a reason code if it fails. |
| 8 | |
| 9 | **Tech Stack:** Go 1.26, `golang.org/x/sys/unix`, cgroup v2, systemd 257 (transient user units for kernel tests). |
| 10 | |
| 11 | **Spec:** /home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md (rev 17), with the approved rev-18 amendment on main: `docs/superpowers/specs/2026-10-12-zero-downtime-deploys-rev18-amendment.md`. §7.3 (containment) is the section this plan serves; the spec does not name a placement mechanism, so no spec text changes. |
| 12 | |
| 13 | **Inputs:** amadan discussion carlos/platform #272, latest reply ("Dev-box checks, round 2", 2026-10-13); command log `/home/paulca/.cache/tmp/claude-1001/zdd-dev-checks2.log`; roll notes `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md`. |
| 14 | |
| 15 | **Base:** branch `zdd-flag-readiness-3` from origin/main `34a4328f`. |
| 16 | |
| 17 | ## Global Constraints |
| 18 | |
| 19 | - Paul's ruling (2026-10-13): do NOT relax `RestrictNamespaces=yes` or any other line of `internal/host/units/carlos-edge.service`. That file is not edited by this plan. |
| 20 | - A generation's tenant code never runs outside its cgroup: no PID-move, no start-then-move, no retry outside the cgroup on the required (generation) path. |
| 21 | - The shim never runs tenant code, never reads tenant-controlled environment before placement, and passes argv, env, cwd, fds 0..2+len(ExtraFiles), and the `SysProcAttr` the caller set to the tenant exactly as a direct `exec.Cmd.Start` would. |
| 22 | - The PID `cmd.Process.Pid` the activator tracks is the shim's PID, which is the tenant's after `execve`; `cmd.Wait` reports the tenant's exit. |
| 23 | - The strict path (`PrepareStrict`, `PrepareStrictParent`, `(*Prepared).Start`, `internal/cgroup/strict*.go`) keeps its clone3 placement unchanged. |
| 24 | - No new member-facing strings unless unavoidable; any that are added carry draft text and a `TODO-COPY` marker and go into `copy-review/strings.json`, and the controller stops and reports that path. |
| 25 | - Commits use the identity `claude <claude~paul@keymail.dev>`: `git -c user.name=claude -c user.email=claude~paul@keymail.dev commit ...`. Never push; the controller pushes. |
| 26 | - Sandboxed Go: `GOCACHE=$TMPDIR/gocache` on every go command. Kernel and socket tests run unsandboxed with `TMPDIR=/tmp/k`. Never set GOCACHE/GOMODCACHE/GOTMPDIR under `/tmp`. |
| 27 | - Scratch for kernel and socket runs lives in `/tmp/k/zddr3/`, which this plan owns; remove it (`rm -rf /tmp/k/zddr3`) at the end of each run, and never touch anything else under `/tmp/k`. The GOCACHE and `/tmp/k` rules are the user's instructions for this run (Ruling 15). |
| 28 | - Kernel tests on DOCS-T1 run in a transient delegated user unit, for example: |
| 29 | `systemd-run --user --wait --pipe --collect -p Delegate=yes -E TMPDIR=/tmp/k/zddr3 -E CARLOS_SHIM_KERNEL=1 <test binary> -test.run '<regex>' -test.v -test.count=1` |
| 30 | (build the binary first with `go test -c -o /tmp/k/zddr3/cgroup.test ./internal/cgroup`). Package test runs that use sockets keep `TMPDIR=/tmp/k` itself, since the activator's socket paths only fit under a root that short, and `t.TempDir` cleans up after itself there. User delegation on DOCS-T1 offers `cpu memory pids`, not `io`. |
| 31 | - Do not run anything on real boxes. Never run `git worktree prune`. Never push to main or run `amadan branch merge`. |
| 32 | - Before committing: `gofmt -l` clean on touched files, `go vet` on touched packages, and `GOOS=darwin go build ./... && GOOS=windows go build ./...` when a touched file is not linux-only (linux-only code needs `//go:build linux` and a `!linux` stub). |
| 33 | - Heavy test loops run under `nice -n 19`. |
| 34 | - Integrity: reports paste real command output only. |
| 35 | |
| 36 | ## Rulings recorded by this plan |
| 37 | |
| 38 | Each is ledgered by the controller as `Ruling: <what> — <why> — <cost if wrong>`. |
| 39 | |
| 40 | 1. **(Paul, 2026-10-13)** The exec-shim replaces clone3 placement; no hardening on carlos-edge is relaxed. — RestrictNamespaces makes systemd's seccomp filter answer clone3 with ENOSYS, so CLONE_INTO_CGROUP is impossible in the agent. — Cost if wrong: one extra exec of the carlos binary per spawn. |
| 41 | 2. Flat starts (the non-required path, flag off included) use the shim too. On a shim *placement* failure, which by the protocol's classification is reported only when non-execution is proven, a flat start falls back to today's start-then-move by PID; a generation start never does. An uncertain start is reported started, never retried, so no tenant runs twice. — The flat path's contract is that accounting never breaks a spawn, and every flat start under the shipped unit already uses start-then-move today. — Cost if wrong: a flag-off tenant can still run for a moment outside its cgroup when the shim fails, as every flag-off start does today. |
| 42 | 3. The clone3 attempt and its latch (`Manager.fallback`) leave the Manager start paths. The strict claims path keeps clone3. — Paul: keep the strict-claims path's existing placement. — Cost if wrong: with host claims on under the shipped unit, strict launches answer `ErrStrictUnsupported` (refused, never uncapped) until that path gets the shim. Claims are off everywhere and claims-on boxes decline arming; the roll notes record it as a claims-on-swaps precondition. |
| 43 | 4. The shim is `/proc/self/exe __cgexec`, used only when `cgroup.RunShimIfRequested()` has run in this process. — `/proc/self/exe` is the running build even after a roll replaces the file on disk; the hook guard stops a test binary without the hook from re-running its own tests as a "shim". — Cost if wrong: a binary that forgets the hook declines arming and its flat starts use start-then-move: loud, never unsafe. |
| 44 | 5. The tenant's path, argv and env travel over a pipe in a NUL-delimited encoding, never in the shim's argv or environment, and the shim runs with an empty environment. — Secrets stay out of `/proc/<pid>/cmdline`; no tenant variable (`GODEBUG`, `LD_*`, `GOMAXPROCS`) shapes the shim before placement; JSON would rewrite invalid UTF-8. — Cost if wrong: one pipe per spawn. |
| 45 | 6. A placement in flight counts as a member of its target cgroup and every ancestor up to the host directory: `Cleanup.Populated` answers true, `Cleanup.Members` answers `ErrPlacementInFlight`, `Cleanup.KillAndEmpty` waits for it to resolve (bounded by its ctx), and the legacy-parent guard counts one aimed at the host directory itself. — With clone3 the child appeared in its cgroup atomically; the shim spends a few milliseconds in the agent's cgroup first, and an emptiness proof must not miss it. — Cost if wrong: a stop that lands during a spawn's few milliseconds waits for it or answers unproven (the lease is kept), never a false empty. |
| 46 | 7. A new decline code `cgroup-placement-failed` reuses the approved declined phrase (`unavailableDeclined`), so it adds no copy. — Same meaning as the other declined codes for a member. — Cost if wrong: a console older than the box shows the generic unknown-code phrase for it and, because it treats an unknown code as not declined (`internal/rolloutwords/words.go:109`, `internal/console/instancepage.go:43`), cannot offer or sign a host retry for that box until it is updated (consoles roll after boxes). |
| 47 | 8. The arm-time placement check is box-wide: a failure declines the whole box, nested rows included. — One flag, one decision, as for `host-claims-on`. — Cost if wrong: a box whose nested rows could swap declines until flat placement works. |
| 48 | 9. The shim's deadline is 10 s (`shimDeadline`), once up to placement and once from the release to exec. Before `R` is seen, expiry SIGKILLs the shim, drains the status pipe, and is a placement failure only if no `R` was read (non-execution proven); with `R` read the start is reported started and nothing is killed. Failed starts are reaped with `os.Process.Wait`, which is bounded once the shim is dead, and exec.Cmd's I/O goroutines are released in the background. — A wake's budget is 30 s; a Go binary that cannot start in 10 s is a box condition. — Cost if wrong: a box that loaded refuses generation starts. |
| 49 | 10. `reset-failed` after a generation stop is best-effort: a failure is logged and the stop still answers 204. — The stop itself succeeded; the failed-list entry is cosmetic. — Cost if wrong: a failed generation unit can linger in `systemctl --failed`. |
| 50 | 11. The roll notes' probe app moves into the module as `cmd/zddprobe`, with a test that launches it with the platform's argv. — A heredoc cannot be tested. — Cost if wrong: one more `cmd/` directory. |
| 51 | 12. The 3000-seed simulation sweep runs only if the branch diff touches `internal/rollout` or the simulation's files; the shim, the arming check and the stop path are outside the modelled scheduler. — Cost if wrong: an unswept scheduler change; the final review checks the diff's paths. |
| 52 | 13. `before` (the caller's last ownership check) moves from before the fork to after placement, immediately before exec, behind a placed/release handshake (Astra plan 2). — `internal/activator/genbackend.go:69` requires the check after preparation and immediately before execution; with the shim, placement is preparation. — Cost if wrong: a refused start now costs a fork and a placement that are killed and reaped. |
| 53 | 14. The shim's exec thread gets an empty signal mask; dispositions are left to `execve`. — systemd starts every service with an empty mask, and Go's own forked child restores the mask the agent started with; the kernel test compares a shim start with a direct start under an inherited ignored SIGHUP, `Pdeathsig`, `Setpgid`, capabilities and `RLIMIT_NOFILE`. — Cost if wrong: a tenant started by an agent that inherited blocked signals would see them unblocked. |
| 54 | 15. `GOCACHE=$TMPDIR/gocache` in the sandbox and `/tmp/k` for socket and kernel runs stand (Astra plan 11). — They are the user's instructions for this run; scratch is confined to `/tmp/k/zddr3`, removed after each run. — Cost if wrong: none. |
| 55 | 16. The tenant's `cmd.Dir` is applied by the shim after placement, not by exec.Cmd at the shim's fork (Astra plan 5). — A bad `Dir` is then the command's own error (`*os.PathError{Op: "chdir"}`), never misread as a placement failure. — Cost if wrong: one `chdir` per spawn moves from Go's fork path to the shim. |
| 56 | |
| 57 | ## Review Focus |
| 58 | |
| 59 | 1. **Exact pass-through.** A tenant env with invalid UTF-8, an empty value, `=` inside a value and a 100 KiB value, and an argv with an empty element, a newline and spaces reach the tenant byte-for-byte, the same as a direct start; a value over Linux's per-string limit fails with the same `E2BIG` a direct start gets (Task 1, `TestShimKernelPassesArgvEnvExactly`). |
| 60 | 2. **Inherited state.** A `build()` that sets `ExtraFiles` (two pipes) gets them at fds 3 and 4 in the tenant, and the tenant has no other fd above 2: none of the shim's spec, directory or status fds, and none the agent opened without close-on-exec. Signal mask and ignored set, capabilities, `Pdeathsig`, process group and `RLIMIT_NOFILE` equal a direct start's (Task 1, `TestShimKernelInheritedStateMatchesADirectStart`). |
| 61 | 3. **The agent's binary replaced on disk.** After a roll replaces `/opt/carlos/carlos`, the running agent's shim still execs (Task 1, `TestShimKernelSurvivesDeletedExecutable`). |
| 62 | 4. **A stop during a spawn.** `KillAndEmpty` or `Populated` on a host whose generation is being placed never reports empty while the shim is in flight (Task 2, `TestCleanupCountsPlacementInFlight`). |
| 63 | 5. **A missing tenant binary.** The start returns the command's own `fork/exec ... no such file or directory` error, not `ErrLimitNotApplied`, nothing ran, the shim is reaped, and an empty generation directory is removed (Task 1 and Task 2, `TestShimExecFailureIsTheCommands`, `TestGenerationMissingBinaryIsNotABoxCondition`). |
| 64 | |
| 65 | --- |
| 66 | |
| 67 | ## File structure |
| 68 | |
| 69 | | File | Task | Responsibility | |
| 70 | |---|---|---| |
| 71 | | `internal/cgroup/shim.go` | 1 | Verbs, the hook (`RunShimIfRequested`, `shimAvailable`), the spec encoding, `placementError`, `ErrPlacementInFlight`. All platforms. | |
| 72 | | `internal/cgroup/shim_linux.go` | 1 | `runShim`, `runProbe`, `startViaShim` (parent side). | |
| 73 | | `internal/cgroup/shim_other.go` | 1 | `!linux` stubs: the shim verbs exit 127, `startViaShim` answers a placement error. | |
| 74 | | `internal/cgroup/main_test.go` | 1 | `TestMain` calling `RunShimIfRequested()`. | |
| 75 | | `internal/cgroup/shim_test.go` | 1 | Encoding round-trips (all platforms). | |
| 76 | | `internal/cgroup/shim_linux_test.go` | 1 | Ungated parent/shim protocol tests that need no delegation. | |
| 77 | | `internal/cgroup/shim_kernel_linux_test.go` | 1 | Gated (`CARLOS_SHIM_KERNEL=1`) real-cgroup shim tests. | |
| 78 | | `main.go` | 1 | `cgroup.RunShimIfRequested()` as the first statement of `main()`. | |
| 79 | | `internal/cgroup/inflight.go` | 1 | The in-flight placement registry on `Manager`. | |
| 80 | | `internal/cgroup/cgroup.go`, `cgroup_linux.go`, `cgroup_other.go`, `gen.go`, `cleanup_linux.go` | 2 | `startIn` on the shim; latch removed; Cleanup and the parent guard count placements in flight. | |
| 81 | | `internal/cgroup/verify_linux.go`, `verify_other.go` | 3 | `Manager.VerifyPlacement`. | |
| 82 | | `internal/host/rollout.go`, `internal/rolloutwords/words.go`, `internal/status/snapshot.go` | 3 | The decline code and its wording. | |
| 83 | | `internal/cgroup/edge_hardening_kernel_linux_test.go` | 4 | The transient-unit test under the edge unit's sandbox lines. | |
| 84 | | `cmd/zddprobe/main.go`, `cmd/zddprobe/main_test.go` | 5 | The roll notes' probe app, tested with the platform's argv. | |
| 85 | | `internal/cgroup/strict_kernel_linux_test.go` | 6 | `logf` on the test's second managers. | |
| 86 | | `internal/instancectl/gen.go` (+ its test) | 6 | `reset-failed` after a generation stop. | |
| 87 | | `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` | 5, 7 | The probe section; the round-3 notes and the dev re-check list. | |
| 88 | |
| 89 | --- |
| 90 | |
| 91 | ### Task 1: The exec-shim (opus) |
| 92 | |
| 93 | **Files:** |
| 94 | - Create: `internal/cgroup/shim.go`, `internal/cgroup/shim_linux.go`, `internal/cgroup/shim_other.go`, `internal/cgroup/inflight.go`, `internal/cgroup/main_test.go`, `internal/cgroup/shim_test.go`, `internal/cgroup/shim_linux_test.go`, `internal/cgroup/inflight_linux_test.go`, `internal/cgroup/shim_kernel_linux_test.go` |
| 95 | - Modify: `main.go:20` (first statement of `main()`); `internal/cgroup/cgroup.go:173-184` (the registry's fields on `Manager`) |
| 96 | |
| 97 | **Interfaces:** |
| 98 | - Produces (used by Tasks 2 to 4): |
| 99 | - `const ShimVerb = "__cgexec"`, `const ProbeVerb = "__cgprobe"` |
| 100 | - `func RunShimIfRequested()` — first statement of `main()` and of `TestMain`. Marks the process as able to serve the shim; if `os.Args[1]` is `ShimVerb` or `ProbeVerb` it runs that and calls `os.Exit`, never returning. |
| 101 | - `func shimAvailable() bool` — the hook ran in this process and the platform is linux. |
| 102 | - `type placementError struct{ err error }` with `Error()` and `Unwrap()`; `func isPlacementError(err error) bool`. |
| 103 | - `var ErrPlacementInFlight = errors.New("cgroup: a process is being placed in this cgroup")` (defined here, used by Task 2). |
| 104 | - `func (m *Manager) startViaShim(dir string, cmd *exec.Cmd, before func() error) (refused bool, err error)` — linux. On nil error the tenant is running in `dir` and `cmd.Process` is it. `refused` means `before` said no and nothing started. Any other error: nothing ran and any shim process is reaped; a `*placementError` means the process could not be placed, anything else is the command's own error (as `exec.Cmd.Start` would have returned it). It registers the placement in flight itself, after `before` agrees. |
| 105 | - The in-flight registry: `beginPlacement`, `placementsUnder`, `placementsAt`, `waitPlacementsUnder` (signatures below). |
| 106 | - `var shimDeadline = 10 * time.Second` |
| 107 | - `var shimExe = "/proc/self/exe"` |
| 108 | - Consumes: `ParseSelfCgroup` (`internal/cgroup/cgroup.go:155`). |
| 109 | |
| 110 | **The protocol** (revised after Astra's plan review, `CODEX-FINDINGS-ZDD-READINESS3-PLAN.md` findings 1, 2, 3 and 5). |
| 111 | |
| 112 | The rule the classification rests on: the shim writes `R` immediately before `execve`, and never earlier. So a status stream with no `R` proves no tenant code ran, and any stream with `R` and no `E` after it means the tenant may have run and the start is reported as started. A start is only ever called a failure when non-execution is proven; an uncertain one is a started process the caller owns (its `Wait` will show how it ended), so the activator's descendant proofs (`internal/activator/backend_exec.go:1037-1106`) still run for it, and the flat fallback never runs a tenant twice. |
| 113 | |
| 114 | Parent, in `startViaShim`: |
| 115 | 1. If `!shimAvailable()`, return `&placementError{errors.New("the placement shim is not available in this process")}`. If `cmd.Err != nil` (a failed `LookPath` in `build()`), return `cmd.Err`. |
| 116 | 2. Open `dir` with `unix.Open(dir, O_RDONLY|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC, 0)`. `unix.Fstatfs` it; anything but `unix.CGROUP2_SUPER_MAGIC` is a placement error. Compute `expected`: `dir` must start with `/sys/fs/cgroup/`; `expected` is `dir` with the `/sys/fs/cgroup` prefix cut (so it starts with `/`). Anything else is a placement error. |
| 117 | 3. Capture the tenant before rewriting anything: `path := cmd.Path`; `argv := cmd.Args`, or `[]string{cmd.Path}` when empty (exec.Cmd's own rule); `env := cmd.Environ()` (exactly what `Start` would pass, `PWD` included); `wd := cmd.Dir`; `keep := 3 + len(cmd.ExtraFiles)`. |
| 118 | 4. `os.Pipe()` twice: spec (parent writes the spec and later the release byte; shim reads) and status (shim writes; parent reads). Rewrite `cmd`: `Path = shimExe`, `Args = []string{"carlos-cgexec", ShimVerb, itoa(keep), itoa(keep+1), itoa(keep+2)}`, `Env = []string{}`, `Dir = ""` (the shim changes directory itself, after placement, so a bad `Dir` is reported as the command's error and never as a placement error), `ExtraFiles = append(original ExtraFiles, specR, dirFile, statusW)`. Leave `Stdin`, `Stdout`, `Stderr`, `SysProcAttr`, `Cancel`, `WaitDelay` untouched. |
| 119 | 5. One absolute deadline `D := now + shimDeadline` covers steps 5 to 7. `done := m.beginPlacement(dir)` (the in-flight registry, below), then `startCmd(cmd)` (`internal/cgroup/cgroup.go:332`). Close the parent's `specR`, `dirFile` and `statusW` whatever happened. A failed `startCmd` means the carlos binary itself did not start (the tenant's `Dir` is not part of this start): `done()`, close `specW` and `statusR`, return a placement error. |
| 120 | 6. Write the spec to `specW` (an 8-byte big-endian length, then the encoding below) with write deadline `D`. Do not close `specW`: the release byte goes on it later. |
| 121 | 7. Read `statusR` line by line with read deadline `D` until `A`, `P`, or EOF: |
| 122 | - `P <reason>`, or EOF: the shim failed or died before placement was confirmed. No tenant code ran. |
| 123 | - deadline: SIGKILL the shim. No release byte was sent, so it cannot have exec'd. No tenant code ran. |
| 124 | - In both cases: `reap(cmd)` (below), `done()`, close `specW` and `statusR`, return a placement error naming the reason, "the shim exited before placement" or "no answer from the shim within 10s". |
| 125 | - `A`: the process is in `dir` now, visible to every reader of the cgroup, so `done()` here. The registry covers only fork-to-placement, and no caller code runs inside it. |
| 126 | 8. Ask `before` (when set), now, after placement and immediately before exec, as `internal/activator/genbackend.go:69` requires. A refusal: close `specW` without the release byte (the shim reads EOF and exits 125 without exec), `reap(cmd)`, close `statusR`, return `true, before's error`. |
| 127 | 9. Write the release byte `X` to `specW` and close it. A second deadline `D2 := now + shimDeadline`. Read `statusR` until EOF with deadline `D2`: |
| 128 | - `E chdir <errno>`: `reap(cmd)`, return `&os.PathError{Op: "chdir", Path: wd, Err: syscall.Errno(n)}`. No tenant code ran. |
| 129 | - `E exec <errno>` (after `R`): `reap(cmd)`, return `&os.PathError{Op: "fork/exec", Path: path, Err: syscall.Errno(n)}`. No tenant code ran: `execve` returned. |
| 130 | - EOF with no `R`: the shim died after placement and before `R`. No tenant code ran. `reap(cmd)`, return a placement error, "the shim exited before exec". |
| 131 | - `R` then EOF: started. Close `statusR`, return `false, nil`. |
| 132 | - deadline with `R` already read: started (an `execve` still loading). Close `statusR`, return `false, nil`, and do not kill. |
| 133 | - deadline with no `R`: SIGKILL the shim, then read `statusR` to EOF (the shim is dead, so the write end is closed). An `R` in what was read: started, return `false, nil`; the caller's `Wait` will show the SIGKILL. No `R`: `reap(cmd)`, placement error. No tenant code ran. |
| 134 | 10. `reap(cmd)`: `cmd.Process.Kill()`, then `cmd.Process.Wait()`, which reaps the PID and is bounded (no tenant ran, so nothing else holds the child's pipes open once it is dead), then `go cmd.Wait()` so exec.Cmd's copying goroutines and pipes are released in the background without the caller's I/O holding this call (`os/exec/exec.go:979`). A failed start's `cmd` is never handed back. |
| 135 | 11. Every return closes `specW` and `statusR` exactly once (a single deferred close of whichever are still open). |
| 136 | |
| 137 | Shim, in `runShim(args []string) int` (args are the three fd numbers): |
| 138 | 1. Parse the three fds; they must be `keep`, `keep+1`, `keep+2` for some `keep >= 3`. Set `FD_CLOEXEC` on the status fd first, so every later path closes it at exec. |
| 139 | 2. Read the 8-byte length (at most 16 MiB) and then exactly that many bytes from the spec fd. Decode them. Keep the spec fd open: the release byte comes on it. |
| 140 | 3. Mark every fd `>= keep` close-on-exec: `unix.CloseRange(uint(keep), math.MaxUint32, unix.CLOSE_RANGE_CLOEXEC)`; if that answers `ENOSYS`, `EPERM` or `EINVAL`, walk `/proc/self/fd` and set `FD_CLOEXEC` on each fd `>= keep`. This covers the spec, directory and status fds and anything the agent leaked without close-on-exec. |
| 141 | 4. `unix.Fstatfs(dirFD)` must be `CGROUP2_SUPER_MAGIC`. |
| 142 | 5. `unix.Openat(dirFD, "cgroup.procs", O_WRONLY|O_NOFOLLOW|O_CLOEXEC, 0)`; write `strconv.Itoa(os.Getpid())`; close. This moves the whole process (every Go runtime thread) while it is still carlos code. |
| 143 | 6. Read `/proc/self/cgroup`; `ParseSelfCgroup`; `"/"+rel` must equal the spec's `Expected`. |
| 144 | 7. `unix.Fstat(dirFD)` and `unix.Lstat("/sys/fs/cgroup"+Expected)` must agree on `Dev` and `Ino`, so the cgroup the kernel reports is the directory the parent validated. |
| 145 | 8. Any failure in 1 to 7: write `P <step>: <error>\n` to the status fd and return 125. Tenant code has not run. |
| 146 | 9. Close the directory fd. Write `A\n`. |
| 147 | 10. Read one byte from the spec fd. Anything but `X` (EOF, an error, another byte): return 125 without writing more. The parent refused or is gone. |
| 148 | 11. If the spec's `Dir` is not empty, `os.Chdir(Dir)`; a failure writes `E chdir <errno>\n` and returns 126. |
| 149 | 12. Restore the exec state a direct start would have: `runtime.LockOSThread()`, then set this thread's signal mask to empty (`unix.PthreadSigmask(unix.SIG_SETMASK, &unix.Sigset_t{}, nil)`). Go's own forked child restores the mask the agent started with (`m.sigmask`), and systemd starts every service with an empty mask, so empty matches a direct start under systemd. Dispositions need no work: `execve` resets every signal the shim's runtime handles to default, and a signal the agent inherited as ignored (`SIGHUP`, `SIGINT`) is inherited ignored by both a direct child and the shim, whose runtime keeps it ignored. `syscall.Exec` restores the original `RLIMIT_NOFILE` (Go source `syscall/exec_unix.go:286`). `Pdeathsig`, the process group, and ambient capabilities are set at the shim's fork and survive both `execve`s exactly as they survive one. |
| 150 | 13. Write `R\n`, then `syscall.Exec(spec.Path, spec.Argv, spec.Env)`. If it returns, write `E exec <errno>\n` and return 126. |
| 151 | |
| 152 | The shim writes nothing to fds 0 to 2. |
| 153 | |
| 154 | `runProbe() int` (for Task 3): read `/proc/self/cgroup`, print `"/"+rel` and a newline to stdout, return 0; return 1 when there is no v2 line. |
| 155 | |
| 156 | **The in-flight registry** (`inflight.go`, used by Task 2's proofs): a mutex-guarded `map[string]int` of target directories (cleaned paths), plus a `chan struct{}` that is closed and replaced whenever a count drops, so waiters wake. |
| 157 | |
| 158 | ```go |
| 159 | func (m *Manager) beginPlacement(dir string) (done func()) // done is idempotent |
| 160 | func (m *Manager) placementsUnder(dir string) int // targets equal to dir or below dir+"/" |
| 161 | func (m *Manager) placementsAt(dir string) int // targets equal to dir |
| 162 | func (m *Manager) waitPlacementsUnder(ctx context.Context, dir string) error // nil at zero, ctx.Err() if ctx ends first |
| 163 | ``` |
| 164 | |
| 165 | A nil `*Manager` answers zero and nil, and its `beginPlacement` returns a no-op. |
| 166 | |
| 167 | **Spec encoding** (`shim.go`): fields each terminated by `\x00`, in order: the magic `carlos-cgexec-1`, `Path`, `Dir`, `Expected`, `KeepFDs` (decimal), `TestHold` (decimal, `0` for none), `len(Argv)` (decimal), each argv element, `len(Env)` (decimal), each env entry. Encoding refuses any string that contains `\x00` (execve could not carry it anyway). Decoding refuses a wrong magic, a short or trailing buffer, or a count above 1<<20. |
| 168 | |
| 169 | ```go |
| 170 | type shimSpec struct { |
| 171 | Path string |
| 172 | Dir string // the tenant's cmd.Dir; the shim changes to it after placement |
| 173 | Argv []string |
| 174 | Env []string |
| 175 | Expected string // "/..." as /proc/self/cgroup shows it after "0::" |
| 176 | KeepFDs int // fds 0..KeepFDs-1 are the tenant's |
| 177 | TestHold int // test binaries only: 0 none, 1 hold before step 4, 2 hold after step 10 (before R) |
| 178 | } |
| 179 | |
| 180 | func encodeShimSpec(s shimSpec) ([]byte, error) |
| 181 | func decodeShimSpec(b []byte) (shimSpec, error) |
| 182 | ``` |
| 183 | |
| 184 | `TestHold` is honoured only when `testing.Testing()` is true in the shim process (the package already imports `testing`, `cgroup.go:30`), so in the carlos binary it is inert. When honoured, the shim sleeps at that point until it is signalled, with default signal dispositions (no `signal.Notify`): a test's SIGTERM or SIGKILL kills it there. The parent sets the field from `var shimTestHold int`, and calls `var shimAfterRelease func()` (nil in production) between writing the release byte and reading the status, which only test files set. |
| 185 | |
| 186 | - [ ] **Step 1: Write the failing tests** |
| 187 | |
| 188 | `internal/cgroup/main_test.go`: |
| 189 | |
| 190 | ```go |
| 191 | package cgroup |
| 192 | |
| 193 | import ( |
| 194 | "os" |
| 195 | "testing" |
| 196 | ) |
| 197 | |
| 198 | func TestMain(m *testing.M) { |
| 199 | RunShimIfRequested() |
| 200 | os.Exit(m.Run()) |
| 201 | } |
| 202 | ``` |
| 203 | |
| 204 | `internal/cgroup/shim_test.go` (all platforms): |
| 205 | |
| 206 | ```go |
| 207 | func TestShimSpecRoundTripsExactBytes(t *testing.T) { |
| 208 | in := shimSpec{ |
| 209 | Path: "/run/carlos/exe/app", |
| 210 | Argv: []string{"app", "", "two words", "line\nbreak", "--db", "/data/x.db"}, |
| 211 | Env: []string{"A=1", "EMPTY=", "EQ=a=b", "BAD=\xff\xfe", "BIG=" + strings.Repeat("x", 1<<20)}, |
| 212 | Expected: "/system.slice/carlos-edge.service/carlos-exec@a.example/g3", |
| 213 | KeepFDs: 5, |
| 214 | } |
| 215 | b, err := encodeShimSpec(in) |
| 216 | if err != nil { |
| 217 | t.Fatal(err) |
| 218 | } |
| 219 | out, err := decodeShimSpec(b) |
| 220 | if err != nil { |
| 221 | t.Fatal(err) |
| 222 | } |
| 223 | if !reflect.DeepEqual(in, out) { |
| 224 | t.Fatalf("round trip changed the spec") |
| 225 | } |
| 226 | } |
| 227 | |
| 228 | func TestShimSpecRefusesNUL(t *testing.T) { |
| 229 | if _, err := encodeShimSpec(shimSpec{Path: "/x", Argv: []string{"a\x00b"}, Expected: "/g", KeepFDs: 3}); err == nil { |
| 230 | t.Fatal("a NUL inside argv was encoded") |
| 231 | } |
| 232 | } |
| 233 | |
| 234 | func TestShimSpecRefusesTruncatedAndTrailing(t *testing.T) { |
| 235 | b, _ := encodeShimSpec(shimSpec{Path: "/x", Argv: []string{"x"}, Env: []string{"A=1"}, Expected: "/g", KeepFDs: 3}) |
| 236 | for _, bad := range [][]byte{b[:len(b)-1], append(append([]byte{}, b...), 'z', 0), []byte("carlos-cgexec-0\x00")} { |
| 237 | if _, err := decodeShimSpec(bad); err == nil { |
| 238 | t.Fatalf("decoded a bad spec %q", bad) |
| 239 | } |
| 240 | } |
| 241 | } |
| 242 | ``` |
| 243 | |
| 244 | `internal/cgroup/shim_linux_test.go` (ungated; a temp directory is not cgroupfs, so these exercise the failure paths): |
| 245 | |
| 246 | ```go |
| 247 | // TestShimRefusesANonCgroupDirectory: the parent's own check (step 2) |
| 248 | // refuses before anything starts, as a placement error. |
| 249 | func TestShimRefusesANonCgroupDirectory(t *testing.T) { |
| 250 | m := &Manager{root: t.TempDir(), logf: t.Logf} |
| 251 | marker := filepath.Join(t.TempDir(), "ran") |
| 252 | cmd := exec.Command("/bin/sh", "-c", "touch "+marker) |
| 253 | refused, err := m.startViaShim(m.root, cmd, nil) |
| 254 | if refused || !isPlacementError(err) { |
| 255 | t.Fatalf("refused=%v err=%v, want a placement error", refused, err) |
| 256 | } |
| 257 | if cmd.Process != nil { |
| 258 | t.Fatal("a process was started") |
| 259 | } |
| 260 | if _, err := os.Stat(marker); err == nil { |
| 261 | t.Fatal("tenant code ran") |
| 262 | } |
| 263 | } |
| 264 | |
| 265 | // TestShimUnavailableWithoutTheHook: a process whose main never ran the |
| 266 | // hook never starts a shim. |
| 267 | func TestShimUnavailableWithoutTheHook(t *testing.T) { ... flip the hooked flag off for the test, restore in Cleanup; startViaShim answers a placement error and cmd.Process stays nil ... } |
| 268 | |
| 269 | // TestShimPlacementFailureIsReported runs a real shim (this test binary, |
| 270 | // through TestMain's hook) against a temp directory. Split startViaShim |
| 271 | // into its checked prologue (steps 1 to 3) and a core, |
| 272 | // func (m *Manager) startShimCore(dir string, dirFile *os.File, expected string, cmd *exec.Cmd, before func() error) (bool, error) |
| 273 | // and call the core with a temp directory's fd, which the prologue would |
| 274 | // have refused. The shim's own fstatfs check (shim step 4) answers "P". |
| 275 | func TestShimPlacementFailureIsReported(t *testing.T) { |
| 276 | m := &Manager{root: t.TempDir(), logf: t.Logf} |
| 277 | f, err := os.Open(m.root) |
| 278 | if err != nil { |
| 279 | t.Fatal(err) |
| 280 | } |
| 281 | defer f.Close() |
| 282 | marker := filepath.Join(t.TempDir(), "ran") |
| 283 | cmd := exec.Command("/bin/sh", "-c", "touch "+marker) |
| 284 | _, err = m.startShimCore(m.root, f, "/nowhere", cmd, nil) |
| 285 | if !isPlacementError(err) { |
| 286 | t.Fatalf("err=%v, want a placement error", err) |
| 287 | } |
| 288 | if _, err := os.Stat(fmt.Sprintf("/proc/%d", cmd.Process.Pid)); err == nil { |
| 289 | t.Fatal("the shim was not reaped") |
| 290 | } |
| 291 | if _, err := os.Stat(marker); err == nil { |
| 292 | t.Fatal("tenant code ran") |
| 293 | } |
| 294 | if m.placementsUnder(m.root) != 0 { |
| 295 | t.Fatal("the placement is still registered") |
| 296 | } |
| 297 | } |
| 298 | |
| 299 | // TestShimBeforeRefusalRegistersNothing: before says no, nothing starts and |
| 300 | // nothing is registered. |
| 301 | func TestShimBeforeRefusalRegistersNothing(t *testing.T) { ... } |
| 302 | |
| 303 | // TestPlacementsUnderCountsTheSubtree and TestWaitPlacementsUnderWakes |
| 304 | // cover the registry (inflight_linux_test.go): |
| 305 | func TestPlacementsUnderCountsTheSubtree(t *testing.T) { |
| 306 | m := &Manager{root: "/r", logf: t.Logf} |
| 307 | done := m.beginPlacement("/r/carlos-exec@a.example/g2") |
| 308 | if m.placementsUnder("/r/carlos-exec@a.example") != 1 || m.placementsAt("/r/carlos-exec@a.example") != 0 { |
| 309 | t.Fatal("a child's placement is under its parent, not at it") |
| 310 | } |
| 311 | if m.placementsUnder("/r/carlos-exec@a.exampleX") != 0 { |
| 312 | t.Fatal("a prefix without a separator matched") |
| 313 | } |
| 314 | done() |
| 315 | done() // idempotent |
| 316 | if m.placementsUnder("/r/carlos-exec@a.example") != 0 { |
| 317 | t.Fatal("done did not drop the count") |
| 318 | } |
| 319 | } |
| 320 | |
| 321 | func TestWaitPlacementsUnderWakes(t *testing.T) { ... one in flight; a waiter with a 50 ms ctx returns context.DeadlineExceeded; a second waiter returns nil within 1 s of done() ... } |
| 322 | ``` |
| 323 | |
| 324 | The exec-failure case needs a placement that succeeds, so it lives in the kernel file below. This file also covers the two status classifiers, which `shim_linux.go` factors out of `startViaShim`: |
| 325 | |
| 326 | ```go |
| 327 | // classifyPlacement reads the stream up to placement (parent step 7). |
| 328 | // placed is true only for "A". timedOut is the read deadline having passed. |
| 329 | func classifyPlacement(b []byte, timedOut bool) (placed bool, err error) |
| 330 | |
| 331 | // classifyExec reads the stream after the release byte (parent step 9). |
| 332 | // killed means the deadline passed with no R, the shim was SIGKILLed, and |
| 333 | // b is everything read until EOF after that. |
| 334 | func classifyExec(path, wd string, b []byte, killed bool) (started bool, err error) |
| 335 | |
| 336 | func TestShimStatusClassification(t *testing.T) { |
| 337 | for _, c := range []struct { |
| 338 | in string |
| 339 | timedOut bool |
| 340 | placed bool |
| 341 | }{ |
| 342 | {"A\n", false, true}, |
| 343 | {"P procs: permission denied\n", false, false}, |
| 344 | {"", false, false}, // died before placement |
| 345 | {"", true, false}, // deadline before placement |
| 346 | {"garbage\n", false, false}, |
| 347 | } { ... placed must match; !placed must give isPlacementError(err) ... } |
| 348 | |
| 349 | for _, c := range []struct { |
| 350 | in string |
| 351 | killed bool |
| 352 | started bool |
| 353 | op string // "" for a placement error, else the *os.PathError's Op |
| 354 | errno syscall.Errno |
| 355 | }{ |
| 356 | {"R\n", false, true, "", 0}, |
| 357 | {"R\n", true, true, "", 0}, // R then killed: may have exec'd, so started |
| 358 | {"", false, false, "", 0}, // died after release, before R: placement error |
| 359 | {"", true, false, "", 0}, // killed before R: placement error |
| 360 | {"E chdir 2\n", false, false, "chdir", syscall.ENOENT}, |
| 361 | {"R\nE exec 2\n", false, false, "fork/exec", syscall.ENOENT}, |
| 362 | {"R\nE exec 7\n", false, false, "fork/exec", syscall.E2BIG}, |
| 363 | {"garbage\n", false, false, "", 0}, |
| 364 | } { ... started must match; when !started, op "" means isPlacementError(err), else errors.As *os.PathError with that Op, Path "/bin/app" for fork/exec and "/wd" for chdir, and Err == errno ... } |
| 365 | } |
| 366 | ``` |
| 367 | |
| 368 | `internal/cgroup/shim_kernel_linux_test.go` (gated: `CARLOS_SHIM_KERNEL=1` inside a delegated unit, with `genKernelManager`'s setup at `gen_kernel_linux_test.go:27`; without the variable every test skips, but WITH it any delegation failure is `t.Fatal`, never a skip, so a run that was asked for cannot pass by skipping). Each test places into `m.root/<name>` made with `os.Mkdir` and a `TierDefault` limit written through `m.applyLimit(..., true)`. Tenants that must not run touch a marker file first thing, and each such test asserts the marker is absent: |
| 369 | |
| 370 | - `TestShimKernelPlacesBeforeExec`: tenant `/bin/sh -c 'cat /proc/self/cgroup; echo $$'`. After `startViaShim` returns: `/proc/<pid>/cgroup` names the target; the tenant's printed cgroup is the target; the tenant's printed `$$` equals `cmd.Process.Pid` (the leader PID is the shim's, now the tenant's); `cmd.Wait()` returns nil. |
| 371 | - `TestShimKernelExitCodePassesThrough`: tenant `/bin/sh -c 'exit 7'`; `cmd.Wait()` gives an `*exec.ExitError` with `ExitCode() == 7`. |
| 372 | - `TestShimKernelPassesArgvEnvExactly`: the test binary as tenant in a child mode (`CARLOS_SHIM_CHILD=dump`) writes `os.Args` and `os.Environ()` as NUL-joined bytes to stdout. The env carries `EMPTY=`, `EQ=a=b`, `BAD=\xff\xfe` and a 100 KiB value; argv carries `""`, `"two words"` and `"line\nbreak"`. Compare byte-for-byte against the same command started directly with `cmd.Start()`, and compare its `os.Getwd()` with `cmd.Dir`. Then a 200 KiB single value (above Linux's 32-page per-string limit): both the shim start and the direct start fail with `E2BIG`, the shim's as `*os.PathError{Op: "fork/exec"}`, not a placement error. |
| 373 | - `TestShimKernelInheritedStateMatchesADirectStart`: before starting, the test process runs `signal.Ignore(syscall.SIGHUP)` (restored with `signal.Reset` in Cleanup), and the command sets `SysProcAttr{Setpgid: true, Pdeathsig: syscall.SIGKILL}` plus two `ExtraFiles` pipes. The agent side also opens one file with `syscall.Open` without `O_CLOEXEC` (closed in Cleanup). The child mode (`CARLOS_SHIM_CHILD=state`) prints: its `/proc/self/fd` listing (minus the listing's own fd), `SigBlk`, `SigIgn`, `CapInh`, `CapPrm`, `CapEff`, `CapBnd`, `CapAmb` and `NoNewPrivs` from `/proc/self/status`, `PR_GET_PDEATHSIG`, `getpgid(0) == getpid()`, and `RLIMIT_NOFILE` soft and hard. Every line equals a direct start's, and the fd listing is exactly `0 1 2 3 4`. |
| 374 | - `TestShimKernelChdirFailureIsTheCommands`: `cmd.Dir` names a missing directory: the error is `*os.PathError{Op: "chdir"}` with `ENOENT`, not a placement error; reaped; marker absent. |
| 375 | - `TestShimExecFailureIsTheCommands`: tenant path `/nonexistent/app` (set `cmd.Path` directly so `LookPath` does not fail first): `*os.PathError{Op: "fork/exec"}` with `ENOENT`, `!isPlacementError(err)`, the PID is gone from `/proc`. |
| 376 | - `TestShimKernelPlacementFailureNeverRunsTenant`: target a cgroup the shim cannot write (`chmod 0444` its `cgroup.procs`, restore in Cleanup): placement error, marker absent, reaped. |
| 377 | - `TestShimKernelBeforeRunsAfterPlacement`: `before` reads the target's `cgroup.events` and records `populated 1` (the shim is in place when it is asked); then a second run whose `before` refuses: `refused == true`, before's error returned, marker absent, reaped, nothing registered. |
| 378 | - `TestShimKernelSignalBeforePlacement`: `shimTestHold = 1`; a `startCmd` seam wrapper (`cgroup.go:332`) starts the shim and sends it SIGTERM; a placement error ("exited before placement"), marker absent, reaped. Then the same with SIGKILL. |
| 379 | - `TestShimKernelSignalBeforeExec`: `shimTestHold = 2` (after the release byte, before `R`); SIGKILL it from `shimAfterRelease`: a placement error ("exited before exec"), marker absent, reaped. |
| 380 | - `TestShimKernelExecBeforeAcknowledgement`: `shimDeadline = 200ms` (restored in Cleanup) and a `shimAfterRelease` hook that waits until the tenant's marker exists and then sleeps 400 ms, past the deadline. The tenant ran, so the start must be reported started: nil error, `cmd.Wait()` succeeds. This pins the rule that an exec that happened is never reported as a pre-exec failure. |
| 381 | - `TestShimKernelDeadlineBeforePlacement`: `shimTestHold = 1`, `shimDeadline = 300ms`, no signal: a placement error naming the deadline, reaped, marker absent. |
| 382 | - `TestShimKernelSurvivesDeletedExecutable`: copy `os.Args[0]` to `t.TempDir()/copy.test`, run the copy as a child test (`-test.run '^TestShimKernelDeletedChild$'`, env `CARLOS_SHIM_KERNEL=1 CARLOS_SHIM_DELETED_CHILD=1 CARLOS_GEN_KERNEL=1`); the child deletes its own executable, then places `/bin/true` through `startViaShim` and must succeed. |
| 383 | - `TestShimKernelOverhead` (informational, never fails on time): 20 shim starts of `/bin/true` against 20 direct starts; `t.Logf` the medians. |
| 384 | |
| 385 | A reported kernel run must show `--- PASS` for every test named here and no `--- SKIP` lines; paste the `-test.v` output. |
| 386 | |
| 387 | - [ ] **Step 2: Run the tests to see them fail** |
| 388 | |
| 389 | Run: `cd <worktree> && GOCACHE=$TMPDIR/gocache go test ./internal/cgroup -run 'Shim' -count=1` |
| 390 | Expected: build failure (`encodeShimSpec`, `startViaShim`, `RunShimIfRequested` undefined). |
| 391 | |
| 392 | - [ ] **Step 3: Implement** |
| 393 | |
| 394 | `shim.go`: |
| 395 | |
| 396 | ```go |
| 397 | package cgroup |
| 398 | |
| 399 | import ( |
| 400 | "errors" |
| 401 | "os" |
| 402 | "sync/atomic" |
| 403 | ) |
| 404 | |
| 405 | // ShimVerb is the carlos binary's hidden placement verb (Paul's ruling, |
| 406 | // #272, 2026-10-13). carlos-edge.service's RestrictNamespaces=yes makes |
| 407 | // systemd's seccomp filter answer clone3 with ENOSYS, so no child can be |
| 408 | // created inside its cgroup. Instead the child starts as this binary, |
| 409 | // writes its own PID into the target cgroup through a directory fd the |
| 410 | // parent validated, checks where it landed, and only then execs the |
| 411 | // tenant: the tenant's code never runs outside its limits. |
| 412 | const ShimVerb = "__cgexec" |
| 413 | |
| 414 | // ProbeVerb prints the process's own cgroup and exits: the tenant the |
| 415 | // arm-time placement check (VerifyPlacement) execs through the shim. |
| 416 | const ProbeVerb = "__cgprobe" |
| 417 | |
| 418 | var shimHooked atomic.Bool |
| 419 | |
| 420 | // RunShimIfRequested must be the first statement of main(), and of TestMain |
| 421 | // in a test binary that places processes. It marks this process as able to |
| 422 | // serve the shim (shimAvailable), and when this process IS a shim or a |
| 423 | // probe it runs it and exits; it returns only when it is neither. |
| 424 | func RunShimIfRequested() { |
| 425 | shimHooked.Store(true) |
| 426 | if len(os.Args) < 2 { |
| 427 | return |
| 428 | } |
| 429 | switch os.Args[1] { |
| 430 | case ShimVerb: |
| 431 | os.Exit(runShim(os.Args[2:])) |
| 432 | case ProbeVerb: |
| 433 | os.Exit(runProbe()) |
| 434 | } |
| 435 | } |
| 436 | |
| 437 | // placementError is a start whose process could not be placed in its |
| 438 | // cgroup before exec. Nothing of the tenant ran. |
| 439 | type placementError struct{ err error } |
| 440 | |
| 441 | func (e *placementError) Error() string { return "placement before exec: " + e.err.Error() } |
| 442 | func (e *placementError) Unwrap() error { return e.err } |
| 443 | |
| 444 | func isPlacementError(err error) bool { |
| 445 | var p *placementError |
| 446 | return errors.As(err, &p) |
| 447 | } |
| 448 | |
| 449 | // ErrPlacementInFlight is a cleanup asked for members while a process is |
| 450 | // being placed in its cgroup (Task 2). |
| 451 | var ErrPlacementInFlight = errors.New("cgroup: a process is being placed in this cgroup") |
| 452 | ``` |
| 453 | |
| 454 | `shimAvailable()` is `shimHooked.Load()` in `shim_linux.go` and `false` in `shim_other.go`. `shim_other.go`'s `runShim`/`runProbe` return 127, and its `startViaShim` returns `false, &placementError{errors.New("placement before exec needs linux")}`. |
| 455 | |
| 456 | `shim_linux.go` implements the parent and shim protocol above. Factor the status decisions into `classifyPlacement` and `classifyExec` so `TestShimStatusClassification` covers them. Keep every comment to what the code does and why. |
| 457 | |
| 458 | `main.go`, first statement of `main()`: |
| 459 | |
| 460 | ```go |
| 461 | func main() { |
| 462 | // The placement shim (internal/cgroup, #272) must run before anything |
| 463 | // else touches the environment or the logger: a shim process carries |
| 464 | // none of the tenant's settings and exits from inside this call. |
| 465 | cgroup.RunShimIfRequested() |
| 466 | rootaws.ScrubEnvironment() |
| 467 | ``` |
| 468 | |
| 469 | (Import `github.com/carlosframework/platform/internal/cgroup` if `main.go` does not already.) |
| 470 | |
| 471 | - [ ] **Step 4: Run the tests** |
| 472 | |
| 473 | Run (sandboxed): `GOCACHE=$TMPDIR/gocache go test ./internal/cgroup -count=1` |
| 474 | Expected: PASS; the kernel tests SKIP. |
| 475 | |
| 476 | Run (unsandboxed, kernel): |
| 477 | ```bash |
| 478 | mkdir -p /tmp/k/zddr3 && GOCACHE=$TMPDIR/gocache go test -c -o /tmp/k/zddr3/cgroup.test ./internal/cgroup && \ |
| 479 | systemd-run --user --wait --pipe --collect -p Delegate=yes -E TMPDIR=/tmp/k/zddr3 -E CARLOS_SHIM_KERNEL=1 -E CARLOS_GEN_KERNEL=1 \ |
| 480 | /tmp/k/zddr3/cgroup.test -test.run 'ShimKernel|ShimExecFailure' -test.v -test.count=1 |
| 481 | ``` |
| 482 | Expected: every `ShimKernel` test PASS, with the overhead medians in the log. Paste the real output into the report. Then `rm -rf /tmp/k/zddr3`. |
| 483 | |
| 484 | Also: `GOOS=darwin GOCACHE=$TMPDIR/gocache go build ./... && GOOS=windows GOCACHE=$TMPDIR/gocache go build ./...` and `go vet ./internal/cgroup .` |
| 485 | |
| 486 | - [ ] **Step 5: Commit** |
| 487 | |
| 488 | ```bash |
| 489 | git add internal/cgroup/shim*.go internal/cgroup/inflight*.go internal/cgroup/main_test.go internal/cgroup/cgroup.go main.go |
| 490 | git -c user.name=claude -c user.email=claude~paul@keymail.dev commit -m "cgroup: the placement exec-shim (#272)" |
| 491 | ``` |
| 492 | |
| 493 | --- |
| 494 | |
| 495 | ### Task 2: Every Manager start places through the shim (opus) |
| 496 | |
| 497 | **Files:** |
| 498 | - Modify: `internal/cgroup/cgroup.go:170-184` (Manager: drop `fallback`), `cgroup.go:330-549` (`startCmd`, `StartGated` doc, `FakePlacementForTest`, `startIn`), `internal/cgroup/cgroup_linux.go` (drop what only the clone3 start used; keep `setInto` and `shouldLatch`, which `strict_linux.go:344-350` uses), `internal/cgroup/cgroup_other.go`, `internal/cgroup/gen.go:171-235` (`ensureParent`), `internal/cgroup/cleanup_linux.go:54-353` (Cleanup), the tests that pinned the clone3 start and its latch (`internal/cgroup/cgroup_linux_test.go`, `gen_test.go`, `cgroup_test.go`), and comments naming the clone3 start in `internal/activator/{swap.go:205-207, backend_exec.go:2132, backend_exec.go:2920, genbackend.go:400, genbackend.go:495, workerbuild.go:102}` and `internal/cgroup/recreate_linux.go:19,118-122`. |
| 499 | |
| 500 | **Interfaces:** |
| 501 | - Consumes (Task 1): `startViaShim`, `shimAvailable`, `isPlacementError`, `ErrPlacementInFlight`, and the registry (`beginPlacement`, `placementsUnder`, `placementsAt`, `waitPlacementsUnder`). |
| 502 | - Produces: |
| 503 | - `var placeStart = func(m *Manager, dir string, cmd *exec.Cmd, before func() error) (bool, error) { return m.startViaShim(dir, cmd, before) }` — the seam `FakePlacementForTest` swaps. |
| 504 | - The start functions' signatures do not change. |
| 505 | |
| 506 | **Behaviour.** |
| 507 | |
| 508 | `startIn(host, dir, limit, required, build, before)`: |
| 509 | - Required (generation): if `!shimAvailable()`, log once and return `ErrLimitNotApplied` before the directory is made or `build()` runs (as the latch refusal does today, `cgroup.go:453-458`). Make the directory and apply the limit as today. `refused, err := placeStart(m, dir, build(), before)` (the shim registers the placement itself). Refused: remove the directory if empty, return `before`'s error. Success: `biasOOM`, return the cmd. A placement error: remove the directory if empty, log `cgroup: <host>: placement before exec failed (<err>) — generation start refused`, return `fmt.Errorf("%w: %s: %w", ErrLimitNotApplied, host, err)`. Any other error is the command's: remove the directory if empty, return it unwrapped. |
| 510 | - Flat: make the directory (failure: start plain, as today `cgroup.go:465-466`) and apply the limit best-effort as today. If `!shimAvailable()`, go straight to `startAndMoveIn` (today's fallback). Otherwise place through the shim. Success: `biasOOM`, return. Refused: as today. A placement error: log `cgroup: <host>: placement before exec failed (<err>) — using the PID-move fallback for this spawn` and `startAndMoveIn` with a fresh `build()` behind `before` (Ruling 2). Any other error is the command's: remove the directory if empty and return it, with no retry. |
| 511 | - `Manager.fallback`, the read-once comment and the unreachable required-path guard at `cgroup.go:538-546` go; keep a single explicit statement that the required path never reaches `startAndMoveIn`. |
| 512 | |
| 513 | `FakePlacementForTest`: keep its name, signature, panic outside tests and no-`t.Parallel` note; it now swaps `placeStart` for one that runs `startGated(cmd, before)` with no shim (as if the kernel had placed it), AND sets a test-scope override that makes `shimAvailable()` answer true, restoring both in Cleanup. The override is safe because the fake never runs a shim, so it cannot make a test binary without the hook re-run itself (Astra plan 7; the activator's `TestMain` at `internal/activator/activator_test.go:5894` has no hook, and `TestSpawnInCgroupAtPutsAGenerationUnderItsParent`, `internal/activator/containment_test.go:113`, must keep passing through the fake). Update its doc. |
| 514 | |
| 515 | `startAndMoveIn` (the flat fallback, `cgroup.go:625`) registers its target too, from after `before` agrees until the PID write has returned: its child runs in the agent's cgroup for that window, and a proof must not miss it either. Give `startGated` an optional register step so both paths take the registry at the same point. |
| 516 | |
| 517 | Cleanup (`cleanup_linux.go`): `openCleanupAt` records the manager and the directory path it pinned (`filepath.Join(m.root, ExecName(host))`, plus `/child` for a child cleanup) in `cleanupState`. The registry goes into the shared primitives, not only the public methods, because `KillAndEmpty` calls them directly (`cleanup_linux.go:334`, `:338`): `populationLocked` answers `true, nil` while `m.placementsUnder(path) > 0`, before reading `cgroup.events`, and `membersLocked` answers `nil, ErrPlacementInFlight` while one is in flight. `KillAndEmpty` first calls `m.waitPlacementsUnder(ctx, path)` (an error there is unproven), and because every emptiness check inside its loop now goes through the registry-aware `populationLocked`, a placement that registers after the wait keeps the proof from finishing until the placed process has been killed with the rest or the ctx ends (unproven). Write the doc comment on `KillAndEmpty` saying so, and that callers still exclude new starts during a proof the way they do today (the lease and the `before` hook); the registry only closes the window a shim spends outside its cgroup. |
| 518 | |
| 519 | `ensureParent` (`gen.go:198`): after `parentProcs`, add `m.placementsAt(dir)` to `n`, so a flat start being placed into the host directory itself counts as a direct process and the generation start is refused with `ErrLegacyParentPopulated`. The `parentPopulated(dir)` branch at `gen.go:213` also counts `m.placementsUnder(dir) > 0` as populated, so a sibling generation being placed is never taken for a cold parent whose limits may be rewritten (`gen.go:219-229`). |
| 520 | |
| 521 | Check every caller of `Cleanup.Members` (`grep -rn "\.Members()" internal`) treats an error as unproven (keeps the lease, records nothing as empty). If one does not, fix that caller in this task and name it in the report. |
| 522 | |
| 523 | - [ ] **Step 1: Write the failing tests** |
| 524 | |
| 525 | `TestCleanupCountsPlacementInFlight` (in `cleanup_linux_test.go`, over the existing fake-root helper at `cleanup_linux_test.go:28`, which builds `&Manager{root: root}` — give it `logf: t.Logf`): with a placement begun on `<host dir>/g1`, a host Cleanup's `Populated()` is `true` even though the fake `cgroup.events` says `populated 0`; `Members()` is `ErrPlacementInFlight`; `KillAndEmpty` with a 50 ms ctx returns an error and writes nothing to `cgroup.kill`; after `done()`, `Populated()` reads the file again. |
| 526 | |
| 527 | `TestKillAndEmptySeesAPlacementRegisteredMidProof`: no placement when `KillAndEmpty` starts; through the kill-write seam the cleanup tests already use (or one added for this test), register a placement on `<host dir>/g1` right after the kill is written. With a 100 ms ctx, `KillAndEmpty` must return an error (unproven), never nil, although the fake `cgroup.events` says `populated 0`. |
| 528 | |
| 529 | `TestEnsureParentCountsAPlacementBelowTheParent` (in `gen_test.go`): a placement in flight on `<host dir>/g1` while `StartChildGated` starts `g2` on a parent whose fake `cgroup.events` says `populated 0`: the parent's limit files are not rewritten (the cold-parent branch is not taken). |
| 530 | |
| 531 | `TestEnsureParentCountsAPlacementAtTheParent` (in `gen_test.go`): a placement begun on the host directory itself makes `StartChildGated` answer `ErrLegacyParentPopulated` and build nothing. |
| 532 | |
| 533 | `TestGenerationMissingBinaryIsNotABoxCondition` (in `gen_test.go`, with a `placeStart` stub returning `&os.PathError{Op: "fork/exec", Path: "/x", Err: syscall.ENOENT}`): `StartChildGated` returns that error, `errors.Is(err, ErrLimitNotApplied)` is false, and the child directory is gone. |
| 534 | |
| 535 | `TestGenerationPlacementFailureIsLimitNotApplied` (stub returns `&placementError{...}`): `errors.Is(err, ErrLimitNotApplied)`, child directory gone, `build` called once. |
| 536 | |
| 537 | `TestGenerationRefusedWithoutTheShim` (flip the hook flag off): `ErrLimitNotApplied`, `build` never called, no child directory made. |
| 538 | |
| 539 | `TestFlatPlacementFailureFallsBackToPIDMove` (stub returns a placement error): `StartGated` succeeds through `startAndMoveIn` (the fake `cgroup.procs` holds the child's PID), `build` called twice, `before` asked twice. |
| 540 | |
| 541 | `TestFlatCommandFailureIsNotRetried` (stub returns a `fork/exec` ENOENT): returned as is, `build` called once. |
| 542 | |
| 543 | Replace the clone3 latch tests in `cgroup_linux_test.go` (those asserting `UseCgroupFD`, `shouldLatch` latching, or `fallback`) with the tests above; keep any test of `shouldLatch` that the strict path still relies on. |
| 544 | |
| 545 | - [ ] **Step 2: Run them to see them fail** |
| 546 | |
| 547 | Run: `GOCACHE=$TMPDIR/gocache go test ./internal/cgroup -count=1` |
| 548 | Expected: FAIL / build errors on the new names. |
| 549 | |
| 550 | - [ ] **Step 3: Implement** the behaviour above. |
| 551 | |
| 552 | - [ ] **Step 4: Run the tests** |
| 553 | |
| 554 | ```bash |
| 555 | GOCACHE=$TMPDIR/gocache go test ./internal/cgroup -count=1 |
| 556 | GOCACHE=$TMPDIR/gocache go vet ./internal/cgroup ./internal/activator |
| 557 | GOOS=darwin GOCACHE=$TMPDIR/gocache go build ./... && GOOS=windows GOCACHE=$TMPDIR/gocache go build ./... |
| 558 | ``` |
| 559 | Then unsandboxed, the activator package and the other cgroup users (sockets need `/tmp/k`): |
| 560 | ```bash |
| 561 | mkdir -p /tmp/k && TMPDIR=/tmp/k GOCACHE=$HOME/.cache/go-build nice -n 19 go test ./internal/activator ./internal/host ./internal/stopevidence ./internal/instancectl -count=1 |
| 562 | ``` |
| 563 | Then the generation kernel tests in a delegated user unit, which now place through the shim: |
| 564 | ```bash |
| 565 | mkdir -p /tmp/k/zddr3 && GOCACHE=$TMPDIR/gocache go test -c -o /tmp/k/zddr3/cgroup.test ./internal/cgroup && \ |
| 566 | systemd-run --user --wait --pipe --collect -p Delegate=yes -E TMPDIR=/tmp/k/zddr3 -E CARLOS_GEN_KERNEL=1 -E CARLOS_SHIM_KERNEL=1 \ |
| 567 | /tmp/k/zddr3/cgroup.test -test.run 'GenKernel|GenOOMKernel|ShimKernel|GenerationStartRefusedBesideADirectParentMember|ClaimContainment' -test.v -test.count=1; rm -rf /tmp/k/zddr3 |
| 568 | ``` |
| 569 | Expected: PASS (paste the real output). A test that asserted clone3-specific kernel behaviour (the `cgroup.kill` respawn kill was clone3's) is updated to assert the shim's behaviour on the same scenario, and the report says what changed and why. |
| 570 | |
| 571 | - [ ] **Step 5: Commit** |
| 572 | |
| 573 | ```bash |
| 574 | git -c user.name=claude -c user.email=claude~paul@keymail.dev commit -am "cgroup: every Manager start places through the shim; placements in flight count as members (#272)" |
| 575 | ``` |
| 576 | (`git add` the new files first.) |
| 577 | |
| 578 | --- |
| 579 | |
| 580 | ### Task 3: Arming verifies placement (opus) |
| 581 | |
| 582 | **Files:** |
| 583 | - Create: `internal/cgroup/verify_linux.go`, `internal/cgroup/verify_other.go`, `internal/cgroup/verify_linux_test.go` |
| 584 | - Modify: `internal/host/rollout.go:22-44` (the code), `rollout.go:150-166` (`swapArming`), `internal/host/rollout_test.go`, `internal/rolloutwords/words.go:84-112` (+ its test), `internal/status/snapshot.go:300-304` (comment) |
| 585 | |
| 586 | **Interfaces:** |
| 587 | - Consumes: Task 1's `startViaShim` (which registers the placement in flight), `ProbeVerb`, `shimAvailable`, `shimExe`. |
| 588 | - Produces: |
| 589 | - `func (m *Manager) VerifyPlacement(ctx context.Context) error` (linux); the `!linux` stub returns an error. |
| 590 | - `const placementCheckName = "carlos-placement-check"` |
| 591 | - In `internal/host`: `rolloutPlacementFailed = "cgroup-placement-failed"` and `var placementCheck = func(m *cgroup.Manager) error` (a 15 s ctx around `VerifyPlacement`), a seam host tests replace. |
| 592 | |
| 593 | **Behaviour.** `VerifyPlacement`: |
| 594 | 1. Nil manager: `ErrNoDelegation`. `!shimAvailable()`: an error saying the shim is not available. |
| 595 | 2. `dir := filepath.Join(m.root, placementCheckName)`. `os.Remove(dir)` (a leftover from an earlier run is empty: the unit's restart killed its members); then `os.Mkdir(dir, 0o755)`; `defer` a removal that waits for the probe to be reaped first. |
| 596 | 3. `m.applyLimit("placement check", dir, TierDefault, true)`; an error is returned. |
| 597 | 4. `cmd := exec.Command(shimExe, ProbeVerb)` with `Env = []string{}` and `Stdout` a buffer. `m.startViaShim(dir, cmd, nil)`. Any error is returned (a placement error or not: either means the check failed). |
| 598 | 5. Wait for the probe with `ctx` (kill and reap on `ctx.Done()`). It must exit 0 and its stdout must be exactly `expected + "\n"` (`expected` is the string `startViaShim` computed: `dir` without `/sys/fs/cgroup`). |
| 599 | 6. `memory.max` in `dir` must read back `TierDefault.MemoryMax`. |
| 600 | 7. Remove `dir`; a removal failure is logged, not returned. |
| 601 | |
| 602 | `swapArming`: after `swapDelegation` passes (`rollout.go:160`), call `placementCheck(eb.Cgroups)`. On error: `log.Printf("agent: REFUSING %s=overlap (%s): a test placement into a scratch cgroup failed (%v), so generations could not be started inside their limits; this box stays on today's stop-then-wake paths", rollout.SwapEnv, rolloutPlacementFailed, err)`, `declineRollout(rolloutPlacementFailed)`, return false. A box that declines this way honours failure records exactly as the other declined codes do (`honourDeclinedFailures`); check that path keys on "declined", not on a list of codes, and if it keys on a list, add the code there. |
| 603 | |
| 604 | `rolloutwords`: `"cgroup-placement-failed": unavailableDeclined` in `unavailablePhrases`, and the code in `UnavailableDeclines`'s case list. `status/snapshot.go`: name the code in the comment that lists the declined codes. Grep for every other list of declined codes (`grep -rn '"host-claims-on"' --include=*.go .`) and add the new code to each, with a test where one exists. |
| 605 | |
| 606 | - [ ] **Step 1: Write the failing tests** |
| 607 | |
| 608 | `internal/host/rollout_test.go`: |
| 609 | |
| 610 | ```go |
| 611 | func TestSwapArmingDeclinesWhenPlacementFails(t *testing.T) { |
| 612 | // build eb as TestSwapArming's armed case does (rollout_test.go:526) |
| 613 | orig := placementCheck |
| 614 | placementCheck = func(*cgroup.Manager) error { return errors.New("probe exited before exec") } |
| 615 | t.Cleanup(func() { placementCheck = orig }) |
| 616 | if swapArming(overlapEnv(pub), eb) { |
| 617 | t.Fatal("armed with a failed placement check") |
| 618 | } |
| 619 | if rolloutUnarmed() != rolloutPlacementFailed { |
| 620 | t.Fatalf("declined as %q", rolloutUnarmed()) |
| 621 | } |
| 622 | } |
| 623 | ``` |
| 624 | |
| 625 | and make the existing armed-case tests (`rollout_test.go:526`, `:568` and any other test that expects `swapArming` true) stub `placementCheck` to nil-returning, in a helper they share. A flag-off `swapArming` must not call `placementCheck` (assert with a stub that fails the test). |
| 626 | |
| 627 | `internal/rolloutwords`: `Unavailable("cgroup-placement-failed")` equals the declined phrase and `UnavailableDeclines` is true for it. |
| 628 | |
| 629 | `internal/cgroup/verify_linux_test.go`: ungated: a manager over a temp root answers an error (the temp directory is no cgroup) and leaves no `carlos-placement-check` behind. Gated (`CARLOS_SHIM_KERNEL=1`, `genKernelManager`): `VerifyPlacement` returns nil and the directory is gone afterwards; with `shimExe` pointed at `/bin/true` (restored in Cleanup) it returns an error and the directory is gone. |
| 630 | |
| 631 | - [ ] **Step 2: Run to see them fail.** `GOCACHE=$TMPDIR/gocache go test ./internal/host ./internal/rolloutwords ./internal/cgroup -run 'Placement|Unavailable|SwapArming' -count=1` — build errors. |
| 632 | |
| 633 | - [ ] **Step 3: Implement.** |
| 634 | |
| 635 | - [ ] **Step 4: Run.** The same packages sandboxed (host tests may need unsandboxed `TMPDIR=/tmp/k` for sockets: use that if a socket error appears), plus the gated verify tests in a delegated user unit as in Task 1. Paste real output. |
| 636 | |
| 637 | - [ ] **Step 5: Commit** — `"host: arming verifies a real shim placement; decline cgroup-placement-failed (#272)"`. |
| 638 | |
| 639 | --- |
| 640 | |
| 641 | ### Task 4: The shim under the edge unit's hardening (opus) |
| 642 | |
| 643 | **Files:** |
| 644 | - Create: `internal/cgroup/edge_hardening_kernel_linux_test.go` |
| 645 | |
| 646 | **Interfaces:** |
| 647 | - Consumes: Task 1 (`startViaShim`), Task 2 (`StartChildGated` on the shim), Task 3 (`VerifyPlacement`), `genKernelManager` (`gen_kernel_linux_test.go:27`). |
| 648 | - Produces: `TestEdgeHardeningPlacement` (the driver) and `TestEdgeHardeningInner` (runs inside the unit). |
| 649 | |
| 650 | **Behaviour.** The driver, gated by `CARLOS_EDGE_HARDENING_KERNEL=1` and run outside any unit: |
| 651 | 1. Read `../host/units/carlos-edge.service` (go test runs in the package directory). Collect every `Key=value` line in `[Service]`. |
| 652 | 2. Sort each key into one of two explicit tables in the test: **applied** (passed as `-p Key=value` to `systemd-run --user`) and **excluded** (with a one-line reason each). A `[Service]` key in neither table fails the test, so a new hardening line forces a decision here. |
| 653 | 3. The applied table must contain at least: `RestrictNamespaces`, `SystemCallFilter`, `SystemCallErrorNumber`, `SystemCallArchitectures`, `NoNewPrivileges`, `MemoryDenyWriteExecute`, `LockPersonality`, `RestrictRealtime`, `RestrictAddressFamilies`, `ProtectProc`, `ProtectClock`, `ProtectHostname`, `ProtectKernelTunables`, `ProtectKernelModules`, `ProtectKernelLogs`, `PrivateDevices`, `ProtectSystem`, `TasksMax`. The test asserts each is present in the unit file and applied. If one of these cannot be applied in a user unit on DOCS-T1, move it to excluded with the real `systemd-run` error as its reason, and say so in the report: Task 7 carries it into the dev re-check. |
| 654 | 4. Expected exclusions, each with its reason: `ExecStart`, `ExecReload`, `Restart`, `RestartSec`, `EnvironmentFile` (process control, not sandbox); `User`, `Group`, `AmbientCapabilities`, `CapabilityBoundingSet` (a user manager cannot change identity or capabilities); `ReadWritePaths` (names box paths absent here); `Slice`, `OOMPolicy`, `Delegate` (the test passes its own `Delegate=yes`); `ProtectHome`, `PrivateTmp` only if the test binary cannot be made visible inside them (prefer keeping them and making the binary visible with `-p BindReadOnlyPaths=<binary>`; say which in the report). |
| 655 | 5. **Unrestricted control first** (Astra plan 6): run the inner test in a unit with only `-p Delegate=yes` and `-E CARLOS_EDGE_HARDENING_INNER=control`. There clone3 must WORK: the raw `clone3` probe answers `EINVAL` (not `ENOSYS`), and a plain `exec.Cmd` with `SysProcAttr{UseCgroupFD: true, CgroupFD: <a scratch cgroup fd>}` starts and lands in that cgroup. If the control fails, the hardened run below proves nothing, so the driver fails. |
| 656 | 6. **Hardened run:** `systemd-run --user --wait --pipe --collect -p Delegate=yes <applied...> -E CARLOS_EDGE_HARDENING_INNER=hardened -E CARLOS_SHIM_KERNEL=1 -E CARLOS_GEN_KERNEL=1 <binary> -test.run '^TestEdgeHardeningInner$' -test.v -test.count=1`; fail unless it passes. Stream both runs' output to the test log. |
| 657 | |
| 658 | The inner test, gated by `CARLOS_EDGE_HARDENING_INNER` (`control` or `hardened`). Once the variable is set, nothing in it skips: a delegation failure, or any precondition it cannot establish, is `t.Fatal`. In `hardened` mode: |
| 659 | 0. **What the unit enforces, observed from inside:** `/proc/self/status` shows `Seccomp: 2` and `NoNewPrivs: 1`; a write under `/usr` fails with `EROFS` (`ProtectSystem=strict`); `/proc/1/cmdline` is unreadable or absent (`ProtectProc=invisible`); `socket(AF_NETLINK, ...)` fails (`RestrictAddressFamilies`). Print a table of every applied property marked `enforced` (observed here) or `untested` (accepted by systemd but not observed). A property in `untested` is listed in the report so Task 7 carries it to the dev re-check on the real unit. |
| 660 | 1. **The filter is on:** the raw `clone3` probe answers `ENOSYS`, and the plain `UseCgroupFD` start fails with `ENOSYS`, where the control run showed both working. |
| 661 | 2. **Placement succeeds:** `StartChildGated("hardening.example", GenChild(1), lim, lim, build, nil)` with `lim := Limit{MemoryMax: 64 << 20}` and a tenant that prints its cgroup then waits on stdin: the cgroup is `.../carlos-exec@hardening.example/g1`, and the PID the tenant prints is `cmd.Process.Pid`. |
| 662 | 3. **The limit applies:** the existing `hog` child mode (`gen_kernel_linux_test.go:65`) started the same way into `g2` is OOM-killed: `memory.events` in `g2` shows `oom_kill` of at least 1, the process dies by SIGKILL, and `g1`'s tenant is still alive. |
| 663 | 4. **Arming would arm:** `m.VerifyPlacement(ctx)` returns nil. |
| 664 | |
| 665 | - [ ] **Step 1: Write the test** (both functions, the two tables). |
| 666 | - [ ] **Step 2: Run it against a branch where it must fail**, to prove it can: temporarily make `placeStart` use the old clone3 start (a local, uncommitted edit, `setInto` + `cmd.Start()`); the inner step 2 must fail with `ENOSYS`/`ErrLimitNotApplied`. Paste that output, then revert the edit (`git diff` must show only the new test file). |
| 667 | - [ ] **Step 3: Run it for real**: |
| 668 | ```bash |
| 669 | mkdir -p /tmp/k/zddr3 && GOCACHE=$TMPDIR/gocache go test -c -o /tmp/k/zddr3/cgroup.test ./internal/cgroup && \ |
| 670 | cd internal/cgroup && CARLOS_EDGE_HARDENING_KERNEL=1 TMPDIR=/tmp/k/zddr3 /tmp/k/zddr3/cgroup.test -test.run '^TestEdgeHardeningPlacement$' -test.v -test.count=1; rm -rf /tmp/k/zddr3 |
| 671 | ``` |
| 672 | (unsandboxed). Expected: PASS, with the applied and excluded tables printed. Paste the real output. |
| 673 | - [ ] **Step 4: Commit** — `"cgroup: kernel test, shim placement under carlos-edge's sandbox lines (#272)"`. |
| 674 | |
| 675 | --- |
| 676 | |
| 677 | ### Task 5: The probe app accepts the platform's argv (sonnet) |
| 678 | |
| 679 | **Files:** |
| 680 | - Create: `cmd/zddprobe/main.go`, `cmd/zddprobe/main_test.go` |
| 681 | - Modify: `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md:841-895` (section "1. Build a probe app") |
| 682 | |
| 683 | **Interfaces:** none shared. |
| 684 | |
| 685 | **Behaviour.** The platform runs an exec-backed instance as `<binary> --socket <sock> --db <db>` (`internal/activator/backend_exec.go:2147`, `:2335`, `internal/activator/claim_snapshot.go:221`), and `carlos deploy`'s vet boots it as `<binary> -socket <sock>` (`internal/vet/vet.go:135`). The probe accepts both: `--socket` (required) and `--db` (accepted and ignored: the probe keeps no data). Everything else is the roll notes' program unchanged: serve HTTP on the unix socket, every request sleeps 50 ms then answers `ok <version>\n`, and SIGTERM or SIGINT drains with a 15 s shutdown. `version` is set by `-ldflags "-X main.version=..."`. Put the server in `func run(args []string, stop <-chan os.Signal) error` so `main` is `if err := run(os.Args[1:], sigs); err != nil { fmt.Fprintln(os.Stderr, err); os.Exit(1) }`. |
| 686 | |
| 687 | - [ ] **Step 1: Write the failing test** — `main_test.go` launches the test binary itself as the probe, the way the platform launches it: |
| 688 | |
| 689 | ```go |
| 690 | func TestMain(m *testing.M) { |
| 691 | if os.Getenv("ZDDPROBE_AS_MAIN") == "1" { |
| 692 | main() |
| 693 | os.Exit(0) |
| 694 | } |
| 695 | os.Exit(m.Run()) |
| 696 | } |
| 697 | |
| 698 | // launch starts this test binary as the probe with argv, as the platform |
| 699 | // starts an instance: argv only, no shell. |
| 700 | func launch(t *testing.T, argv ...string) *exec.Cmd { |
| 701 | cmd := exec.Command(os.Args[0], argv...) |
| 702 | cmd.Env = append(os.Environ(), "ZDDPROBE_AS_MAIN=1") |
| 703 | cmd.Stderr = os.Stderr |
| 704 | if err := cmd.Start(); err != nil { |
| 705 | t.Fatal(err) |
| 706 | } |
| 707 | t.Cleanup(func() { _ = cmd.Process.Kill(); _, _ = cmd.Process.Wait() }) |
| 708 | return cmd |
| 709 | } |
| 710 | |
| 711 | func TestProbeServesWithThePlatformsRunArgv(t *testing.T) { |
| 712 | dir := t.TempDir() |
| 713 | sock, db := filepath.Join(dir, "app.sock"), filepath.Join(dir, "app.db") |
| 714 | cmd := launch(t, "--socket", sock, "--db", db) // backend_exec.go:2335 |
| 715 | body := getOverUnix(t, sock, "/healthz") // retries until the socket answers, 10 s cap |
| 716 | if !strings.HasPrefix(body, "ok ") { |
| 717 | t.Fatalf("body %q", body) |
| 718 | } |
| 719 | // SIGTERM drains: an in-flight request still completes, then exit 0. |
| 720 | ... |
| 721 | if err := cmd.Wait(); err != nil { |
| 722 | t.Fatalf("exit after SIGTERM: %v", err) |
| 723 | } |
| 724 | } |
| 725 | |
| 726 | func TestProbeServesWithVetsArgv(t *testing.T) { ... launch(t, "-socket", sock) (vet.go:135) answers ok ... } |
| 727 | ``` |
| 728 | |
| 729 | `getOverUnix` is a small helper with an `http.Client` whose transport dials the unix socket. The drain check must know the request is inside the handler before it signals (Astra plan 8): the probe writes one line `serving` to stdout at handler entry when `ZDDPROBE_TRACE=1` is set (nothing otherwise). The test sets it, starts a request, reads `serving` from the probe's stdout pipe, then sends SIGTERM, and asserts the request answers 200 and `cmd.Wait()` returns nil. |
| 730 | |
| 731 | - [ ] **Step 2: Run to see it fail:** `TMPDIR=/tmp/k GOCACHE=$TMPDIR/gocache go test ./cmd/zddprobe -count=1` (socket paths: if the sandbox refuses the socket, rerun unsandboxed with `TMPDIR=/tmp/k`). Expected: build failure. |
| 732 | - [ ] **Step 3: Implement `main.go`.** Then replace the roll notes' heredoc with: build from a platform checkout at the release commit, `for v in zdd-1 zdd-2 zdd-3 zdd-4; do CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -ldflags "-X main.version=$v" -o "$TMPDIR/zdd-probe/probe-$v" ./cmd/zddprobe; done`, and a sentence that the probe takes the platform's `--socket` and `--db` (the round-2 dev check found the old one exited at once on `--db`). Keep the rest of "The probe" as it is, and fix its `cd`/path lines to match. |
| 733 | - [ ] **Step 4: Run** the test (paste output), `go vet ./cmd/zddprobe`, gofmt. |
| 734 | - [ ] **Step 5: Commit** — `"cmd/zddprobe: the roll notes' probe, taking the platform's --socket and --db (#272)"`. |
| 735 | |
| 736 | --- |
| 737 | |
| 738 | ### Task 6: TestStrictKernel's managers, and reset-failed after a generation stop (sonnet) |
| 739 | |
| 740 | **Files:** |
| 741 | - Modify: `internal/cgroup/strict_kernel_linux_test.go:160`, `:190` |
| 742 | - Modify: `internal/instancectl/gen.go:185-238` (`abandonGen`, `stopGen`) and its test file (find it with `grep -rln "stopGen\|StopGen" internal/instancectl/*_test.go`) |
| 743 | |
| 744 | **Behaviour.** |
| 745 | |
| 746 | (a) `TestStrictKernel` full mode builds second managers with no `logf` (`&Manager{root: m.root}` at `:160` and `:190`); production always sets one, and `freshenOne` (`recreate_linux.go:156`) logs through it, so the test panics. Give both `logf: t.Logf`. No production change. |
| 747 | |
| 748 | (b) After a generation unit is successfully stopped or killed, and before its files go, reset its failed state. There are two such paths: `stopGen` (`gen.go:224`) and `abandonGen`'s successful `killGen` followed by `removeGenFiles` (`gen.go:189-197`, Astra plan 9). Share one helper, `s.resetGenFailed(ctx, unit)`, which runs `s.Systemctl(ctx, "reset-failed", unit)`. An unproven stop or kill is unchanged and does not reset. An answer containing `not loaded` is ignored; any other error is logged with `s.logf("instance-control: reset-failed %s: %v", unit, err)` and the stop still answers 204 (Ruling 10). `reset-failed` is already one of the control service's verbs (`server.go:641`). |
| 749 | |
| 750 | - [ ] **Step 1: Write the failing test** for (b): with the recording fake `Systemctl` the instancectl tests already use, a `/stop?gen=N` (and `&kill=1`) records `reset-failed <unit>` after the stop or kill; with `reset-failed` answering an error, the response is still 204 and a log line was written; a stop that fails records no `reset-failed`. An abandoned generation start whose kill succeeds records `reset-failed <unit>` too; one whose kill fails does not. |
| 751 | - [ ] **Step 2: Run to see it fail:** `GOCACHE=$TMPDIR/gocache go test ./internal/instancectl -run StopGen -count=1` (unsandboxed with `TMPDIR=/tmp/k` if sockets are refused). |
| 752 | - [ ] **Step 3: Implement (a) and (b).** |
| 753 | - [ ] **Step 4: Run** the instancectl package, then TestStrictKernel's full mode for real, in a delegated user unit with no RestrictNamespaces (the strict path keeps clone3, Ruling 3): |
| 754 | ```bash |
| 755 | mkdir -p /tmp/k/zddr3 && GOCACHE=$TMPDIR/gocache go test -c -o /tmp/k/zddr3/cgroup.test ./internal/cgroup && mkdir -p /tmp/k/zddr3/strictproof && \ |
| 756 | systemd-run --user --wait --pipe --collect -p Delegate=yes -E TMPDIR=/tmp/k/zddr3 -E CARLOS_STRICT_KERNEL=full -E CARLOS_STRICT_PROOF_DIR=/tmp/k/zddr3/strictproof \ |
| 757 | /tmp/k/zddr3/cgroup.test -test.run '^TestStrictKernel$' -test.v -test.count=1; rm -rf /tmp/k/zddr3 |
| 758 | ``` |
| 759 | Expected: all five subtests PASS (round 2 saw this pass with the same fix in scratch). Paste real output. If the mode needs other variables, read `strict_kernel_linux_test.go:30-60` and set them; say which in the report. |
| 760 | - [ ] **Step 5: Commit** — `"instancectl: reset-failed a stopped generation unit; strict kernel test managers log (#272)"`. |
| 761 | |
| 762 | --- |
| 763 | |
| 764 | ### Task 7: Roll notes (sonnet) |
| 765 | |
| 766 | **Files:** |
| 767 | - Modify: `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` |
| 768 | |
| 769 | **Behaviour.** Add a section `### Roll notes from zdd-flag-readiness-3` after `### Roll notes from zdd-flag-readiness` (`:336`), in the file's plain style (full stops, no em dashes), covering: |
| 770 | |
| 771 | 1. The four round-2 blockers and where each is closed (Tasks 1 to 6 by commit; the controller passes the commit list). |
| 772 | 2. The shim, in two sentences, and Paul's ruling that the edge unit's hardening stays. |
| 773 | 3. **Behaviour change on roll, flag off:** every exec child now starts through the shim. What the agent log shows on a healthy box (no `placement before exec failed` lines, no PID-move lines), and what a failing shim looks like (`placement before exec failed (...) — using the PID-move fallback for this spawn`): the box still serves, as today. |
| 774 | 4. The new decline code `cgroup-placement-failed`, its log line, and that a console older than the box shows the unknown-code phrase until consoles roll. |
| 775 | 5. Host claims: the strict path keeps clone3, so under the shipped unit a claims-on box's strict launches answer `ErrStrictUnsupported`; that is now a precondition of claims-on swaps. |
| 776 | 6. **Dev re-check for Paul** (replacing round 2's blocked steps): roll; flag-off wakes answer 200 with no fallback lines; set `CARLOS_SWAP=overlap` and restart; the log shows `overlap swaps armed` and no `REFUSING`; the probe (built from `cmd/zddprobe`) under load through zdd-1 → zdd-3 with 0 failed; the hold control; a generation unit stopped by the control service is not in `systemctl --failed`; and any edge-unit line Task 4 had to exclude or could only mark `untested` (the controller passes the list), checked on the real unit. |
| 777 | 7. The DOCS-T1 kernel-test recipe (`systemd-run --user ... -p Delegate=yes`) and the Task 4 command. |
| 778 | |
| 779 | Update the "Still blocks flag-on" lists that name the four round-2 blockers so they point to this section. |
| 780 | |
| 781 | - [ ] **Step 1: Write the section and the list edits.** |
| 782 | - [ ] **Step 2: Check names against the code:** every identifier, log line and flag the section quotes is grepped in the tree; paste the grep output in the report. |
| 783 | - [ ] **Step 3: Commit** — `"roll notes: zdd-flag-readiness-3 (#272)"`. |
| 784 | |
amadan