amadan
plan: zdd flag readiness round 3 (draft)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+725
−0
docs/superpowers/plans/2026-10-13-zero-downtime-flag-readiness-3.md+725 −0
diff --git a/docs/superpowers/plans/2026-10-13-zero-downtime-flag-readiness-3.md b/docs/superpowers/plans/2026-10-13-zero-downtime-flag-readiness-3.md| new file mode 100644 |
| index 000000000..da55125c4 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-13-zero-downtime-flag-readiness-3.md |
| @@ -0,0 +1,725 @@ |
| +# Zero-downtime flag readiness, round 3: placement under the hardened edge unit |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Make armed generation starts work on the shipped, hardened `carlos-edge.service` by replacing clone3 `CLONE_INTO_CGROUP` placement with an exec-shim, make arming prove placement before it arms, and fix the three smaller defects the second dev-box check found. |
| + |
| +**Architecture:** Every `cgroup.Manager` start (flat, generation, worker, legacy sidecar, seed) starts the carlos binary itself as `/proc/self/exe __cgexec`. The shim receives the tenant's exec triple over a pipe, writes its own PID into the target cgroup's `cgroup.procs` through an already-validated directory fd, checks `/proc/self/cgroup` and the directory's identity, then `execve`s the tenant. The parent learns the outcome over a close-on-exec status pipe before `Start` returns, so a shim that exits before exec is a placement failure (`cgroup.ErrLimitNotApplied` for generations) and the tenant never runs outside its cgroup. Arming does one real shim placement into a scratch cgroup and declines with a reason code if it fails. |
| + |
| +**Tech Stack:** Go 1.26, `golang.org/x/sys/unix`, cgroup v2, systemd 257 (transient user units for kernel tests). |
| + |
| +**Spec:** /home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md (rev 17), with the approved rev-18 amendment on main: `docs/superpowers/specs/2026-10-12-zero-downtime-deploys-rev18-amendment.md`. §7.3 (containment) is the section this plan serves; the spec does not name a placement mechanism, so no spec text changes. |
| + |
| +**Inputs:** amadan discussion carlos/platform #272, latest reply ("Dev-box checks, round 2", 2026-10-13); command log `/home/paulca/.cache/tmp/claude-1001/zdd-dev-checks2.log`; roll notes `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md`. |
| + |
| +**Base:** branch `zdd-flag-readiness-3` from origin/main `34a4328f`. |
| + |
| +## Global Constraints |
| + |
| +- Paul's ruling (2026-10-13): do NOT relax `RestrictNamespaces=yes` or any other line of `internal/host/units/carlos-edge.service`. That file is not edited by this plan. |
| +- A generation's tenant code never runs outside its cgroup: no PID-move, no start-then-move, no retry outside the cgroup on the required (generation) path. |
| +- The shim never runs tenant code, never reads tenant-controlled environment before placement, and passes argv, env, cwd, fds 0..2+len(ExtraFiles), and the `SysProcAttr` the caller set to the tenant exactly as a direct `exec.Cmd.Start` would. |
| +- The PID `cmd.Process.Pid` the activator tracks is the shim's PID, which is the tenant's after `execve`; `cmd.Wait` reports the tenant's exit. |
| +- The strict path (`PrepareStrict`, `PrepareStrictParent`, `(*Prepared).Start`, `internal/cgroup/strict*.go`) keeps its clone3 placement unchanged. |
| +- No new member-facing strings unless unavoidable; any that are added carry draft text and a `TODO-COPY` marker and go into `copy-review/strings.json`, and the controller stops and reports that path. |
| +- Commits use the identity `claude <claude~paul@keymail.dev>`: `git -c user.name=claude -c user.email=claude~paul@keymail.dev commit ...`. Never push; the controller pushes. |
| +- Sandboxed Go: `GOCACHE=$TMPDIR/gocache` on every go command. Kernel and socket tests run unsandboxed with `TMPDIR=/tmp/k`. Never set GOCACHE/GOMODCACHE/GOTMPDIR under `/tmp`. |
| +- Kernel tests on DOCS-T1 run in a transient delegated user unit, for example: |
| + `systemd-run --user --wait --pipe --collect -p Delegate=yes -E TMPDIR=/tmp/k -E CARLOS_SHIM_KERNEL=1 <test binary> -test.run '<regex>' -test.v -test.count=1` |
| + (build the binary first with `go test -c -o $TMPDIR/cgroup.test ./internal/cgroup`, and copy it to `/tmp/k` if a unit's sandbox hides `$TMPDIR`). User delegation on DOCS-T1 offers `cpu memory pids`, not `io`. |
| +- Do not run anything on real boxes. Never run `git worktree prune`. Never push to main or run `amadan branch merge`. |
| +- Before committing: `gofmt -l` clean on touched files, `go vet` on touched packages, and `GOOS=darwin go build ./... && GOOS=windows go build ./...` when a touched file is not linux-only (linux-only code needs `//go:build linux` and a `!linux` stub). |
| +- Heavy test loops run under `nice -n 19`. |
| +- Integrity: reports paste real command output only. |
| + |
| +## Rulings recorded by this plan |
| + |
| +Each is ledgered by the controller as `Ruling: <what> — <why> — <cost if wrong>`. |
| + |
| +1. **(Paul, 2026-10-13)** The exec-shim replaces clone3 placement; no hardening on carlos-edge is relaxed. — RestrictNamespaces makes systemd's seccomp filter answer clone3 with ENOSYS, so CLONE_INTO_CGROUP is impossible in the agent. — Cost if wrong: one extra exec of the carlos binary per spawn. |
| +2. Flat starts (the non-required path, flag off included) use the shim too. On a shim *placement* failure, a flat start falls back to today's start-then-move by PID; a generation start never does. — The flat path's contract is that accounting never breaks a spawn, and every flat start under the shipped unit already uses start-then-move today. — Cost if wrong: a flag-off tenant can still run for a moment outside its cgroup when the shim fails, as every flag-off start does today. |
| +3. The clone3 attempt and its latch (`Manager.fallback`) leave the Manager start paths. The strict claims path keeps clone3. — Paul: keep the strict-claims path's existing placement. — Cost if wrong: with host claims on under the shipped unit, strict launches answer `ErrStrictUnsupported` (refused, never uncapped) until that path gets the shim. Claims are off everywhere and claims-on boxes decline arming; the roll notes record it as a claims-on-swaps precondition. |
| +4. The shim is `/proc/self/exe __cgexec`, used only when `cgroup.RunShimIfRequested()` has run in this process. — `/proc/self/exe` is the running build even after a roll replaces the file on disk; the hook guard stops a test binary without the hook from re-running its own tests as a "shim". — Cost if wrong: a binary that forgets the hook declines arming and its flat starts use start-then-move: loud, never unsafe. |
| +5. The tenant's path, argv and env travel over a pipe in a NUL-delimited encoding, never in the shim's argv or environment, and the shim runs with an empty environment. — Secrets stay out of `/proc/<pid>/cmdline`; no tenant variable (`GODEBUG`, `LD_*`, `GOMAXPROCS`) shapes the shim before placement; JSON would rewrite invalid UTF-8. — Cost if wrong: one pipe per spawn. |
| +6. A placement in flight counts as a member of its target cgroup and every ancestor up to the host directory: `Cleanup.Populated` answers true, `Cleanup.Members` answers `ErrPlacementInFlight`, `Cleanup.KillAndEmpty` waits for it to resolve (bounded by its ctx), and the legacy-parent guard counts one aimed at the host directory itself. — With clone3 the child appeared in its cgroup atomically; the shim spends a few milliseconds in the agent's cgroup first, and an emptiness proof must not miss it. — Cost if wrong: a stop that lands during a spawn's few milliseconds waits for it or answers unproven (the lease is kept), never a false empty. |
| +7. A new decline code `cgroup-placement-failed` reuses the approved declined phrase (`unavailableDeclined`), so it adds no copy. — Same meaning as the other declined codes for a member. — Cost if wrong: a console older than the box shows the generic unknown-code phrase for it until consoles roll (they roll after boxes). |
| +8. The arm-time placement check is box-wide: a failure declines the whole box, nested rows included. — One flag, one decision, as for `host-claims-on`. — Cost if wrong: a box whose nested rows could swap declines until flat placement works. |
| +9. The shim's status deadline is 10 s (`shimDeadline`). On expiry the shim is SIGKILLed and reaped, and the start is a placement failure. — A wake's budget is 30 s; a Go binary that cannot start in 10 s is a box condition. — Cost if wrong: a box that loaded refuses generation starts. |
| +10. `reset-failed` after a generation stop is best-effort: a failure is logged and the stop still answers 204. — The stop itself succeeded; the failed-list entry is cosmetic. — Cost if wrong: a failed generation unit can linger in `systemctl --failed`. |
| +11. The roll notes' probe app moves into the module as `cmd/zddprobe`, with a test that launches it with the platform's argv. — A heredoc cannot be tested. — Cost if wrong: one more `cmd/` directory. |
| +12. The 3000-seed simulation sweep runs only if the branch diff touches `internal/rollout` or the simulation's files; the shim, the arming check and the stop path are outside the modelled scheduler. — Cost if wrong: an unswept scheduler change; the final review checks the diff's paths. |
| + |
| +## Review Focus |
| + |
| +1. **Exact pass-through.** A tenant env with invalid UTF-8, an empty string, `=` inside a value, a 1 MiB value, and an argv with an empty element, a newline and spaces reach the tenant byte-for-byte, the same as a direct start (Task 1, `TestShimKernelPassesArgvEnvExactly`). |
| +2. **Inherited descriptors.** A `build()` that sets `ExtraFiles` (two pipes) gets them at fds 3 and 4 in the tenant, and the tenant has no other fd above 2: none of the shim's spec, directory or status fds, and none the agent opened without close-on-exec (Task 1, `TestShimKernelFDsExact`). |
| +3. **The agent's binary replaced on disk.** After a roll replaces `/opt/carlos/carlos`, the running agent's shim still execs (Task 1, `TestShimKernelSurvivesDeletedExecutable`). |
| +4. **A stop during a spawn.** `KillAndEmpty` or `Populated` on a host whose generation is being placed never reports empty while the shim is in flight (Task 2, `TestCleanupCountsPlacementInFlight`). |
| +5. **A missing tenant binary.** The start returns the command's own `fork/exec ... no such file or directory` error, not `ErrLimitNotApplied`, nothing ran, the shim is reaped, and an empty generation directory is removed (Task 1 and Task 2, `TestShimExecFailureIsTheCommands`, `TestGenerationMissingBinaryIsNotABoxCondition`). |
| + |
| +--- |
| + |
| +## File structure |
| + |
| +| File | Task | Responsibility | |
| +|---|---|---| |
| +| `internal/cgroup/shim.go` | 1 | Verbs, the hook (`RunShimIfRequested`, `shimAvailable`), the spec encoding, `placementError`, `ErrPlacementInFlight`. All platforms. | |
| +| `internal/cgroup/shim_linux.go` | 1 | `runShim`, `runProbe`, `startViaShim` (parent side). | |
| +| `internal/cgroup/shim_other.go` | 1 | `!linux` stubs: the shim verbs exit 127, `startViaShim` answers a placement error. | |
| +| `internal/cgroup/main_test.go` | 1 | `TestMain` calling `RunShimIfRequested()`. | |
| +| `internal/cgroup/shim_test.go` | 1 | Encoding round-trips (all platforms). | |
| +| `internal/cgroup/shim_linux_test.go` | 1 | Ungated parent/shim protocol tests that need no delegation. | |
| +| `internal/cgroup/shim_kernel_linux_test.go` | 1 | Gated (`CARLOS_SHIM_KERNEL=1`) real-cgroup shim tests. | |
| +| `main.go` | 1 | `cgroup.RunShimIfRequested()` as the first statement of `main()`. | |
| +| `internal/cgroup/inflight.go` | 1 | The in-flight placement registry on `Manager`. | |
| +| `internal/cgroup/cgroup.go`, `cgroup_linux.go`, `cgroup_other.go`, `gen.go`, `cleanup_linux.go` | 2 | `startIn` on the shim; latch removed; Cleanup and the parent guard count placements in flight. | |
| +| `internal/cgroup/verify_linux.go`, `verify_other.go` | 3 | `Manager.VerifyPlacement`. | |
| +| `internal/host/rollout.go`, `internal/rolloutwords/words.go`, `internal/status/snapshot.go` | 3 | The decline code and its wording. | |
| +| `internal/cgroup/edge_hardening_kernel_linux_test.go` | 4 | The transient-unit test under the edge unit's sandbox lines. | |
| +| `cmd/zddprobe/main.go`, `cmd/zddprobe/main_test.go` | 5 | The roll notes' probe app, tested with the platform's argv. | |
| +| `internal/cgroup/strict_kernel_linux_test.go` | 6 | `logf` on the test's second managers. | |
| +| `internal/instancectl/gen.go` (+ its test) | 6 | `reset-failed` after a generation stop. | |
| +| `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` | 5, 7 | The probe section; the round-3 notes and the dev re-check list. | |
| + |
| +--- |
| + |
| +### Task 1: The exec-shim (opus) |
| + |
| +**Files:** |
| +- Create: `internal/cgroup/shim.go`, `internal/cgroup/shim_linux.go`, `internal/cgroup/shim_other.go`, `internal/cgroup/inflight.go`, `internal/cgroup/main_test.go`, `internal/cgroup/shim_test.go`, `internal/cgroup/shim_linux_test.go`, `internal/cgroup/inflight_linux_test.go`, `internal/cgroup/shim_kernel_linux_test.go` |
| +- Modify: `main.go:20` (first statement of `main()`); `internal/cgroup/cgroup.go:173-184` (the registry's fields on `Manager`) |
| + |
| +**Interfaces:** |
| +- Produces (used by Tasks 2 to 4): |
| + - `const ShimVerb = "__cgexec"`, `const ProbeVerb = "__cgprobe"` |
| + - `func RunShimIfRequested()` — first statement of `main()` and of `TestMain`. Marks the process as able to serve the shim; if `os.Args[1]` is `ShimVerb` or `ProbeVerb` it runs that and calls `os.Exit`, never returning. |
| + - `func shimAvailable() bool` — the hook ran in this process and the platform is linux. |
| + - `type placementError struct{ err error }` with `Error()` and `Unwrap()`; `func isPlacementError(err error) bool`. |
| + - `var ErrPlacementInFlight = errors.New("cgroup: a process is being placed in this cgroup")` (defined here, used by Task 2). |
| + - `func (m *Manager) startViaShim(dir string, cmd *exec.Cmd, before func() error) (refused bool, err error)` — linux. On nil error the tenant is running in `dir` and `cmd.Process` is it. `refused` means `before` said no and nothing started. Any other error: nothing ran and any shim process is reaped; a `*placementError` means the process could not be placed, anything else is the command's own error (as `exec.Cmd.Start` would have returned it). It registers the placement in flight itself, after `before` agrees. |
| + - The in-flight registry: `beginPlacement`, `placementsUnder`, `placementsAt`, `waitPlacementsUnder` (signatures below). |
| + - `var shimDeadline = 10 * time.Second` |
| + - `var shimExe = "/proc/self/exe"` |
| +- Consumes: `ParseSelfCgroup` (`internal/cgroup/cgroup.go:155`). |
| + |
| +**The protocol.** |
| + |
| +Parent, in `startViaShim`: |
| +1. If `!shimAvailable()`, return `&placementError{errors.New("the placement shim is not available in this process")}`. If `cmd.Err != nil` (a failed `LookPath` in `build()`), return `cmd.Err`. |
| +2. Open `dir` with `unix.Open(dir, O_RDONLY|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC, 0)`. `unix.Fstatfs` it; anything but `unix.CGROUP2_SUPER_MAGIC` is a placement error. Compute `expected`: `dir` must start with `/sys/fs/cgroup/`; `expected` is `dir` with the `/sys/fs/cgroup` prefix cut (so it starts with `/`). Anything else is a placement error. |
| +3. Capture the tenant before rewriting anything: `path := cmd.Path`; `argv := cmd.Args`, or `[]string{cmd.Path}` when empty (exec.Cmd's own rule); `env := cmd.Environ()` (exactly what `Start` would pass, `PWD` included); `keep := 3 + len(cmd.ExtraFiles)`. |
| +4. `os.Pipe()` twice: spec (shim reads) and status (shim writes). Rewrite `cmd`: `Path = shimExe`, `Args = []string{"carlos-cgexec", ShimVerb, itoa(keep), itoa(keep+1), itoa(keep+2)}`, `Env = []string{}`, `ExtraFiles = append(original ExtraFiles, specR, dirFile, statusW)`. Leave `Dir`, `Stdin`, `Stdout`, `Stderr`, `SysProcAttr`, `Cancel`, `WaitDelay` untouched. |
| +5. Ask `before` (when set); a refusal returns `true, before's error` with nothing started. Only then `done := m.beginPlacement(dir)` (the in-flight registry, below), and `startCmd(cmd)` (`internal/cgroup/cgroup.go:332`). The registry is taken after `before` and never across it: a stop that waits on the registry may hold a lock `before` needs. Close the parent's `specR`, `dirFile` and `statusW` whatever happened. A failed `startCmd` (the carlos binary itself did not start) is a placement error; `done()` and return it. |
| +6. Write the encoded spec to `specW` with a write deadline at `shimDeadline`, then close it. Read `statusR` to EOF with a read deadline at `shimDeadline` (pipes from `os.Pipe` support deadlines). |
| +7. Decide from what was read: a line `E <errno>` is the command's own failure: `&os.PathError{Op: "fork/exec", Path: path, Err: syscall.Errno(n)}`. A line `P <reason>` is a placement error with that reason. `R` with nothing after it is success. Nothing at all (EOF without `R`) is a placement error ("the shim exited before exec"). A deadline is a placement error ("no answer from the shim within 10s"). |
| +8. On any failure: `cmd.Process.Kill()` then `cmd.Wait()` (ignore its error), so no shim outlives the call, then `done()`. On success `done()` and return `false, nil`: the process is in `dir` by then, so a proof that reads `dir` sees it. |
| + |
| +**The in-flight registry** (`inflight.go`, used by Task 2's proofs): a mutex-guarded `map[string]int` of target directories (cleaned paths), plus a `chan struct{}` that is closed and replaced whenever a count drops, so waiters wake. |
| + |
| +```go |
| +func (m *Manager) beginPlacement(dir string) (done func()) // done is idempotent |
| +func (m *Manager) placementsUnder(dir string) int // targets equal to dir or below dir+"/" |
| +func (m *Manager) placementsAt(dir string) int // targets equal to dir |
| +func (m *Manager) waitPlacementsUnder(ctx context.Context, dir string) error // nil at zero, ctx.Err() if ctx ends first |
| +``` |
| + |
| +A nil `*Manager` answers zero and nil, and its `beginPlacement` returns a no-op. |
| + |
| +Shim, in `runShim(args []string) int` (args are the three fd numbers): |
| +1. Parse the three fds; they must be `keep`, `keep+1`, `keep+2` for some `keep >= 3`. Set `FD_CLOEXEC` on the status fd first, so every later path closes it at exec. |
| +2. Read the spec fd to EOF, at most 16 MiB, and close it. Decode it. |
| +3. Mark every fd `>= keep` close-on-exec: `unix.CloseRange(uint(keep), math.MaxUint32, unix.CLOSE_RANGE_CLOEXEC)`; if that answers `ENOSYS`, `EPERM` or `EINVAL`, walk `/proc/self/fd` and set `FD_CLOEXEC` on each fd `>= keep`. This covers the spec, directory and status fds and anything the agent leaked without close-on-exec. |
| +4. `unix.Fstatfs(dirFD)` must be `CGROUP2_SUPER_MAGIC`. |
| +5. `unix.Openat(dirFD, "cgroup.procs", O_WRONLY|O_NOFOLLOW|O_CLOEXEC, 0)`; write `strconv.Itoa(os.Getpid())`; close. This moves the whole process (every Go runtime thread) while it is still carlos code. |
| +6. Read `/proc/self/cgroup`; `ParseSelfCgroup`; `"/"+rel` must equal the spec's `Expected`. |
| +7. `unix.Fstat(dirFD)` and `unix.Lstat("/sys/fs/cgroup"+Expected)` must agree on `Dev` and `Ino`, so the cgroup the kernel reports is the directory the parent validated. |
| +8. Any failure in 1 to 7: write `P <step>: <error>\n` to the status fd and return 125. Tenant code has not run. |
| +9. Write `R\n`, then `syscall.Exec(spec.Path, spec.Argv, spec.Env)`. `syscall.Exec` restores the original `RLIMIT_NOFILE` (Go source `syscall/exec_unix.go:286`). If it returns, write `E <errno>\n` and return 126. |
| + |
| +The shim writes nothing to fds 0 to 2. |
| + |
| +`runProbe() int` (for Task 3): read `/proc/self/cgroup`, print `"/"+rel` and a newline to stdout, return 0; return 1 when there is no v2 line. |
| + |
| +**Spec encoding** (`shim.go`): fields each terminated by `\x00`, in order: the magic `carlos-cgexec-1`, `Path`, `Expected`, `KeepFDs` (decimal), `TestHold` (`0` or `1`), `len(Argv)` (decimal), each argv element, `len(Env)` (decimal), each env entry. Encoding refuses any string that contains `\x00` (execve could not carry it anyway). Decoding refuses a wrong magic, a short or trailing buffer, or a count above 1<<20. |
| + |
| +```go |
| +type shimSpec struct { |
| + Path string |
| + Argv []string |
| + Env []string |
| + Expected string // "/..." as /proc/self/cgroup shows it after "0::" |
| + KeepFDs int // fds 0..KeepFDs-1 are the tenant's |
| + TestHold bool // test binaries only: the shim waits for SIGUSR1 before step 4 |
| +} |
| + |
| +func encodeShimSpec(s shimSpec) ([]byte, error) |
| +func decodeShimSpec(b []byte) (shimSpec, error) |
| +``` |
| + |
| +`TestHold` is honoured only when `testing.Testing()` is true in the shim process (the package already imports `testing`, `cgroup.go:30`), so in the carlos binary it is inert. When honoured, the shim sleeps before step 4 until it is signalled, with default signal dispositions (no `signal.Notify`): a test's SIGTERM or SIGKILL then kills it before placement, which is the case under test. The parent sets the field from `var shimTestHold bool`, which only test files flip. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +`internal/cgroup/main_test.go`: |
| + |
| +```go |
| +package cgroup |
| + |
| +import ( |
| + "os" |
| + "testing" |
| +) |
| + |
| +func TestMain(m *testing.M) { |
| + RunShimIfRequested() |
| + os.Exit(m.Run()) |
| +} |
| +``` |
| + |
| +`internal/cgroup/shim_test.go` (all platforms): |
| + |
| +```go |
| +func TestShimSpecRoundTripsExactBytes(t *testing.T) { |
| + in := shimSpec{ |
| + Path: "/run/carlos/exe/app", |
| + Argv: []string{"app", "", "two words", "line\nbreak", "--db", "/data/x.db"}, |
| + Env: []string{"A=1", "EMPTY=", "EQ=a=b", "BAD=\xff\xfe", "BIG=" + strings.Repeat("x", 1<<20)}, |
| + Expected: "/system.slice/carlos-edge.service/carlos-exec@a.example/g3", |
| + KeepFDs: 5, |
| + } |
| + b, err := encodeShimSpec(in) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + out, err := decodeShimSpec(b) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if !reflect.DeepEqual(in, out) { |
| + t.Fatalf("round trip changed the spec") |
| + } |
| +} |
| + |
| +func TestShimSpecRefusesNUL(t *testing.T) { |
| + if _, err := encodeShimSpec(shimSpec{Path: "/x", Argv: []string{"a\x00b"}, Expected: "/g", KeepFDs: 3}); err == nil { |
| + t.Fatal("a NUL inside argv was encoded") |
| + } |
| +} |
| + |
| +func TestShimSpecRefusesTruncatedAndTrailing(t *testing.T) { |
| + b, _ := encodeShimSpec(shimSpec{Path: "/x", Argv: []string{"x"}, Env: []string{"A=1"}, Expected: "/g", KeepFDs: 3}) |
| + for _, bad := range [][]byte{b[:len(b)-1], append(append([]byte{}, b...), 'z', 0), []byte("carlos-cgexec-0\x00")} { |
| + if _, err := decodeShimSpec(bad); err == nil { |
| + t.Fatalf("decoded a bad spec %q", bad) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +`internal/cgroup/shim_linux_test.go` (ungated; a temp directory is not cgroupfs, so these exercise the failure paths): |
| + |
| +```go |
| +// TestShimRefusesANonCgroupDirectory: the parent's own check (step 2) |
| +// refuses before anything starts, as a placement error. |
| +func TestShimRefusesANonCgroupDirectory(t *testing.T) { |
| + m := &Manager{root: t.TempDir(), logf: t.Logf} |
| + marker := filepath.Join(t.TempDir(), "ran") |
| + cmd := exec.Command("/bin/sh", "-c", "touch "+marker) |
| + refused, err := m.startViaShim(m.root, cmd, nil) |
| + if refused || !isPlacementError(err) { |
| + t.Fatalf("refused=%v err=%v, want a placement error", refused, err) |
| + } |
| + if cmd.Process != nil { |
| + t.Fatal("a process was started") |
| + } |
| + if _, err := os.Stat(marker); err == nil { |
| + t.Fatal("tenant code ran") |
| + } |
| +} |
| + |
| +// TestShimUnavailableWithoutTheHook: a process whose main never ran the |
| +// hook never starts a shim. |
| +func TestShimUnavailableWithoutTheHook(t *testing.T) { ... flip the hooked flag off for the test, restore in Cleanup; startViaShim answers a placement error and cmd.Process stays nil ... } |
| + |
| +// TestShimPlacementFailureIsReported runs a real shim (this test binary, |
| +// through TestMain's hook) against a temp directory. Split startViaShim |
| +// into its checked prologue (steps 1 to 3) and a core, |
| +// func (m *Manager) startShimCore(dir string, dirFile *os.File, expected string, cmd *exec.Cmd, before func() error) (bool, error) |
| +// and call the core with a temp directory's fd, which the prologue would |
| +// have refused. The shim's own fstatfs check (shim step 4) answers "P". |
| +func TestShimPlacementFailureIsReported(t *testing.T) { |
| + m := &Manager{root: t.TempDir(), logf: t.Logf} |
| + f, err := os.Open(m.root) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer f.Close() |
| + marker := filepath.Join(t.TempDir(), "ran") |
| + cmd := exec.Command("/bin/sh", "-c", "touch "+marker) |
| + _, err = m.startShimCore(m.root, f, "/nowhere", cmd, nil) |
| + if !isPlacementError(err) { |
| + t.Fatalf("err=%v, want a placement error", err) |
| + } |
| + if cmd.ProcessState == nil { |
| + t.Fatal("the shim was not reaped") |
| + } |
| + if _, err := os.Stat(marker); err == nil { |
| + t.Fatal("tenant code ran") |
| + } |
| + if m.placementsUnder(m.root) != 0 { |
| + t.Fatal("the placement is still registered") |
| + } |
| +} |
| + |
| +// TestShimBeforeRefusalRegistersNothing: before says no, nothing starts and |
| +// nothing is registered. |
| +func TestShimBeforeRefusalRegistersNothing(t *testing.T) { ... } |
| + |
| +// TestPlacementsUnderCountsTheSubtree and TestWaitPlacementsUnderWakes |
| +// cover the registry (inflight_linux_test.go): |
| +func TestPlacementsUnderCountsTheSubtree(t *testing.T) { |
| + m := &Manager{root: "/r", logf: t.Logf} |
| + done := m.beginPlacement("/r/carlos-exec@a.example/g2") |
| + if m.placementsUnder("/r/carlos-exec@a.example") != 1 || m.placementsAt("/r/carlos-exec@a.example") != 0 { |
| + t.Fatal("a child's placement is under its parent, not at it") |
| + } |
| + if m.placementsUnder("/r/carlos-exec@a.exampleX") != 0 { |
| + t.Fatal("a prefix without a separator matched") |
| + } |
| + done() |
| + done() // idempotent |
| + if m.placementsUnder("/r/carlos-exec@a.example") != 0 { |
| + t.Fatal("done did not drop the count") |
| + } |
| +} |
| + |
| +func TestWaitPlacementsUnderWakes(t *testing.T) { ... one in flight; a waiter with a 50 ms ctx returns context.DeadlineExceeded; a second waiter returns nil within 1 s of done() ... } |
| +``` |
| + |
| +The exec-failure case needs a placement that succeeds, so it lives in the kernel file below. This file also covers the status parser: |
| + |
| +```go |
| +func TestShimStatusParsing(t *testing.T) { |
| + for _, c := range []struct { |
| + in string |
| + placement bool |
| + execErrno syscall.Errno |
| + ok bool |
| + }{ |
| + {"R\n", false, 0, true}, |
| + {"", true, 0, false}, |
| + {"P procs: permission denied\n", true, 0, false}, |
| + {"R\nE 2\n", false, syscall.ENOENT, false}, |
| + {"R\nE 13\n", false, syscall.EACCES, false}, |
| + {"garbage", true, 0, false}, |
| + } { |
| + err := parseShimStatus("/bin/app", []byte(c.in)) |
| + ... assert ok => err == nil; placement => isPlacementError(err); execErrno => errors.As *os.PathError with Op "fork/exec", Path "/bin/app", Err == execErrno ... |
| + } |
| +} |
| +``` |
| + |
| +`internal/cgroup/shim_kernel_linux_test.go` (gated: `CARLOS_SHIM_KERNEL=1` inside a delegated unit; reuse `genKernelManager`'s setup at `gen_kernel_linux_test.go:27`, and skip without the variable). Each test places into `m.root/<name>` made with `os.Mkdir` and a `TierDefault` limit written through `m.applyLimit(..., true)`: |
| + |
| +- `TestShimKernelPlacesBeforeExec`: tenant `/bin/sh -c 'cat /proc/self/cgroup; echo $$'`. After `startViaShim` returns: `/proc/<pid>/cgroup` names the target; the tenant's printed cgroup is the target; the tenant's printed `$$` equals `cmd.Process.Pid` (the leader PID is the shim's, now the tenant's); `cmd.Wait()` returns nil. |
| +- `TestShimKernelExitCodePassesThrough`: tenant `/bin/sh -c 'exit 7'`; `cmd.Wait()` gives an `*exec.ExitError` with `ExitCode() == 7`. |
| +- `TestShimKernelPassesArgvEnvExactly`: the test binary as tenant in a child mode (`CARLOS_SHIM_CHILD=dump`, env carrying Review Focus 1's values) writes `os.Args` and `os.Environ()` as NUL-joined bytes to stdout; compare against the same command started directly with `cmd.Start()`; also compare its `os.Getwd()` with `cmd.Dir`. |
| +- `TestShimKernelFDsExact`: `ExtraFiles` of two pipes; the agent side also opens one file with `syscall.Open` without `O_CLOEXEC` before the start (closed in Cleanup). The child mode lists `/proc/self/fd` (excluding the fd its own listing opens): exactly `0 1 2 3 4`. Also `SigBlk`, `SigIgn` from `/proc/self/status` equal a direct start's. |
| +- `TestShimExecFailureIsTheCommands`: tenant path `/nonexistent/app` (set `cmd.Path` directly so `LookPath` does not fail first): error is `*os.PathError` with `Err == syscall.ENOENT`, `!isPlacementError(err)`, `cmd.ProcessState != nil`. |
| +- `TestShimKernelPlacementFailureNeverRunsTenant`: target a cgroup the shim cannot write (`chmod 0444` its `cgroup.procs`, restore in Cleanup): placement error, marker file absent, reaped. |
| +- `TestShimKernelSignalBeforeExec`: `shimTestHold = true`; a `startCmd` seam wrapper (see `cgroup.go:332`) starts the shim and then sends SIGTERM to it; the result is a placement error ("exited before exec"), the marker is absent, `cmd.ProcessState` is set. Then the same with SIGKILL. |
| +- `TestShimKernelDeadline`: `shimTestHold = true`, `shimDeadline = 300ms` (restored in Cleanup), no signal: a placement error naming the deadline, the shim reaped, marker absent. |
| +- `TestShimKernelSurvivesDeletedExecutable`: copy `os.Args[0]` to `t.TempDir()/copy.test`, run the copy as a child test (`-test.run '^TestShimKernelDeletedChild$'`, env `CARLOS_SHIM_KERNEL=1 CARLOS_SHIM_DELETED_CHILD=1`); the child deletes its own executable, then places `/bin/true` through `startViaShim` and must succeed. |
| +- `TestShimKernelOverhead` (informational, never fails on time): 20 shim starts of `/bin/true` against 20 direct starts; `t.Logf` the medians. |
| + |
| +- [ ] **Step 2: Run the tests to see them fail** |
| + |
| +Run: `cd <worktree> && GOCACHE=$TMPDIR/gocache go test ./internal/cgroup -run 'Shim' -count=1` |
| +Expected: build failure (`encodeShimSpec`, `startViaShim`, `RunShimIfRequested` undefined). |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +`shim.go`: |
| + |
| +```go |
| +package cgroup |
| + |
| +import ( |
| + "errors" |
| + "os" |
| + "sync/atomic" |
| +) |
| + |
| +// ShimVerb is the carlos binary's hidden placement verb (Paul's ruling, |
| +// #272, 2026-10-13). carlos-edge.service's RestrictNamespaces=yes makes |
| +// systemd's seccomp filter answer clone3 with ENOSYS, so no child can be |
| +// created inside its cgroup. Instead the child starts as this binary, |
| +// writes its own PID into the target cgroup through a directory fd the |
| +// parent validated, checks where it landed, and only then execs the |
| +// tenant: the tenant's code never runs outside its limits. |
| +const ShimVerb = "__cgexec" |
| + |
| +// ProbeVerb prints the process's own cgroup and exits: the tenant the |
| +// arm-time placement check (VerifyPlacement) execs through the shim. |
| +const ProbeVerb = "__cgprobe" |
| + |
| +var shimHooked atomic.Bool |
| + |
| +// RunShimIfRequested must be the first statement of main(), and of TestMain |
| +// in a test binary that places processes. It marks this process as able to |
| +// serve the shim (shimAvailable), and when this process IS a shim or a |
| +// probe it runs it and exits; it returns only when it is neither. |
| +func RunShimIfRequested() { |
| + shimHooked.Store(true) |
| + if len(os.Args) < 2 { |
| + return |
| + } |
| + switch os.Args[1] { |
| + case ShimVerb: |
| + os.Exit(runShim(os.Args[2:])) |
| + case ProbeVerb: |
| + os.Exit(runProbe()) |
| + } |
| +} |
| + |
| +// placementError is a start whose process could not be placed in its |
| +// cgroup before exec. Nothing of the tenant ran. |
| +type placementError struct{ err error } |
| + |
| +func (e *placementError) Error() string { return "placement before exec: " + e.err.Error() } |
| +func (e *placementError) Unwrap() error { return e.err } |
| + |
| +func isPlacementError(err error) bool { |
| + var p *placementError |
| + return errors.As(err, &p) |
| +} |
| + |
| +// ErrPlacementInFlight is a cleanup asked for members while a process is |
| +// being placed in its cgroup (Task 2). |
| +var ErrPlacementInFlight = errors.New("cgroup: a process is being placed in this cgroup") |
| +``` |
| + |
| +`shimAvailable()` is `shimHooked.Load()` in `shim_linux.go` and `false` in `shim_other.go`. `shim_other.go`'s `runShim`/`runProbe` return 127, and its `startViaShim` returns `false, &placementError{errors.New("placement before exec needs linux")}`. |
| + |
| +`shim_linux.go` implements the parent and shim protocol above. Factor the status decision into `parseShimStatus(path string, b []byte) error` so `TestShimStatusParsing` covers it. Keep every comment to what the code does and why. |
| + |
| +`main.go`, first statement of `main()`: |
| + |
| +```go |
| +func main() { |
| + // The placement shim (internal/cgroup, #272) must run before anything |
| + // else touches the environment or the logger: a shim process carries |
| + // none of the tenant's settings and exits from inside this call. |
| + cgroup.RunShimIfRequested() |
| + rootaws.ScrubEnvironment() |
| +``` |
| + |
| +(Import `github.com/carlosframework/platform/internal/cgroup` if `main.go` does not already.) |
| + |
| +- [ ] **Step 4: Run the tests** |
| + |
| +Run (sandboxed): `GOCACHE=$TMPDIR/gocache go test ./internal/cgroup -count=1` |
| +Expected: PASS; the kernel tests SKIP. |
| + |
| +Run (unsandboxed, kernel): |
| +```bash |
| +GOCACHE=$TMPDIR/gocache go test -c -o /tmp/k/cgroup.test ./internal/cgroup && \ |
| +systemd-run --user --wait --pipe --collect -p Delegate=yes -E TMPDIR=/tmp/k -E CARLOS_SHIM_KERNEL=1 -E CARLOS_GEN_KERNEL=1 \ |
| + /tmp/k/cgroup.test -test.run 'ShimKernel|ShimExecFailure' -test.v -test.count=1 |
| +``` |
| +Expected: every `ShimKernel` test PASS, with the overhead medians in the log. Paste the real output into the report. Then `rm /tmp/k/cgroup.test`. |
| + |
| +Also: `GOOS=darwin GOCACHE=$TMPDIR/gocache go build ./... && GOOS=windows GOCACHE=$TMPDIR/gocache go build ./...` and `go vet ./internal/cgroup .` |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/cgroup/shim*.go internal/cgroup/main_test.go main.go |
| +git -c user.name=claude -c user.email=claude~paul@keymail.dev commit -m "cgroup: the placement exec-shim (#272)" |
| +``` |
| + |
| +--- |
| + |
| +### Task 2: Every Manager start places through the shim (opus) |
| + |
| +**Files:** |
| +- Modify: `internal/cgroup/cgroup.go:170-184` (Manager: drop `fallback`), `cgroup.go:330-549` (`startCmd`, `StartGated` doc, `FakePlacementForTest`, `startIn`), `internal/cgroup/cgroup_linux.go` (drop what only the clone3 start used; keep `setInto` and `shouldLatch`, which `strict_linux.go:344-350` uses), `internal/cgroup/cgroup_other.go`, `internal/cgroup/gen.go:171-235` (`ensureParent`), `internal/cgroup/cleanup_linux.go:54-353` (Cleanup), the tests that pinned the clone3 start and its latch (`internal/cgroup/cgroup_linux_test.go`, `gen_test.go`, `cgroup_test.go`), and comments naming the clone3 start in `internal/activator/{swap.go:205-207, backend_exec.go:2132, backend_exec.go:2920, genbackend.go:400, genbackend.go:495, workerbuild.go:102}` and `internal/cgroup/recreate_linux.go:19,118-122`. |
| + |
| +**Interfaces:** |
| +- Consumes (Task 1): `startViaShim`, `shimAvailable`, `isPlacementError`, `ErrPlacementInFlight`, and the registry (`beginPlacement`, `placementsUnder`, `placementsAt`, `waitPlacementsUnder`). |
| +- Produces: |
| + - `var placeStart = func(m *Manager, dir string, cmd *exec.Cmd, before func() error) (bool, error) { return m.startViaShim(dir, cmd, before) }` — the seam `FakePlacementForTest` swaps. |
| + - The start functions' signatures do not change. |
| + |
| +**Behaviour.** |
| + |
| +`startIn(host, dir, limit, required, build, before)`: |
| +- Required (generation): if `!shimAvailable()`, log once and return `ErrLimitNotApplied` before the directory is made or `build()` runs (as the latch refusal does today, `cgroup.go:453-458`). Make the directory and apply the limit as today. `refused, err := placeStart(m, dir, build(), before)` (the shim registers the placement itself). Refused: remove the directory if empty, return `before`'s error. Success: `biasOOM`, return the cmd. A placement error: remove the directory if empty, log `cgroup: <host>: placement before exec failed (<err>) — generation start refused`, return `fmt.Errorf("%w: %s: %w", ErrLimitNotApplied, host, err)`. Any other error is the command's: remove the directory if empty, return it unwrapped. |
| +- Flat: make the directory (failure: start plain, as today `cgroup.go:465-466`) and apply the limit best-effort as today. If `!shimAvailable()`, go straight to `startAndMoveIn` (today's fallback). Otherwise place through the shim. Success: `biasOOM`, return. Refused: as today. A placement error: log `cgroup: <host>: placement before exec failed (<err>) — using the PID-move fallback for this spawn` and `startAndMoveIn` with a fresh `build()` behind `before` (Ruling 2). Any other error is the command's: remove the directory if empty and return it, with no retry. |
| +- `Manager.fallback`, the read-once comment and the unreachable required-path guard at `cgroup.go:538-546` go; keep a single explicit statement that the required path never reaches `startAndMoveIn`. |
| + |
| +`FakePlacementForTest`: keep its name, signature, panic outside tests and no-`t.Parallel` note; it now swaps `placeStart` for one that runs `startGated(cmd, before)` with no shim (as if the kernel had placed it) and restores it in Cleanup. Update its doc. |
| + |
| +`startAndMoveIn` (the flat fallback, `cgroup.go:625`) registers its target too, from after `before` agrees until the PID write has returned: its child runs in the agent's cgroup for that window, and a proof must not miss it either. Give `startGated` an optional register step so both paths take the registry at the same point. |
| + |
| +Cleanup (`cleanup_linux.go`): `openCleanupAt` records the manager and the directory path it pinned (`filepath.Join(m.root, ExecName(host))`, plus `/child` for a child cleanup) in `cleanupState`. `Populated` answers `true, nil` while `m.placementsUnder(path) > 0`, before reading `cgroup.events`. `Members` answers `nil, ErrPlacementInFlight` while one is in flight. `KillAndEmpty` first calls `m.waitPlacementsUnder(ctx, path)`, returning its error (unproven) if ctx ends first, then does what it does today. |
| + |
| +`ensureParent` (`gen.go:198`): after `parentProcs`, add `m.placementsAt(dir)` to `n`, so a flat start being placed into the host directory itself counts as a direct process and the generation start is refused with `ErrLegacyParentPopulated`. |
| + |
| +Check every caller of `Cleanup.Members` (`grep -rn "\.Members()" internal`) treats an error as unproven (keeps the lease, records nothing as empty). If one does not, fix that caller in this task and name it in the report. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +`TestCleanupCountsPlacementInFlight` (in `cleanup_linux_test.go`, over the existing fake-root helper at `cleanup_linux_test.go:28`, which builds `&Manager{root: root}` — give it `logf: t.Logf`): with a placement begun on `<host dir>/g1`, a host Cleanup's `Populated()` is `true` even though the fake `cgroup.events` says `populated 0`; `Members()` is `ErrPlacementInFlight`; `KillAndEmpty` with a 50 ms ctx returns an error and writes nothing to `cgroup.kill`; after `done()`, `Populated()` reads the file again. |
| + |
| +`TestEnsureParentCountsAPlacementAtTheParent` (in `gen_test.go`): a placement begun on the host directory itself makes `StartChildGated` answer `ErrLegacyParentPopulated` and build nothing. |
| + |
| +`TestGenerationMissingBinaryIsNotABoxCondition` (in `gen_test.go`, with a `placeStart` stub returning `&os.PathError{Op: "fork/exec", Path: "/x", Err: syscall.ENOENT}`): `StartChildGated` returns that error, `errors.Is(err, ErrLimitNotApplied)` is false, and the child directory is gone. |
| + |
| +`TestGenerationPlacementFailureIsLimitNotApplied` (stub returns `&placementError{...}`): `errors.Is(err, ErrLimitNotApplied)`, child directory gone, `build` called once. |
| + |
| +`TestGenerationRefusedWithoutTheShim` (flip the hook flag off): `ErrLimitNotApplied`, `build` never called, no child directory made. |
| + |
| +`TestFlatPlacementFailureFallsBackToPIDMove` (stub returns a placement error): `StartGated` succeeds through `startAndMoveIn` (the fake `cgroup.procs` holds the child's PID), `build` called twice, `before` asked twice. |
| + |
| +`TestFlatCommandFailureIsNotRetried` (stub returns a `fork/exec` ENOENT): returned as is, `build` called once. |
| + |
| +Replace the clone3 latch tests in `cgroup_linux_test.go` (those asserting `UseCgroupFD`, `shouldLatch` latching, or `fallback`) with the tests above; keep any test of `shouldLatch` that the strict path still relies on. |
| + |
| +- [ ] **Step 2: Run them to see them fail** |
| + |
| +Run: `GOCACHE=$TMPDIR/gocache go test ./internal/cgroup -count=1` |
| +Expected: FAIL / build errors on the new names. |
| + |
| +- [ ] **Step 3: Implement** the behaviour above. |
| + |
| +- [ ] **Step 4: Run the tests** |
| + |
| +```bash |
| +GOCACHE=$TMPDIR/gocache go test ./internal/cgroup -count=1 |
| +GOCACHE=$TMPDIR/gocache go vet ./internal/cgroup ./internal/activator |
| +GOOS=darwin GOCACHE=$TMPDIR/gocache go build ./... && GOOS=windows GOCACHE=$TMPDIR/gocache go build ./... |
| +``` |
| +Then unsandboxed, the activator package and the other cgroup users (sockets need `/tmp/k`): |
| +```bash |
| +mkdir -p /tmp/k && TMPDIR=/tmp/k GOCACHE=$HOME/.cache/go-build nice -n 19 go test ./internal/activator ./internal/host ./internal/stopevidence ./internal/instancectl -count=1 |
| +``` |
| +Then the generation kernel tests in a delegated user unit, which now place through the shim: |
| +```bash |
| +GOCACHE=$TMPDIR/gocache go test -c -o /tmp/k/cgroup.test ./internal/cgroup && \ |
| +systemd-run --user --wait --pipe --collect -p Delegate=yes -E TMPDIR=/tmp/k -E CARLOS_GEN_KERNEL=1 -E CARLOS_SHIM_KERNEL=1 \ |
| + /tmp/k/cgroup.test -test.run 'GenKernel|GenOOMKernel|ShimKernel|GenerationStartRefusedBesideADirectParentMember|ClaimContainment' -test.v -test.count=1; rm -f /tmp/k/cgroup.test |
| +``` |
| +Expected: PASS (paste the real output). A test that asserted clone3-specific kernel behaviour (the `cgroup.kill` respawn kill was clone3's) is updated to assert the shim's behaviour on the same scenario, and the report says what changed and why. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git -c user.name=claude -c user.email=claude~paul@keymail.dev commit -am "cgroup: every Manager start places through the shim; placements in flight count as members (#272)" |
| +``` |
| +(`git add` the new files first.) |
| + |
| +--- |
| + |
| +### Task 3: Arming verifies placement (opus) |
| + |
| +**Files:** |
| +- Create: `internal/cgroup/verify_linux.go`, `internal/cgroup/verify_other.go`, `internal/cgroup/verify_linux_test.go` |
| +- Modify: `internal/host/rollout.go:22-44` (the code), `rollout.go:150-166` (`swapArming`), `internal/host/rollout_test.go`, `internal/rolloutwords/words.go:84-112` (+ its test), `internal/status/snapshot.go:300-304` (comment) |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1's `startViaShim` (which registers the placement in flight), `ProbeVerb`, `shimAvailable`, `shimExe`. |
| +- Produces: |
| + - `func (m *Manager) VerifyPlacement(ctx context.Context) error` (linux); the `!linux` stub returns an error. |
| + - `const placementCheckName = "carlos-placement-check"` |
| + - In `internal/host`: `rolloutPlacementFailed = "cgroup-placement-failed"` and `var placementCheck = func(m *cgroup.Manager) error` (a 15 s ctx around `VerifyPlacement`), a seam host tests replace. |
| + |
| +**Behaviour.** `VerifyPlacement`: |
| +1. Nil manager: `ErrNoDelegation`. `!shimAvailable()`: an error saying the shim is not available. |
| +2. `dir := filepath.Join(m.root, placementCheckName)`. `os.Remove(dir)` (a leftover from an earlier run is empty: the unit's restart killed its members); then `os.Mkdir(dir, 0o755)`; `defer` a removal that waits for the probe to be reaped first. |
| +3. `m.applyLimit("placement check", dir, TierDefault, true)`; an error is returned. |
| +4. `cmd := exec.Command(shimExe, ProbeVerb)` with `Env = []string{}` and `Stdout` a buffer. `m.startViaShim(dir, cmd, nil)`. Any error is returned (a placement error or not: either means the check failed). |
| +5. Wait for the probe with `ctx` (kill and reap on `ctx.Done()`). It must exit 0 and print exactly `expected` (the same string `startViaShim` computed: `dir` without `/sys/fs/cgroup`). |
| +6. `memory.max` in `dir` must read back `TierDefault.MemoryMax`. |
| +7. Remove `dir`; a removal failure is logged, not returned. |
| + |
| +`swapArming`: after `swapDelegation` passes (`rollout.go:160`), call `placementCheck(eb.Cgroups)`. On error: `log.Printf("agent: REFUSING %s=overlap (%s): a test placement into a scratch cgroup failed (%v), so generations could not be started inside their limits; this box stays on today's stop-then-wake paths", rollout.SwapEnv, rolloutPlacementFailed, err)`, `declineRollout(rolloutPlacementFailed)`, return false. A box that declines this way honours failure records exactly as the other declined codes do (`honourDeclinedFailures`); check that path keys on "declined", not on a list of codes, and if it keys on a list, add the code there. |
| + |
| +`rolloutwords`: `"cgroup-placement-failed": unavailableDeclined` in `unavailablePhrases`, and the code in `UnavailableDeclines`'s case list. `status/snapshot.go`: name the code in the comment that lists the declined codes. Grep for every other list of declined codes (`grep -rn '"host-claims-on"' --include=*.go .`) and add the new code to each, with a test where one exists. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +`internal/host/rollout_test.go`: |
| + |
| +```go |
| +func TestSwapArmingDeclinesWhenPlacementFails(t *testing.T) { |
| + // build eb as TestSwapArming's armed case does (rollout_test.go:526) |
| + orig := placementCheck |
| + placementCheck = func(*cgroup.Manager) error { return errors.New("probe exited before exec") } |
| + t.Cleanup(func() { placementCheck = orig }) |
| + if swapArming(overlapEnv(pub), eb) { |
| + t.Fatal("armed with a failed placement check") |
| + } |
| + if rolloutUnarmed() != rolloutPlacementFailed { |
| + t.Fatalf("declined as %q", rolloutUnarmed()) |
| + } |
| +} |
| +``` |
| + |
| +and make the existing armed-case tests (`rollout_test.go:526`, `:568` and any other test that expects `swapArming` true) stub `placementCheck` to nil-returning, in a helper they share. A flag-off `swapArming` must not call `placementCheck` (assert with a stub that fails the test). |
| + |
| +`internal/rolloutwords`: `Unavailable("cgroup-placement-failed")` equals the declined phrase and `UnavailableDeclines` is true for it. |
| + |
| +`internal/cgroup/verify_linux_test.go`: ungated: a manager over a temp root answers an error (the temp directory is no cgroup) and leaves no `carlos-placement-check` behind. Gated (`CARLOS_SHIM_KERNEL=1`, `genKernelManager`): `VerifyPlacement` returns nil and the directory is gone afterwards; with `shimExe` pointed at `/bin/true` (restored in Cleanup) it returns an error and the directory is gone. |
| + |
| +- [ ] **Step 2: Run to see them fail.** `GOCACHE=$TMPDIR/gocache go test ./internal/host ./internal/rolloutwords ./internal/cgroup -run 'Placement|Unavailable|SwapArming' -count=1` — build errors. |
| + |
| +- [ ] **Step 3: Implement.** |
| + |
| +- [ ] **Step 4: Run.** The same packages sandboxed (host tests may need unsandboxed `TMPDIR=/tmp/k` for sockets: use that if a socket error appears), plus the gated verify tests in a delegated user unit as in Task 1. Paste real output. |
| + |
| +- [ ] **Step 5: Commit** — `"host: arming verifies a real shim placement; decline cgroup-placement-failed (#272)"`. |
| + |
| +--- |
| + |
| +### Task 4: The shim under the edge unit's hardening (opus) |
| + |
| +**Files:** |
| +- Create: `internal/cgroup/edge_hardening_kernel_linux_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1 (`startViaShim`), Task 2 (`StartChildGated` on the shim), Task 3 (`VerifyPlacement`), `genKernelManager` (`gen_kernel_linux_test.go:27`). |
| +- Produces: `TestEdgeHardeningPlacement` (the driver) and `TestEdgeHardeningInner` (runs inside the unit). |
| + |
| +**Behaviour.** The driver, gated by `CARLOS_EDGE_HARDENING_KERNEL=1` and run outside any unit: |
| +1. Read `../host/units/carlos-edge.service` (go test runs in the package directory). Collect every `Key=value` line in `[Service]`. |
| +2. Sort each key into one of two explicit tables in the test: **applied** (passed as `-p Key=value` to `systemd-run --user`) and **excluded** (with a one-line reason each). A `[Service]` key in neither table fails the test, so a new hardening line forces a decision here. |
| +3. The applied table must contain at least: `RestrictNamespaces`, `SystemCallFilter`, `SystemCallErrorNumber`, `SystemCallArchitectures`, `NoNewPrivileges`, `MemoryDenyWriteExecute`, `LockPersonality`, `RestrictRealtime`, `RestrictAddressFamilies`, `ProtectProc`, `ProtectClock`, `ProtectHostname`, `ProtectKernelTunables`, `ProtectKernelModules`, `ProtectKernelLogs`, `PrivateDevices`, `ProtectSystem`, `TasksMax`. The test asserts each is present in the unit file and applied. If one of these cannot be applied in a user unit on DOCS-T1, move it to excluded with the real `systemd-run` error as its reason, and say so in the report: Task 7 carries it into the dev re-check. |
| +4. Expected exclusions, each with its reason: `ExecStart`, `ExecReload`, `Restart`, `RestartSec`, `EnvironmentFile` (process control, not sandbox); `User`, `Group`, `AmbientCapabilities`, `CapabilityBoundingSet` (a user manager cannot change identity or capabilities); `ReadWritePaths` (names box paths absent here); `Slice`, `OOMPolicy`, `Delegate` (the test passes its own `Delegate=yes`); `ProtectHome`, `PrivateTmp` only if the test binary cannot be made visible inside them (prefer keeping them and making the binary visible with `-p BindReadOnlyPaths=<binary>`; say which in the report). |
| +5. Run `systemd-run --user --wait --pipe --collect -p Delegate=yes <applied...> -E CARLOS_EDGE_HARDENING_INNER=1 -E CARLOS_SHIM_KERNEL=1 -E CARLOS_GEN_KERNEL=1 <binary> -test.run '^TestEdgeHardeningInner$' -test.v -test.count=1` and fail unless the inner run passes; stream its output to the test log. |
| + |
| +The inner test, gated by `CARLOS_EDGE_HARDENING_INNER=1`: |
| +1. **Positive control, the filter is on:** `unix.Syscall(unix.SYS_CLONE3, 0, 0, 0)` answers `ENOSYS`. And a plain `exec.Cmd` with `SysProcAttr{UseCgroupFD: true, CgroupFD: <a scratch cgroup fd>}` fails to start with `ENOSYS`. If either succeeds the test fails: the unit is not applying `RestrictNamespaces`, and nothing else here would prove anything. |
| +2. **Placement succeeds:** `StartChildGated("hardening.example", GenChild(1), lim, lim, build, nil)` with `lim := Limit{MemoryMax: 64 << 20}` and a tenant that prints its cgroup then waits on stdin: the cgroup is `.../carlos-exec@hardening.example/g1`, and the PID the tenant prints is `cmd.Process.Pid`. |
| +3. **The limit applies:** the existing `hog` child mode (`gen_kernel_linux_test.go:65`) started the same way into `g2` is OOM-killed: `memory.events` in `g2` shows `oom_kill` of at least 1, the process dies by SIGKILL, and `g1`'s tenant is still alive. |
| +4. **Arming would arm:** `m.VerifyPlacement(ctx)` returns nil. |
| + |
| +- [ ] **Step 1: Write the test** (both functions, the two tables). |
| +- [ ] **Step 2: Run it against a branch where it must fail**, to prove it can: temporarily make `placeStart` use the old clone3 start (a local, uncommitted edit, `setInto` + `cmd.Start()`); the inner step 2 must fail with `ENOSYS`/`ErrLimitNotApplied`. Paste that output, then revert the edit (`git diff` must show only the new test file). |
| +- [ ] **Step 3: Run it for real**: |
| +```bash |
| +GOCACHE=$TMPDIR/gocache go test -c -o /tmp/k/cgroup.test ./internal/cgroup && \ |
| +cd internal/cgroup && CARLOS_EDGE_HARDENING_KERNEL=1 TMPDIR=/tmp/k /tmp/k/cgroup.test -test.run '^TestEdgeHardeningPlacement$' -test.v -test.count=1; rm -f /tmp/k/cgroup.test |
| +``` |
| +(unsandboxed). Expected: PASS, with the applied and excluded tables printed. Paste the real output. |
| +- [ ] **Step 4: Commit** — `"cgroup: kernel test, shim placement under carlos-edge's sandbox lines (#272)"`. |
| + |
| +--- |
| + |
| +### Task 5: The probe app accepts the platform's argv (sonnet) |
| + |
| +**Files:** |
| +- Create: `cmd/zddprobe/main.go`, `cmd/zddprobe/main_test.go` |
| +- Modify: `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md:841-895` (section "1. Build a probe app") |
| + |
| +**Interfaces:** none shared. |
| + |
| +**Behaviour.** The platform runs an exec-backed instance as `<binary> --socket <sock> --db <db>` (`internal/activator/backend_exec.go:2147`, `:2335`, `internal/activator/claim_snapshot.go:221`), and `carlos deploy`'s vet boots it as `<binary> -socket <sock>` (`internal/vet/vet.go:135`). The probe accepts both: `--socket` (required) and `--db` (accepted and ignored: the probe keeps no data). Everything else is the roll notes' program unchanged: serve HTTP on the unix socket, every request sleeps 50 ms then answers `ok <version>\n`, and SIGTERM or SIGINT drains with a 15 s shutdown. `version` is set by `-ldflags "-X main.version=..."`. Put the server in `func run(args []string, stop <-chan os.Signal) error` so `main` is `if err := run(os.Args[1:], sigs); err != nil { fmt.Fprintln(os.Stderr, err); os.Exit(1) }`. |
| + |
| +- [ ] **Step 1: Write the failing test** — `main_test.go` launches the test binary itself as the probe, the way the platform launches it: |
| + |
| +```go |
| +func TestMain(m *testing.M) { |
| + if os.Getenv("ZDDPROBE_AS_MAIN") == "1" { |
| + main() |
| + os.Exit(0) |
| + } |
| + os.Exit(m.Run()) |
| +} |
| + |
| +// launch starts this test binary as the probe with argv, as the platform |
| +// starts an instance: argv only, no shell. |
| +func launch(t *testing.T, argv ...string) *exec.Cmd { |
| + cmd := exec.Command(os.Args[0], argv...) |
| + cmd.Env = append(os.Environ(), "ZDDPROBE_AS_MAIN=1") |
| + cmd.Stderr = os.Stderr |
| + if err := cmd.Start(); err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { _ = cmd.Process.Kill(); _, _ = cmd.Process.Wait() }) |
| + return cmd |
| +} |
| + |
| +func TestProbeServesWithThePlatformsRunArgv(t *testing.T) { |
| + dir := t.TempDir() |
| + sock, db := filepath.Join(dir, "app.sock"), filepath.Join(dir, "app.db") |
| + cmd := launch(t, "--socket", sock, "--db", db) // backend_exec.go:2335 |
| + body := getOverUnix(t, sock, "/healthz") // retries until the socket answers, 10 s cap |
| + if !strings.HasPrefix(body, "ok ") { |
| + t.Fatalf("body %q", body) |
| + } |
| + // SIGTERM drains: an in-flight request still completes, then exit 0. |
| + ... |
| + if err := cmd.Wait(); err != nil { |
| + t.Fatalf("exit after SIGTERM: %v", err) |
| + } |
| +} |
| + |
| +func TestProbeServesWithVetsArgv(t *testing.T) { ... launch(t, "-socket", sock) (vet.go:135) answers ok ... } |
| +``` |
| + |
| +`getOverUnix` is a small helper with an `http.Client` whose transport dials the unix socket. The drain check: start a request, send SIGTERM 10 ms later, and assert the request answers 200 and `cmd.Wait()` returns nil. |
| + |
| +- [ ] **Step 2: Run to see it fail:** `TMPDIR=/tmp/k GOCACHE=$TMPDIR/gocache go test ./cmd/zddprobe -count=1` (socket paths: if the sandbox refuses the socket, rerun unsandboxed with `TMPDIR=/tmp/k`). Expected: build failure. |
| +- [ ] **Step 3: Implement `main.go`.** Then replace the roll notes' heredoc with: build from a platform checkout at the release commit, `for v in zdd-1 zdd-2 zdd-3 zdd-4; do CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -ldflags "-X main.version=$v" -o "$TMPDIR/zdd-probe/probe-$v" ./cmd/zddprobe; done`, and a sentence that the probe takes the platform's `--socket` and `--db` (the round-2 dev check found the old one exited at once on `--db`). Keep the rest of "The probe" as it is, and fix its `cd`/path lines to match. |
| +- [ ] **Step 4: Run** the test (paste output), `go vet ./cmd/zddprobe`, gofmt. |
| +- [ ] **Step 5: Commit** — `"cmd/zddprobe: the roll notes' probe, taking the platform's --socket and --db (#272)"`. |
| + |
| +--- |
| + |
| +### Task 6: TestStrictKernel's managers, and reset-failed after a generation stop (sonnet) |
| + |
| +**Files:** |
| +- Modify: `internal/cgroup/strict_kernel_linux_test.go:160`, `:190` |
| +- Modify: `internal/instancectl/gen.go:224-238` (`stopGen`) and its test file (find it with `grep -rln "stopGen\|StopGen" internal/instancectl/*_test.go`) |
| + |
| +**Behaviour.** |
| + |
| +(a) `TestStrictKernel` full mode builds second managers with no `logf` (`&Manager{root: m.root}` at `:160` and `:190`); production always sets one, and `freshenOne` (`recreate_linux.go:156`) logs through it, so the test panics. Give both `logf: t.Logf`. No production change. |
| + |
| +(b) After `stopGen`'s stop or kill succeeds, and before `s.removeGenFiles(unit)`, run `s.Systemctl(ctx, "reset-failed", unit)`. An answer containing `not loaded` is ignored; any other error is logged with `s.logf("instance-control: reset-failed %s: %v", unit, err)` and the stop still answers 204 (Ruling 10). `reset-failed` is already one of the control service's verbs (`server.go:641`). |
| + |
| +- [ ] **Step 1: Write the failing test** for (b): with the recording fake `Systemctl` the instancectl tests already use, a `/stop?gen=N` (and `&kill=1`) records `reset-failed <unit>` after the stop or kill; with `reset-failed` answering an error, the response is still 204 and a log line was written; a stop that fails records no `reset-failed`. |
| +- [ ] **Step 2: Run to see it fail:** `GOCACHE=$TMPDIR/gocache go test ./internal/instancectl -run StopGen -count=1` (unsandboxed with `TMPDIR=/tmp/k` if sockets are refused). |
| +- [ ] **Step 3: Implement (a) and (b).** |
| +- [ ] **Step 4: Run** the instancectl package, then TestStrictKernel's full mode for real, in a delegated user unit with no RestrictNamespaces (the strict path keeps clone3, Ruling 3): |
| +```bash |
| +GOCACHE=$TMPDIR/gocache go test -c -o /tmp/k/cgroup.test ./internal/cgroup && mkdir -p /tmp/k/strictproof && \ |
| +systemd-run --user --wait --pipe --collect -p Delegate=yes -E TMPDIR=/tmp/k -E CARLOS_STRICT_KERNEL=full -E CARLOS_STRICT_PROOF_DIR=/tmp/k/strictproof \ |
| + /tmp/k/cgroup.test -test.run '^TestStrictKernel$' -test.v -test.count=1; rm -rf /tmp/k/cgroup.test /tmp/k/strictproof |
| +``` |
| +Expected: all five subtests PASS (round 2 saw this pass with the same fix in scratch). Paste real output. If the mode needs other variables, read `strict_kernel_linux_test.go:30-60` and set them; say which in the report. |
| +- [ ] **Step 5: Commit** — `"instancectl: reset-failed a stopped generation unit; strict kernel test managers log (#272)"`. |
| + |
| +--- |
| + |
| +### Task 7: Roll notes (sonnet) |
| + |
| +**Files:** |
| +- Modify: `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` |
| + |
| +**Behaviour.** Add a section `### Roll notes from zdd-flag-readiness-3` after `### Roll notes from zdd-flag-readiness` (`:336`), in the file's plain style (full stops, no em dashes), covering: |
| + |
| +1. The four round-2 blockers and where each is closed (Tasks 1 to 6 by commit; the controller passes the commit list). |
| +2. The shim, in two sentences, and Paul's ruling that the edge unit's hardening stays. |
| +3. **Behaviour change on roll, flag off:** every exec child now starts through the shim. What the agent log shows on a healthy box (no `placement before exec failed` lines, no PID-move lines), and what a failing shim looks like (`placement before exec failed (...) — using the PID-move fallback for this spawn`): the box still serves, as today. |
| +4. The new decline code `cgroup-placement-failed`, its log line, and that a console older than the box shows the unknown-code phrase until consoles roll. |
| +5. Host claims: the strict path keeps clone3, so under the shipped unit a claims-on box's strict launches answer `ErrStrictUnsupported`; that is now a precondition of claims-on swaps. |
| +6. **Dev re-check for Paul** (replacing round 2's blocked steps): roll; flag-off wakes answer 200 with no fallback lines; set `CARLOS_SWAP=overlap` and restart; the log shows `overlap swaps armed` and no `REFUSING`; the probe (built from `cmd/zddprobe`) under load through zdd-1 → zdd-3 with 0 failed; the hold control; a generation unit stopped by the control service is not in `systemctl --failed`; and any edge-unit line Task 4 had to exclude (the controller passes the list), checked on the real unit. |
| +7. The DOCS-T1 kernel-test recipe (`systemd-run --user ... -p Delegate=yes`) and the Task 4 command. |
| + |
| +Update the "Still blocks flag-on" lists that name the four round-2 blockers so they point to this section. |
| + |
| +- [ ] **Step 1: Write the section and the list edits.** |
| +- [ ] **Step 2: Check names against the code:** every identifier, log line and flag the section quotes is grepped in the tree; paste the grep output in the report. |
| +- [ ] **Step 3: Commit** — `"roll notes: zdd-flag-readiness-3 (#272)"`. |