amadan
roll notes: readiness-3 review fixes, the activator kernel-test recipe (#272)
- Finding 5: the general kernel-test recipe makes its own directory with mktemp under $TMPDIR and removes only that, and says to rerun the shim tests under RestrictNamespaces=yes without TestGenStrictKernel. - Finding 4: the arming line says the 4 s bounds the check's waits, not a reap held by stalled kernel I/O. - Dev re-check step 7: runnable commands for TestClaimContainmentCleanupRespawn and TestGenerationStartRefusedBesideADirectParentMember on the dev box (a delegated system unit run as the user, for io), with the strict_primitive subtest run apart and without RestrictNamespaces, since it calls (*Prepared).Start, which keeps clone3 and is expected to fail there by Ruling 3. The Real-box checks entries point at it. - The new operator line for a start whose context ended during placement, and the shim's restoring of the agent's ignored signals.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+102
−16
docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md+102 −16
diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md| index dec929058..996bfe31c 100644 |
| --- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| +++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| @@ -583,6 +583,25 @@ review and CI come after these notes. Nothing here was run on a real box. |
| and the refusal is the box's condition, not a failure recorded against the |
| build. The "armed box without `CLONE_INTO_CGROUP`" note above is replaced by |
| this one. |
| + |
| + A start whose own context ended while its shim was being placed (a seed's |
| + wake that timed out or was stopped) is not a shim fault. It logs: |
| + |
| + ```text |
| + cgroup: <host>: the start's context ended during placement (<why>); nothing ran |
| + ``` |
| + |
| + and answers the context's own error, as a direct start would. Flat or |
| + generation, it does not fall back and is not the box's condition. The |
| + member's wake event reads as such a wake always read (`wake failed: seeding |
| + <host>: context canceled`). |
| +- **The shim keeps the agent's ignored signals.** A tenant started through |
| + the shim has the same ignored signals as one started directly: the agent |
| + sends its current set (`SigIgn` in `/proc/self/status`) with the spawn, and |
| + the shim ignores those again just before it execs the tenant. Without |
| + that, the shim's own Go runtime would have reset every ignored signal but |
| + `SIGHUP` and `SIGINT` to its default. This changes nothing on a box whose |
| + agent ignores no other signal. |
| - **The new decline code `cgroup-placement-failed`.** At arm time (the agent's |
| start with `CARLOS_SWAP=overlap`) the agent starts a real shim placement |
| into a scratch cgroup named `carlos-placement-check`, under its delegated |
| @@ -595,9 +614,15 @@ review and CI come after these notes. Nothing here was run on a real box. |
| The code is in the instance's JSON record (`rollout.unavailable`) and uses |
| the approved declined wording, so there is no new copy. A box whose check |
| - fails declines in at most about 4 s, inside the 20 s `carlos system update` |
| - gate. A box that passes logs `agent: overlap swaps armed (CARLOS_SWAP=overlap)` |
| - as before. |
| + fails declines in about 4 s, inside the 20 s `carlos system update` gate. |
| + The 4 s bounds the check's own waits. After them the agent kills and |
| + reaps what it started, which takes milliseconds when the kernel ends a |
| + killed process promptly. It does not when that process is stuck in |
| + uninterruptible kernel I/O: SIGKILL ends it only when the I/O returns, and |
| + the decline waits for that. So on a box whose disk or cgroupfs is |
| + stalled, the decline can take longer than 4 s, and the update gate can |
| + time out. A box that passes logs |
| + `agent: overlap swaps armed (CARLOS_SWAP=overlap)` as before. |
| *Mixed versions.* Consoles and CLIs roll after boxes, so for a while a |
| rolled box can decline with a code the others do not know. A console older |
| @@ -650,23 +675,80 @@ review and CI come after these notes. Nothing here was run on a real box. |
| 7. Run the two activator kernel tests that now start through the shim and |
| could not run on DOCS-T1, because user delegation there has no `io` |
| controller: `TestClaimContainmentCleanupRespawn` and |
| - `TestGenerationStartRefusedBesideADirectParentMember` |
| - (`CARLOS_CLAIM_SPAWN_KERNEL=1`, as "Real-box checks" says). They are owed |
| - on the dev box. |
| + `TestGenerationStartRefusedBesideADirectParentMember`. They are owed on |
| + the dev box. Both skip unless `CARLOS_CLAIM_SPAWN_KERNEL=1`, and both |
| + call `cgroup.Setup`, which needs a delegated cgroup offering `cpu`, |
| + `memory`, `io` and `pids`; without one they fail with |
| + `missing delegated cgroup`. |
| + |
| + Build the binary for the dev box from the checkout root on DOCS-T1, and |
| + copy it to the dev box by the route the roll uses for binaries: |
| + |
| + ```bash |
| + GOOS=linux GOARCH=arm64 go test -c -o activator.test ./internal/activator |
| + ``` |
| + |
| + On the dev box, in a fresh directory under `/tmp` (the activator's |
| + socket paths need a short root), first the two shim checks, with the |
| + shipped unit's `RestrictNamespaces=yes`: |
| + |
| + ```bash |
| + d=$(mktemp -d /tmp/zddk.XXXXXX) |
| + install -m 0755 activator.test "$d/" |
| + sudo systemd-run --wait --pipe --collect -p Delegate=yes -p User="$(id -un)" \ |
| + -p RestrictNamespaces=yes -E TMPDIR="$d" -E CARLOS_CLAIM_SPAWN_KERNEL=1 \ |
| + "$d/activator.test" -test.run '^TestGenerationStartRefusedBesideADirectParentMember$' -test.v -test.count=1 |
| + sudo systemd-run --wait --pipe --collect -p Delegate=yes -p User="$(id -un)" \ |
| + -p RestrictNamespaces=yes -E TMPDIR="$d" -E CARLOS_CLAIM_SPAWN_KERNEL=1 \ |
| + "$d/activator.test" -test.run '^TestClaimContainmentCleanupRespawn$/^Kill' -test.v -test.count=1 |
| + ``` |
| + |
| + The pass is `--- PASS` for `TestGenerationStartRefusedBesideADirectParentMember` |
| + and its `sibling_control` and `recovery_control`, and for |
| + `KillHost_then_respawn` and `KillSidecar_then_respawn`, with no |
| + `--- SKIP`. These use a system unit run as you (`sudo`, `-p User=`) |
| + because systemd delegates `io` to it. Where the user manager offers |
| + `io` too, the same commands as `systemd-run --user`, without `sudo` and |
| + `-p User=`, do as well; where it does not, they fail with |
| + `missing delegated cgroup`. |
| + |
| + Then the strict subtest, on its own and **without** |
| + `RestrictNamespaces`: |
| + |
| + ```bash |
| + sudo systemd-run --wait --pipe --collect -p Delegate=yes -p User="$(id -un)" \ |
| + -E TMPDIR="$d" -E CARLOS_CLAIM_SPAWN_KERNEL=1 \ |
| + "$d/activator.test" -test.run '^TestClaimContainmentCleanupRespawn$/^strict_primitive$' -test.v -test.count=1 |
| + rm -rf "$d" |
| + ``` |
| + |
| + `strict_primitive` calls `(*Prepared).Start`, the strict claims path, |
| + which keeps its `clone3` placement by Ruling 3. Under |
| + `RestrictNamespaces=yes` `clone3` answers ENOSYS, so that subtest is |
| + expected to fail there (`cgroup.ErrStrictUnsupported`). That is the |
| + claims-on precondition above, not a regression, and it is why it runs |
| + without the line. Run as one `-test.run '^TestClaimContainmentCleanupRespawn$'` |
| + under `RestrictNamespaces=yes`, the test fails on that subtest alone. |
| - **Kernel tests on DOCS-T1.** They skip unless asked for, and they need a |
| - delegated unit. Build the test binary, then run it in a transient user unit: |
| + delegated unit. From the checkout root, build the test binary into a |
| + directory of its own, run it in a transient user unit, and remove that |
| + directory and nothing else: |
| ```bash |
| - mkdir -p "$TMPDIR/k" |
| - go test -c -o "$TMPDIR/k/cgroup.test" ./internal/cgroup |
| + d=$(mktemp -d "$TMPDIR/zddk.XXXXXX") |
| + go test -c -o "$d/cgroup.test" ./internal/cgroup |
| systemd-run --user --wait --pipe --collect -p Delegate=yes \ |
| - -E TMPDIR="$TMPDIR/k" -E CARLOS_GEN_KERNEL=1 -E CARLOS_SHIM_KERNEL=1 \ |
| - "$TMPDIR/k/cgroup.test" -test.run '<regex>' -test.v -test.count=1 |
| + -E TMPDIR="$d" -E CARLOS_GEN_KERNEL=1 -E CARLOS_SHIM_KERNEL=1 \ |
| + "$d/cgroup.test" -test.run '<regex>' -test.v -test.count=1 |
| + rm -rf "$d" |
| ``` |
| - (Remove `$TMPDIR/k` afterwards. User delegation offers `cpu memory pids`, |
| - not `io`.) Task 4's edge-hardening test is a |
| - driver that starts its own units, so it runs outside any unit: |
| + Run the shim, generation and placement-check tests a second time with |
| + `-p RestrictNamespaces=yes`, the shipped unit's line, and leave |
| + `TestGenStrictKernel` out of that run: it is the strict path, which keeps |
| + `clone3` (Ruling 3). User delegation offers `cpu memory pids`, not `io`. |
| + Task 4's edge-hardening test is a driver that starts its own units, so it |
| + runs outside any unit: |
| ```bash |
| mkdir -p /tmp/k/zddr3 && go test -c -o /tmp/k/zddr3/cgroup.test ./internal/cgroup |
| @@ -729,13 +811,16 @@ the unit tests cannot stand in for a real cgroupfs. |
| - Readiness round 3's dev re-check, replacing round 2's blocked steps: see |
| "Roll notes from zdd-flag-readiness-3". It includes the two activator |
| kernel tests below, which now place through the shim and need the dev box's |
| - `io` controller. |
| + `io` controller. Its step 7 has the commands, and runs the strict subtest |
| + of `TestClaimContainmentCleanupRespawn` apart, without |
| + `RestrictNamespaces=yes`. |
| - Round 1's kernel checks, still as listed next. |
| `TestClaimContainmentCleanupRespawn` (now the |
| KillHost and KillSidecar respawn) and |
| `TestGenerationStartRefusedBesideADirectParentMember` on the dev box |
| - (arm64, `CARLOS_CLAIM_SPAWN_KERNEL=1`, delegated user service). Also check |
| + (arm64, `CARLOS_CLAIM_SPAWN_KERNEL=1`, a delegated unit with `io`; the |
| + commands are step 7 of the readiness-3 dev re-check). Also check |
| whether children of a killed kept cgroup, and cgroups systemd kills on |
| `carlos-edge` stop, are affected. |
| - The plan 2 kernel suite rerun. |
| @@ -1134,6 +1219,7 @@ Added by zdd-flag-readiness-3 (checked on that branch): `__cgexec` |
| `cgroup-placement-failed`, `cgroup.ErrLimitNotApplied`, |
| `cgroup.ErrStrictUnsupported`, `PrepareStrict`, `PrepareStrictParent`, the |
| agent log lines `placement before exec failed`, `generation start refused`, |
| +`the start's context ended during placement`, |
| `REFUSING CARLOS_SWAP=overlap (<code>)` and `overlap swaps armed`, the control |
| service's `reset-failed` log line, `classifyRollout` and `follows` in |
| `deploywatch.go`, `cmd/zddprobe`, and the kernel-test variables |