amadan
roll notes: zdd-flag-readiness-3 (#272)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+199
−8
docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md+199 −8
diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md| index 6d142cff1..43d591d88 100644 |
| --- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| +++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| @@ -143,7 +143,9 @@ dormant and `CARLOS_SWAP` must stay unset until everything below is closed. |
| Two defects blocked the flag on the dev box. Both are fixed in code on |
| zdd-flag-readiness, pending merge. Their tests are kernel-gated and skip on |
| the build machine, so neither fix has been seen working on a real kernel. |
| -The dev-box rerun is owed. |
| +The dev-box rerun is owed. Both kernel tests now start through the shim, and |
| +zdd-flag-readiness-3 notes that they cannot run on DOCS-T1 (see "Roll notes |
| +from zdd-flag-readiness-3", dev re-check step 7). |
| - The legacy-parent guard (2c5adbe8). Code fixed. Its kernel test |
| `TestGenerationStartRefusedBesideADirectParentMember` needs the rerun. |
| @@ -260,6 +262,16 @@ commit on the branch, "copy: readiness round 2's seven strings, approved by |
| Paul unchanged; rev-18 amendment approved"). The code for the other items |
| is closed on the branch, pending merge. |
| +Readiness round 2's dev check (2026-10-13) then found four blockers, and a |
| +fifth defect that was only untidy. All of them are closed in code on |
| +zdd-flag-readiness-3 (`ff4b814a` to `496fb6ed`), pending merge, and they still |
| +block the flag until Paul's dev re-check passes. They are: the edge unit's |
| +`RestrictNamespaces=yes` made every generation placement fail (the exec-shim); |
| +arming did not check placement (`cgroup-placement-failed`); the probe app |
| +rejected `--db` (`cmd/zddprobe`); and `TestStrictKernel` had a defect. The |
| +untidy one is failed generation units lingering in `systemctl --failed`. |
| +Commits and the re-check are in "Roll notes from zdd-flag-readiness-3". |
| + |
| - Nested live-link race on a declined box: needs an expected identity on |
| `Control.Start`, root-side (task 9). Closed on zdd-flag-readiness-2 |
| (`903ed7ca`, `c910d020`), pending merge: a declined box pins the live link |
| @@ -413,7 +425,11 @@ by 076bd9c0 (final review round 1), pending merge. |
| and any copy made by hand. Each of those must drop the `book` key as |
| above before the flag goes on. |
| - **An armed box without `CLONE_INTO_CGROUP` refuses every generation |
| - start** (`98593909`, `8530210e`). A generation start must be placed in its |
| + start** (`98593909`, `8530210e`). Superseded on zdd-flag-readiness-3: the |
| + Manager no longer uses `clone3` and places through the shim, so the shipped |
| + unit no longer causes this refusal. See "Roll notes from |
| + zdd-flag-readiness-3". The rest holds, with "the shim cannot place" in |
| + place of "clone3 is unavailable". A generation start must be placed in its |
| cgroup before exec. If clone3 placement is unavailable (the fallback has |
| latched, or the cgroup cannot be opened), the start is refused with |
| `cgroup.ErrLimitNotApplied` and is never done by moving the PID after |
| @@ -502,6 +518,164 @@ by 076bd9c0 (final review round 1), pending merge. |
| restart it. Use `carlos system update` (main's system apply restarts it, |
| 0bb57a31), or restart it by hand after the old script. |
| +### Roll notes from zdd-flag-readiness-3 |
| + |
| +Branch `zdd-flag-readiness-3`, from `main` at `34a4328f`, on amadan |
| +discussion #272. Built and task-reviewed on the branch: not merged, not |
| +released, not rolled, and only Paul closes it on #272. The whole-branch |
| +review and CI come after these notes. Nothing here was run on a real box. |
| + |
| +- **The four round-2 blockers, and where each is closed.** Paul's dev check |
| + on 2026-10-13 ("Dev-box checks, round 2" on #272) found four things that |
| + stopped the flag. They are closed in code on this branch, pending merge: |
| + 1. `RestrictNamespaces=yes` on `carlos-edge` makes systemd's seccomp filter |
| + answer `clone3` with ENOSYS, so `CLONE_INTO_CGROUP` never worked under |
| + the shipped unit and every generation start was refused. Closed by the |
| + exec-shim (`ff4b814a`, `a34013c6`), by routing every Manager start |
| + through it (`961aed71`), and by a kernel test under the edge unit's |
| + sandbox lines (`891c0fb7`). |
| + 2. Arming did not check that placement works, so a box that could not place |
| + armed anyway and then failed every generation start. Closed by an |
| + arm-time check (`7fbafe7e`, `1f9d6a83`): see the decline code below. |
| + 3. The roll notes' probe app rejected the `--db` argument the platform |
| + passes, so it exited at once. Closed by `cmd/zddprobe` (`59b1895a`). |
| + 4. `TestStrictKernel` had a defect of its own and could not pass in its full |
| + mode. Closed by `496fb6ed`. |
| + |
| + The minor from the same check, failed generation units lingering in |
| + `systemctl --failed` after the control service stopped them, is closed by |
| + `496fb6ed` too: the control service now runs `reset-failed` on a generation |
| + unit after a stop or an abandon-kill that proved it gone. It is best-effort. |
| + A failure is logged as `instance-control: reset-failed <unit>: ...` and the |
| + stop still answers 204. |
| +- **The shim.** A spawn starts `/proc/self/exe __cgexec`. The shim places |
| + itself in the target cgroup, then `execve`s the tenant. The tenant's |
| + path, argv and environment travel over a pipe, never in the shim's own argv |
| + or environment, so the PID the agent tracks is the shim's and then the |
| + tenant's. Paul's ruling (2026-10-13): no line of |
| + `internal/host/units/carlos-edge.service` is relaxed. `RestrictNamespaces=yes` |
| + stays, and so does every other sandbox line. The unit file is not edited |
| + on this branch. The cost is one extra exec of the `carlos` binary per |
| + spawn, measured on DOCS-T1 at about 4.5 ms (median 4.9 ms against 0.42 ms |
| + for a direct start). |
| +- **Behaviour change on roll, flag off.** Every exec child on a rolled box |
| + now starts through the shim, flag on or off, not only generations. Healthy |
| + looks like today: no new log lines at all. In particular the agent log has |
| + no `placement before exec failed` line and no PID-move line. Wakes answer |
| + as before, a few milliseconds slower. |
| + |
| + A failing shim on a flat (flag-off) start logs: |
| + |
| + ```text |
| + cgroup: <host>: placement before exec failed (<why>) — using the PID-move fallback for this spawn |
| + ``` |
| + |
| + and the spawn goes ahead by starting the process and then moving its PID |
| + into the cgroup. That is exactly what every flag-off start does on the |
| + shipped unit today, so the box still serves. Only a placement the shim |
| + proved happened before exec falls back. A start that may have run is |
| + reported started and never retried, so a tenant never runs twice. If you see |
| + the line on a healthy box, stop and look: the shim is not working there and |
| + the box is on the old path. |
| + |
| + A generation start never falls back. The same words end with |
| + `— generation start refused`, the start answers `cgroup.ErrLimitNotApplied`, |
| + and the refusal is the box's condition, not a failure recorded against the |
| + build. The "armed box without `CLONE_INTO_CGROUP`" note above is replaced by |
| + this one. |
| +- **The new decline code `cgroup-placement-failed`.** At arm time (the agent's |
| + start with `CARLOS_SWAP=overlap`) the agent starts a real shim placement |
| + into a scratch cgroup named `carlos-placement-check`, under its delegated |
| + root, within one 4 s budget for the whole check. If that fails the box |
| + declines the flag for every row, nested rows included, and logs: |
| + |
| + ```text |
| + agent: REFUSING CARLOS_SWAP=overlap (cgroup-placement-failed): a test placement into a scratch cgroup failed (<why>), so generations could not be started inside their limits; this box stays on today's stop-then-wake paths |
| + ``` |
| + |
| + The code is in the instance's JSON record (`rollout.unavailable`) and uses |
| + the approved declined wording, so there is no new copy. A box whose check |
| + fails declines in at most about 4 s, inside the 20 s `carlos system update` |
| + gate. A box that passes logs `agent: overlap swaps armed (CARLOS_SWAP=overlap)` |
| + as before. |
| + |
| + *Mixed versions.* Consoles and CLIs roll after boxes, so for a while a |
| + rolled box can decline with a code the others do not know. A console older |
| + than the box shows the unknown-code phrase for `cgroup-placement-failed`. It |
| + treats an unknown code as not declined, so it cannot offer or sign a host |
| + retry for that box until it is updated. An older `carlos` CLI's |
| + `deploy --watch` does the same: it does not follow the box as declined, so |
| + it shows that box as pending until the watch times out (`classifyRollout` |
| + and `follows`, `deploywatch.go`). The deploy itself is unaffected. Update the |
| + console and the CLI and ask again. |
| +- **Claims-on swaps: a new precondition.** The strict claims path |
| + (`PrepareStrict`, `PrepareStrictParent`, `(*Prepared).Start`) keeps its |
| + `clone3` placement, by Paul's ruling. Under the shipped unit `clone3` |
| + answers ENOSYS, so on a box with host claims on every strict launch answers |
| + `cgroup.ErrStrictUnsupported`. It is refused, never run uncapped. Claims are |
| + off everywhere today and a claims-on box declines arming (`host-claims-on`), |
| + so nothing changes now. But claims-on swaps, a future feature, cannot work |
| + until the strict path gets the shim too. Count that as a precondition of |
| + them. |
| +- **Dev re-check for Paul, replacing round 2's blocked steps.** Needs Paul's |
| + approval, naming the box, as Gate E does. In order: |
| + 1. Roll the build to the dev box with the flag still off. Wake an |
| + instance. It answers 200, and the agent log (`journalctl -u carlos-edge`) |
| + has no `placement before exec failed` line. |
| + 2. Set `CARLOS_SWAP=overlap` in `/etc/carlos/host.env` and restart |
| + `carlos-edge`, as Gate E does. The log shows |
| + `agent: overlap swaps armed (CARLOS_SWAP=overlap)` and no `REFUSING` |
| + line. If it shows `(cgroup-placement-failed)`, stop and read the cause in |
| + the line. |
| + 3. Run the probe (built from `cmd/zddprobe`, see "The probe") under load |
| + through `zdd-1` to `zdd-3`. The pass is 0 failed requests and Gate E's |
| + other conditions. |
| + 4. Run the probe's `hold` control. It must show the pause. |
| + 5. After the swaps, `systemctl --failed` on the dev box lists no |
| + `carlos-instance-gen@` unit that the control service stopped. |
| + 6. Check on the real unit the edge-unit lines the kernel test could not. |
| + Task 4's test ran in a user unit on DOCS-T1, not the shipped system unit. |
| + Its report marks these `untested` there: `ProtectProc`, `ProtectClock`, |
| + `RestrictRealtime` and `SystemCallArchitectures`. `ProtectSystem` was |
| + observed only as a read-only `/usr`, so `strict` was not told from |
| + `full`. `NoNewPrivileges` is implied by the seccomp lines. These it |
| + excluded, because only the real unit has them: `User`, `Group`, |
| + `AmbientCapabilities`, `CapabilityBoundingSet`, `ReadWritePaths`, `Slice`, |
| + `OOMPolicy`, and `Delegate=cpu memory io pids` (the user manager gives |
| + `cpu memory pids`, no `io`). The user unit also adds a user namespace the |
| + real unit lacks. On dev, read what systemd applied with |
| + `systemctl show carlos-edge -p ProtectProc -p ProtectClock -p RestrictRealtime -p SystemCallArchitectures -p ProtectSystem -p NoNewPrivileges`, |
| + and treat steps 2 and 3 passing as the proof that placement works under |
| + the unit as shipped, as `carlos` with its real `ReadWritePaths` and slice. |
| + 7. Run the two activator kernel tests that now start through the shim and |
| + could not run on DOCS-T1, because user delegation there has no `io` |
| + controller: `TestClaimContainmentCleanupRespawn` and |
| + `TestGenerationStartRefusedBesideADirectParentMember` |
| + (`CARLOS_CLAIM_SPAWN_KERNEL=1`, as "Real-box checks" says). They are owed |
| + on the dev box. |
| +- **Kernel tests on DOCS-T1.** They skip unless asked for, and they need a |
| + delegated unit. Build the test binary, then run it in a transient user unit: |
| + |
| + ```bash |
| + mkdir -p "$TMPDIR/k" |
| + go test -c -o "$TMPDIR/k/cgroup.test" ./internal/cgroup |
| + systemd-run --user --wait --pipe --collect -p Delegate=yes \ |
| + -E TMPDIR="$TMPDIR/k" -E CARLOS_GEN_KERNEL=1 -E CARLOS_SHIM_KERNEL=1 \ |
| + "$TMPDIR/k/cgroup.test" -test.run '<regex>' -test.v -test.count=1 |
| + ``` |
| + |
| + (Remove `$TMPDIR/k` afterwards. User delegation offers `cpu memory pids`, |
| + not `io`.) Task 4's edge-hardening test is a |
| + driver that starts its own units, so it runs outside any unit: |
| + |
| + ```bash |
| + cd internal/cgroup && CARLOS_EDGE_HARDENING_KERNEL=1 TMPDIR="$TMPDIR/k" \ |
| + "$TMPDIR/k/cgroup.test" -test.run '^TestEdgeHardeningPlacement$' -test.v -test.count=1 |
| + ``` |
| + |
| + `TestStrictKernel` takes `CARLOS_STRICT_KERNEL=<mode>` and a delegated unit |
| + with no `RestrictNamespaces`, since the strict path keeps `clone3`. |
| + |
| ### Real-box checks (need Paul's approval; not run by this branch) |
| Readiness round 2 adds these to the list. None of them was run. The kernel |
| @@ -517,9 +691,10 @@ the unit tests cannot stand in for a real cgroupfs. |
| `internal/activator/claimspawn_kernel_linux_test.go`, run by |
| `TestClaimContainmentCleanupRespawn`: a live g1 survives each sidecar |
| respawn and the parent's `cgroup.procs` stays empty. The same checks need |
| - to hold with the |
| - `CLONE_INTO_CGROUP` placement the box really has, and the refusal needs |
| - watching on a kernel where it is unavailable. |
| + to hold with the placement the box really has. Since zdd-flag-readiness-3 |
| + that is the shim, not `CLONE_INTO_CGROUP`, and the refusal to watch for is |
| + `cgroup-placement-failed` at arming or `generation start refused` at a |
| + start. |
| - Task 8's seed and normal-stop kernel control: a forked helper killed by |
| `cgroup.kill` before the lease goes, for a flat seed, a nested seed unit |
| and a flag-off stop. No kernel test of it exists yet. `KillAndEmpty`, the |
| @@ -534,6 +709,10 @@ the unit tests cannot stand in for a real cgroupfs. |
| Include a data-dir copy with the `book` key dropped, and a downgrade onto a |
| schema 3 book to see the refusal. |
| - The flagship probe's gates E, F and G, as before. |
| +- Readiness round 3's dev re-check, replacing round 2's blocked steps: see |
| + "Roll notes from zdd-flag-readiness-3". It includes the two activator |
| + kernel tests below, which now place through the shim and need the dev box's |
| + `io` controller. |
| - Round 1's kernel checks, still as listed next. |
| `TestClaimContainmentCleanupRespawn` (now the |
| @@ -842,12 +1021,13 @@ held longer than a normal response. Run it at Gate E (dev) and Gate F |
| The probe is `cmd/zddprobe` in this repository. It serves on the unix socket, |
| sleeps 50 ms per request so some are always in flight, answers `ok <version>`, |
| -and drains on SIGTERM. It takes the platform's own argv, `--socket <sock> --db |
| -<db>` (it ignores `--db`), and vet's `-socket <sock>`. The first dev check |
| +and drains on SIGTERM. It takes the platform's own argv, `--socket <sock>` and |
| +`--db <db>` (it ignores `--db`), and vet's `-socket <sock>`. The first dev check |
| used a probe that parsed only `-socket`, so it exited at once when the |
| platform launched it with `--db`. |
| -Build it from a platform checkout at the release commit. The binaries go to |
| +Build it from the root of a platform checkout at the release commit |
| +(`cd` there first: the path `./cmd/zddprobe` is relative). The binaries go to |
| scratch: |
| ```bash |
| @@ -932,6 +1112,17 @@ the verbs and flags `carlos system update|status`, `carlos apps set|create|delet |
| `docs/site/changelog.json`, `docs/site/changelog-links.json` and the "Two |
| builds at once" section. |
| +Added by zdd-flag-readiness-3 (checked on that branch): `__cgexec` |
| +(`cgroup.ShimVerb`), `carlos-placement-check`, the decline code |
| +`cgroup-placement-failed`, `cgroup.ErrLimitNotApplied`, |
| +`cgroup.ErrStrictUnsupported`, `PrepareStrict`, `PrepareStrictParent`, the |
| +agent log lines `placement before exec failed`, `generation start refused`, |
| +`REFUSING CARLOS_SWAP=overlap (<code>)` and `overlap swaps armed`, the control |
| +service's `reset-failed` log line, `classifyRollout` and `follows` in |
| +`deploywatch.go`, `cmd/zddprobe`, and the kernel-test variables |
| +`CARLOS_GEN_KERNEL`, `CARLOS_SHIM_KERNEL`, `CARLOS_EDGE_HARDENING_KERNEL`, |
| +`CARLOS_STRICT_KERNEL` and `CARLOS_CLAIM_SPAWN_KERNEL`. |
| + |
| Not checked here (not in this repo, or not verifiable without running |
| something): `roll-console.sh` and `update-platform-binary.sh` (platform-infrastructure); |
| `hack/sync-docs.mjs` and the website's AGENTS.md (carlosframework/website); |