carlos / platform Public

Zero-downtime deploys: owed before CARLOS_SWAP can go on open

Zero-downtime deploys are being merged dormant, and CARLOS_SWAP must stay unset until everything below is closed.

Spec: docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md (rev 17), plans 0–6 on design-zero-downtime-deploys.

On main:

  • plan 1, generations: 8e066e2
  • plan 4, release-side: db8c3f0
  • plan 2, containment: 218b745
  • plan 3, swap: a3bf7d6

Plan 5, the scheduler (zdd-5-scheduler), is being merged. Plan 6, the surfaces, is not started.

Owed

*Correctness, from Astra's plan 5 review (`CODEX-FINDINGS-ZDD-IMPL-5.md`):**

  • N5–N7: gaps in the hold-ran marker. N7 is a disagreement over how it treats an existing failure record.
  • F5: failure-record durability across a failed write followed by a restart.
  • Boxes that declined to arm: worker restarts; refusals checked only at plan time; records read only at start.
  • The 15-item list in round 3, §5.

Carried from plans 2 and 3:

  • With the flag off, a retained wake is proven by its leader's exit only; forked descendants aren't checked (F6, pre-existing).
  • Leftover executable files aren't swept from /run.
  • Claims-backed swaps are refused (P2-9). Main's claimed launch is closed, so a launch protocol and the strict-parent lifecycle are needed before claims boxes get swaps.
  • Strict hosts get no swaps until a claim path exists (P2-8).

Dev-box checks (need Paul's approval to run):

  • D1: daemon-reload cost at titogo-scale unit counts. Above about 1 s per swap means transient units instead of drop-ins.
  • The plan 2 kernel cgroup suite on arm64 with User=carlos.
  • Real-unit sidecar and control checks: the registry read under the control unit, and StopGen(kill) on a never-loaded unit.
  • A guard against legacy cgroup parents that still hold processes.

Surfaces (plan 6):

  • Copy review of all event and refusal wording. That includes the WakePlan.Refuse texts, "restart refused: rolling restarts are unavailable on this box", the refused-wake texts, and main's "sidecar could not be restarted — retrying", which has an em dash and now fires more often.
  • A member-safe wake reason for ErrBuildNotPublished (plan 4 R22).
  • carlos sidecars ls shows ServingVersion.
  • Roll notes: boxes first, then console policy, then dev, then bab/adopt-probe under load with zero failed requests, then the fleet.

Update: plan 5 (the scheduler) is merged DORMANT as d905a67, after merging main d6f66ef3 (#268 lease margin, region-HA). Astra's merge review (CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md): ACCEPT WITH FIXES for a dormant merge, no flag-off defect.

Added to the owed list: four claims-on gaps. Main's RoleLaunch.Root is wired in production, so claims-on launches can execute. Before the flag goes on, either decline arming on boxes with host claims on, or close all four:

  1. Recovery selection and fallback can't be honoured for claimed planned wakes.
  2. A claimed hold launch can lose its execution evidence and outcome fence.
  3. Retained claim processes are missing from the generations file.
  4. Successful claimed wakes can't record their good build or serving version.

Settings plan 2 (promotion records) is merged as 70f8119. Held staging checks authorization inside stageHeld, so every entry point is covered, including main's claim-path MaterializeHeldUnder.

Wiring caution, whoever wires zero-downtime plan 5 Task 9: when movesProbeOwners.Release is connected to MaterializeHeldUnder, it must pass the parsed CARLOS_RELEASE_PUBKEY and the agent's digest ratchet. A missing key or ratchet refuses every stage (fail closed); that is not a bug to "fix" by passing nil.

Added by Paul's ruling on plan 6 Q4 (2026-10-11): carlos instances retry <host> clears a host's failure hold through the unsigned restart-scope log, the same trust as carlos restart. That stays for now. Before the flag goes on anywhere hold mode is used, retries must be signed in the console, like promote --retry. A forged retry in hold mode could otherwise take an instance down.

Also from plan 6: with the flag on, a publish-only deploy's watch can fall back to web polling and wake sleeping instances when an instance record and its fleet snapshot are both unreadable. That blocks flag-on.

Dev-box checks run 2026-10-11 (Paul's approval). Dev edge box only (i-046c7fe43620944bc).

Inventory. carlos 40b6f50c, Debian 13 arm64, kernel 6.12.105, systemd 257, cgroup v2 (cpu io memory pids), 298 units. Edge healthy. No exec instances; three proof routes already answering 502. Claims and swap both off. 18G disk and 660M RAM free.

D1. Median / p90 of 10 daemon-reloads, in ms:

  • baseline: 309 / 364
  • 75 synthetic units, loaded (368 units): 344 / 366
  • 250 synthetic units, loaded (543 units): 424 / 439

Unloaded files cost nothing. Verdict: well under 1 s, so keep drop-ins.

Kernel suite. arm64, User=carlos, Delegate=yes.

  • Pass: TestGenOOMKernel (the hog was OOM-killed in g2 and the incumbent kept answering), TestGenStrictKernel, TestStrictKernel (full, permission-denied, unsupported), stopevidence Reconstruct (20 tests), host ClaimKernel* (8), the hostexec Plan9 test and the full cgroup, stopevidence and instancectl suites.
  • Skips are honest. TestStrictKernelSupervisor wasn't run (it needs an external kill).
  • Fail, 3 of 3 runs: TestClaimContainmentCleanupRespawn. The test is right. On this kernel, once cgroup.kill has been written to a cgroup, every later clone3 CLONEINTOCGROUP into it is SIGKILLed at once. Recreating the cgroup fixes it. Exposed: KillSidecar and KillHost, which kill a cgroup and keep it. Generation kills remove theirs.

Roll. Old road (no dev console login). 40b6f50c → 5da1cc9b. The script doesn't restart instance-control (the edge flagged it stale); I did. All healthy.

Real-unit checks. The real socket admits only the edge's MainPID, so these ran the real instancectl.Server against real systemd in a unit copying the control unit's sandbox.

  • (a) Pass: the registry read through the installed binary works under ProtectSystem=strict (17 ms).
  • (b) Pass: StopGen(kill) on a never-loaded unit answers 204 and creates nothing.
  • (c) Pass: StartGen sets the slice, WorkingDirectory, binds, the /opt/carlos mounts and the 512M limits. Stop removes the drop-in and the exe layout.
  • (d) Fail, the guard is absent: a generation under a populated legacy carlos-exec@ parent starts silently with no memory limit. The strict path refuses.

Step 4. Skipped: steps 2 and 3(d) are not green, and there is no dev console login.

Final state. The edge box runs 5da1cc9b with CARLOS_SWAP unset, 0 failed units and the routes unchanged. bootstrap/carlos is now 5da1cc9b. All scratch is removed.

Flag blockers:

  • the legacy-parent guard;
  • the cgroup.kill respawn kill;
  • the old roll road must restart instance-control.

Follow-up to the dev-box report: both kernel/unit defects (the cgroup.kill respawn kill in KillSidecar/KillHost, and the missing legacy-parent guard) are now tasks in the zdd-flag-readiness round.

Roll note: platform-infrastructure/scripts/update-platform-binary.sh does not restart carlos-instance-control. Main's carlos system update / system apply does (0bb57a31). Use the system-update road, or restart instance-control by hand after the old script.

To look into: the 5da1cc9b arm64 binary is 74 MB against 36.5 MB for 40b6f50c.

Flag-readiness round: built on zdd-flag-readiness (head 13dffeab, CI green), not merged. CARLOS_SWAP stays unset. Astra's final review, round 3: ACCEPT for a dormant merge, REJECT for flag-on. Findings are committed as docs/superpowers/plans/CODEX-FINDINGS-ZDD-READINESS{,-R2,-R3}.md. The plan is docs/superpowers/plans/2026-10-11-zero-downtime-flag-readiness.md. The roll notes' "Before Gate E: owed" section carries the full state.

Ruling (Paul's): claims-on swaps. A box with host claims on now declines to arm. The reason code is host-claims-on, the agent logs REFUSING CARLOS_SWAP=overlap (host-claims-on), and the box stays on today's paths with failure records honoured (cd7de723). The four claims-on gaps are not closed. They become a separate future feature, "claims-on swaps".

Closed on the branch (pending merge)

  • Plan 6 watch finding. An unreadable record never drops a fleet from the completeness check, and a target missing from the list is never URL-polled (f9be4718).
  • N5, N7 and ruling 205 (b402bc0e, 42865b81): - markers keep failure records; - a worker carries no fallback while a marker stands; - the good build boots over a marker only after a signed rollback newer than the hold run.
  • N6 and retry ordering. A persisted book sequence, per-execution markers, and outcomes that began before a retry never undo it (0d6029b3, 3e875ff2). Execution IDs carry a per-process incarnation (1e842f89). A marker-only retry also records its boundary (c327f703).
  • F5. Write-ahead intents, a schema-2 book and startup reconciliation (8836e521). The rollout manager joins in-flight work at shutdown, so a planned restart leaves no intent behind (f46d3142, 4416872e).
  • Ruling 206. An uncertain nested start is a hold failure (c453a798).
  • Declined boxes: - the gate runs at every actual launch: flat, nested, worker, legacy sidecar, claims and move launches; - markers are honoured; - signed retries are applied, and promote --retry and rollback pointers are read.

Commits: fa7d7374, e32ac0c6, e327ae0e, 076bd9c0.

  • Q4 retry signing. carlos instances retry is a root-signed grant bound to the failure book's identity and the host's restriction epoch. Every box ignores an unsigned retry (d68a1dde, c12c35c3, e72e68d2, fb55d199).
  • Round-3 §5 code defects: - execution admission (7daf32ac); - bounded reads and acks (f46d3142); - clean one-shot good build, WakeWorker budget and halt event (5ad55485); - GoodVersion retention, and the hold build published before its incumbent stops (d3d464cd); - stage relay give-up (2c5adbe8); - transient status (98c1763d, 61b83e27).
  • Dev-box defects: - the legacy-parent guard: a generation is refused while carlos-exec@<host> holds processes directly (2c5adbe8); - the cgroup.kill respawn kill: a killed kept cgroup is recreated before the next spawn, including retaining-setup seeding (c76a1d44, 66c92efe, 13dffeab).

Both have kernel-gated tests that skip on the build box.

  • F6 for failed wakes. A flag-off flat failed wake keeps its lease until the host cgroup is proven empty (7c0f2543, 23d42f0f, bbb7fc8f). Leftover exe directories in /run are swept (7c0f2543).
  • Simulation. New invariants I5–I8 and mutants M7–M12 (44800ab4). The 3000-seed sweep caught all of M1–M12 on fb55d199 and again on 13dffeab. On 13dffeab the sim baseline ran past the script's 20-minute run timeout, so it was rerun alone: 3000 seeds passed in 22 minutes with no violation.

Still blocks flag-on (code and contract), per Astra round 3

  • An explicit bound, or Paul's ruling, on crash-epoch allocation. The 1<<20 open-time sequence advance is a bound, not a guarantee.
  • Declined boxes: - the nested live-link race (needs an expected identity on Control.Start); - seeds that exec the live link ungated; - recovery of a stranded legacy sidecar.
  • One-shot good publication after a fence or lost lease, and WakeWorker's duplicate-wake window.
  • Uncapped ensureParent error paths (no +memory, EBUSY/EACCES on subtree_control, the mkdir fallback, applyLimit on g<n>).
  • Failed-seed descendant proof, and the flag-off normal-stop half of F6.
  • The declined box's console hold/Retry surface, unless waived.
  • A spec revision: §11 still says a retry is a re-signed pointer.
  • Mixed-version retry sequencing: boxes before consoles. An older binary appending to restart-scope.json drops signed.
  • Copy review of three new strings (copy-review/strings.json, zdd-readiness.retry-no-key.*).

Still blocks flag-on (real box, needs Paul's approval)

  • The delegated arm64 kernel tests as User=carlos: - TestGenerationStartRefusedBesideADirectParentMember, with positive controls; - TestClaimContainmentCleanupRespawn, now covering the KillHost and KillSidecar respawn; - killed-parent children, retained directories, and systemd-stop behaviour.
  • The real-unit control and sidecar checks.
  • Dev rollout and retry sequencing.
  • The flagship probe gates (E–G).

D1 is done: well under 1 s, so keep drop-ins.

Settings plan 4 (root applies the settings) is merged as 8916558.

Owed before O8 (replica keys) is switched on: a cold-wake build-state record is written before the replica-key gate, and two of main's new cgroup refusals can fire after the record is written. A refused launch would then leave a record for a build that never ran.

Owed before host claims are switched on: the leftover consumed/ request files would make the claims census refuse re-installation.

Follow-up: release.ResolveLiveLink and ResolveLivePath now have identical bodies.

Settings plan 5 parked a plan 4 issue: fix before update settings go live. After a hostname moves to a different tenant on the same box, root still trusts the old tenant's stored settings decision, and a registry upsert keeps the old settings columns.

  • A stale pin of the old app's build is refused by the §4.1 authorization (wrong app), so no wrong bytes install.
  • But a stale manual or major hold can freeze the new tenant's instance.

Fix: clear or ignore the settings columns and root's stored decision when the route's account/app changes, keyed on account/app as plan 5's report tagging now is.

Settings plan 5 is merged as d850865. All five instance-update-settings plans are now on main:

| Plan | Commit | |---|---| | 1, #253 | 8d33cef | | 2 | 70f8119 | | 3 | 3086625 | | 4 | 8916558 | | 5 | d850865 |

The tenant-change fix (settings columns and root's stored decision reset when a route's account or app changes) landed with plan 5, accepted by Astra.

Update settings do not depend on CARLOS_SWAP. The console refuses pin, manual and major until every box reports a binary that protects builds, so they go live once the fleet rolls to a build containing these plans.

Flag-readiness round 2: built on zdd-flag-readiness-2 (head 96b5f9a1, CI green), not merged. CARLOS_SWAP stays unset. Astra's whole-branch review, round 2: ACCEPT WITH FIXES for a dormant merge (the fixes are in), NOT YET SAFE for flag-on, with no runtime defect left in the branch. What remains is Paul's sign-off and copy, a deployment-repo audit and the real-box checks. Findings: docs/superpowers/plans/CODEX-FINDINGS-ZDD-READINESS2{-PLAN,,-R2}.md. Plan: docs/superpowers/plans/2026-10-12-zero-downtime-flag-readiness-2.md. The roll notes' "Still owed" and "Real-box checks" sections carry the full state.

Closed on the branch (pending merge)

  • Crash-epoch hard limit (Paul's 2026-10-12 ruling). The 2^20 advance is gone. A restriction's generation is a per-app counter that every change to that app's restrictions moves to a fresh value. It is published or accepted only once it is on disk. A reopen continues from the disk's sequence. The book identity is rotated when a pre-rule file is adopted, and is never taken from the clock. The failure book is schema 3, so a round-1 agent refuses it. The sequence fails closed before it could wrap. Commits 7c7c6141, 191c12b9, 692d1533, a6eb8b35. The FailureBook doc carries the proof and its two assumptions: the file is written only by this rule, and it is never restored from an older copy.
  • Simulation. New invariants I9 (no stale grant clears a later failure) and I10 (only on-disk values are published), and mutants M13 to M17. I9 was rebuilt after Astra's round 1 so that only an applied clear discharges a fact. The 3000-seed sweep caught all of M1 to M17 on 4c9bc525 and again on bcdf4fe9. mutate.sh's per-run timeout now scales with seeds (110 minutes at 3000).
  • Mixed-version retries, by construction. Signed host retries ride their own object, host-retry.json, which no released binary reads or writes. A box on this branch still applies a signed grant found in restart-scope.json, with the same checks. Grants are not seed-skipped across an agent restart. A console writes a retry only for a box that reports retry_v and a saved epoch; otherwise it answers 409. Commits 6e9e84bd, 9f27cdfa, 4d8957c1, 18bde560.
  • Declined box's hold and Retry on the console page. Commits 6e6ad000 and 65d4e135. The new strings await copy review.
  • Declined boxes. A live-link start pins the link once, gates that version and runs that file. This covers nested run and sidecar starts and flat and nested seeds. A sidecar refused because the link names a held build runs the instance's own build. Commits 903ed7ca, c910d020.
  • One-shot good build. A one-shot counts as good only after the sweep parked it, following that sweep's own fence and lease checks. WakeWorker and begin no longer start a second wake inside the extension window. Commit deb4c811.
  • Uncapped cgroup paths. An armed generation whose limits cannot be applied does not start (ErrLimitNotApplied, a box condition). The armed path never uses the PID-move fallback, and the latch is read once. Kernel positive controls were added; they skip here. Commits 98593909, 8530210e.
  • Failed-seed proof and the normal-stop half of F6. A seed writes a durable <db>.seed-unproven obligation before it runs, nested seeds included. No later wake serves that database warm until the seed's processes are proven gone. A flag-off terminal stop proves the host cgroup empty before the lease goes, including every co-stopped or returned sidecar. A flag-off wake proves it again before restoring. Any unknown termination keeps the lease. Commits 4c5e43b5 through 4771d147, then 70849825 and e22f840d.
  • Three flaky tests fixed at their cause (73878213, bcdf4fe9).
  • Data-dir copy audit: no path in platform or platform-infrastructure carries the failure book (a967da80).

For Paul

  • Spec rev-18 amendment for signed host retries (§3, §8.4, §11): docs/superpowers/specs/2026-10-12-zero-downtime-deploys-rev18-amendment.md. It records the Q4 and hard-limit rulings and needs your sign-off. The design branch is untouched.
  • Copy review: copy-review/strings.json on the branch, 7 strings, ids zdd-readiness2.*.

Still blocks flag-on

  • Paul's rev-18 sign-off and copy approval (above).
  • The copy/restore audit of carloku-infrastructure, teamtito/carlos-infrastructure and any backup configuration. A copied failure book must get a new identity.
  • Real box (needs Paul's approval): - delegated placement and sibling limits; - legacy-parent recovery; - live-g1 sidecar respawn; - the unsupported-clone3 refusal; - flat and nested seed and flag-off-stop helper termination before the lease goes; - real-unit control and sidecar fallback; - mixed-version retries and unsaved-book recovery on dev; - downgrade and clone behaviour; - round 1's outstanding kernel checks; - flagship gates E to G.

The list is in the roll notes under "Real-box checks".

  • Claims-on boxes keep declining. Claims-on swaps remain a future feature.

Failure-book audit, flagship: done 2026-10-13. Nothing carries /data/swap-failures.json.

  • AWS (keymail): no AWS Backup plans, no DLM lifecycle policies, and 0 EBS snapshots in eu-west-1, us-east-1, eu-west-2 and ap-southeast-2.
  • carlosframework/carloku-infrastructure (bf3f98b): no EBS volumes, AMIs, snapshots or whole-/data copies. Boxes come from the region-host module, already audited in round 2. The only restores, scripts/rehearse-kass.sh and rehearse-keymail.sh, sync app data into /data/kass/ and keymail paths, never the /data root.
  • Tito (teamtito/carlos-infrastructure): not audited. Tito is a third party and the flag is not ours to turn on there.

Update: Paul approved the seven readiness-2 strings unchanged and signed off the rev-18 amendment (2026-10-13). Applied on zdd-flag-readiness-2 in d15fd8ab: the markers are removed, the record is docs/superpowers/plans/2026-10-12-zero-downtime-flag-readiness-2-copy.json, the pin is TestZeroDowntimeReadiness2CopyIsApplied, copy-review/ is deleted, and the amendment is marked approved. origin/main 67698c9c is merged. Head 7c81e031, CI green. Still blocks flag-on: the deployment-repo copy/restore audit and the real-box checks listed in the roll notes. The design branch's spec file still reads rev 17 until the amendment is copied in.

Dev-box checks, round 2 (2026-10-13, Paul's approval). Dev edge box only (i-046c7fe43620944bc). Result: the flag cannot work on the shipped edge unit.

Inventory. carlos 5da1cc9b, Debian 13 arm64, kernel 6.12.105, systemd 257, 290 units, 0 failed. Edge healthy. The three proof routes answer 502, as before. Swap and claims both off.

Roll. Old road, 5da1cc9b → ba52d5ef. Instance-control restarted by hand. All healthy, routes unchanged.

Kernel suite. arm64, User=carlos, Delegate=yes, transient units.

  • Pass: TestClaimContainmentCleanupRespawn 3/3, all three subtests. TestGenerationStartRefusedBesideADirectParentMember 3/3, with the sibling and recovery controls.
  • Pass: TestGenOOMKernel, TestGenStrictKernel, and TestStrictKernel in permission-denied and unsupported modes.
  • Pass: TestStrictKernelSupervisor (first run; supervisor SIGKILLed, its tree died, the unit restarted). The host Kernel tests (8), the stopevidence suite (85, plus WholeTree with CARLOS_CGROUP_TEST), hostexec (36), instancectl (123) and cgroup (140).
  • Fail: TestStrictKernel full mode. It panics at recreate_linux.go:156, because the test's other := &Manager{root: m.root} has no logf. It is a test defect; production always sets one. With logf: t.Logf added in scratch, all five subtests pass.
  • Probe: after cgroup.kill, a child that already existed dies on clone3 too; a new child lives. systemd's stop kills per process and recreates the unit cgroup.

Real-unit checks (scratch harnesses, not committed):

  • Placement and limits: pass. The real StartGen drop-in had the slice, workdir, binds, exe mounts and 512M limits; Parent raise and restore worked.
  • Seed and stop: pass. Flat failed seed: the SIGTERM-ignoring helper was killed, the db and obligation removed, the reseed landed fresh. Flat good seed: EmptyHost killed the helper. Flag-on seeds: the role-seed helper was killed. The nested seed was not run (nested layout is not in force here).
  • Legacy-parent guard: refuses, pass. With clone3 blocked, a generation is refused with ErrLimitNotApplied.
  • StopGen(kill) on a never-loaded unit: pass. The declined-box sidecar check was not run (it needs claims on).

Swap under load: blocked. Armed (overlap swaps armed), the box refused every generation start: clone3-into-cgroup … function not implemented. That gave 196/196 requests 503 ("instance failed to wake"), worst 1.03 s, with no failure record (correct). Cause: carlos-edge.service has RestrictNamespaces=yes, and systemd answers clone3 with ENOSYS under it. A transient unit confirmed it: with =no the spawn lives, with =yes it gets ENOSYS. Flag off, the same build answered 200 with its legacy sidecar; flat starts latch the PID-move fallback. Changing the unit was outside the approval, so builds B–D were not run. Reverted at once.

Mixed-version retries. Not run: they need a console and a working armed swap.

Final state. ba52d5ef, CARLOS_SWAP unset, 0 failed units, routes unchanged. bootstrap/carlos is now ba52d5ef. The test app, route, staged build, book and S3 objects are deleted; generations.seq was left at 109. All scratch is removed.

Blocks the flagship:

  1. RestrictNamespaces=yes on carlos-edge blocks CLONE_INTO_CGROUP on every box.
  2. Arming doesn't check for clone3, so a box reports armed and fails every wake. It should decline with a reason code.
  3. The roll notes' probe app parses only -socket, but the platform passes --db too, so it exits at once.
  4. The TestStrictKernel test defect.

Flag-readiness round 3: built on zdd-flag-readiness-3 (head f4f9547b, origin/main 645a7d05 merged), not merged. CI is red on one test that needs your ruling (below). CARLOS_SWAP stays unset.

Needs Paul: the Plan 14 root-read census

TestPlan14RootReadCensus arrived with claim Plan 14 Part 1 (645a7d05, #280) and fails on this branch with 28 discrepancies. Everything else in the gate passes. (TestPlan9B8RecordWriteCensus also tripped on the shim hook in main(); it was reviewed and re-pinned in f4f9547b.)

  • The shim's and VerifyPlacement's new reads are agent-side cgroupfs and procfs reads that gate a tenant exec or the arming decision. No kept family A to G owns them, so the honest class is legacy. That matches how fix1's merge review classed the same delegated-cgroup code.
  • startIn, openCleanupAt, StartWorkerStaged and spawnChildFrom need their legacy rows refreshed.
  • The deleted placementLive needs an @removed row with an owner.
  • Plan 14 says any new or refreshed legacy row changes the census seal, which "requires a separate explicit scope ruling, never routine census maintenance". So I have not resealed it.

Asked: either allow a plan14-root-reads-zddr3.tsv overlay that re-freezes these rows as legacy, changes the seal and removes placementLive, or name the kept family that should own agent cgroup placement. Once ruled, it is one commit, then CI. Astra's whole-branch review, round 2: ACCEPT for a dormant merge, and no further code blocker to flag-on. The real-box checks are still owed. Findings: docs/superpowers/plans/CODEX-FINDINGS-ZDD-READINESS3{-PLAN,,-R2}.md. Plan: docs/superpowers/plans/2026-10-13-zero-downtime-flag-readiness-3.md. The roll notes' "Roll notes from zdd-flag-readiness-3" section has the dev re-check list.

Ruling (Paul's): the edge unit keeps RestrictNamespaces=yes and all its other hardening. clone3 placement is replaced by an exec-shim.

Closed on the branch (pending merge)

  • The RestrictNamespaces blocker. Every cgroup.Manager start now runs carlos __cgexec: flat, generation, worker, legacy sidecar, seed and the fallbacks. The shim: 1. writes its own PID into the target cgroup.procs through a validated directory fd, without following symlinks; 2. checks /proc/self/cgroup and the directory's identity; 3. reports "placed"; 4. waits for the parent's release, so the caller's ownership check runs after placement and right before exec; 5. only then execs the tenant.

Argv, env and dir travel over a pipe, byte-exact, and the shim's own environment is empty. Fds above the tenant's are marked close-on-exec. The signal mask is emptied, the ignored signals are restored, and the rlimits match a direct start. The leader PID is the tenant's.

A start is reported failed only when the shim proves no exec happened. A generation that cannot be placed answers ErrLimitNotApplied and never moves by PID. A flat start falls back to today's PID move only after a proven pre-exec failure, so no tenant runs twice. A placement still in flight counts as a member for every emptiness and coldness check. Cost: about 4.5 ms per spawn. Commits ff4b814a, a34013c6, 961aed71, 306aa76d, 24887f97, 281711c9.

  • Arming verifies placement. With the flag on, the agent does one real shim placement into a scratch carlos-placement-check cgroup, with a 4 s budget inside the 20 s update gate. If it fails, the box declines with the new code cgroup-placement-failed. The member sees the existing approved "declined" phrase, so there is no new copy. The agent logs REFUSING CARLOS_SWAP=overlap (cgroup-placement-failed). Commits 7fbafe7e, 1f9d6a83.
  • Kernel proof under the edge unit's own sandbox lines (891c0fb7). This is a transient-unit test that reads carlos-edge.service. - An unrestricted control shows clone3 works. - The hardened run shows clone3 gets ENOSYS, while the shim still places the process and the 64M limit OOM-kills a hog alone. - VerifyPlacement passes in the hardened run.
  • The probe app is now cmd/zddprobe. It takes the platform's --socket … --db … and vet's -socket, and is tested by launching it that way (59b1895a).
  • TestStrictKernel full mode was a test fix (logf); it passes 5/5 in a delegated unit. reset-failed now runs after a successful generation stop or abandon-kill (496fb6ed).

For the dev re-check (Paul's approval)

The list is in the roll notes. In short:

  • the roll;
  • flag-off wakes with no placement before exec failed lines;
  • arm, then overlap swaps armed with no REFUSING;
  • the zddprobe under load through zdd-1 to zdd-3, plus the hold control;
  • no generation unit left in systemctl --failed;
  • the two activator kernel tests that need the io controller (TestClaimContainmentCleanupRespawn, TestGenerationStartRefusedBesideADirectParentMember);
  • the edge-unit lines a user unit could not test: ProtectProc, ProtectClock, RestrictRealtime, SystemCallArchitectures, ProtectSystem=strict versus full, and User/Group/capabilities/Slice/io delegation.

Still owed or noted

  • Claims-on: the strict claims path keeps clone3 (Paul's instruction). Under the shipped unit it answers ErrStrictUnsupported, which refuses the launch and never runs it uncapped. That is a precondition for claims-on swaps.
  • Mixed versions: a console older than the box shows the generic phrase for the new code and cannot offer a host retry for that box. An older CLI's deploy --watch shows that box as pending until the watch times out. Roll boxes before consoles and CLIs.
  • The 4 s arming bound assumes SIGKILL takes effect promptly. A box stuck in kernel I/O could decline late.
  • No simulation sweep: the branch touches no swap or gate logic. Activator production changes are comment-only.

Readiness round 3: CI green at ee603636 on zdd-flag-readiness-3. Not merged. The Plan 14 root-read census is settled under platform-1e's census ruling zddr3 (rev3=census-ruling-zddr3).

  • New overlay plan14-root-reads-zddr3.tsv, loaded after fix6. No earlier overlay or the baseline was edited. It covers 28 of 28 rows: - the shim's and VerifyPlacement's reads, accepted as legacy within the boundary the agent's own delegated cgroup subtree (including its scratch probe child), /proc/self, and pipes to and from its own parent, running as carlos; - the three changed legacy rows; - the eight envelopes; - H3 retirements of placementLive and Manager.startIn.
  • Narrow format change. An @removed owner H3 is accepted only with a census-ruling- basis. A unit that is still declared can be retired only when the scan finds it no longer reads anything. The basis text is never taken as evidence. Tests cover the passing case, a retired unit that still reads, non-H3 removals, and a retired unit that starts reading again (it shows up as unclassified).
  • New legacy seal c748e02b…c568, pinned beside the zddr3 load.
  • Shim comment. If the shim ever runs as root, or reads outside that boundary, it stops being legacy and needs the Plan 14 owner's ruling.
  • Caller check: passes. No root-run path reaches startViaShimBy, VerifyPlacement or runShim. - The only cgroup.Manager is built in runAgent, which only carlos agent reaches, and only carlos-edge.service runs that, as User=carlos. - runShim runs only when the binary is started with __cgexec, and only startShimCore starts it that way.

The dev re-check list from the earlier reply is unchanged.

Dev-box checks, round 3 (2026-10-13, Paul's approval). leo-climbing only. The swap works under load, with 0 failed requests in every phase.

Inventory and roll. Leo ran v0.26.1 (stable), 292 units, 0 failed, healthy; swap and claims off. I stopped carlos-system-update.path for the test (runtime masks lose to the /etc unit files). Old road to 2e00b17e, instance-control restarted by hand. Healthy, with no placement before exec failed lines.

Kernel checks (arm64, transient units). All pass:

  • TestEdgeHardeningInner under every [Service] line of the installed edge unit. The shim places under clone3 ENOSYS, the hog is OOM-killed alone, and VerifyPlacement passes. ProtectProc is now seen enforced.
  • 22/22 shim and generation tests under the same lines (7.1 ms overhead).
  • TestStrictKernel full, 5/5, and the step-7 activator tests.

Swap. Load came from curl on leo via --resolve 127.0.0.1 against a .invalid host, so sophie could never wake it (#286).

  • 3a: armed, no REFUSING. Baseline: 1567 requests, 0 failed, worst 1.29 s (cold wake).
  • 3c: B, then C. 13275 requests, 0 failed, worst 0.19 s. The header flipped a→b→c, the canary settled in 10 s, and the drains took 1 s and 0.5 s. Limits were 1024M only during each overlap, then 512M again.
  • 3d: D never listens. Health failed at 30 s; cohort.halted and swap.failed were logged, the record was written, and D was not retried. C served throughout: 8281 requests, 0 failed, worst 0.16 s.
  • 3e: restart, done as a restart-gen.json touch plus a serial bump. It ran as a rolling swap. 4945 requests, 0 failed, worst 0.14 s.

Final state. Reverted. Leo runs v0.26.1 (the original bytes), with CARLOS_SWAP unset, 0 failed units and the system units as before. The test app, route, builds, book and S3 objects are gone. wakecheck was not touched. bootstrap/carlos has its prior bytes.

Findings. None block the swap. Still owed: mixed-version retries and the hold control, both of which need a console.

  1. Leo runs instances as carlos-instance-gen@ units, so the shim here ran only at arm time. Check the flagship's layout before Gate F.
  2. After an old-road roll the system leg never returns a box to its pointer (result.json "applied" is terminal). I restored v0.26.1 by hand. Add this to the roll notes.
  3. Direct-mode promote doesn't bump control/serial, so I bumped it by hand.
  4. Minor: an edge restart leaves the flag-off carlos-instance@ unit failed. The first control-socket sweep after each restart got "connection reset".

Dev-box checks, round 4 (2026-10-13, Paul's approval). leo-climbing only. The hold control passes, and every retry transition I ran behaves as the roll notes say.

Inventory. v0.27.0, 292 units, 0 failed, flag unset. Armed cleanly, no REFUSING.

#286. The instance domain was zdd-check.invalid, which never resolves. Load ran on leo via 127.0.0.1. The CLI's sandbox allowed only the console, and refused zdd-1's watch URL poll.

Door check. zdd-probe4 swaps in overlap mode, 1 at a time on each box, plus the migrations note. instances enable was store mode (no apps domain on the dev console).

Swap. Baseline: 3270 requests, 0 failed, worst 1.27 s (cold wake). zdd-2: 6746, 0 failed, worst 0.16 s; swapped, box leo-climbing: 1 swapped, exit 0. No sophie line (unarmed).

Hold control: pass. zdd-3 under --swap hold: 6613 requests, 0 failed, worst 1.99 s. Four requests took over 0.5 s, all at the stop-start.

Retries.

  • a: pass. zdd-4 never listens. The watch said failed … 30 seconds, and zdd-3 served 5552 requests, 0 failed. The API showed held_by zdd-4; the snapshot had epoch 1, book and retry_v 1, so Retry is offered. Page not captured (browser session only). instances retry: leo applied the grant within 1 s. The canary failed and was held again.
  • b: pass. chattr +i on the book during the canary's health wait put epoch 0 in the snapshot. The console said "has not saved its failure yet". After the unlock the save landed in 2 s.
  • c: pass. v0.26.1 by the old road, flag on, epoch-7 hold: unknown schema 3, managed starts refused (a wake got 503), book untouched. The console said "not held … nothing to retry". With a schema-2 copy of the hold, which v0.26.1 reports without retry_v, the console said "too old to take a retry" (409). No refusal wrote host-retry.json. Back on v0.27.0 the book took a new identity; spent grants cleared nothing.
  • d: not run. It needs the console box.
  • e: skipped. A data-dir copy needs a second agent.

Final state. v0.27.0 restored by hand, flag unset, 0 failed units, system units back. system status shows leo current. wakecheck untouched. The app is trashed, and the rest is deleted.

Findings (none block Gate F):

  1. "Too old" needs a readable book. A real flag-on downgrade says "not held". Note it in the roll notes.
  2. Status republishes every 5 min, so "not saved" outlasted its save by 4 m 40 s, despite "try again in a minute".
  3. carlos-stage restarts system-update.path on every staged build, so stopping the path doesn't hold. That undid round 3's hold too.
  4. A first deploy to a fresh instance URL-polls. On dev that hits sophie.

Gate F, flagship probe on annie-welding (IE), 2026-10-13: FAILED and REVERTED. Arming broke three tenant instances, so annie is not armed.

Pre-check. annie was on v0.27.1 (current), edge active, NRestarts 0. host.env had no CARLOS_SWAP line. 39 carlos-instance@ units: 12 running and the known 27 failed (plus amadan-backup.service, 28 failed in all). There were no gen units. probe1 showed REGION empty, INSTALLED p12, SERVING "—".

Door check. adopt-probe swaps in overlap mode, 1 at a time on each box, plus the migrations note, exit 0.

Arm (22:08:11Z). Started at 22:08:13.290 and overlap swaps armed (CARLOS_SWAP=overlap) at 22:08:14.870 (+1.6 s). The steering probe listened at +0.17 s and the hibernation activator armed at +3.6 s. No REFUSING. NRestarts 0, console 200.

Probe. Only partly run:

| phase | requests | failed | slowest | | --- | --- | --- | --- | | baseline (about 28 s, p12) | 865 | 0 | 0.09 s | | overlap | not run | | | | hold | not run | | |

zdd-1's watch timed out with its box has p12 installed, not this build, exit 1. The edge couldn't stage it (open directory /opt/carlos/versions/zdd-1: read-only file system, then needs a root pass), and probe1's DB was missing (booting fresh). I stopped the run before zdd-2, so only one of the four deploys was used. INSTALLED is p12 and SERVING is p12.

Why I reverted. From 22:09:04 to 22:12:42 every generation wake of canada, ips and kpmg (.bdn) failed: IPS_ORIGIN is set to "https://canada.bdn.oncarlos.com" and the platform says this instance is "https://app.3" (from -socket). The app exits 1 and fails health at 30 s with app.N.sock. Generation sockets are app.<N>.sock, and the IPS framework reads the socket's base name as the instance's origin. This is a blocker for any IPS-framework app.

Reverted at 22:12:32: removed the line and restarted. All three started as carlos-instance@ at 22:13:14 and are healthy.

annie now. Not armed, active, NRestarts 0. 31 failed units: the 27, amadan-backup, and movement, printcard-app and sheets. Those three were hibernated by the restart and are in the same signal state as the 27. ipsdocs is still crash-looping, as it was before.

Findings.

  1. BLOCKER: app.<N>.sock breaks apps that derive their origin from -socket. If an app has no IPS_ORIGIN, it may start silently as https://app.N. memoria, printcard-app, proving and rootline-staging woke as generations; their origins weren't checked.
  2. /data/swap-failures.json holds bdn/ips 2d577ebda on canada (mode overlap). That's the live build. It does nothing while the flag is off, but it must be cleared before annie is re-armed.
  3. An edge restart SIGKILLs the flag-off units it is stopping, leaving them failed, so "failed count beyond the 27" can't be the revert test.
  4. On the flagship the edge can't stage a new build itself. The first deploy waits on a root pass, and that outlasted the 2-min watch.
  5. adopt-probe's swap policy is overlap. zdd-1 is promoted but not installed.

Dev-box checks, round 5 (2026-10-14, Paul's approval). leo-climbing only. The rev 19 fix works. Every generation launched on g<n>/app.sock with CARLOS_HOST set, and the v0.27.2 probe passed its rule-5 check and served.

Inventory. v0.27.2 current, 292 units, 0 failed, flag unset. Armed cleanly, no REFUSING.

Evidence (p1.zdd-check.invalid, nested):

  • Mid-overlap, the run dir held g1/app.sock and g2/app.sock. The canonical app.sock -> …/p1.zdd-check.invalid/g1/app.sock flipped to g2/app.sock at 05:04:45.2.
  • The app's argv was /opt/carlos/live/p1.zdd-check.invalid --socket /run/carlos/instances/p1.zdd-check.invalid/g2/app.sock --db …, under carlos instance-exec run p1.zdd-check.invalid:2. g1 to g4 matched.
  • The app's environment had CARLOS_HOST=p1.zdd-check.invalid.

| phase | requests | failed | slowest | watch | | --- | --- | --- | --- | --- | | baseline zdd-1 | 3233 | 0 | 1.250 s (cold) | URL poll refused by the sandbox (#286 guard) | | overlap zdd-2 | 6660 | 0 | 0.165 s | swapped, box leo-climbing: 1 swapped, exit 0 | | overlap zdd-3 | 6664 | 0 | 0.137 s | the same | | failing zdd-4 | 5497 | 0 | 0.162 s | failed … 30 seconds, box leo-climbing: 1 failed, exit 1 | | hold zdd-5 | 6544 | 0 | 2.015 s | swapped, exit 0 |

Directories. g1 went 0.5 s after the link flip and g2 1 s after; in the hold phase, g3 went at the stop. None lingered.

Failing build. zdd-4 never listens. Health failed at 30 s on …/g4/app.sock, followed by cohort.halted and swap.failed ("Still serving build zdd-3"). The record was written (overlap), and the console showed held: build zdd-4 failed on this box. g4 held only app.sock.control, and it was removed as the failure landed. zdd-5 lifted the halt.

Hold: pass. 2.0 s worst, against 0.17 s for overlap. Four requests took over 0.5 s, all at the stop-start.

Final state. Flag unset, host.env as before, edge NRestarts 0, 0 failed units. System-update units as recorded. system status shows leo current on v0.27.2. wakecheck untouched. The app is trashed. The instance, builds, book, run dir and bucket object are deleted. sophie was not inspected.

Findings (none block Gate F):

  1. The flag-off edge restart left g5/ with a stale app.sock. Rule 3 calls it harmless, but with the flag off nothing reuses it, so it stays until reboot. I removed it by hand. Add it to the revert steps.
  2. A new generation's app.sock is srwxr-xr-x for about a second before it becomes srw-rw----. The 0750 directory covers it.
  3. Still owed before Gate F: clearing annie's stale bdn/ips record (needs Paul).

Gate F retry, flagship probe on annie-welding (IE), 2026-10-14: STOPPED at the pre-check. Nothing changed on annie, and it is not armed.

Pre-check (05:20Z). annie is on v0.27.2, current. Edge is active since 04:58:00Z, NRestarts 0. host.env has no CARLOS_SWAP line. There are no gen units. The console answers 200.

  • 33 failed units: 31 stale carlos-instance@ units in signal, hello.bdf in timeout (04:58:21, the roll), and ipsdocs.bdn crash-looping (60 failures in the last hour).
  • amadan-backup.service is no longer failed (success at 05:05Z).
  • canada, ips and kpmg (.bdn) have been active as carlos-instance@ since 04:58:39Z, with no errors in the last hour.
  • probe1 shows INSTALLED p12 and SERVING "—".

Blocker: the box refuses every zddprobe build for probe1.

  • /data/programs/probe1.bab.oncarlos.com.json lists only adoptprobe. p12 is that program, not zddprobe.
  • Root wrote a wrong-program refusal for zdd-1 at 2026-10-13T22:10:04Z: "wrong program: …/cmd/zddprobe", restored p12. Every pass since repeats refused: zdd-1 is the wrong program for this route … ship a new version with carlos ship --program-change.
  • So last night's zdd-1 timeout was this refusal, not a slow root pass. Waiting on INSTALLED would never succeed.

The D2 check (adopt.go:1435) lets a zddprobe build through only as a new version shipped with carlos ship --program-change. carlos deploy has no such flag. zdd-1's name is burned, because its manifest is immutable. That is outside the approval, so I did not move the failure book or arm annie.

Needs Paul, one of:

  1. Approve shipping the first probe build under a new name (e.g. zdd-0) with carlos ship --program-change, then carlos promote it to edge. This permanently changes adopt-probe's recorded program to cmd/zddprobe. zdd-2 to zdd-4 then deploy as the roll notes say.
  2. Run the probe against a fresh app with no program history, as dev did (needs its own approval and an instance on annie).

annie now: unchanged. Not armed, /data/swap-failures.json still in place. adopt-probe is untouched; its swap policy was left at overlap last night.

Findings.

  1. The roll notes' Gate F probe cannot run on adopt-probe as written. Its program is adoptprobe.
  2. Correction to the first Gate F report: zdd-1 was refused as the wrong program, not left waiting on staging.
  3. Root's refusal record for zdd-1 stays on annie until a different build is promoted.

Log: $TMPDIR/zdd-gateF2.log on DOCS-T1.

Gate F retry (option 2, a fresh app), 2026-10-14: PASS. annie-welding stays armed.

Setup. I created bab/zdd-probe through the console with no region, enabled its instances, and created zddprobe1.bab.oncarlos.com ("Ireland · Dublin"). Only annie's edges file lists it. The host did not resolve from DOCS-T1, so the load used curl --resolve …:443:99.81.104.219.

Pre-check (13:18Z). v0.27.2 current. Edge NRestarts 0. No CARLOS_SWAP line. 30 failed units, all signal. ipsdocs was crash-looping (60 an hour).

Failure book. The bdn/ips 2d577ebda record (sha256 9ddaed8d…) is now /data/swap-failures.json.gatef-20261014.bak.

Arm (13:18:48Z). overlap swaps armed at +1.6 s, after the placement check passed. No REFUSING. NRestarts 0.

Tenant watch. No tenant failed under the flag.

  • canada, ips and kpmg woke as generations. Each logged msg=origin with the IPS_ORIGIN fallback for …/g6|g5|g4/app.sock, at INFO with no error.
  • printcard-app, rootline, seapointish, slides, woodfamilytrust, hello and infinity also woke cleanly as generations.
  • bunkhouse: the old edge recycled it while shutting down (209/STDOUT). The stale lease then refused its wakes from 13:18:52 to 13:33:35 (227 refusals). It then woke healthy on g33/app.sock. This is the restart, not the flag.
  • ipsdocs: it exits 1 just as it did with the flag off. The book now holds bdn/ipsdocs c201821.

| phase | requests | failed | slowest | | --- | --- | --- | --- | | baseline (zdd-1) | 2063 | 0 | 0.208 s | | overlap (zdd-2, zdd-3) | 8092 | 0 | 0.290 s | | hold (zdd-4) | 3932 | 0 | 2.808 s |

Watches.

  • zdd-1: installed, box annie-welding: 1 asleep with it installed, exit 0, 41 s.
  • zdd-2 to zdd-4: each printed swapped https://zddprobe1… and box annie-welding: 1 swapped, exit 0, about 57 s each.

Sockets. The link went g11 → g16 → g19 → g23/app.sock, and argv was --socket …/g<n>/app.sock. The old directory went at each swap; g11 went 0.4 s after the flip.

annie now. Armed, NRestarts 0, 35 failed units. Five are new since the arm: bunkhouse (exit-code) and ipsdocs, plus rootline, seapointish and woodfamilytrust, flag-off units the restart stopped (signal). All but ipsdocs run as generations.

zdd-probe is trashed. On annie I removed its instance, builds, links, data and programs file, and its S3 object; no g<n> was left. Not removed: an empty carlos-instances-zddprobe1…slice. adopt-probe is untouched.

For the roll notes.

  • Gate F needs a fresh app, not adopt-probe. adopt-probe's program is adoptprobe, and root refuses zddprobe builds there without carlos ship --program-change.
  • An edge restart can leave an instance's lease held for about 15 minutes.
  • The book will hold a crash-looping tenant's live build.

Gate G: PASS. All four flagship boxes are now armed with CARLOS_SWAP=overlap.

The boxes were done one at a time in the approved order. Each got the Gate E flag command (append only if there is no CARLOS_SWAP= line, then one systemctl restart carlos-edge). Each ran v0.27.3. The serve check curled edge.oncarlos.com through --resolve to the box's public IP, so no tenant was woken. Nothing else changed: no deploys, apps or instances. annie, the scanner, Tito and dev were not touched.

1. us i-0ed0641d8e476f1b8 = adi-knitting (journal: this box is "adi-knitting")

  • Pre-check: edge active, NRestarts 0, no CARLOSSWAP line, 0 failed units. No instances homed here (only `probe), 35 routes, no /data/swap-failures.json`.
  • Armed: restart 19:03:02 to 19:03:12Z.
  • Verify: placement check: memory.max=512M memory.high=460M, overlap swaps armed (CARLOS_SWAP=overlap) and hibernation activator armed all logged. 0 REFUSING, 0 LEASE HELD, NRestarts 0, curl 200. The arm created an empty book (records: null).
  • Watch: 19:03 to 19:13Z, 10 polls, clean, nothing woke. Final: active, 0 failed units (was 0).

2. uk i-03c310ceaf50a9bab = iris-clockmaking

  • Pre-check: same as us. No instances, 31 routes, 0 failed units, no book.
  • Armed: restart 19:14:04 to 19:14:05Z.
  • Verify: the same three lines; 0 REFUSING, 0 LEASE HELD, NRestarts 0, curl 200, empty book.
  • Watch: 19:14 to 19:24Z, clean, nothing woke. Final: active, 0 failed units (was 0).

3. au i-022288023f5e7496d = barbara-tinkering

  • Pre-check: clean, 37 routes, 0 failed units, no book. One homed instance, au.bdk.oncarlos.com, asleep (no activity in 7 days).
  • Armed: restart 19:24:33 to 19:24:35Z.
  • Verify: the same three lines; 0 REFUSING, 0 LEASE HELD, NRestarts 0, curl 200. The only error lines, domains: skipping ... context canceled, came from the old edge process during its shutdown.
  • Watch: 19:24 to 19:35Z, clean, nothing woke. Final: active, 0 failed units (was 0).

No watch saw a msg=origin or g<n>/app.sock line, and no restart stopped an instance, because nothing was awake. So the flag is armed on these three boxes but no swap has run under it there yet. The first real wake or deploy will be the first test.

The roll notes' "one ordinary deploy of adopt-probe after each box" was not run, because this approval excluded deploys. It is still owed if wanted.

Log: $TMPDIR/zdd-gateG.log (tokens redacted; host.env values never read).

Kernel tests owed on #272, run 2026-10-14 (Paul's approval). origin/main 2683f16a. Throwaway t4g.small, Debian 13 arm64, kernel 6.12.96, systemd 257. Dev account, SSM only. Each run used a transient unit with User=carlos and Delegate=yes, plus a short /tmp TMPDIR. No build tags.

Result: all pass, none skipped.

  1. TestClaimContainmentCleanupRespawn (CARLOS_CLAIM_SPAWN_KERNEL=1): PASS. All three subtests pass: strictprimitive, KillHostthenrespawn and KillSidecarthen_respawn. Each respawn logs "took cgroup.kill; removed so the next spawn gets a fresh cgroup".
  2. TestGenerationStartRefusedBesideADirectParentMember (same run): PASS, including siblingcontrol and recoverycontrol. It logged "generation start refused: the parent cgroup holds 1 process(es) directly".
  3. ensureParent positive control: internal/cgroup TestGenKernelGenerationRefusedBesideAFlatMember and TestGenKernelShimRespawnAfterKill (CARLOS_GEN_KERNEL=1 CARLOS_SHIM_KERNEL=1): both PASS. The refusal fires, g1 starts capped once the flat member has gone, and three respawns each get a fresh cgroup.

subtree_control. Inside the unit, root reads cpuset cpu io memory pids. Before any delegated unit it read memory pids, because systemd turns controllers on only when a unit asks. My first run checked too early and aborted without running tests. The check now runs inside the unit.

Runs. 20261014T230356-66c210 (aborted at the check), 20261014T230527-7805b0 (tests 1-2), 20261014T230634-2a20f2 (test 3).

Cleanup. All three instances are terminated. My S3 prefixes are deleted. Another session's runner, i-0ea8a329d4ae3879a (run 20261014T222248-c3b06b), was still running. I left it alone.

This kernel is 6.12.96, not the dev box's 6.12.105.

Dev Gate: CARLOS_SWAP=overlap is armed on both dev boxes. Both PASS.

leo-climbing (i-046c7fe43620944bc)

  • Pre-check: v0.27.3 current; edge active, NRestarts 0; no CARLOSSWAP line; 0 failed units; 2 instance dirs (probe, wakecheck), none active; no swap-failures.json; no g<n> dirs.
  • Armed 22:01:59Z. One edge restart, PID 430708.
  • Verify: placement check (memory.max=512M), overlap swaps armed (CARLOS_SWAP=overlap), hibernation activator armed. 0 REFUSING, 0 LEASE HELD, NRestarts 0. A new empty book was written (records null).
  • Watch: 5 polls (22:03 to 22:07), 0 strong lines, 0 failed.

sophie-knitting (i-05498eada6f0afd23, the dev console box; name taken from the edge journal)

  • Pre-check: same as leo. console.service: PID 466172, active since 18:11:31, not bound to carlos-edge. Its drop-ins (apps-domain.conf, release-key.conf) were left alone.
  • Armed 22:08:18Z. One edge restart, PID 472951.
  • Verify: same three lines. 0 REFUSING, 0 LEASE HELD, NRestarts 0.
  • Console: still PID 466172, same start time, NRestarts 0, answering 200.
  • Watch: 5 polls (22:09 to 22:13), 0 strong lines. The console stayed 200 and was not restarted.

wakecheck (one wake through DNS, 22:14:07Z): 200, host=ip-172-31-14-16 (sophie), hits=551. It started carlos-instance-gen@wakecheck…:1, g1/ is live, and the edge logged no errors.

Final state: both boxes have CARLOS_SWAP=overlap, edge active, NRestarts 0, 0 failed units.

Not touched: carlos system status also lists adi-sailing as UNREACHABLE (2aaf747).

platform-d8 and platform-1e have been told.

Owed items: status as of 2026-10-15.

  • Armed: all four flagship boxes (Gates F and G, 2026-10-14) and both dev boxes. adi-sailing (carlos-dev-uk) needs CARLOS_SWAP=overlap once it's bootstrapped.
  • Merged (zdd-owed-items): - The cold-wake build-state record is now written at the first exec, after the replica-key gate and cgroup refusals, with a recheck of admission and the exec gate after the write. A nested seed's obligation is now written before its record. - ResolveLivePath delegates to ResolveLiveLink. - The roll notes are brought up to date. - Astra ACCEPTED it; the census ruling is census-ruling-owed.
  • Kernel tests: the io-controller activator tests and the ensureParent positive control PASS on the root runner (reply above).
  • Moved to #250 (platform-1e, Plan 14): the claims consumed/, lock and record leftovers that block re-installation.
  • Closed separately: #292 (edge-restart lease race, v0.27.3) and #275 (binary 77.7 → 45.8 MB, with a CI ceiling).
  • Still open: - The new box + old console retry transition was not run on dev. It is covered by construction, since grants ride host-retry.json, which no older binary reads or writes. I've asked Paul to waive it. - Claims-on swaps remain a future feature.

Waived by Paul (2026-10-15): the new box + old console host-retry transition is not run on dev. It is covered by construction: signed host retries ride host-retry.json, which no older binary reads or writes, so no ordering of box and console rolls can apply a grant unsafely. The other three transitions passed on dev round 4.

With that, nothing owed on this thread blocks the flag. All four flagship boxes and both dev boxes are armed. What's left:

  • claims-on swaps, a future feature;
  • the claims leftovers, moved to #250;
  • arming adi-sailing (carlos-dev-uk) once it's bootstrapped.