amadan

carlos / platform Public

Clone 
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

Zero-downtime deploys: roll notes (plan 6, task 17)

These are the notes for putting zero-downtime deploys into production, gate by gate. They are written, not run. Nothing in this file has been executed against a box, a bucket, a console or an AWS account, and every command below is quoted for Paul to run or approve.

Spec: 2026-10-07-zero-downtime-deploys-design.md (rev 17), section 15 for the order. The plans are 2026-10-07-zero-downtime-deploys-0-index.md and plans 1 to 6, in platform-design-zero-downtime-deploys.

Ground rules

  • Approval. Every gate that touches dev or the flagship is production infrastructure (AGENTS.md, "Production infrastructure needs scoped approval"). Before running one, get Paul's explicit approval for that gate, naming the boxes. Approval for one gate is not approval for the next.
  • AWS profiles. Assumed-role profiles need the Bash sandbox off for that command. Dev is profile kass. The flagship is profile keymail.
  • Order. Boxes first, then the console's swap-policy settings, then the flag: dev, then one flagship box with a fresh probe app (Gate F), then the rest of the flagship. The deploy docs and release notes carry the migrations rule before the flag goes on across the fleet. Plan 5 keeps the flag off outside dev until this plan has merged.
  • The flag. CARLOS_SWAP=overlap in a box's /etc/carlos/host.env. Only the exact word overlap arms it. Anything else, or nothing, leaves the box on today's stop-then-start paths.
  • Scratch. The probe's scratch lives under $TMPDIR, never /tmp.

Before Gate E: owed

Current state, 2026-10-14. This list is copied from amadan discussion #272 ("Zero-downtime deploys: owed before CARLOS_SWAP can go on"). Every item on it that blocked the flag is now closed, or is one of the open items named below. The subsections after this one are the history of how each item closed, kept for their commit references. Where they say "pending merge", read the state here instead.

  • Merged and released. Readiness rounds 1 to 3 (zdd-flag-readiness, zdd-flag-readiness-2 and zdd-flag-readiness-3). Rev 19, the generation socket names, is merged as ebddcfc and released in v0.27.2. #292 is merged as e15ad6b and released in v0.27.3.
  • The flag. Gate F passed on annie-welding, and Gate G armed the other three flagship boxes. All four flagship boxes are armed as of 2026-10-14. Dev is being armed on 2026-10-14.
  • Kernel checks: closed. On 2026-10-14 a root test runner ran them on Debian 13 arm64, kernel 6.12.96, origin/main 2683f16a, as User=carlos with Delegate=yes and cpuset cpu io memory pids in the unit's subtree_control. All passed and none skipped: - TestClaimContainmentCleanupRespawn, all three subtests; - TestGenerationStartRefusedBesideADirectParentMember, with its sibling and recovery controls; - the ensureParent positive control, which is internal/cgroup's TestGenKernelGenerationRefusedBesideAFlatMember (CARLOS_GEN_KERNEL=1) and TestGenKernelShimRespawnAfterKill (CARLOS_SHIM_KERNEL=1).

The details are on #272.

  • Mixed-version retries. Dev round 4 passed three cases: a new box with a new console, a book that had not saved, and an old box with a new console. A new box with an old console was not run. That case is covered by construction. An old console can only write an unsigned retry, and every new box ignores an unsigned retry, so the worst it can do is a retry that does nothing.
  • Owed items from settings plan 4 (branch zdd-owed-items): - Closed: a cold wake's build-state record is now written at the exec, after the replica-key gate and the cgroup refusals, so a refused launch leaves no record. The exec is decided again after the write, and a nested seed writes its obligation before its record (Astra's review of the branch, P1 and P2). This was owed before O8 (replica keys) is switched on. - Closed: release.ResolveLivePath now calls release.ResolveLiveLink. The two had identical bodies. - MOVED to #250: leftover consumed/ files refuse the claims census. See "Still owed (code)".
  • Claims-on swaps remain a future feature. A box with host claims on declines the flag with the reason code host-claims-on.

The flag stays off on any box that has not passed its gate, and only Paul closes items on #272.

Correctness, from Astra's plan 5 review (CODEX-FINDINGS-ZDD-IMPL-5*.md)

  • N5 to N7: gaps in the hold-ran marker. N7 is a disagreement over how it treats an existing failure record. Closed on zdd-flag-readiness (b402bc0e, 42865b81; N6 with 0d6029b3, 3e875ff2; final review round 1: 1e842f89, c327f703, e72e68d2; final review round 2: fb55d199), pending merge. That covers markers keeping records, blocking the worker fallback and signed-rollback provenance, with marker = installed = good boots (ruling 205). N6 and the retry ordering use a persisted book sequence and per-execution markers, so a pre-retry outcome never undoes a retry. The round 1 fixes: an execution's identity carries the agent process's incarnation, so a run after a restart is never taken for the run before it, and a rollback or retry that matched the old run clears nothing of the new one (1e842f89); a retry that removes only a marker still orders a late failure of an execution that began before it (c327f703); and a host retry of a build the instance does not want yet keeps its canary for that build (e72e68d2). Round 2 found e72e68d2 kept only one entitlement per host, so a retry of a second restricted build took away the first build's canary, and the first build could then never be retried again (the book already listed the host as retried for it). Since fb55d199 entitlements are kept per host and build: each is spent only by the adoption onto its own build, comes back if that adoption is proven to have run nothing, and is dropped once the book no longer lists the host as retried for that build.
  • F5: failure-record durability across a failed write followed by a restart. Closed on zdd-flag-readiness (8836e521: write-ahead intents, schema-2 book, startup reconciliation; f46d3142, 4416872e: a shutdown join so planned restarts leave no intent), pending merge. A box that declined the flag also reconciles an armed run's intents into records, but its save of the reconciled book is an attempt: a failed save is logged and the box carries on with the records in memory, and reconciles the intents again on its next start.
  • Ruling 206: an uncertain nested start is a hold failure. Closed on zdd-flag-readiness (c453a798), pending merge.
  • Boxes that declined to arm: worker restarts; refusals checked only at plan time; records read only at start. Closed on zdd-flag-readiness (fa7d7374, e32ac0c6, e327ae0e; final review round 1: 076bd9c0), pending merge. A declined box now gates every launch at exec time, including workers, legacy sidecars, claims and move launches. A move launch runs on a row the move holds (landing, restoring), and until 076bd9c0 such a row read as unrestricted; it is now restricted as its ready row is, and a signed retry that arrives while the row is held is applied rather than dropped. It honours markers, and signed retries and the pointer pass. Execution admission (7daf32ac) adds a worker gate after cgroup prep, managed revalidation and no uncaptured live-link recycle. Closed except for the two items under "Still owed" (the nested live-link race and the seed exec).
  • The 15-item list in round 3, section 5. These code defects on the flag-on path are closed on zdd-flag-readiness, pending merge: policy cache, bounded reads and acks (f46d3142, 4416872e); clean one-shot good (but see "Still owed": its good publication does not yet check ownership or fencing), the WakeWorker budget and the halt event (5ad55485); GoodVersion bucket retention and the hold build published before its incumbent stops (d3d464cd); the stage relay give-up (2c5adbe8); transient status (98c1763d, 61b83e27). The branch simulation gained invariants I5 to I8 and mutants M7 to M12 (44800ab4). The 3000-seed sweep passed and M1 to M12 were all caught; it was run again after the final review round 1 fixes, which changed gate logic, on 3ee4f1d1: both baselines passed and M1 to M12 were all caught again. It was run once more after the final review round 2 fix, which changed queue admission, on fb55d199: both baselines passed and M1 to M12 were all caught. It does not cover the restart and replay sequences of round 1's findings 1 and 3, which have unit tests instead. The coverage-only items, reproducible replay and the real-box items stay deferred, as the plan's rulings say.

Carried from plans 2 and 3

  • With the flag off, a retained wake is proven by its leader's exit only; forked descendants are not checked (F6, pre-existing). The failed-wake half is closed on zdd-flag-readiness (7c0f2543, 23d42f0f; final review round 1: bbb7fc8f), pending merge: a failed wake now proves its host cgroup empty, deciding from the layout its launch used rather than the layout the row reads at cleanup (bbb7fc8f). The failed-seed and normal-stop halves are still open, see "Still owed".
  • Leftover executable files are not swept from /run. Closed on zdd-flag-readiness (7c0f2543, 23d42f0f), pending merge.
  • Claims-backed swaps are refused (P2-9). Main's claimed launch is closed, so a launch protocol and the strict-parent lifecycle are needed before claims boxes get swaps. Handled on zdd-flag-readiness (cd7de723) by declining the flag on a claims-on box, see the claims reply below. Claims-on swaps themselves are a future feature.
  • Strict hosts get no swaps until a claim path exists (P2-8).

Dev-box checks (need Paul's approval to run)

  • D1: daemon-reload cost at titogo-scale unit counts. Above about 1 s per swap means transient units instead of drop-ins. (Gate E below runs this measurement.) Done: well under 1 s, so keep the drop-ins.
  • The plan 2 kernel cgroup suite on arm64 with User=carlos.
  • Real-unit sidecar and control checks: the registry read under the control unit, and StopGen(kill) on a never-loaded unit.
  • A guard against legacy cgroup parents that still hold processes. Code fixed on zdd-flag-readiness (2c5adbe8), pending merge. See "Dev-box flag blockers" below for what is still owed on the dev box.

Dev-box flag blockers

Two defects blocked the flag on the dev box. Both are fixed, merged and released, and both kernel tests passed on a real kernel on 2026-10-14 (see "Kernel checks" at the top of this section).

  • The legacy-parent guard (2c5adbe8). Its kernel test is TestGenerationStartRefusedBesideADirectParentMember.
  • The cgroup.kill respawn (c76a1d44, 66c92efe). A cgroup that took cgroup.kill is recreated, and KillHost and KillSidecar respawn. Its kernel test is TestClaimContainmentCleanupRespawn.

Surfaces (plan 6)

  • Copy review of all event and refusal wording is closed. Paul approved all 149 user-facing strings on 2026-10-11. The approved text is pinned in docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-copy.json and guarded by approved_copy_zdd_test.go. That includes the WakePlan.Refuse texts, "restart refused: rolling restarts are unavailable on this box", the refused-wake texts, and main's "sidecar could not be restarted. retrying" line, which now fires more often.
  • A member-safe wake reason for ErrBuildNotPublished (plan 4 R22).
  • carlos sidecars ls shows ServingVersion.
  • Roll notes: boxes first, then console policy, then dev, then bab/adopt-probe under load with zero failed requests, then the fleet. (This file.)

Added by plan 6's final review (not in #272)

This item comes from Astra's final review of this branch, round 1, not from the discussion. The controller parked it as a flag-on blocker, so it belongs on #272 too.

  • Host retries ride the unsigned restart-scope log; a box that can write the bucket can clear another box's failure hold (Astra plan 6 R1 #5, plan Q4). Superseded on zdd-flag-readiness-2 (6e9e84bd): grants ride host-retry.json, not the restart-scope log. Paul's ruling (2026-10-11): host retries stay on the unsigned restart-scope log for now, with the same trust as carlos restart. Signing retries in the console, as carlos promote --retry is signed, is owed before the flag goes on anywhere hold mode is used. Paul is adding this to #272. Closed on zdd-flag-readiness (d68a1dde; final review round 1: c12c35c3), pending merge: the console signs a host retry as a grant bound to the restriction's epoch and to the identity of the box's failure book, and every box ignores an unsigned retry. The book identity (c12c35c3) stops a grant for one book's restriction clearing another book's at the same epoch. Every open of the book also advances its sequence by 2^20 (about a million), so a run does not hand out again an epoch an earlier run handed out but never saved. That is a bound, not a guarantee: an epoch could be reused only if an earlier run made more than a million book changes without one successful save, and nothing enforces that limit (see "Still owed"). An armed agent saves the advance when it opens the book, or refuses managed starts. A box that declined the flag keeps the advance in memory until its next write of the book, and that write is an attempt: a failed save is logged and the box continues. A console with no signing key answers 503, and the text of that refusal is new copy awaiting review (copy-review/strings.json). Superseded on zdd-flag-readiness-2 (7c7c6141, pending merge): the 2^20 advance and its "bound, not a guarantee" caveat are gone, replaced by the durable per-app generation rule (see "Still owed (code)").
  • provision.ListRecords silently skips unreadable instance records (Astra plan 6 R3). A relevant fleet can then drop out of the deploy watch's completeness check, so a sleeper on it may be URL-polled when its fleet snapshot is also unreadable. This is a flag-on blocker. Owed: expose record-read completeness to the instance list (docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-6-R3.md). Closed on zdd-flag-readiness (f9be4718), pending merge.

Reply: four claims-on gaps

Plan 5 was merged dormant as d905a67, after merging main d6f66ef3 (#268 lease margin, region-HA). Astra's merge review (CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md) said ACCEPT WITH FIXES for a dormant merge, with no flag-off defect.

Main's RoleLaunch.Root is wired in production, so claims-on launches can execute. When this reply was written, the flag could not go on until the box either declined arming with host claims on, or all four of these were closed:

  1. Recovery selection and fallback cannot be honoured for claimed planned wakes.
  2. A claimed hold launch can lose its execution evidence and outcome fence.
  3. Retained claim processes are missing from the generations file.
  4. Successful claimed wakes cannot record their good build or serving version.

That reply found no claims check in swapArming (internal/host/rollout.go:145-156).

Conclusion (2026-10-12): decline, do not close. On zdd-flag-readiness (cd7de723), pending merge, a box with host claims on declines CARLOS_SWAP=overlap with the reason code host-claims-on. It runs today's stop-then-start paths. The four gaps above are not closed. They become the future feature "claims-on swaps", which is not scheduled. The source for the four gaps is CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md. After Gate E on any claims-on box, the operator sees a dash under SERVING and, in REASON, "its box cannot run two builds side by side, so it restarts instances the old way". The code host-claims-on is in the instance's JSON record (rollout.unavailable) and in the agent log line REFUSING CARLOS_SWAP=overlap (host-claims-on).

Still owed (code)

Current state, 2026-10-14. Readiness rounds 1 to 3 are merged and released, so every item this section listed after zdd-flag-readiness is closed on main. Rev 19 (ebddcfc, v0.27.2) and #292 (e15ad6b, v0.27.3) are merged and released too. The kernel checks that were owed here are closed (see "Kernel checks" at the top of "Before Gate E: owed").

Closed, with the commits that closed them:

  • The nested live-link race on a declined box (903ed7ca, c910d020). A declined box pins the live link once, and nested starts and seeds exec the pinned build.
  • Seeds that exec'd the live link ungated, and the failed seed's proof (903ed7ca; 4c5e43b5, 5e51c008, 4771d147). A seed's processes are proven gone before its database goes, and an unproven seed leaves a durable obligation.
  • The stranded legacy sidecar (c910d020). A refused sidecar restarts on the instance's own build.
  • The one-shot's good build and WakeWorker's duplicate wake (deb4c811).
  • The flag-off normal-stop half of F6 (d6996537, ac9f3fef, 097ca930, 79032948).
  • ensureParent's uncapped paths (98593909, 8530210e). A generation whose limits cannot be applied does not start (cgroup.ErrLimitNotApplied). Its kernel positive control passed on 2026-10-14.
  • The declined box's hold row and Retry button (6e6ad000, 65d4e135). Paul approved the seven strings on 2026-10-13.
  • The rev-18 spec amendment for console-signed host retries, approved by Paul on 2026-10-13. The design branch's spec still has to be raised to rev 18 there.
  • The reopen epoch, now the hard rule (7c7c6141, 191c12b9, 692d1533, a6eb8b35). The proof and its two assumptions are on FailureBook (internal/rollout/failures.go).
  • Mixed-version retry sequencing (6e9e84bd, 9f27cdfa, 4d8957c1, 18bde560). Grants ride their own object. Dev round 4 checked the transitions; the one not run is covered by construction (see "Mixed-version retries" at the top of "Before Gate E: owed").
  • Readiness round 2's dev blockers, closed by readiness round 3: the exec-shim for RestrictNamespaces=yes, the cgroup-placement-failed arming check, cmd/zddprobe and the TestStrictKernel fix.
  • Task 10's grant consumed while the row is moving or landing (076bd9c0).
  • From settings plan 4, on branch zdd-owed-items: a cold wake's build-state record is written at the exec, after the replica-key gate and the cgroup refusals, so a refused launch leaves no record (owed before O8). The exec is decided again after the write, and a nested seed writes its obligation first.
  • Also from settings plan 4, on the same branch: release.ResolveLivePath calls release.ResolveLiveLink.

Still open:

  • Claims-on swaps (the four gaps from CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md). A future feature, not scheduled. A box with host claims on declines the flag.
  • MOVED to #250. Leftover consumed/ files refuse the claims census. Root's request record /opt/carlos/consumed/<host> and its fence lock .<host>.lock outlive the installation, and the empty-installation census refuses both. It also refuses another host's record, which is an unclassified file. platform-1e owns it now, under Plan 14. Root's retirement removes the consumed file, its lock and the record atomically, so a re-install passes the empty-installation census. A runner test covers install, retire and re-install. It is owed before host claims are switched on, and does not block the flag.

Roll notes from zdd-flag-readiness

  • Retries: boxes before consoles. Superseded by the next bullet. Kept for history: on zdd-flag-readiness a retry rode restart-scope.json, and an older binary that appended to it stripped signed from pending retries. That fails closed, but a box from before the branch applied a retry with no book or epoch check, so the roll order mattered. Since zdd-flag-readiness-2 it does not (see below).
  • Retries: supported transitions (zdd-flag-readiness-2, 6e9e84bd, 9f27cdfa, 4d8957c1, 18bde560, pending merge). A signed host retry is written to apps/<acct>/<app>/host-retry.json, not to restart-scope.json. No released binary reads or writes it, and restart-scope.json no longer takes a retry or a signature. A retry is released nowhere before this branch, so there is no released retry behaviour to preserve. The transitions: - Old console, new box. A released console has no host retry at all, so the CLI and the page offer none until the console is updated. Its restarts rewrite restart-scope.json only, which holds no grant now and so cannot lose one. - New console, old box. The old box's report carries no retry_v and the old box never reads host-retry.json. The console refuses the retry with a message (409 on the API, a flash in the browser, no Retry button on the page) and writes nothing. Update the box, then ask again. - New console, new box, book not saved. A held host at epoch 0 is refused with "the box has not saved its failure yet". Ask again in a minute. The agent retries its own save. - New console, new box. The console signs a grant for the epoch and book the box reported, the box verifies it on the next serial move, and a grant gone stale is a no-op. - A grant written before an agent restart is checked again after it (grants are not seed-skipped). A current one applies, a spent or stale one does nothing. - A dev box on an intermediate build (between 5da1cc9b and this branch) loses the retry path until it is updated. It is unreleased, which is why it is acceptable. - A console with no signing key answers 503 and writes nothing. - A box that declined the flag and keeps a failure book honours a grant too (no canary: its next wake starts the build as today).
  • Schema 3 failure book. The failure book file is written as schema 3 (it was 2 on zdd-flag-readiness). Downgrading an agent with the flag on makes that box refuse managed starts, and that includes a round 1 agent (on main since 8ebd9d8c), which accepts only schema 1 and 2. Recover by upgrading the agent again. Do not delete the file to get past the refusal: it holds the failure records and hold-ran markers that keep instances off builds that failed or ran on them. A file of schema 1 or 2 is adopted on first open and given a new book identity, so grants issued against the old identity clear nothing.
  • Copying or cloning a data dir: drop the book key. The failure book's identity is in the file's book key. A copy that keeps it gives two boxes one identity, and a copy restored from an older snapshot can hand out again a generation already published. Before the flag goes on, any path that clones, restores or relocates an agent data dir (a rebuilt box from an image or snapshot, a hand copy) must delete the book key from swap-failures.json in the copy. The next open then makes a new identity.

Audit (fix wave 1, item 7, at 929dfc39). The book is /data/swap-failures.json (CARLOS_DATA_DIR, default /data). Every path found in this repo and in platform-infrastructure (at 841e94b) that copies, snapshots, restores, rsyncs or images agent data:

| Path | What it carries | Can the book travel? | |---|---|---| | Box rebuild or replace (infra/modules/region-host, tofu apply -replace) | Stock Debian AMI (data.aws_ami); root volume delete_on_termination = true; no other volume | No. The new box's /data is empty, and the first open makes a new identity. | | Cold-standby box (platform-infrastructure/dev/standby-ie.tf) | A fresh box that converges from the bucket like any new box | No. Same as a rebuild. | | Box init (host-init.sh, host-init.sh.tftpl, scripts/bootstrap-box.sh) | Creates /data and fetches the binary | No. Nothing is copied into /data. | | Instance move (internal/host/movespass.go FilesDir, internal/moves/archive) | /data/<host>/ and /data/<host>.env only | No. The book is at the /data root and is not in the archive. | | Litestream replication and carlos ops restore-verify | *.db files | No. The book is JSON, not a database. | | Hibernate wake restore (Restore, internal/lease), claims cold restore (internal/hostbox ColdRestore) | One instance's database | No. | | Layout migration (internal/layout) | Instance directories inside one box's /data | No. Same box, same file. | | Box volume records (internal/boxvolumes) | Records of EBS mappings; nothing is reattached | No. A terminated box leaves no disk (region HA spec 1 §5.9). | | Platform update (scripts/update-platform-binary.sh, carlos system update, carlos-system-recover.service) | /opt/carlos binaries | No. |

No path carries the book, so no code change was needed. Not audited, and outside these two repos: carlosframework/carloku-infrastructure (the flagship root since 2026-10-01) and teamtito/carlos-infrastructure (RUNBOOK-edge-failover.md), which have no local checkout here; AWS Backup or DLM policies made outside tofu (none appear in either repo); and any copy made by hand. Each of those must drop the book key as above before the flag goes on.

  • An armed box without CLONE_INTO_CGROUP refuses every generation start (98593909, 8530210e). Superseded on zdd-flag-readiness-3: the Manager no longer uses clone3 and places through the shim, so the shipped unit no longer causes this refusal. See "Roll notes from zdd-flag-readiness-3". The rest holds, with "the shim cannot place" in place of "clone3 is unavailable". A generation start must be placed in its cgroup before exec. If clone3 placement is unavailable (the fallback has latched, or the cgroup cannot be opened), the start is refused with cgroup.ErrLimitNotApplied and is never done by moving the PID after exec. Wakes of that box show the maintenance text, and the refusal is the box's condition, not a failure recorded against the build; a swap abandons its attempt and is retried, as for the populated legacy parent below. The flat (flag-off) start is unchanged. Recovery: turn the flag off and restart the agent, or fix the kernel or cgroup state. The same refusal covers a parent that cannot be made, does not delegate memory, or whose backstop or child limit cannot be written. The log line begins cgroup: and says "generation start refused".
  • The seed obligation file <db>.seed-unproven (5e51c008, 4c5e43b5, 4771d147). Before a seed runs, the agent writes <db>.seed-unproven beside the instance's database (three lines: carlos seed obligation v1, db=owned-by-seed or db=there-before-seed, seed=flat or seed=nested). A write that fails refuses the seed before anything runs. The file is removed only when the seed's processes are proven gone. After a failed seed, a database the seed created goes with it; a successful seed's discharge keeps the database. While the file is there no wake treats the database as warm, and a wake that cannot prove the seed gone fails with activator: seed unproven; processes remain and holds the lease to its TTL. The obligation is a file, so it survives an agent restart. It does not end in .db, so the litestream glob never takes it. A box on a kernel without cgroup.kill proves a seed by its leader's exit alone (logged).

Normally do nothing by hand. Once the processes are gone, the next wake proves it, discharges the obligation itself, removes a seed-owned database and seeds again. The agent removes the database (<db>, <db>-wal, <db>-shm) first, syncs the directory, and removes the obligation last, so a crash or a wake in between never sees a database without its obligation. Prefer that path: clear one by hand only for a proof the agent cannot make.

A hand clear must not race a wake. Between your proof and your delete, a wake can discharge the obligation itself and start a new seed (Restore, settleSeedObligation), and you would then delete that new seed's database. There is no per-host way for an operator to hold off wakes. The activator's maintenance fence (FenceAndStop / LiftFence, internal/activator/maintenance.go) is used only by the move worker, the layout pass and the claim-stop coordinator, and nothing exposes it to an operator. A layout journal (<data>/layout/<host>.json) fences wakes too, but the layout pass reads and acts on it, so do not plant one. A row status other than ready is set only by the move worker and does not stop a wake already in flight. So clear an obligation by hand only with the agent stopped. Stopping carlos-edge stops every instance on the box, so plan for a short box-wide outage:

1. Stop the agent: systemctl stop carlos-edge. Wait until it is inactive. No wake can run until it starts again. Its delegated cgroups and the units bound to it (carlos-instance-seed@, carlos-instance@, carlos-instance-gen@, carlos-instance-sidecar@) stop with it. 2. Prove the seed's processes gone. The host's carlos-exec@<host> cgroup (and for a nested layout the host slice's seed unit) must show an empty cgroup.procs and populated 0 in cgroup.events, or be absent, and no process may still hold the database open. 3. Only when the obligation's body says db=owned-by-seed, delete <db>, <db>-wal and <db>-shm. 4. Sync the directory (sync "$(dirname <db>)", which fsyncs it) so the deletes are on disk before the obligation goes. 5. Remove <db>.seed-unproven last, never first. Deleting it first leaves a half-built database that the next wake serves as warm. Sync the directory again. 6. Start the agent: systemctl start carlos-edge.

A body of db=there-before-seed, or one the agent does not recognise, keeps the database, and the next wake serves it warm, by design. Never delete the obligation to get past a refusal while a process may still be writing the database. A planted non-empty directory at that name keeps that tenant's row down until it is removed by hand.

  • The armed agent writes its failure book on start. Since c12c35c3 an agent with the flag on saves the book when it opens it, even when there was no file, and refuses managed starts if it cannot (the same refusal as an unreadable book). A box whose data dir is read-only will show that.
  • A populated legacy parent cgroup. A generation start is refused while carlos-exec@<host> holds a process directly (2c5adbe8). The agent logs a line beginning cgroup: <host>: generation start refused: the parent cgroup holds N process(es) directly, wakes of that host fail, and swaps abandon and are retried every 15 s. Nothing lifts this automatically. Clear it by hand: kill the processes listed in that cgroup's own cgroup.procs (under the agent's delegated cgroup), or restart carlos-edge, which kills the delegated subtree. The next wake or retried swap then starts normally.
  • carlos-instance-control is not restarted by the old script. platform-infrastructure/scripts/update-platform-binary.sh does not restart it. Use carlos system update (main's system apply restarts it, 0bb57a31), or restart it by hand after the old script.

Roll notes from zdd-flag-readiness-3

Branch zdd-flag-readiness-3, from main at 34a4328f, on amadan discussion #272. Built and task-reviewed on the branch: not merged, not released, not rolled, and only Paul closes it on #272. The whole-branch review and CI come after these notes. Nothing here was run on a real box.

  • The four round-2 blockers, and where each is closed. Paul's dev check on 2026-10-13 ("Dev-box checks, round 2" on #272) found four things that stopped the flag. They are closed in code on this branch, pending merge: 1. RestrictNamespaces=yes on carlos-edge makes systemd's seccomp filter answer clone3 with ENOSYS, so CLONE_INTO_CGROUP never worked under the shipped unit and every generation start was refused. Closed by the exec-shim (ff4b814a, a34013c6), by routing every Manager start through it (961aed71), and by a kernel test under the edge unit's sandbox lines (891c0fb7). 2. Arming did not check that placement works, so a box that could not place armed anyway and then failed every generation start. Closed by an arm-time check (7fbafe7e, 1f9d6a83): see the decline code below. 3. The roll notes' probe app rejected the --db argument the platform passes, so it exited at once. Closed by cmd/zddprobe (59b1895a). 4. TestStrictKernel had a defect of its own and could not pass in its full mode. Closed by 496fb6ed.

The minor from the same check, failed generation units lingering in systemctl --failed after the control service stopped them, is closed by 496fb6ed too: the control service now runs reset-failed on a generation unit after a stop or an abandon-kill that proved it gone. It is best-effort. A failure is logged as instance-control: reset-failed <unit>: ... and the stop still answers 204.

  • The shim. A spawn starts /proc/self/exe __cgexec. The shim places itself in the target cgroup, then execves the tenant. The tenant's path, argv and environment travel over a pipe, never in the shim's own argv or environment, so the PID the agent tracks is the shim's and then the tenant's. Paul's ruling (2026-10-13): no line of internal/host/units/carlos-edge.service is relaxed. RestrictNamespaces=yes stays, and so does every other sandbox line. The unit file is not edited on this branch. The cost is one extra exec of the carlos binary per spawn, measured on DOCS-T1 at about 4.5 ms (median 4.9 ms against 0.42 ms for a direct start).
  • Behaviour change on roll, flag off. Every exec child on a rolled box now starts through the shim, flag on or off, not only generations. Healthy looks like today: no new log lines at all. In particular the agent log has no placement before exec failed line and no PID-move line. Wakes answer as before, a few milliseconds slower.

A failing shim on a flat (flag-off) start logs:

``text cgroup: <host>: placement before exec failed (<why>) — using the PID-move fallback for this spawn ``

and the spawn goes ahead by starting the process and then moving its PID into the cgroup. That is exactly what every flag-off start does on the shipped unit today, so the box still serves. Only a placement the shim proved happened before exec falls back. A start that may have run is reported started and never retried, so a tenant never runs twice. If you see the line on a healthy box, stop and look: the shim is not working there and the box is on the old path.

A generation start never falls back. The same words end with — generation start refused, the start answers cgroup.ErrLimitNotApplied, and the refusal is the box's condition, not a failure recorded against the build. The "armed box without CLONE_INTO_CGROUP" note above is replaced by this one.

A start whose own context ended while its shim was being placed (a seed's wake that timed out or was stopped) is not a shim fault. It logs:

``text cgroup: <host>: the start's context ended during placement (<why>); nothing ran ``

and answers the context's own error, as a direct start would. Flat or generation, it does not fall back and is not the box's condition. The member's wake event reads as such a wake always read (wake failed: seeding <host>: context canceled).

  • The shim keeps the agent's ignored signals. A tenant started through the shim has the same ignored signals as one started directly: the agent sends its current set (SigIgn in /proc/self/status) with the spawn, and the shim ignores those again just before it execs the tenant. Without that, the shim's own Go runtime would have reset every ignored signal but SIGHUP and SIGINT to its default. This changes nothing on a box whose agent ignores no other signal.
  • The new decline code cgroup-placement-failed. At arm time (the agent's start with CARLOS_SWAP=overlap) the agent starts a real shim placement into a scratch cgroup named carlos-placement-check, under its delegated root, within one 4 s budget for the whole check. If that fails the box declines the flag for every row, nested rows included, and logs:

``text agent: REFUSING CARLOS_SWAP=overlap (cgroup-placement-failed): a test placement into a scratch cgroup failed (<why>), so generations could not be started inside their limits; this box stays on today's stop-then-wake paths ``

The code is in the instance's JSON record (rollout.unavailable) and uses the approved declined wording, so there is no new copy. A box whose check fails declines in about 4 s, inside the 20 s carlos system update gate. The 4 s bounds the check's own waits. After them the agent kills and reaps what it started, which takes milliseconds when the kernel ends a killed process promptly. It does not when that process is stuck in uninterruptible kernel I/O: SIGKILL ends it only when the I/O returns, and the decline waits for that. So on a box whose disk or cgroupfs is stalled, the decline can take longer than 4 s, and the update gate can time out. A box that passes logs agent: overlap swaps armed (CARLOS_SWAP=overlap) as before.

Mixed versions. Consoles and CLIs roll after boxes, so for a while a rolled box can decline with a code the others do not know. A console older than the box shows the unknown-code phrase for cgroup-placement-failed. It treats an unknown code as not declined, so it cannot offer or sign a host retry for that box until it is updated. An older carlos CLI's deploy --watch does the same: it does not follow the box as declined, so it shows that box as pending until the watch times out (classifyRollout and follows, deploywatch.go). The deploy itself is unaffected. Update the console and the CLI and ask again.

  • Claims-on swaps: a new precondition. The strict claims path (PrepareStrict, PrepareStrictParent, (*Prepared).Start) keeps its clone3 placement, by Paul's ruling. Under the shipped unit clone3 answers ENOSYS, so on a box with host claims on every strict launch answers cgroup.ErrStrictUnsupported. It is refused, never run uncapped. Claims are off everywhere today and a claims-on box declines arming (host-claims-on), so nothing changes now. But claims-on swaps, a future feature, cannot work until the strict path gets the shim too. Count that as a precondition of them.
  • Dev re-check for Paul, replacing round 2's blocked steps. Needs Paul's approval, naming the box, as Gate E does. In order: 1. Roll the build to the dev box with the flag still off. Wake an instance. It answers 200, and the agent log (journalctl -u carlos-edge) has no placement before exec failed line. 2. Set CARLOS_SWAP=overlap in /etc/carlos/host.env and restart carlos-edge, as Gate E does. The log shows agent: overlap swaps armed (CARLOS_SWAP=overlap) and no REFUSING line. If it shows (cgroup-placement-failed), stop and read the cause in the line. 3. Run the probe (built from cmd/zddprobe, see "The probe") under load through zdd-1 to zdd-3. The pass is 0 failed requests and Gate E's other conditions. 4. Run the probe's hold control. It must show the pause. 5. After the swaps, systemctl --failed on the dev box lists no carlos-instance-gen@ unit that the control service stopped. 6. Check on the real unit the edge-unit lines the kernel test could not. Task 4's test ran in a user unit on DOCS-T1, not the shipped system unit. Its report marks these untested there: ProtectProc, ProtectClock, RestrictRealtime and SystemCallArchitectures. ProtectSystem was observed only as a read-only /usr, so strict was not told from full. NoNewPrivileges is implied by the seccomp lines. These it excluded, because only the real unit has them: User, Group, AmbientCapabilities, CapabilityBoundingSet, ReadWritePaths, Slice, OOMPolicy, and Delegate=cpu memory io pids (the user manager gives cpu memory pids, no io). The user unit also adds a user namespace the real unit lacks. On dev, read what systemd applied with systemctl show carlos-edge -p ProtectProc -p ProtectClock -p RestrictRealtime -p SystemCallArchitectures -p ProtectSystem -p NoNewPrivileges, and treat steps 2 and 3 passing as the proof that placement works under the unit as shipped, as carlos with its real ReadWritePaths and slice. 7. Run the two activator kernel tests that now start through the shim and could not run on DOCS-T1, because user delegation there has no io controller: TestClaimContainmentCleanupRespawn and TestGenerationStartRefusedBesideADirectParentMember. They are owed on the dev box. Both skip unless CARLOS_CLAIM_SPAWN_KERNEL=1, and both call cgroup.Setup, which needs a delegated cgroup offering cpu, memory, io and pids; without one they fail with missing delegated cgroup.

Build the binary for the dev box from the checkout root on DOCS-T1, and copy it to the dev box by the route the roll uses for binaries:

``bash GOOS=linux GOARCH=arm64 go test -c -o activator.test ./internal/activator ``

On the dev box, in a fresh directory under /tmp (the activator's socket paths need a short root), first the two shim checks, with the shipped unit's RestrictNamespaces=yes:

``bash d=$(mktemp -d /tmp/zddk.XXXXXX) install -m 0755 activator.test "$d/" sudo systemd-run --wait --pipe --collect -p Delegate=yes -p User="$(id -un)" \ -p RestrictNamespaces=yes -E TMPDIR="$d" -E CARLOS_CLAIM_SPAWN_KERNEL=1 \ "$d/activator.test" -test.run '^TestGenerationStartRefusedBesideADirectParentMember$' -test.v -test.count=1 sudo systemd-run --wait --pipe --collect -p Delegate=yes -p User="$(id -un)" \ -p RestrictNamespaces=yes -E TMPDIR="$d" -E CARLOS_CLAIM_SPAWN_KERNEL=1 \ "$d/activator.test" -test.run '^TestClaimContainmentCleanupRespawn$/^Kill' -test.v -test.count=1 ``

The pass is --- PASS for TestGenerationStartRefusedBesideADirectParentMember and its sibling_control and recovery_control, and for KillHost_then_respawn and KillSidecar_then_respawn, with no --- SKIP. These use a system unit run as you (sudo, -p User=) because systemd delegates io to it. Where the user manager offers io too, the same commands as systemd-run --user, without sudo and -p User=, do as well; where it does not, they fail with missing delegated cgroup.

Then the strict subtest, on its own and without RestrictNamespaces:

``bash sudo systemd-run --wait --pipe --collect -p Delegate=yes -p User="$(id -un)" \ -E TMPDIR="$d" -E CARLOS_CLAIM_SPAWN_KERNEL=1 \ "$d/activator.test" -test.run '^TestClaimContainmentCleanupRespawn$/^strict_primitive$' -test.v -test.count=1 rm -rf "$d" ``

strict_primitive calls (*Prepared).Start, the strict claims path, which keeps its clone3 placement by Ruling 3. Under RestrictNamespaces=yes clone3 answers ENOSYS, so that subtest is expected to fail there (cgroup.ErrStrictUnsupported). That is the claims-on precondition above, not a regression, and it is why it runs without the line. Run as one -test.run '^TestClaimContainmentCleanupRespawn$' under RestrictNamespaces=yes, the test fails on that subtest alone.

  • Kernel tests on DOCS-T1. They skip unless asked for, and they need a delegated unit. From the checkout root, build the test binary into a directory of its own, run it in a transient user unit, and remove that directory and nothing else:

``bash d=$(mktemp -d "$TMPDIR/zddk.XXXXXX") go test -c -o "$d/cgroup.test" ./internal/cgroup systemd-run --user --wait --pipe --collect -p Delegate=yes \ -E TMPDIR="$d" -E CARLOS_GEN_KERNEL=1 -E CARLOS_SHIM_KERNEL=1 \ "$d/cgroup.test" -test.run '<regex>' -test.v -test.count=1 rm -rf "$d" ``

Run the shim, generation and placement-check tests a second time with -p RestrictNamespaces=yes, the shipped unit's line, and leave TestGenStrictKernel out of that run: it is the strict path, which keeps clone3 (Ruling 3). User delegation offers cpu memory pids, not io. Task 4's edge-hardening test is a driver that starts its own units, so it runs outside any unit:

``bash mkdir -p /tmp/k/zddr3 && go test -c -o /tmp/k/zddr3/cgroup.test ./internal/cgroup (cd internal/cgroup && CARLOS_EDGE_HARDENING_KERNEL=1 TMPDIR=/tmp/k/zddr3 \ /tmp/k/zddr3/cgroup.test -test.run '^TestEdgeHardeningPlacement$' -test.v -test.count=1) rm -rf /tmp/k/zddr3 ``

This test needs a TMPDIR under /tmp, so it can check PrivateTmp. It fails with "needs TMPDIR under /tmp" otherwise, which is why this command does not use the on-disk $TMPDIR of the recipe above.

TestStrictKernel takes a mode in CARLOS_STRICT_KERNEL and a proof directory in CARLOS_STRICT_PROOF_DIR. This is the full mode. It runs in a delegated unit without RestrictNamespaces, because the strict path keeps clone3:

``bash mkdir -p /tmp/k/zddr3/strictproof && go test -c -o /tmp/k/zddr3/cgroup.test ./internal/cgroup && \ systemd-run --user --wait --pipe --collect -p Delegate=yes \ -E TMPDIR=/tmp/k/zddr3 -E CARLOS_STRICT_KERNEL=full \ -E CARLOS_STRICT_PROOF_DIR=/tmp/k/zddr3/strictproof \ /tmp/k/zddr3/cgroup.test -test.run '^TestStrictKernel$' -test.v -test.count=1 rm -rf /tmp/k/zddr3 ``

Real-box checks (need Paul's approval; not run by this branch)

Readiness round 2 adds these to the list. None of them was run. The kernel suites skip unless CARLOS_CLAIM_SPAWN_KERNEL=1, and the temp-dir cgroups of the unit tests cannot stand in for a real cgroupfs.

  • Task 7's kernel tests, in internal/activator/genparent_kernel_linux_test.go: TestGenerationStartRefusedBesideADirectParentMember's "sibling control" (g1 starts and stays up, is capped in g1, and g2 starts beside it) and "recovery control" (kill the legacy process, the parent empties, g1 starts). The live-g1 respawn control is killSidecarRespawn, in internal/activator/claimspawn_kernel_linux_test.go, run by TestClaimContainmentCleanupRespawn: a live g1 survives each sidecar respawn and the parent's cgroup.procs stays empty. The same checks need to hold with the placement the box really has. Since zdd-flag-readiness-3 that is the shim, not CLONE_INTO_CGROUP, and the refusal to watch for is cgroup-placement-failed at arming or generation start refused at a start.
  • Task 8's seed and normal-stop kernel control: a forked helper killed by cgroup.kill before the lease goes, for a flat seed, a nested seed unit and a flag-off stop. No kernel test of it exists yet. KillAndEmpty, the wake-time EmptyHost and a nested seed unit stopped by root were only modelled through seams.
  • The real-unit control and the sidecar checks (a unit route and the sidecar members on dev, including a refused sidecar restarting on the instance's own build).
  • Dev rollout, and a retry across a mixed-version roll on dev: each transition under "Retries: supported transitions", with an old box and a new console, a new box and an old console, and a book that has not saved. Include a data-dir copy with the book key dropped, and a downgrade onto a schema 3 book to see the refusal.
  • The flagship probe's gates E, F and G, as before.
  • Readiness round 3's dev re-check, replacing round 2's blocked steps: see "Roll notes from zdd-flag-readiness-3". It includes the two activator kernel tests below, which now place through the shim and need the dev box's io controller. Its step 7 has the commands, and runs the strict subtest of TestClaimContainmentCleanupRespawn apart, without RestrictNamespaces=yes.
  • Round 1's kernel checks, still as listed next.

TestClaimContainmentCleanupRespawn (now the KillHost and KillSidecar respawn) and TestGenerationStartRefusedBesideADirectParentMember on the dev box (arm64, CARLOS_CLAIM_SPAWN_KERNEL=1, a delegated unit with io; the commands are step 7 of the readiness-3 dev re-check). Also check whether children of a killed kept cgroup, and cgroups systemd kills on carlos-edge stop, are affected.

  • The plan 2 kernel suite rerun.
  • The flagship probe (Gates E to G).
  • D1 is done: well under 1 s, keep drop-ins.

Reply: wiring caution (settings plan 2, merged as 70f8119)

Held staging checks authorization inside stageHeld, so every entry point is covered, including main's claim-path MaterializeHeldUnder.

Whoever wires zero-downtime plan 5 Task 9: when movesProbeOwners.Release is connected to MaterializeHeldUnder, it must pass the parsed CARLOS_RELEASE_PUBKEY and the agent's digest ratchet. A missing key or ratchet refuses every stage (fail closed). That is not a bug to "fix" by passing nil.

Where plan 6 stands against that list (for information)

The discussion text predates plan 6 ("not started"). On this branch the carlos sidecars ls change is built (task 13), and the refusal and wake wording is approved (task 14b). This file is task 17. Only Paul closes the discussion's items.

What plan 6 changed that the roll should check

  • The copy is approved and pinned. Paul approved every new user-facing string on 2026-10-11. The record is docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-copy.json, and approved_copy_zdd_test.go fails if any string in it drifts from the code. Where this file says "the approved line", it means the text in that record.
  • carlos sidecars ls changes with the flag off. It reads ServingVersion and prints "unknown" for a sidecar that has never started anywhere. Plan 4 already changed what the registry Version means for a sidecar, so the old column would lie. Look at it on dev after Gate B and expect different output from the last release. The code is in internal/console/apisidecars.go:121-130 and sidecars_cli.go:368-378.
  • The swap-policy door is person-only. PUT /api/cli/apps/{account}/{app}/swap is registered with the zero role set (internal/console/authcli.go:863, reasoning at internal/console/apiswap.go:1-8). A signed-in member can run carlos apps set. A CI token or any credential the app itself holds cannot. If CI needs to set the policy, that is a one-line role change and Paul's call.
  • The first non-default swap policy creates config.json but restarts nothing (plan Q3, Astra plan 6 R1 #1). Setting the defaults (overlap, 1) on an app with no config.json writes nothing (internal/appconfig/swap.go:47-57). The first hold or concurrency above 1 creates the file with no vars in it. Config sync leaves an instance's config file absent when its bundle renders nothing and it has no instance overlay (internal/appconfig/syncexec.go:1397), so no instance restarts, no config revision reaches the release scheduler and no config ack is written. That is the state every never-configured app's instances are in. An instance that already has a config file converges on its bytes as before. This answers Q3's restart concern; closing Q3 is Paul's call.
  • A host retry rides the unsigned restart-scope log (plan Q4). Superseded on zdd-flag-readiness-2 (6e9e84bd): grants ride host-retry.json, not the restart-scope log; see "Retries: supported transitions" above. carlos instances retry asks the box to retry through the restart-scope log. That log is not signed, and the box's RetryHost (internal/rollout/manager.go:501-515) trusts any request that names a host on that box. Only the app's record and the rollout report gate it, at the console, not at the box. It is on the owed list above as a flag-on blocker. Closed on zdd-flag-readiness (d68a1dde), pending merge: retries are console-signed root grants bound to the host's restriction epoch, and every box ignores an unsigned retry.
  • FailureBook is not rendered anywhere. The box publishes it (internal/status/snapshot.go:268-275) but no carlos surface or console page shows it. It is parked in #272. What carlos shows for a failed or held build comes from the per-host rollout report instead.
  • The deploy watch reads the instance list every 5 s, not every 2 s. It falls back to polling the URL (today's watchVersion) only on evidence that no targeted instance is on a reporting box: the list answers with a 4xx other than 429 (a console with no list for the app), or every targeted instance is on the list, the console read every box's report (the list carries no rollout_incomplete), and none of them has a report. A list cut short for the whole app neither causes nor blocks the fallback. A 5xx, a 429 or a network error is retried every 5 s until the deadline, and the timeout then says the instance list could not be read. So a flag-off deploy against a console whose list is failing times out instead of URL-watching. One deadline covers the list reads, the waits between them and the fallback. A box that refuses to arm is named in the timeout, and its instances are never URL-polled. A box that declined overlap runs today's stop-then-start paths, so its instances are URL-watched, as with the flag off. An instance counted as done goes back to pending if its box later reports a different build installed. The constants are deployWatchListEvery and deployWatchPollEvery (deploywatch.go:140-142); the fallback rule is listAbsent and legacyOnly (deploywatch.go:107, deploywatch.go:413).
  • A dash under SERVING means the build is not reported, for either of two reasons. No rollout report (flag off, or an older console), or a box that set the flag and declined (Unavailable). servingCell is at instances_cli.go:450-455. After Gate E, a dev instance still showing a dash means the box did not arm. Look for the box's reason code and the agent log line REFUSING CARLOS_SWAP=overlap (internal/host/rollout.go:150).
  • The migrations rule. The new "Two builds at once" section of the public CLI page (docs/site/cli.md) and the release notes carry it: add in one release and remove in a later one, and keep the names of scheduled work the same across the two builds. --swap hold and carlos apps set --swap hold are the way out for a release that cannot follow it.

Gate A: release

Needs Paul's approval, naming the release. This gate touches no box.

Plans 1 to 6 are merged to amadan main and main's CI is green. Cut a platform release from that commit, following README "Releasing the platform" (steps 1 to 9). Its release notes (step 5, copy-reviewed) lead with the migrations rule (spec section 9.5) and say the flag ships off. Call the release vX.Y.Z below.

README's "Releasing the platform" section ends with a place for release specific operator notes. Add one for this release naming the flag-on preconditions above, so the person rolling it reads them.

Gate B: boxes, flag off

Needs Paul's approval, naming the boxes: dev (platform.dev.carlosframework.com, channel canary) and the flagship (console.carloku.com, channel stable), which is ie, us, uk and au.

carlos system update --console https://platform.dev.carlosframework.com --channel canary --version vX.Y.Z --yes
carlos system status --console https://platform.dev.carlosframework.com
carlos system update --console https://console.carloku.com --channel stable --version vX.Y.Z --yes
carlos system status --console https://console.carloku.com

Expected: every box reports vX.Y.Z. Sydney is slow, so give it the full gate. With CARLOS_SWAP unset nothing changes: a carlos deploy still prints live https://... lines, and carlos instances list shows a dash under SERVING.

Gate C: consoles

Needs Paul's approval, naming the consoles and the app.

Roll each console after its boxes. The script is platform-infrastructure's roll-console.sh; it is not in this repo (not checked here). Or use direct mode:

carlos ship --app console --kind binary --version <sha> console-linux-arm64
carlos promote --app console <sha> canary/console

with CARLOS_DEPLOYMENT_BUCKET set for both. Then check the door:

carlos apps set adopt-probe --account bab --swap overlap --console https://console.carloku.com

Expected: the approved zdd.apps-set.done line for adopt-probe, overlap, 1: adopt-probe swaps in overlap mode, 1 at a time on each box, then a muted note about backward-compatible migrations. Setting the default writes nothing (see the Q3 note above).

A console that predates the door answers that the app is not in this account or that the console predates carlos apps set. Roll the console and run it again. In direct bucket mode (CARLOS_DEPLOYMENT_BUCKET or CARLOS_DEPLOYMENT_DIR set in the shell) carlos apps set refuses and says so, so run the door check from a shell that is logged in to the console and has neither variable set.

Gate D: docs and changelog

Needs Paul's approval to publish. This gate touches no box.

Run README step 10: go run ./cmd/changelog in a full checkout of main (after git fetch origin --tags '+refs/amadan/closed/*:refs/amadan/closed/*', with gh and amadan signed in), map the release's bullets in docs/site/changelog-links.json, and land docs/site/changelog.json and the links file on a branch. Then run node hack/sync-docs.mjs <checkout> in carlosframework/website and ship the site the way its AGENTS.md describes. hack/sync-docs.mjs and that AGENTS.md live in the website repo (not checked here).

Check that carlosframework.com/platform/changelog lists the release and the public CLI page shows "Two builds at once". The page's source is docs/site/cli.md:330 here; where the live page sits in the website is not checked here. Both must be live before Gate G.

Gate E: dev, the daemon-reload check, then the flag

Needs Paul's approval, naming the boxes: dev box i-05498eada6f0afd23 (the dev console box, eu-west-1) and dev box i-046c7fe43620944bc (eu-west-1), profile kass. Both box ids come from the brief; only the first appears in another repo document (2026-10-06-console-release-defences-plan-4-d4.md), and the second is not checked here. The approval also names the probe's app actions on dev: creating the zdd-probe app, flipping its swap policy to hold and back, and four deploys of it. Do not start this gate while the "Before Gate E: owed" list has an open item that Paul has not waived.

First the index's D1 guard: measure systemctl daemon-reload on a dev box carrying titogo-scale unit counts. Done 2026-10-11 on dev: median 309 to 424 ms, so keep the drop-ins. Repeat it only if unit counts grow a lot; the commands are kept below for that case.

aws --profile kass --region eu-west-1 ssm send-command --instance-ids i-046c7fe43620944bc \
  --document-name AWS-RunShellScript --query Command.CommandId --output text \
  --parameters 'commands=["systemctl list-units --all --no-legend | wc -l","for i in 1 2 3; do s=$(date +%s.%N); systemctl daemon-reload; e=$(date +%s.%N); echo \"$e - $s\" | bc; done"]'
aws --profile kass --region eu-west-1 ssm get-command-invocation --instance-id i-046c7fe43620944bc --command-id <the id printed> --query StandardOutputContent --output text

Expected: a unit count, then three times. If any reload takes more than about 1 s, stop: plan 2's D1 says nested generation units move to transient units in a follow-up before the flag goes on. Otherwise set the flag on both dev boxes:

aws --profile kass --region eu-west-1 ssm send-command \
  --instance-ids i-05498eada6f0afd23 i-046c7fe43620944bc --document-name AWS-RunShellScript \
  --parameters 'commands=["grep -q ^CARLOS_SWAP= /etc/carlos/host.env || echo CARLOS_SWAP=overlap >> /etc/carlos/host.env","systemctl restart carlos-edge"]'

If host.env already has a CARLOS_SWAP= line, even an empty or different one, the command leaves it alone and the box does not arm. Read the file first. The unit is carlos-edge and it loads /etc/carlos/host.env (internal/host/units/carlos-edge.service:18).

Restarting carlos-edge stops every instance on the box (spec section 10). Each wakes on its next request.

To revert a box, remove the CARLOS_SWAP= line from /etc/carlos/host.env and restart carlos-edge. Then remove the last generation's leftover g<n>/ directory under each instance's run directory, /run/carlos/instances/<host>/g<n>/. Dev round 5 (#272) found that the flag-off restart leaves it behind with a stale app.sock. Nothing uses a g<n>/ directory with the flag off, so it stays until a reboot. List what is left with ls -d /run/carlos/instances/*/g[0-9]*/ after the restart, and remove each one.

Then run the probe below against a dev app. Create one with:

carlos apps create --app zdd-probe --console https://platform.dev.carlosframework.com

then an operator's carlos instances enable --app zdd-probe, and:

carlos instances create --app zdd-probe --host <label>.<the account's platform domain>

Use the host it prints as HOST. For this gate: APP=zdd-probe, CONSOLE=https://platform.dev.carlosframework.com, and ACCOUNT is the dev account (an empty --account uses your only account).

Check the flag took before trusting the probe: carlos instances list --app zdd-probe should show a build under SERVING once the box has reported. A dash means the box did not arm (see the dash note above).

Gate F: the flagship probe box

Status: passed on annie-welding (ie) on 2026-10-14 (#272). annie stays armed. The first attempt, on 2026-10-13, was reverted. The retry ran the probe against a fresh app, as below. Its arm restart left one tenant's lease held for 15 minutes; that is the edge restart, not the flag (#292).

Needs Paul's approval, naming the box and its region. The approval also names the probe's app actions: creating a throwaway app in bab with one instance on that box, flipping its swap policy to hold and back, four deploys of it, and deleting it afterwards. Do not start this gate while the "Before Gate E: owed" list has an open item that Paul has not waived.

Use a fresh app with no program history, not bab/adopt-probe. adopt-probe's recorded program is adoptprobe, so root refuses a zddprobe build for it as the wrong program unless that build is shipped with carlos ship --program-change, which carlos deploy cannot do. That refusal, not a slow root pass, is what stopped zdd-1 in the first Gate F: root logged wrong program and restored the old build, and the watch timed out waiting for an install that would never come.

The 2026-10-14 retry set the app up like this:

  1. Create the app in bab through the console's New app form, with no region. It was bab/zdd-probe.
  2. An operator runs carlos instances enable --app zdd-probe --account bab, then creates one instance:

``bash carlos instances create --app zdd-probe --account bab --host zddprobe1.bab.oncarlos.com --console https://console.carloku.com ``

  1. Check it landed on the gate's box: carlos instances list --app zdd-probe --account bab --console https://console.carloku.com shows that box's region (the retry's was "Ireland · Dublin"), and only that box's edges file lists the host.
  2. A new host may not resolve yet. While DNS lags, point the load at the box's public IPv4 with RESOLVE (see "The probe"): RESOLVE=zddprobe1.bab.oncarlos.com:443:<box IP>.

The flagship boxes, profile keymail:

| box | instance id | region | | --- | --- | --- | | ie | i-092c0c1eea75723cb | eu-west-1 | | us | i-0ed0641d8e476f1b8 | us-east-1 | | uk | i-03c310ceaf50a9bab | eu-west-2 | | au | i-022288023f5e7496d | ap-southeast-2 |

ie's id matches the flagship console box id in 2026-10-05-scanner-host-rollout.md. The other three are not checked here.

Run Gate E's flag command against that one box with --profile keymail and its region. Then run the probe with:

APP=zdd-probe ACCOUNT=bab CONSOLE=https://console.carloku.com HOST=zddprobe1.bab.oncarlos.com
RESOLVE=zddprobe1.bab.oncarlos.com:443:<box IP>   # only while DNS lags

Afterwards, delete the app: carlos apps delete zdd-probe --account bab --console https://console.carloku.com. On 2026-10-14 the retry then removed the instance's leftovers on annie by hand (its instance, builds, links, data and /data/programs/ file) and its bucket object, and checked that no g<n>/ directory was left. An empty carlos-instances-zddprobe1…slice stayed.

Gate G: the rest of the flagship

Needs Paul's approval, naming each box before it is done. Do not start this gate while the "Before Gate E: owed" list has an open item that Paul has not waived.

One box at a time, with Gate E's flag command. After each box, do one ordinary deploy of adopt-probe and run carlos instances list --app adopt-probe --account bab, which must show INSTALLED equal to SERVING. Tito is Tito's operator's call; do not touch Tito's boxes from here.

The probe

Spec section 12: zero failed requests under continuous load, and no request held longer than a normal response. Run it at Gate E (dev) and Gate F (a fresh app on the flagship box, never adopt-probe; see Gate F for why).

1. Build the probe app, in scratch

The probe is cmd/zddprobe in this repository. It serves on the unix socket, sleeps 50 ms per request so some are always in flight, answers ok <version>, and drains on SIGTERM. It takes the platform's own argv, --socket <sock> and --db <db> (it ignores --db), and vet's -socket <sock>. The first dev check used a probe that parsed only -socket, so it exited at once when the platform launched it with --db.

Build it from the root of a platform checkout at the release commit (cd there first: the path ./cmd/zddprobe is relative). The binaries go to scratch:

mkdir -p "$TMPDIR/zdd-probe"
for v in zdd-1 zdd-2 zdd-3 zdd-4; do
  CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -ldflags "-X main.version=$v" -o "$TMPDIR/zdd-probe/probe-$v" ./cmd/zddprobe
done

2. A load phase

Four request loops for the given seconds, cache-busted, logging status and time. With RESOLVE set (<host>:443:<box IP>), curl skips DNS and goes straight to that box:

load() { # usage: load <phase> <secs>
  mkdir -p "$TMPDIR/zdd-probe/$1"
  for i in 1 2 3 4; do
    ( end=$((SECONDS+$2)); n=0
      while [ $SECONDS -lt $end ]; do n=$((n+1))
        curl -s ${RESOLVE:+--resolve "$RESOLVE"} -o /dev/null -m 35 -w '%{http_code} %{time_total}\n' "https://$HOST/?zdd=$1-$i-$n" >> "$TMPDIR/zdd-probe/$1/load-$i.log"
      done ) &
  done
  wait
}
summary() { cat "$TMPDIR/zdd-probe/$1"/load-*.log | awk -v p="$1" '$1 !~ /^2/ {bad++} {if ($2>max) max=$2; n++} END {printf "%s: %d requests, %d failed, slowest %.2fs\n", p, n, bad+0, max}'; }

3. Run it

cd "$TMPDIR/zdd-probe"
carlos deploy --app "$APP" --account "$ACCOUNT" --console "$CONSOLE" --version zdd-1 probe-zdd-1
load baseline 60; summary baseline
( load overlap 240 ) & sleep 20
carlos deploy --app "$APP" --account "$ACCOUNT" --console "$CONSOLE" --version zdd-2 probe-zdd-2
sleep 40
carlos deploy --app "$APP" --account "$ACCOUNT" --console "$CONSOLE" --version zdd-3 probe-zdd-3
wait; summary overlap
# Control: hold mode must show the pause, or the probe proves nothing.
carlos apps set "$APP" --account "$ACCOUNT" --console "$CONSOLE" --swap hold
( load hold 120 ) & sleep 20
carlos deploy --app "$APP" --account "$ACCOUNT" --console "$CONSOLE" --version zdd-4 probe-zdd-4
wait; summary hold
carlos apps set "$APP" --account "$ACCOUNT" --console "$CONSOLE" --swap overlap

Pass. overlap has 0 failed and its slowest is at most the larger of 1 s and twice the baseline's slowest. Each deploy's watch ends with the approved swapped https://$HOST line and a box line, exit 0. The hold control's slowest is clearly above the baseline. If it is not, the probe cannot see a pause and the overlap result means nothing.

The watch prints swapped https://<host> (the approved word swapped, deploywatch.go:569) and box <label>: ... (deploywatch.go:44). With the flag off it prints live https://<host> instead (deploywatch.go:564), so a live line during the overlap phase means that box is not armed.

On the control: the first --swap hold on an app creates config.json, and on boxes running this release that restarts nothing (the Q3 note above), so the hold phase's pause is the zdd-4 swap's. If the numbers look odd anyway, rerun the hold phase once with the policy already set.

4. Clean up

rm -rf "$TMPDIR/zdd-probe"

On dev, run carlos apps delete zdd-probe once Paul is done with it. On the flagship, delete the Gate F app as Gate F says.

Names checked against this branch

Checked in the code on zdd-6-surfaces (HEAD 77d77b3a plus this file; the anchors final fix wave 1 moved were re-checked after it): the verbs and flags carlos system update|status, carlos apps set|create|delete, carlos instances list|create|enable|retry, carlos deploy, carlos ship, carlos promote, carlos promote --retry, carlos rollback; the flag CARLOS_SWAP and its exact value overlap; the unit carlos-edge and /etc/carlos/host.env; the SERVING, INSTALLED and REGION columns; the swapped, live and box watch lines; cmd/changelog; docs/site/changelog.json, docs/site/changelog-links.json and the "Two builds at once" section.

Added by zdd-flag-readiness-3 (checked on that branch): __cgexec (cgroup.ShimVerb), carlos-placement-check, the decline code cgroup-placement-failed, cgroup.ErrLimitNotApplied, cgroup.ErrStrictUnsupported, PrepareStrict, PrepareStrictParent, the agent log lines placement before exec failed, generation start refused, the start's context ended during placement, REFUSING CARLOS_SWAP=overlap (<code>) and overlap swaps armed, the control service's reset-failed log line, classifyRollout and follows in deploywatch.go, cmd/zddprobe, and the kernel-test variables CARLOS_GEN_KERNEL, CARLOS_SHIM_KERNEL, CARLOS_EDGE_HARDENING_KERNEL, CARLOS_STRICT_KERNEL and CARLOS_CLAIM_SPAWN_KERNEL.

Not checked here (not in this repo, or not verifiable without running something): roll-console.sh and update-platform-binary.sh (platform-infrastructure); hack/sync-docs.mjs and the website's AGENTS.md (carlosframework/website); the live URLs; the AWS instance ids except i-092c0c1eea75723cb and i-05498eada6f0afd23; the SSM AWS-RunShellScript documents; that bc exists on the boxes; the dev account and the flagship's channel names; the behaviour of the running boxes.