| index be5a843b5..dc385057c 100644 |
| --- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| +++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| @@ -32,9 +32,13 @@ plans 1 to 6, in `platform-design-zero-downtime-deploys`. |
| This is the flag-on precondition. It is copied from amadan discussion #272 |
| ("Zero-downtime deploys: owed before CARLOS_SWAP can go on", opened by |
| claude~paul@keymail.dev), as read on 2026-10-11, including its replies. |
| -Nothing on this list is closed. **The flag stays off on any box until every |
| -item is closed or Paul waives it in writing.** Gates A to D do not turn the |
| -flag on and can go ahead while this list is open. |
| +As of 2026-10-12, items this branch closes carry a line saying "closed on |
| +zdd-flag-readiness (refs), pending merge". Those are built and reviewed on |
| +the branch, not merged, not released and not rolled, and only Paul closes |
| +them on #272. Everything else is still open, and the open items are |
| +collected under "Still owed" below. **The flag stays off on any box until |
| +every item is closed or Paul waives it in writing.** Gates A to D do not |
| +turn the flag on and can go ahead while this list is open. |
| |
| The discussion's opening text says zero-downtime deploys are merged |
| dormant and `CARLOS_SWAP` must stay unset until everything below is closed. |
| @@ -42,31 +46,75 @@ dormant and `CARLOS_SWAP` must stay unset until everything below is closed. |
| ### Correctness, from Astra's plan 5 review (`CODEX-FINDINGS-ZDD-IMPL-5*.md`) |
| |
| - N5 to N7: gaps in the hold-ran marker. N7 is a disagreement over how it |
| - treats an existing failure record. |
| + treats an existing failure record. Closed on zdd-flag-readiness (b402bc0e, |
| + 42865b81; N6 with 0d6029b3, 3e875ff2), pending merge. That covers markers |
| + keeping records, blocking the worker fallback and signed-rollback |
| + provenance, with marker = installed = good boots (ruling 205). N6 and the |
| + retry ordering use a persisted book sequence and per-execution markers, so |
| + a pre-retry outcome never undoes a retry. |
| - F5: failure-record durability across a failed write followed by a restart. |
| + Closed on zdd-flag-readiness (8836e521: write-ahead intents, schema-2 book, |
| + startup reconciliation; f46d3142, 4416872e: a shutdown join so planned |
| + restarts leave no intent), pending merge. |
| +- Ruling 206: an uncertain nested start is a hold failure. Closed on |
| + zdd-flag-readiness (c453a798), pending merge. |
| - Boxes that declined to arm: worker restarts; refusals checked only at plan |
| - time; records read only at start. |
| -- The 15-item list in round 3, section 5. |
| + time; records read only at start. Closed on zdd-flag-readiness (fa7d7374, |
| + e32ac0c6, e327ae0e), pending merge. A declined box now gates every launch at |
| + exec time, including workers, legacy sidecars, claims and move launches. It |
| + honours markers, and signed retries and the pointer pass. Execution |
| + admission (7daf32ac) adds a worker gate after cgroup prep, managed |
| + revalidation and no uncaptured live-link recycle. Two parts of this are |
| + still open, see "Still owed". |
| +- The 15-item list in round 3, section 5. The code defects are closed on |
| + zdd-flag-readiness, pending merge: policy cache, bounded reads and acks |
| + (f46d3142, 4416872e); clean one-shot good, the WakeWorker budget and the |
| + halt event (5ad55485); GoodVersion bucket retention and the hold build |
| + published before its incumbent stops (d3d464cd); the stage relay give-up |
| + (2c5adbe8); transient status (98c1763d, 61b83e27). The branch simulation |
| + gained invariants I5 to I8 and mutants M7 to M12 (44800ab4). The 3000-seed |
| + sweep passed and M1 to M12 were all caught. |
| |
| ### Carried from plans 2 and 3 |
| |
| - With the flag off, a retained wake is proven by its leader's exit only; |
| - forked descendants are not checked (F6, pre-existing). |
| -- Leftover executable files are not swept from `/run`. |
| + forked descendants are not checked (F6, pre-existing). The failed-wake half |
| + is closed on zdd-flag-readiness (7c0f2543, 23d42f0f), pending merge: a |
| + failed wake now proves its host cgroup empty. The normal-stop half is still |
| + open, see "Still owed". |
| +- Leftover executable files are not swept from `/run`. Closed on |
| + zdd-flag-readiness (7c0f2543, 23d42f0f), pending merge. |
| - Claims-backed swaps are refused (P2-9). Main's claimed launch is closed, so |
| a launch protocol and the strict-parent lifecycle are needed before claims |
| - boxes get swaps. |
| + boxes get swaps. Handled on zdd-flag-readiness (cd7de723) by declining the |
| + flag on a claims-on box, see the claims reply below. Claims-on swaps |
| + themselves are a future feature. |
| - Strict hosts get no swaps until a claim path exists (P2-8). |
| |
| ### Dev-box checks (need Paul's approval to run) |
| |
| - D1: `daemon-reload` cost at titogo-scale unit counts. Above about 1 s per |
| swap means transient units instead of drop-ins. (Gate E below runs this |
| - measurement.) |
| + measurement.) Done: well under 1 s, so keep the drop-ins. |
| - The plan 2 kernel cgroup suite on arm64 with `User=carlos`. |
| - Real-unit sidecar and control checks: the registry read under the control |
| unit, and StopGen(kill) on a never-loaded unit. |
| -- A guard against legacy cgroup parents that still hold processes. |
| +- A guard against legacy cgroup parents that still hold processes. Code |
| + fixed on zdd-flag-readiness (2c5adbe8), pending merge. See "Dev-box flag |
| + blockers" below for what is still owed on the dev box. |
| + |
| +#### Dev-box flag blockers |
| + |
| +Two defects blocked the flag on the dev box. Both are fixed in code on |
| +zdd-flag-readiness, pending merge. Their tests are kernel-gated and skip on |
| +the build machine, so neither fix has been seen working on a real kernel. |
| +The dev-box rerun is owed. |
| + |
| +- The legacy-parent guard (2c5adbe8). Code fixed. Its kernel test |
| + `TestGenerationStartRefusedBesideADirectParentMember` needs the rerun. |
| +- The `cgroup.kill` respawn (c76a1d44, 66c92efe). Code fixed: a cgroup that |
| + took `cgroup.kill` is recreated, and KillHost and KillSidecar respawn. |
| + Its kernel test `TestClaimContainmentCleanupRespawn` needs the rerun. |
| |
| ### Surfaces (plan 6) |
| |
| @@ -94,13 +142,18 @@ on #272 too. |
| Paul's ruling (2026-10-11): host retries stay on the unsigned restart-scope |
| log for now, with the same trust as `carlos restart`. Signing retries in the |
| console, as `carlos promote --retry` is signed, is owed before the flag goes |
| - on anywhere hold mode is used. Paul is adding this to #272. |
| + on anywhere hold mode is used. Paul is adding this to #272. Closed on |
| + zdd-flag-readiness (d68a1dde), pending merge: the console signs a host |
| + retry as an epoch-bound grant, and every box ignores an unsigned retry. A |
| + console with no signing key answers 503, and the text of that refusal is |
| + new copy awaiting review (`copy-review/strings.json`). |
| - `provision.ListRecords` silently skips unreadable instance records (Astra |
| plan 6 R3). A relevant fleet can then drop out of the deploy watch's |
| completeness check, so a sleeper on it may be URL-polled when its fleet |
| snapshot is also unreadable. This is a flag-on blocker. Owed: expose |
| record-read completeness to the instance list |
| - (docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-6-R3.md). |
| + (docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-6-R3.md). Closed on |
| + zdd-flag-readiness (f9be4718), pending merge. |
| |
| ### Reply: four claims-on gaps |
| |
| @@ -110,8 +163,8 @@ lease margin, region-HA). Astra's merge review |
| merge, with no flag-off defect. |
| |
| Main's `RoleLaunch.Root` is wired in production, so claims-on launches can |
| -execute. **Before the flag goes on, either decline arming on boxes with host |
| -claims on, or close all four:** |
| +execute. When this reply was written, the flag could not go on until the box |
| +either declined arming with host claims on, or all four of these were closed: |
| |
| 1. Recovery selection and fallback cannot be honoured for claimed planned |
| wakes. |
| @@ -120,11 +173,64 @@ claims on, or close all four:** |
| 4. Successful claimed wakes cannot record their good build or serving |
| version. |
| |
| -No claims check was found in `swapArming` |
| -(`internal/host/rollout.go:145-156`), which is where the arming decision is |
| -made. Only that function was read, so this is not proof that nothing |
| -declines elsewhere. Treat "decline arming on claims boxes" as unbuilt until |
| -someone shows the code. |
| +That reply found no claims check in `swapArming` |
| +(`internal/host/rollout.go:145-156`). |
| + |
| +**Conclusion (2026-10-12): decline, do not close.** On zdd-flag-readiness |
| +(cd7de723), pending merge, a box with host claims on declines |
| +`CARLOS_SWAP=overlap` with the reason code `host-claims-on`. It runs today's |
| +stop-then-start paths, and the agent log says `REFUSING CARLOS_SWAP=overlap`. |
| +The four gaps above are not closed. They become the future feature "claims-on |
| +swaps", which is not scheduled. The source for the four gaps is |
| +`CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md`. After Gate E on any claims-on box, |
| +expect a dash under SERVING and the `host-claims-on` reason, not an armed box. |
| + |
| +### Still owed (code), after zdd-flag-readiness |
| + |
| +These stay open after the branch. They come from the rulings of this round, |
| +and each one still blocks the flag unless Paul waives it. |
| + |
| +- Nested live-link race on a declined box: needs an expected identity on |
| + `Control.Start`, root-side. |
| +- Seeds exec the live link ungated on a declined box. A failed seed's |
| + descendants are not proven (flag off and on), and `removeSeedDB` can delete |
| + a DB a descendant still writes. |
| +- The flag-off normal-stop half of F6: a normal stop still proves only the |
| + leader. |
| +- `ensureParent`'s other failures still run a generation uncapped: no |
| + `+memory`, EBUSY or EACCES on `subtree_control`, the mkdir fallback, and |
| + `applyLimit` on `g<n>`. |
| +- The console instance page shows no hold row or Retry button for a declined |
| + box's hold. This needs copy. |
| +- Claims-on swaps (the four gaps from `CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md`). |
| + A future feature. |
| +- Spec section 11 still says a retry is a re-signed pointer. The spec needs a |
| + revision for console-signed host-retry grants. |
| + |
| +### Roll notes from zdd-flag-readiness |
| + |
| +- **Retries: boxes before consoles.** A new box ignores an old console's |
| + unsigned retry. An older binary that appends to `restart-scope.json` strips |
| + `signed` from pending retries, so they are dropped. That fails closed. |
| +- **Schema 2 failure book.** The failure book file is written as schema 2. |
| + Downgrading an agent with the flag on makes that box refuse managed starts |
| + until the file is upgraded or removed. |
| +- **`carlos-instance-control` is not restarted by the old script.** |
| + `platform-infrastructure/scripts/update-platform-binary.sh` does not |
| + restart it. Use `carlos system update` (main's system apply restarts it, |
| + 0bb57a31), or restart it by hand after the old script. |
| + |
| +### Real-box checks (need Paul's approval; not run by this branch) |
| + |
| +- The kernel-gated tests `TestClaimContainmentCleanupRespawn` (now the |
| + KillHost and KillSidecar respawn) and |
| + `TestGenerationStartRefusedBesideADirectParentMember` on the dev box |
| + (arm64, `CARLOS_CLAIM_SPAWN_KERNEL=1`, delegated user service). Also check |
| + whether children of a killed kept cgroup, and cgroups systemd kills on |
| + `carlos-edge` stop, are affected. |
| +- The plan 2 kernel suite rerun. |
| +- The flagship probe (Gates E to G). |
| +- D1 is done: well under 1 s, keep drop-ins. |
| |
| ### Reply: wiring caution (settings plan 2, merged as `70f8119`) |
| |
amadan