amadan
zdd-readiness: final review round 1, roll notes stop overclaiming
Merge item 9: the closures this wave completes (N6, rollback provenance and retry ordering; the claims move gate; F6's failed wake; signed retries) cite its commits; the flag blockers Astra's triage lists and this wave did not fix are under Still owed; the schema-2 recovery no longer advises deleting the book; the populated legacy parent gets an accurate manual recovery (kill its direct processes or restart carlos-edge; nothing lifts it automatically); the armed agent's book write on start and the book-bound grant's mixed-version behaviour are noted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+88
−28
docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md+88 −28
diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md| index 447f68e82..5e416b708 100644 |
| --- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| +++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| @@ -34,8 +34,10 @@ This is the flag-on precondition. It is copied from amadan discussion #272 |
| claude~paul@keymail.dev), as read on 2026-10-11, including its replies. |
| As of 2026-10-12, items this branch closes carry a line saying "closed on |
| zdd-flag-readiness (refs), pending merge". Those are built and task-reviewed on |
| -the branch (the whole-branch review comes next), not merged, not released and not rolled, and only Paul closes |
| -them on #272. Everything else is still open, and the open items are |
| +the branch, and the whole-branch review's first round (Astra, |
| +`CODEX-FINDINGS-ZDD-READINESS.md`) found gaps in several of them, which its |
| +fix wave closed (the commits cited as "final review round 1"). They are not |
| +merged, not released and not rolled, and only Paul closes them on #272. Everything else is still open, and the open items are |
| collected under "Still owed" below. **The flag stays off on any box until |
| every item is closed or Paul waives it in writing.** Gates A to D do not |
| turn the flag on and can go ahead while this list is open. |
| @@ -47,11 +49,18 @@ dormant and `CARLOS_SWAP` must stay unset until everything below is closed. |
| - N5 to N7: gaps in the hold-ran marker. N7 is a disagreement over how it |
| treats an existing failure record. Closed on zdd-flag-readiness (b402bc0e, |
| - 42865b81; N6 with 0d6029b3, 3e875ff2), pending merge. That covers markers |
| - keeping records, blocking the worker fallback and signed-rollback |
| - provenance, with marker = installed = good boots (ruling 205). N6 and the |
| - retry ordering use a persisted book sequence and per-execution markers, so |
| - a pre-retry outcome never undoes a retry. |
| + 42865b81; N6 with 0d6029b3, 3e875ff2; final review round 1: 1e842f89, |
| + c327f703, e72e68d2), pending merge. That covers markers keeping records, |
| + blocking the worker fallback and signed-rollback provenance, with marker = |
| + installed = good boots (ruling 205). N6 and the retry ordering use a |
| + persisted book sequence and per-execution markers, so a pre-retry outcome |
| + never undoes a retry. The round 1 fixes: an execution's identity carries |
| + the agent process's incarnation, so a run after a restart is never taken |
| + for the run before it, and a rollback or retry that matched the old run |
| + clears nothing of the new one (1e842f89); a retry that removes only a |
| + marker still orders a late failure of an execution that began before it |
| + (c327f703); and a host retry of a build the instance does not want yet |
| + keeps its canary for that build (e72e68d2). |
| - F5: failure-record durability across a failed write followed by a restart. |
| Closed on zdd-flag-readiness (8836e521: write-ahead intents, schema-2 book, |
| startup reconciliation; f46d3142, 4416872e: a shutdown join so planned |
| @@ -60,20 +69,28 @@ dormant and `CARLOS_SWAP` must stay unset until everything below is closed. |
| zdd-flag-readiness (c453a798), pending merge. |
| - Boxes that declined to arm: worker restarts; refusals checked only at plan |
| time; records read only at start. Closed on zdd-flag-readiness (fa7d7374, |
| - e32ac0c6, e327ae0e), pending merge. A declined box now gates every launch at |
| - exec time, including workers, legacy sidecars, claims and move launches. It |
| + e32ac0c6, e327ae0e; final review round 1: 076bd9c0), pending merge. A |
| + declined box now gates every launch at exec time, including workers, |
| + legacy sidecars, claims and move launches. A move launch runs on a row the |
| + move holds (landing, restoring), and until 076bd9c0 such a row read as |
| + unrestricted; it is now restricted as its ready row is, and a signed retry |
| + that arrives while the row is held is applied rather than dropped. It |
| honours markers, and signed retries and the pointer pass. Execution |
| admission (7daf32ac) adds a worker gate after cgroup prep, managed |
| revalidation and no uncaptured live-link recycle. Closed except for the two |
| items under "Still owed" (the nested live-link race and the seed exec). |
| -- The 15-item list in round 3, section 5. The code defects on the flag-on |
| +- The 15-item list in round 3, section 5. These code defects on the flag-on |
| path are closed on zdd-flag-readiness, pending merge: policy cache, bounded reads and acks |
| - (f46d3142, 4416872e); clean one-shot good, the WakeWorker budget and the |
| - halt event (5ad55485); GoodVersion bucket retention and the hold build |
| + (f46d3142, 4416872e); clean one-shot good (but see "Still owed": its good |
| + publication does not yet check ownership or fencing), the WakeWorker budget |
| + and the halt event (5ad55485); GoodVersion bucket retention and the hold build |
| published before its incumbent stops (d3d464cd); the stage relay give-up |
| (2c5adbe8); transient status (98c1763d, 61b83e27). The branch simulation |
| gained invariants I5 to I8 and mutants M7 to M12 (44800ab4). The 3000-seed |
| - sweep passed and M1 to M12 were all caught. The coverage-only items, |
| + sweep passed and M1 to M12 were all caught; it was run again after the |
| + final review round 1 fixes, which changed gate logic. It |
| + does not cover the restart and replay sequences of round 1's findings 1 |
| + and 3, which have unit tests instead. The coverage-only items, |
| reproducible replay and the real-box items stay deferred, as the plan's |
| rulings say. |
| @@ -81,9 +98,11 @@ dormant and `CARLOS_SWAP` must stay unset until everything below is closed. |
| - With the flag off, a retained wake is proven by its leader's exit only; |
| forked descendants are not checked (F6, pre-existing). The failed-wake half |
| - is closed on zdd-flag-readiness (7c0f2543, 23d42f0f), pending merge: a |
| - failed wake now proves its host cgroup empty. The normal-stop half is still |
| - open, see "Still owed". |
| + is closed on zdd-flag-readiness (7c0f2543, 23d42f0f; final review round 1: |
| + bbb7fc8f), pending merge: a failed wake now proves its host cgroup empty, |
| + deciding from the layout its launch used rather than the layout the row |
| + reads at cleanup (bbb7fc8f). The failed-seed and normal-stop halves are |
| + still open, see "Still owed". |
| - Leftover executable files are not swept from `/run`. Closed on |
| zdd-flag-readiness (7c0f2543, 23d42f0f), pending merge. |
| - Claims-backed swaps are refused (P2-9). Main's claimed launch is closed, so |
| @@ -145,8 +164,13 @@ on #272 too. |
| log for now, with the same trust as `carlos restart`. Signing retries in the |
| console, as `carlos promote --retry` is signed, is owed before the flag goes |
| on anywhere hold mode is used. Paul is adding this to #272. Closed on |
| - zdd-flag-readiness (d68a1dde), pending merge: the console signs a host |
| - retry as an epoch-bound grant, and every box ignores an unsigned retry. A |
| + zdd-flag-readiness (d68a1dde; final review round 1: c12c35c3), pending |
| + merge: the console signs a host retry as a grant bound to the |
| + restriction's epoch and to the identity of the box's failure book, and |
| + every box ignores an unsigned retry. The book identity (c12c35c3) stops a |
| + grant for one book's restriction clearing another book's at the same |
| + epoch, and every open of the book moves its sequence past any epoch an |
| + earlier run handed out unsaved. A |
| console with no signing key answers 503, and the text of that refusal is |
| new copy awaiting review (`copy-review/strings.json`). |
| - `provision.ListRecords` silently skips unreadable instance records (Astra |
| @@ -192,34 +216,70 @@ in the agent log line `REFUSING CARLOS_SWAP=overlap (host-claims-on)`. |
| ### Still owed (code), after zdd-flag-readiness |
| -These stay open after the branch. They come from the rulings of this round, |
| -and each one still blocks the flag unless Paul waives it. |
| +These stay open after the branch, and each one still blocks the flag |
| +unless Paul waives it. Astra's whole-branch review, round 1 |
| +(`CODEX-FINDINGS-ZDD-READINESS.md`), classes each of them as a flag |
| +blocker, and the fix wave for that round did not close them. |
| - Nested live-link race on a declined box: needs an expected identity on |
| - `Control.Start`, root-side. |
| -- Seeds exec the live link ungated on a declined box. A failed seed's |
| - descendants are not proven (flag off and on), and `removeSeedDB` can delete |
| - a DB a descendant still writes. |
| + `Control.Start`, root-side (task 9). |
| +- Seeds exec the live link ungated on a declined box (task 9). A failed |
| + seed's descendants are not proven (flag off and on), and `removeSeedDB` |
| + can delete a DB a descendant still writes (task 15). |
| +- A legacy sidecar left stranded beside a retained incumbent has no |
| + recovery path yet (task 9). |
| +- A clean one-shot's good build is published without checking that the |
| + activator still owns the host and that no fence or lost lease came first, |
| + and a WakeWorker can still start a duplicate wake in a narrow window |
| + (task 12). |
| - The flag-off normal-stop half of F6: a normal stop still proves only the |
| - leader. |
| + leader (task 15). |
| - `ensureParent`'s other failures still run a generation uncapped: no |
| `+memory`, EBUSY or EACCES on `subtree_control`, the mkdir fallback, and |
| - `applyLimit` on `g<n>`. |
| + `applyLimit` on `g<n>`. The kernel positive control (a real cgroupfs |
| + showing the populated-parent refusal and the respawn work) is also owed |
| + (task 14). |
| - The console instance page shows no hold row or Retry button for a declined |
| box's hold. This needs copy. |
| - Claims-on swaps (the four gaps from `CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md`). |
| A future feature. |
| - Spec section 11 still says a retry is a re-signed pointer. The spec needs a |
| revision for console-signed host-retry grants. |
| +- Mixed-version retry sequencing: the order in which boxes and consoles take |
| + the signed, book-bound grants needs writing down and checking on dev before |
| + the flag goes on (see "Retries: boxes before consoles" below). |
| + |
| +Task 10's "a grant consumed while the row is moving or landing" is closed |
| +by 076bd9c0 (final review round 1), pending merge. |
| ### Roll notes from zdd-flag-readiness |
| - **Retries: boxes before consoles.** A new box ignores an old console's |
| unsigned retry. An older binary that appends to `restart-scope.json` strips |
| `signed` from pending retries, so they are dropped. That fails closed. |
| + Since c12c35c3 a grant must also name the box's failure book: a box on |
| + this branch drops a grant from a console that does not sign one, and the |
| + log line says the grant is for book "". A box from before this branch |
| + handles a retry as it does today. Nothing is cleared that should not be, |
| + but a retry asked between the two rolls may have to be asked again once |
| + the console is updated. |
| - **Schema 2 failure book.** The failure book file is written as schema 2. |
| - Downgrading an agent with the flag on makes that box refuse managed starts |
| - until the file is upgraded or removed. |
| + Downgrading an agent with the flag on makes that box refuse managed starts. |
| + Recover by upgrading the agent again. Do not delete the file to get past |
| + the refusal: it holds the failure records and hold-ran markers that keep |
| + instances off builds that failed or ran on them. |
| +- **The armed agent writes its failure book on start.** Since c12c35c3 an |
| + agent with the flag on saves the book when it opens it, even when there |
| + was no file, and refuses managed starts if it cannot (the same refusal as |
| + an unreadable book). A box whose data dir is read-only will show that. |
| +- **A populated legacy parent cgroup.** A generation start is refused while |
| + `carlos-exec@<host>` holds a process directly (2c5adbe8). The agent logs a |
| + line beginning `cgroup: <host>: generation start refused: the parent |
| + cgroup holds N process(es) directly`, wakes of that host fail, and swaps |
| + abandon and are retried every 15 s. Nothing lifts this automatically. Clear it by hand: |
| + kill the processes listed in that cgroup's own `cgroup.procs` (under the |
| + agent's delegated cgroup), or restart `carlos-edge`, which kills the |
| + delegated subtree. The next wake or retried swap then starts normally. |
| - **`carlos-instance-control` is not restarted by the old script.** |
| `platform-infrastructure/scripts/update-platform-binary.sh` does not |
| restart it. Use `carlos system update` (main's system apply restarts it, |