| index 90214056a..26a212213 100644 |
| --- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| +++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| @@ -193,6 +193,9 @@ on #272 too. |
| memory until its next write of the book, and that write is an attempt: a |
| failed save is logged and the box continues. A console with no signing key answers 503, and the text of that refusal is |
| new copy awaiting review (`copy-review/strings.json`). |
| + Superseded on zdd-flag-readiness-2 (`7c7c6141`, pending merge): the 2^20 |
| + advance and its "bound, not a guarantee" caveat are gone, replaced by the |
| + durable per-app generation rule (see "Still owed (code)"). |
| - `provision.ListRecords` silently skips unreadable instance records (Astra |
| plan 6 R3). A relevant fleet can then drop out of the deploy watch's |
| completeness check, so a sleeper on it may be URL-polled when its fleet |
| @@ -241,62 +244,173 @@ unless Paul waives it. Astra's whole-branch review, round 1 |
| (`CODEX-FINDINGS-ZDD-READINESS.md`), classes each of them as a flag |
| blocker, and the fix wave for that round did not close them. |
| |
| +Readiness round 2 (branch `zdd-flag-readiness-2`, from `650e6152`, pending |
| +merge) closes most of them. A closed item below says "Closed on |
| +zdd-flag-readiness-2 (<commit>), pending merge". That means built and |
| +task-reviewed on the branch: not merged, not released, not rolled, and only |
| +Paul closes it on #272. The branch's whole-branch review (Astra) and the |
| +3000-seed rollout-simulation sweep come after these notes. Two items stay |
| +open: claims-on swaps, and Paul's sign-off of the spec amendment. |
| + |
| - Nested live-link race on a declined box: needs an expected identity on |
| - `Control.Start`, root-side (task 9). |
| + `Control.Start`, root-side (task 9). Closed on zdd-flag-readiness-2 |
| + (`903ed7ca`, `c910d020`), pending merge: a declined box pins the live link |
| + once (its identity and the exec gate's verdict) and nested starts and |
| + seeds exec the pinned build. |
| - Seeds exec the live link ungated on a declined box (task 9). A failed |
| seed's descendants are not proven (flag off and on), and `removeSeedDB` |
| - can delete a DB a descendant still writes (task 15). |
| + can delete a DB a descendant still writes (task 15). Closed on |
| + zdd-flag-readiness-2 (`903ed7ca` for the gate; `4c5e43b5`, `5e51c008`, |
| + `4771d147` for the proof), pending merge: a seed's processes are proven |
| + gone before its database goes, nested seeds included, and an unproven seed |
| + leaves a durable obligation (see the roll note on `.seed-unproven`). |
| - A legacy sidecar left stranded beside a retained incumbent has no |
| - recovery path yet (task 9). |
| + recovery path yet (task 9). Closed on zdd-flag-readiness-2 (`c910d020`), |
| + pending merge: a refused sidecar restarts on the instance's own build. |
| - A clean one-shot's good build is published without checking that the |
| activator still owns the host and that no fence or lost lease came first, |
| and a WakeWorker can still start a duplicate wake in a narrow window |
| - (task 12). |
| + (task 12). Closed on zdd-flag-readiness-2 (`deb4c811`), pending merge: a |
| + one-shot is recorded good only after a clean park, and a waiting wake |
| + never starts a second. |
| - The flag-off normal-stop half of F6: a normal stop still proves only the |
| - leader (task 15). |
| + leader (task 15). Closed on zdd-flag-readiness-2 (`d6996537`, `ac9f3fef`, |
| + `097ca930`, `79032948`), pending merge: seeds and flag-off stops prove |
| + their processes gone before the lease goes, and a flag-off wake proves its |
| + host cgroup empty before it restores. This is an authorized flag-off |
| + change: one `cgroup.events` read per flag-off stop. |
| - `ensureParent`'s other failures still run a generation uncapped: no |
| `+memory`, EBUSY or EACCES on `subtree_control`, the mkdir fallback, and |
| `applyLimit` on `g<n>`. The kernel positive control (a real cgroupfs |
| showing the populated-parent refusal and the respawn work) is also owed |
| - (task 14). |
| + (task 14). The code is closed on zdd-flag-readiness-2 (`98593909`, |
| + `8530210e`), pending merge: a generation whose limits cannot be applied |
| + does not start (`cgroup.ErrLimitNotApplied`). The kernel positive control |
| + is still owed (see "Real-box checks"). |
| - The console instance page shows no hold row or Retry button for a declined |
| - box's hold. This needs copy. |
| + box's hold. This needs copy. Closed on zdd-flag-readiness-2 (`6e6ad000`, |
| + `65d4e135`), pending merge, except the copy: the seven new strings |
| + (`copy-review/strings.json`) wait for Paul's copy review. |
| - Claims-on swaps (the four gaps from `CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md`). |
| - A future feature. |
| + A future feature. Still open and not scheduled. |
| - Spec section 11 still says a retry is a re-signed pointer. The spec needs a |
| - revision for console-signed host-retry grants. |
| -- The reopen epoch bound (c12c35c3): the 2^20 advance is not enforced, so a |
| - run with more than a million unsaved book changes could let a later run |
| - reuse an epoch. Before the flag goes on, enforce the bound, reserve epochs |
| - durably, or get Paul's ruling that the limit is acceptable (Astra, final |
| - review round 2). |
| + revision for console-signed host-retry grants. Drafted on |
| + zdd-flag-readiness-2 as |
| + `docs/superpowers/specs/2026-10-12-zero-downtime-deploys-rev18-amendment.md` |
| + (this commit), pending Paul's sign-off. The design branch's spec is not |
| + edited until he signs it off. |
| +- The reopen epoch bound (c12c35c3), now the hard rule. Paul's ruling |
| + (2026-10-12): a stale grant must never clear a later failure, including |
| + across a crash. Closed on zdd-flag-readiness-2 (`7c7c6141`, `191c12b9`), |
| + pending merge. The rule: the box keeps one restriction generation per app, |
| + moved by every change to any of the app's restrictions; the book is saved |
| + before a generation is published, a save that fails leaves it unpublished |
| + (epoch 0, so no Retry is offered) and is retried on its own; a reopen |
| + starts past every saved value; and a failure book written by an earlier |
| + agent is given a new identity when adopted, so earlier grants name the old |
| + one. The 2^20 advance on every open is gone. The proof sketch and its two |
| + assumptions are in the doc comment on `FailureBook` |
| + (`internal/rollout/failures.go`). The cost if the ruling is wrong: a box |
| + with a failing disk shows holds with no Retry until a save lands. |
| - Mixed-version retry sequencing: the order in which boxes and consoles take |
| the signed, book-bound grants needs writing down and checking on dev before |
| - the flag goes on (see "Retries: boxes before consoles" below). |
| + the flag goes on (see "Retries: supported transitions" below). Closed on |
| + zdd-flag-readiness-2 (`6e9e84bd`, `9f27cdfa`, `4d8957c1`, `18bde560`), |
| + pending merge, by construction: grants ride their own object, so no |
| + ordering of box and console rolls applies one unsafely. The dev-box check |
| + across a mixed-version roll is still owed (see "Real-box checks"). |
| |
| Task 10's "a grant consumed while the row is moving or landing" is closed |
| by 076bd9c0 (final review round 1), pending merge. |
| |
| ### Roll notes from zdd-flag-readiness |
| |
| -- **Retries: boxes before consoles.** A new box ignores an old console's |
| - unsigned retry. An older binary that appends to `restart-scope.json` strips |
| - `signed` from pending retries, so they are dropped. That fails closed. |
| - Since c12c35c3 a grant must also name the box's failure book: a box on |
| - this branch drops a grant from a console that does not sign one, and the |
| - log line says the grant is for book "". A box from before this branch |
| - handles a retry as it does today: it does not check the grant's book or |
| - epoch, so receiving the extra field gives it none of this protection. Until |
| - every box runs this branch, an older box can apply a retry that a box on |
| - this branch would refuse (one for a stale epoch or for another book), so |
| - retry sequencing across a mixed-version roll stays a flag gate (see |
| - "Still owed"). A retry asked between the two rolls may also have to be |
| - asked again once the console is updated. |
| -- **Schema 2 failure book.** The failure book file is written as schema 2. |
| - Downgrading an agent with the flag on makes that box refuse managed starts. |
| - Recover by upgrading the agent again. Do not delete the file to get past |
| - the refusal: it holds the failure records and hold-ran markers that keep |
| - instances off builds that failed or ran on them. |
| +- **Retries: boxes before consoles.** Superseded by the next bullet. Kept |
| + for history: on zdd-flag-readiness a retry rode `restart-scope.json`, and |
| + an older binary that appended to it stripped `signed` from pending |
| + retries. That fails closed, but a box from before the branch applied a |
| + retry with no book or epoch check, so the roll order mattered. Since |
| + zdd-flag-readiness-2 it does not (see below). |
| +- **Retries: supported transitions** (zdd-flag-readiness-2, `6e9e84bd`, |
| + `9f27cdfa`, `4d8957c1`, `18bde560`, pending merge). A signed host retry is |
| + written to `apps/<acct>/<app>/host-retry.json`, not to `restart-scope.json`. |
| + No released binary reads or writes it, and `restart-scope.json` no longer |
| + takes a retry or a signature. A retry is released nowhere before this |
| + branch, so there is no released retry behaviour to preserve. The |
| + transitions: |
| + - *Old console, new box.* A released console has no host retry at all, so |
| + the CLI and the page offer none until the console is updated. Its |
| + restarts rewrite `restart-scope.json` only, which holds no grant now and |
| + so cannot lose one. |
| + - *New console, old box.* The old box's report carries no `retry_v` and the |
| + old box never reads `host-retry.json`. The console refuses the retry with |
| + a message (409 on the API, a flash in the browser, no Retry button on the |
| + page) and writes nothing. Update the box, then ask again. |
| + - *New console, new box, book not saved.* A held host at epoch 0 is |
| + refused with "the box has not saved its failure yet". Ask again in a |
| + minute. The agent retries its own save. |
| + - *New console, new box.* The console signs a grant for the epoch and book |
| + the box reported, the box verifies it on the next serial move, and a |
| + grant gone stale is a no-op. |
| + - *A grant written before an agent restart* is checked again after it |
| + (grants are not seed-skipped). A current one applies, a spent or stale |
| + one does nothing. |
| + - *A dev box on an intermediate build* (between `5da1cc9b` and this |
| + branch) loses the retry path until it is updated. It is unreleased, |
| + which is why it is acceptable. |
| + - A console with no signing key answers 503 and writes nothing. |
| + - A box that declined the flag and keeps a failure book honours a grant too |
| + (no canary: its next wake starts the build as today). |
| +- **Schema 3 failure book.** The failure book file is written as schema 3 |
| + (it was 2 on zdd-flag-readiness). Downgrading an agent with the flag on |
| + makes that box refuse managed starts, and that includes a round 1 agent |
| + (on `main` since `8ebd9d8c`), which accepts only schema 1 and 2. Recover by upgrading the agent again. Do not |
| + delete the file to get past the refusal: it holds the failure records and |
| + hold-ran markers that keep instances off builds that failed or ran on |
| + them. A file of schema 1 or 2 is adopted on first open and given a new |
| + book identity, so grants issued against the old identity clear nothing. |
| +- **Copying or cloning a data dir: drop the book key.** The failure book's |
| + identity is in the file's `book` key. A copy that keeps it gives two boxes |
| + one identity, and a copy restored from an older snapshot can hand out again |
| + a generation already published. Before the flag goes on, any path that |
| + clones, restores or relocates an agent data dir (a rebuilt box from an |
| + image or snapshot, a hand copy) must delete the `book` key from |
| + `swap-failures.json` in the copy. The next open then makes a new identity. |
| + This is unaudited: nothing yet checks the fleet's rebuild and restore paths |
| + for it. |
| +- **An armed box without `CLONE_INTO_CGROUP` refuses every generation |
| + start** (`98593909`, `8530210e`). A generation start must be placed in its |
| + cgroup before exec. If clone3 placement is unavailable (the fallback has |
| + latched, or the cgroup cannot be opened), the start is refused with |
| + `cgroup.ErrLimitNotApplied` and is never done by moving the PID after |
| + exec. Wakes of that box show the maintenance text, and the refusal is the |
| + box's condition, not a failure recorded against the build; a swap abandons |
| + its attempt and is retried, as for the populated legacy parent below. The flat (flag-off) start is unchanged. Recovery: turn the |
| + flag off and restart the agent, or fix the kernel or cgroup state. The same |
| + refusal covers a parent that cannot be made, does not delegate memory, or |
| + whose backstop or child limit cannot be written. The log line begins |
| + `cgroup:` and says "generation start refused". |
| +- **The seed obligation file `<db>.seed-unproven`** (`5e51c008`, `4c5e43b5`, |
| + `4771d147`). Before a seed runs, the agent writes `<db>.seed-unproven` |
| + beside the instance's database (three lines: `carlos seed obligation v1`, |
| + `db=owned-by-seed` or `db=there-before-seed`, `seed=flat` or |
| + `seed=nested`). A write that fails refuses the seed before anything runs. |
| + The file is removed only when the seed's processes are proven gone, and a |
| + database the seed created goes with it. While the file is there no wake |
| + treats the database as warm, and a wake that cannot prove the seed gone |
| + fails with `activator: seed unproven; processes remain` and holds the lease |
| + to its TTL. The obligation is a file, so it survives an agent restart. It |
| + does not end in `.db`, so the litestream glob never takes it. A box on a |
| + kernel without `cgroup.kill` proves a seed by its leader's exit alone |
| + (logged). To clear a stuck one, first prove the seed's processes gone |
| + yourself: the host's `carlos-exec@<host>` cgroup (and for a nested layout |
| + the host slice's seed unit) must show an empty `cgroup.procs` and |
| + `populated 0` in `cgroup.events`, and no process may still hold the |
| + database open. Only then delete `<db>.seed-unproven` (and, if its body says |
| + `db=owned-by-seed`, the database, `-wal` and `-shm` beside it, so the next |
| + wake seeds again). Never delete it to get past a refusal while a process |
| + may still be writing the database. A planted non-empty directory at that |
| + name keeps that tenant's row down until it is removed by hand. |
| - **The armed agent writes its failure book on start.** Since c12c35c3 an |
| agent with the flag on saves the book when it opens it, even when there |
| was no file, and refuses managed starts if it cannot (the same refusal as |
| @@ -316,7 +430,36 @@ by 076bd9c0 (final review round 1), pending merge. |
| |
| ### Real-box checks (need Paul's approval; not run by this branch) |
| |
| -- The kernel-gated tests `TestClaimContainmentCleanupRespawn` (now the |
| +Readiness round 2 adds these to the list. None of them was run. The kernel |
| +suites skip unless `CARLOS_CLAIM_SPAWN_KERNEL=1`, and the temp-dir cgroups of |
| +the unit tests cannot stand in for a real cgroupfs. |
| + |
| +- Task 7's kernel tests, in |
| + `internal/activator/genparent_kernel_linux_test.go`: |
| + `TestGenerationStartRefusedBesideADirectParentMember`'s "sibling control" |
| + (g1 starts and stays up, is capped in `g1`, and g2 starts beside it) and |
| + "recovery control" (kill the legacy process, the parent empties, g1 starts), |
| + and the live-g1 respawn: a live g1 survives each sidecar respawn and the |
| + parent's `cgroup.procs` stays empty. The same checks need to hold with the |
| + `CLONE_INTO_CGROUP` placement the box really has, and the refusal needs |
| + watching on a kernel where it is unavailable. |
| +- Task 8's seed and normal-stop kernel control: a forked helper killed by |
| + `cgroup.kill` before the lease goes, for a flat seed, a nested seed unit |
| + and a flag-off stop. No kernel test of it exists yet. `KillAndEmpty`, the |
| + wake-time `EmptyHost` and a nested seed unit stopped by root were only |
| + modelled through seams. |
| +- The real-unit control and the sidecar checks (a unit route and the |
| + sidecar members on dev, including a refused sidecar restarting on the |
| + instance's own build). |
| +- Dev rollout, and a retry across a mixed-version roll on dev: each |
| + transition under "Retries: supported transitions", with an old box and a |
| + new console, a new box and an old console, and a book that has not saved. |
| + Include a data-dir copy with the `book` key dropped, and a downgrade onto a |
| + schema 3 book to see the refusal. |
| +- The flagship probe's gates E, F and G, as before. |
| +- Round 1's kernel checks, still as listed next. |
| + |
| + `TestClaimContainmentCleanupRespawn` (now the |
| KillHost and KillSidecar respawn) and |
| `TestGenerationStartRefusedBesideADirectParentMember` on the dev box |
| (arm64, `CARLOS_CLAIM_SPAWN_KERNEL=1`, delegated user service). Also check |
amadan