| 1 | # Zero-downtime flag readiness, round 2: Implementation Plan |
| 2 | |
| 3 | > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| 4 | |
| 5 | **Goal:** Close every remaining CODE blocker to turning `CARLOS_SWAP` on, as listed by Astra's readiness round 3 and the "Still owed (code)" section of the roll notes, with Paul's 2026-10-12 ruling on the crash-epoch guard. |
| 6 | |
| 7 | **Architecture:** Nine tasks, each small and self-contained. Task 1 replaces the bounded crash-epoch advance with a provable rule: a restriction's generation is a per-app counter that every change of the app's restrictions moves to a fresh value, published or accepted only once it is on disk, under a book identity rotated when the rule is adopted. Task 2 proves it in the rollout simulation with a model-side invariant and mutants. Tasks 3 and 4 move signed host retries to their own object and show a declined box's hold in the console. Tasks 5 to 8 close the declined-box, one-shot, cgroup and process-proof residuals. Task 9 writes the spec rev-18 amendment and the roll notes. |
| 8 | |
| 9 | **Tech Stack:** Go; `internal/rollout`, `internal/activator`, `internal/cgroup`, `internal/release`, `internal/host`, `internal/console`; `internal/rollout/testdata/rolloutsim/mutate.sh`. |
| 10 | |
| 11 | **Spec:** /home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md (rev 17, binding). Task 9 drafts rev 18 for Paul's sign-off. |
| 12 | |
| 13 | **Base:** origin/main 650e6152. Branch `zdd-flag-readiness-2`. Plan revision 2 answers Astra's plan review (`docs/superpowers/plans/CODEX-FINDINGS-ZDD-READINESS2-PLAN.md`); "(plan N)" marks the finding a change answers. |
| 14 | |
| 15 | **Inputs:** `docs/superpowers/plans/CODEX-FINDINGS-ZDD-READINESS{,-R2,-R3}.md`; roll notes `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` §"Still owed (code), after zdd-flag-readiness"; round 1's ledger `~/.cache/zdd-impl/readiness-ledger.md`; amadan discussion #272. Recon notes: `~/.cache/zdd-impl/readiness2/recon-{A-declined,B-good-f6,C-cgroup,D-retry-console}.md` (each task names its file; read it, it carries the line anchors; where this plan and a recon note differ, this plan wins). |
| 16 | |
| 17 | ## Global Constraints |
| 18 | |
| 19 | - `CARLOS_SWAP` stays unset. Nothing in this plan runs on a real box. |
| 20 | - Flag-off behaviour does not change, except where a task says it is an authorized flag-off change (Task 8's failed-seed and normal-stop proofs; Task 3's move of retries, which no released binary ever wrote). |
| 21 | - No new disk I/O under `activator.Service.mu`. Keep `rollout.Manager`'s documented lock order (`internal/rollout/manager.go`, the lock-order comment). |
| 22 | - Wire changes are additive: an older reader ignores a new field. |
| 23 | - No liveness trap: no instance may be left down forever with no supported recovery. |
| 24 | - New member-facing strings carry `// TODO-COPY(<id>)` on the literal's line, ids prefixed `zdd-readiness2.`. Draft text only: no em dashes (a full stop and a new sentence instead). Operator log lines are not copy. |
| 25 | - Cross-compile: linux-only syscalls need `//go:build linux` and a `!linux` stub; CI builds darwin and windows (`GOOS=darwin go build ./...`, `GOOS=windows go build ./...`). |
| 26 | - Tests run serially and niced (`nice -n 19`). The environment follows the user's global scratch rules (scratch not directly in `/tmp`, which is a shared RAM-backed tmpfs; never `GOCACHE` or `GOTMPDIR` under `/tmp`): |
| 27 | - Sandboxed Go uses `export GOCACHE=$TMPDIR/gocache GOFLAGS=-mod=mod GOPROXY=off` in every command. That cache is on disk, not in `/tmp`: the sandbox's `$TMPDIR` is under `~/.cache` (`~/.cache/tmp/claude-<uid>`). It is shared; never delete it. |
| 28 | - Packages that bind unix sockets (activator, edge, host, some console tests) or touch cgroupfs run unsandboxed with the default Go cache (`~/.cache/go-build`) and `TMPDIR=/tmp/k GOFLAGS=-buildvcs=false`. `/tmp/k` is used only because a socket test needs a short real path: `sun_path` is 108 bytes, and a test temp dir under `~/.cache` is too long for it. It holds only the tests' own temp dirs, never a cache or a build. `mkdir -p /tmp/k` first, and empty what you created there after the run. |
| 29 | - Commits: the worktree's identity is set (`claude` / `claude~paul@keymail.dev`); check `git config user.email` before the first commit. Plain `git add <files>` and `git commit -m`; never `git stash`, `git checkout -- <file>`, `git worktree prune`, `git reset --hard`. |
| 30 | - Integrity: every test result reported is pasted from a real run. |
| 31 | - Simulation: `mutate.sh` tests `git archive HEAD`, so **commit before every mutate.sh run** and name the commit it tested (plan 11). `mutate.sh 300` inside a task; the one 3000-seed sweep per branch revision is the controller's, at `nice -n 19`, GOMAXPROCS=2. |
| 32 | |
| 33 | ## Review Focus |
| 34 | |
| 35 | 1. A box whose failure-book save keeps failing (full or read-only disk): its holds stay visible, no epoch is offered for signing, the save is retried on its own until the disk recovers, and nothing a console signed earlier can clear a failure recorded later, across any number of agent restarts. (Task 1.) |
| 36 | 2. A box upgraded from a round-1 binary that had published an unsaved epoch: grants issued before the upgrade clear nothing after it. (Task 1, migration test.) |
| 37 | 3. A retry pressed during a mixed-version roll: applied with full checks, or refused with a message by a new console, or (old console, new box) honoured only if its signed grant passes the same checks. Never applied unchecked. A retry requested just before an agent restart is applied after it. (Task 3.) |
| 38 | 4. A one-shot worker that exits 0 just as the box loses its lease or is fenced: its build is not recorded as good. (Task 6.) |
| 39 | 5. A flag-off instance that forks a helper, or whose sidecar start is still in flight, and is put to sleep: the lease is not released while anything of it still runs; a failed seed's partial database is never served by the next wake. (Task 8.) |
| 40 | |
| 41 | --- |
| 42 | |
| 43 | ## Rulings recorded by this plan |
| 44 | |
| 45 | - Ruling: the crash-epoch guard is a hard rule, not the 2^20 advance. A restriction's generation is a per-(account, app) counter in the book, moved to a fresh sequence value by every change to any restriction of that app (records created, merged, retried, cleared by pointer or pruned; markers set, renewed, replaced or cleared; intents reconciled); it is published (status) or accepted (`RetryHostAt`) only while the book's memory equals its last landed save; the book identity is rotated, durably, the first time a book is opened under the rule; identities come from crypto/rand, never the clock. `reopenAdvance` is removed. — Paul's ruling 2026-10-12; per-app rather than per-host because a record restricts every host of its app, so per-host values cannot be kept exact on removals (plan 1) — cost if wrong: a retry pressed while another instance of the same app changes state is refused once and must be pressed again; a box with a failing disk shows holds with no Retry until a save lands. |
| 46 | - Ruling: signed host retries move to their own object `apps/<acct>/<app>/host-retry.json`, which new consoles write. New boxes also honour a signed grant found in `restart-scope.json` (a console built from main between 5da1cc9b and this branch writes there), with the same checks. The console writes a retry only to a host whose report carries `retry_v >= 1` and a non-zero epoch. — no released binary writes or applies a retry (v0.25.1 has none) — cost if wrong: a box on an intermediate dev build cannot be retried from a new console until it is updated. |
| 47 | - Ruling: retry grants are not seed-skipped at agent restart; every grant still inside its TTL is applied on first sight, because a spent or stale grant is a no-op under Task 1's generation rule (plan 9). — cost if wrong: one signature check per pending grant per changed log read. |
| 48 | - Ruling: the rev-18 spec amendment is a new file on this branch (`docs/superpowers/specs/2026-10-12-zero-downtime-deploys-rev18-amendment.md`), not an edit to the design branch, until Paul signs it off. — cost if wrong: one copy step after sign-off. |
| 49 | - Ruling: a stranded legacy sidecar recovers by running the instance's own build when the link's build is refused (recon A §3a). The recovery probe's dropped staged file (§3b) is not changed: that would change flag-off behaviour on claims-on boxes, and §3a closes the stranding. — cost if wrong: one extra hash per refused sidecar restart. |
| 50 | - Ruling: the failed-seed proof and the normal-stop half of F6 are authorized flag-off changes (Paul's scope item 5). A stop proves the host cgroup empty before the lease goes, killing only when something is left, and waits out any sidecar start in flight; a box with no delegation keeps the leader-only proof (round 1 ruling, ledger line 127). — cost if wrong: one `cgroup.events` read per flag-off stop on delegated boxes, and a lease held to its TTL when the proof fails. |
| 51 | - Ruling: an unproven failed seed leaves a durable obligation beside its database; no later wake treats that database as warm until the host's processes are proven gone and the partial database is removed (plan 3). — cost if wrong: a fresh instance waits one lease TTL and a proof before the seed is retried. |
| 52 | - Ruling: on the armed generation path, a start that cannot place the process in its cgroup before it executes (the clone3 fallback is latched, or the cgroup cannot be opened) is refused before exec with `ErrLimitNotApplied`; the PID-move fallback is never used there (plan 4). — cost if wrong: an armed box on a kernel without CLONE_INTO_CGROUP refuses every generation start (a box condition; recovery is flag off and an agent restart). |
| 53 | - Ruling: a clean one-shot exit counts as good only when the sweep parked it after its own fence and lease checks; any other stop of an exited one-shot is abandoned. — cost if wrong: a one-shot whose renewal fails just after a clean exit is retried instead of recorded good. |
| 54 | - Ruling: real-box checks are out of scope and go in the roll notes (Task 9). |
| 55 | |
| 56 | --- |
| 57 | |
| 58 | ### Task 1: The crash-epoch hard limit |
| 59 | |
| 60 | Model: opus (durability, security). No recon note: this section is the brief. |
| 61 | |
| 62 | **Files:** |
| 63 | - Modify: `internal/rollout/failures.go` (`failureFile`, `FailureBook`, `reopenAdvance`, `newBookID`, `OpenFailureBook`, `saveLocked`, `RestrictionEpoch`, `restrictionEpochLocked`, `RetryHostAt`, `Record`/`recordLocked`, `markLocked`, `ClearHoldRan`, `ClearHoldRanBefore`, `ClearHoldRanOnPointer`, `ClearOnPointer`, `Prune`, `retryHostLocked`, `reconcileLocked`, `epochOlderLocked`) |
| 64 | - Modify: `internal/rollout/status.go` (the two `hr.HeldEpoch = ...` producers), `internal/rollout/manager.go` (the periodic pass calls `SaveIfDirty`), `internal/host/rolloutdeclined.go` (`failureHolds.report`; startup flush; the periodic declined pass calls `SaveIfDirty`) |
| 65 | - Modify: `internal/release/hostretry.go` (doc only) |
| 66 | - Test: `internal/rollout/failures_epoch_test.go` (new), `internal/host/rolloutdeclined_test.go` |
| 67 | |
| 68 | **Interfaces:** |
| 69 | - Produces: `func (b *FailureBook) PublishedEpoch(acct, app, build, host string) uint64` — the restriction's generation when something of `build` restricts `host` and the book's memory equals its last landed save; else 0. Status and the declined report publish this, never `RestrictionEpoch`. |
| 70 | - Produces: `func (b *FailureBook) SaveIfDirty() error` — saves when a save is owed, else nil. |
| 71 | - Produces: `failureFile.Gens map[string]uint64 \`json:"gens,omitempty"\`` keyed `acct + "/" + app`; `failureFile.Durable bool \`json:"durable_epochs,omitempty"\``. |
| 72 | - Produces: `func newBookID() (string, error)` through a seam `var randRead = rand.Read`; `OpenFailureBook` returns its error. |
| 73 | - Task 3 consumes: status's `HeldEpoch == 0` on a held host means "not saved yet". |
| 74 | |
| 75 | **The rule (put it, with the proof, in the FailureBook doc comment):** |
| 76 | 1. **One generation per app.** `b.gens[acct/app]` takes `b.bumpLocked()` on every change to any restriction of that app: a record created; a merge that changes the record (`Changed`); a host retried (`retryHostLocked`); a record removed (`ClearOnPointer`, `Prune`); a marker of the app set, renewed (`markLocked`, including replacement of one build's marker by another's), or removed (`ClearHoldRan`, `ClearHoldRanBefore`, `ClearHoldRanOnPointer`, `RetryHostAt`'s removal); an intent reconciled into a record; an epoch given to an older record or marker (`epochOlderLocked`). A call that changes nothing (`PreRetry`, `Unchanged`, a duplicate marker of the same execution) moves nothing. The restriction's generation for (build, host) is `b.gens[acct/app]` when the build's record blocks the host or the host's marker names the build, else 0. Prune drops an app's generation only together with the app's last record and marker; a later restriction of that app takes a fresh value, which is larger than every earlier one. |
| 77 | 2. **Durable before use.** `b.dirty` is true whenever memory differs from the last landed save. Audit every function that changes `recs`, `held`, `intents`, `gens` or `seq`: each saves under the same hold of `b.mu`, or leaves `dirty` true. `PublishedEpoch` answers 0 while dirty. `RetryHostAt` first retries an owed save; if the book is still dirty it clears nothing. |
| 78 | 3. **Monotonic across restarts.** Every landed save writes the in-memory `seq`, which only grows, and a reopen starts from the file's `seq` (maxed with every saved generation, marker and epoch, as today). So the disk's `seq` only grows, every published generation is at most the disk's `seq`, and every value a later run hands out is larger. Remove `reopenAdvance`, its `+=` lines and the comment calling it a bound. |
| 79 | 4. **Adoption rotates the identity (plan 2).** A file without `durable_epochs` (written by an earlier binary, which may have published epochs it never saved) is opened with a new book identity and `durable_epochs` set, and owes a save (dirty, so nothing is published until it lands). Every grant issued before names the old identity and clears nothing. |
| 80 | 5. **Identity from crypto/rand only.** `newBookID` returns an error when `randRead` fails; `OpenFailureBook` returns it (an armed box refuses managed starts, as for an unreadable book; a declined box logs and honours nothing new). A memory-only book (empty path) may keep a fallback for tests. |
| 81 | 6. **The owed save is retried on its own (plan 10).** The armed manager's periodic pass and the declined box's periodic pass call `SaveIfDirty`; a declined box flushes a dirty book at startup whatever reconciliation found. |
| 82 | |
| 83 | Proof sketch: a grant G names book X, app A and generation E, which status published, so E was on X's disk under the rule (2, 4). `RetryHostAt` clears only when X is this book, memory equals disk, and A's generation now equals E. Within a run every change takes a fresh larger value (1), so equality means no change of A's restrictions since publication. Across a crash the reopened book starts at the disk's `seq` ≥ E and hands out larger values (3). A book written before the rule has another identity (4), and another book has another random identity (5). So G clears only the restriction state it was issued for. |
| 84 | |
| 85 | - [ ] **Step 1: Write the failing tests** in `internal/rollout/failures_epoch_test.go`. Use a real temp-dir book and make a save fail by making the directory read-only (`os.Chmod(dir, 0o500)`, restored in `t.Cleanup`; `t.Skip` when `os.Geteuid() == 0`). Exact sequences (plan 8): |
| 86 | |
| 87 | ```go |
| 88 | func TestAFailingDiskPublishesNoEpoch(t *testing.T) |
| 89 | // Record{v2 on h1, HeldDown h1} saved; e1 := PublishedEpoch(a, app, v2, h1) > 0. |
| 90 | // chmod read-only; Record a NEW build v3 failing on h2 (creates a record: a change of the app); save fails. |
| 91 | // PublishedEpoch(.., v2, h1) == 0 while dirty; RetryHostAt(.., v2, h1, id, e1) clears nothing (rec=false, mark=false). |
| 92 | // chmod back; SaveIfDirty() lands; PublishedEpoch(.., v2, h1) > e1; a grant at that value clears h1. |
| 93 | |
| 94 | func TestEveryRemovalMovesTheGeneration(t *testing.T) // one subtest per path (plan 1) |
| 95 | // For each: e := PublishedEpoch(...); perform the change; PublishedEpoch != e and RetryHostAt(.., e) clears nothing. |
| 96 | // - marker of v2 on h1 replaced by a marker of v3 on h1 while v2's record holds h1 |
| 97 | // - v2's record removed by ClearOnPointer while v2's marker on h1 stands |
| 98 | // - v2's record pruned while v2's marker on h1 stands |
| 99 | // - marker of v2 on h1 cleared by ClearHoldRanBefore, ClearHoldRanOnPointer, ClearHoldRan |
| 100 | // - a merge that changes v2's record because another host failed with a newer PromotedAt |
| 101 | |
| 102 | func TestANoOpOutcomeDoesNotMoveTheGeneration(t *testing.T) |
| 103 | // PreRetry, Unchanged and a duplicate marker of the same execution leave PublishedEpoch unchanged, |
| 104 | // and a grant at that epoch still clears. |
| 105 | |
| 106 | func TestABookFromBeforeTheRuleGetsANewIdentity(t *testing.T) // plan 2 |
| 107 | // Write a schema-2 file by hand without durable_epochs, with book "old", seq 10, a record of v2 holding h1 |
| 108 | // with epochs {"": 11, h1: 11}. Open: ID() != "old"; PublishedEpoch == 0 until Flush lands; |
| 109 | // RetryHostAt(.., "old", 11) and RetryHostAt(.., "old", 12) clear nothing; after Flush, a grant naming the new |
| 110 | // identity and the published generation clears h1. Reopen: the identity and durable_epochs persist. |
| 111 | |
| 112 | func TestAReopenNeverReissuesAPublishedGeneration(t *testing.T) |
| 113 | // Publish e1; make three changes that save; reopen from disk; make three more changes; |
| 114 | // every PublishedEpoch read after the reopen is > the largest read before it. |
| 115 | |
| 116 | func TestAGrantForAnotherBookClearsNothing(t *testing.T) // two books opened with equal numeric state |
| 117 | func TestABookThatCannotGetAnIdentityDoesNotOpen(t *testing.T) // randRead fails -> OpenFailureBook errors |
| 118 | ``` |
| 119 | |
| 120 | In `internal/host/rolloutdeclined_test.go`: a declined box whose book opened dirty and whose disk recovers publishes an epoch after the next periodic pass with no pointer or grant arriving (plan 10). |
| 121 | |
| 122 | - [ ] **Step 2: Run them and watch them fail.** `nice -n 19 go test ./internal/rollout/ -run 'TestAFailingDisk|TestEveryRemoval|TestANoOp|TestABookFromBefore|TestAReopenNever|TestAGrantForAnother|TestABookThatCannot' -count=1` (sandboxed env). Paste the failures. |
| 123 | - [ ] **Step 3: Implement rules 1 to 6.** Switch both status producers and `failureHolds.report` to `PublishedEpoch`. Keep `RestrictionEpoch` as the in-memory value for logs, or remove it if nothing else needs it. Update `HostRetryGrant.Epoch`'s doc (it is the app's restriction generation as published). |
| 124 | - [ ] **Step 4: Run** the new tests, `go test ./internal/rollout/ -count=1`, and unsandboxed `TMPDIR=/tmp/k nice -n 19 go test ./internal/host/ -run 'Retry|Declined|Rollout' -count=1`. Tests that relied on `reopenAdvance` or per-host epochs are updated to the new rule, not deleted; list them in the report. |
| 125 | - [ ] **Step 5: Commit** `rollout: a restriction's generation is durable before use and never repeats (Paul's hard limit)`. |
| 126 | - [ ] **Step 6: Run `internal/rollout/testdata/rolloutsim/mutate.sh 300`** on that commit (baseline plus M1–M12 CAUGHT). Paste the summary and the commit it tested. |
| 127 | |
| 128 | ### Task 2: Simulation: stale grants, crashes and failed saves; mutants; sweep timeout |
| 129 | |
| 130 | Model: opus. |
| 131 | |
| 132 | **Files:** |
| 133 | - Modify: `internal/rollout/sim_test.go`, `internal/rollout/testdata/rolloutsim/mutate.sh`, and `// rolloutsim:Mn` marker comments on the lines Task 1 wrote in `internal/rollout/failures.go`. |
| 134 | |
| 135 | **Interfaces:** |
| 136 | - Consumes: Task 1's `PublishedEpoch`, `RetryHostAt`, `SaveIfDirty`, `OpenFailureBook`, the generation and durability rules. |
| 137 | - Produces: invariants I9 and I10; mutants M13–M16. |
| 138 | |
| 139 | The simulation does not model stale grants today (round 1 ledger, Task 8 minor 8). For the seeds whose book is on disk (the I8 machinery), add: |
| 140 | - **Grants.** At random steps, read a held host's published (book, generation) as status does and queue a grant, stamped with the model step it was issued at. Deliver queued grants at later random steps through `RetryHostAt`, some after many events and some after a restart. The queue lives in the model and survives restarts and book recreation. |
| 141 | - **Failed saves and crashes.** Random read-only windows (as I8 does), and random agent restarts that reopen the book from disk under a new Manager. A restart drops in-flight executions as a real one does. |
| 142 | - **Book recreation** (for M16): at a random step, remove one seed's book file and open a new book (a box rebuild). |
| 143 | |
| 144 | Model-side facts, kept outside the book and never derived from the book's verdicts (plan 7): for every execution the model starts it knows the account, app, build, host and its start step; for every failure or hold-down the model delivers, it records a **failure fact** (app, build, host, start step). A failure fact is **answered** by any clear the model issued after the fact's start step: a grant for that (build, host) issued after it, a signed rollback or `promote --retry` pointer for that build signed after it. These are the clears the spec allows to lift it; a grant or pointer issued before the execution began is not one. |
| 145 | |
| 146 | Invariants: |
| 147 | - **I9 (a stale grant never clears a later failure):** when `RetryHostAt` for a grant issued at step t clears anything, no unanswered failure fact of the grant's (build, host) has a start step after t. Message `rolloutsim: I9 violated at seed %d: ...`. |
| 148 | - **I10 (durable before use):** every non-zero value `PublishedEpoch` returns equals the app's generation in the book's last landed file, which the model re-reads from disk at that moment. Message `rolloutsim: I10 violated at seed %d: ...`. |
| 149 | |
| 150 | Mutants (one `// rolloutsim:Mn` marker on exactly one line each, a perl substitution in `MUTANTS`; give each a witness, plan 8): |
| 151 | - **M13** `PublishedEpoch` ignores the dirty check → I10: publish while a save has failed, compared with the last landed file. |
| 152 | - **M14** one removal path (pick the marker replacement in `markLocked`) does not move the generation → I9: witness is a grant issued under v2's marker, the marker replaced by a failed hold run of v3 on the same host, then v3's marker cleared by a pointer, then the grant delivered. |
| 153 | - **M15** the reopen ignores the saved generations and markers when it sets `seq` (keep only `f.Seq`, then subtract enough to force reuse, for example `b.seq = f.Seq / 2`): witness removes the high-valued components before the reopen so the max cannot repair it, then forces numerical reuse → I9. |
| 154 | - **M16** `RetryHostAt` skips the book-identity comparison → I9 through book recreation. If the simulation cannot reach it, make it a `test:` mutant on `TestAGrantForAnotherBookClearsNothing` and say why in the list comment, as M12 does. |
| 155 | |
| 156 | mutate.sh: update the header and verdict comments to name I1 to I10 (the check regex already accepts any digits; plan 12). Replace the fixed `-timeout 20m` with a seeds-scaled value, `TIMEOUT_MIN=$(( SEEDS * 3 / 100 + 20 ))` (3000 seeds → 110 minutes), used as `-timeout ${TIMEOUT_MIN}m`, and print it in the header. Round 1 found 20 minutes too short for 3000 seeds. |
| 157 | |
| 158 | - [ ] **Step 1:** Add the model state, I9 and I10. Commit (`rollout sim: stale grants, crashes and failed saves; I9, I10`). Run `ROLLOUT_SIM_SEEDS=300 GOMAXPROCS=2 nice -n 19 go test ./internal/rollout/ -run '^TestSimInvariants$' -count=1`; it must pass. Paste the line. |
| 159 | - [ ] **Step 2:** Add the markers and the four mutants, commit, run `mutate.sh 300 M13 M14 M15 M16`. Each must read CAUGHT on its stated invariant. Paste the summary and the tested commit. |
| 160 | - [ ] **Step 3:** Run `mutate.sh 300` (all mutants) on the final commit. Paste the summary. |
| 161 | |
| 162 | ### Task 3: Retry grants in their own object, and the console's capability check |
| 163 | |
| 164 | Model: opus (security). Recon: `recon-D-retry-console.md` §2, with this plan's corrections. |
| 165 | |
| 166 | **Files:** |
| 167 | - Modify: `internal/release/restartscope.go` (`ValidRestartScope` refuses kind retry and `Signed` for new writes), new `internal/release/hostretrylog.go` (`HostRetryKey`, `LoadHostRetries`, `AppendHostRetry`, `claimRawAppendHostRetry`) |
| 168 | - Modify: `internal/host/restart.go`, `internal/host/restartscope.go` (a `hostRetryPass` over the new object; legacy retry entries in `restart-scope.json` are still verified and applied with the same checks; neither is seed-skipped at restart) |
| 169 | - Modify: `internal/status/snapshot.go` (`HostRollout.RetryV int \`json:"retry_v,omitempty"\``), set to 1 beside `HeldBook` at both producers in `internal/rollout/status.go` and in `internal/host/rolloutdeclined.go` |
| 170 | - Modify: `internal/console/apiinstances.go` (`rolloutView.retryV`, copied where `heldEpoch` and `heldBook` are), `internal/console/apiretry.go` (`retryHostBuild` reads the restriction and refuses; `requestHostRetry` performs the append, now through `AppendHostRetry`; plan 12), `internal/console/browserretry.go` |
| 171 | - Modify: `internal/provision/plan9_b8_census_test.go` (a digest row for `claimRawAppendHostRetry`, and any changed `claimRaw*` digest) |
| 172 | - Test: `internal/release/hostretrylog_test.go` (new), `internal/host/restartscope_test.go`, `internal/console/apiretry_test.go`, `internal/console/browserretry_test.go` |
| 173 | |
| 174 | **Interfaces:** |
| 175 | - Consumes: Task 1's published generation (`HeldEpoch` 0 on a held host while the book is not saved). |
| 176 | - Produces: |
| 177 | - `func HostRetryKey(acct, app string) string` → `apps/<acct>/<app>/host-retry.json` |
| 178 | - `func LoadHostRetries(ctx context.Context, s Store, acct, app string) (*ScopedRestartGeneration, bool, error)` |
| 179 | - `func AppendHostRetry(ctx context.Context, s Store, acct, app, host, signed string) (RestartRequest, error)` (same JSON shape and CAS loop as the restart-scope log, keep 32, op name `release.AppendHostRetry`) |
| 180 | - `status.HostRollout.RetryV`; `rolloutView.retryV` |
| 181 | - console errors `errRetryUnsupported` (report has `retry_v == 0` or no `held_book`) and `errRetryUnsaved` (`retry_v >= 1`, held, `held_epoch == 0`) |
| 182 | |
| 183 | Supported transitions (plan 9), each pinned by a test: |
| 184 | - new console, new box: the grant rides `host-retry.json`, the box verifies and applies it. |
| 185 | - new console, older box (no `retry_v`): the console refuses with `errRetryUnsupported` and writes nothing. |
| 186 | - console built from main before this branch, new box: its signed, book-bound grant in `restart-scope.json` is verified and applied with the same checks (an unsigned one is dropped with a log line, as today). A v0.25.1 console cannot write a retry at all. |
| 187 | - a grant written before an agent restart and not yet applied is applied after it. |
| 188 | Write the guarantee as: never applied unchecked; refused with a message by a new console when the box cannot take it; an unsigned or stale request is dropped with an operator log line. |
| 189 | |
| 190 | New strings (draft, TODO-COPY): |
| 191 | - `zdd-readiness2.retry-unsupported.api`: "this instance's box is too old to take a retry. update the box, then try again" |
| 192 | - `zdd-readiness2.retry-unsupported.browser`: "This instance's box is too old to take a retry. Update the box, then try again." |
| 193 | - `zdd-readiness2.retry-unsaved.api`: "this instance's box has not saved its failure yet. try again in a minute" |
| 194 | - `zdd-readiness2.retry-unsaved.browser`: "This instance's box has not saved its failure yet. Try again in a minute." |
| 195 | The API door answers 409 with these; the browser door flashes them. Neither writes anything. |
| 196 | |
| 197 | Tests (write first, watch fail): release round trip; a restart-scope rewrite by a struct without `Signed` leaves `host-retry.json` untouched; `ValidRestartScope` refuses a new retry kind. Host: the three existing retry tests (`TestRestartLegHandsARetryToTheRolloutManager`, `...DropsARetryThatIsNotTheRootsGrant`, `...RetryGrantsAgainstTheBook`) cover the new object; a legacy signed grant in `restart-scope.json` is applied; a legacy unsigned one clears nothing; a grant written before an agent restart is applied after it (the restart memory is rebuilt). Console: no `retry_v` → 409 unsupported, nothing written; `retry_v: 1`, held, `held_epoch: 0` → 409 unsaved; `retry_v: 1` with epoch and book → the signed entry lands in `host-retry.json` and nothing in `restart-scope.json` (update `lastScoped`). |
| 198 | |
| 199 | - [ ] Steps: failing tests; implement; `go test ./internal/release/ ./internal/console/ -count=1` sandboxed; unsandboxed `TMPDIR=/tmp/k nice -n 19 go test ./internal/host/ -run 'Restart|Retry' -count=1` and `go test ./internal/provision/ -run Census -count=1`; commit `retries: signed host retries ride their own object; the console asks the box's report first`. |
| 200 | |
| 201 | ### Task 4: A declined box's hold and Retry in the console |
| 202 | |
| 203 | Model: sonnet. Recon: `recon-D-retry-console.md` §1. |
| 204 | |
| 205 | **Files:** |
| 206 | - Modify: `internal/console/instancepage.go` (`HoldText`, `PromoteRetryText`, `RolloutNote`; `CanRetry` on the rollout view and a forwarding `instanceDetail.CanRetry`, since the template's dot is `instanceDetail`; plan 12), `internal/console/apiretry.go` (`heldRestriction.declined`), `internal/console/browserretry.go`, `internal/console/swapcopy.go` (two new strings), `internal/console/templates/app_instance.html` (Retry form on `.CanRetry`) |
| 207 | - Create: `copy-review/strings.json` (force-added with `git add -f`; `copy-review/` is excluded), covering every TODO-COPY string on the branch (Task 3's four and this task's two), rows `{id, section, label, text, context, notes}` |
| 208 | - Test: `internal/console/browserretry_test.go`, `internal/console/apiretry_test.go` |
| 209 | |
| 210 | **Interfaces:** |
| 211 | - Consumes: Task 3's `rolloutView.retryV`, `errRetryUnsupported`, `errRetryUnsaved`. |
| 212 | - Produces: `func (r *rolloutView) retryable() bool { return r != nil && r.HeldBy != "" && (r.Unavailable == "" || rolloutwords.UnavailableDeclines(r.Unavailable)) }`, used by both `retryHostBuild` (replacing its inline test) and the page. `CanRetry()` is `retryable() && retryV >= 1 && heldEpoch != 0 && heldBook != ""`. |
| 213 | |
| 214 | Page rules: |
| 215 | - The hold row shows whenever `retryable()`. Text: `Attempt == AttemptDown` → the approved `zdd.instance.hold-down-text` verbatim; declined (`Unavailable != ""`) → new `HoldDeclinedText`; otherwise today's `hold-text`. Never format `Reason` for a declined host (it is empty there). |
| 216 | - The Retry form shows on `CanRetry()`. The confirmation line for a declined host is new `RetryLineDeclined` (no canary runs there). |
| 217 | |
| 218 | New strings (draft, TODO-COPY): |
| 219 | - `zdd-readiness2.hold-declined`: "Build %s failed on this box. This box cannot run two builds side by side, so it will not start this instance on that build. Retry it, or ship a newer build." |
| 220 | - `zdd-readiness2.retry-line-declined`: "This clears the failure hold on {host}. Its box cannot run two builds side by side, so its next start runs {build} the old way, with no canary." |
| 221 | |
| 222 | Tests: a declined fixture (`Unavailable: "host-claims-on"`, a failed host and a down host, with `HeldBy`, `HeldEpoch`, `HeldBook`, `RetryV: 1`) shows the hold row, the Retry form and the declined confirmation; a refusing code (`no-deploy-store`) with a held host shows neither; `TestInstancePageOfAnUnavailableBoxShowsNoHoldAndNoRetry` stays green (rename it if it now means "refusing"); a declined host with `HeldEpoch: 0` shows the hold row and no form. |
| 223 | |
| 224 | - [ ] Steps: failing tests; implement; `go test ./internal/console/ -run 'Retry|InstancePage|Copy' -count=1` sandboxed; write `copy-review/strings.json`; commit `console: a declined box's hold and retry on the instance page (copy pending review)`. |
| 225 | |
| 226 | ### Task 5: Declined boxes: pin the live link once, and recover a stranded sidecar |
| 227 | |
| 228 | Model: opus (security). Recon: `recon-A-declined.md` (read all of it: settings plan 4, 8916558d, already closed most of the production path; this task makes the backend's live-link arms safe on their own and closes the sidecar). |
| 229 | |
| 230 | **Files:** |
| 231 | - Modify: `internal/activator/backend_exec.go` (`declinedLivePin`, `spawnUnitFrom`, `startSidecarUnitFrom`, `seedChild`, `seedUnit`, `BuildIdentity`; rewrite the residual comments the recon names), `internal/activator/activator.go` (`startSidecar`), the `SplitBackend` forward |
| 232 | - Test: `internal/activator/identity_test.go`, `internal/activator/exec_gate_test.go`, `internal/activator/declinedgate_identity_test.go`, `internal/activator/wakeidentity_test.go` (fake backend gains `BuildIdentity`) |
| 233 | |
| 234 | **Interfaces:** |
| 235 | - Produces: `func (b *ExecBackend) declinedLivePin(ctx context.Context, rt registry.Route) (release.StagedIdentity, error)`; `type buildIdentifier interface { BuildIdentity(rt registry.Route, build string) (release.StagedIdentity, error) }`; `func (b *ExecBackend) BuildIdentity(rt registry.Route, build string) (release.StagedIdentity, error)`. |
| 236 | |
| 237 | Design (recon A): when `LiveExecGate != nil` and a start has no staged file, resolve and hash the link once, ask the gate about that version, and run that identity by path (flat) or through `StartExe`/`StartSidecarExe`/`SeedExe` (nested: root pins the file). In seeds, do it before the `seedOwnsDB` stat so a refusal never deletes a database. In `startSidecar`, when the gate refuses the link's version and the generation's build differs, try the instance's own build through `BuildIdentity` and the gate; run it if both pass; otherwise keep the refusal. |
| 238 | |
| 239 | Tests (write first): a nested repoint between read and start runs v1 (`startexe ... v1`, gate asked `[v1]`, no plain `start`); the same for the sidecar; a link outside the store asks root for nothing; flat and nested seeds run the gated file and a refused seed creates no database; a sidecar under a link naming held v2 runs v1 (gate asked `[v2 v1]`), and with v1 refused too it stays down while the instance serves. |
| 240 | |
| 241 | - [ ] Steps: failing tests; implement; unsandboxed `TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -run 'Identity|Gate|Sidecar|Seed' -count=1`, then the whole package once; commit `activator: a declined box pins the live link once, and a refused sidecar runs the instance's build`. |
| 242 | |
| 243 | ### Task 6: One-shot good build after a fence or lost lease; the duplicate-wake window |
| 244 | |
| 245 | Model: opus (concurrency). Recon: `recon-B-good-f6.md` §1 and §2. |
| 246 | |
| 247 | **Files:** |
| 248 | - Modify: `internal/activator/workerswap.go` (`ReplaceWorker`'s forgiveness of `errSwapNotAwake`), `internal/activator/activator.go` (`actEntry.parkedInc`, `markCleanParkLocked`, the sweep's clean-exit park, `WakeWorker`, `begin`, new `awaitWake`) |
| 249 | - Test: `internal/activator/workerswap_test.go` |
| 250 | |
| 251 | **Interfaces:** |
| 252 | - Produces: `func (a *Service) markCleanParkLocked(e *actEntry)`; `func (a *Service) awaitWake(ctx context.Context, e *actEntry, ready <-chan struct{}, deadline *time.Timer, arrival time.Time, extended *bool) error`; test seam `var wakeBoundExtended func()` (nil in production). |
| 253 | |
| 254 | Design (recon B §1, §2): forgive `errSwapNotAwake` only when `e.parkedInc == inc1`, no fence stands and the service is not shutting down; every other stop of an exited one-shot is abandoned. `WakeWorker` and `begin` wait again on the same `ready` after extending the bound, never re-reading the entries map. The optional `servedOneShot` fence check in recon B is not needed and is not added. |
| 255 | |
| 256 | Tests (write first, watch fail): exit 0 then a renewal failure and a sweep → `SwapAbandoned`; exit 0 then `FenceAndStop` → `SwapAbandoned`; positive control, exit 0 then a sweep → `SwapDone` with `CleanExit`; a wake that fails inside the extension window → `WakeWorker` returns the wake's error and the backend counts one lease acquire (today two); the same for a request through `begin`. |
| 257 | |
| 258 | - [ ] Steps: failing tests; implement; unsandboxed `TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -run 'Worker|OneShot|Wake' -count=1`, then `-race` on the new tests, then the package once; commit `activator: a one-shot is good only after a clean park, and a waiting wake never starts a second`. |
| 259 | |
| 260 | ### Task 7: Fail closed when a generation's limits cannot be applied; kernel positive controls |
| 261 | |
| 262 | Model: opus. Recon: `recon-C-cgroup.md`, with case G replaced below. |
| 263 | |
| 264 | **Files:** |
| 265 | - Modify: `internal/cgroup/gen.go` (`ErrLimitNotApplied`, `ensureParent` branches A–D, `delegatesMemory`, `writeSubtreeControl` seam), `internal/cgroup/cgroup.go` (`startIn(..., required bool, ...)`, `applyLimit` returns its first failure; with `required`, never the PID-move fallback), `internal/activator/swap.go` (`verdictOf`), `internal/activator/logsink.go` (`memberSafeWakeReason` → `ErrFenced`; fix the stale comment the recon names) |
| 266 | - Test: `internal/cgroup/gen_test.go`, `internal/cgroup/cgroup_test.go`, `internal/activator/swap_overlap_test.go`, `internal/activator/swap_executed_test.go`, `internal/activator/logsink_test.go`, `internal/activator/genparent_kernel_linux_test.go`, `internal/activator/claimspawn_kernel_linux_test.go` |
| 267 | |
| 268 | **Interfaces:** |
| 269 | - Produces: `var ErrLimitNotApplied = errors.New("cgroup: a generation's limits could not be applied")`; `func (m *Manager) startIn(host, dir string, limit Limit, required bool, build func() *exec.Cmd, before func() error) (*exec.Cmd, error)`; `func (m *Manager) applyLimit(host, dir string, limit Limit) error`. |
| 270 | |
| 271 | Design: recon C §2 for A–F. `StartGated` passes `required=false` (the flag-off path is byte-for-byte the same); `StartChildGated` passes `true`. EBUSY on `subtree_control` maps to `ErrLegacyParentPopulated`. **Case G (plan 4):** with `required`, the start must place the process in its cgroup before it executes (clone3 `CLONE_INTO_CGROUP`). When the fallback is latched, or the cgroup directory cannot be opened for the clone, return `ErrLimitNotApplied` before `build()` runs; never start and move by PID. Fixtures that write `populated 1` with no `subtree_control` get one (`cpu memory io pids`). |
| 272 | |
| 273 | Tests: one unit test per branch A–F plus G without a kernel (the recon names the fixture trick for each; for G, store the fallback latch true), each asserting nothing started and no generation child left; `ErrLimitNotApplied` rows in `TestVerdictOf`, `TestClassifyStep`, `TestSwapResultReportsWhetherTheAttemptsBuildExecuted` (`SwapAbandoned`, not executed) and the logsink table. Kernel controls (skip-gated by `CARLOS_CLAIM_SPAWN_KERNEL=1`; they compile here and run on the dev box later): the sibling control and the recovery control in `TestGenerationStartRefusedBesideADirectParentMember`; a live g1 kept running through `killSidecarRespawn`'s loop. |
| 274 | |
| 275 | - [ ] Steps: failing tests; implement; unsandboxed `TMPDIR=/tmp/k nice -n 19 go test ./internal/cgroup/ -count=1`, then `go test ./internal/activator/ -run 'Verdict|Classify|SwapResult|Logsink|MemberSafe' -count=1`; `go vet ./internal/activator/` (compiles the linux kernel test files); `GOOS=darwin go build ./...`; commit `cgroup: a generation whose limits cannot be applied does not start`. |
| 276 | |
| 277 | ### Task 8: Prove a seed's processes gone; the flag-off normal-stop half of F6 |
| 278 | |
| 279 | Model: opus (concurrency, durability). Recon: `recon-B-good-f6.md` §3 and §4, with the corrections below (plan 3, 5, 6). |
| 280 | |
| 281 | **Files:** |
| 282 | - Modify: `internal/activator/backend_exec.go` (`seedDescendantsGone`, `seedChild`'s failure and success branches, `EmptyHost`, the unproven-seed obligation file, `Restore`'s warm-cache check), `internal/activator/activator.go` (`doWake`'s seed decision; `actEntry.launchLayout` set at install; `stop()`; the pending-stop path through new `reapUncontainedStop`; the in-flight sidecar start), `internal/activator/opset.go` (`stopHostProven`, `hostEmptier`), the `SplitBackend` forward |
| 283 | - Test: `internal/activator/activator_test.go`, `internal/activator/containment_test.go`, `internal/activator/wakeplan_test.go` (the `plannedHostKill` double pattern) |
| 284 | |
| 285 | **Interfaces:** |
| 286 | - Produces: `func (b *ExecBackend) seedDescendantsGone(ctx context.Context, rt registry.Route, child string) error`; `func (b *ExecBackend) EmptyHost(ctx context.Context, rt registry.Route) error`; `type hostEmptier interface{ EmptyHost(context.Context, registry.Route) error }`; `func (a *Service) stopHostProven(e *actEntry) bool`; `func (a *Service) reapUncontainedStop(e *actEntry)`; `const seedUnprovenSuffix = ".seed-unproven"`. |
| 287 | |
| 288 | Design: |
| 289 | 1. **Failed seed.** A failed flat seed's cgroup (flag on: its role child; flag off: the host cgroup) is killed and proven empty before `RemoveChild` and `removeSeedDB`. `ErrNoDelegation` keeps the leader-only proof with a log line. When the proof fails, write `<db>.seed-unproven` durably (safefile) beside the database, keep the database, and return an error wrapping `ErrSeedUnproven` (the lease is kept to its TTL, as today). |
| 290 | 2. **The obligation survives (plan 3).** Before any wake treats an existing database as warm (`Restore`, and `doWake`'s `fresh` decision), it checks for `<db>.seed-unproven`. If present: prove the host's processes gone (`seedDescendantsGone` with the same child rule); on success remove the database, its `-wal`/`-shm` files and the obligation, and seed afresh; on failure refuse the wake with `ErrSeedUnproven`. The obligation is a file, so it survives an agent restart. |
| 291 | 3. **Successful seed, flag on (plan 6).** On every successful completion, including a plain nil `Wait`, the seed's role child is killed and proven empty before `RemoveChild` (a helper that redirected its output and outlived the seed is killed). Flag off, a successful seed's helper stays in the host cgroup and is caught by the normal-stop proof. |
| 292 | 4. **Normal stop, flag off.** A flat or foreign exec row's terminal stop proves the host cgroup empty before `finishStop` releases the lease: `EmptyHost` reads `cgroup.events` and kills only when populated, so a clean sleep leaves no kill mark. `ErrNoDelegation` keeps the leader-only proof. Nothing runs under `a.mu`. |
| 293 | 5. **In-flight sidecar start (plan 5).** A terminal stop with a sidecar start still in flight past `stop()`'s wait keeps the stop pending, with the lease, until that start returns; its child, if any, is stopped, and only then does the host proof run. |
| 294 | |
| 295 | Tests (write first, watch fail): |
| 296 | - a failing seed on `delegatedBackend` whose cleanup cannot be proven returns `ErrSeedUnproven`, keeps the database and writes the obligation; `TestExecBackendSeedFailureRemovesHalfBuiltDB` stays green; |
| 297 | - **the next wake** after that: with the proof still failing it is refused and the database is not opened as warm; with the proof passing the database and obligation are removed and the seed runs again (assert the seed command ran twice); |
| 298 | - flag on, a seed that forks a helper which redirects its output, then exits 0: the helper is killed before the seed returns (fake cgroup double records `KillAndEmpty` of `role-seed`); |
| 299 | - flag-off idle stop with a failing `EmptyHost` double keeps the lease (`lastRep().isClosed() == false`); a passing double releases it after the proof (order asserted with `setOnClose`); `ErrNoDelegation` releases at once; the pending-stop path the same; |
| 300 | - a sidecar start delayed beyond `stop()`'s wait keeps the lease until it returns, and its child is stopped before the release. |
| 301 | |
| 302 | - [ ] Steps: failing tests; implement; unsandboxed `TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -count=1`; `GOOS=windows go build ./...`; commit `activator: seeds and flag-off stops prove their processes gone before the lease goes`. |
| 303 | |
| 304 | ### Task 9: Spec rev-18 amendment and roll notes |
| 305 | |
| 306 | Model: sonnet. Recon: `recon-D-retry-console.md` §3. |
| 307 | |
| 308 | **Files:** |
| 309 | - Create: `docs/superpowers/specs/2026-10-12-zero-downtime-deploys-rev18-amendment.md` |
| 310 | - Modify: `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` ("Still owed (code)", "Roll notes from zdd-flag-readiness", "Real-box checks") |
| 311 | |
| 312 | The amendment states, for Paul's sign-off: the status line for rev 18; replacements for spec line 3 (the rev-14 sentence), lines 704–705 (§8.4 `carlos instances retry`) and lines 836–837 (§11 "A retry is a re-signed channel pointer"); a new §11 paragraph with the grant as built (`HostRetryGrant` fields; the root-signed envelope and the scoped-key refusal; the 24 h TTL as a backstop and the 1-minute skew; book identity plus the app's restriction generation; Task 1's rule and proof sketch, including the identity rotation on adoption; unsigned or stale grants dropped with a log line; 503 with no signing key; declined boxes honour grants with no canary; grants ride `host-retry.json`, legacy signed grants in `restart-scope.json` are checked the same way; the console checks `retry_v`), and Paul's Q4 ruling (2026-10-11) and hard-limit ruling (2026-10-12) by date. `carlos promote --retry` stays the app-wide re-signed pointer (lines 761–762 and the line-998 table row are unchanged; say so). |
| 313 | |
| 314 | Roll notes: mark each closed item with its commit; replace the "reopen epoch bound" bullet with the hard rule; replace "Retries: boxes before consoles" with the supported transitions from Task 3; add the armed-box refusal when CLONE_INTO_CGROUP is unavailable (Task 7) and the seed obligation file (Task 8) as roll notes; list the real-box checks still owed: the kernel tests from Task 7 (sibling, recovery, live-g1 respawn) and round 1's, the seed and normal-stop kernel control from Task 8 (a forked helper killed before the lease goes), the real-unit control and sidecar checks, dev rollout and retry across a mixed-version roll, and the flagship probe gates E–G. |
| 315 | |
| 316 | - [ ] Steps: write both; check every commit hash cited with `git log --oneline`; commit `docs: rev-18 amendment for signed host retries (pending Paul); roll notes for readiness round 2`. |
| 317 | |
| 318 | --- |
| 319 | |
| 320 | ## Execution order and gates |
| 321 | |
| 322 | 1, 2, 3, 4, 5, 6, 7, 8, 9. Tasks 3 and 4 share console files (sequential). Tasks 5, 6 and 8 share `activator.go` and `backend_exec.go` (sequential). After Task 9 the controller runs the one 3000-seed sweep (`mutate.sh 3000`, nice 19, GOMAXPROCS=2) on the committed tip and the full gate, then Astra's whole-branch review. |
| 323 | |
amadan