Zero-downtime flag readiness, round 2: Implementation Plan

> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Close every remaining CODE blocker to turning CARLOS_SWAP on, as listed by Astra's readiness round 3 and the "Still owed (code)" section of the roll notes, with Paul's 2026-10-12 ruling on the crash-epoch guard.

Architecture: Nine tasks, each small and self-contained. Task 1 replaces the bounded crash-epoch advance with a provable rule: a restriction's generation is a per-app counter that every change of the app's restrictions moves to a fresh value, published or accepted only once it is on disk, under a book identity rotated when the rule is adopted. Task 2 proves it in the rollout simulation with a model-side invariant and mutants. Tasks 3 and 4 move signed host retries to their own object and show a declined box's hold in the console. Tasks 5 to 8 close the declined-box, one-shot, cgroup and process-proof residuals. Task 9 writes the spec rev-18 amendment and the roll notes.

Tech Stack: Go; internal/rollout, internal/activator, internal/cgroup, internal/release, internal/host, internal/console; internal/rollout/testdata/rolloutsim/mutate.sh.

Spec: /home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md (rev 17, binding). Task 9 drafts rev 18 for Paul's sign-off.

Base: origin/main 650e6152. Branch zdd-flag-readiness-2. Plan revision 2 answers Astra's plan review (docs/superpowers/plans/CODEX-FINDINGS-ZDD-READINESS2-PLAN.md); "(plan N)" marks the finding a change answers.

Inputs: docs/superpowers/plans/CODEX-FINDINGS-ZDD-READINESS{,-R2,-R3}.md; roll notes docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md §"Still owed (code), after zdd-flag-readiness"; round 1's ledger ~/.cache/zdd-impl/readiness-ledger.md; amadan discussion #272. Recon notes: ~/.cache/zdd-impl/readiness2/recon-{A-declined,B-good-f6,C-cgroup,D-retry-console}.md (each task names its file; read it, it carries the line anchors; where this plan and a recon note differ, this plan wins).

Global Constraints

• CARLOS_SWAP stays unset. Nothing in this plan runs on a real box.
• Flag-off behaviour does not change, except where a task says it is an authorized flag-off change (Task 8's failed-seed and normal-stop proofs; Task 3's move of retries, which no released binary ever wrote).
• No new disk I/O under activator.Service.mu. Keep rollout.Manager's documented lock order (internal/rollout/manager.go, the lock-order comment).
• Wire changes are additive: an older reader ignores a new field.
• No liveness trap: no instance may be left down forever with no supported recovery.
• New member-facing strings carry // TODO-COPY(<id>) on the literal's line, ids prefixed zdd-readiness2.. Draft text only: no em dashes (a full stop and a new sentence instead). Operator log lines are not copy.
• Cross-compile: linux-only syscalls need //go:build linux and a !linux stub; CI builds darwin and windows (GOOS=darwin go build ./..., GOOS=windows go build ./...).
• Tests run serially and niced (nice -n 19). The environment follows the user's global scratch rules (scratch not directly in /tmp, which is a shared RAM-backed tmpfs; never GOCACHE or GOTMPDIR under /tmp): - Sandboxed Go uses export GOCACHE=$TMPDIR/gocache GOFLAGS=-mod=mod GOPROXY=off in every command. That cache is on disk, not in /tmp: the sandbox's $TMPDIR is under ~/.cache (~/.cache/tmp/claude-<uid>). It is shared; never delete it. - Packages that bind unix sockets (activator, edge, host, some console tests) or touch cgroupfs run unsandboxed with the default Go cache (~/.cache/go-build) and TMPDIR=/tmp/k GOFLAGS=-buildvcs=false. /tmp/k is used only because a socket test needs a short real path: sun_path is 108 bytes, and a test temp dir under ~/.cache is too long for it. It holds only the tests' own temp dirs, never a cache or a build. mkdir -p /tmp/k first, and empty what you created there after the run.
• Commits: the worktree's identity is set (claude / claude~paul@keymail.dev); check git config user.email before the first commit. Plain git add <files> and git commit -m; never git stash, git checkout -- <file>, git worktree prune, git reset --hard.
• Integrity: every test result reported is pasted from a real run.
• Simulation: mutate.sh tests git archive HEAD, so commit before every mutate.sh run and name the commit it tested (plan 11). mutate.sh 300 inside a task; the one 3000-seed sweep per branch revision is the controller's, at nice -n 19, GOMAXPROCS=2.

Review Focus

1. A box whose failure-book save keeps failing (full or read-only disk): its holds stay visible, no epoch is offered for signing, the save is retried on its own until the disk recovers, and nothing a console signed earlier can clear a failure recorded later, across any number of agent restarts. (Task 1.)
2. A box upgraded from a round-1 binary that had published an unsaved epoch: grants issued before the upgrade clear nothing after it. (Task 1, migration test.)
3. A retry pressed during a mixed-version roll: applied with full checks, or refused with a message by a new console, or (old console, new box) honoured only if its signed grant passes the same checks. Never applied unchecked. A retry requested just before an agent restart is applied after it. (Task 3.)
4. A one-shot worker that exits 0 just as the box loses its lease or is fenced: its build is not recorded as good. (Task 6.)
5. A flag-off instance that forks a helper, or whose sidecar start is still in flight, and is put to sleep: the lease is not released while anything of it still runs; a failed seed's partial database is never served by the next wake. (Task 8.)

---

Rulings recorded by this plan

• Ruling: the crash-epoch guard is a hard rule, not the 2^20 advance. A restriction's generation is a per-(account, app) counter in the book, moved to a fresh sequence value by every change to any restriction of that app (records created, merged, retried, cleared by pointer or pruned; markers set, renewed, replaced or cleared; intents reconciled); it is published (status) or accepted (RetryHostAt) only while the book's memory equals its last landed save; the book identity is rotated, durably, the first time a book is opened under the rule; identities come from crypto/rand, never the clock. reopenAdvance is removed. — Paul's ruling 2026-10-12; per-app rather than per-host because a record restricts every host of its app, so per-host values cannot be kept exact on removals (plan 1) — cost if wrong: a retry pressed while another instance of the same app changes state is refused once and must be pressed again; a box with a failing disk shows holds with no Retry until a save lands.
• Ruling: signed host retries move to their own object apps/<acct>/<app>/host-retry.json, which new consoles write. New boxes also honour a signed grant found in restart-scope.json (a console built from main between 5da1cc9b and this branch writes there), with the same checks. The console writes a retry only to a host whose report carries retry_v >= 1 and a non-zero epoch. — no released binary writes or applies a retry (v0.25.1 has none) — cost if wrong: a box on an intermediate dev build cannot be retried from a new console until it is updated.
• Ruling: retry grants are not seed-skipped at agent restart; every grant still inside its TTL is applied on first sight, because a spent or stale grant is a no-op under Task 1's generation rule (plan 9). — cost if wrong: one signature check per pending grant per changed log read.
• Ruling: the rev-18 spec amendment is a new file on this branch (docs/superpowers/specs/2026-10-12-zero-downtime-deploys-rev18-amendment.md), not an edit to the design branch, until Paul signs it off. — cost if wrong: one copy step after sign-off.
• Ruling: a stranded legacy sidecar recovers by running the instance's own build when the link's build is refused (recon A §3a). The recovery probe's dropped staged file (§3b) is not changed: that would change flag-off behaviour on claims-on boxes, and §3a closes the stranding. — cost if wrong: one extra hash per refused sidecar restart.
• Ruling: the failed-seed proof and the normal-stop half of F6 are authorized flag-off changes (Paul's scope item 5). A stop proves the host cgroup empty before the lease goes, killing only when something is left, and waits out any sidecar start in flight; a box with no delegation keeps the leader-only proof (round 1 ruling, ledger line 127). — cost if wrong: one cgroup.events read per flag-off stop on delegated boxes, and a lease held to its TTL when the proof fails.
• Ruling: an unproven failed seed leaves a durable obligation beside its database; no later wake treats that database as warm until the host's processes are proven gone and the partial database is removed (plan 3). — cost if wrong: a fresh instance waits one lease TTL and a proof before the seed is retried.
• Ruling: on the armed generation path, a start that cannot place the process in its cgroup before it executes (the clone3 fallback is latched, or the cgroup cannot be opened) is refused before exec with ErrLimitNotApplied; the PID-move fallback is never used there (plan 4). — cost if wrong: an armed box on a kernel without CLONEINTOCGROUP refuses every generation start (a box condition; recovery is flag off and an agent restart).
• Ruling: a clean one-shot exit counts as good only when the sweep parked it after its own fence and lease checks; any other stop of an exited one-shot is abandoned. — cost if wrong: a one-shot whose renewal fails just after a clean exit is retried instead of recorded good.
• Ruling: real-box checks are out of scope and go in the roll notes (Task 9).

---

Task 1: The crash-epoch hard limit

Model: opus (durability, security). No recon note: this section is the brief.

Files:

• Modify: internal/rollout/failures.go (failureFile, FailureBook, reopenAdvance, newBookID, OpenFailureBook, saveLocked, RestrictionEpoch, restrictionEpochLocked, RetryHostAt, Record/recordLocked, markLocked, ClearHoldRan, ClearHoldRanBefore, ClearHoldRanOnPointer, ClearOnPointer, Prune, retryHostLocked, reconcileLocked, epochOlderLocked)
• Modify: internal/rollout/status.go (the two hr.HeldEpoch = ... producers), internal/rollout/manager.go (the periodic pass calls SaveIfDirty), internal/host/rolloutdeclined.go (failureHolds.report; startup flush; the periodic declined pass calls SaveIfDirty)
• Modify: internal/release/hostretry.go (doc only)
• Test: internal/rollout/failures_epoch_test.go (new), internal/host/rolloutdeclined_test.go

Interfaces:

• Produces: func (b *FailureBook) PublishedEpoch(acct, app, build, host string) uint64 — the restriction's generation when something of build restricts host and the book's memory equals its last landed save; else 0. Status and the declined report publish this, never RestrictionEpoch.
• Produces: func (b *FailureBook) SaveIfDirty() error — saves when a save is owed, else nil.
• Produces: failureFile.Gens map[string]uint64 \json:"gens,omitempty"\` keyed acct + "/" + app; failureFile.Durable bool \json:"durable_epochs,omitempty"\`.
• Produces: func newBookID() (string, error) through a seam var randRead = rand.Read; OpenFailureBook returns its error.
• Task 3 consumes: status's HeldEpoch == 0 on a held host means "not saved yet".

The rule (put it, with the proof, in the FailureBook doc comment):

1. One generation per app. b.gens[acct/app] takes b.bumpLocked() on every change to any restriction of that app: a record created; a merge that changes the record (Changed); a host retried (retryHostLocked); a record removed (ClearOnPointer, Prune); a marker of the app set, renewed (markLocked, including replacement of one build's marker by another's), or removed (ClearHoldRan, ClearHoldRanBefore, ClearHoldRanOnPointer, RetryHostAt's removal); an intent reconciled into a record; an epoch given to an older record or marker (epochOlderLocked). A call that changes nothing (PreRetry, Unchanged, a duplicate marker of the same execution) moves nothing. The restriction's generation for (build, host) is b.gens[acct/app] when the build's record blocks the host or the host's marker names the build, else 0. Prune drops an app's generation only together with the app's last record and marker; a later restriction of that app takes a fresh value, which is larger than every earlier one.
2. Durable before use. b.dirty is true whenever memory differs from the last landed save. Audit every function that changes recs, held, intents, gens or seq: each saves under the same hold of b.mu, or leaves dirty true. PublishedEpoch answers 0 while dirty. RetryHostAt first retries an owed save; if the book is still dirty it clears nothing.
3. Monotonic across restarts. Every landed save writes the in-memory seq, which only grows, and a reopen starts from the file's seq (maxed with every saved generation, marker and epoch, as today). So the disk's seq only grows, every published generation is at most the disk's seq, and every value a later run hands out is larger. Remove reopenAdvance, its += lines and the comment calling it a bound.
4. Adoption rotates the identity (plan 2). A file without durable_epochs (written by an earlier binary, which may have published epochs it never saved) is opened with a new book identity and durable_epochs set, and owes a save (dirty, so nothing is published until it lands). Every grant issued before names the old identity and clears nothing.
5. Identity from crypto/rand only. newBookID returns an error when randRead fails; OpenFailureBook returns it (an armed box refuses managed starts, as for an unreadable book; a declined box logs and honours nothing new). A memory-only book (empty path) may keep a fallback for tests.
6. The owed save is retried on its own (plan 10). The armed manager's periodic pass and the declined box's periodic pass call SaveIfDirty; a declined box flushes a dirty book at startup whatever reconciliation found.

Proof sketch: a grant G names book X, app A and generation E, which status published, so E was on X's disk under the rule (2, 4). RetryHostAt clears only when X is this book, memory equals disk, and A's generation now equals E. Within a run every change takes a fresh larger value (1), so equality means no change of A's restrictions since publication. Across a crash the reopened book starts at the disk's seq ≥ E and hands out larger values (3). A book written before the rule has another identity (4), and another book has another random identity (5). So G clears only the restriction state it was issued for.

• [ ] Step 1: Write the failing tests in internal/rollout/failures_epoch_test.go. Use a real temp-dir book and make a save fail by making the directory read-only (os.Chmod(dir, 0o500), restored in t.Cleanup; t.Skip when os.Geteuid() == 0). Exact sequences (plan 8):

func TestAFailingDiskPublishesNoEpoch(t *testing.T)
// Record{v2 on h1, HeldDown h1} saved; e1 := PublishedEpoch(a, app, v2, h1) > 0.
// chmod read-only; Record a NEW build v3 failing on h2 (creates a record: a change of the app); save fails.
// PublishedEpoch(.., v2, h1) == 0 while dirty; RetryHostAt(.., v2, h1, id, e1) clears nothing (rec=false, mark=false).
// chmod back; SaveIfDirty() lands; PublishedEpoch(.., v2, h1) > e1; a grant at that value clears h1.

func TestEveryRemovalMovesTheGeneration(t *testing.T) // one subtest per path (plan 1)
// For each: e := PublishedEpoch(...); perform the change; PublishedEpoch != e and RetryHostAt(.., e) clears nothing.
//  - marker of v2 on h1 replaced by a marker of v3 on h1 while v2's record holds h1
//  - v2's record removed by ClearOnPointer while v2's marker on h1 stands
//  - v2's record pruned while v2's marker on h1 stands
//  - marker of v2 on h1 cleared by ClearHoldRanBefore, ClearHoldRanOnPointer, ClearHoldRan
//  - a merge that changes v2's record because another host failed with a newer PromotedAt

func TestANoOpOutcomeDoesNotMoveTheGeneration(t *testing.T)
// PreRetry, Unchanged and a duplicate marker of the same execution leave PublishedEpoch unchanged,
// and a grant at that epoch still clears.

func TestABookFromBeforeTheRuleGetsANewIdentity(t *testing.T) // plan 2
// Write a schema-2 file by hand without durable_epochs, with book "old", seq 10, a record of v2 holding h1
// with epochs {"": 11, h1: 11}. Open: ID() != "old"; PublishedEpoch == 0 until Flush lands;
// RetryHostAt(.., "old", 11) and RetryHostAt(.., "old", 12) clear nothing; after Flush, a grant naming the new
// identity and the published generation clears h1. Reopen: the identity and durable_epochs persist.

func TestAReopenNeverReissuesAPublishedGeneration(t *testing.T)
// Publish e1; make three changes that save; reopen from disk; make three more changes;
// every PublishedEpoch read after the reopen is > the largest read before it.

func TestAGrantForAnotherBookClearsNothing(t *testing.T) // two books opened with equal numeric state
func TestABookThatCannotGetAnIdentityDoesNotOpen(t *testing.T) // randRead fails -> OpenFailureBook errors

In internal/host/rolloutdeclined_test.go: a declined box whose book opened dirty and whose disk recovers publishes an epoch after the next periodic pass with no pointer or grant arriving (plan 10).

• [ ] Step 2: Run them and watch them fail. nice -n 19 go test ./internal/rollout/ -run 'TestAFailingDisk|TestEveryRemoval|TestANoOp|TestABookFromBefore|TestAReopenNever|TestAGrantForAnother|TestABookThatCannot' -count=1 (sandboxed env). Paste the failures.
• [ ] Step 3: Implement rules 1 to 6. Switch both status producers and failureHolds.report to PublishedEpoch. Keep RestrictionEpoch as the in-memory value for logs, or remove it if nothing else needs it. Update HostRetryGrant.Epoch's doc (it is the app's restriction generation as published).
• [ ] Step 4: Run the new tests, go test ./internal/rollout/ -count=1, and unsandboxed TMPDIR=/tmp/k nice -n 19 go test ./internal/host/ -run 'Retry|Declined|Rollout' -count=1. Tests that relied on reopenAdvance or per-host epochs are updated to the new rule, not deleted; list them in the report.
• [ ] Step 5: Commit rollout: a restriction's generation is durable before use and never repeats (Paul's hard limit).
• [ ] Step 6: Run internal/rollout/testdata/rolloutsim/mutate.sh 300 on that commit (baseline plus M1–M12 CAUGHT). Paste the summary and the commit it tested.

Task 2: Simulation: stale grants, crashes and failed saves; mutants; sweep timeout

Model: opus.

Files:

• Modify: internal/rollout/sim_test.go, internal/rollout/testdata/rolloutsim/mutate.sh, and // rolloutsim:Mn marker comments on the lines Task 1 wrote in internal/rollout/failures.go.

Interfaces:

• Consumes: Task 1's PublishedEpoch, RetryHostAt, SaveIfDirty, OpenFailureBook, the generation and durability rules.
• Produces: invariants I9 and I10; mutants M13–M16.

The simulation does not model stale grants today (round 1 ledger, Task 8 minor 8). For the seeds whose book is on disk (the I8 machinery), add:

• Grants. At random steps, read a held host's published (book, generation) as status does and queue a grant, stamped with the model step it was issued at. Deliver queued grants at later random steps through RetryHostAt, some after many events and some after a restart. The queue lives in the model and survives restarts and book recreation.
• Failed saves and crashes. Random read-only windows (as I8 does), and random agent restarts that reopen the book from disk under a new Manager. A restart drops in-flight executions as a real one does.
• Book recreation (for M16): at a random step, remove one seed's book file and open a new book (a box rebuild).

Model-side facts, kept outside the book and never derived from the book's verdicts (plan 7): for every execution the model starts it knows the account, app, build, host and its start step; for every failure or hold-down the model delivers, it records a failure fact (app, build, host, start step). A failure fact is answered by any clear the model issued after the fact's start step: a grant for that (build, host) issued after it, a signed rollback or promote --retry pointer for that build signed after it. These are the clears the spec allows to lift it; a grant or pointer issued before the execution began is not one.

Invariants:

• I9 (a stale grant never clears a later failure): when RetryHostAt for a grant issued at step t clears anything, no unanswered failure fact of the grant's (build, host) has a start step after t. Message rolloutsim: I9 violated at seed %d: ....
• I10 (durable before use): every non-zero value PublishedEpoch returns equals the app's generation in the book's last landed file, which the model re-reads from disk at that moment. Message rolloutsim: I10 violated at seed %d: ....

Mutants (one // rolloutsim:Mn marker on exactly one line each, a perl substitution in MUTANTS; give each a witness, plan 8):

• M13 PublishedEpoch ignores the dirty check → I10: publish while a save has failed, compared with the last landed file.
• M14 one removal path (pick the marker replacement in markLocked) does not move the generation → I9: witness is a grant issued under v2's marker, the marker replaced by a failed hold run of v3 on the same host, then v3's marker cleared by a pointer, then the grant delivered.
• M15 the reopen ignores the saved generations and markers when it sets seq (keep only f.Seq, then subtract enough to force reuse, for example b.seq = f.Seq / 2): witness removes the high-valued components before the reopen so the max cannot repair it, then forces numerical reuse → I9.
• M16 RetryHostAt skips the book-identity comparison → I9 through book recreation. If the simulation cannot reach it, make it a test: mutant on TestAGrantForAnotherBookClearsNothing and say why in the list comment, as M12 does.

mutate.sh: update the header and verdict comments to name I1 to I10 (the check regex already accepts any digits; plan 12). Replace the fixed -timeout 20m with a seeds-scaled value, TIMEOUT_MIN=$(( SEEDS * 3 / 100 + 20 )) (3000 seeds → 110 minutes), used as -timeout ${TIMEOUT_MIN}m, and print it in the header. Round 1 found 20 minutes too short for 3000 seeds.

• [ ] Step 1: Add the model state, I9 and I10. Commit (rollout sim: stale grants, crashes and failed saves; I9, I10). Run ROLLOUT_SIM_SEEDS=300 GOMAXPROCS=2 nice -n 19 go test ./internal/rollout/ -run '^TestSimInvariants$' -count=1; it must pass. Paste the line.
• [ ] Step 2: Add the markers and the four mutants, commit, run mutate.sh 300 M13 M14 M15 M16. Each must read CAUGHT on its stated invariant. Paste the summary and the tested commit.
• [ ] Step 3: Run mutate.sh 300 (all mutants) on the final commit. Paste the summary.

Task 3: Retry grants in their own object, and the console's capability check

Model: opus (security). Recon: recon-D-retry-console.md §2, with this plan's corrections.

Files:

• Modify: internal/release/restartscope.go (ValidRestartScope refuses kind retry and Signed for new writes), new internal/release/hostretrylog.go (HostRetryKey, LoadHostRetries, AppendHostRetry, claimRawAppendHostRetry)
• Modify: internal/host/restart.go, internal/host/restartscope.go (a hostRetryPass over the new object; legacy retry entries in restart-scope.json are still verified and applied with the same checks; neither is seed-skipped at restart)
• Modify: internal/status/snapshot.go (HostRollout.RetryV int \json:"retry_v,omitempty"\`), set to 1 beside HeldBook at both producers in internal/rollout/status.go and in internal/host/rolloutdeclined.go`
• Modify: internal/console/apiinstances.go (rolloutView.retryV, copied where heldEpoch and heldBook are), internal/console/apiretry.go (retryHostBuild reads the restriction and refuses; requestHostRetry performs the append, now through AppendHostRetry; plan 12), internal/console/browserretry.go
• Modify: internal/provision/plan9_b8_census_test.go (a digest row for claimRawAppendHostRetry, and any changed claimRaw* digest)
• Test: internal/release/hostretrylog_test.go (new), internal/host/restartscope_test.go, internal/console/apiretry_test.go, internal/console/browserretry_test.go

Interfaces:

• Consumes: Task 1's published generation (HeldEpoch 0 on a held host while the book is not saved).
• Produces: - func HostRetryKey(acct, app string) string → apps/<acct>/<app>/host-retry.json - func LoadHostRetries(ctx context.Context, s Store, acct, app string) (*ScopedRestartGeneration, bool, error) - func AppendHostRetry(ctx context.Context, s Store, acct, app, host, signed string) (RestartRequest, error) (same JSON shape and CAS loop as the restart-scope log, keep 32, op name release.AppendHostRetry) - status.HostRollout.RetryV; rolloutView.retryV - console errors errRetryUnsupported (report has retry_v == 0 or no held_book) and errRetryUnsaved (retry_v >= 1, held, held_epoch == 0)

Supported transitions (plan 9), each pinned by a test:

• new console, new box: the grant rides host-retry.json, the box verifies and applies it.
• new console, older box (no retry_v): the console refuses with errRetryUnsupported and writes nothing.
• console built from main before this branch, new box: its signed, book-bound grant in restart-scope.json is verified and applied with the same checks (an unsigned one is dropped with a log line, as today). A v0.25.1 console cannot write a retry at all.
• a grant written before an agent restart and not yet applied is applied after it. Write the guarantee as: never applied unchecked; refused with a message by a new console when the box cannot take it; an unsigned or stale request is dropped with an operator log line.

New strings (draft, TODO-COPY):

• zdd-readiness2.retry-unsupported.api: "this instance's box is too old to take a retry. update the box, then try again"
• zdd-readiness2.retry-unsupported.browser: "This instance's box is too old to take a retry. Update the box, then try again."
• zdd-readiness2.retry-unsaved.api: "this instance's box has not saved its failure yet. try again in a minute"
• zdd-readiness2.retry-unsaved.browser: "This instance's box has not saved its failure yet. Try again in a minute." The API door answers 409 with these; the browser door flashes them. Neither writes anything.

Tests (write first, watch fail): release round trip; a restart-scope rewrite by a struct without Signed leaves host-retry.json untouched; ValidRestartScope refuses a new retry kind. Host: the three existing retry tests (TestRestartLegHandsARetryToTheRolloutManager, ...DropsARetryThatIsNotTheRootsGrant, ...RetryGrantsAgainstTheBook) cover the new object; a legacy signed grant in restart-scope.json is applied; a legacy unsigned one clears nothing; a grant written before an agent restart is applied after it (the restart memory is rebuilt). Console: no retry_v → 409 unsupported, nothing written; retry_v: 1, held, held_epoch: 0 → 409 unsaved; retry_v: 1 with epoch and book → the signed entry lands in host-retry.json and nothing in restart-scope.json (update lastScoped).

• [ ] Steps: failing tests; implement; go test ./internal/release/ ./internal/console/ -count=1 sandboxed; unsandboxed TMPDIR=/tmp/k nice -n 19 go test ./internal/host/ -run 'Restart|Retry' -count=1 and go test ./internal/provision/ -run Census -count=1; commit retries: signed host retries ride their own object; the console asks the box's report first.

Task 4: A declined box's hold and Retry in the console

Model: sonnet. Recon: recon-D-retry-console.md §1.

Files:

• Modify: internal/console/instancepage.go (HoldText, PromoteRetryText, RolloutNote; CanRetry on the rollout view and a forwarding instanceDetail.CanRetry, since the template's dot is instanceDetail; plan 12), internal/console/apiretry.go (heldRestriction.declined), internal/console/browserretry.go, internal/console/swapcopy.go (two new strings), internal/console/templates/app_instance.html (Retry form on .CanRetry)
• Create: copy-review/strings.json (force-added with git add -f; copy-review/ is excluded), covering every TODO-COPY string on the branch (Task 3's four and this task's two), rows {id, section, label, text, context, notes}
• Test: internal/console/browserretry_test.go, internal/console/apiretry_test.go

Interfaces:

• Consumes: Task 3's rolloutView.retryV, errRetryUnsupported, errRetryUnsaved.
• Produces: func (r *rolloutView) retryable() bool { return r != nil && r.HeldBy != "" && (r.Unavailable == "" || rolloutwords.UnavailableDeclines(r.Unavailable)) }, used by both retryHostBuild (replacing its inline test) and the page. CanRetry() is retryable() && retryV >= 1 && heldEpoch != 0 && heldBook != "".

Page rules:

• The hold row shows whenever retryable(). Text: Attempt == AttemptDown → the approved zdd.instance.hold-down-text verbatim; declined (Unavailable != "") → new HoldDeclinedText; otherwise today's hold-text. Never format Reason for a declined host (it is empty there).
• The Retry form shows on CanRetry(). The confirmation line for a declined host is new RetryLineDeclined (no canary runs there).

New strings (draft, TODO-COPY):

• zdd-readiness2.hold-declined: "Build %s failed on this box. This box cannot run two builds side by side, so it will not start this instance on that build. Retry it, or ship a newer build."
• zdd-readiness2.retry-line-declined: "This clears the failure hold on {host}. Its box cannot run two builds side by side, so its next start runs {build} the old way, with no canary."

Tests: a declined fixture (Unavailable: "host-claims-on", a failed host and a down host, with HeldBy, HeldEpoch, HeldBook, RetryV: 1) shows the hold row, the Retry form and the declined confirmation; a refusing code (no-deploy-store) with a held host shows neither; TestInstancePageOfAnUnavailableBoxShowsNoHoldAndNoRetry stays green (rename it if it now means "refusing"); a declined host with HeldEpoch: 0 shows the hold row and no form.

• [ ] Steps: failing tests; implement; go test ./internal/console/ -run 'Retry|InstancePage|Copy' -count=1 sandboxed; write copy-review/strings.json; commit console: a declined box's hold and retry on the instance page (copy pending review).

Task 5: Declined boxes: pin the live link once, and recover a stranded sidecar

Model: opus (security). Recon: recon-A-declined.md (read all of it: settings plan 4, 8916558d, already closed most of the production path; this task makes the backend's live-link arms safe on their own and closes the sidecar).

Files:

• Modify: internal/activator/backend_exec.go (declinedLivePin, spawnUnitFrom, startSidecarUnitFrom, seedChild, seedUnit, BuildIdentity; rewrite the residual comments the recon names), internal/activator/activator.go (startSidecar), the SplitBackend forward
• Test: internal/activator/identity_test.go, internal/activator/exec_gate_test.go, internal/activator/declinedgate_identity_test.go, internal/activator/wakeidentity_test.go (fake backend gains BuildIdentity)

Interfaces:

• Produces: func (b *ExecBackend) declinedLivePin(ctx context.Context, rt registry.Route) (release.StagedIdentity, error); type buildIdentifier interface { BuildIdentity(rt registry.Route, build string) (release.StagedIdentity, error) }; func (b *ExecBackend) BuildIdentity(rt registry.Route, build string) (release.StagedIdentity, error).

Design (recon A): when LiveExecGate != nil and a start has no staged file, resolve and hash the link once, ask the gate about that version, and run that identity by path (flat) or through StartExe/StartSidecarExe/SeedExe (nested: root pins the file). In seeds, do it before the seedOwnsDB stat so a refusal never deletes a database. In startSidecar, when the gate refuses the link's version and the generation's build differs, try the instance's own build through BuildIdentity and the gate; run it if both pass; otherwise keep the refusal.

Tests (write first): a nested repoint between read and start runs v1 (startexe ... v1, gate asked [v1], no plain start); the same for the sidecar; a link outside the store asks root for nothing; flat and nested seeds run the gated file and a refused seed creates no database; a sidecar under a link naming held v2 runs v1 (gate asked [v2 v1]), and with v1 refused too it stays down while the instance serves.

• [ ] Steps: failing tests; implement; unsandboxed TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -run 'Identity|Gate|Sidecar|Seed' -count=1, then the whole package once; commit activator: a declined box pins the live link once, and a refused sidecar runs the instance's build.

Task 6: One-shot good build after a fence or lost lease; the duplicate-wake window

Model: opus (concurrency). Recon: recon-B-good-f6.md §1 and §2.

Files:

• Modify: internal/activator/workerswap.go (ReplaceWorker's forgiveness of errSwapNotAwake), internal/activator/activator.go (actEntry.parkedInc, markCleanParkLocked, the sweep's clean-exit park, WakeWorker, begin, new awaitWake)
• Test: internal/activator/workerswap_test.go

Interfaces:

• Produces: func (a *Service) markCleanParkLocked(e *actEntry); func (a *Service) awaitWake(ctx context.Context, e *actEntry, ready <-chan struct{}, deadline *time.Timer, arrival time.Time, extended *bool) error; test seam var wakeBoundExtended func() (nil in production).

Design (recon B §1, §2): forgive errSwapNotAwake only when e.parkedInc == inc1, no fence stands and the service is not shutting down; every other stop of an exited one-shot is abandoned. WakeWorker and begin wait again on the same ready after extending the bound, never re-reading the entries map. The optional servedOneShot fence check in recon B is not needed and is not added.

Tests (write first, watch fail): exit 0 then a renewal failure and a sweep → SwapAbandoned; exit 0 then FenceAndStop → SwapAbandoned; positive control, exit 0 then a sweep → SwapDone with CleanExit; a wake that fails inside the extension window → WakeWorker returns the wake's error and the backend counts one lease acquire (today two); the same for a request through begin.

• [ ] Steps: failing tests; implement; unsandboxed TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -run 'Worker|OneShot|Wake' -count=1, then -race on the new tests, then the package once; commit activator: a one-shot is good only after a clean park, and a waiting wake never starts a second.

Task 7: Fail closed when a generation's limits cannot be applied; kernel positive controls

Model: opus. Recon: recon-C-cgroup.md, with case G replaced below.

Files:

• Modify: internal/cgroup/gen.go (ErrLimitNotApplied, ensureParent branches A–D, delegatesMemory, writeSubtreeControl seam), internal/cgroup/cgroup.go (startIn(..., required bool, ...), applyLimit returns its first failure; with required, never the PID-move fallback), internal/activator/swap.go (verdictOf), internal/activator/logsink.go (memberSafeWakeReason → ErrFenced; fix the stale comment the recon names)
• Test: internal/cgroup/gen_test.go, internal/cgroup/cgroup_test.go, internal/activator/swap_overlap_test.go, internal/activator/swap_executed_test.go, internal/activator/logsink_test.go, internal/activator/genparent_kernel_linux_test.go, internal/activator/claimspawn_kernel_linux_test.go

Interfaces:

• Produces: var ErrLimitNotApplied = errors.New("cgroup: a generation's limits could not be applied"); func (m *Manager) startIn(host, dir string, limit Limit, required bool, build func() *exec.Cmd, before func() error) (*exec.Cmd, error); func (m *Manager) applyLimit(host, dir string, limit Limit) error.

Design: recon C §2 for A–F. StartGated passes required=false (the flag-off path is byte-for-byte the same); StartChildGated passes true. EBUSY on subtree_control maps to ErrLegacyParentPopulated. Case G (plan 4): with required, the start must place the process in its cgroup before it executes (clone3 CLONE_INTO_CGROUP). When the fallback is latched, or the cgroup directory cannot be opened for the clone, return ErrLimitNotApplied before build() runs; never start and move by PID. Fixtures that write populated 1 with no subtree_control get one (cpu memory io pids).

Tests: one unit test per branch A–F plus G without a kernel (the recon names the fixture trick for each; for G, store the fallback latch true), each asserting nothing started and no generation child left; ErrLimitNotApplied rows in TestVerdictOf, TestClassifyStep, TestSwapResultReportsWhetherTheAttemptsBuildExecuted (SwapAbandoned, not executed) and the logsink table. Kernel controls (skip-gated by CARLOS_CLAIM_SPAWN_KERNEL=1; they compile here and run on the dev box later): the sibling control and the recovery control in TestGenerationStartRefusedBesideADirectParentMember; a live g1 kept running through killSidecarRespawn's loop.

• [ ] Steps: failing tests; implement; unsandboxed TMPDIR=/tmp/k nice -n 19 go test ./internal/cgroup/ -count=1, then go test ./internal/activator/ -run 'Verdict|Classify|SwapResult|Logsink|MemberSafe' -count=1; go vet ./internal/activator/ (compiles the linux kernel test files); GOOS=darwin go build ./...; commit cgroup: a generation whose limits cannot be applied does not start.

Task 8: Prove a seed's processes gone; the flag-off normal-stop half of F6

Model: opus (concurrency, durability). Recon: recon-B-good-f6.md §3 and §4, with the corrections below (plan 3, 5, 6).

Files:

• Modify: internal/activator/backend_exec.go (seedDescendantsGone, seedChild's failure and success branches, EmptyHost, the unproven-seed obligation file, Restore's warm-cache check), internal/activator/activator.go (doWake's seed decision; actEntry.launchLayout set at install; stop(); the pending-stop path through new reapUncontainedStop; the in-flight sidecar start), internal/activator/opset.go (stopHostProven, hostEmptier), the SplitBackend forward
• Test: internal/activator/activator_test.go, internal/activator/containment_test.go, internal/activator/wakeplan_test.go (the plannedHostKill double pattern)

Interfaces:

• Produces: func (b *ExecBackend) seedDescendantsGone(ctx context.Context, rt registry.Route, child string) error; func (b *ExecBackend) EmptyHost(ctx context.Context, rt registry.Route) error; type hostEmptier interface{ EmptyHost(context.Context, registry.Route) error }; func (a *Service) stopHostProven(e *actEntry) bool; func (a *Service) reapUncontainedStop(e *actEntry); const seedUnprovenSuffix = ".seed-unproven".

Design:

1. Failed seed. A failed flat seed's cgroup (flag on: its role child; flag off: the host cgroup) is killed and proven empty before RemoveChild and removeSeedDB. ErrNoDelegation keeps the leader-only proof with a log line. When the proof fails, write <db>.seed-unproven durably (safefile) beside the database, keep the database, and return an error wrapping ErrSeedUnproven (the lease is kept to its TTL, as today).
2. The obligation survives (plan 3). Before any wake treats an existing database as warm (Restore, and doWake's fresh decision), it checks for <db>.seed-unproven. If present: prove the host's processes gone (seedDescendantsGone with the same child rule); on success remove the database, its -wal/-shm files and the obligation, and seed afresh; on failure refuse the wake with ErrSeedUnproven. The obligation is a file, so it survives an agent restart.
3. Successful seed, flag on (plan 6). On every successful completion, including a plain nil Wait, the seed's role child is killed and proven empty before RemoveChild (a helper that redirected its output and outlived the seed is killed). Flag off, a successful seed's helper stays in the host cgroup and is caught by the normal-stop proof.
4. Normal stop, flag off. A flat or foreign exec row's terminal stop proves the host cgroup empty before finishStop releases the lease: EmptyHost reads cgroup.events and kills only when populated, so a clean sleep leaves no kill mark. ErrNoDelegation keeps the leader-only proof. Nothing runs under a.mu.
5. In-flight sidecar start (plan 5). A terminal stop with a sidecar start still in flight past stop()'s wait keeps the stop pending, with the lease, until that start returns; its child, if any, is stopped, and only then does the host proof run.

Tests (write first, watch fail):

• a failing seed on delegatedBackend whose cleanup cannot be proven returns ErrSeedUnproven, keeps the database and writes the obligation; TestExecBackendSeedFailureRemovesHalfBuiltDB stays green;
• the next wake after that: with the proof still failing it is refused and the database is not opened as warm; with the proof passing the database and obligation are removed and the seed runs again (assert the seed command ran twice);
• flag on, a seed that forks a helper which redirects its output, then exits 0: the helper is killed before the seed returns (fake cgroup double records KillAndEmpty of role-seed);
• flag-off idle stop with a failing EmptyHost double keeps the lease (lastRep().isClosed() == false); a passing double releases it after the proof (order asserted with setOnClose); ErrNoDelegation releases at once; the pending-stop path the same;
• a sidecar start delayed beyond stop()'s wait keeps the lease until it returns, and its child is stopped before the release.

• [ ] Steps: failing tests; implement; unsandboxed TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -count=1; GOOS=windows go build ./...; commit activator: seeds and flag-off stops prove their processes gone before the lease goes.

Task 9: Spec rev-18 amendment and roll notes

Model: sonnet. Recon: recon-D-retry-console.md §3.

Files:

• Create: docs/superpowers/specs/2026-10-12-zero-downtime-deploys-rev18-amendment.md
• Modify: docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md ("Still owed (code)", "Roll notes from zdd-flag-readiness", "Real-box checks")

The amendment states, for Paul's sign-off: the status line for rev 18; replacements for spec line 3 (the rev-14 sentence), lines 704–705 (§8.4 carlos instances retry) and lines 836–837 (§11 "A retry is a re-signed channel pointer"); a new §11 paragraph with the grant as built (HostRetryGrant fields; the root-signed envelope and the scoped-key refusal; the 24 h TTL as a backstop and the 1-minute skew; book identity plus the app's restriction generation; Task 1's rule and proof sketch, including the identity rotation on adoption; unsigned or stale grants dropped with a log line; 503 with no signing key; declined boxes honour grants with no canary; grants ride host-retry.json, legacy signed grants in restart-scope.json are checked the same way; the console checks retry_v), and Paul's Q4 ruling (2026-10-11) and hard-limit ruling (2026-10-12) by date. carlos promote --retry stays the app-wide re-signed pointer (lines 761–762 and the line-998 table row are unchanged; say so).

Roll notes: mark each closed item with its commit; replace the "reopen epoch bound" bullet with the hard rule; replace "Retries: boxes before consoles" with the supported transitions from Task 3; add the armed-box refusal when CLONEINTOCGROUP is unavailable (Task 7) and the seed obligation file (Task 8) as roll notes; list the real-box checks still owed: the kernel tests from Task 7 (sibling, recovery, live-g1 respawn) and round 1's, the seed and normal-stop kernel control from Task 8 (a forked helper killed before the lease goes), the real-unit control and sidecar checks, dev rollout and retry across a mixed-version roll, and the flagship probe gates E–G.

• [ ] Steps: write both; check every commit hash cited with git log --oneline; commit docs: rev-18 amendment for signed host retries (pending Paul); roll notes for readiness round 2.

---

Execution order and gates

1, 2, 3, 4, 5, 6, 7, 8, 9. Tasks 3 and 4 share console files (sequential). Tasks 5, 6 and 8 share activator.go and backend_exec.go (sequential). After Task 9 the controller runs the one 3000-seed sweep (mutate.sh 3000, nice 19, GOMAXPROCS=2) on the committed tip and the full gate, then Astra's whole-branch review.