amadan
plan: zero-downtime flag readiness, round 2 (draft for Astra's review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+291
−0
docs/superpowers/plans/2026-10-12-zero-downtime-flag-readiness-2.md+291 −0
diff --git a/docs/superpowers/plans/2026-10-12-zero-downtime-flag-readiness-2.md b/docs/superpowers/plans/2026-10-12-zero-downtime-flag-readiness-2.md| new file mode 100644 |
| index 000000000..63cce2c52 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-12-zero-downtime-flag-readiness-2.md |
| @@ -0,0 +1,291 @@ |
| +# Zero-downtime flag readiness, round 2: Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Close every remaining CODE blocker to turning `CARLOS_SWAP` on, as listed by Astra's readiness round 3 and the "Still owed (code)" section of the roll notes, with Paul's 2026-10-12 ruling on the crash-epoch guard. |
| + |
| +**Architecture:** Nine tasks, each small and self-contained. Task 1 replaces the bounded crash-epoch advance with a provable rule: an epoch is published or accepted only once it is on disk, and a restriction's generation strictly increases on every change. Task 2 proves it in the rollout simulation with new invariants and mutants. Tasks 3 and 4 move signed host retries to their own object and show a declined box's hold in the console. Tasks 5 to 8 close the declined-box, one-shot, cgroup and process-proof residuals. Task 9 writes the spec rev-18 amendment and the roll notes. |
| + |
| +**Tech Stack:** Go 1.x, the existing `internal/rollout`, `internal/activator`, `internal/cgroup`, `internal/release`, `internal/host`, `internal/console` packages; `internal/rollout/testdata/rolloutsim/mutate.sh`. |
| + |
| +**Spec:** /home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md (rev 17, binding). Task 9 drafts rev 18 for Paul's sign-off. |
| + |
| +**Base:** origin/main 650e6152. Branch `zdd-flag-readiness-2`. |
| + |
| +**Inputs:** `docs/superpowers/plans/CODEX-FINDINGS-ZDD-READINESS{,-R2,-R3}.md`; roll notes `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` §"Still owed (code), after zdd-flag-readiness"; round 1's ledger `~/.cache/zdd-impl/readiness-ledger.md`; amadan discussion #272. Recon notes for every task: `~/.cache/zdd-impl/readiness2/recon-{A-declined,B-good-f6,C-cgroup,D-retry-console}.md` (each task names its file; read it, it carries the line anchors). |
| + |
| +## Global Constraints |
| + |
| +- `CARLOS_SWAP` stays unset. Nothing in this plan runs on a real box. |
| +- Flag-off behaviour does not change, except where a task says it is an authorized flag-off change (Task 8's failed-seed proof and normal-stop proof; Task 3's move of retries, which only unreleased builds ever wrote). |
| +- No new disk I/O under `activator.Service.mu`. Keep `rollout.Manager`'s documented lock order (`internal/rollout/manager.go`, the lock-order comment). |
| +- Wire changes are additive: an older reader ignores a new field. |
| +- No liveness trap: no instance may be left down forever with no supported recovery. |
| +- New member-facing strings carry `// TODO-COPY(<id>)` on the literal's line, ids prefixed `zdd-readiness2.`. Draft text only: lower case after a full stop where the house style does, no em dashes (a full stop and a new sentence instead). Operator log lines are not copy. |
| +- Cross-compile: linux-only syscalls need `//go:build linux` and a `!linux` stub; CI builds darwin and windows (`GOOS=darwin go build ./...`, `GOOS=windows go build ./...`). |
| +- Tests run serially and niced. Sandboxed: `export GOCACHE=$TMPDIR/gocache GOFLAGS=-mod=mod GOPROXY=off` in every command. Packages that bind unix sockets (activator, edge, host, console's socket tests) or touch cgroupfs run unsandboxed with `TMPDIR=/tmp/k GOFLAGS=-buildvcs=false` (mkdir it first; default GOCACHE there). |
| +- Commits: identity is already set in the worktree (`claude` / `claude~paul@keymail.dev`); check `git config user.email` before the first commit. Plain `git add <files>` and `git commit -m`; never `git stash`, `git checkout -- <file>`, `git worktree prune`, `git reset --hard`. |
| +- Integrity: every test result reported is pasted from a real run. |
| +- Simulation budget: `mutate.sh 300` inside a task; the one 3000-seed sweep per branch revision is the controller's, at `nice -n 19`, GOMAXPROCS=2 (PAR=2). |
| + |
| +## Review Focus |
| + |
| +1. A box whose failure-book save keeps failing (full or read-only disk): its holds stay visible, no epoch is offered for signing, and nothing a console signed earlier can clear a failure recorded later, across any number of agent restarts. (Task 1, test "a failing disk publishes no epoch and a stale grant stays stale across reopen".) |
| +2. A retry pressed in the console during a mixed-version roll (old console, new box; new console, old box): it is either applied with full checks or refused with a message, never silently dropped and never applied unchecked. (Task 3, tests for each pairing.) |
| +3. A declined box (claims on) whose instance runs v1 while its live link names a held v2: the instance, its sidecar and any seed run only a gated build, and nothing stays down with no recovery. (Task 5.) |
| +4. A one-shot worker that exits 0 just as the box loses its lease or is fenced: its build is not recorded as good. (Task 6.) |
| +5. A flag-off instance that forks a helper and is put to sleep: the lease is not released while the helper still runs. (Task 8.) |
| + |
| +--- |
| + |
| +## Rulings recorded by this plan |
| + |
| +- Ruling: the crash-epoch guard is a hard rule, not the 2^20 advance. An epoch is published (status) or accepted (`RetryHostAt`) only while the book's memory equals its last landed save; a restriction's generation takes a fresh sequence value on every change of any of its parts, clears included, so it never returns to an earlier value; the book identity comes from crypto/rand and a file-backed book refuses to open if it cannot get one. `reopenAdvance` is removed so the mutants can show the rule is what carries the guarantee. — Paul's ruling 2026-10-12 — cost if wrong: a box with a failing disk shows its holds with no Retry until a save lands. |
| +- Ruling: signed host retries move out of `restart-scope.json` into their own object `apps/<acct>/<app>/host-retry.json`; the console writes a retry only to a host whose report carries `retry_v >= 1` and a non-zero epoch. — No released binary writes or applies a retry (v0.25.1 has none), so moving them now makes every mixed-version pairing safe by construction instead of by roll order — cost if wrong: a dev box on an intermediate build cannot be retried until it is updated. |
| +- Ruling: the rev-18 spec amendment is a new file on this branch (`docs/superpowers/specs/2026-10-12-zero-downtime-deploys-rev18-amendment.md`), not an edit to the design branch, until Paul signs it off. — the design branch belongs to the spec's own worktree — cost if wrong: one copy step after sign-off. |
| +- Ruling: a stranded legacy sidecar recovers by running the instance's own build when the link's build is refused (recon A §3a). The recovery probe's dropped staged file (§3b) is not changed: it would change flag-off behaviour on claims-on boxes and §3a already closes the stranding. — cost if wrong: on a claims-on box the sidecar runs the build the instance runs, as intended, through one extra hash. |
| +- Ruling: the normal-stop half of F6 is an authorized flag-off change (Paul's scope item 5). A stop proves the host cgroup empty before the lease goes, killing only when something is left; a box with no delegation keeps the leader-only proof (round 1 ruling at ledger line 127). — cost if wrong: one `cgroup.events` read per flag-off stop on delegated boxes, and a lease held to its TTL when the proof fails. |
| +- Ruling: an unproven failed seed keeps its lease and its half-seeded database (`ErrSeedUnproven`), as the nested path already does. — cost if wrong: a fresh instance waits one lease TTL before the next wake can retry the seed. |
| +- Ruling: on the armed generation path, a `cgroup.procs` move that fails after the process started (recon C case G) kills and reaps the process and reports a start that may have run (`markStartUncertain`), so an attempt treats it as executed. — cost if wrong: a hold attempt is recorded as a failure that may have touched the database. |
| +- Ruling: a clean one-shot exit counts as good only when the sweep parked it after its own fence and lease checks; any other stop of an exited one-shot is abandoned. — cost if wrong: a one-shot whose renewal fails just after a clean exit is retried instead of recorded good. |
| +- Ruling: real-box checks are out of scope and go in the roll notes (Task 9). |
| + |
| +--- |
| + |
| +### Task 1: The crash-epoch hard limit |
| + |
| +Model: opus (durability, security). Recon: none; this plan's design is the brief. |
| + |
| +**Files:** |
| +- Modify: `internal/rollout/failures.go` (FailureBook: `reopenAdvance`, `newBookID`, `OpenFailureBook`, `saveLocked`, `RestrictionEpoch`, `restrictionEpochLocked`, `RetryHostAt`, `ClearHoldRan`, `ClearHoldRanBefore`, `ClearHoldRanOnPointer`, the marker setters) |
| +- Modify: `internal/rollout/status.go` (the two `HeldEpoch` producers), `internal/host/rolloutdeclined.go` (`failureHolds.report`) |
| +- Modify: `internal/release/hostretry.go` (doc only: Epoch is the restriction's persisted generation) |
| +- Test: `internal/rollout/failures_epoch_test.go` (new) |
| + |
| +**Interfaces:** |
| +- Produces: `func (b *FailureBook) PublishedEpoch(acct, app, build, host string) uint64` — the restriction's generation if the book's memory equals its last landed save, else 0. Status and the declined report call this instead of `RestrictionEpoch`. Task 3's console treats `HeldEpoch == 0` on a host that is held as "not saved yet". |
| +- Produces: `RetryHostAt` refuses (clears nothing, returns `now` as today) while the book is dirty. |
| +- Produces: `func newBookID() (string, error)`; `OpenFailureBook` returns its error. A memory-only book (empty path) may keep a clock fallback. |
| + |
| +**The rule, stated once (put it in the FailureBook doc comment):** |
| +1. Durable before use. `b.dirty` is true whenever memory differs from the last landed save. Audit every function that changes `recs`, `held`, `intents` or `seq`: each either saves under the same hold of `b.mu` or leaves `dirty` true. Epochs are published and accepted only when `!b.dirty`. |
| +2. Monotonic across restarts. The disk's `seq` only grows, because every landed save writes the in-memory `seq`, which only grows, and a reopen starts from the disk's value (max with the file's markers and epochs, as today). With rule 1, every published epoch is at most the disk's `seq`, and every value a later run hands out is above it. Remove `reopenAdvance` and the `+= reopenAdvance` lines, and delete the comment that calls it a bound. |
| +3. Exact generation. A restriction of `host` by `build` is the record's record-wide and host epochs and the host's marker when it names `build`. Every change to any of these takes a fresh `bumpLocked()` value that becomes the restriction's generation, including the marker clears (`ClearHoldRan`, `ClearHoldRanBefore`, `ClearHoldRanOnPointer`, and `RetryHostAt`'s own marker removal) when a record of that build still stands for the host. The simplest form: when a marker of build B on host H is removed and B's record exists, set `rec.Epochs[H] = b.bumpLocked()`. The generation never returns to a value it had before. |
| +4. Book identity. `newBookID` returns an error when `crypto/rand` fails; `OpenFailureBook` returns it (an armed box then refuses managed starts, as for an unreadable book). Never fall back to the clock for a file-backed book. |
| + |
| +Proof sketch to put in the doc comment: a grant G names book X and epoch E, which status published, so E was on X's disk (rule 1). `RetryHostAt` clears only when the current generation equals E, X is this book, and memory equals disk. Within one run, generations are fresh values (rule 3), so equality means no change since publication. Across a crash, the reopened book starts at the disk's `seq` ≥ E and every later change takes a larger value (rule 2), so no later change can have generation E. Another book has another 128-bit random identity (rule 4). Therefore G clears only the restriction state it was issued for. |
| + |
| +- [ ] **Step 1: Write the failing tests** in `internal/rollout/failures_epoch_test.go`. Use a real temp-dir book (`OpenFailureBook(t.TempDir())`) and make a save fail by making the directory read-only (`os.Chmod(dir, 0o500)`, restored in `t.Cleanup`; skip when `os.Geteuid() == 0`, as I8 does). |
| + |
| +```go |
| +func TestAFailingDiskPublishesNoEpoch(t *testing.T) |
| +// record a failure of v2 on h1 (saved), PublishedEpoch > 0; |
| +// chmod the dir read-only; record a second failure merging h2 (save fails, dirty); |
| +// PublishedEpoch(.., "v2", "h1") == 0 while dirty; RetryHostAt with the old epoch clears nothing; |
| +// chmod back; any mutating call lands the save; PublishedEpoch > old epoch. |
| + |
| +func TestAStaleGrantStaysStaleAcrossReopen(t *testing.T) |
| +// e1 := PublishedEpoch for (v2,h1); dir read-only; a later failure of v2 on h1 changes the |
| +// restriction in memory only; reopen the book from disk (OpenFailureBook) without the change; |
| +// then record the later failure again through the reopened book (and enough further events |
| +// to pass e1 by 3 values); RetryHostAt(.., book.ID(), e1) must clear nothing. |
| +// Under the old code (reopenAdvance removed, rule 1 absent) this test must fail: show it. |
| + |
| +func TestAMarkerClearMovesTheGeneration(t *testing.T) |
| +// record v2 failure on h1 (epoch e1); set a marker for v2 on h1; clear it (ClearHoldRanBefore with a |
| +// later snapshot); PublishedEpoch != e1 and RetryHostAt(.., e1) clears nothing; a fresh read clears. |
| + |
| +func TestAGrantForAnotherBookClearsNothing(t *testing.T) // two books, equal numeric epochs |
| + |
| +func TestABookThatCannotGetAnIdentityDoesNotOpen(t *testing.T) |
| +// swap the package's rand reader seam (add `var randRead = rand.Read` in failures.go) to fail; |
| +// OpenFailureBook on an empty dir returns an error. |
| +``` |
| + |
| +- [ ] **Step 2: Run them and watch them fail** (`nice -n 19 go test ./internal/rollout/ -run 'TestAFailingDisk|TestAStaleGrant|TestAMarkerClear|TestAGrantForAnother|TestABookThatCannot' -count=1`, sandboxed env). Paste the failures. |
| +- [ ] **Step 3: Implement rules 1 to 4.** Switch `internal/rollout/status.go` (both `hr.HeldEpoch = ...` lines) and `internal/host/rolloutdeclined.go` (`HeldEpoch:` in `failureHolds.report`) to `PublishedEpoch`. Keep `RestrictionEpoch` for logs and tests if used. Update `HostRetryGrant.Epoch`'s doc. |
| +- [ ] **Step 4: Run the new tests, then the package** (`go test ./internal/rollout/ -count=1`) and `go test ./internal/host/ -run 'Retry|Declined|Rollout' -count=1` unsandboxed with `TMPDIR=/tmp/k`. Existing tests that relied on `reopenAdvance` (grep `reopenAdvance` and `1 << 20` in `_test.go`) are updated to the new rule, not deleted; say which in the report. |
| +- [ ] **Step 5: Run `internal/rollout/testdata/rolloutsim/mutate.sh 300`** (baseline plus M1–M12 still CAUGHT). Paste the summary lines. |
| +- [ ] **Step 6: Commit** `rollout: crash epochs are published only once on disk, and a restriction's generation never repeats`. |
| + |
| +### Task 2: Simulation invariants and mutants for the hard limit; sweep timeout |
| + |
| +Model: opus. |
| + |
| +**Files:** |
| +- Modify: `internal/rollout/sim_test.go`, `internal/rollout/testdata/rolloutsim/mutate.sh`, and the `// rolloutsim:Mn` marker lines in `internal/rollout/failures.go` (and `status.go` if M13's line lives there). |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1's `PublishedEpoch`, `RetryHostAt`, the dirty rule, the generation rule, `OpenFailureBook`. |
| +- Produces: invariants I9 and I10; mutants M13–M16 in `mutate.sh`. |
| + |
| +The simulation does not model stale grants today (round 1 ledger, Task 8 minor 8). Add, for the seeds whose book is on disk (the I8 machinery): |
| +- **Grants.** At random steps, read a held host's published (book, epoch) the way status does and queue a grant; deliver queued grants at later random steps (some after many events, some after a reopen) through `RetryHostAt`. |
| +- **Failed saves and crashes.** Random read-only windows (as I8 does) and random agent restarts that reopen the book from disk with a new Manager. Grants queued before a restart survive it (the console holds them). |
| +- **Ghost generations.** The model keeps, outside the book, a counter per (account, app, build, host) bumped on every *failure event* for that host and build: a record created or merged with that host failing, a marker set or renewed for that build on that host, an intent reconciled on that host. A queued grant remembers the ghost value at issue. |
| + |
| +Invariants: |
| +- **I9 (no stale grant):** a grant whose `RetryHostAt` clears anything had the same ghost value at issue as at delivery. Message `rolloutsim: I9 violated at seed %d: ...`. |
| +- **I10 (durable before use):** every non-zero epoch read from `PublishedEpoch` equals the restriction's generation in the book's last landed file (parse the file the model reads back). Message `rolloutsim: I10 violated at seed %d: ...`. |
| + |
| +Mutants (add a `// rolloutsim:Mn` marker on exactly one line each; perl substitution in `MUTANTS`): |
| +- **M13** `PublishedEpoch` ignores the dirty check → I10 (and I9). |
| +- **M14** a marker clear does not move the generation (the `rec.Epochs[H] = b.bumpLocked()` line of rule 3 in one clear path) → I9. |
| +- **M15** the reopen starts below the disk's sequence (for example `b.seq = f.Seq` becomes `b.seq = f.Seq / 2`) → I9. |
| +- **M16** `RetryHostAt` skips the book-identity comparison → I9, with the model recreating one host's book (a box rebuild: the book file removed and opened anew) at a random step. If the simulation cannot reach M16, make it a `test:` mutant on `TestAGrantForAnotherBookClearsNothing` and say why in the list comment, as M12 does. |
| + |
| +mutate.sh: extend the verdict text and the I-regex from "I1 to I8" to "I1 to I10". Raise the per-run timeout: replace the fixed `-timeout 20m` with a value scaled by seeds, `TIMEOUT_MIN=$(( SEEDS * 3 / 100 + 20 ))` (3000 seeds → 110 minutes), and print it in the header. Round 1 found 20 minutes too short for 3000 seeds. |
| + |
| +- [ ] **Step 1:** Add the model state, I9 and I10, and run `ROLLOUT_SIM_SEEDS=300 GOMAXPROCS=2 nice -n 19 go test ./internal/rollout/ -run '^TestSimInvariants$' -count=1` on the Task 1 code. It must pass. Paste the line. |
| +- [ ] **Step 2:** For each of M13–M16, apply the substitution by hand in a scratch copy only (never the checkout), or run `mutate.sh 300 M13 M14 M15 M16` after adding them. Every one must read CAUGHT on its stated invariant. Paste the summary. |
| +- [ ] **Step 3:** Run `mutate.sh 300` (all mutants). Paste the summary. |
| +- [ ] **Step 4: Commit** `rollout sim: stale grants, crashes and failed saves; I9, I10, M13–M16; sweep timeout scales with seeds`. |
| + |
| +### Task 3: Retry grants in their own object, and the console's capability check |
| + |
| +Model: opus (security). Recon: `~/.cache/zdd-impl/readiness2/recon-D-retry-console.md` §2. |
| + |
| +**Files:** |
| +- Modify: `internal/release/restartscope.go` (`ValidRestartScope` refuses kind retry and `Signed`), new `internal/release/hostretrylog.go` (`HostRetryKey`, `LoadHostRetries`, `AppendHostRetry`, `claimRawAppendHostRetry`) |
| +- Modify: `internal/host/restart.go` and `internal/host/restartscope.go` (a `hostRetryPass` over the new object, seeded like `restartScopePass`; a retry kind found in `restart-scope.json` clears nothing and logs once) |
| +- Modify: `internal/status/snapshot.go` (`HostRollout.RetryV int \`json:"retry_v,omitempty"\``), set to 1 beside `HeldBook` in `internal/rollout/status.go` (both producers) and `internal/host/rolloutdeclined.go` |
| +- Modify: `internal/console/apiinstances.go` (`rolloutView.retryV`), `internal/console/apiretry.go` (`retryHostBuild` writes with `AppendHostRetry`; refusals), `internal/console/browserretry.go` |
| +- Modify: `internal/provision/plan9_b8_census_test.go` (a digest row for `claimRawAppendHostRetry`, and any changed `claimRaw*` digest) |
| +- Test: `internal/release/hostretrylog_test.go` (new), `internal/host/restartscope_test.go`, `internal/console/apiretry_test.go`, `internal/console/browserretry_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1's `PublishedEpoch` (status's `HeldEpoch` is 0 while the box's book is not saved). |
| +- Produces: |
| + - `func HostRetryKey(acct, app string) string` → `apps/<acct>/<app>/host-retry.json` |
| + - `func LoadHostRetries(ctx context.Context, s Store, acct, app string) (*ScopedRestartGeneration, bool, error)` |
| + - `func AppendHostRetry(ctx context.Context, s Store, acct, app, host, signed string) (RestartRequest, error)` (same JSON shape and CAS loop as the restart-scope log, keep 32, op name `release.AppendHostRetry`) |
| + - `status.HostRollout.RetryV`; `rolloutView.retryV` |
| + - console errors `errRetryUnsupported` (report has `retry_v == 0` or no `held_book`) and `errRetryUnsaved` (`retry_v >= 1`, held, `held_epoch == 0`) |
| +- Task 4 consumes: `func (r *rolloutView) retryable() bool` is Task 4's; this task only adds `retryV` and the two errors. |
| + |
| +New strings (draft, TODO-COPY): |
| +- `zdd-readiness2.retry-unsupported.api`: "this instance's box is too old to take a retry. update the box, then try again" |
| +- `zdd-readiness2.retry-unsupported.browser`: "This instance's box is too old to take a retry. Update the box, then try again." |
| +- `zdd-readiness2.retry-unsaved.api`: "this instance's box has not saved its failure yet. try again in a minute" |
| +- `zdd-readiness2.retry-unsaved.browser`: "This instance's box has not saved its failure yet. Try again in a minute." |
| +The API door answers 409 with these; the browser door flashes them. Neither writes anything. |
| + |
| +Tests (write first, watch fail): |
| +- release: append/load round trip; a restart-scope rewrite by a struct without `Signed` leaves `host-retry.json` untouched; `ValidRestartScope` refuses `{Kind: retry}` and any `Signed`. |
| +- host: move `TestRestartLegHandsARetryToTheRolloutManager`, `...DropsARetryThatIsNotTheRootsGrant`, `...RetryGrantsAgainstTheBook` to the new object; add "a retry kind in restart-scope.json clears nothing"; a grant seen once is not replayed after an agent restart (the seed marks). |
| +- console: report without `retry_v` → 409 unsupported and nothing written; `retry_v: 1`, held, `held_epoch: 0` → 409 unsaved; `retry_v: 1` with epoch and book → the signed entry lands in `host-retry.json` and nothing in `restart-scope.json` (update `lastScoped`). |
| + |
| +- [ ] Steps: failing tests; implement; run `go test ./internal/release/ ./internal/console/ -count=1` sandboxed, `go test ./internal/host/ -run 'Restart|Retry' -count=1` and `go test ./internal/provision/ -run Census -count=1` unsandboxed (`TMPDIR=/tmp/k`); commit `retries: signed host retries ride their own object; the console asks the box's report first`. |
| + |
| +### Task 4: A declined box's hold and Retry in the console |
| + |
| +Model: sonnet. Recon: `recon-D-retry-console.md` §1. |
| + |
| +**Files:** |
| +- Modify: `internal/console/instancepage.go` (`HoldText`, `PromoteRetryText`, `RolloutNote`, a `CanRetry`), `internal/console/apiretry.go` (`heldRestriction.declined`), `internal/console/browserretry.go`, `internal/console/swapcopy.go` (two new strings), `internal/console/templates/app_instance.html` (Retry form on `CanRetry`) |
| +- Create: `copy-review/strings.json` (force-added: `git add -f`), covering every TODO-COPY string on the branch (Task 3's four and this task's two), rows `{id, section, label, text, context, notes}` |
| +- Test: `internal/console/browserretry_test.go`, `internal/console/apiretry_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 3's `rolloutView.retryV`, `errRetryUnsupported`, `errRetryUnsaved`. |
| +- Produces: `func (r *rolloutView) retryable() bool { return r != nil && r.HeldBy != "" && (r.Unavailable == "" || rolloutwords.UnavailableDeclines(r.Unavailable)) }`, used by both the door (`retryHostBuild`, replacing its inline test) and the page, so they cannot disagree. `CanRetry()` is `retryable() && retryV >= 1 && heldEpoch != 0 && heldBook != ""`. |
| + |
| +Page rules: |
| +- The hold row shows whenever `retryable()`. Text: `Attempt == AttemptDown` → the approved `zdd.instance.hold-down-text` verbatim; declined (`Unavailable != ""`) → new `HoldDeclinedText`; otherwise today's `hold-text`. Never format `Reason` for a declined host (it is empty there). |
| +- The Retry form shows on `CanRetry()`. The confirmation line for a declined host is new `RetryLineDeclined` (no canary runs there). |
| + |
| +New strings (draft, TODO-COPY): |
| +- `zdd-readiness2.hold-declined`: "Build %s failed on this box. This box cannot run two builds side by side, so it will not start this instance on that build. Retry it, or ship a newer build." |
| +- `zdd-readiness2.retry-line-declined`: "This clears the failure hold on {host}. Its box cannot run two builds side by side, so its next start runs {build} the old way, with no canary." |
| + |
| +Tests: a declined fixture (`Unavailable: "host-claims-on"`, a failed host and a down host, with `HeldBy`, `HeldEpoch`, `HeldBook`, `RetryV: 1`) shows the hold row, the Retry form and the declined confirmation; a refusing code (`no-deploy-store`) with a held host shows neither; `TestInstancePageOfAnUnavailableBoxShowsNoHoldAndNoRetry` stays green (adjust its fixture name if it now means "refusing"); a declined host with `HeldEpoch: 0` shows the hold row and no form. |
| + |
| +- [ ] Steps: failing tests; implement; `go test ./internal/console/ -run 'Retry|InstancePage|Copy' -count=1` sandboxed; write `copy-review/strings.json`; commit `console: a declined box's hold and retry on the instance page (copy pending review)`. |
| + |
| +### Task 5: Declined boxes: pin the live link once, and recover a stranded sidecar |
| + |
| +Model: opus (security). Recon: `recon-A-declined.md` (read all of it: settings plan 4 already closed most of the production path; this task makes the backend's live-link arms safe on their own and closes the sidecar). |
| + |
| +**Files:** |
| +- Modify: `internal/activator/backend_exec.go` (`declinedLivePin`, `spawnUnitFrom`, `startSidecarUnitFrom`, `seedChild`, `seedUnit`, `BuildIdentity`; rewrite the residual comments the recon names), `internal/activator/activator.go` (`startSidecar`), the `SplitBackend` forward |
| +- Test: `internal/activator/identity_test.go`, `internal/activator/exec_gate_test.go`, `internal/activator/declinedgate_identity_test.go`, `internal/activator/wakeidentity_test.go` (fake backend gains `BuildIdentity`) |
| + |
| +**Interfaces:** |
| +- Produces: `func (b *ExecBackend) declinedLivePin(ctx context.Context, rt registry.Route) (release.StagedIdentity, error)`; `type buildIdentifier interface { BuildIdentity(rt registry.Route, build string) (release.StagedIdentity, error) }` with `func (b *ExecBackend) BuildIdentity(rt registry.Route, build string) (release.StagedIdentity, error)`. |
| + |
| +Design (from the recon): when `LiveExecGate != nil` and a start has no staged file, resolve and hash the link once, ask the gate about that version, and run that identity by path (flat) or through `StartExe`/`StartSidecarExe`/`SeedExe` (nested, root pins the file). In seeds, do it before the `seedOwnsDB` stat so a refusal never deletes a database. In `startSidecar`, when the gate refuses the link's version and the generation's build differs, try the instance's own build through `BuildIdentity` and the gate; run it if both pass; otherwise keep the refusal. |
| + |
| +Tests (from the recon; write first): nested repoint between read and start runs v1 (`startexe ... v1`, gate asked `[v1]`, no plain `start`); the same for the sidecar; a link outside the store asks root for nothing; flat and nested seeds run the gated file and a refused seed creates no database; a sidecar under a link naming held v2 runs v1 (gate asked `[v2 v1]`), and with v1 refused too it stays down while the instance serves. |
| + |
| +- [ ] Steps: failing tests; implement; unsandboxed `TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -run 'Identity|Gate|Sidecar|Seed' -count=1`, then the whole package once; commit `activator: a declined box pins the live link once, and a refused sidecar runs the instance's build`. |
| + |
| +### Task 6: One-shot good build after a fence or lost lease; the duplicate-wake window |
| + |
| +Model: opus (concurrency). Recon: `recon-B-good-f6.md` §1 and §2. |
| + |
| +**Files:** |
| +- Modify: `internal/activator/workerswap.go` (`ReplaceWorker`'s forgiveness of `errSwapNotAwake`), `internal/activator/activator.go` (`actEntry.parkedInc`, `markCleanParkLocked`, the sweep's clean-exit park, `WakeWorker`, `begin`, new `awaitWake`), optionally `internal/rollout/manager.go` (`servedOneShot` refuses when the activator reports the host fenced; `Fenced` on `rollout.Activator`) |
| +- Test: `internal/activator/workerswap_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `func (a *Service) markCleanParkLocked(e *actEntry)`; `func (a *Service) awaitWake(ctx context.Context, e *actEntry, ready <-chan struct{}, deadline *time.Timer, arrival time.Time, extended *bool) error`; test seam `var wakeBoundExtended func()` (nil in production). |
| + |
| +Design: as recon B §1 and §2. Forgive `errSwapNotAwake` only when `e.parkedInc == inc1`, no fence stands and the service is not shutting down; every other stop is abandoned. `WakeWorker` and `begin` wait again on the same `ready` after extending the bound, never re-reading the entries map. |
| + |
| +Tests (write first, watch fail): exit 0 then a renewal failure and a sweep → `SwapAbandoned`; exit 0 then `FenceAndStop` → `SwapAbandoned`; positive control, exit 0 then a sweep → `SwapDone` with `CleanExit`; a wake that fails inside the extension window → `WakeWorker` returns the wake's error and the backend counts one lease acquire (today two); the same for a request through `begin`. |
| + |
| +- [ ] Steps: failing tests; implement; `TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -run 'Worker|OneShot|Wake' -count=1` unsandboxed, then `go test -race` on the new tests, then the package once; commit `activator: a one-shot is good only after a clean park, and a waiting wake never starts a second`. |
| + |
| +### Task 7: Fail closed when a generation's limits cannot be applied; kernel positive controls |
| + |
| +Model: opus. Recon: `recon-C-cgroup.md`. |
| + |
| +**Files:** |
| +- Modify: `internal/cgroup/gen.go` (`ErrLimitNotApplied`, `ensureParent` branches A–D, `delegatesMemory`, `writeSubtreeControl` seam), `internal/cgroup/cgroup.go` (`startIn(..., required bool, ...)`, `applyLimit` returns its first failure, case G), `internal/activator/swap.go` (`verdictOf`), `internal/activator/logsink.go` (`memberSafeWakeReason` → `ErrFenced`; fix the stale comment the recon names), `internal/activator/containment.go` (case G → `markStartUncertain`) |
| +- Test: `internal/cgroup/gen_test.go`, `internal/cgroup/cgroup_test.go`, `internal/activator/swap_overlap_test.go`, `internal/activator/swap_executed_test.go`, `internal/activator/logsink_test.go`, `internal/activator/genparent_kernel_linux_test.go`, `internal/activator/claimspawn_kernel_linux_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `var ErrLimitNotApplied = errors.New("cgroup: a generation's limits could not be applied")`; `func (m *Manager) startIn(host, dir string, limit Limit, required bool, build func() *exec.Cmd, before func() error) (*exec.Cmd, error)`; `func (m *Manager) applyLimit(host, dir string, limit Limit) error`. |
| + |
| +Design: as recon C §2. `StartGated` passes `required=false` (flag-off path byte-for-byte the same); `StartChildGated` passes `true`. EBUSY on `subtree_control` maps to `ErrLegacyParentPopulated`. Fixtures that write `populated 1` with no `subtree_control` get one (`cpu memory io pids`). |
| + |
| +Tests: one unit test per branch A–G without a kernel (the recon names the fixture trick for each), asserting nothing started (except G, which asserts the process was killed and the error is uncertain) and no generation child left; `ErrLimitNotApplied` rows in `TestVerdictOf`, `TestClassifyStep`, `TestSwapResultReportsWhetherTheAttemptsBuildExecuted` and the logsink table. Kernel controls (skip-gated by `CARLOS_CLAIM_SPAWN_KERNEL=1`, compile here, run on the dev box later): the sibling control and the recovery control in `TestGenerationStartRefusedBesideADirectParentMember`; a live g1 kept running through `killSidecarRespawn`'s loop. |
| + |
| +- [ ] Steps: failing tests; implement; `go test ./internal/cgroup/ -count=1` unsandboxed with `TMPDIR=/tmp/k`, then `go test ./internal/activator/ -run 'Verdict|Classify|SwapResult|Logsink|MemberSafe' -count=1`; `go vet -tags linux ./internal/activator/` compiles the kernel files; `GOOS=darwin go build ./...`; commit `cgroup: a generation whose limits cannot be applied does not start`. |
| + |
| +### Task 8: Prove a failed seed's processes gone; the flag-off normal-stop half of F6 |
| + |
| +Model: opus (concurrency, durability). Recon: `recon-B-good-f6.md` §3 and §4. |
| + |
| +**Files:** |
| +- Modify: `internal/activator/backend_exec.go` (`seedDescendantsGone`, `seedChild`'s failure branch and its lingering-helper success branch, `EmptyHost`), `internal/activator/opset.go` (`stopHostProven`, `hostEmptier`), `internal/activator/activator.go` (`actEntry.launchLayout` set at install; `stop()`; the pending-stop path through new `reapUncontainedStop`), the `SplitBackend` forward |
| +- Test: `internal/activator/activator_test.go`, `internal/activator/containment_test.go`, `internal/activator/wakeplan_test.go` (the `plannedHostKill` double pattern) |
| + |
| +**Interfaces:** |
| +- Produces: `func (b *ExecBackend) seedDescendantsGone(ctx context.Context, rt registry.Route, child string) error`; `func (b *ExecBackend) EmptyHost(ctx context.Context, rt registry.Route) error`; `type hostEmptier interface{ EmptyHost(context.Context, registry.Route) error }`; `func (a *Service) stopHostProven(e *actEntry) bool`; `func (a *Service) reapUncontainedStop(e *actEntry)`. |
| + |
| +Design: as recon B §3 and §4. A failed flat seed's cgroup (flag on: its role child; flag off: the host cgroup) is killed and proven empty before `RemoveChild` and `removeSeedDB`; an unproven seed returns an error wrapping `ErrSeedUnproven`, which keeps the lease and the database. Flag off, a flat or foreign exec row's terminal stop proves the host cgroup empty before `finishStop` releases the lease: `EmptyHost` reads `cgroup.events` and kills only when populated, so a clean sleep leaves no kill mark. `ErrNoDelegation` keeps the leader-only proof with a log line. Nothing runs under `a.mu`. |
| + |
| +Tests (write first, watch fail): a failing seed on `delegatedBackend` whose cleanup cannot be proven returns `ErrSeedUnproven` and keeps the database; `TestExecBackendSeedFailureRemovesHalfBuiltDB` stays green; flag-off idle stop with a failing `EmptyHost` double keeps the lease (`lastRep().isClosed() == false`), a passing double releases it after the proof (order asserted with `setOnClose`), `ErrNoDelegation` releases at once; the pending-stop path the same. |
| + |
| +- [ ] Steps: failing tests; implement; `TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -count=1` unsandboxed; `GOOS=windows go build ./...`; commit `activator: a failed seed and a flag-off stop prove their processes gone before the lease goes`. |
| + |
| +### Task 9: Spec rev-18 amendment and roll notes |
| + |
| +Model: sonnet. Recon: `recon-D-retry-console.md` §3. |
| + |
| +**Files:** |
| +- Create: `docs/superpowers/specs/2026-10-12-zero-downtime-deploys-rev18-amendment.md` |
| +- Modify: `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` ("Still owed (code)", "Roll notes from zdd-flag-readiness", "Real-box checks") |
| + |
| +The amendment states, for Paul's sign-off: the status line for rev 18; replacements for spec lines 3 (rev-14 sentence), 704–705 (§8.4 `carlos instances retry`) and 836–837 (§11 "A retry is a re-signed channel pointer"); a new §11 paragraph with the grant as built (`HostRetryGrant` fields, root-signed envelope and the scoped-key refusal, the 24 h TTL as a backstop and the 1-minute skew, book identity plus epoch, `PublishedEpoch`'s durable-before-use rule and the generation rule from Task 1 with the proof sketch, unsigned or stale grants dropped with a log line, 503 with no signing key, declined boxes honour grants with no canary, grants ride `host-retry.json`, the console checks `retry_v`), and Paul's Q4 ruling (2026-10-11) and hard-limit ruling (2026-10-12) by date. `carlos promote --retry` stays the app-wide re-signed pointer (lines 761–762 and the line-998 table row are unchanged; say so). |
| + |
| +Roll notes: mark each closed item with its commit; replace the "reopen epoch bound" bullet with the hard rule; shrink "Retries: boxes before consoles" to the by-construction facts; list the real-box checks still owed: the kernel tests from Task 7 (sibling, recovery, live-g1 respawn) and round 1's, the failed-seed and normal-stop kernel control from Task 8 (a forked helper killed before the lease goes), the real-unit control and sidecar checks, dev rollout and retry across a mixed-version roll, and the flagship probe gates E–G. |
| + |
| +- [ ] Steps: write both; check every commit hash cited with `git log --oneline`; commit `docs: rev-18 amendment for signed host retries (pending Paul); roll notes for readiness round 2`. |
| + |
| +--- |
| + |
| +## Execution order and gates |
| + |
| +1, 2, 3, 4, 5, 6, 7, 8, 9. Tasks 3 and 4 share console files (sequential). Tasks 5, 6 and 8 share `activator.go` and `backend_exec.go` (sequential). After Task 9 the controller runs the one 3000-seed sweep (`mutate.sh 3000`, nice 19, GOMAXPROCS=2) and the full gate, then Astra's whole-branch review. |