| index 63cce2c52..c72648797 100644 |
| --- a/docs/superpowers/plans/2026-10-12-zero-downtime-flag-readiness-2.md |
| +++ b/docs/superpowers/plans/2026-10-12-zero-downtime-flag-readiness-2.md |
| @@ -4,49 +4,50 @@ |
| |
| **Goal:** Close every remaining CODE blocker to turning `CARLOS_SWAP` on, as listed by Astra's readiness round 3 and the "Still owed (code)" section of the roll notes, with Paul's 2026-10-12 ruling on the crash-epoch guard. |
| |
| -**Architecture:** Nine tasks, each small and self-contained. Task 1 replaces the bounded crash-epoch advance with a provable rule: an epoch is published or accepted only once it is on disk, and a restriction's generation strictly increases on every change. Task 2 proves it in the rollout simulation with new invariants and mutants. Tasks 3 and 4 move signed host retries to their own object and show a declined box's hold in the console. Tasks 5 to 8 close the declined-box, one-shot, cgroup and process-proof residuals. Task 9 writes the spec rev-18 amendment and the roll notes. |
| +**Architecture:** Nine tasks, each small and self-contained. Task 1 replaces the bounded crash-epoch advance with a provable rule: a restriction's generation is a per-app counter that every change of the app's restrictions moves to a fresh value, published or accepted only once it is on disk, under a book identity rotated when the rule is adopted. Task 2 proves it in the rollout simulation with a model-side invariant and mutants. Tasks 3 and 4 move signed host retries to their own object and show a declined box's hold in the console. Tasks 5 to 8 close the declined-box, one-shot, cgroup and process-proof residuals. Task 9 writes the spec rev-18 amendment and the roll notes. |
| |
| -**Tech Stack:** Go 1.x, the existing `internal/rollout`, `internal/activator`, `internal/cgroup`, `internal/release`, `internal/host`, `internal/console` packages; `internal/rollout/testdata/rolloutsim/mutate.sh`. |
| +**Tech Stack:** Go; `internal/rollout`, `internal/activator`, `internal/cgroup`, `internal/release`, `internal/host`, `internal/console`; `internal/rollout/testdata/rolloutsim/mutate.sh`. |
| |
| **Spec:** /home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md (rev 17, binding). Task 9 drafts rev 18 for Paul's sign-off. |
| |
| -**Base:** origin/main 650e6152. Branch `zdd-flag-readiness-2`. |
| +**Base:** origin/main 650e6152. Branch `zdd-flag-readiness-2`. Plan revision 2 answers Astra's plan review (`docs/superpowers/plans/CODEX-FINDINGS-ZDD-READINESS2-PLAN.md`); "(plan N)" marks the finding a change answers. |
| |
| -**Inputs:** `docs/superpowers/plans/CODEX-FINDINGS-ZDD-READINESS{,-R2,-R3}.md`; roll notes `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` §"Still owed (code), after zdd-flag-readiness"; round 1's ledger `~/.cache/zdd-impl/readiness-ledger.md`; amadan discussion #272. Recon notes for every task: `~/.cache/zdd-impl/readiness2/recon-{A-declined,B-good-f6,C-cgroup,D-retry-console}.md` (each task names its file; read it, it carries the line anchors). |
| +**Inputs:** `docs/superpowers/plans/CODEX-FINDINGS-ZDD-READINESS{,-R2,-R3}.md`; roll notes `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` §"Still owed (code), after zdd-flag-readiness"; round 1's ledger `~/.cache/zdd-impl/readiness-ledger.md`; amadan discussion #272. Recon notes: `~/.cache/zdd-impl/readiness2/recon-{A-declined,B-good-f6,C-cgroup,D-retry-console}.md` (each task names its file; read it, it carries the line anchors; where this plan and a recon note differ, this plan wins). |
| |
| ## Global Constraints |
| |
| - `CARLOS_SWAP` stays unset. Nothing in this plan runs on a real box. |
| -- Flag-off behaviour does not change, except where a task says it is an authorized flag-off change (Task 8's failed-seed proof and normal-stop proof; Task 3's move of retries, which only unreleased builds ever wrote). |
| +- Flag-off behaviour does not change, except where a task says it is an authorized flag-off change (Task 8's failed-seed and normal-stop proofs; Task 3's move of retries, which no released binary ever wrote). |
| - No new disk I/O under `activator.Service.mu`. Keep `rollout.Manager`'s documented lock order (`internal/rollout/manager.go`, the lock-order comment). |
| - Wire changes are additive: an older reader ignores a new field. |
| - No liveness trap: no instance may be left down forever with no supported recovery. |
| -- New member-facing strings carry `// TODO-COPY(<id>)` on the literal's line, ids prefixed `zdd-readiness2.`. Draft text only: lower case after a full stop where the house style does, no em dashes (a full stop and a new sentence instead). Operator log lines are not copy. |
| +- New member-facing strings carry `// TODO-COPY(<id>)` on the literal's line, ids prefixed `zdd-readiness2.`. Draft text only: no em dashes (a full stop and a new sentence instead). Operator log lines are not copy. |
| - Cross-compile: linux-only syscalls need `//go:build linux` and a `!linux` stub; CI builds darwin and windows (`GOOS=darwin go build ./...`, `GOOS=windows go build ./...`). |
| -- Tests run serially and niced. Sandboxed: `export GOCACHE=$TMPDIR/gocache GOFLAGS=-mod=mod GOPROXY=off` in every command. Packages that bind unix sockets (activator, edge, host, console's socket tests) or touch cgroupfs run unsandboxed with `TMPDIR=/tmp/k GOFLAGS=-buildvcs=false` (mkdir it first; default GOCACHE there). |
| -- Commits: identity is already set in the worktree (`claude` / `claude~paul@keymail.dev`); check `git config user.email` before the first commit. Plain `git add <files>` and `git commit -m`; never `git stash`, `git checkout -- <file>`, `git worktree prune`, `git reset --hard`. |
| +- Tests run serially and niced (`nice -n 19`). Session house rules for the environment: sandboxed Go uses `export GOCACHE=$TMPDIR/gocache GOFLAGS=-mod=mod GOPROXY=off` in every command (never delete that cache, it is shared); packages that bind unix sockets (activator, edge, host, some console tests) or touch cgroupfs run unsandboxed with `TMPDIR=/tmp/k GOFLAGS=-buildvcs=false` and the default Go cache (`mkdir -p /tmp/k` first; remove only what you created there afterwards). |
| +- Commits: the worktree's identity is set (`claude` / `claude~paul@keymail.dev`); check `git config user.email` before the first commit. Plain `git add <files>` and `git commit -m`; never `git stash`, `git checkout -- <file>`, `git worktree prune`, `git reset --hard`. |
| - Integrity: every test result reported is pasted from a real run. |
| -- Simulation budget: `mutate.sh 300` inside a task; the one 3000-seed sweep per branch revision is the controller's, at `nice -n 19`, GOMAXPROCS=2 (PAR=2). |
| +- Simulation: `mutate.sh` tests `git archive HEAD`, so **commit before every mutate.sh run** and name the commit it tested (plan 11). `mutate.sh 300` inside a task; the one 3000-seed sweep per branch revision is the controller's, at `nice -n 19`, GOMAXPROCS=2. |
| |
| ## Review Focus |
| |
| -1. A box whose failure-book save keeps failing (full or read-only disk): its holds stay visible, no epoch is offered for signing, and nothing a console signed earlier can clear a failure recorded later, across any number of agent restarts. (Task 1, test "a failing disk publishes no epoch and a stale grant stays stale across reopen".) |
| -2. A retry pressed in the console during a mixed-version roll (old console, new box; new console, old box): it is either applied with full checks or refused with a message, never silently dropped and never applied unchecked. (Task 3, tests for each pairing.) |
| -3. A declined box (claims on) whose instance runs v1 while its live link names a held v2: the instance, its sidecar and any seed run only a gated build, and nothing stays down with no recovery. (Task 5.) |
| +1. A box whose failure-book save keeps failing (full or read-only disk): its holds stay visible, no epoch is offered for signing, the save is retried on its own until the disk recovers, and nothing a console signed earlier can clear a failure recorded later, across any number of agent restarts. (Task 1.) |
| +2. A box upgraded from a round-1 binary that had published an unsaved epoch: grants issued before the upgrade clear nothing after it. (Task 1, migration test.) |
| +3. A retry pressed during a mixed-version roll: applied with full checks, or refused with a message by a new console, or (old console, new box) honoured only if its signed grant passes the same checks. Never applied unchecked. A retry requested just before an agent restart is applied after it. (Task 3.) |
| 4. A one-shot worker that exits 0 just as the box loses its lease or is fenced: its build is not recorded as good. (Task 6.) |
| -5. A flag-off instance that forks a helper and is put to sleep: the lease is not released while the helper still runs. (Task 8.) |
| +5. A flag-off instance that forks a helper, or whose sidecar start is still in flight, and is put to sleep: the lease is not released while anything of it still runs; a failed seed's partial database is never served by the next wake. (Task 8.) |
| |
| --- |
| |
| ## Rulings recorded by this plan |
| |
| -- Ruling: the crash-epoch guard is a hard rule, not the 2^20 advance. An epoch is published (status) or accepted (`RetryHostAt`) only while the book's memory equals its last landed save; a restriction's generation takes a fresh sequence value on every change of any of its parts, clears included, so it never returns to an earlier value; the book identity comes from crypto/rand and a file-backed book refuses to open if it cannot get one. `reopenAdvance` is removed so the mutants can show the rule is what carries the guarantee. — Paul's ruling 2026-10-12 — cost if wrong: a box with a failing disk shows its holds with no Retry until a save lands. |
| -- Ruling: signed host retries move out of `restart-scope.json` into their own object `apps/<acct>/<app>/host-retry.json`; the console writes a retry only to a host whose report carries `retry_v >= 1` and a non-zero epoch. — No released binary writes or applies a retry (v0.25.1 has none), so moving them now makes every mixed-version pairing safe by construction instead of by roll order — cost if wrong: a dev box on an intermediate build cannot be retried until it is updated. |
| -- Ruling: the rev-18 spec amendment is a new file on this branch (`docs/superpowers/specs/2026-10-12-zero-downtime-deploys-rev18-amendment.md`), not an edit to the design branch, until Paul signs it off. — the design branch belongs to the spec's own worktree — cost if wrong: one copy step after sign-off. |
| -- Ruling: a stranded legacy sidecar recovers by running the instance's own build when the link's build is refused (recon A §3a). The recovery probe's dropped staged file (§3b) is not changed: it would change flag-off behaviour on claims-on boxes and §3a already closes the stranding. — cost if wrong: on a claims-on box the sidecar runs the build the instance runs, as intended, through one extra hash. |
| -- Ruling: the normal-stop half of F6 is an authorized flag-off change (Paul's scope item 5). A stop proves the host cgroup empty before the lease goes, killing only when something is left; a box with no delegation keeps the leader-only proof (round 1 ruling at ledger line 127). — cost if wrong: one `cgroup.events` read per flag-off stop on delegated boxes, and a lease held to its TTL when the proof fails. |
| -- Ruling: an unproven failed seed keeps its lease and its half-seeded database (`ErrSeedUnproven`), as the nested path already does. — cost if wrong: a fresh instance waits one lease TTL before the next wake can retry the seed. |
| -- Ruling: on the armed generation path, a `cgroup.procs` move that fails after the process started (recon C case G) kills and reaps the process and reports a start that may have run (`markStartUncertain`), so an attempt treats it as executed. — cost if wrong: a hold attempt is recorded as a failure that may have touched the database. |
| +- Ruling: the crash-epoch guard is a hard rule, not the 2^20 advance. A restriction's generation is a per-(account, app) counter in the book, moved to a fresh sequence value by every change to any restriction of that app (records created, merged, retried, cleared by pointer or pruned; markers set, renewed, replaced or cleared; intents reconciled); it is published (status) or accepted (`RetryHostAt`) only while the book's memory equals its last landed save; the book identity is rotated, durably, the first time a book is opened under the rule; identities come from crypto/rand, never the clock. `reopenAdvance` is removed. — Paul's ruling 2026-10-12; per-app rather than per-host because a record restricts every host of its app, so per-host values cannot be kept exact on removals (plan 1) — cost if wrong: a retry pressed while another instance of the same app changes state is refused once and must be pressed again; a box with a failing disk shows holds with no Retry until a save lands. |
| +- Ruling: signed host retries move to their own object `apps/<acct>/<app>/host-retry.json`, which new consoles write. New boxes also honour a signed grant found in `restart-scope.json` (a console built from main between 5da1cc9b and this branch writes there), with the same checks. The console writes a retry only to a host whose report carries `retry_v >= 1` and a non-zero epoch. — no released binary writes or applies a retry (v0.25.1 has none) — cost if wrong: a box on an intermediate dev build cannot be retried from a new console until it is updated. |
| +- Ruling: retry grants are not seed-skipped at agent restart; every grant still inside its TTL is applied on first sight, because a spent or stale grant is a no-op under Task 1's generation rule (plan 9). — cost if wrong: one signature check per pending grant per changed log read. |
| +- Ruling: the rev-18 spec amendment is a new file on this branch (`docs/superpowers/specs/2026-10-12-zero-downtime-deploys-rev18-amendment.md`), not an edit to the design branch, until Paul signs it off. — cost if wrong: one copy step after sign-off. |
| +- Ruling: a stranded legacy sidecar recovers by running the instance's own build when the link's build is refused (recon A §3a). The recovery probe's dropped staged file (§3b) is not changed: that would change flag-off behaviour on claims-on boxes, and §3a closes the stranding. — cost if wrong: one extra hash per refused sidecar restart. |
| +- Ruling: the failed-seed proof and the normal-stop half of F6 are authorized flag-off changes (Paul's scope item 5). A stop proves the host cgroup empty before the lease goes, killing only when something is left, and waits out any sidecar start in flight; a box with no delegation keeps the leader-only proof (round 1 ruling, ledger line 127). — cost if wrong: one `cgroup.events` read per flag-off stop on delegated boxes, and a lease held to its TTL when the proof fails. |
| +- Ruling: an unproven failed seed leaves a durable obligation beside its database; no later wake treats that database as warm until the host's processes are proven gone and the partial database is removed (plan 3). — cost if wrong: a fresh instance waits one lease TTL and a proof before the seed is retried. |
| +- Ruling: on the armed generation path, a start that cannot place the process in its cgroup before it executes (the clone3 fallback is latched, or the cgroup cannot be opened) is refused before exec with `ErrLimitNotApplied`; the PID-move fallback is never used there (plan 4). — cost if wrong: an armed box on a kernel without CLONE_INTO_CGROUP refuses every generation start (a box condition; recovery is flag off and an agent restart). |
| - Ruling: a clean one-shot exit counts as good only when the sweep parked it after its own fence and lease checks; any other stop of an exited one-shot is abandoned. — cost if wrong: a one-shot whose renewal fails just after a clean exit is retried instead of recorded good. |
| - Ruling: real-box checks are out of scope and go in the roll notes (Task 9). |
| |
| @@ -54,114 +55,135 @@ |
| |
| ### Task 1: The crash-epoch hard limit |
| |
| -Model: opus (durability, security). Recon: none; this plan's design is the brief. |
| +Model: opus (durability, security). No recon note: this section is the brief. |
| |
| **Files:** |
| -- Modify: `internal/rollout/failures.go` (FailureBook: `reopenAdvance`, `newBookID`, `OpenFailureBook`, `saveLocked`, `RestrictionEpoch`, `restrictionEpochLocked`, `RetryHostAt`, `ClearHoldRan`, `ClearHoldRanBefore`, `ClearHoldRanOnPointer`, the marker setters) |
| -- Modify: `internal/rollout/status.go` (the two `HeldEpoch` producers), `internal/host/rolloutdeclined.go` (`failureHolds.report`) |
| -- Modify: `internal/release/hostretry.go` (doc only: Epoch is the restriction's persisted generation) |
| -- Test: `internal/rollout/failures_epoch_test.go` (new) |
| +- Modify: `internal/rollout/failures.go` (`failureFile`, `FailureBook`, `reopenAdvance`, `newBookID`, `OpenFailureBook`, `saveLocked`, `RestrictionEpoch`, `restrictionEpochLocked`, `RetryHostAt`, `Record`/`recordLocked`, `markLocked`, `ClearHoldRan`, `ClearHoldRanBefore`, `ClearHoldRanOnPointer`, `ClearOnPointer`, `Prune`, `retryHostLocked`, `reconcileLocked`, `epochOlderLocked`) |
| +- Modify: `internal/rollout/status.go` (the two `hr.HeldEpoch = ...` producers), `internal/rollout/manager.go` (the periodic pass calls `SaveIfDirty`), `internal/host/rolloutdeclined.go` (`failureHolds.report`; startup flush; the periodic declined pass calls `SaveIfDirty`) |
| +- Modify: `internal/release/hostretry.go` (doc only) |
| +- Test: `internal/rollout/failures_epoch_test.go` (new), `internal/host/rolloutdeclined_test.go` |
| |
| **Interfaces:** |
| -- Produces: `func (b *FailureBook) PublishedEpoch(acct, app, build, host string) uint64` — the restriction's generation if the book's memory equals its last landed save, else 0. Status and the declined report call this instead of `RestrictionEpoch`. Task 3's console treats `HeldEpoch == 0` on a host that is held as "not saved yet". |
| -- Produces: `RetryHostAt` refuses (clears nothing, returns `now` as today) while the book is dirty. |
| -- Produces: `func newBookID() (string, error)`; `OpenFailureBook` returns its error. A memory-only book (empty path) may keep a clock fallback. |
| +- Produces: `func (b *FailureBook) PublishedEpoch(acct, app, build, host string) uint64` — the restriction's generation when something of `build` restricts `host` and the book's memory equals its last landed save; else 0. Status and the declined report publish this, never `RestrictionEpoch`. |
| +- Produces: `func (b *FailureBook) SaveIfDirty() error` — saves when a save is owed, else nil. |
| +- Produces: `failureFile.Gens map[string]uint64 \`json:"gens,omitempty"\`` keyed `acct + "/" + app`; `failureFile.Durable bool \`json:"durable_epochs,omitempty"\``. |
| +- Produces: `func newBookID() (string, error)` through a seam `var randRead = rand.Read`; `OpenFailureBook` returns its error. |
| +- Task 3 consumes: status's `HeldEpoch == 0` on a held host means "not saved yet". |
| |
| -**The rule, stated once (put it in the FailureBook doc comment):** |
| -1. Durable before use. `b.dirty` is true whenever memory differs from the last landed save. Audit every function that changes `recs`, `held`, `intents` or `seq`: each either saves under the same hold of `b.mu` or leaves `dirty` true. Epochs are published and accepted only when `!b.dirty`. |
| -2. Monotonic across restarts. The disk's `seq` only grows, because every landed save writes the in-memory `seq`, which only grows, and a reopen starts from the disk's value (max with the file's markers and epochs, as today). With rule 1, every published epoch is at most the disk's `seq`, and every value a later run hands out is above it. Remove `reopenAdvance` and the `+= reopenAdvance` lines, and delete the comment that calls it a bound. |
| -3. Exact generation. A restriction of `host` by `build` is the record's record-wide and host epochs and the host's marker when it names `build`. Every change to any of these takes a fresh `bumpLocked()` value that becomes the restriction's generation, including the marker clears (`ClearHoldRan`, `ClearHoldRanBefore`, `ClearHoldRanOnPointer`, and `RetryHostAt`'s own marker removal) when a record of that build still stands for the host. The simplest form: when a marker of build B on host H is removed and B's record exists, set `rec.Epochs[H] = b.bumpLocked()`. The generation never returns to a value it had before. |
| -4. Book identity. `newBookID` returns an error when `crypto/rand` fails; `OpenFailureBook` returns it (an armed box then refuses managed starts, as for an unreadable book). Never fall back to the clock for a file-backed book. |
| +**The rule (put it, with the proof, in the FailureBook doc comment):** |
| +1. **One generation per app.** `b.gens[acct/app]` takes `b.bumpLocked()` on every change to any restriction of that app: a record created; a merge that changes the record (`Changed`); a host retried (`retryHostLocked`); a record removed (`ClearOnPointer`, `Prune`); a marker of the app set, renewed (`markLocked`, including replacement of one build's marker by another's), or removed (`ClearHoldRan`, `ClearHoldRanBefore`, `ClearHoldRanOnPointer`, `RetryHostAt`'s removal); an intent reconciled into a record; an epoch given to an older record or marker (`epochOlderLocked`). A call that changes nothing (`PreRetry`, `Unchanged`, a duplicate marker of the same execution) moves nothing. The restriction's generation for (build, host) is `b.gens[acct/app]` when the build's record blocks the host or the host's marker names the build, else 0. Prune drops an app's generation only together with the app's last record and marker; a later restriction of that app takes a fresh value, which is larger than every earlier one. |
| +2. **Durable before use.** `b.dirty` is true whenever memory differs from the last landed save. Audit every function that changes `recs`, `held`, `intents`, `gens` or `seq`: each saves under the same hold of `b.mu`, or leaves `dirty` true. `PublishedEpoch` answers 0 while dirty. `RetryHostAt` first retries an owed save; if the book is still dirty it clears nothing. |
| +3. **Monotonic across restarts.** Every landed save writes the in-memory `seq`, which only grows, and a reopen starts from the file's `seq` (maxed with every saved generation, marker and epoch, as today). So the disk's `seq` only grows, every published generation is at most the disk's `seq`, and every value a later run hands out is larger. Remove `reopenAdvance`, its `+=` lines and the comment calling it a bound. |
| +4. **Adoption rotates the identity (plan 2).** A file without `durable_epochs` (written by an earlier binary, which may have published epochs it never saved) is opened with a new book identity and `durable_epochs` set, and owes a save (dirty, so nothing is published until it lands). Every grant issued before names the old identity and clears nothing. |
| +5. **Identity from crypto/rand only.** `newBookID` returns an error when `randRead` fails; `OpenFailureBook` returns it (an armed box refuses managed starts, as for an unreadable book; a declined box logs and honours nothing new). A memory-only book (empty path) may keep a fallback for tests. |
| +6. **The owed save is retried on its own (plan 10).** The armed manager's periodic pass and the declined box's periodic pass call `SaveIfDirty`; a declined box flushes a dirty book at startup whatever reconciliation found. |
| |
| -Proof sketch to put in the doc comment: a grant G names book X and epoch E, which status published, so E was on X's disk (rule 1). `RetryHostAt` clears only when the current generation equals E, X is this book, and memory equals disk. Within one run, generations are fresh values (rule 3), so equality means no change since publication. Across a crash, the reopened book starts at the disk's `seq` ≥ E and every later change takes a larger value (rule 2), so no later change can have generation E. Another book has another 128-bit random identity (rule 4). Therefore G clears only the restriction state it was issued for. |
| +Proof sketch: a grant G names book X, app A and generation E, which status published, so E was on X's disk under the rule (2, 4). `RetryHostAt` clears only when X is this book, memory equals disk, and A's generation now equals E. Within a run every change takes a fresh larger value (1), so equality means no change of A's restrictions since publication. Across a crash the reopened book starts at the disk's `seq` ≥ E and hands out larger values (3). A book written before the rule has another identity (4), and another book has another random identity (5). So G clears only the restriction state it was issued for. |
| |
| -- [ ] **Step 1: Write the failing tests** in `internal/rollout/failures_epoch_test.go`. Use a real temp-dir book (`OpenFailureBook(t.TempDir())`) and make a save fail by making the directory read-only (`os.Chmod(dir, 0o500)`, restored in `t.Cleanup`; skip when `os.Geteuid() == 0`, as I8 does). |
| +- [ ] **Step 1: Write the failing tests** in `internal/rollout/failures_epoch_test.go`. Use a real temp-dir book and make a save fail by making the directory read-only (`os.Chmod(dir, 0o500)`, restored in `t.Cleanup`; `t.Skip` when `os.Geteuid() == 0`). Exact sequences (plan 8): |
| |
| ```go |
| func TestAFailingDiskPublishesNoEpoch(t *testing.T) |
| -// record a failure of v2 on h1 (saved), PublishedEpoch > 0; |
| -// chmod the dir read-only; record a second failure merging h2 (save fails, dirty); |
| -// PublishedEpoch(.., "v2", "h1") == 0 while dirty; RetryHostAt with the old epoch clears nothing; |
| -// chmod back; any mutating call lands the save; PublishedEpoch > old epoch. |
| - |
| -func TestAStaleGrantStaysStaleAcrossReopen(t *testing.T) |
| -// e1 := PublishedEpoch for (v2,h1); dir read-only; a later failure of v2 on h1 changes the |
| -// restriction in memory only; reopen the book from disk (OpenFailureBook) without the change; |
| -// then record the later failure again through the reopened book (and enough further events |
| -// to pass e1 by 3 values); RetryHostAt(.., book.ID(), e1) must clear nothing. |
| -// Under the old code (reopenAdvance removed, rule 1 absent) this test must fail: show it. |
| - |
| -func TestAMarkerClearMovesTheGeneration(t *testing.T) |
| -// record v2 failure on h1 (epoch e1); set a marker for v2 on h1; clear it (ClearHoldRanBefore with a |
| -// later snapshot); PublishedEpoch != e1 and RetryHostAt(.., e1) clears nothing; a fresh read clears. |
| - |
| -func TestAGrantForAnotherBookClearsNothing(t *testing.T) // two books, equal numeric epochs |
| - |
| -func TestABookThatCannotGetAnIdentityDoesNotOpen(t *testing.T) |
| -// swap the package's rand reader seam (add `var randRead = rand.Read` in failures.go) to fail; |
| -// OpenFailureBook on an empty dir returns an error. |
| +// Record{v2 on h1, HeldDown h1} saved; e1 := PublishedEpoch(a, app, v2, h1) > 0. |
| +// chmod read-only; Record a NEW build v3 failing on h2 (creates a record: a change of the app); save fails. |
| +// PublishedEpoch(.., v2, h1) == 0 while dirty; RetryHostAt(.., v2, h1, id, e1) clears nothing (rec=false, mark=false). |
| +// chmod back; SaveIfDirty() lands; PublishedEpoch(.., v2, h1) > e1; a grant at that value clears h1. |
| + |
| +func TestEveryRemovalMovesTheGeneration(t *testing.T) // one subtest per path (plan 1) |
| +// For each: e := PublishedEpoch(...); perform the change; PublishedEpoch != e and RetryHostAt(.., e) clears nothing. |
| +// - marker of v2 on h1 replaced by a marker of v3 on h1 while v2's record holds h1 |
| +// - v2's record removed by ClearOnPointer while v2's marker on h1 stands |
| +// - v2's record pruned while v2's marker on h1 stands |
| +// - marker of v2 on h1 cleared by ClearHoldRanBefore, ClearHoldRanOnPointer, ClearHoldRan |
| +// - a merge that changes v2's record because another host failed with a newer PromotedAt |
| + |
| +func TestANoOpOutcomeDoesNotMoveTheGeneration(t *testing.T) |
| +// PreRetry, Unchanged and a duplicate marker of the same execution leave PublishedEpoch unchanged, |
| +// and a grant at that epoch still clears. |
| + |
| +func TestABookFromBeforeTheRuleGetsANewIdentity(t *testing.T) // plan 2 |
| +// Write a schema-2 file by hand without durable_epochs, with book "old", seq 10, a record of v2 holding h1 |
| +// with epochs {"": 11, h1: 11}. Open: ID() != "old"; PublishedEpoch == 0 until Flush lands; |
| +// RetryHostAt(.., "old", 11) and RetryHostAt(.., "old", 12) clear nothing; after Flush, a grant naming the new |
| +// identity and the published generation clears h1. Reopen: the identity and durable_epochs persist. |
| + |
| +func TestAReopenNeverReissuesAPublishedGeneration(t *testing.T) |
| +// Publish e1; make three changes that save; reopen from disk; make three more changes; |
| +// every PublishedEpoch read after the reopen is > the largest read before it. |
| + |
| +func TestAGrantForAnotherBookClearsNothing(t *testing.T) // two books opened with equal numeric state |
| +func TestABookThatCannotGetAnIdentityDoesNotOpen(t *testing.T) // randRead fails -> OpenFailureBook errors |
| ``` |
| |
| -- [ ] **Step 2: Run them and watch them fail** (`nice -n 19 go test ./internal/rollout/ -run 'TestAFailingDisk|TestAStaleGrant|TestAMarkerClear|TestAGrantForAnother|TestABookThatCannot' -count=1`, sandboxed env). Paste the failures. |
| -- [ ] **Step 3: Implement rules 1 to 4.** Switch `internal/rollout/status.go` (both `hr.HeldEpoch = ...` lines) and `internal/host/rolloutdeclined.go` (`HeldEpoch:` in `failureHolds.report`) to `PublishedEpoch`. Keep `RestrictionEpoch` for logs and tests if used. Update `HostRetryGrant.Epoch`'s doc. |
| -- [ ] **Step 4: Run the new tests, then the package** (`go test ./internal/rollout/ -count=1`) and `go test ./internal/host/ -run 'Retry|Declined|Rollout' -count=1` unsandboxed with `TMPDIR=/tmp/k`. Existing tests that relied on `reopenAdvance` (grep `reopenAdvance` and `1 << 20` in `_test.go`) are updated to the new rule, not deleted; say which in the report. |
| -- [ ] **Step 5: Run `internal/rollout/testdata/rolloutsim/mutate.sh 300`** (baseline plus M1–M12 still CAUGHT). Paste the summary lines. |
| -- [ ] **Step 6: Commit** `rollout: crash epochs are published only once on disk, and a restriction's generation never repeats`. |
| +In `internal/host/rolloutdeclined_test.go`: a declined box whose book opened dirty and whose disk recovers publishes an epoch after the next periodic pass with no pointer or grant arriving (plan 10). |
| |
| -### Task 2: Simulation invariants and mutants for the hard limit; sweep timeout |
| +- [ ] **Step 2: Run them and watch them fail.** `nice -n 19 go test ./internal/rollout/ -run 'TestAFailingDisk|TestEveryRemoval|TestANoOp|TestABookFromBefore|TestAReopenNever|TestAGrantForAnother|TestABookThatCannot' -count=1` (sandboxed env). Paste the failures. |
| +- [ ] **Step 3: Implement rules 1 to 6.** Switch both status producers and `failureHolds.report` to `PublishedEpoch`. Keep `RestrictionEpoch` as the in-memory value for logs, or remove it if nothing else needs it. Update `HostRetryGrant.Epoch`'s doc (it is the app's restriction generation as published). |
| +- [ ] **Step 4: Run** the new tests, `go test ./internal/rollout/ -count=1`, and unsandboxed `TMPDIR=/tmp/k nice -n 19 go test ./internal/host/ -run 'Retry|Declined|Rollout' -count=1`. Tests that relied on `reopenAdvance` or per-host epochs are updated to the new rule, not deleted; list them in the report. |
| +- [ ] **Step 5: Commit** `rollout: a restriction's generation is durable before use and never repeats (Paul's hard limit)`. |
| +- [ ] **Step 6: Run `internal/rollout/testdata/rolloutsim/mutate.sh 300`** on that commit (baseline plus M1–M12 CAUGHT). Paste the summary and the commit it tested. |
| + |
| +### Task 2: Simulation: stale grants, crashes and failed saves; mutants; sweep timeout |
| |
| Model: opus. |
| |
| **Files:** |
| -- Modify: `internal/rollout/sim_test.go`, `internal/rollout/testdata/rolloutsim/mutate.sh`, and the `// rolloutsim:Mn` marker lines in `internal/rollout/failures.go` (and `status.go` if M13's line lives there). |
| +- Modify: `internal/rollout/sim_test.go`, `internal/rollout/testdata/rolloutsim/mutate.sh`, and `// rolloutsim:Mn` marker comments on the lines Task 1 wrote in `internal/rollout/failures.go`. |
| |
| **Interfaces:** |
| -- Consumes: Task 1's `PublishedEpoch`, `RetryHostAt`, the dirty rule, the generation rule, `OpenFailureBook`. |
| -- Produces: invariants I9 and I10; mutants M13–M16 in `mutate.sh`. |
| +- Consumes: Task 1's `PublishedEpoch`, `RetryHostAt`, `SaveIfDirty`, `OpenFailureBook`, the generation and durability rules. |
| +- Produces: invariants I9 and I10; mutants M13–M16. |
| + |
| +The simulation does not model stale grants today (round 1 ledger, Task 8 minor 8). For the seeds whose book is on disk (the I8 machinery), add: |
| +- **Grants.** At random steps, read a held host's published (book, generation) as status does and queue a grant, stamped with the model step it was issued at. Deliver queued grants at later random steps through `RetryHostAt`, some after many events and some after a restart. The queue lives in the model and survives restarts and book recreation. |
| +- **Failed saves and crashes.** Random read-only windows (as I8 does), and random agent restarts that reopen the book from disk under a new Manager. A restart drops in-flight executions as a real one does. |
| +- **Book recreation** (for M16): at a random step, remove one seed's book file and open a new book (a box rebuild). |
| |
| -The simulation does not model stale grants today (round 1 ledger, Task 8 minor 8). Add, for the seeds whose book is on disk (the I8 machinery): |
| -- **Grants.** At random steps, read a held host's published (book, epoch) the way status does and queue a grant; deliver queued grants at later random steps (some after many events, some after a reopen) through `RetryHostAt`. |
| -- **Failed saves and crashes.** Random read-only windows (as I8 does) and random agent restarts that reopen the book from disk with a new Manager. Grants queued before a restart survive it (the console holds them). |
| -- **Ghost generations.** The model keeps, outside the book, a counter per (account, app, build, host) bumped on every *failure event* for that host and build: a record created or merged with that host failing, a marker set or renewed for that build on that host, an intent reconciled on that host. A queued grant remembers the ghost value at issue. |
| +Model-side facts, kept outside the book and never derived from the book's verdicts (plan 7): for every execution the model starts it knows the account, app, build, host and its start step; for every failure or hold-down the model delivers, it records a **failure fact** (app, build, host, start step). A failure fact is **answered** by any clear the model issued after the fact's start step: a grant for that (build, host) issued after it, a signed rollback or `promote --retry` pointer for that build signed after it. These are the clears the spec allows to lift it; a grant or pointer issued before the execution began is not one. |
| |
| Invariants: |
| -- **I9 (no stale grant):** a grant whose `RetryHostAt` clears anything had the same ghost value at issue as at delivery. Message `rolloutsim: I9 violated at seed %d: ...`. |
| -- **I10 (durable before use):** every non-zero epoch read from `PublishedEpoch` equals the restriction's generation in the book's last landed file (parse the file the model reads back). Message `rolloutsim: I10 violated at seed %d: ...`. |
| +- **I9 (a stale grant never clears a later failure):** when `RetryHostAt` for a grant issued at step t clears anything, no unanswered failure fact of the grant's (build, host) has a start step after t. Message `rolloutsim: I9 violated at seed %d: ...`. |
| +- **I10 (durable before use):** every non-zero value `PublishedEpoch` returns equals the app's generation in the book's last landed file, which the model re-reads from disk at that moment. Message `rolloutsim: I10 violated at seed %d: ...`. |
| |
| -Mutants (add a `// rolloutsim:Mn` marker on exactly one line each; perl substitution in `MUTANTS`): |
| -- **M13** `PublishedEpoch` ignores the dirty check → I10 (and I9). |
| -- **M14** a marker clear does not move the generation (the `rec.Epochs[H] = b.bumpLocked()` line of rule 3 in one clear path) → I9. |
| -- **M15** the reopen starts below the disk's sequence (for example `b.seq = f.Seq` becomes `b.seq = f.Seq / 2`) → I9. |
| -- **M16** `RetryHostAt` skips the book-identity comparison → I9, with the model recreating one host's book (a box rebuild: the book file removed and opened anew) at a random step. If the simulation cannot reach M16, make it a `test:` mutant on `TestAGrantForAnotherBookClearsNothing` and say why in the list comment, as M12 does. |
| +Mutants (one `// rolloutsim:Mn` marker on exactly one line each, a perl substitution in `MUTANTS`; give each a witness, plan 8): |
| +- **M13** `PublishedEpoch` ignores the dirty check → I10: publish while a save has failed, compared with the last landed file. |
| +- **M14** one removal path (pick the marker replacement in `markLocked`) does not move the generation → I9: witness is a grant issued under v2's marker, the marker replaced by a failed hold run of v3 on the same host, then v3's marker cleared by a pointer, then the grant delivered. |
| +- **M15** the reopen ignores the saved generations and markers when it sets `seq` (keep only `f.Seq`, then subtract enough to force reuse, for example `b.seq = f.Seq / 2`): witness removes the high-valued components before the reopen so the max cannot repair it, then forces numerical reuse → I9. |
| +- **M16** `RetryHostAt` skips the book-identity comparison → I9 through book recreation. If the simulation cannot reach it, make it a `test:` mutant on `TestAGrantForAnotherBookClearsNothing` and say why in the list comment, as M12 does. |
| |
| -mutate.sh: extend the verdict text and the I-regex from "I1 to I8" to "I1 to I10". Raise the per-run timeout: replace the fixed `-timeout 20m` with a value scaled by seeds, `TIMEOUT_MIN=$(( SEEDS * 3 / 100 + 20 ))` (3000 seeds → 110 minutes), and print it in the header. Round 1 found 20 minutes too short for 3000 seeds. |
| +mutate.sh: update the header and verdict comments to name I1 to I10 (the check regex already accepts any digits; plan 12). Replace the fixed `-timeout 20m` with a seeds-scaled value, `TIMEOUT_MIN=$(( SEEDS * 3 / 100 + 20 ))` (3000 seeds → 110 minutes), used as `-timeout ${TIMEOUT_MIN}m`, and print it in the header. Round 1 found 20 minutes too short for 3000 seeds. |
| |
| -- [ ] **Step 1:** Add the model state, I9 and I10, and run `ROLLOUT_SIM_SEEDS=300 GOMAXPROCS=2 nice -n 19 go test ./internal/rollout/ -run '^TestSimInvariants$' -count=1` on the Task 1 code. It must pass. Paste the line. |
| -- [ ] **Step 2:** For each of M13–M16, apply the substitution by hand in a scratch copy only (never the checkout), or run `mutate.sh 300 M13 M14 M15 M16` after adding them. Every one must read CAUGHT on its stated invariant. Paste the summary. |
| -- [ ] **Step 3:** Run `mutate.sh 300` (all mutants). Paste the summary. |
| -- [ ] **Step 4: Commit** `rollout sim: stale grants, crashes and failed saves; I9, I10, M13–M16; sweep timeout scales with seeds`. |
| +- [ ] **Step 1:** Add the model state, I9 and I10. Commit (`rollout sim: stale grants, crashes and failed saves; I9, I10`). Run `ROLLOUT_SIM_SEEDS=300 GOMAXPROCS=2 nice -n 19 go test ./internal/rollout/ -run '^TestSimInvariants$' -count=1`; it must pass. Paste the line. |
| +- [ ] **Step 2:** Add the markers and the four mutants, commit, run `mutate.sh 300 M13 M14 M15 M16`. Each must read CAUGHT on its stated invariant. Paste the summary and the tested commit. |
| +- [ ] **Step 3:** Run `mutate.sh 300` (all mutants) on the final commit. Paste the summary. |
| |
| ### Task 3: Retry grants in their own object, and the console's capability check |
| |
| -Model: opus (security). Recon: `~/.cache/zdd-impl/readiness2/recon-D-retry-console.md` §2. |
| +Model: opus (security). Recon: `recon-D-retry-console.md` §2, with this plan's corrections. |
| |
| **Files:** |
| -- Modify: `internal/release/restartscope.go` (`ValidRestartScope` refuses kind retry and `Signed`), new `internal/release/hostretrylog.go` (`HostRetryKey`, `LoadHostRetries`, `AppendHostRetry`, `claimRawAppendHostRetry`) |
| -- Modify: `internal/host/restart.go` and `internal/host/restartscope.go` (a `hostRetryPass` over the new object, seeded like `restartScopePass`; a retry kind found in `restart-scope.json` clears nothing and logs once) |
| -- Modify: `internal/status/snapshot.go` (`HostRollout.RetryV int \`json:"retry_v,omitempty"\``), set to 1 beside `HeldBook` in `internal/rollout/status.go` (both producers) and `internal/host/rolloutdeclined.go` |
| -- Modify: `internal/console/apiinstances.go` (`rolloutView.retryV`), `internal/console/apiretry.go` (`retryHostBuild` writes with `AppendHostRetry`; refusals), `internal/console/browserretry.go` |
| +- Modify: `internal/release/restartscope.go` (`ValidRestartScope` refuses kind retry and `Signed` for new writes), new `internal/release/hostretrylog.go` (`HostRetryKey`, `LoadHostRetries`, `AppendHostRetry`, `claimRawAppendHostRetry`) |
| +- Modify: `internal/host/restart.go`, `internal/host/restartscope.go` (a `hostRetryPass` over the new object; legacy retry entries in `restart-scope.json` are still verified and applied with the same checks; neither is seed-skipped at restart) |
| +- Modify: `internal/status/snapshot.go` (`HostRollout.RetryV int \`json:"retry_v,omitempty"\``), set to 1 beside `HeldBook` at both producers in `internal/rollout/status.go` and in `internal/host/rolloutdeclined.go` |
| +- Modify: `internal/console/apiinstances.go` (`rolloutView.retryV`, copied where `heldEpoch` and `heldBook` are), `internal/console/apiretry.go` (`retryHostBuild` reads the restriction and refuses; `requestHostRetry` performs the append, now through `AppendHostRetry`; plan 12), `internal/console/browserretry.go` |
| - Modify: `internal/provision/plan9_b8_census_test.go` (a digest row for `claimRawAppendHostRetry`, and any changed `claimRaw*` digest) |
| - Test: `internal/release/hostretrylog_test.go` (new), `internal/host/restartscope_test.go`, `internal/console/apiretry_test.go`, `internal/console/browserretry_test.go` |
| |
| **Interfaces:** |
| -- Consumes: Task 1's `PublishedEpoch` (status's `HeldEpoch` is 0 while the box's book is not saved). |
| +- Consumes: Task 1's published generation (`HeldEpoch` 0 on a held host while the book is not saved). |
| - Produces: |
| - `func HostRetryKey(acct, app string) string` → `apps/<acct>/<app>/host-retry.json` |
| - `func LoadHostRetries(ctx context.Context, s Store, acct, app string) (*ScopedRestartGeneration, bool, error)` |
| - `func AppendHostRetry(ctx context.Context, s Store, acct, app, host, signed string) (RestartRequest, error)` (same JSON shape and CAS loop as the restart-scope log, keep 32, op name `release.AppendHostRetry`) |
| - `status.HostRollout.RetryV`; `rolloutView.retryV` |
| - console errors `errRetryUnsupported` (report has `retry_v == 0` or no `held_book`) and `errRetryUnsaved` (`retry_v >= 1`, held, `held_epoch == 0`) |
| -- Task 4 consumes: `func (r *rolloutView) retryable() bool` is Task 4's; this task only adds `retryV` and the two errors. |
| + |
| +Supported transitions (plan 9), each pinned by a test: |
| +- new console, new box: the grant rides `host-retry.json`, the box verifies and applies it. |
| +- new console, older box (no `retry_v`): the console refuses with `errRetryUnsupported` and writes nothing. |
| +- console built from main before this branch, new box: its signed, book-bound grant in `restart-scope.json` is verified and applied with the same checks (an unsigned one is dropped with a log line, as today). A v0.25.1 console cannot write a retry at all. |
| +- a grant written before an agent restart and not yet applied is applied after it. |
| +Write the guarantee as: never applied unchecked; refused with a message by a new console when the box cannot take it; an unsigned or stale request is dropped with an operator log line. |
| |
| New strings (draft, TODO-COPY): |
| - `zdd-readiness2.retry-unsupported.api`: "this instance's box is too old to take a retry. update the box, then try again" |
| @@ -170,25 +192,22 @@ New strings (draft, TODO-COPY): |
| - `zdd-readiness2.retry-unsaved.browser`: "This instance's box has not saved its failure yet. Try again in a minute." |
| The API door answers 409 with these; the browser door flashes them. Neither writes anything. |
| |
| -Tests (write first, watch fail): |
| -- release: append/load round trip; a restart-scope rewrite by a struct without `Signed` leaves `host-retry.json` untouched; `ValidRestartScope` refuses `{Kind: retry}` and any `Signed`. |
| -- host: move `TestRestartLegHandsARetryToTheRolloutManager`, `...DropsARetryThatIsNotTheRootsGrant`, `...RetryGrantsAgainstTheBook` to the new object; add "a retry kind in restart-scope.json clears nothing"; a grant seen once is not replayed after an agent restart (the seed marks). |
| -- console: report without `retry_v` → 409 unsupported and nothing written; `retry_v: 1`, held, `held_epoch: 0` → 409 unsaved; `retry_v: 1` with epoch and book → the signed entry lands in `host-retry.json` and nothing in `restart-scope.json` (update `lastScoped`). |
| +Tests (write first, watch fail): release round trip; a restart-scope rewrite by a struct without `Signed` leaves `host-retry.json` untouched; `ValidRestartScope` refuses a new retry kind. Host: the three existing retry tests (`TestRestartLegHandsARetryToTheRolloutManager`, `...DropsARetryThatIsNotTheRootsGrant`, `...RetryGrantsAgainstTheBook`) cover the new object; a legacy signed grant in `restart-scope.json` is applied; a legacy unsigned one clears nothing; a grant written before an agent restart is applied after it (the restart memory is rebuilt). Console: no `retry_v` → 409 unsupported, nothing written; `retry_v: 1`, held, `held_epoch: 0` → 409 unsaved; `retry_v: 1` with epoch and book → the signed entry lands in `host-retry.json` and nothing in `restart-scope.json` (update `lastScoped`). |
| |
| -- [ ] Steps: failing tests; implement; run `go test ./internal/release/ ./internal/console/ -count=1` sandboxed, `go test ./internal/host/ -run 'Restart|Retry' -count=1` and `go test ./internal/provision/ -run Census -count=1` unsandboxed (`TMPDIR=/tmp/k`); commit `retries: signed host retries ride their own object; the console asks the box's report first`. |
| +- [ ] Steps: failing tests; implement; `go test ./internal/release/ ./internal/console/ -count=1` sandboxed; unsandboxed `TMPDIR=/tmp/k nice -n 19 go test ./internal/host/ -run 'Restart|Retry' -count=1` and `go test ./internal/provision/ -run Census -count=1`; commit `retries: signed host retries ride their own object; the console asks the box's report first`. |
| |
| ### Task 4: A declined box's hold and Retry in the console |
| |
| Model: sonnet. Recon: `recon-D-retry-console.md` §1. |
| |
| **Files:** |
| -- Modify: `internal/console/instancepage.go` (`HoldText`, `PromoteRetryText`, `RolloutNote`, a `CanRetry`), `internal/console/apiretry.go` (`heldRestriction.declined`), `internal/console/browserretry.go`, `internal/console/swapcopy.go` (two new strings), `internal/console/templates/app_instance.html` (Retry form on `CanRetry`) |
| -- Create: `copy-review/strings.json` (force-added: `git add -f`), covering every TODO-COPY string on the branch (Task 3's four and this task's two), rows `{id, section, label, text, context, notes}` |
| +- Modify: `internal/console/instancepage.go` (`HoldText`, `PromoteRetryText`, `RolloutNote`; `CanRetry` on the rollout view and a forwarding `instanceDetail.CanRetry`, since the template's dot is `instanceDetail`; plan 12), `internal/console/apiretry.go` (`heldRestriction.declined`), `internal/console/browserretry.go`, `internal/console/swapcopy.go` (two new strings), `internal/console/templates/app_instance.html` (Retry form on `.CanRetry`) |
| +- Create: `copy-review/strings.json` (force-added with `git add -f`; `copy-review/` is excluded), covering every TODO-COPY string on the branch (Task 3's four and this task's two), rows `{id, section, label, text, context, notes}` |
| - Test: `internal/console/browserretry_test.go`, `internal/console/apiretry_test.go` |
| |
| **Interfaces:** |
| - Consumes: Task 3's `rolloutView.retryV`, `errRetryUnsupported`, `errRetryUnsaved`. |
| -- Produces: `func (r *rolloutView) retryable() bool { return r != nil && r.HeldBy != "" && (r.Unavailable == "" || rolloutwords.UnavailableDeclines(r.Unavailable)) }`, used by both the door (`retryHostBuild`, replacing its inline test) and the page, so they cannot disagree. `CanRetry()` is `retryable() && retryV >= 1 && heldEpoch != 0 && heldBook != ""`. |
| +- Produces: `func (r *rolloutView) retryable() bool { return r != nil && r.HeldBy != "" && (r.Unavailable == "" || rolloutwords.UnavailableDeclines(r.Unavailable)) }`, used by both `retryHostBuild` (replacing its inline test) and the page. `CanRetry()` is `retryable() && retryV >= 1 && heldEpoch != 0 && heldBook != ""`. |
| |
| Page rules: |
| - The hold row shows whenever `retryable()`. Text: `Attempt == AttemptDown` → the approved `zdd.instance.hold-down-text` verbatim; declined (`Unavailable != ""`) → new `HoldDeclinedText`; otherwise today's `hold-text`. Never format `Reason` for a declined host (it is empty there). |
| @@ -198,24 +217,24 @@ New strings (draft, TODO-COPY): |
| - `zdd-readiness2.hold-declined`: "Build %s failed on this box. This box cannot run two builds side by side, so it will not start this instance on that build. Retry it, or ship a newer build." |
| - `zdd-readiness2.retry-line-declined`: "This clears the failure hold on {host}. Its box cannot run two builds side by side, so its next start runs {build} the old way, with no canary." |
| |
| -Tests: a declined fixture (`Unavailable: "host-claims-on"`, a failed host and a down host, with `HeldBy`, `HeldEpoch`, `HeldBook`, `RetryV: 1`) shows the hold row, the Retry form and the declined confirmation; a refusing code (`no-deploy-store`) with a held host shows neither; `TestInstancePageOfAnUnavailableBoxShowsNoHoldAndNoRetry` stays green (adjust its fixture name if it now means "refusing"); a declined host with `HeldEpoch: 0` shows the hold row and no form. |
| +Tests: a declined fixture (`Unavailable: "host-claims-on"`, a failed host and a down host, with `HeldBy`, `HeldEpoch`, `HeldBook`, `RetryV: 1`) shows the hold row, the Retry form and the declined confirmation; a refusing code (`no-deploy-store`) with a held host shows neither; `TestInstancePageOfAnUnavailableBoxShowsNoHoldAndNoRetry` stays green (rename it if it now means "refusing"); a declined host with `HeldEpoch: 0` shows the hold row and no form. |
| |
| - [ ] Steps: failing tests; implement; `go test ./internal/console/ -run 'Retry|InstancePage|Copy' -count=1` sandboxed; write `copy-review/strings.json`; commit `console: a declined box's hold and retry on the instance page (copy pending review)`. |
| |
| ### Task 5: Declined boxes: pin the live link once, and recover a stranded sidecar |
| |
| -Model: opus (security). Recon: `recon-A-declined.md` (read all of it: settings plan 4 already closed most of the production path; this task makes the backend's live-link arms safe on their own and closes the sidecar). |
| +Model: opus (security). Recon: `recon-A-declined.md` (read all of it: settings plan 4, 8916558d, already closed most of the production path; this task makes the backend's live-link arms safe on their own and closes the sidecar). |
| |
| **Files:** |
| - Modify: `internal/activator/backend_exec.go` (`declinedLivePin`, `spawnUnitFrom`, `startSidecarUnitFrom`, `seedChild`, `seedUnit`, `BuildIdentity`; rewrite the residual comments the recon names), `internal/activator/activator.go` (`startSidecar`), the `SplitBackend` forward |
| - Test: `internal/activator/identity_test.go`, `internal/activator/exec_gate_test.go`, `internal/activator/declinedgate_identity_test.go`, `internal/activator/wakeidentity_test.go` (fake backend gains `BuildIdentity`) |
| |
| **Interfaces:** |
| -- Produces: `func (b *ExecBackend) declinedLivePin(ctx context.Context, rt registry.Route) (release.StagedIdentity, error)`; `type buildIdentifier interface { BuildIdentity(rt registry.Route, build string) (release.StagedIdentity, error) }` with `func (b *ExecBackend) BuildIdentity(rt registry.Route, build string) (release.StagedIdentity, error)`. |
| +- Produces: `func (b *ExecBackend) declinedLivePin(ctx context.Context, rt registry.Route) (release.StagedIdentity, error)`; `type buildIdentifier interface { BuildIdentity(rt registry.Route, build string) (release.StagedIdentity, error) }`; `func (b *ExecBackend) BuildIdentity(rt registry.Route, build string) (release.StagedIdentity, error)`. |
| |
| -Design (from the recon): when `LiveExecGate != nil` and a start has no staged file, resolve and hash the link once, ask the gate about that version, and run that identity by path (flat) or through `StartExe`/`StartSidecarExe`/`SeedExe` (nested, root pins the file). In seeds, do it before the `seedOwnsDB` stat so a refusal never deletes a database. In `startSidecar`, when the gate refuses the link's version and the generation's build differs, try the instance's own build through `BuildIdentity` and the gate; run it if both pass; otherwise keep the refusal. |
| +Design (recon A): when `LiveExecGate != nil` and a start has no staged file, resolve and hash the link once, ask the gate about that version, and run that identity by path (flat) or through `StartExe`/`StartSidecarExe`/`SeedExe` (nested: root pins the file). In seeds, do it before the `seedOwnsDB` stat so a refusal never deletes a database. In `startSidecar`, when the gate refuses the link's version and the generation's build differs, try the instance's own build through `BuildIdentity` and the gate; run it if both pass; otherwise keep the refusal. |
| |
| -Tests (from the recon; write first): nested repoint between read and start runs v1 (`startexe ... v1`, gate asked `[v1]`, no plain `start`); the same for the sidecar; a link outside the store asks root for nothing; flat and nested seeds run the gated file and a refused seed creates no database; a sidecar under a link naming held v2 runs v1 (gate asked `[v2 v1]`), and with v1 refused too it stays down while the instance serves. |
| +Tests (write first): a nested repoint between read and start runs v1 (`startexe ... v1`, gate asked `[v1]`, no plain `start`); the same for the sidecar; a link outside the store asks root for nothing; flat and nested seeds run the gated file and a refused seed creates no database; a sidecar under a link naming held v2 runs v1 (gate asked `[v2 v1]`), and with v1 refused too it stays down while the instance serves. |
| |
| - [ ] Steps: failing tests; implement; unsandboxed `TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -run 'Identity|Gate|Sidecar|Seed' -count=1`, then the whole package once; commit `activator: a declined box pins the live link once, and a refused sidecar runs the instance's build`. |
| |
| @@ -224,51 +243,61 @@ Tests (from the recon; write first): nested repoint between read and start runs |
| Model: opus (concurrency). Recon: `recon-B-good-f6.md` §1 and §2. |
| |
| **Files:** |
| -- Modify: `internal/activator/workerswap.go` (`ReplaceWorker`'s forgiveness of `errSwapNotAwake`), `internal/activator/activator.go` (`actEntry.parkedInc`, `markCleanParkLocked`, the sweep's clean-exit park, `WakeWorker`, `begin`, new `awaitWake`), optionally `internal/rollout/manager.go` (`servedOneShot` refuses when the activator reports the host fenced; `Fenced` on `rollout.Activator`) |
| +- Modify: `internal/activator/workerswap.go` (`ReplaceWorker`'s forgiveness of `errSwapNotAwake`), `internal/activator/activator.go` (`actEntry.parkedInc`, `markCleanParkLocked`, the sweep's clean-exit park, `WakeWorker`, `begin`, new `awaitWake`) |
| - Test: `internal/activator/workerswap_test.go` |
| |
| **Interfaces:** |
| - Produces: `func (a *Service) markCleanParkLocked(e *actEntry)`; `func (a *Service) awaitWake(ctx context.Context, e *actEntry, ready <-chan struct{}, deadline *time.Timer, arrival time.Time, extended *bool) error`; test seam `var wakeBoundExtended func()` (nil in production). |
| |
| -Design: as recon B §1 and §2. Forgive `errSwapNotAwake` only when `e.parkedInc == inc1`, no fence stands and the service is not shutting down; every other stop is abandoned. `WakeWorker` and `begin` wait again on the same `ready` after extending the bound, never re-reading the entries map. |
| +Design (recon B §1, §2): forgive `errSwapNotAwake` only when `e.parkedInc == inc1`, no fence stands and the service is not shutting down; every other stop of an exited one-shot is abandoned. `WakeWorker` and `begin` wait again on the same `ready` after extending the bound, never re-reading the entries map. The optional `servedOneShot` fence check in recon B is not needed and is not added. |
| |
| Tests (write first, watch fail): exit 0 then a renewal failure and a sweep → `SwapAbandoned`; exit 0 then `FenceAndStop` → `SwapAbandoned`; positive control, exit 0 then a sweep → `SwapDone` with `CleanExit`; a wake that fails inside the extension window → `WakeWorker` returns the wake's error and the backend counts one lease acquire (today two); the same for a request through `begin`. |
| |
| -- [ ] Steps: failing tests; implement; `TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -run 'Worker|OneShot|Wake' -count=1` unsandboxed, then `go test -race` on the new tests, then the package once; commit `activator: a one-shot is good only after a clean park, and a waiting wake never starts a second`. |
| +- [ ] Steps: failing tests; implement; unsandboxed `TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -run 'Worker|OneShot|Wake' -count=1`, then `-race` on the new tests, then the package once; commit `activator: a one-shot is good only after a clean park, and a waiting wake never starts a second`. |
| |
| ### Task 7: Fail closed when a generation's limits cannot be applied; kernel positive controls |
| |
| -Model: opus. Recon: `recon-C-cgroup.md`. |
| +Model: opus. Recon: `recon-C-cgroup.md`, with case G replaced below. |
| |
| **Files:** |
| -- Modify: `internal/cgroup/gen.go` (`ErrLimitNotApplied`, `ensureParent` branches A–D, `delegatesMemory`, `writeSubtreeControl` seam), `internal/cgroup/cgroup.go` (`startIn(..., required bool, ...)`, `applyLimit` returns its first failure, case G), `internal/activator/swap.go` (`verdictOf`), `internal/activator/logsink.go` (`memberSafeWakeReason` → `ErrFenced`; fix the stale comment the recon names), `internal/activator/containment.go` (case G → `markStartUncertain`) |
| +- Modify: `internal/cgroup/gen.go` (`ErrLimitNotApplied`, `ensureParent` branches A–D, `delegatesMemory`, `writeSubtreeControl` seam), `internal/cgroup/cgroup.go` (`startIn(..., required bool, ...)`, `applyLimit` returns its first failure; with `required`, never the PID-move fallback), `internal/activator/swap.go` (`verdictOf`), `internal/activator/logsink.go` (`memberSafeWakeReason` → `ErrFenced`; fix the stale comment the recon names) |
| - Test: `internal/cgroup/gen_test.go`, `internal/cgroup/cgroup_test.go`, `internal/activator/swap_overlap_test.go`, `internal/activator/swap_executed_test.go`, `internal/activator/logsink_test.go`, `internal/activator/genparent_kernel_linux_test.go`, `internal/activator/claimspawn_kernel_linux_test.go` |
| |
| **Interfaces:** |
| - Produces: `var ErrLimitNotApplied = errors.New("cgroup: a generation's limits could not be applied")`; `func (m *Manager) startIn(host, dir string, limit Limit, required bool, build func() *exec.Cmd, before func() error) (*exec.Cmd, error)`; `func (m *Manager) applyLimit(host, dir string, limit Limit) error`. |
| |
| -Design: as recon C §2. `StartGated` passes `required=false` (flag-off path byte-for-byte the same); `StartChildGated` passes `true`. EBUSY on `subtree_control` maps to `ErrLegacyParentPopulated`. Fixtures that write `populated 1` with no `subtree_control` get one (`cpu memory io pids`). |
| +Design: recon C §2 for A–F. `StartGated` passes `required=false` (the flag-off path is byte-for-byte the same); `StartChildGated` passes `true`. EBUSY on `subtree_control` maps to `ErrLegacyParentPopulated`. **Case G (plan 4):** with `required`, the start must place the process in its cgroup before it executes (clone3 `CLONE_INTO_CGROUP`). When the fallback is latched, or the cgroup directory cannot be opened for the clone, return `ErrLimitNotApplied` before `build()` runs; never start and move by PID. Fixtures that write `populated 1` with no `subtree_control` get one (`cpu memory io pids`). |
| |
| -Tests: one unit test per branch A–G without a kernel (the recon names the fixture trick for each), asserting nothing started (except G, which asserts the process was killed and the error is uncertain) and no generation child left; `ErrLimitNotApplied` rows in `TestVerdictOf`, `TestClassifyStep`, `TestSwapResultReportsWhetherTheAttemptsBuildExecuted` and the logsink table. Kernel controls (skip-gated by `CARLOS_CLAIM_SPAWN_KERNEL=1`, compile here, run on the dev box later): the sibling control and the recovery control in `TestGenerationStartRefusedBesideADirectParentMember`; a live g1 kept running through `killSidecarRespawn`'s loop. |
| +Tests: one unit test per branch A–F plus G without a kernel (the recon names the fixture trick for each; for G, store the fallback latch true), each asserting nothing started and no generation child left; `ErrLimitNotApplied` rows in `TestVerdictOf`, `TestClassifyStep`, `TestSwapResultReportsWhetherTheAttemptsBuildExecuted` (`SwapAbandoned`, not executed) and the logsink table. Kernel controls (skip-gated by `CARLOS_CLAIM_SPAWN_KERNEL=1`; they compile here and run on the dev box later): the sibling control and the recovery control in `TestGenerationStartRefusedBesideADirectParentMember`; a live g1 kept running through `killSidecarRespawn`'s loop. |
| |
| -- [ ] Steps: failing tests; implement; `go test ./internal/cgroup/ -count=1` unsandboxed with `TMPDIR=/tmp/k`, then `go test ./internal/activator/ -run 'Verdict|Classify|SwapResult|Logsink|MemberSafe' -count=1`; `go vet -tags linux ./internal/activator/` compiles the kernel files; `GOOS=darwin go build ./...`; commit `cgroup: a generation whose limits cannot be applied does not start`. |
| +- [ ] Steps: failing tests; implement; unsandboxed `TMPDIR=/tmp/k nice -n 19 go test ./internal/cgroup/ -count=1`, then `go test ./internal/activator/ -run 'Verdict|Classify|SwapResult|Logsink|MemberSafe' -count=1`; `go vet ./internal/activator/` (compiles the linux kernel test files); `GOOS=darwin go build ./...`; commit `cgroup: a generation whose limits cannot be applied does not start`. |
| |
| -### Task 8: Prove a failed seed's processes gone; the flag-off normal-stop half of F6 |
| +### Task 8: Prove a seed's processes gone; the flag-off normal-stop half of F6 |
| |
| -Model: opus (concurrency, durability). Recon: `recon-B-good-f6.md` §3 and §4. |
| +Model: opus (concurrency, durability). Recon: `recon-B-good-f6.md` §3 and §4, with the corrections below (plan 3, 5, 6). |
| |
| **Files:** |
| -- Modify: `internal/activator/backend_exec.go` (`seedDescendantsGone`, `seedChild`'s failure branch and its lingering-helper success branch, `EmptyHost`), `internal/activator/opset.go` (`stopHostProven`, `hostEmptier`), `internal/activator/activator.go` (`actEntry.launchLayout` set at install; `stop()`; the pending-stop path through new `reapUncontainedStop`), the `SplitBackend` forward |
| +- Modify: `internal/activator/backend_exec.go` (`seedDescendantsGone`, `seedChild`'s failure and success branches, `EmptyHost`, the unproven-seed obligation file, `Restore`'s warm-cache check), `internal/activator/activator.go` (`doWake`'s seed decision; `actEntry.launchLayout` set at install; `stop()`; the pending-stop path through new `reapUncontainedStop`; the in-flight sidecar start), `internal/activator/opset.go` (`stopHostProven`, `hostEmptier`), the `SplitBackend` forward |
| - Test: `internal/activator/activator_test.go`, `internal/activator/containment_test.go`, `internal/activator/wakeplan_test.go` (the `plannedHostKill` double pattern) |
| |
| **Interfaces:** |
| -- Produces: `func (b *ExecBackend) seedDescendantsGone(ctx context.Context, rt registry.Route, child string) error`; `func (b *ExecBackend) EmptyHost(ctx context.Context, rt registry.Route) error`; `type hostEmptier interface{ EmptyHost(context.Context, registry.Route) error }`; `func (a *Service) stopHostProven(e *actEntry) bool`; `func (a *Service) reapUncontainedStop(e *actEntry)`. |
| +- Produces: `func (b *ExecBackend) seedDescendantsGone(ctx context.Context, rt registry.Route, child string) error`; `func (b *ExecBackend) EmptyHost(ctx context.Context, rt registry.Route) error`; `type hostEmptier interface{ EmptyHost(context.Context, registry.Route) error }`; `func (a *Service) stopHostProven(e *actEntry) bool`; `func (a *Service) reapUncontainedStop(e *actEntry)`; `const seedUnprovenSuffix = ".seed-unproven"`. |
| |
| -Design: as recon B §3 and §4. A failed flat seed's cgroup (flag on: its role child; flag off: the host cgroup) is killed and proven empty before `RemoveChild` and `removeSeedDB`; an unproven seed returns an error wrapping `ErrSeedUnproven`, which keeps the lease and the database. Flag off, a flat or foreign exec row's terminal stop proves the host cgroup empty before `finishStop` releases the lease: `EmptyHost` reads `cgroup.events` and kills only when populated, so a clean sleep leaves no kill mark. `ErrNoDelegation` keeps the leader-only proof with a log line. Nothing runs under `a.mu`. |
| +Design: |
| +1. **Failed seed.** A failed flat seed's cgroup (flag on: its role child; flag off: the host cgroup) is killed and proven empty before `RemoveChild` and `removeSeedDB`. `ErrNoDelegation` keeps the leader-only proof with a log line. When the proof fails, write `<db>.seed-unproven` durably (safefile) beside the database, keep the database, and return an error wrapping `ErrSeedUnproven` (the lease is kept to its TTL, as today). |
| +2. **The obligation survives (plan 3).** Before any wake treats an existing database as warm (`Restore`, and `doWake`'s `fresh` decision), it checks for `<db>.seed-unproven`. If present: prove the host's processes gone (`seedDescendantsGone` with the same child rule); on success remove the database, its `-wal`/`-shm` files and the obligation, and seed afresh; on failure refuse the wake with `ErrSeedUnproven`. The obligation is a file, so it survives an agent restart. |
| +3. **Successful seed, flag on (plan 6).** On every successful completion, including a plain nil `Wait`, the seed's role child is killed and proven empty before `RemoveChild` (a helper that redirected its output and outlived the seed is killed). Flag off, a successful seed's helper stays in the host cgroup and is caught by the normal-stop proof. |
| +4. **Normal stop, flag off.** A flat or foreign exec row's terminal stop proves the host cgroup empty before `finishStop` releases the lease: `EmptyHost` reads `cgroup.events` and kills only when populated, so a clean sleep leaves no kill mark. `ErrNoDelegation` keeps the leader-only proof. Nothing runs under `a.mu`. |
| +5. **In-flight sidecar start (plan 5).** A terminal stop with a sidecar start still in flight past `stop()`'s wait keeps the stop pending, with the lease, until that start returns; its child, if any, is stopped, and only then does the host proof run. |
| |
| -Tests (write first, watch fail): a failing seed on `delegatedBackend` whose cleanup cannot be proven returns `ErrSeedUnproven` and keeps the database; `TestExecBackendSeedFailureRemovesHalfBuiltDB` stays green; flag-off idle stop with a failing `EmptyHost` double keeps the lease (`lastRep().isClosed() == false`), a passing double releases it after the proof (order asserted with `setOnClose`), `ErrNoDelegation` releases at once; the pending-stop path the same. |
| +Tests (write first, watch fail): |
| +- a failing seed on `delegatedBackend` whose cleanup cannot be proven returns `ErrSeedUnproven`, keeps the database and writes the obligation; `TestExecBackendSeedFailureRemovesHalfBuiltDB` stays green; |
| +- **the next wake** after that: with the proof still failing it is refused and the database is not opened as warm; with the proof passing the database and obligation are removed and the seed runs again (assert the seed command ran twice); |
| +- flag on, a seed that forks a helper which redirects its output, then exits 0: the helper is killed before the seed returns (fake cgroup double records `KillAndEmpty` of `role-seed`); |
| +- flag-off idle stop with a failing `EmptyHost` double keeps the lease (`lastRep().isClosed() == false`); a passing double releases it after the proof (order asserted with `setOnClose`); `ErrNoDelegation` releases at once; the pending-stop path the same; |
| +- a sidecar start delayed beyond `stop()`'s wait keeps the lease until it returns, and its child is stopped before the release. |
| |
| -- [ ] Steps: failing tests; implement; `TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -count=1` unsandboxed; `GOOS=windows go build ./...`; commit `activator: a failed seed and a flag-off stop prove their processes gone before the lease goes`. |
| +- [ ] Steps: failing tests; implement; unsandboxed `TMPDIR=/tmp/k nice -n 19 go test ./internal/activator/ -count=1`; `GOOS=windows go build ./...`; commit `activator: seeds and flag-off stops prove their processes gone before the lease goes`. |
| |
| ### Task 9: Spec rev-18 amendment and roll notes |
| |
| @@ -278,9 +307,9 @@ Model: sonnet. Recon: `recon-D-retry-console.md` §3. |
| - Create: `docs/superpowers/specs/2026-10-12-zero-downtime-deploys-rev18-amendment.md` |
| - Modify: `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` ("Still owed (code)", "Roll notes from zdd-flag-readiness", "Real-box checks") |
| |
| -The amendment states, for Paul's sign-off: the status line for rev 18; replacements for spec lines 3 (rev-14 sentence), 704–705 (§8.4 `carlos instances retry`) and 836–837 (§11 "A retry is a re-signed channel pointer"); a new §11 paragraph with the grant as built (`HostRetryGrant` fields, root-signed envelope and the scoped-key refusal, the 24 h TTL as a backstop and the 1-minute skew, book identity plus epoch, `PublishedEpoch`'s durable-before-use rule and the generation rule from Task 1 with the proof sketch, unsigned or stale grants dropped with a log line, 503 with no signing key, declined boxes honour grants with no canary, grants ride `host-retry.json`, the console checks `retry_v`), and Paul's Q4 ruling (2026-10-11) and hard-limit ruling (2026-10-12) by date. `carlos promote --retry` stays the app-wide re-signed pointer (lines 761–762 and the line-998 table row are unchanged; say so). |
| +The amendment states, for Paul's sign-off: the status line for rev 18; replacements for spec line 3 (the rev-14 sentence), lines 704–705 (§8.4 `carlos instances retry`) and lines 836–837 (§11 "A retry is a re-signed channel pointer"); a new §11 paragraph with the grant as built (`HostRetryGrant` fields; the root-signed envelope and the scoped-key refusal; the 24 h TTL as a backstop and the 1-minute skew; book identity plus the app's restriction generation; Task 1's rule and proof sketch, including the identity rotation on adoption; unsigned or stale grants dropped with a log line; 503 with no signing key; declined boxes honour grants with no canary; grants ride `host-retry.json`, legacy signed grants in `restart-scope.json` are checked the same way; the console checks `retry_v`), and Paul's Q4 ruling (2026-10-11) and hard-limit ruling (2026-10-12) by date. `carlos promote --retry` stays the app-wide re-signed pointer (lines 761–762 and the line-998 table row are unchanged; say so). |
| |
| -Roll notes: mark each closed item with its commit; replace the "reopen epoch bound" bullet with the hard rule; shrink "Retries: boxes before consoles" to the by-construction facts; list the real-box checks still owed: the kernel tests from Task 7 (sibling, recovery, live-g1 respawn) and round 1's, the failed-seed and normal-stop kernel control from Task 8 (a forked helper killed before the lease goes), the real-unit control and sidecar checks, dev rollout and retry across a mixed-version roll, and the flagship probe gates E–G. |
| +Roll notes: mark each closed item with its commit; replace the "reopen epoch bound" bullet with the hard rule; replace "Retries: boxes before consoles" with the supported transitions from Task 3; add the armed-box refusal when CLONE_INTO_CGROUP is unavailable (Task 7) and the seed obligation file (Task 8) as roll notes; list the real-box checks still owed: the kernel tests from Task 7 (sibling, recovery, live-g1 respawn) and round 1's, the seed and normal-stop kernel control from Task 8 (a forked helper killed before the lease goes), the real-unit control and sidecar checks, dev rollout and retry across a mixed-version roll, and the flagship probe gates E–G. |
| |
| - [ ] Steps: write both; check every commit hash cited with `git log --oneline`; commit `docs: rev-18 amendment for signed host retries (pending Paul); roll notes for readiness round 2`. |
| |
| @@ -288,4 +317,4 @@ Roll notes: mark each closed item with its commit; replace the "reopen epoch bou |
| |
| ## Execution order and gates |
| |
| -1, 2, 3, 4, 5, 6, 7, 8, 9. Tasks 3 and 4 share console files (sequential). Tasks 5, 6 and 8 share `activator.go` and `backend_exec.go` (sequential). After Task 9 the controller runs the one 3000-seed sweep (`mutate.sh 3000`, nice 19, GOMAXPROCS=2) and the full gate, then Astra's whole-branch review. |
| +1, 2, 3, 4, 5, 6, 7, 8, 9. Tasks 3 and 4 share console files (sequential). Tasks 5, 6 and 8 share `activator.go` and `backend_exec.go` (sequential). After Task 9 the controller runs the one 3000-seed sweep (`mutate.sh 3000`, nice 19, GOMAXPROCS=2) on the committed tip and the full gate, then Astra's whole-branch review. |
amadan