| index d1c30b875..23a993036 100644 |
| --- a/docs/superpowers/plans/2026-10-11-zero-downtime-flag-readiness.md |
| +++ b/docs/superpowers/plans/2026-10-11-zero-downtime-flag-readiness.md |
| @@ -12,8 +12,8 @@ |
| |
| ## Global Constraints |
| |
| -- **Flag off is unchanged.** With `CARLOS_SWAP` unset, behaviour stays as on origin/main, except Task 15's F6 change, which is flag-off by nature. Every new hook defaults to nil and is set only on an armed or declined box. |
| -- **Wire shapes are additive** (AGENTS.md). Restart-scope entries gain an optional `signed` field. The status rollout report gains nothing in this plan. |
| +- **Flag off is unchanged.** With `CARLOS_SWAP` unset, behaviour stays as on origin/main, with exactly three authorized exceptions: Task 13's bucket retention also pins `GoodVersion` (it can only keep more, never delete more); Task 15's F6 change (flag-off by nature); Task 15's `/run` sweep (flag-independent cleanup). Every new hook defaults to nil and is set only on an armed or declined box. A changed shared helper (for example `configAckFunc`) keeps its flag-off behaviour; new behaviour goes on the armed path only. |
| +- **Wire shapes are additive** (AGENTS.md). Restart-scope entries gain an optional `signed` field. The status rollout report gains per-host `held_epoch` (Task 8) and `transient` (Task 17), both `omitempty`; the console's instance-list rollout view passes them through. |
| - **The failure book file** (`<dataDir>/swap-failures.json`) is written as `"schema": 2` from Task 5 on. Readers accept 1 and 2. A schema-1 file must read unchanged. |
| - **Lock order** (`internal/rollout/manager.go:98`): goodMu, then activator.Service.mu, then Manager.mu, then scheduler.mu, then FailureBook.mu. No disk write under activator.Service.mu or scheduler.mu. |
| - **Copy.** No new member- or developer-facing text without a `// TODO-COPY(<id>)` comment on its line and a row in `copy-review/strings.json` (Task 17). Reuse approved text first: `internal/rolloutwords/words.go`, `rollout.HeldDownRefusal`, `host.refuseNoSwaps`, `host.refuseTexts`, the approved record `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-copy.json`. No em dashes in any new string. Agent log lines are operator text and need no review. |
| @@ -38,16 +38,19 @@ |
| |
| - Ruling (Paul's, 2026-10-11): claims-on boxes do not get swaps. Arming declines on any box with host claims enabled (`CARLOS_HOST_CLAIM` non-empty), reason code `host-claims-on`. The four claims-on gaps from `CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md` stay open as the future feature "claims-on swaps" (#272). |
| - Ruling: "refuse to arm" means **decline** (today's paths, failure records honoured), not `refuseManaged` (every managed wake refused). Why: refusing would make the flag an outage switch on claims boxes; declining runs nothing new. Cost if wrong: a claims box with the flag set serves on today's stop-then-start paths instead of refusing, which is the flag-off behaviour. |
| -- Ruling: host retries are signed in the console (`release.SignRootEnvelope` with the console's root key), verified on the box (`release.VerifyRootEnvelope`), and an unsigned or invalid retry is ignored on every box. The grant names account, app, host and build, and is accepted for one hour after it was issued, newest-first per host. Cost if wrong: a box replays a captured grant inside its hour after a new failure of the same build on the same host; the residual needs bucket write access and an in-window capture. |
| +- Ruling: host retries are signed in the console (`release.SignRootEnvelope` with the console's root key), verified on the box (`release.VerifyRootEnvelope`), and an unsigned or invalid retry is ignored on every box. The grant names account, app, host, build and the **restriction epoch** the box published for that host (Task 8): any change to the host's restriction since the console read it (a new failure, a re-mark) voids the grant, and the epoch is persisted, so a replay after a restart clears nothing new. A 24-hour issue window is a backstop. Cost if wrong: a member whose retry raced a second failure has to retry again. |
| - Ruling: an unresolved write-ahead intent found at startup is a failure of its build on that instance (held down in hold mode). Cost if wrong: an agent crash in the middle of a swap halts that build's cohort on the box until `carlos promote --retry` or a signed host retry. |
| - Ruling: the failure book is always written as schema 2. Cost if wrong: downgrading an agent with the flag on leaves that box refusing managed starts until the file is upgraded or removed. |
| -- Ruling: on a declined box, an installed good build never boots over a hold-ran marker; recovery is a signed host retry or a newer build. Cost if wrong: an explicit `carlos rollback` on a declined box needs a retry as well. |
| -- Ruling: deferred (not code defects, or not on the flag-on path): round-3 §5 coverage-only items (config delivery order/removal coverage, shutdown join tests, blocked-publication tests, admitting `CoalesceRecycle` coverage, concurrency-one/two-cohort coverage), reproducible simulation replay, the stale serving/awake status race (spec §9.1 already reports a cold instance as "installed, awaiting first wake"), and every real-box check (D1, kernel suite, real-unit checks, flagship probe). |
| +- Ruling (replaces the plan-time pointer read, plan review #7): explicit rollback evidence clears the marker. The pointer pass (armed `applyPointers`, declined Task 10) clears a host's hold-ran marker when the route's channel pointer, signature-verified, names the route's installed good build with `PromotedAt` after the marker's `At`. `carlos rollback` re-signs the pointer, so it qualifies; root's automatic sibling repoint does not. While a marker stands, the good build never boots in place of another build, at plan or at exec, whatever is installed. Cost if wrong: a wake in the seconds between `carlos rollback` and the next pointer pass is refused once with the approved "roll back or retry" text. |
| +- Ruling: deferred (not code defects on the flag-on path): round-3 §5 coverage-only items (config delivery order/removal coverage, shutdown join tests, blocked-publication tests, admitting `CoalesceRecycle` coverage, concurrency-one/two-cohort coverage), reproducible simulation replay, and every real-box check (D1, kernel suite, real-unit checks, flagship probe). The status waking/stopping race is NOT deferred: Task 17. |
| |
| --- |
| |
| ### Task 1: Claims-on boxes decline to arm |
| |
| +**Corrections (plan review round 1; these override the text below):** |
| +- `overlapEnv` takes a string (`internal/host/rollout_test.go:58`): call `overlapEnv("")`. |
| + |
| **Files:** |
| - Modify: `internal/host/rollout.go:25-37` (code), `:142-153` (`swapArming`) |
| - Modify: `internal/rolloutwords/words.go:83-121` (`unavailablePhrases`, `UnavailableDeclines`) |
| @@ -149,6 +152,10 @@ Update the comment at `internal/status/snapshot.go:261-262` to list `"host-claim |
| |
| ### Task 2: The deploy watch never drops a fleet or polls an unread host |
| |
| +**Corrections (plan review round 1; these override the text below):** |
| +- Seed the provision test the way `TestListRecords` does (`CreateRecord`, `internal/provision/provision_test.go:413`); `DirStore.Put` returns two values (`internal/release/store.go:402`). There is no `putTestRecord`. |
| +- An absent host must also stop being polled if it was already in `modeURL`: the poll loop (`deploywatch.go:317`) checks only `mode` and `urlLive`. Give `watchedHost` an `absent bool`, set it in the new `case !ok:` arm (cleared whenever the host is listed again), and make the poll loop skip absent hosts. Add `TestWatchRolloutStopsPollingAURLHostThatDropsOffTheList`: a host first listed with no report on a complete join (so it is URL-watched), then missing from every later list; after the transition, zero further origin requests. |
| + |
| **Files:** |
| - Modify: `internal/provision/provision.go:2610-2657` (`ListRecords`), and the census pin `internal/provision/plan9_b8_census_test.go:849` if `TestPlan9B8RecordWriteCensus` fails on the moved body |
| - Modify: `internal/console/apiinstances.go:630-636` (`handleAPIInstanceList`) |
| @@ -268,6 +275,14 @@ Update `apply`'s doc comment bullets (`deploywatch.go:430-446`) to match. |
| |
| ### Task 3: Hold-ran markers honour records, workers and rollback provenance (N5, N7, ruling 205) |
| |
| +**Corrections (plan review round 1; these override the text below):** |
| +- **Rollback provenance moves out of the planner** (plan review #7). Drop `wakeInputs.RolledBack` and the plan-time pointer read. Instead: |
| + - `decideHoldRan`'s `installed == rt.GoodVersion` arm refuses with `HeldDownRefusal(mark.Build)` whenever a marker stands. |
| + - `planCheck`'s good-build arm refuses the good build whenever a marker stands, whatever `rt.Version` is: delete `&& rt.Version != rt.GoodVersion` (`wakeplan.go:317`). No I/O in the check. |
| + - `applyPointers` (`manager.go:418`) also runs for an app with a marker on any of its hosts (not only one with records). For each such host whose route installs its good build (`rt.Version == rt.GoodVersion`), a pointer for the route's channel naming `rt.GoodVersion` with `PromotedAt` after `mark.At` clears the marker (`ClearHoldRan`) and logs it. An installed-build change seen by `observe` on a marked host makes that app's pointer read due at once (reset `m.pointerRead[k]`). |
| + - Test: `TestASignedRollbackClearsTheMarkerAndTheGoodBuildBoots` replaces `TestAnInstalledGoodBuildBootsOverAMarkerOnlyAfterASignedRollback`: before the pointer, the plan is refused and `p.Check("v1")` (from any earlier plan) refuses; after `h.pointer("stable", "v1", later)` and `h.pass()`, the marker is gone and the plan boots v1. An older pointer clears nothing. The harness pointer is an unsigned fixture (`fake_test.go:352`); signature verification is `release.Channel`'s, already tested in `internal/release`. |
| +- The book's clock: give `FailureBook` a `now func() time.Time` (nil means `time.Now`), and set `h.book.now = h.clock` in `newHarness` so `At` follows the harness clock. |
| + |
| **Files:** |
| - Modify: `internal/rollout/failures.go:62-66` (`HoldRan` gains `At`), `:197-221` (`markLocked`/`MarkHoldRan` stamp `At`) |
| - Modify: `internal/rollout/wakeplan.go:31-40` (`wakeInputs`), `:127-152` (`decideHoldRan`), `:254-283` (`planRow`), `:300-329` (`planCheck`) |
| @@ -407,6 +422,10 @@ In `sim_test.go`, delete the N7 exemption (`&& build != s.holdRan[h.name]` in `c |
| |
| ### Task 4: A book sequence orders marker clears and retries (N6, retry ordering) |
| |
| +**Corrections (plan review round 1; these override the text below):** |
| +- **A stale outcome suppresses the whole failure transition** (plan review #3). `Record` returns a disposition, `type RecordResult int` with `Recorded`, `Merged`, `Unchanged`, `PreRetry`. `failCohort` (`queue.go:~480-500`) halts and supersedes only for `Recorded`/`Merged`; for `PreRetry` it neither halts the cohort nor supersedes attempts. Add Manager-level tests through `done()` and `OnGen`: after `RetryHost` (and, separately, after a pointer clear), a delayed failure of a pre-retry attempt and of a pre-retry wake leave the cohort unhalted, the retried host's queued adoption admitted, and no running attempt's `SchedRev` changed. |
| +- **Markers are per execution, not per build** (plan review #4). `HoldRan` gains `Exec string` (`json:"exec,omitempty"`): the execution's identity, `"a<SchedRev>"` for an attempt and `"w<token>"` for a planned wake (the same IDs Task 5's intents use). `markLocked` keeps `Seq`/`At` only when `Exec` matches the standing marker (a repeated report of one execution); a different execution of the same build gets a fresh `Seq = ++seq` and `At`. Extend the N6 regression: marker for B (exec a1), snapshot C, B runs again (exec a2, re-marked), C's good write clears nothing. |
| + |
| **Files:** |
| - Modify: `internal/rollout/failures.go` (sequence, `HoldRan.Seq`, clears, `Record`) |
| - Modify: `internal/rollout/manager.go:892-1003` (`served`, `writeGood`, `writeGoodLocked`, `goodRecorded`), `:515-553` (`RetryHost`), `:418-466` (`applyPointers`), `:581-587` (`newWakeToken`), `:732-741` (`Bootstrap`) |
| @@ -544,6 +563,11 @@ In `Record`: after the marker step, `if b.preRetryLocked(r) { return false, save |
| |
| ### Task 5: Failure state survives a failed write and a restart (F5, schema 2) |
| |
| +**Corrections (plan review round 1; these override the text below):** |
| +- **Never end an intent on `!Owns`** (plan review #1). A failed wake deletes its activator entry before its `OnGen` is delivered (`activator.go:1453`, `:1549`), so `dropUnownedWakes` must not call `End`. Instead, **every planned first wake ends with exactly one token-carrying terminal event**: in `internal/activator`, the planned-wake exits that today send no event (refused at the lease, abandoned at the exec, refused by `planCheck`) emit `GenCancelled{Executed: false, Token: plan.Token}`; only for a plan with a `Token` and only when `OnGen` is set, so flag-off is untouched. `End` runs on the token's first event that settles the planned build's fate: `GenSettled`, `GenFailed`, `GenFellBack`, `GenCancelled`, and `GenHealthy` with `FirstWake == false` (the verdict floor booted the good build and the planned build never ran, `activator/wakeplan.go:667`). `Begin`'s replacement of an older wake intent for the same host stays (the outcome fence keeps a newer plan from starting before the older wake's outcome is delivered); say so in its comment, with the fence's anchor. |
| +- **Persist the effective mode** (plan review #2). `Swap` turns an overlap attempt on a foreign-layout row into a hold (`activator/swap.go:264`, `a.foreignRow`). Add `EffectiveMode(host string, m activator.SwapMode) activator.SwapMode` to the rollout `Activator` interface (`*activator.Service` implements it with `foreignRow`; the fake returns `m` unless a test sets it), and write the intent's `Mode` from it. Check the planned-wake path for the same override and do likewise. Add `TestAForeignLayoutHoldCrashReopensHeldDown`: an overlap attempt on a foreign row writes a `hold` intent; reopening the book with that intent unresolved holds the host down and marks it. |
| +- The marker a reconciled hold intent sets carries `Exec` = the intent's ID (Task 4). |
| + |
| **Files:** |
| - Modify: `internal/rollout/failures.go` (intents, schema 2, reconciliation) |
| - Modify: `internal/rollout/scheduler.go:96-129` (`execute`), `:133-287` (`done`) |
| @@ -663,6 +687,9 @@ Also add, in `intent_test.go`, Manager-level tests: (a) an adoption whose `Begin |
| |
| ### Task 6: An uncertain start in hold mode is a failure (ruling 206) |
| |
| +**Corrections (plan review round 1; these override the text below):** |
| +- **Fallback needs termination evidence** (plan review #9). A nested start error can wrap both `errStartedThenExited` and a failed termination proof (`backend_exec.go:1709`), and `spawnPlanned` refuses a fallback when `proc == nil` (`activator/wakeplan.go:393`). So in overlap, `errStartedThenExited` is a failure of the boot build (`w.failed`), and the fallback runs **only when the generation's termination is proven** (no `genStartUnproven` in the error chain, or the activator's own proof succeeded). Otherwise the wake records the failure (`GenFailed`), serves nothing, and keeps the lease/containment as an unproven wake does today. Test both cleanup outcomes. |
| + |
| **Files:** |
| - Modify: `internal/activator/backend_exec.go:~1702-1725` (the `markStartUncertain` sites; give the `inv == ""` case its own sentinel `errStartedThenExited`) |
| - Modify: `internal/activator/swaphold.go:~190-205` (`holdResult`) |
| @@ -714,6 +741,13 @@ Rules (Astra R3 §5 item 4, ruling 206 "park worker launch hook"): |
| |
| ### Task 8: Host retries are signed by the console |
| |
| +**Corrections (plan review round 1; these override the text below):** |
| +- **Bind the grant to the restriction's epoch, not to a watermark** (plan review #13). Drop `restartMemory.retryAt`. `HostRetryGrant` gains `Epoch uint64 \`json:"epoch"\``. The epoch of a host's restriction is a persisted book sequence number: `FailureRecord` gains `Epochs map[string]uint64 \`json:"epochs,omitempty"\`` (per host: `++seq` each time `Record` changes that host's standing in the record, and the record-wide entry under key `""` each time the record is created or merged), and a marker's epoch is its `Seq` (Task 4). `func (b *FailureBook) RestrictionEpoch(acct, app, build, host string) uint64` returns the larger of the record's `""` and host entries, or the marker's `Seq` when the marker names `build`. |
| +- **Status publishes it.** `status.RolloutHost` gains `HeldEpoch uint64 \`json:"held_epoch,omitempty"\``, set beside `HeldBy` (`internal/rollout/status.go:74-93`). The console's rollout join and `retryHostBuild` (`internal/console/apiretry.go:41-58`) return `(build string, epoch uint64)`, and the door signs both. An absent epoch (an older box) is 0; the console still signs it, and the box refuses a 0 epoch. |
| +- **Which build a grant clears** (plan review #5). `Manager.RetryHost(host, build string, epoch uint64) bool` clears (a) the record of `build` for the host, when one holds the host, and (b) the host's marker, when it names `build`; each only if `RestrictionEpoch(..., build, host) == epoch`. A grant naming the marker's build B while the slot wants A clears the marker and nothing else. Drop the `wantBuild` equality rule in the body below. |
| +- The box's verification keeps the 24-hour issue window (`HostRetryGrantTTL = 24 * time.Hour`) as a backstop. |
| +- Tests replace the replay test: a grant whose epoch predates a second failure of the same build on the same host clears nothing; reopening the book and replaying a used grant clears nothing new. |
| + |
| **Files:** |
| - Create: `internal/release/hostretry.go`, `internal/release/hostretry_test.go` |
| - Modify: `internal/release/restartscope.go:107-110` (`RestartScope.Signed`), `:161-175` (`ValidRestartScope`), `AppendRestartScope` if it compares scopes |
| @@ -793,6 +827,11 @@ Rules: |
| |
| ### Task 9: Declined boxes gate the actual launch and honour markers |
| |
| +**Corrections (plan review round 1; these override the text below):** |
| +- **Every declined launch mechanism is gated against the identity it actually starts** (plan review #6): flat instance (`launchGate`), nested instance (`runExecGate`, `backend_exec.go:1682`), worker (`StartWorkerStaged`, `onto == nil`), and a claims-backed wake (after the claim snapshot resolves the live link, `claim_snapshot.go:87`, before the root request; Task 1 makes a claims box decline, so this is a declined box's path). Each passes the version it will start. |
| +- **No flat read/exec race.** When `LiveExecGate` is set, the flat live-link spawn resolves the link once (`release.ResolveLive`), gates that version, and execs the resolved staged path instead of the link. With the gate nil (every other box) the spawn is byte-for-byte today's. Drop the "Residual" sentence in rule 1. |
| +- **Markers on a declined box**: the installed good build is refused while a marker stands (as in the body); Task 10's pointer pass clears the marker on a signed rollback, as the armed box does (Task 3), so `HeldDownRefusal`'s "roll back or retry" stays true on a declined box. |
| + |
| **Files:** |
| - Modify: `internal/activator/genbackend.go:~65-114` (`launchGate`), `internal/activator/backend_exec.go:~1465-1500` (live-link spawn), `:~2113-2150` (`StartWorkerStaged`, `onto == nil`), `internal/activator/activator.go:~2641` (`convergeWorkerLocked`), `internal/activator/workerbuild.go:~160-186` (`recycleWorkerLocked` converge branch) |
| - Modify: `internal/host/rolloutdeclined.go` (`failureHolds`: markers, the gate, activation on markers or records) |
| @@ -821,6 +860,11 @@ Rules (Astra R1 #6, R2/R3 declined residuals; spec §6.4, §8.4): |
| |
| ### Task 10: Declined boxes process retries |
| |
| +**Corrections (plan review round 1; these override the text below):** |
| +- **A declined box publishes its holds, and the console lets them be retried** (plan review #5). `rolloutStatusProbe`'s declined report (`internal/host/rollout.go:481-490`) gains `Hosts` entries for each managed route `failureHolds` holds: `HeldBy` (the held build), `HeldEpoch` (`RestrictionEpoch`, Task 8), `Attempt` `status.AttemptDown` when held down or when a marker refuses the installed build, else `status.AttemptFailed`. `retryHostBuild` (`internal/console/apiretry.go:50`) accepts a report whose `Unavailable` is a decline code (`rolloutwords.UnavailableDeclines`) when the host has a `HeldBy`; it still refuses every other `Unavailable`. Add an API-to-box test: a declined box's held host gets a signed grant from the door, and the declined box's restart leg clears it. |
| +- Retry clearing follows Task 8's epoch rule (`retryHost(host, build, epoch)`). |
| +- The pointer pass also clears a marker on a signed rollback (Task 3's rule), and runs for apps with markers as well as records. |
| + |
| **Files:** |
| - Modify: `internal/host/rolloutdeclined.go` (`failureHolds.retryHost`, `failureHolds.applyPointers`; the comment that says the book is never written) |
| - Modify: `internal/host/agent.go:~2259-2306, ~2390-2396` (`fastConvergeLoop`: set `restartMem.retryHost` for a declined box; run the pointer pass) |
| @@ -845,6 +889,10 @@ Rules (Astra R2 ruling 194 "startup-only records"): |
| |
| ### Task 11: Bounded reads and acknowledgement writes |
| |
| +**Corrections (plan review round 1; these override the text below):** |
| +- **Ack order is fixed at the transition** (plan review #10). `configTrack.activate` (called under `m.mu`) hands back the ack together with a per-host sequence taken under `m.mu`; the writer (`Manager.acks`) keeps the newest pending ack per host (latest sequence wins, older ones are dropped, never written after a newer one), submission never blocks, and one drain goroutine writes. When the manager's context ends, the drain finishes the write in progress and stops. |
| +- **Flag off keeps `configAckFunc` as it is** (plan review #11): put the 10 s bound in the armed path only (the writer wraps each write in `context.WithTimeout`), not in `configAckFunc`. |
| + |
| **Files:** |
| - Modify: `internal/rollout/policy.go:~60-117` (`policyCache`) |
| - Modify: `internal/rollout/manager.go:418-466, 555-568, 831-887, 1017-1057` (`applyPointers`, `failedUnder`, `resolve`/`stagedDemand`, `activationDone`, `configSlept`), `internal/rollout/scheduler.go:46-50, 113-117`, `internal/rollout/wakeplan.go:254-283` |
| @@ -892,6 +940,10 @@ Rules: |
| |
| ### Task 13: Fallback artifacts stay, and a hold build is published before its incumbent stops |
| |
| +**Corrections (plan review round 1; these override the text below):** |
| +- **Publish outside `a.mu`** (plan review #10). `persistGenerations` takes `genMu` then `a.mu` (`generations_file.go:238`), and `swapHold`'s admit body runs under `a.mu` (`swaphold.go:107`). So: register the pending generation under `a.mu`, release it, persist, re-acquire `a.mu`, and re-run the full admission checks (ownership, newest, the entry's state) before `beginStopLocked`. Any check that fails withdraws the pending entry and abandons with `Executed: false`. |
| +- The retention change is an authorized flag-off exception (Global Constraints). |
| + |
| **Files:** |
| - Modify: `internal/host/retention.go:41-46` |
| - Modify: `internal/activator/swaphold.go:~110-135, ~245-255`, `internal/activator/generations_file.go:~151-181` |
| @@ -911,6 +963,9 @@ Rules (Astra R1 T12, R3 §5 item 9): |
| |
| ### Task 14: The stage relay gives up, and a populated legacy parent is refused |
| |
| +**Corrections (plan review round 1; these override the text below):** |
| +- **Refuse direct parent members whatever children exist** (plan review #12). `ensureParent` (`internal/cgroup/gen.go:127`) returns an error now (its signature changes; update every caller, including `StartChildGated` at `:168`). It reads the parent's own `cgroup.procs` (direct members only, not `cgroup.events`' descendant population) and returns `ErrLegacyParentPopulated` whenever that file lists a process, with or without `g*` children. Replace the "one with a `g1` child still passes" test with `TestAPopulatedParentWithAStaleEmptyGenerationIsRefused`; keep a test that an empty parent with live `g*` children passes. |
| + |
| **Files:** |
| - Modify: `internal/host/stagerelay.go` (`pendingHead` gains `since time.Time`; `relayGiveUp = time.Hour`) |
| - Modify: `internal/cgroup/gen.go:127-146` (`ensureParent`), and its caller's error path in `internal/activator` (abandon before exec) |
| @@ -959,15 +1014,37 @@ Rules: |
| - **M8** (N6): `ClearHoldRanBefore`'s sequence test disabled. Needs an invariant that a good write decided before a hold build ran never clears that build's marker; add it to the simulation as I5 if the simulation models good writes racing marks, otherwise make M8's kill a named unit test (rule 2). |
| - **M9** (retry ordering): `preRetryLocked` returns false. Invariant: a retry is never undone by an outcome that began before it (I6). |
| - **M10** (F5): a failed `Begin` does not refuse. Invariant: after an injected save failure and a simulated restart, no failed or held build boots on an instance it failed on (I7). Add a fault-injecting book path to the simulation if needed. |
| + - **M11** (plan review #3): `failCohort` halts on a `PreRetry` disposition. Invariant I6 (a retry is never undone) must cover the cohort halt, not only the book. |
| + - **M12** (plan review #4): `markLocked` keeps the old `Seq` for a different execution of the same build. Caught by I5 or by the extended N6 unit test (rule 2). |
| 2. If a mutant's invariant cannot be modelled in the simulation, extend `mutate.sh` so a mutant line may name its own test (`-run`) instead of `TestSimInvariants`, keep the integrity rules, and say so in the script's header. |
| -3. Run `uptime` first (back off above load 12), then once: `nice -n 19 internal/rollout/testdata/rolloutsim/mutate.sh 3000` with the sandboxed exports. Want a PASSED baseline and CAUGHT for M1 to M10. Paste the verdict lines. |
| +3. Run `uptime` first (back off above load 12), then once: `nice -n 19 internal/rollout/testdata/rolloutsim/mutate.sh 3000` with the sandboxed exports. Want a PASSED baseline and CAUGHT for M1 to M12. Paste the verdict lines. |
| |
| - [ ] Steps: add invariants and mutants, run `mutate.sh 300` until clean, commit, run the 3000 sweep, commit any fixes and the evidence in the report. |
| - [ ] **Commit** `zdd-readiness: simulation invariants and mutants for markers, retries and intents` |
| |
| --- |
| |
| -### Task 17: Roll notes, the owed list, and the copy index |
| +### Task 17: A host that is starting or stopping is never counted done |
| + |
| +**Files:** |
| +- Modify: `internal/status/snapshot.go` (`RolloutHost.Transient`), `internal/rollout/status.go:~40-95` (`Manager.Status`) |
| +- Modify: the console's rollout view struct and its join (`internal/console/apiinstances.go`, search `apiRollout` / the rollout view the list returns), the CLI's `apiRollout` and `classifyRollout` (`deploywatch.go:~182`) |
| +- Test: `internal/rollout/status_test.go`, `internal/console/rolloutjoin_test.go`, `deploywatch_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `status.RolloutHost.Transient bool \`json:"transient,omitempty"\``: the activator owns the host (`Owns`) but has no current generation for it (`Current` false), so its process is starting or stopping and the cached serving build says nothing settled. The console view carries it as `transient`. |
| + |
| +Rules (Astra R3 §5 item 10; plan review #8): spec §9.1 counts a **sleeping** instance as "installed, awaiting first wake"; a host caught mid-wake or mid-stop is neither asleep nor awake. |
| +1. `Manager.Status` asks `Owns` and `Current` for each host with no manager or scheduler lock held (the lock order on `Manager`), and for a transient host reports `Transient: true`, `Awake: false`, and no `Serving` from the stale cache. |
| +2. `classifyRollout` returns pending for a transient host, so the watch neither counts it done as asleep nor as swapped. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `Status` reports a host the fake activator owns with no current generation as transient with no serving build; the console list carries `"transient": true`; the watch keeps a transient host pending through a sweep and completes once it reports asleep or swapped. |
| +- [ ] **Step 2–4:** run to fail, implement, run rollout `-run Status`, console `-run Rollout`, root `-run TestWatch`; paste. |
| +- [ ] **Step 5: Commit** `zdd-readiness: a host starting or stopping is reported transient and never counted done` |
| + |
| +--- |
| + |
| +### Task 18: Roll notes, the owed list, and the copy index |
| |
| **Files:** |
| - Modify: `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` ("Before Gate E: owed", the claims reply, a new "Claims-on boxes" note) |
amadan