amadan
zdd-readiness: implementation plan for the flag-readiness round (#272)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+978
−0
docs/superpowers/plans/2026-10-11-zero-downtime-flag-readiness.md+978 −0
diff --git a/docs/superpowers/plans/2026-10-11-zero-downtime-flag-readiness.md b/docs/superpowers/plans/2026-10-11-zero-downtime-flag-readiness.md| new file mode 100644 |
| index 000000000..d1c30b875 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-11-zero-downtime-flag-readiness.md |
| @@ -0,0 +1,978 @@ |
| +# Zero-downtime deploys: flag-readiness Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Close the code items amadan #272 owes before `CARLOS_SWAP=overlap` may be set on any box, and turn the claims-on gaps into one documented limitation. |
| + |
| +**Architecture:** Every fix lands in the dormant zero-downtime code on `main` (anchored on origin/main `5da1cc9b`). The failure book (`internal/rollout/failures.go`) gains a persisted sequence, write-ahead intents and schema 2. The planner and scheduler honour hold-ran markers everywhere a good build could run. Declined boxes get an exec-time gate and process signed retries. Host retries become console-signed root envelopes. Claims-on boxes decline to arm. |
| + |
| +**Tech Stack:** Go 1.x, stdlib only. Tests: `go test`, the rollout simulation (`internal/rollout/sim_test.go`) and its mutation script. |
| + |
| +**Spec:** `/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md` (rev 17, binding). Inputs: `docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-5*.md`, `CODEX-FINDINGS-ZDD-IMPL-6*.md`, amadan discussion #272. |
| + |
| +## Global Constraints |
| + |
| +- **Flag off is unchanged.** With `CARLOS_SWAP` unset, behaviour stays as on origin/main, except Task 15's F6 change, which is flag-off by nature. Every new hook defaults to nil and is set only on an armed or declined box. |
| +- **Wire shapes are additive** (AGENTS.md). Restart-scope entries gain an optional `signed` field. The status rollout report gains nothing in this plan. |
| +- **The failure book file** (`<dataDir>/swap-failures.json`) is written as `"schema": 2` from Task 5 on. Readers accept 1 and 2. A schema-1 file must read unchanged. |
| +- **Lock order** (`internal/rollout/manager.go:98`): goodMu, then activator.Service.mu, then Manager.mu, then scheduler.mu, then FailureBook.mu. No disk write under activator.Service.mu or scheduler.mu. |
| +- **Copy.** No new member- or developer-facing text without a `// TODO-COPY(<id>)` comment on its line and a row in `copy-review/strings.json` (Task 17). Reuse approved text first: `internal/rolloutwords/words.go`, `rollout.HeldDownRefusal`, `host.refuseNoSwaps`, `host.refuseTexts`, the approved record `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-copy.json`. No em dashes in any new string. Agent log lines are operator text and need no review. |
| +- **Commands.** Sandboxed packages (`internal/rollout`, `internal/release`, `internal/provision`, `internal/rolloutwords`, `internal/status`): |
| + `export GOCACHE=$TMPDIR/gocache GOFLAGS=-mod=mod GOPROXY=off; nice -n 19 go test ./internal/rollout/ -run 'Name' -count=1`. |
| + Every other package (activator, host, console, cgroup, instancectl, root package) runs **unsandboxed** (`dangerouslyDisableSandbox: true`) with a short temp root: |
| + `mkdir -p /tmp/k && cd /home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-flag-readiness && TMPDIR=/tmp/k GOFLAGS='-mod=mod -buildvcs=false' GOPROXY=off nice -n 19 go test ./internal/host/ -run 'Name' -count=1`. |
| + Run tests serially. Never `go test ./...` inside a task. `gofmt -l` on touched files must be empty. |
| +- **Commits:** plain `git commit -m` in the worktree (identity is configured there). End messages with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. |
| +- **Simulation gate.** Tasks 3, 4, 5 and 7 touch scheduler or gate logic: run `nice -n 19 internal/rollout/testdata/rolloutsim/mutate.sh 300` (sandboxed exports as above) after committing, and paste its verdict lines. Task 16 adds mutants and runs the one 3000-seed sweep for the revision. |
| +- **Integrity:** reports paste real command output only. |
| + |
| +## Review Focus |
| + |
| +1. **A failed write of the failure book followed by an agent restart** (disk full, then a crash). Expected: no build that failed, and no hold build that ran, is eligible again after the restart. Task 5 pins it with `TestAFailedSaveThenRestartKeepsTheFailure`. |
| +2. **An older agent binary started over a schema-2 book.** Expected: it refuses managed starts loudly rather than erase markers. Task 5 pins the reader side with `TestASchema2BookIsUnreadableToASchema1Reader` (an old-reader stub). |
| +3. **A box with `CARLOS_HOST_CLAIM` set and `CARLOS_SWAP=overlap`.** Expected: it declines with `host-claims-on`, runs today's paths, and the deploy watch URL-watches it like any declined box. Task 1 pins it. |
| +4. **A retry entry replayed from an old log, forged unsigned by a box, or naming another build.** Expected: ignored with a log line. Task 8 pins all three. |
| +5. **A late outcome of an older wake or attempt arriving after a newer retry, marker or good write.** Expected: it never undoes the newer state. Task 4 pins the two sequences Astra gave (N6, retry ordering). |
| + |
| +## Rulings recorded by this plan |
| + |
| +- Ruling (Paul's, 2026-10-11): claims-on boxes do not get swaps. Arming declines on any box with host claims enabled (`CARLOS_HOST_CLAIM` non-empty), reason code `host-claims-on`. The four claims-on gaps from `CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md` stay open as the future feature "claims-on swaps" (#272). |
| +- Ruling: "refuse to arm" means **decline** (today's paths, failure records honoured), not `refuseManaged` (every managed wake refused). Why: refusing would make the flag an outage switch on claims boxes; declining runs nothing new. Cost if wrong: a claims box with the flag set serves on today's stop-then-start paths instead of refusing, which is the flag-off behaviour. |
| +- Ruling: host retries are signed in the console (`release.SignRootEnvelope` with the console's root key), verified on the box (`release.VerifyRootEnvelope`), and an unsigned or invalid retry is ignored on every box. The grant names account, app, host and build, and is accepted for one hour after it was issued, newest-first per host. Cost if wrong: a box replays a captured grant inside its hour after a new failure of the same build on the same host; the residual needs bucket write access and an in-window capture. |
| +- Ruling: an unresolved write-ahead intent found at startup is a failure of its build on that instance (held down in hold mode). Cost if wrong: an agent crash in the middle of a swap halts that build's cohort on the box until `carlos promote --retry` or a signed host retry. |
| +- Ruling: the failure book is always written as schema 2. Cost if wrong: downgrading an agent with the flag on leaves that box refusing managed starts until the file is upgraded or removed. |
| +- Ruling: on a declined box, an installed good build never boots over a hold-ran marker; recovery is a signed host retry or a newer build. Cost if wrong: an explicit `carlos rollback` on a declined box needs a retry as well. |
| +- Ruling: deferred (not code defects, or not on the flag-on path): round-3 §5 coverage-only items (config delivery order/removal coverage, shutdown join tests, blocked-publication tests, admitting `CoalesceRecycle` coverage, concurrency-one/two-cohort coverage), reproducible simulation replay, the stale serving/awake status race (spec §9.1 already reports a cold instance as "installed, awaiting first wake"), and every real-box check (D1, kernel suite, real-unit checks, flagship probe). |
| + |
| +--- |
| + |
| +### Task 1: Claims-on boxes decline to arm |
| + |
| +**Files:** |
| +- Modify: `internal/host/rollout.go:25-37` (code), `:142-153` (`swapArming`) |
| +- Modify: `internal/rolloutwords/words.go:83-121` (`unavailablePhrases`, `UnavailableDeclines`) |
| +- Modify: `internal/status/snapshot.go:261-262` (comment lists the decline codes) |
| +- Test: `internal/host/rollout_test.go`, `internal/rolloutwords/words_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `const rolloutHostClaims = "host-claims-on"` in package host. `rolloutwords.Unavailable("host-claims-on")` returns the approved `unavailableDeclined`, and `UnavailableDeclines("host-claims-on")` is true, so the deploy watch URL-watches a claims box exactly as a declined one. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +// internal/host/rollout_test.go |
| +func TestSwapArmingDeclinesWhenHostClaimsAreOn(t *testing.T) { |
| + resetRollout(t) |
| + var buf bytes.Buffer |
| + log.SetOutput(&buf) |
| + t.Cleanup(func() { log.SetOutput(os.Stderr) }) |
| + // No cgroups either: the claims check must come first, so the code |
| + // says claims, not delegation. |
| + eb := &activator.ExecBackend{Cfg: activator.Config{HostWrites: hostwrite.Config{Enabled: true}}} |
| + if swapArming(overlapEnv(nil), eb) { |
| + t.Fatal("armed a box with host claims on") |
| + } |
| + if got := rolloutUnarmed(); got != rolloutHostClaims { |
| + t.Fatalf("decline code %q, want %q", got, rolloutHostClaims) |
| + } |
| + if !strings.Contains(buf.String(), "REFUSING CARLOS_SWAP=overlap (host-claims-on)") { |
| + t.Fatalf("no operator line naming the claims refusal: %q", buf.String()) |
| + } |
| + r := rolloutStatusProbe() |
| + if r == nil || r.Unavailable != rolloutHostClaims { |
| + t.Fatalf("status %+v: want Unavailable %q", r, rolloutHostClaims) |
| + } |
| +} |
| + |
| +func TestSwapArmingIgnoresClaimsWithTheFlagOff(t *testing.T) { |
| + resetRollout(t) |
| + eb := &activator.ExecBackend{Cfg: activator.Config{HostWrites: hostwrite.Config{Enabled: true}}} |
| + if swapArming(noEnv, eb) || rolloutUnarmed() != "" { |
| + t.Fatal("flag off: a claims box must say nothing") |
| + } |
| +} |
| +``` |
| + |
| +```go |
| +// internal/rolloutwords/words_test.go |
| +func TestHostClaimsDeclineReadsAsADecline(t *testing.T) { |
| + if Unavailable("host-claims-on") != unavailableDeclined || !UnavailableDeclines("host-claims-on") || UnavailableRefuses("host-claims-on") { |
| + t.Fatal("host-claims-on must read and act as a decline") |
| + } |
| +} |
| +``` |
| + |
| +`overlapEnv` takes the public key the existing tests pass; pass whatever its signature needs (nil or a test key). Use the imports the file already has plus `hostwrite`. |
| + |
| +- [ ] **Step 2: Run to verify they fail** |
| + |
| +Run (unsandboxed): `TMPDIR=/tmp/k GOFLAGS='-mod=mod -buildvcs=false' GOPROXY=off nice -n 19 go test ./internal/host/ -run 'TestSwapArming' -count=1` |
| +Run (sandboxed): `nice -n 19 go test ./internal/rolloutwords/ -run TestHostClaims -count=1` |
| +Expected: FAIL (`rolloutHostClaims` undefined; the words test fails). |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +```go |
| +// internal/host/rollout.go, in the const block beside rolloutNoDelegation |
| + // The flag is on but host claims are on: claims-backed launches cannot |
| + // yet carry a swap's recovery choice, its execution evidence, the |
| + // generations file or the good build (#272, "claims-on swaps"), so the |
| + // box declines and stays on today's paths. |
| + rolloutHostClaims = "host-claims-on" |
| +``` |
| + |
| +```go |
| +// internal/host/rollout.go, swapArming, first check after rollout.Enabled |
| + if eb != nil && eb.Cfg.HostWrites.Enabled { |
| + log.Printf("agent: REFUSING %s=overlap (%s): host claims are on, and claims-backed launches cannot yet keep a swap's "+ |
| + "recovery choice, execution evidence, generations file or good build, so this box stays on today's stop-then-wake paths", |
| + rollout.SwapEnv, rolloutHostClaims) |
| + declineRollout(rolloutHostClaims) |
| + return false |
| + } |
| +``` |
| + |
| +```go |
| +// internal/rolloutwords/words.go |
| + "host-claims-on": unavailableDeclined, |
| +... |
| + case "no-cgroup-delegation", "no-generation-control", "generations-off", "host-claims-on": |
| +``` |
| + |
| +Update the comment at `internal/status/snapshot.go:261-262` to list `"host-claims-on"`. |
| + |
| +- [ ] **Step 4: Run to verify they pass**, plus the packages' existing rollout tests: `-run 'Rollout|Swap|Declin'` in host and the whole rolloutwords package. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: claims-on boxes decline CARLOS_SWAP=overlap (host-claims-on)` |
| + |
| +--- |
| + |
| +### Task 2: The deploy watch never drops a fleet or polls an unread host |
| + |
| +**Files:** |
| +- Modify: `internal/provision/provision.go:2610-2657` (`ListRecords`), and the census pin `internal/provision/plan9_b8_census_test.go:849` if `TestPlan9B8RecordWriteCensus` fails on the moved body |
| +- Modify: `internal/console/apiinstances.go:630-636` (`handleAPIInstanceList`) |
| +- Modify: `deploywatch.go:474-475` (`rolloutWatch.apply`), and `pendingDetail` (~593) only if the absent-host line needs it |
| +- Test: `internal/provision/provision_test.go`, `internal/console/rolloutfleet_test.go`, `deploywatch_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `provision.ListRecordsComplete(ctx, s release.Store, acct, app string, max int) (recs []Record, truncated bool, skipped []string, err error)`. `ListRecords` becomes a wrapper that drops `skipped`; no other caller changes. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +// internal/provision/provision_test.go |
| +func TestListRecordsCompleteNamesTheSkippedHost(t *testing.T) { |
| + ctx := context.Background() |
| + s := release.NewDirStore(t.TempDir()) // the store TestListRecords uses; match it |
| + putTestRecord(t, s, "acct", "app", "a.example.com") // use TestListRecords' own seeding helper |
| + if err := s.Put(ctx, RecordKey("acct", "app", "b.example.com"), []byte("not json"), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + recs, truncated, skipped, err := ListRecordsComplete(ctx, s, "acct", "app", 10) |
| + if err != nil || truncated || len(recs) != 1 || len(skipped) != 1 || skipped[0] != "b.example.com" { |
| + t.Fatalf("recs %d truncated %v skipped %v err %v", len(recs), truncated, skipped, err) |
| + } |
| +} |
| +``` |
| + |
| +```go |
| +// internal/console/rolloutfleet_test.go (helpers: placedRecord, writeFleetHeldBox, listRollout) |
| +// Astra plan 6 R3: an unreadable record must never drop its fleet from the |
| +// completeness check. The record's host sits on fleet/pi, whose only |
| +// snapshot is unreadable too. |
| +func TestAnUnreadableRecordKeepsEveryFleetInTheCompletenessCheck(t *testing.T) { |
| + // 1. placedRecord a readable host on the platform and a second host on fleet/pi. |
| + // 2. register fleet pi (writeFleetHeldBox or UpsertFleet, as the neighbouring tests do). |
| + // 3. overwrite the pi host's record with "{not json" at provision.RecordKey(acct, app, host). |
| + // 4. put "{not json" at fleets/pi/status/boxes/broken.json. |
| + // 5. GET the instance list: want "rollout_incomplete": true. |
| +} |
| +``` |
| + |
| +Write the test body with the file's own helpers, following `TestFleetBoxHoldShowsOnTheListPageAndRetryDoor`; the five numbered steps are the arrangement and the assertion. |
| + |
| +```go |
| +// deploywatch_test.go (helpers: scripted, sweep, noURL, testWatch) |
| +// A target missing from a complete, untruncated list is not evidence that it |
| +// uses the old path: it is never polled, and the timeout names it. |
| +func TestWatchRolloutNeverPollsATargetMissingFromACompleteList(t *testing.T) { |
| + // targets: a (on a reporting box) and b; every sweep lists only a, with a |
| + // followed report still settling, incomplete=false, truncated=false. |
| + // noURL(t) fails the test on any origin request. |
| + // Want: a timeout error naming b, and zero origin requests. |
| +} |
| +``` |
| + |
| +Model the deploywatch test on `TestWatchRolloutIncompleteJoinIsNotNoReports` (`deploywatch_test.go:708`). |
| + |
| +- [ ] **Step 2: Run to verify they fail** (provision sandboxed; console and root package unsandboxed with `TMPDIR=/tmp/k`). |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +```go |
| +// internal/provision/provision.go |
| +func ListRecords(ctx context.Context, s release.Store, acct, app string, max int) (recs []Record, truncated bool, err error) { |
| + recs, truncated, _, err = ListRecordsComplete(ctx, s, acct, app, max) |
| + return recs, truncated, err |
| +} |
| + |
| +// ListRecordsComplete is ListRecords that also names each host whose record |
| +// could not be read (a store error, a malformed body, a body that disagrees |
| +// with its path). A caller that must know whether the list is the whole app |
| +// (the deploy watch's completeness, Astra plan 6 R3) reads skipped. |
| +func ListRecordsComplete(ctx context.Context, s release.Store, acct, app string, max int) (recs []Record, truncated bool, skipped []string, err error) { |
| + // ListRecords' body moved here unchanged, with the skip recorded: |
| + // rec, err := GetRecord(ctx, s, acct, app, host) |
| + // if err != nil { |
| + // skipped = append(skipped, host) |
| + // continue |
| + // } |
| +} |
| +``` |
| + |
| +```go |
| +// internal/console/apiinstances.go, handleAPIInstanceList |
| + recs, truncated, skipped, err := provision.ListRecordsComplete(ctx, s.cfg.Releases, acct.Sqid, app, instanceListMax) |
| + ... |
| + relevant := recordFleets(recs) |
| + if len(skipped) > 0 { |
| + // A skipped record's placement is unknown, so every fleet the account |
| + // has, and the platform, counts toward completeness (nil means all). |
| + log.Printf("console: api instances list %s/%s: %d record(s) unreadable %v; every fleet counts toward the rollout join", |
| + acct.Sqid, app, len(skipped), skipped) |
| + relevant = nil |
| + } |
| + join, joinComplete := s.loadRolloutJoin(ctx, acct.Sqid, time.Now(), relevant) |
| +``` |
| + |
| +```go |
| +// deploywatch.go, rolloutWatch.apply: replace `case !ok && l.truncated:` |
| + case !ok: |
| + // Absent from this list: cut short, or a record the console could |
| + // not read. Neither says the host uses the old path, so it keeps |
| + // its mode and state and is never moved to URL watching here |
| + // (Astra plan 6 R3). |
| + h.unread = !l.truncated |
| +``` |
| + |
| +Check `pendingDetail`: an absent host on an untruncated list now has `unread` true. If `watchPendingUnread`'s approved text does not fit "missing from the list", reuse `watchPendingUnlisted` by leaving `unread` false instead, and say which in the report; do not add new text. |
| + |
| +Update `apply`'s doc comment bullets (`deploywatch.go:430-446`) to match. |
| + |
| +- [ ] **Step 4: Run to verify they pass**, plus `-run 'TestListRecords|TestPlan9B8'` (provision), `-run 'Rollout|Fleet|InstanceList'` (console), `-run 'TestWatch|TestDeployWatch'` (root). |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: an unreadable record keeps every fleet in the watch's completeness check, and is never URL-polled` |
| + |
| +--- |
| + |
| +### Task 3: Hold-ran markers honour records, workers and rollback provenance (N5, N7, ruling 205) |
| + |
| +**Files:** |
| +- Modify: `internal/rollout/failures.go:62-66` (`HoldRan` gains `At`), `:197-221` (`markLocked`/`MarkHoldRan` stamp `At`) |
| +- Modify: `internal/rollout/wakeplan.go:31-40` (`wakeInputs`), `:127-152` (`decideHoldRan`), `:254-283` (`planRow`), `:300-329` (`planCheck`) |
| +- Modify: `internal/rollout/worker.go:21-30` (`executeWorker`) |
| +- Modify: `internal/rollout/holdran_seq_test.go:58-71`, `internal/rollout/sim_test.go:~826-840` (remove the N7 exemption) |
| +- Test: `internal/rollout/holdran_test.go`, `internal/rollout/worker_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `HoldRan{Account, App, Build string; At time.Time}` (`At` is `json:"at,omitempty"`, set by the book from its clock, `time.Now` unless a test injects one). `wakeInputs.RolledBack func(good string, since time.Time) bool`. |
| + |
| +Rules (spec §6.4, §8.3, §8.4; Astra R3 N5, N7, ruling 205): |
| +1. A marked build is planned only when nothing holds it for the instance: held down refuses (as today), and **a record that `Blocks` it while the instance has a good build refuses too**, with `HeldDownRefusal(mark.Build)` (approved text; no new copy). Without a good build the §8.5 rule stands: a record alone refuses nothing. |
| +2. The installed good build boots over a marker only on explicit rollback evidence: the route's channel pointer, signature-verified (`Deps.Pointer`), names `rt.GoodVersion` with `PromotedAt` after `mark.At`. `carlos rollback` re-signs the pointer (`internal/release/release.go:1252`, `signPointer` stamps a fresh `PromotedAt`), so a member's rollback passes. Root's automatic sibling repoint (`internal/release/rollback.go:96`) does not. No pointer, an unreadable one, or an older one refuses with `HeldDownRefusal(mark.Build)`. |
| +3. A scheduled worker replacement carries no fallback while a marker of the same app stands on the host. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +// internal/rollout/holdran_test.go |
| +// N7: a marker never turns into permission to run a build a sibling's |
| +// failure record holds. The instance has a good build, so it stays down. |
| +func TestAMarkedBuildStillHonoursASiblingsFailureRecord(t *testing.T) { |
| + ctx := context.Background() |
| + h := newHarness(t, "h", "s") |
| + h.stage("v2") |
| + h.install("v2", "h", "s") |
| + h.setGood("h", "v1") // the good-build helper the file's other tests use |
| + h.markHoldRan("h", "v2") |
| + h.recordFailure("v2", "s") // a sibling's plain failure record of v2 |
| + h.sleepHost("h") |
| + h.wakeHost("h") |
| + p := h.m.Plan(ctx, h.rows.get("h")) |
| + if p.Refuse != HeldDownRefusal("v2") { |
| + t.Fatalf("plan %+v: want the hold-down refusal for v2", p) |
| + } |
| +} |
| + |
| +// Ruling 205: the installed good build boots over a marker only after a |
| +// signed rollback newer than the marker. |
| +func TestAnInstalledGoodBuildBootsOverAMarkerOnlyAfterASignedRollback(t *testing.T) { |
| + ctx := context.Background() |
| + h := newHarness(t, "h") |
| + h.setGood("h", "v1") |
| + h.markHoldRan("h", "v2") // stamped at the harness clock |
| + h.install("v1", "h") // root repointed h to its good build |
| + h.sleepHost("h") |
| + h.wakeHost("h") |
| + if p := h.m.Plan(ctx, h.rows.get("h")); p.Refuse != HeldDownRefusal("v2") { |
| + t.Fatalf("no rollback pointer: plan %+v, want refused", p) |
| + } |
| + h.advance(time.Minute) |
| + h.pointer("stable", "v1", h.clock()) |
| + if p := h.m.Plan(ctx, h.rows.get("h")); p.Refuse != "" || p.Boot.Version != "v1" { |
| + t.Fatalf("after a signed rollback: plan %+v, want v1", p) |
| + } |
| +} |
| +``` |
| + |
| +```go |
| +// internal/rollout/worker_test.go |
| +// N5: a worker replacement on a host with a hold-ran marker carries no |
| +// fallback onto the good build. |
| +func TestAWorkerReplacementCarriesNoFallbackWhileAMarkerStands(t *testing.T) { |
| + // Arrange as the file's existing fallback test does (a worker host with |
| + // good v1, an overlap adoption of v3), then mark the host hold-ran on v2 |
| + // before the pass. Want: h.act.replacements()[0].fallback == nil. |
| +} |
| +``` |
| + |
| +If the harness lacks `setGood`, `markHoldRan` or `recordFailure`, add them to `fake_test.go` as thin wrappers (`h.rows.put` with `GoodVersion`/`GoodDigest`; `h.book.MarkHoldRan`; `h.book.Record`). |
| + |
| +Change `TestASupersededHoldThatRanKeepsTheInstanceOffItsGoodBuild` (`holdran_seq_test.go:58-71`): with s's record of v2 standing and h holding good v1, h's plan must now be `Refuse == HeldDownRefusal("v2")`, and its comment must say the instance stays down until a retry or a newer build (N7). |
| + |
| +- [ ] **Step 2: Run to verify they fail**: `nice -n 19 go test ./internal/rollout/ -run 'Marker|HoldRan|Superseded|Worker' -count=1` |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +```go |
| +// wakeplan.go, decideHoldRan |
| + switch { |
| + case installed == mark.Build: |
| + if hasGood && in.Book.Blocks(acct, app, installed, host, false) { |
| + // N7: the marker stops a fallback; it never lifts a record. |
| + return activator.WakePlan{Refuse: HeldDownRefusal(mark.Build)}, nil |
| + } |
| + case hasGood && installed == rt.GoodVersion: |
| + if in.RolledBack == nil || !in.RolledBack(rt.GoodVersion, mark.At) { |
| + // Ruling 205: only a signed rollback newer than the marker is an |
| + // explicit recovery; root's sibling repoint is not. |
| + return activator.WakePlan{Refuse: HeldDownRefusal(mark.Build)}, nil |
| + } |
| + return recovery, nil |
| + case hasGood && in.Book.Blocks(acct, app, installed, host, false): |
| + return activator.WakePlan{Refuse: HeldDownRefusal(mark.Build)}, nil |
| + } |
| +``` |
| + |
| +```go |
| +// wakeplan.go, planRow: wakeInputs gains |
| + RolledBack: func(good string, since time.Time) bool { |
| + if m.d.Pointer == nil { |
| + return false |
| + } |
| + ptr, err := m.d.Pointer(ctx, acct, app, routeChannel(rt)) |
| + return err == nil && ptr != nil && ptr.Version == good && ptr.PromotedAt.After(since) |
| + }, |
| +``` |
| + |
| +```go |
| +// wakeplan.go, planCheck, the marker arm |
| + case build == mark.Build: |
| + if m.d.Book.HeldDown(acct, app, build, host) || (hasGood && m.d.Book.Blocks(acct, app, build, host, false)) { |
| + return held(build) |
| + } |
| + return nil |
| +``` |
| + |
| +```go |
| +// worker.go, executeWorker |
| + acct := release.RouteAccount(rt) |
| + if _, marked := holdRanFor(m.d.Book, acct, rt.App, rt.Host); !marked { |
| + if g := rt.Good(); g.Version != "" && g.Digest != "" && g.Version != a.Build { |
| + fallback = &release.StagedIdentity{...as today...} |
| + } |
| + } |
| +``` |
| + |
| +Rewrite `decideHoldRan`'s and `planCheck`'s doc comments to state the three rules above. |
| + |
| +In `sim_test.go`, delete the N7 exemption (`&& build != s.holdRan[h.name]` in `checkBoot`, ~840) and make `checkWakeRefused` (~826-832) expect a refusal when the marked build is held for a host with a good build. Model the rollback rule if the simulation installs the good build over a marker; otherwise say so in the report. |
| + |
| +- [ ] **Step 4: Run** the rollout package (`nice -n 19 go test ./internal/rollout/ -count=1`) and `nice -n 19 internal/rollout/testdata/rolloutsim/mutate.sh 300`; paste both. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: hold-ran markers keep records, block worker fallback, and need a signed rollback (N5, N7, ruling 205)` |
| + |
| +--- |
| + |
| +### Task 4: A book sequence orders marker clears and retries (N6, retry ordering) |
| + |
| +**Files:** |
| +- Modify: `internal/rollout/failures.go` (sequence, `HoldRan.Seq`, clears, `Record`) |
| +- Modify: `internal/rollout/manager.go:892-1003` (`served`, `writeGood`, `writeGoodLocked`, `goodRecorded`), `:515-553` (`RetryHost`), `:418-466` (`applyPointers`), `:581-587` (`newWakeToken`), `:732-741` (`Bootstrap`) |
| +- Modify: `internal/rollout/queue.go:34-50` (`dispatch.BookSeq`), `internal/rollout/scheduler.go:96-287` (`execute`, `done`), `internal/rollout/wakeplan.go:186-246, 362-477` (`Plan`, `OnGen`, `buildFailed`) |
| +- Test: `internal/rollout/failures_test.go`, new `internal/rollout/bookseq_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `func (b *FailureBook) Snapshot() uint64`: the book's sequence plus one. Taken before a generation can execute. Zero means "no snapshot" and clears nothing. |
| + - `HoldRan.Seq uint64` (`json:"seq,omitempty"`), set from the book's sequence when the marker is written. |
| + - `func (b *FailureBook) ClearHoldRanBefore(host string, snap uint64) (bool, error)`: drops the marker only if `snap != 0 && marker.Seq < snap`. |
| + - `FailureRecord.Seq uint64` (`json:"-"`): the snapshot the failing execution started under; zero when unknown. |
| + - The book's sequence is persisted (`failureFile.Seq`, `json:"seq,omitempty"`), so snapshots after a restart exceed every marker before it. |
| + - `Manager.served(host string, g activator.Gen, snap uint64)` and `writeGood(host string, g activator.Gen, snap uint64)`. |
| + |
| +Rules: |
| +1. **N6.** A good write clears only markers set before its generation's execution began. Attempts take `d.BookSeq = Book.Snapshot()` in `execute` immediately before `Act.Swap` or `executeWorker`. Planned wakes take a snapshot in `Plan` right after `newWakeToken`, kept per token (`Manager.wakeSnap map[uint64]uint64`, dropped when that token's terminal event arrives, when `forget` drops the host, and when a newer plan for the host replaces an entry with no event). `Bootstrap` passes zero. |
| +2. **Retry ordering.** `RetryHost` records a host clear at `++seq`, and `ClearOnPointer` records an app-wide clear of the build at `++seq` (in memory, `map[[4]string]uint64` keyed by account, app, build and host, `""` for app-wide; `Prune` drops entries for builds no route installs). `Record(r)` with `r.Seq != 0 && r.Seq <= clear` for its host or app-wide key began before that retry: it creates no record, leaves `Retried` alone, adds no `HeldDown`. It **still sets** a requested hold-ran marker, because the marker is a fact about the database, not a failure. |
| +3. A marker written by `markLocked` gets `Seq = ++b.seq` and `At = b.now()`. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +// internal/rollout/bookseq_test.go |
| +// N6, Astra's sequence: A's good write was decided before hold B ran on the |
| +// host; B's marker is newer, and A's write must not clear it. |
| +func TestAnOlderGoodWriteNeverClearsANewerMarker(t *testing.T) { |
| + b := &FailureBook{} |
| + snapA := b.Snapshot() |
| + if err := b.MarkHoldRan("h", HoldRan{Account: "bab", App: "blog", Build: "v2"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if cleared, _ := b.ClearHoldRanBefore("h", snapA); cleared { |
| + t.Fatal("a write decided before the marker cleared it") |
| + } |
| + if cleared, _ := b.ClearHoldRanBefore("h", b.Snapshot()); !cleared { |
| + t.Fatal("a write decided after the marker must clear it") |
| + } |
| +} |
| + |
| +func TestAZeroSnapshotClearsNothing(t *testing.T) { |
| + b := &FailureBook{} |
| + _ = b.MarkHoldRan("h", HoldRan{Account: "bab", App: "blog", Build: "v2"}) |
| + if cleared, _ := b.ClearHoldRanBefore("h", 0); cleared { |
| + t.Fatal("no snapshot must clear nothing") |
| + } |
| +} |
| + |
| +// Retry ordering: a failure of an attempt that began before the host's retry |
| +// never takes the retry back. |
| +func TestAPreRetryFailureLeavesTheRetryStanding(t *testing.T) { |
| + b := &FailureBook{} |
| + before := b.Snapshot() |
| + b.Record(FailureRecord{Account: "bab", App: "blog", Build: "v2", Host: "s", Seq: before}) |
| + if ok, _ := b.RetryHost("bab", "blog", "v2", "h"); !ok { |
| + t.Fatal("retry") |
| + } |
| + b.Record(FailureRecord{Account: "bab", App: "blog", Build: "v2", Host: "h", Seq: before, HeldDown: []string{"h"}}) |
| + if b.Blocks("bab", "blog", "v2", "h", false) || b.HeldDown("bab", "blog", "v2", "h") { |
| + t.Fatal("a pre-retry failure undid the retry") |
| + } |
| + b.Record(FailureRecord{Account: "bab", App: "blog", Build: "v2", Host: "h", Seq: b.Snapshot()}) |
| + if !b.Blocks("bab", "blog", "v2", "h", false) { |
| + t.Fatal("a failure after the retry must stand") |
| + } |
| +} |
| + |
| +func TestAPreRetryFailureNeverRecreatesAClearedRecord(t *testing.T) { |
| + b := &FailureBook{} |
| + before := b.Snapshot() |
| + b.Record(FailureRecord{Account: "bab", App: "blog", Build: "v2", Host: "s", Seq: before}) |
| + if ok, _ := b.ClearOnPointer("bab", "blog", "v2", time.Now().Add(time.Hour)); !ok { |
| + t.Fatal("clear") |
| + } |
| + b.Record(FailureRecord{Account: "bab", App: "blog", Build: "v2", Host: "t", Seq: before, MarkHoldRan: true, Mode: "hold"}) |
| + if _, held := b.Lookup("bab", "blog", "v2"); held { |
| + t.Fatal("a pre-retry failure recreated the record") |
| + } |
| + if _, marked := b.HoldRanOn("t"); !marked { |
| + t.Fatal("the marker is a fact about t's database and must still be set") |
| + } |
| +} |
| + |
| +func TestTheSequenceSurvivesAReopen(t *testing.T) { |
| + dd := t.TempDir() |
| + b, _ := OpenFailureBook(dd) |
| + _ = b.MarkHoldRan("h", HoldRan{Account: "bab", App: "blog", Build: "v2"}) |
| + b2, err := OpenFailureBook(dd) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if cleared, _ := b2.ClearHoldRanBefore("h", b2.Snapshot()); !cleared { |
| + t.Fatal("after a reopen a fresh snapshot must exceed the old marker") |
| + } |
| +} |
| +``` |
| + |
| +Add one Manager-level regression in `bookseq_test.go` for N6 through `OnGen`: plan a first wake of v1 on h (snapshot taken), mark h hold-ran on v2 through a scheduled hold attempt that is superseded (as `holdran_seq_test.go` does), then deliver v1's `GenSettled` with the first plan's token; assert `h.book.HoldRanOn("h")` still names v2. |
| + |
| +- [ ] **Step 2: Run to verify they fail**: `-run 'Snapshot|Marker|PreRetry|Sequence|OlderGood'` |
| + |
| +- [ ] **Step 3: Implement** the rules above. Key code: |
| + |
| +```go |
| +// failures.go |
| +type FailureBook struct { |
| + // ...existing fields... |
| + seq uint64 // persisted; every marker and clear takes ++seq |
| + cleared map[[4]string]uint64 // in memory: retry clears, host "" for app-wide |
| + now func() time.Time // nil: time.Now |
| +} |
| + |
| +func (b *FailureBook) Snapshot() uint64 { |
| + b.mu.Lock() |
| + defer b.mu.Unlock() |
| + return b.seq + 1 |
| +} |
| + |
| +func (b *FailureBook) preRetryLocked(r FailureRecord) bool { |
| + if r.Seq == 0 { |
| + return false |
| + } |
| + return r.Seq <= b.cleared[[4]string{r.Account, r.App, r.Build, r.Host}] || |
| + r.Seq <= b.cleared[[4]string{r.Account, r.App, r.Build, ""}] |
| +} |
| +``` |
| + |
| +In `Record`: after the marker step, `if b.preRetryLocked(r) { return false, save-or-retry }`. `markLocked` sets `h.Seq = b.bumpLocked()` and `h.At = b.nowLocked()`; compare old and new markers on `Account/App/Build` only, so re-marking the same build keeps its first `Seq` and `At`. `goodRecorded(host, ver, snap)` calls `ClearHoldRanBefore`. `RetryHost` keeps its unconditional `ClearHoldRan` (the retry is the explicit recovery). |
| + |
| +- [ ] **Step 4: Run** the rollout package and `mutate.sh 300`; paste both. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: a persisted book sequence orders marker clears and pre-retry outcomes (N6, retry ordering)` |
| + |
| +--- |
| + |
| +### Task 5: Failure state survives a failed write and a restart (F5, schema 2) |
| + |
| +**Files:** |
| +- Modify: `internal/rollout/failures.go` (intents, schema 2, reconciliation) |
| +- Modify: `internal/rollout/scheduler.go:96-129` (`execute`), `:133-287` (`done`) |
| +- Modify: `internal/rollout/wakeplan.go:186-246` (`Plan`), `:362-426` (`OnGen`), `internal/rollout/manager.go:638-661` (`dropUnownedWakes`) |
| +- Modify: `internal/host/rollout.go:315-320` (`armRollout`), `internal/host/rolloutdeclined.go:68-84` (`honourDeclinedFailures`) |
| +- Test: `internal/rollout/failures_test.go`, new `internal/rollout/intent_test.go`, `internal/host/rollout_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `type Intent struct { Account, App, Build, Mode, ID string }` (JSON keys lower-case), stored in `failureFile.Intents map[string]Intent` by host. |
| + - `func (b *FailureBook) Begin(host string, in Intent) error`: writes the intent and saves; on a save error the intent is removed from memory and the error returned, and the caller must not execute. |
| + - `func (b *FailureBook) End(host, id string) error`: removes the host's intent if its ID matches, and saves. |
| + - `func (b *FailureBook) Reconciled() int`: how many intents `OpenFailureBook` turned into records. |
| + - `func (b *FailureBook) Flush() error`: saves now (used after reconciliation). |
| + |
| +Rules (Astra R1 #5, R3 §5 item 2): |
| +1. **Write-ahead.** An adoption attempt (`d.Class == classAdopt`) calls `Begin(host, Intent{..., ID: "a" + strconv.FormatUint(d.SchedRev, 10)})` in `execute` after `resolve` and before `Swap`/`executeWorker`. A planned first wake (`plan.FirstWake && plan.Boot.Version != "" && plan.Refuse == ""`) calls `Begin(host, Intent{..., ID: "w" + strconv.FormatUint(tok, 10)})` at the end of `Plan` (no activator lock is held there). A failed `Begin` refuses: the attempt finishes `SwapAbandoned` with `Executed: false` and reason `"failure records could not be written"`; the wake is refused with the approved text `"this box cannot start instances right now: its record of failed builds cannot be read"` (the same string as `host.refuseTexts[rolloutBookUnreadable]`; share it, do not copy-edit it). |
| +2. **End** runs after the outcome's own book write (`Record`/`MarkHoldRan`), never before: attempts in `done()`, wakes on the token's terminal `OnGen` event (`GenSettled`, `GenFailed`, `GenFellBack`, `GenCancelled`), on `dropUnownedWakes` for the dropped entry's token, and in `Begin` itself when a newer wake intent replaces an older wake intent for the host. A `Begin` that finds an attempt intent (`"a..."`) for the host returns an error. |
| +3. **Reconcile at open.** Each intent left in the file becomes a failure record for its build on that host (`Cause` `release`, `Mode` the intent's, `HeldDown` and a hold-ran marker when `Mode == "hold"`, `PromotedAt` and `At` the open time, `Reason` a string `reasonWord` maps to an existing word; check `internal/rollout/events.go`). Intents are then deleted in memory and the book is marked dirty. |
| +4. **Schema 2.** `saveLocked` writes `Schema: 2`; `OpenFailureBook` accepts 1 and 2 and errors on anything else. |
| +5. `armRollout`: after `OpenFailureBook`, if `book.Reconciled() > 0`, log the count and `Flush()`; a flush error is `refuseManaged(svc, rolloutBookUnreadable)`. `honourDeclinedFailures`: the reconciled records are honoured; a flush error is logged. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +// internal/rollout/intent_test.go |
| +// A save fails by making the data dir read-only (WriteFileAtomic writes its |
| +// temp file beside the target). Running as root defeats it: the tests skip. |
| + |
| +// Review Focus 1: B's failure is held only in memory because the save |
| +// failed; the agent restarts. The intent written before B ran is still on |
| +// disk and comes back as B's failure. |
| +func TestAFailedSaveThenRestartKeepsTheFailure(t *testing.T) { |
| + dd := t.TempDir() |
| + b, _ := OpenFailureBook(dd) |
| + if err := b.Begin("h", Intent{Account: "bab", App: "blog", Build: "v2", Mode: "hold", ID: "a7"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Chmod(dd, 0o500); err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { os.Chmod(dd, 0o700) }) |
| + if _, err := b.Record(FailureRecord{Account: "bab", App: "blog", Build: "v2", Host: "h", Mode: "hold", HeldDown: []string{"h"}}); err == nil { |
| + t.Skip("the save did not fail (running as root?)") |
| + } |
| + _ = b.End("h", "a7") // fails too: the intent stays on disk |
| + os.Chmod(dd, 0o700) |
| + b2, err := OpenFailureBook(dd) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if b2.Reconciled() != 1 || !b2.HeldDown("bab", "blog", "v2", "h") { |
| + t.Fatalf("after restart: reconciled %d, held down %v", b2.Reconciled(), b2.HeldDown("bab", "blog", "v2", "h")) |
| + } |
| + if m, ok := b2.HoldRanOn("h"); !ok || m.Build != "v2" { |
| + t.Fatal("a hold intent must leave its marker") |
| + } |
| +} |
| + |
| +func TestABeginThatCannotSaveRefusesAndLeavesNothing(t *testing.T) { |
| + dd := t.TempDir() |
| + b, _ := OpenFailureBook(dd) |
| + os.Chmod(dd, 0o500) |
| + t.Cleanup(func() { os.Chmod(dd, 0o700) }) |
| + if err := b.Begin("h", Intent{Account: "bab", App: "blog", Build: "v2", Mode: "overlap", ID: "a1"}); err == nil { |
| + t.Skip("the save did not fail") |
| + } |
| + os.Chmod(dd, 0o700) |
| + b2, _ := OpenFailureBook(dd) |
| + if b2.Reconciled() != 0 { |
| + t.Fatal("a refused begin left an intent") |
| + } |
| +} |
| + |
| +func TestASchema1BookReadsAndIsWrittenAsSchema2(t *testing.T) { |
| + dd := t.TempDir() |
| + raw := `{"schema":1,"records":[{"account":"bab","app":"blog","build":"v2","host":"h","cause":"release","mode":"overlap","reason":"x","promoted_at":"2026-10-01T00:00:00Z","at":"2026-10-01T00:00:00Z"}]}` |
| + os.WriteFile(filepath.Join(dd, FailureFile), []byte(raw), 0o644) |
| + b, err := OpenFailureBook(dd) |
| + if err != nil || !b.Blocks("bab", "blog", "v2", "x", false) { |
| + t.Fatalf("schema 1: %v", err) |
| + } |
| + if err := b.Flush(); err != nil { |
| + t.Fatal(err) |
| + } |
| + out, _ := os.ReadFile(filepath.Join(dd, FailureFile)) |
| + if !strings.Contains(string(out), `"schema": 2`) { |
| + t.Fatalf("written as %s", out) |
| + } |
| +} |
| + |
| +// Review Focus 2: what an older agent sees. Its reader refused schema 2. |
| +func TestASchema2BookIsUnreadableToASchema1Reader(t *testing.T) { |
| + var f struct{ Schema int `json:"schema"` } |
| + dd := t.TempDir() |
| + b, _ := OpenFailureBook(dd) |
| + _ = b.Flush() |
| + raw, _ := os.ReadFile(filepath.Join(dd, FailureFile)) |
| + json.Unmarshal(raw, &f) |
| + if f.Schema == 1 { |
| + t.Fatal("an old reader would accept the file and drop markers and intents") |
| + } |
| +} |
| +``` |
| + |
| +Also add, in `intent_test.go`, Manager-level tests: (a) an adoption whose `Begin` fails never reaches `Act.Swap` (`h.act.snapshot()` shows no call) and is retried after the abandon deferral; (b) a first-wake plan whose `Begin` fails is refused with the shared text; (c) a planned wake's intent is ended by its `GenSettled` and by `dropUnownedWakes`. In `internal/host/rollout_test.go`: an armed box whose book holds an intent reconciles it, and a flush failure refuses managed starts (`rolloutUnavailable() == rolloutBookUnreadable`). |
| + |
| +- [ ] **Step 2: Run to verify they fail.** |
| + |
| +- [ ] **Step 3: Implement** the rules. `Begin`/`End` never run under `scheduler.mu` or `activator.Service.mu`. Reword the `dirty` comment (`failures.go:89-92`) and `Record`'s comment (`:155-163`) to say what now protects a failed save: the intent on disk. |
| + |
| +- [ ] **Step 4: Run** the rollout package, the host tests `-run 'Rollout|Declin|Book'` (unsandboxed), and `mutate.sh 300`; paste all three. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: write-ahead intents and schema 2 keep failures across a failed save and a restart (F5)` |
| + |
| +--- |
| + |
| +### Task 6: An uncertain start in hold mode is a failure (ruling 206) |
| + |
| +**Files:** |
| +- Modify: `internal/activator/backend_exec.go:~1702-1725` (the `markStartUncertain` sites; give the `inv == ""` case its own sentinel `errStartedThenExited`) |
| +- Modify: `internal/activator/swaphold.go:~190-205` (`holdResult`) |
| +- Modify: `internal/activator/wakeplan.go:~355-390` (`spawnPlanned`) |
| +- Test: `internal/activator/swaphold_test.go` or the file with `holdResult` tests; `internal/activator/wakeplan_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `var errStartedThenExited = errors.New("generation exited before it was observed")` wrapped by `markStartUncertain` for the `inv == ""` case, so `errors.Is` finds both. |
| + |
| +Rules (Astra R3 ruling 206): a nested start that root accepted and whose outcome is uncertain may have run its build. |
| +1. **Hold swap:** `holdResult` returns `SwapDown` (held down by `done()`, cohort halted) when `werr != nil`, the proc is nil and `startMayHaveRun(nil, werr)` is true. It never returns abandoned with `Executed: true` for such a start, so the abandon deferral cannot re-run the build. |
| +2. **Planned wake:** `spawnPlanned` treats an uncertain start as a failure of the boot build (`w.failed`) in hold mode always, and in overlap mode only for `errStartedThenExited`. Other uncertain overlap starts keep today's `GenCancelled{Executed: true}`. |
| + |
| +- [ ] **Step 1: Write the failing tests** against the nested fakes the existing uncertain-start tests use (search `markStartUncertain` in `internal/activator/*_test.go`): a hold swap whose nested start returns `markStartUncertain(errStartedThenExited)` yields `SwapDown` with `Executed` true; one returning another uncertain error also yields `SwapDown`; a hold planned wake with an uncertain start emits `GenFailed` (not `GenCancelled`); an overlap planned wake with `errStartedThenExited` emits `GenFellBack` when it has a fallback, and with another uncertain error still emits `GenCancelled{Executed: true}`. |
| + |
| +- [ ] **Step 2: Run to verify they fail** (unsandboxed, `TMPDIR=/tmp/k`, `-run 'Uncertain'`). |
| + |
| +- [ ] **Step 3: Implement.** |
| + |
| +- [ ] **Step 4: Run** `-run 'Uncertain|Hold|Planned|Swap'` in activator; paste. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: an uncertain nested start is a hold failure, never a re-runnable abandon (ruling 206)` |
| + |
| +--- |
| + |
| +### Task 7: Execution admission: worker gate, managed revalidation, uncaptured recycle |
| + |
| +**Files:** |
| +- Modify: `internal/activator/backend_exec.go:~2113-2160` (`StartWorkerStaged`), `internal/activator/workerbuild.go:~51-118` (`startWorker`, `runBeforeExec`), `:~160-186` (`recycleWorkerLocked`) |
| +- Modify: `internal/rollout/scheduler.go:96-103` (`execute`) |
| +- Test: `internal/activator/exec_gate_test.go`, `internal/rollout/manager_test.go` |
| + |
| +Rules (Astra R3 §5 item 4, ruling 206 "park worker launch hook"): |
| +1. **Worker gate after preparation.** `runBeforeExec` keeps only the generations-file publication. The caller's ownership gate rides the context (`withExecGate(ctx, gate)`), and `StartWorkerStaged` spawns with `b.spawnInCgroupGated(rt, build, b.launchGate(ctx, rt.Host))`, so the gate runs after cgroup preparation and before every start attempt, as the instance path does (`cgroup.go:~354/367`). |
| +2. **Managed revalidation.** `execute`: after reading the row, `if !Managed(*rt) || release.RouteAccount(*rt) != s.key.acct || rt.App != s.key.app` the attempt finishes `SwapAbandoned` with `Executed: false` and reason `"route no longer managed"`, before `resolve`. |
| +3. **Uncaptured recycle.** In `recycleWorkerLocked`, when `a.CoalesceRecycle != nil` (an armed or refusing box) and the worker has no captured build (`e.workerServing.StagedPath == ""`), it never respawns from the live link: with a live process (`stopOld != nil`) the recycle is skipped and the process keeps running; with none, the entry stays down and the agent logs once per host. Today's behaviour stays when `CoalesceRecycle` is nil. |
| + |
| +- [ ] **Step 1: Write the failing tests**: (a) in `exec_gate_test.go`, modelled on the instance tests at `:419`, a worker start whose gate starts refusing during cgroup preparation never execs (the fake cgroup's prepare hook flips the gate); (b) a worker start retry after a failed PID move re-asks the gate; (c) in `manager_test.go`, a queued host whose row becomes a unit route before dispatch gets no `Swap` call and its outcome is abandoned; (d) an armed box's recycle of a worker with no captured build makes no live-link spawn. |
| + |
| +- [ ] **Step 2: Run to verify they fail.** |
| + |
| +- [ ] **Step 3: Implement.** |
| + |
| +- [ ] **Step 4: Run** activator `-run 'Gate|Worker|Recycle'` (unsandboxed), rollout package, `mutate.sh 300`; paste. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: worker gates after cgroup preparation, managed revalidation at dispatch, no uncaptured live-link recycle` |
| + |
| +--- |
| + |
| +### Task 8: Host retries are signed by the console |
| + |
| +**Files:** |
| +- Create: `internal/release/hostretry.go`, `internal/release/hostretry_test.go` |
| +- Modify: `internal/release/restartscope.go:107-110` (`RestartScope.Signed`), `:161-175` (`ValidRestartScope`), `AppendRestartScope` if it compares scopes |
| +- Modify: `internal/console/apiretry.go:61-71` (`requestHostRetry`), `:76-122`, and the browser twin `internal/console/browserretry.go` |
| +- Modify: `internal/host/restartscope.go:~179-200` (`retryRequests` returns the signed field), `internal/host/restart.go:~838-863` (verify before `retryHost`), `internal/host/agent.go:~2296-2306` (`restartMem.pub`, `retryHost` signature) |
| +- Modify: `internal/rollout/manager.go:515-553` (`RetryHost` takes the build) |
| +- Test: `internal/console/apiretry_test.go` (or the existing retry test file), `internal/host/restart_test.go` (or the retry tests' file), `internal/rollout/manager_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + |
| +```go |
| +// internal/release/hostretry.go |
| +package release |
| + |
| +// HostRetryGrant is the console's signed `carlos instances retry <host>` |
| +// (spec §8.4, §11): a box clears a host's failure hold only on a grant the |
| +// deployment root signed, naming that account, app, host and build. |
| +type HostRetryGrant struct { |
| + V int `json:"v"` |
| + Account string `json:"account"` |
| + App string `json:"app"` |
| + Host string `json:"host"` |
| + Build string `json:"build"` |
| + IssuedAt string `json:"issued_at"` // RFC 3339, UTC, whole seconds |
| +} |
| + |
| +// HostRetryGrantTTL is how long a box accepts a grant after it was issued. |
| +const HostRetryGrantTTL = time.Hour |
| + |
| +var ErrHostRetryGrant = errors.New("host retry grant: not valid for this request") |
| + |
| +func SignHostRetry(sg Signer, acct, app, host, build string, now time.Time) (string, error) { |
| + g := HostRetryGrant{V: 1, Account: acct, App: app, Host: host, Build: build, |
| + IssuedAt: now.UTC().Truncate(time.Second).Format(time.RFC3339)} |
| + wire, err := SignRootEnvelope(sg, g) |
| + return string(wire), err |
| +} |
| + |
| +// VerifyHostRetry checks wire against the root key and the entry it rode in: |
| +// same account, app and host, issued no later than now plus a minute of skew |
| +// and no earlier than now minus HostRetryGrantTTL. |
| +func VerifyHostRetry(pub ed25519.PublicKey, wire, acct, app, host string, now time.Time) (HostRetryGrant, error) { |
| + var g HostRetryGrant |
| + if err := VerifyRootEnvelope(pub, []byte(wire), &g); err != nil { |
| + return HostRetryGrant{}, err |
| + } |
| + at, err := time.Parse(time.RFC3339, g.IssuedAt) |
| + if err != nil || g.V != 1 || g.Account != acct || g.App != app || g.Host != host || g.Build == "" || |
| + at.After(now.Add(time.Minute)) || at.Before(now.Add(-HostRetryGrantTTL)) { |
| + return HostRetryGrant{}, ErrHostRetryGrant |
| + } |
| + return g, nil |
| +} |
| +``` |
| + |
| + - `RestartScope.Signed string \`json:"signed,omitempty"\``: only a `retry` scope may carry it; `ValidRestartScope` refuses it on other kinds and above `MaxRootEnvelopeBytes`. |
| + - `Manager.RetryHost(host, build string) bool`: clears only when `build` is the build the host's queue slot wants (`h.wantBuild`) or, with no slot build, the build a record holds the host on; a mismatch clears nothing (and leaves the marker). |
| + - `restartMemory.pub ed25519.PublicKey`; `restartMemory.retryHost func(host, build string) bool`; `restartMemory.retryAt map[string]time.Time` (newest accepted `IssuedAt` per host). |
| + |
| +Rules: |
| +1. **Console.** `requestHostRetry(ctx, acct, app, host, build)` signs with `release.SignHostRetry(release.RootSigner(s.cfg.Key), ...)` and appends `RestartScope{Kind: retry, Value: host, Signed: wire}`. With `s.cfg.Key == nil` the door answers 503 and appends nothing. For its text, reuse the read-only console's existing 503 wording for promote (`internal/console/server.go:~4135`) if it reads right for a retry; otherwise add a draft with `// TODO-COPY(zdd-readiness.retry-no-key)`. |
| +2. **Box.** Every retry entry is verified with `mem.pub` against the entry's account, app and host at the box's clock. Unsigned, unverifiable, expired, not newer than `retryAt[host]`, or a build that does not match: logged (`agent: retry requested for %s, dropped: %s`) and skipped. A box with no public key drops every retry. Verified: `retryAt[host] = IssuedAt`, then `retryHost(host, g.Build)`. |
| +3. Old entries stay readable; a new box reading an old console's unsigned entry drops it. |
| + |
| +- [ ] **Step 1: Write the failing tests**: release (sign/verify round trip; wrong host, wrong app, expired, future, tampered payload, scoped signer refused); console (the API and browser doors append an entry whose `signed` verifies for the held build; a console with no key answers 503 and appends nothing); host (Review Focus 4: an unsigned entry, a replay of the same grant under a new entry id, and a grant for another build each leave `RetryHost` uncalled; a valid grant calls it once with the grant's build); rollout (`RetryHost("h", "v9")` with the slot wanting v2 clears nothing). |
| + |
| +- [ ] **Step 2: Run to verify they fail.** |
| + |
| +- [ ] **Step 3: Implement.** Keep the restart-scope id/mark logic as is. |
| + |
| +- [ ] **Step 4: Run** release, rollout (sandboxed), console and host `-run 'Retry|RestartScope'` (unsandboxed); paste. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: host retries are root-signed in the console and verified on the box (Q4, spec §11)` |
| + |
| +--- |
| + |
| +### Task 9: Declined boxes gate the actual launch and honour markers |
| + |
| +**Files:** |
| +- Modify: `internal/activator/genbackend.go:~65-114` (`launchGate`), `internal/activator/backend_exec.go:~1465-1500` (live-link spawn), `:~2113-2150` (`StartWorkerStaged`, `onto == nil`), `internal/activator/activator.go:~2641` (`convergeWorkerLocked`), `internal/activator/workerbuild.go:~160-186` (`recycleWorkerLocked` converge branch) |
| +- Modify: `internal/host/rolloutdeclined.go` (`failureHolds`: markers, the gate, activation on markers or records) |
| +- Test: `internal/host/rolloutdeclined_test.go`, `internal/activator/exec_gate_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `ExecBackend.LiveExecGate func(rt registry.Route, build string) error` (nil everywhere except a declined box with records or markers). It is asked (a) in `launchGate` when the context carries no gate, with the version read from the live link immediately before the start; (b) in `StartWorkerStaged` with `onto == nil`, with the resolved `id.Version`, before the spawn; (c) in `convergeWorkerLocked` and the converge branch of `recycleWorkerLocked` with the live link's version, **before** the old process is stopped: a refusal leaves the worker running its current build (a recycle of a dead worker respawns its captured build when it has one). |
| +- `failureHolds.heldBuild(rt)` also answers for markers (rule below), and `failureHolds.gate(rt, build) error` returns `errors.New(rollout.HeldDownRefusal(b))` or `errors.New(refuseNoSwaps)` when `build` is held for `rt`. |
| + |
| +Rules (Astra R1 #6, R2/R3 declined residuals; spec §6.4, §8.4): |
| +1. **Exec-time check.** The plan-time refusal stays; the gate repeats it at the real launch with the build actually being started. Residual, recorded in the ledger: a flat instance execs the live-link path, so a root repoint landing between the gate's read and `exec` is not caught. |
| +2. **Markers on a declined box.** With a marker `m` for the route's account and app: the marked build may start unless held down, or blocked by a record while the route has a good build; any other build a record blocks is refused; **the installed good build is refused** (a declined box has no pointer read in this path; recovery is a signed retry, Task 10, or a newer build); any other build starts. |
| +3. `honourDeclinedFailures` installs the planner and the gate when the book holds records **or** markers. |
| + |
| +- [ ] **Step 1: Write the failing tests**: (a) `TestDeclinedBoxRefusesAHeldBuildAtLaunch`: plan with the live link on the good build, repoint to the held build before the start, assert no exec (use `TestDeclinedBoxKeepsAHeldBuildOffARealActivator`, `rolloutdeclined_test.go:314`, as the template); (b) a declined worker whose live link moves to a held build is not stopped by the sweep's converge and keeps its process; (c) a declined worker that crashes with the live link on a held build respawns its captured build, or stays down with none; (d) markers: a declined box with only a marker refuses the installed good build and the marked build when a record blocks it; (e) flag off: `LiveExecGate` is nil and no leg changes (`legsAreTodays`). |
| + |
| +- [ ] **Step 2: Run to verify they fail** (unsandboxed). |
| + |
| +- [ ] **Step 3: Implement.** `LiveExecGate` is consulted under the activator's lock in (c); it must only read memory and the registry row and the link, as `heldBuild` does today. |
| + |
| +- [ ] **Step 4: Run** host `-run 'Declin|Rollout'` and activator `-run 'Gate|Worker|Converge'`; paste. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: a declined box gates the actual launch, workers included, and honours hold-ran markers` |
| + |
| +--- |
| + |
| +### Task 10: Declined boxes process retries |
| + |
| +**Files:** |
| +- Modify: `internal/host/rolloutdeclined.go` (`failureHolds.retryHost`, `failureHolds.applyPointers`; the comment that says the book is never written) |
| +- Modify: `internal/host/agent.go:~2259-2306, ~2390-2396` (`fastConvergeLoop`: set `restartMem.retryHost` for a declined box; run the pointer pass) |
| +- Test: `internal/host/rolloutdeclined_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 8's `restartMemory.retryHost func(host, build string) bool` and signed-entry verification; Task 4's `FailureBook.RetryHost`, `ClearOnPointer`, `ClearHoldRan`. |
| +- Produces: `func (h *failureHolds) retryHost(host, build string) bool` and `func (h *failureHolds) applyPointers(ctx context.Context, read func(ctx context.Context, acct, app, channel string) (*release.ChannelPointer, error), routes []registry.Route)`. |
| + |
| +Rules (Astra R2 ruling 194 "startup-only records"): |
| +1. A verified signed retry for a host clears that host from the record for `build` (`RetryHost`) and its marker (`ClearHoldRan`), and saves. |
| +2. At most every 30 s (the armed `pointerReadEvery`), for each app with a standing record, the declined box reads each followed channel's signed pointer (`release.Channel` with the agent's store and key, both already open in `fastConvergeLoop`) and clears a record whose build the pointer names with a newer `PromotedAt` (`ClearOnPointer`). A read error is logged and skipped. No store or no key: no pointer pass, logged once. |
| +3. Records and markers now change in the running process, so `heldBuild` reads the live book (it does today); no restart is needed. |
| + |
| +- [ ] **Step 1: Write the failing tests**: a declined box with a hold-down of v2 on h wakes h onto v2 after a verified signed retry for (h, v2) (planner returns today's plan); after a pointer naming v2 with a newer `PromotedAt` the record is gone; an older pointer clears nothing; a retry for another build clears nothing. |
| + |
| +- [ ] **Step 2–4:** run to fail, implement, run `-run 'Declin|Retry'` (unsandboxed); paste. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: declined boxes process signed retries and promote --retry pointers` |
| + |
| +--- |
| + |
| +### Task 11: Bounded reads and acknowledgement writes |
| + |
| +**Files:** |
| +- Modify: `internal/rollout/policy.go:~60-117` (`policyCache`) |
| +- Modify: `internal/rollout/manager.go:418-466, 555-568, 831-887, 1017-1057` (`applyPointers`, `failedUnder`, `resolve`/`stagedDemand`, `activationDone`, `configSlept`), `internal/rollout/scheduler.go:46-50, 113-117`, `internal/rollout/wakeplan.go:254-283` |
| +- Modify: `internal/host/agent.go:~1166-1173` (`configAckFunc`) |
| +- Test: `internal/rollout/policy_test.go`, `internal/rollout/manager_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `const readBound = 5 * time.Second` (rollout); `failedUnder(ctx context.Context, rt registry.Route, build string) time.Time`; `Manager.acks` (an ordered asynchronous writer: one drain goroutine, FIFO, no lock held while writing). |
| + |
| +Rules (Astra R1 Flag triage T3/T8/T10, R3 §5 items 6-7): |
| +1. **Policy cache.** Each fill takes a per-key generation before `load`; the result is stored only if no newer fill has stored since. A failed read stamps `failedAt` and is not retried for `policyRetryAfter = 5 * time.Second`; meanwhile `get` answers the last value, or hold for an app never read. Missing (an empty manifest from `appconfig.Get`) is the defaults; an error is never the defaults. |
| +2. **Bounds.** `policies.get`'s load, `Deps.Pointer` and `Deps.Demand` run under `context.WithTimeout(ctx, readBound)`. `failedUnder` takes the caller's context: the wake's in `planRow`, the attempt's in `execute`. |
| +3. **Acks.** `activationDone` and `configSlept` hand `write` to `m.acks` instead of calling it, so neither a scheduler slot (`done()` calls `configActivated` before `finish`) nor the scheduler loop waits on the store. `configAckFunc` bounds its store write with a 10 s timeout. |
| + |
| +- [ ] **Step 1: Write the failing tests**: (a) an older slow fill finishing after a newer one does not overwrite it (gate the loader on channels); (b) a failing loader is called at most once within `policyRetryAfter`; (c) a blocked ack writer does not stop `done()` from freeing the slot (the next attempt is admitted while the writer blocks); (d) a blocked pointer read in `failedUnder` returns zero once the wake's context is cancelled. |
| + |
| +- [ ] **Step 2–4:** run to fail, implement, run the rollout package and host `-run 'ConfigAck|Rollout'`; paste. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: versioned policy fills, bounded store reads, acks off the scheduler's path` |
| + |
| +--- |
| + |
| +### Task 12: Workers record clean one-shots and wait out a fallback; halts are always announced |
| + |
| +**Files:** |
| +- Modify: `internal/activator/workerswap.go:~173-192` (`SwapResult.CleanExit`), `internal/activator/activator.go:~1209-1222, ~2484-2562` (`WakeWorker` budget) |
| +- Modify: `internal/rollout/scheduler.go:181-194` (`done`), `internal/rollout/manager.go` (a good write for a clean exit), `internal/rollout/wakeplan.go:469-476` (`buildFailed` with no scheduler) |
| +- Test: `internal/activator/workerswap_test.go` (or the worker test file), `internal/rollout/worker_test.go`, `internal/rollout/wakeplan_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `activator.SwapResult.CleanExit bool`: the replacement's build exited cleanly as a one-shot (spec §6.5 success). |
| + |
| +Rules: |
| +1. **One-shot good** (spec §6.5, §8.5): `done()` for `SwapDone` with `res.CleanExit` writes the good build from `res.Gen` without `acknowledged` (the attempt still holds the host's slot, so no newer generation can have completed). Snapshot rules from Task 4 apply (`d.BookSeq`). |
| +2. **WakeWorker budget:** while the entry is a first wake, `WakeWorker`'s wait extends once to `arrival + firstWakeBudget`, as request waits do (`activator.go:~1209-1222`). |
| +3. **Halt event:** `buildFailed` with no scheduler emits `EvCohortHalted` with the approved text when `Record` reports the record new, as `cohortFailed` does. |
| + |
| +- [ ] **Step 1: Write the failing tests** for each rule (a parked one-shot replacement records good; a first-wake worker whose fallback takes 40 s returns success to `WakeWorker`; a first-wake failure on a host with no scheduler emits the halt event). |
| + |
| +- [ ] **Step 2–4:** run to fail, implement, run activator `-run 'Worker'` and the rollout package; paste. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: clean one-shot replacements record good, WakeWorker waits a fallback out, halts are announced` |
| + |
| +--- |
| + |
| +### Task 13: Fallback artifacts stay, and a hold build is published before its incumbent stops |
| + |
| +**Files:** |
| +- Modify: `internal/host/retention.go:41-46` |
| +- Modify: `internal/activator/swaphold.go:~110-135, ~245-255`, `internal/activator/generations_file.go:~151-181` |
| +- Test: `internal/host/retention_test.go`, `internal/activator/swaphold_test.go` (or the hold tests' file) |
| + |
| +Rules (Astra R1 T12, R3 §5 item 9): |
| +1. Bucket retention pins `rt.GoodVersion` beside `Version` and `PrevVersion`. |
| +2. `swapHold` registers the hold build's generation as pending (as `startWorker` does with `genPending`) and persists the generations file **before** `beginStopLocked` stops the incumbent; a failed persist abandons the swap (`Executed: false`) with the incumbent still serving. The pending entry is dropped when the hold withdraws. |
| + |
| +- [ ] **Step 1: Write the failing tests**: retention keeps a good build no route installs; a hold swap whose generations write fails leaves the incumbent running and returns abandoned; the generations file names the hold build before the incumbent's stop begins (record the order in the fakes). |
| + |
| +- [ ] **Step 2–4:** run to fail, implement, run host `-run Retention` and activator `-run 'Hold|Generations'`; paste. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: retention pins the good build; a hold build is published before its incumbent stops` |
| + |
| +--- |
| + |
| +### Task 14: The stage relay gives up, and a populated legacy parent is refused |
| + |
| +**Files:** |
| +- Modify: `internal/host/stagerelay.go` (`pendingHead` gains `since time.Time`; `relayGiveUp = time.Hour`) |
| +- Modify: `internal/cgroup/gen.go:127-146` (`ensureParent`), and its caller's error path in `internal/activator` (abandon before exec) |
| +- Test: `internal/host/stagerelay_test.go`, `internal/cgroup/gen_test.go` |
| + |
| +Rules (Astra R1 T15 and T14 L146): |
| +1. A relay head that root has neither installed nor refused for `relayGiveUp` is dropped with a log line; the next blocked pass notes it again. |
| +2. `ensureParent` reads the parent's own `cgroup.procs`; when it holds a process and no generation child exists, it returns `ErrLegacyParentPopulated` and creates no child. The generation start fails before exec (the swap abandons with `Executed: false`; a wake fails as a start failure before exec). |
| + |
| +- [ ] **Step 1: Write the failing tests** (a relay head older than `relayGiveUp` stops touching the marker; a fake cgroupfs parent with a pid in `cgroup.procs` and no `g*` child makes `ensureParent` return `ErrLegacyParentPopulated`; one with a `g1` child still passes). |
| + |
| +- [ ] **Step 2–4:** run to fail, implement, run host `-run Relay`, cgroup `-run Parent` (unsandboxed); paste. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: the stage relay gives up after an hour; a populated legacy parent refuses generation placement` |
| + |
| +--- |
| + |
| +### Task 15: F6 and leftover executable files in /run |
| + |
| +**Files:** |
| +- Modify: `internal/activator/opset.go:~244-285` (`genChildEmpty`) |
| +- Modify: `internal/instancectl/gen.go:~259-300` (`sweepGenLeftovers`) |
| +- Test: `internal/activator/opset_test.go`, `internal/instancectl/gen_test.go` |
| + |
| +Rules (Astra IMPL-3 #F6 `CODEX-FINDINGS-ZDD-IMPL-3.md:44-49`; IMPL-2-R2 `:45`): |
| +1. **F6** (flag off): when `KillGen` answers `errNoGenContainment` for a flat exec row's retained failed wake, `genChildEmpty` requires the host cgroup proven empty (`hostKiller.KillHost`) before the wake counts as stopped. **Before writing it, establish** what else can live in `carlos-exec@<host>` at that moment (co-spawned sidecar members, another live entry). If a live process of another entry can be there, use the non-killing `Cleanup.Populated()` check instead of `KillHost`, so the wake stays retained (and its lease kept) until the cgroup empties on its own. If neither is safe, stop and report BLOCKED with what you found; the item is then listed as owed. |
| +2. **/run sweep:** `sweepGenLeftovers` also walks `ExeRoot` for generation exe directories (`<iid>@<n>`, parsed with the names helpers) that have no drop-in, are not starting (`genStarting`), whose unit systemd reports inactive or failed and whose cgroup is empty, and removes them under `genMu`. Host directories (no `@`) are never touched. |
| + |
| +- [ ] **Step 1: Write the failing tests** (F6: a flag-off retained wake whose leader exited is not released while the host cgroup is populated, and is released once it is empty; sweep: an exe-only leftover is removed, a starting one and a host dir are kept). |
| + |
| +- [ ] **Step 2–4:** run to fail, implement, run activator `-run 'Unproven|Retained|GenChild'` and instancectl `-run Sweep` (unsandboxed); paste. |
| + |
| +- [ ] **Step 5: Commit** `zdd-readiness: F6 proves the host cgroup empty for a retained flag-off wake; /run exe leftovers are swept` |
| + |
| +--- |
| + |
| +### Task 16: Simulation: new invariants, new mutants, one 3000-seed sweep |
| + |
| +**Files:** |
| +- Modify: `internal/rollout/sim_test.go`, `internal/rollout/testdata/rolloutsim/mutate.sh:58-63` |
| +- Modify: the marked lines in `internal/rollout/wakeplan.go`, `failures.go`, `scheduler.go` (add `// rolloutsim:Mn` markers) |
| + |
| +Rules: |
| +1. Add mutants for this plan's invariants, each marked on exactly one line, each with the invariant that must catch it: |
| + - **M7** (N7): the marked-build `Blocks` check in `decideHoldRan` disabled (`if false && hasGood && in.Book.Blocks(`). Caught by I1 (no instance boots a build a record holds for it) now that the exemption is gone. |
| + - **M8** (N6): `ClearHoldRanBefore`'s sequence test disabled. Needs an invariant that a good write decided before a hold build ran never clears that build's marker; add it to the simulation as I5 if the simulation models good writes racing marks, otherwise make M8's kill a named unit test (rule 2). |
| + - **M9** (retry ordering): `preRetryLocked` returns false. Invariant: a retry is never undone by an outcome that began before it (I6). |
| + - **M10** (F5): a failed `Begin` does not refuse. Invariant: after an injected save failure and a simulated restart, no failed or held build boots on an instance it failed on (I7). Add a fault-injecting book path to the simulation if needed. |
| +2. If a mutant's invariant cannot be modelled in the simulation, extend `mutate.sh` so a mutant line may name its own test (`-run`) instead of `TestSimInvariants`, keep the integrity rules, and say so in the script's header. |
| +3. Run `uptime` first (back off above load 12), then once: `nice -n 19 internal/rollout/testdata/rolloutsim/mutate.sh 3000` with the sandboxed exports. Want a PASSED baseline and CAUGHT for M1 to M10. Paste the verdict lines. |
| + |
| +- [ ] Steps: add invariants and mutants, run `mutate.sh 300` until clean, commit, run the 3000 sweep, commit any fixes and the evidence in the report. |
| +- [ ] **Commit** `zdd-readiness: simulation invariants and mutants for markers, retries and intents` |
| + |
| +--- |
| + |
| +### Task 17: Roll notes, the owed list, and the copy index |
| + |
| +**Files:** |
| +- Modify: `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` ("Before Gate E: owed", the claims reply, a new "Claims-on boxes" note) |
| +- Create: `copy-review/strings.json` (only if any `TODO-COPY` marker exists on the branch) |
| + |
| +- [ ] **Step 1:** `git grep -n 'TODO-COPY'` and build `copy-review/strings.json` in the copy-review skill's format (`id`, `section`, `label`, `text`, `context`, `notes`), one row per marker, the `id` matching the marker. If there are none, write none and say so. |
| +- [ ] **Step 2:** Update the roll notes: which #272 items this branch closes (task commit refs filled in by the controller), the claims-on limitation and its reason code, the retry-signing roll order (boxes before consoles; unsigned retries are ignored), the schema-2 downgrade note, and the real-box checks unchanged (D1, the kernel suite, real-unit checks, the flagship probe). |
| +- [ ] **Step 3: Commit** `zdd-readiness: roll notes and copy index` |