carlos / platform Public

branch:

Clone

git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1	# Zero-downtime deploys: flag-readiness Implementation Plan
2
3	> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5	Goal: Close the code items amadan #272 owes before `CARLOS_SWAP=overlap` may be set on any box, and turn the claims-on gaps into one documented limitation.
6
7	Architecture: Every fix lands in the dormant zero-downtime code on `main` (anchored on origin/main `5da1cc9b`). The failure book (`internal/rollout/failures.go`) gains a persisted sequence, write-ahead intents and schema 2. The planner and scheduler honour hold-ran markers everywhere a good build could run. Declined boxes get an exec-time gate and process signed retries. Host retries become console-signed root envelopes. Claims-on boxes decline to arm.
8
9	Tech Stack: Go 1.x, stdlib only. Tests: `go test`, the rollout simulation (`internal/rollout/sim_test.go`) and its mutation script.
10
11	Spec: `/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md` (rev 17, binding). Inputs: `docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-5.md`, `CODEX-FINDINGS-ZDD-IMPL-6.md`, amadan discussion #272.
12
13	## Global Constraints
14
15	- Flag off is unchanged. With `CARLOS_SWAP` unset, behaviour stays as on origin/main, with exactly four authorized exceptions: Task 13's bucket retention also pins `GoodVersion` (it can only keep more, never delete more); Task 15's F6 change (flag-off by nature); Task 15's `/run` sweep (flag-independent cleanup); Task 19's recreation of a cgroup that took `cgroup.kill` (dev-box kernel defect). Every new hook defaults to nil and is set only on an armed or declined box. A changed shared helper (for example `configAckFunc`) keeps its flag-off behaviour; new behaviour goes on the armed path only.
16	- Wire shapes are additive (AGENTS.md). Restart-scope entries gain an optional `signed` field. The status rollout report gains per-host `held_epoch` (Task 8) and `transient` (Task 17), both `omitempty`; the console's instance-list rollout view passes them through.
17	- The failure book file (`<dataDir>/swap-failures.json`) is written as `"schema": 2` from Task 5 on. Readers accept 1 and 2. A schema-1 file must read unchanged.
18	- Lock order (`internal/rollout/manager.go:98`): goodMu, then activator.Service.mu, then Manager.mu, then scheduler.mu, then FailureBook.mu. No disk write under activator.Service.mu or scheduler.mu.
19	- Copy. No new member- or developer-facing text without a `// TODO-COPY(<id>)` comment on its line and a row in `copy-review/strings.json` (Task 17). Reuse approved text first: `internal/rolloutwords/words.go`, `rollout.HeldDownRefusal`, `host.refuseNoSwaps`, `host.refuseTexts`, the approved record `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-copy.json`. No em dashes in any new string. Agent log lines are operator text and need no review.
20	- Commands. Sandboxed packages (`internal/rollout`, `internal/release`, `internal/provision`, `internal/rolloutwords`, `internal/status`):
21	`export GOCACHE=$TMPDIR/gocache GOFLAGS=-mod=mod GOPROXY=off; nice -n 19 go test ./internal/rollout/ -run 'Name' -count=1`.
22	Every other package (activator, host, console, cgroup, instancectl, root package) runs unsandboxed (`dangerouslyDisableSandbox: true`) with a short temp root:
23	`mkdir -p /tmp/k && cd /home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-flag-readiness && TMPDIR=/tmp/k GOFLAGS='-mod=mod -buildvcs=false' GOPROXY=off nice -n 19 go test ./internal/host/ -run 'Name' -count=1`.
24	Run tests serially. Never `go test ./...` inside a task. `gofmt -l` on touched files must be empty.
25	- Commits: plain `git commit -m` in the worktree (identity is configured there). End messages with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`.
26	- Simulation gate. Tasks 3, 4, 5 and 7 touch scheduler or gate logic: run `nice -n 19 internal/rollout/testdata/rolloutsim/mutate.sh 300` (sandboxed exports as above) after committing, and paste its verdict lines. Task 16 adds mutants and runs the one 3000-seed sweep for the revision.
27	- Integrity: reports paste real command output only.
28
29	## Review Focus
30
31	1. A failed write of the failure book followed by an agent restart (disk full, then a crash). Expected: no build that failed, and no hold build that ran, is eligible again after the restart. Task 5 pins it with `TestAFailedSaveThenRestartKeepsTheFailure`.
32	2. An older agent binary started over a schema-2 book. Expected: it refuses managed starts loudly rather than erase markers. Task 5 pins the reader side with `TestASchema2BookIsUnreadableToASchema1Reader` (an old-reader stub).
33	3. A box with `CARLOS_HOST_CLAIM` set and `CARLOS_SWAP=overlap`. Expected: it declines with `host-claims-on`, runs today's paths, and the deploy watch URL-watches it like any declined box. Task 1 pins it.
34	4. A retry entry replayed from an old log, forged unsigned by a box, or naming another build. Expected: ignored with a log line. Task 8 pins all three.
35	5. A late outcome of an older wake or attempt arriving after a newer retry, marker or good write. Expected: it never undoes the newer state. Task 4 pins the two sequences Astra gave (N6, retry ordering).
36
37	## Rulings recorded by this plan
38
39	- Ruling (Paul's, 2026-10-11): claims-on boxes do not get swaps. Arming declines on any box with host claims enabled (`CARLOS_HOST_CLAIM` non-empty), reason code `host-claims-on`. The four claims-on gaps from `CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md` stay open as the future feature "claims-on swaps" (#272).
40	- Ruling: "refuse to arm" means decline (today's paths, failure records honoured), not `refuseManaged` (every managed wake refused). Why: refusing would make the flag an outage switch on claims boxes; declining runs nothing new. Cost if wrong: a claims box with the flag set serves on today's stop-then-start paths instead of refusing, which is the flag-off behaviour.
41	- Ruling: host retries are signed in the console (`release.SignRootEnvelope` with the console's root key), verified on the box (`release.VerifyRootEnvelope`), and an unsigned or invalid retry is ignored on every box. The grant names account, app, host, build and the restriction epoch the box published for that host (Task 8): any change to the host's restriction since the console read it (a new failure, a re-mark) voids the grant, and the epoch is persisted, so a replay after a restart clears nothing new. A 24-hour issue window is a backstop. Cost if wrong: a member whose retry raced a second failure has to retry again.
42	- Ruling: an unresolved write-ahead intent found at startup is a failure of its build on that instance (held down in hold mode). Cost if wrong: an agent crash in the middle of a swap halts that build's cohort on the box until `carlos promote --retry` or a signed host retry.
43	- Ruling: the failure book is always written as schema 2. Cost if wrong: downgrading an agent with the flag on leaves that box refusing managed starts until the file is upgraded or removed.
44	- Ruling (replaces the plan-time pointer read, plan review #7): explicit rollback evidence clears the marker. The pointer pass (armed `applyPointers`, declined Task 10) clears a host's hold-ran marker when the route's channel pointer, signature-verified, names the route's installed good build with `PromotedAt` after the marker's `At`. `carlos rollback` re-signs the pointer, so it qualifies; root's automatic sibling repoint does not. While a marker stands, the good build never boots in place of another build, at plan or at exec, whatever is installed. Cost if wrong: a wake in the seconds between `carlos rollback` and the next pointer pass is refused once with the approved "roll back or retry" text.
45	- Ruling: deferred (not code defects on the flag-on path): round-3 §5 coverage-only items (config delivery order/removal coverage, shutdown join tests, blocked-publication tests, admitting `CoalesceRecycle` coverage, concurrency-one/two-cohort coverage), reproducible simulation replay, and every real-box check (D1, kernel suite, real-unit checks, flagship probe). The status waking/stopping race is NOT deferred: Task 17.
46
47	---
48
49	### Task 1: Claims-on boxes decline to arm
50
51	Corrections (plan review round 1; these override the text below):
52	- `overlapEnv` takes a string (`internal/host/rollout_test.go:58`): call `overlapEnv("")`.
53
54	Files:
55	- Modify: `internal/host/rollout.go:25-37` (code), `:142-153` (`swapArming`)
56	- Modify: `internal/rolloutwords/words.go:83-121` (`unavailablePhrases`, `UnavailableDeclines`)
57	- Modify: `internal/status/snapshot.go:261-262` (comment lists the decline codes)
58	- Test: `internal/host/rollout_test.go`, `internal/rolloutwords/words_test.go`
59
60	Interfaces:
61	- Produces: `const rolloutHostClaims = "host-claims-on"` in package host. `rolloutwords.Unavailable("host-claims-on")` returns the approved `unavailableDeclined`, and `UnavailableDeclines("host-claims-on")` is true, so the deploy watch URL-watches a claims box exactly as a declined one.
62
63	- [ ] Step 1: Write the failing tests
64
65	```go
66	// internal/host/rollout_test.go
67	func TestSwapArmingDeclinesWhenHostClaimsAreOn(t *testing.T) {
68	resetRollout(t)
69	var buf bytes.Buffer
70	log.SetOutput(&buf)
71	t.Cleanup(func() { log.SetOutput(os.Stderr) })
72	// No cgroups either: the claims check must come first, so the code
73	// says claims, not delegation.
74	eb := &activator.ExecBackend{Cfg: activator.Config{HostWrites: hostwrite.Config{Enabled: true}}}
75	if swapArming(overlapEnv(nil), eb) {
76	t.Fatal("armed a box with host claims on")
77	}
78	if got := rolloutUnarmed(); got != rolloutHostClaims {
79	t.Fatalf("decline code %q, want %q", got, rolloutHostClaims)
80	}
81	if !strings.Contains(buf.String(), "REFUSING CARLOS_SWAP=overlap (host-claims-on)") {
82	t.Fatalf("no operator line naming the claims refusal: %q", buf.String())
83	}
84	r := rolloutStatusProbe()
85	if r == nil \|\| r.Unavailable != rolloutHostClaims {
86	t.Fatalf("status %+v: want Unavailable %q", r, rolloutHostClaims)
87	}
88	}
89
90	func TestSwapArmingIgnoresClaimsWithTheFlagOff(t *testing.T) {
91	resetRollout(t)
92	eb := &activator.ExecBackend{Cfg: activator.Config{HostWrites: hostwrite.Config{Enabled: true}}}
93	if swapArming(noEnv, eb) \|\| rolloutUnarmed() != "" {
94	t.Fatal("flag off: a claims box must say nothing")
95	}
96	}
97	```
98
99	```go
100	// internal/rolloutwords/words_test.go
101	func TestHostClaimsDeclineReadsAsADecline(t *testing.T) {
102	if Unavailable("host-claims-on") != unavailableDeclined \|\| !UnavailableDeclines("host-claims-on") \|\| UnavailableRefuses("host-claims-on") {
103	t.Fatal("host-claims-on must read and act as a decline")
104	}
105	}
106	```
107
108	`overlapEnv` takes the public key the existing tests pass; pass whatever its signature needs (nil or a test key). Use the imports the file already has plus `hostwrite`.
109
110	- [ ] Step 2: Run to verify they fail
111
112	Run (unsandboxed): `TMPDIR=/tmp/k GOFLAGS='-mod=mod -buildvcs=false' GOPROXY=off nice -n 19 go test ./internal/host/ -run 'TestSwapArming' -count=1`
113	Run (sandboxed): `nice -n 19 go test ./internal/rolloutwords/ -run TestHostClaims -count=1`
114	Expected: FAIL (`rolloutHostClaims` undefined; the words test fails).
115
116	- [ ] Step 3: Implement
117
118	```go
119	// internal/host/rollout.go, in the const block beside rolloutNoDelegation
120	// The flag is on but host claims are on: claims-backed launches cannot
121	// yet carry a swap's recovery choice, its execution evidence, the
122	// generations file or the good build (#272, "claims-on swaps"), so the
123	// box declines and stays on today's paths.
124	rolloutHostClaims = "host-claims-on"
125	```
126
127	```go
128	// internal/host/rollout.go, swapArming, first check after rollout.Enabled
129	if eb != nil && eb.Cfg.HostWrites.Enabled {
130	log.Printf("agent: REFUSING %s=overlap (%s): host claims are on, and claims-backed launches cannot yet keep a swap's "+
131	"recovery choice, execution evidence, generations file or good build, so this box stays on today's stop-then-wake paths",
132	rollout.SwapEnv, rolloutHostClaims)
133	declineRollout(rolloutHostClaims)
134	return false
135	}
136	```
137
138	```go
139	// internal/rolloutwords/words.go
140	"host-claims-on": unavailableDeclined,
141	...
142	case "no-cgroup-delegation", "no-generation-control", "generations-off", "host-claims-on":
143	```
144
145	Update the comment at `internal/status/snapshot.go:261-262` to list `"host-claims-on"`.
146
147	- [ ] Step 4: Run to verify they pass, plus the packages' existing rollout tests: `-run 'Rollout\|Swap\|Declin'` in host and the whole rolloutwords package.
148
149	- [ ] Step 5: Commit `zdd-readiness: claims-on boxes decline CARLOS_SWAP=overlap (host-claims-on)`
150
151	---
152
153	### Task 2: The deploy watch never drops a fleet or polls an unread host
154
155	Corrections (plan review round 1; these override the text below):
156	- Seed the provision test the way `TestListRecords` does (`CreateRecord`, `internal/provision/provision_test.go:413`); `DirStore.Put` returns two values (`internal/release/store.go:402`). There is no `putTestRecord`.
157	- An absent host must also stop being polled if it was already in `modeURL`: the poll loop (`deploywatch.go:317`) checks only `mode` and `urlLive`. Give `watchedHost` an `absent bool`, set it in the new `case !ok:` arm (cleared whenever the host is listed again), and make the poll loop skip absent hosts. Add `TestWatchRolloutStopsPollingAURLHostThatDropsOffTheList`: a host first listed with no report on a complete join (so it is URL-watched), then missing from every later list; after the transition, zero further origin requests.
158
159	Files:
160	- Modify: `internal/provision/provision.go:2610-2657` (`ListRecords`), and the census pin `internal/provision/plan9_b8_census_test.go:849` if `TestPlan9B8RecordWriteCensus` fails on the moved body
161	- Modify: `internal/console/apiinstances.go:630-636` (`handleAPIInstanceList`)
162	- Modify: `deploywatch.go:474-475` (`rolloutWatch.apply`), and `pendingDetail` (~593) only if the absent-host line needs it
163	- Test: `internal/provision/provision_test.go`, `internal/console/rolloutfleet_test.go`, `deploywatch_test.go`
164
165	Interfaces:
166	- Produces: `provision.ListRecordsComplete(ctx, s release.Store, acct, app string, max int) (recs []Record, truncated bool, skipped []string, err error)`. `ListRecords` becomes a wrapper that drops `skipped`; no other caller changes.
167
168	- [ ] Step 1: Write the failing tests
169
170	```go
171	// internal/provision/provision_test.go
172	func TestListRecordsCompleteNamesTheSkippedHost(t *testing.T) {
173	ctx := context.Background()
174	s := release.NewDirStore(t.TempDir()) // the store TestListRecords uses; match it
175	putTestRecord(t, s, "acct", "app", "a.example.com") // use TestListRecords' own seeding helper
176	if err := s.Put(ctx, RecordKey("acct", "app", "b.example.com"), []byte("not json"), ""); err != nil {
177	t.Fatal(err)
178	}
179	recs, truncated, skipped, err := ListRecordsComplete(ctx, s, "acct", "app", 10)
180	if err != nil \|\| truncated \|\| len(recs) != 1 \|\| len(skipped) != 1 \|\| skipped[0] != "b.example.com" {
181	t.Fatalf("recs %d truncated %v skipped %v err %v", len(recs), truncated, skipped, err)
182	}
183	}
184	```
185
186	```go
187	// internal/console/rolloutfleet_test.go (helpers: placedRecord, writeFleetHeldBox, listRollout)
188	// Astra plan 6 R3: an unreadable record must never drop its fleet from the
189	// completeness check. The record's host sits on fleet/pi, whose only
190	// snapshot is unreadable too.
191	func TestAnUnreadableRecordKeepsEveryFleetInTheCompletenessCheck(t *testing.T) {
192	// 1. placedRecord a readable host on the platform and a second host on fleet/pi.
193	// 2. register fleet pi (writeFleetHeldBox or UpsertFleet, as the neighbouring tests do).
194	// 3. overwrite the pi host's record with "{not json" at provision.RecordKey(acct, app, host).
195	// 4. put "{not json" at fleets/pi/status/boxes/broken.json.
196	// 5. GET the instance list: want "rollout_incomplete": true.
197	}
198	```
199
200	Write the test body with the file's own helpers, following `TestFleetBoxHoldShowsOnTheListPageAndRetryDoor`; the five numbered steps are the arrangement and the assertion.
201
202	```go
203	// deploywatch_test.go (helpers: scripted, sweep, noURL, testWatch)
204	// A target missing from a complete, untruncated list is not evidence that it
205	// uses the old path: it is never polled, and the timeout names it.
206	func TestWatchRolloutNeverPollsATargetMissingFromACompleteList(t *testing.T) {
207	// targets: a (on a reporting box) and b; every sweep lists only a, with a
208	// followed report still settling, incomplete=false, truncated=false.
209	// noURL(t) fails the test on any origin request.
210	// Want: a timeout error naming b, and zero origin requests.
211	}
212	```
213
214	Model the deploywatch test on `TestWatchRolloutIncompleteJoinIsNotNoReports` (`deploywatch_test.go:708`).
215
216	- [ ] Step 2: Run to verify they fail (provision sandboxed; console and root package unsandboxed with `TMPDIR=/tmp/k`).
217
218	- [ ] Step 3: Implement
219
220	```go
221	// internal/provision/provision.go
222	func ListRecords(ctx context.Context, s release.Store, acct, app string, max int) (recs []Record, truncated bool, err error) {
223	recs, truncated, _, err = ListRecordsComplete(ctx, s, acct, app, max)
224	return recs, truncated, err
225	}
226
227	// ListRecordsComplete is ListRecords that also names each host whose record
228	// could not be read (a store error, a malformed body, a body that disagrees
229	// with its path). A caller that must know whether the list is the whole app
230	// (the deploy watch's completeness, Astra plan 6 R3) reads skipped.
231	func ListRecordsComplete(ctx context.Context, s release.Store, acct, app string, max int) (recs []Record, truncated bool, skipped []string, err error) {
232	// ListRecords' body moved here unchanged, with the skip recorded:
233	// rec, err := GetRecord(ctx, s, acct, app, host)
234	// if err != nil {
235	// skipped = append(skipped, host)
236	// continue
237	// }
238	}
239	```
240
241	```go
242	// internal/console/apiinstances.go, handleAPIInstanceList
243	recs, truncated, skipped, err := provision.ListRecordsComplete(ctx, s.cfg.Releases, acct.Sqid, app, instanceListMax)
244	...
245	relevant := recordFleets(recs)
246	if len(skipped) > 0 {
247	// A skipped record's placement is unknown, so every fleet the account
248	// has, and the platform, counts toward completeness (nil means all).
249	log.Printf("console: api instances list %s/%s: %d record(s) unreadable %v; every fleet counts toward the rollout join",
250	acct.Sqid, app, len(skipped), skipped)
251	relevant = nil
252	}
253	join, joinComplete := s.loadRolloutJoin(ctx, acct.Sqid, time.Now(), relevant)
254	```
255
256	```go
257	// deploywatch.go, rolloutWatch.apply: replace `case !ok && l.truncated:`
258	case !ok:
259	// Absent from this list: cut short, or a record the console could
260	// not read. Neither says the host uses the old path, so it keeps
261	// its mode and state and is never moved to URL watching here
262	// (Astra plan 6 R3).
263	h.unread = !l.truncated
264	```
265
266	Check `pendingDetail`: an absent host on an untruncated list now has `unread` true. If `watchPendingUnread`'s approved text does not fit "missing from the list", reuse `watchPendingUnlisted` by leaving `unread` false instead, and say which in the report; do not add new text.
267
268	Update `apply`'s doc comment bullets (`deploywatch.go:430-446`) to match.
269
270	- [ ] Step 4: Run to verify they pass, plus `-run 'TestListRecords\|TestPlan9B8'` (provision), `-run 'Rollout\|Fleet\|InstanceList'` (console), `-run 'TestWatch\|TestDeployWatch'` (root).
271
272	- [ ] Step 5: Commit `zdd-readiness: an unreadable record keeps every fleet in the watch's completeness check, and is never URL-polled`
273
274	---
275
276	### Task 3: Hold-ran markers honour records, workers and rollback provenance (N5, N7, ruling 205)
277
278	Corrections (plan review round 1; these override the text below):
279	- Rollback provenance moves out of the planner (plan review #7). Drop `wakeInputs.RolledBack` and the plan-time pointer read. Instead:
280	- `decideHoldRan`'s `installed == rt.GoodVersion` arm refuses with `HeldDownRefusal(mark.Build)` whenever a marker stands.
281	- `planCheck`'s good-build arm refuses the good build whenever a marker stands, whatever `rt.Version` is: delete `&& rt.Version != rt.GoodVersion` (`wakeplan.go:317`). No I/O in the check.
282	- `applyPointers` (`manager.go:418`) also runs for an app with a marker on any of its hosts (not only one with records). For each such host whose route installs its good build (`rt.Version == rt.GoodVersion`), a pointer for the route's channel naming `rt.GoodVersion` with `PromotedAt` after `mark.At` clears the marker (`ClearHoldRan`) and logs it. An installed-build change seen by `observe` on a marked host makes that app's pointer read due at once (reset `m.pointerRead[k]`).
283	- Test: `TestASignedRollbackClearsTheMarkerAndTheGoodBuildBoots` replaces `TestAnInstalledGoodBuildBootsOverAMarkerOnlyAfterASignedRollback`: before the pointer, the plan is refused and `p.Check("v1")` (from any earlier plan) refuses; after `h.pointer("stable", "v1", later)` and `h.pass()`, the marker is gone and the plan boots v1. An older pointer clears nothing. The harness pointer is an unsigned fixture (`fake_test.go:352`); signature verification is `release.Channel`'s, already tested in `internal/release`.
284	- The book's clock: give `FailureBook` a `now func() time.Time` (nil means `time.Now`), and set `h.book.now = h.clock` in `newHarness` so `At` follows the harness clock.
285
286	Files:
287	- Modify: `internal/rollout/failures.go:62-66` (`HoldRan` gains `At`), `:197-221` (`markLocked`/`MarkHoldRan` stamp `At`)
288	- Modify: `internal/rollout/wakeplan.go:31-40` (`wakeInputs`), `:127-152` (`decideHoldRan`), `:254-283` (`planRow`), `:300-329` (`planCheck`)
289	- Modify: `internal/rollout/worker.go:21-30` (`executeWorker`)
290	- Modify: `internal/rollout/holdran_seq_test.go:58-71`, `internal/rollout/sim_test.go:~826-840` (remove the N7 exemption)
291	- Test: `internal/rollout/holdran_test.go`, `internal/rollout/worker_test.go`
292
293	Interfaces:
294	- Produces: `HoldRan{Account, App, Build string; At time.Time}` (`At` is `json:"at,omitempty"`, set by the book from its clock, `time.Now` unless a test injects one). `wakeInputs.RolledBack func(good string, since time.Time) bool`.
295
296	Rules (spec §6.4, §8.3, §8.4; Astra R3 N5, N7, ruling 205):
297	1. A marked build is planned only when nothing holds it for the instance: held down refuses (as today), and a record that `Blocks` it while the instance has a good build refuses too, with `HeldDownRefusal(mark.Build)` (approved text; no new copy). Without a good build the §8.5 rule stands: a record alone refuses nothing.
298	2. The installed good build boots over a marker only on explicit rollback evidence: the route's channel pointer, signature-verified (`Deps.Pointer`), names `rt.GoodVersion` with `PromotedAt` after `mark.At`. `carlos rollback` re-signs the pointer (`internal/release/release.go:1252`, `signPointer` stamps a fresh `PromotedAt`), so a member's rollback passes. Root's automatic sibling repoint (`internal/release/rollback.go:96`) does not. No pointer, an unreadable one, or an older one refuses with `HeldDownRefusal(mark.Build)`.
299	3. A scheduled worker replacement carries no fallback while a marker of the same app stands on the host.
300
301	- [ ] Step 1: Write the failing tests
302
303	```go
304	// internal/rollout/holdran_test.go
305	// N7: a marker never turns into permission to run a build a sibling's
306	// failure record holds. The instance has a good build, so it stays down.
307	func TestAMarkedBuildStillHonoursASiblingsFailureRecord(t *testing.T) {
308	ctx := context.Background()
309	h := newHarness(t, "h", "s")
310	h.stage("v2")
311	h.install("v2", "h", "s")
312	h.setGood("h", "v1") // the good-build helper the file's other tests use
313	h.markHoldRan("h", "v2")
314	h.recordFailure("v2", "s") // a sibling's plain failure record of v2
315	h.sleepHost("h")
316	h.wakeHost("h")
317	p := h.m.Plan(ctx, h.rows.get("h"))
318	if p.Refuse != HeldDownRefusal("v2") {
319	t.Fatalf("plan %+v: want the hold-down refusal for v2", p)
320	}
321	}
322
323	// Ruling 205: the installed good build boots over a marker only after a
324	// signed rollback newer than the marker.
325	func TestAnInstalledGoodBuildBootsOverAMarkerOnlyAfterASignedRollback(t *testing.T) {
326	ctx := context.Background()
327	h := newHarness(t, "h")
328	h.setGood("h", "v1")
329	h.markHoldRan("h", "v2") // stamped at the harness clock
330	h.install("v1", "h") // root repointed h to its good build
331	h.sleepHost("h")
332	h.wakeHost("h")
333	if p := h.m.Plan(ctx, h.rows.get("h")); p.Refuse != HeldDownRefusal("v2") {
334	t.Fatalf("no rollback pointer: plan %+v, want refused", p)
335	}
336	h.advance(time.Minute)
337	h.pointer("stable", "v1", h.clock())
338	if p := h.m.Plan(ctx, h.rows.get("h")); p.Refuse != "" \|\| p.Boot.Version != "v1" {
339	t.Fatalf("after a signed rollback: plan %+v, want v1", p)
340	}
341	}
342	```
343
344	```go
345	// internal/rollout/worker_test.go
346	// N5: a worker replacement on a host with a hold-ran marker carries no
347	// fallback onto the good build.
348	func TestAWorkerReplacementCarriesNoFallbackWhileAMarkerStands(t *testing.T) {
349	// Arrange as the file's existing fallback test does (a worker host with
350	// good v1, an overlap adoption of v3), then mark the host hold-ran on v2
351	// before the pass. Want: h.act.replacements()[0].fallback == nil.
352	}
353	```
354
355	If the harness lacks `setGood`, `markHoldRan` or `recordFailure`, add them to `fake_test.go` as thin wrappers (`h.rows.put` with `GoodVersion`/`GoodDigest`; `h.book.MarkHoldRan`; `h.book.Record`).
356
357	Change `TestASupersededHoldThatRanKeepsTheInstanceOffItsGoodBuild` (`holdran_seq_test.go:58-71`): with s's record of v2 standing and h holding good v1, h's plan must now be `Refuse == HeldDownRefusal("v2")`, and its comment must say the instance stays down until a retry or a newer build (N7).
358
359	- [ ] Step 2: Run to verify they fail: `nice -n 19 go test ./internal/rollout/ -run 'Marker\|HoldRan\|Superseded\|Worker' -count=1`
360
361	- [ ] Step 3: Implement
362
363	```go
364	// wakeplan.go, decideHoldRan
365	switch {
366	case installed == mark.Build:
367	if hasGood && in.Book.Blocks(acct, app, installed, host, false) {
368	// N7: the marker stops a fallback; it never lifts a record.
369	return activator.WakePlan{Refuse: HeldDownRefusal(mark.Build)}, nil
370	}
371	case hasGood && installed == rt.GoodVersion:
372	if in.RolledBack == nil \|\| !in.RolledBack(rt.GoodVersion, mark.At) {
373	// Ruling 205: only a signed rollback newer than the marker is an
374	// explicit recovery; root's sibling repoint is not.
375	return activator.WakePlan{Refuse: HeldDownRefusal(mark.Build)}, nil
376	}
377	return recovery, nil
378	case hasGood && in.Book.Blocks(acct, app, installed, host, false):
379	return activator.WakePlan{Refuse: HeldDownRefusal(mark.Build)}, nil
380	}
381	```
382
383	```go
384	// wakeplan.go, planRow: wakeInputs gains
385	RolledBack: func(good string, since time.Time) bool {
386	if m.d.Pointer == nil {
387	return false
388	}
389	ptr, err := m.d.Pointer(ctx, acct, app, routeChannel(rt))
390	return err == nil && ptr != nil && ptr.Version == good && ptr.PromotedAt.After(since)
391	},
392	```
393
394	```go
395	// wakeplan.go, planCheck, the marker arm
396	case build == mark.Build:
397	if m.d.Book.HeldDown(acct, app, build, host) \|\| (hasGood && m.d.Book.Blocks(acct, app, build, host, false)) {
398	return held(build)
399	}
400	return nil
401	```
402
403	```go
404	// worker.go, executeWorker
405	acct := release.RouteAccount(rt)
406	if _, marked := holdRanFor(m.d.Book, acct, rt.App, rt.Host); !marked {
407	if g := rt.Good(); g.Version != "" && g.Digest != "" && g.Version != a.Build {
408	fallback = &release.StagedIdentity{...as today...}
409	}
410	}
411	```
412
413	Rewrite `decideHoldRan`'s and `planCheck`'s doc comments to state the three rules above.
414
415	In `sim_test.go`, delete the N7 exemption (`&& build != s.holdRan[h.name]` in `checkBoot`, ~840) and make `checkWakeRefused` (~826-832) expect a refusal when the marked build is held for a host with a good build. Model the rollback rule if the simulation installs the good build over a marker; otherwise say so in the report.
416
417	- [ ] Step 4: Run the rollout package (`nice -n 19 go test ./internal/rollout/ -count=1`) and `nice -n 19 internal/rollout/testdata/rolloutsim/mutate.sh 300`; paste both.
418
419	- [ ] Step 5: Commit `zdd-readiness: hold-ran markers keep records, block worker fallback, and need a signed rollback (N5, N7, ruling 205)`
420
421	---
422
423	### Task 4: A book sequence orders marker clears and retries (N6, retry ordering)
424
425	Corrections (plan review round 1; these override the text below):
426	- A stale outcome suppresses the whole failure transition (plan review #3). `Record` returns a disposition, `type RecordResult int` with `Recorded`, `Merged`, `Unchanged`, `PreRetry`. `failCohort` (`queue.go:~480-500`) halts and supersedes only for `Recorded`/`Merged`; for `PreRetry` it neither halts the cohort nor supersedes attempts. Add Manager-level tests through `done()` and `OnGen`: after `RetryHost` (and, separately, after a pointer clear), a delayed failure of a pre-retry attempt and of a pre-retry wake leave the cohort unhalted, the retried host's queued adoption admitted, and no running attempt's `SchedRev` changed.
427	- Markers are per execution, not per build (plan review #4). `HoldRan` gains `Exec string` (`json:"exec,omitempty"`): the execution's identity, `"a<SchedRev>"` for an attempt and `"w<token>"` for a planned wake (the same IDs Task 5's intents use). `markLocked` keeps `Seq`/`At` only when `Exec` matches the standing marker (a repeated report of one execution); a different execution of the same build gets a fresh `Seq = ++seq` and `At`. Extend the N6 regression: marker for B (exec a1), snapshot C, B runs again (exec a2, re-marked), C's good write clears nothing.
428
429	Files:
430	- Modify: `internal/rollout/failures.go` (sequence, `HoldRan.Seq`, clears, `Record`)
431	- Modify: `internal/rollout/manager.go:892-1003` (`served`, `writeGood`, `writeGoodLocked`, `goodRecorded`), `:515-553` (`RetryHost`), `:418-466` (`applyPointers`), `:581-587` (`newWakeToken`), `:732-741` (`Bootstrap`)
432	- Modify: `internal/rollout/queue.go:34-50` (`dispatch.BookSeq`), `internal/rollout/scheduler.go:96-287` (`execute`, `done`), `internal/rollout/wakeplan.go:186-246, 362-477` (`Plan`, `OnGen`, `buildFailed`)
433	- Test: `internal/rollout/failures_test.go`, new `internal/rollout/bookseq_test.go`
434
435	Interfaces:
436	- Produces:
437	- `func (b *FailureBook) Snapshot() uint64`: the book's sequence plus one. Taken before a generation can execute. Zero means "no snapshot" and clears nothing.
438	- `HoldRan.Seq uint64` (`json:"seq,omitempty"`), set from the book's sequence when the marker is written.
439	- `func (b *FailureBook) ClearHoldRanBefore(host string, snap uint64) (bool, error)`: drops the marker only if `snap != 0 && marker.Seq < snap`.
440	- `FailureRecord.Seq uint64` (`json:"-"`): the snapshot the failing execution started under; zero when unknown.
441	- The book's sequence is persisted (`failureFile.Seq`, `json:"seq,omitempty"`), so snapshots after a restart exceed every marker before it.
442	- `Manager.served(host string, g activator.Gen, snap uint64)` and `writeGood(host string, g activator.Gen, snap uint64)`.
443
444	Rules:
445	1. N6. A good write clears only markers set before its generation's execution began. Attempts take `d.BookSeq = Book.Snapshot()` in `execute` immediately before `Act.Swap` or `executeWorker`. Planned wakes take a snapshot in `Plan` right after `newWakeToken`, kept per token (`Manager.wakeSnap map[uint64]uint64`, dropped when that token's terminal event arrives, when `forget` drops the host, and when a newer plan for the host replaces an entry with no event). `Bootstrap` passes zero.
446	2. Retry ordering. `RetryHost` records a host clear at `++seq`, and `ClearOnPointer` records an app-wide clear of the build at `++seq` (in memory, `map[[4]string]uint64` keyed by account, app, build and host, `""` for app-wide; `Prune` drops entries for builds no route installs). `Record(r)` with `r.Seq != 0 && r.Seq <= clear` for its host or app-wide key began before that retry: it creates no record, leaves `Retried` alone, adds no `HeldDown`. It still sets a requested hold-ran marker, because the marker is a fact about the database, not a failure.
447	3. A marker written by `markLocked` gets `Seq = ++b.seq` and `At = b.now()`.
448
449	- [ ] Step 1: Write the failing tests
450
451	```go
452	// internal/rollout/bookseq_test.go
453	// N6, Astra's sequence: A's good write was decided before hold B ran on the
454	// host; B's marker is newer, and A's write must not clear it.
455	func TestAnOlderGoodWriteNeverClearsANewerMarker(t *testing.T) {
456	b := &FailureBook{}
457	snapA := b.Snapshot()
458	if err := b.MarkHoldRan("h", HoldRan{Account: "bab", App: "blog", Build: "v2"}); err != nil {
459	t.Fatal(err)
460	}
461	if cleared, _ := b.ClearHoldRanBefore("h", snapA); cleared {
462	t.Fatal("a write decided before the marker cleared it")
463	}
464	if cleared, _ := b.ClearHoldRanBefore("h", b.Snapshot()); !cleared {
465	t.Fatal("a write decided after the marker must clear it")
466	}
467	}
468
469	func TestAZeroSnapshotClearsNothing(t *testing.T) {
470	b := &FailureBook{}
471	_ = b.MarkHoldRan("h", HoldRan{Account: "bab", App: "blog", Build: "v2"})
472	if cleared, _ := b.ClearHoldRanBefore("h", 0); cleared {
473	t.Fatal("no snapshot must clear nothing")
474	}
475	}
476
477	// Retry ordering: a failure of an attempt that began before the host's retry
478	// never takes the retry back.
479	func TestAPreRetryFailureLeavesTheRetryStanding(t *testing.T) {
480	b := &FailureBook{}
481	before := b.Snapshot()
482	b.Record(FailureRecord{Account: "bab", App: "blog", Build: "v2", Host: "s", Seq: before})
483	if ok, _ := b.RetryHost("bab", "blog", "v2", "h"); !ok {
484	t.Fatal("retry")
485	}
486	b.Record(FailureRecord{Account: "bab", App: "blog", Build: "v2", Host: "h", Seq: before, HeldDown: []string{"h"}})
487	if b.Blocks("bab", "blog", "v2", "h", false) \|\| b.HeldDown("bab", "blog", "v2", "h") {
488	t.Fatal("a pre-retry failure undid the retry")
489	}
490	b.Record(FailureRecord{Account: "bab", App: "blog", Build: "v2", Host: "h", Seq: b.Snapshot()})
491	if !b.Blocks("bab", "blog", "v2", "h", false) {
492	t.Fatal("a failure after the retry must stand")
493	}
494	}
495
496	func TestAPreRetryFailureNeverRecreatesAClearedRecord(t *testing.T) {
497	b := &FailureBook{}
498	before := b.Snapshot()
499	b.Record(FailureRecord{Account: "bab", App: "blog", Build: "v2", Host: "s", Seq: before})
500	if ok, _ := b.ClearOnPointer("bab", "blog", "v2", time.Now().Add(time.Hour)); !ok {
501	t.Fatal("clear")
502	}
503	b.Record(FailureRecord{Account: "bab", App: "blog", Build: "v2", Host: "t", Seq: before, MarkHoldRan: true, Mode: "hold"})
504	if _, held := b.Lookup("bab", "blog", "v2"); held {
505	t.Fatal("a pre-retry failure recreated the record")
506	}
507	if _, marked := b.HoldRanOn("t"); !marked {
508	t.Fatal("the marker is a fact about t's database and must still be set")
509	}
510	}
511
512	func TestTheSequenceSurvivesAReopen(t *testing.T) {
513	dd := t.TempDir()
514	b, _ := OpenFailureBook(dd)
515	_ = b.MarkHoldRan("h", HoldRan{Account: "bab", App: "blog", Build: "v2"})
516	b2, err := OpenFailureBook(dd)
517	if err != nil {
518	t.Fatal(err)
519	}
520	if cleared, _ := b2.ClearHoldRanBefore("h", b2.Snapshot()); !cleared {
521	t.Fatal("after a reopen a fresh snapshot must exceed the old marker")
522	}
523	}
524	```
525
526	Add one Manager-level regression in `bookseq_test.go` for N6 through `OnGen`: plan a first wake of v1 on h (snapshot taken), mark h hold-ran on v2 through a scheduled hold attempt that is superseded (as `holdran_seq_test.go` does), then deliver v1's `GenSettled` with the first plan's token; assert `h.book.HoldRanOn("h")` still names v2.
527
528	- [ ] Step 2: Run to verify they fail: `-run 'Snapshot\|Marker\|PreRetry\|Sequence\|OlderGood'`
529
530	- [ ] Step 3: Implement the rules above. Key code:
531
532	```go
533	// failures.go
534	type FailureBook struct {
535	// ...existing fields...
536	seq uint64 // persisted; every marker and clear takes ++seq
537	cleared map[[4]string]uint64 // in memory: retry clears, host "" for app-wide
538	now func() time.Time // nil: time.Now
539	}
540
541	func (b *FailureBook) Snapshot() uint64 {
542	b.mu.Lock()
543	defer b.mu.Unlock()
544	return b.seq + 1
545	}
546
547	func (b *FailureBook) preRetryLocked(r FailureRecord) bool {
548	if r.Seq == 0 {
549	return false
550	}
551	return r.Seq <= b.cleared[[4]string{r.Account, r.App, r.Build, r.Host}] \|\|
552	r.Seq <= b.cleared[[4]string{r.Account, r.App, r.Build, ""}]
553	}
554	```
555
556	In `Record`: after the marker step, `if b.preRetryLocked(r) { return false, save-or-retry }`. `markLocked` sets `h.Seq = b.bumpLocked()` and `h.At = b.nowLocked()`; compare old and new markers on `Account/App/Build` only, so re-marking the same build keeps its first `Seq` and `At`. `goodRecorded(host, ver, snap)` calls `ClearHoldRanBefore`. `RetryHost` keeps its unconditional `ClearHoldRan` (the retry is the explicit recovery).
557
558	- [ ] Step 4: Run the rollout package and `mutate.sh 300`; paste both.
559
560	- [ ] Step 5: Commit `zdd-readiness: a persisted book sequence orders marker clears and pre-retry outcomes (N6, retry ordering)`
561
562	---
563
564	### Task 5: Failure state survives a failed write and a restart (F5, schema 2)
565
566	Corrections (plan review round 1; these override the text below):
567	- Never end an intent on `!Owns` (plan review #1). A failed wake deletes its activator entry before its `OnGen` is delivered (`activator.go:1453`, `:1549`), so `dropUnownedWakes` must not call `End`. Instead, every planned first wake ends with exactly one token-carrying terminal event: in `internal/activator`, the planned-wake exits that today send no event (refused at the lease, abandoned at the exec, refused by `planCheck`) emit `GenCancelled{Executed: false, Token: plan.Token}`; only for a plan with a `Token` and only when `OnGen` is set, so flag-off is untouched. `End` runs on the token's first event that settles the planned build's fate: `GenSettled`, `GenFailed`, `GenFellBack`, `GenCancelled`, and `GenHealthy` with `FirstWake == false` (the verdict floor booted the good build and the planned build never ran, `activator/wakeplan.go:667`). `Begin`'s replacement of an older wake intent for the same host stays (the outcome fence keeps a newer plan from starting before the older wake's outcome is delivered); say so in its comment, with the fence's anchor.
568	- Persist the effective mode (plan review #2). `Swap` turns an overlap attempt on a foreign-layout row into a hold (`activator/swap.go:264`, `a.foreignRow`). Add `EffectiveMode(host string, m activator.SwapMode) activator.SwapMode` to the rollout `Activator` interface (`*activator.Service` implements it with `foreignRow`; the fake returns `m` unless a test sets it), and write the intent's `Mode` from it. Check the planned-wake path for the same override and do likewise. Add `TestAForeignLayoutHoldCrashReopensHeldDown`: an overlap attempt on a foreign row writes a `hold` intent; reopening the book with that intent unresolved holds the host down and marks it.
569	- The marker a reconciled hold intent sets carries `Exec` = the intent's ID (Task 4).
570
571	Files:
572	- Modify: `internal/rollout/failures.go` (intents, schema 2, reconciliation)
573	- Modify: `internal/rollout/scheduler.go:96-129` (`execute`), `:133-287` (`done`)
574	- Modify: `internal/rollout/wakeplan.go:186-246` (`Plan`), `:362-426` (`OnGen`), `internal/rollout/manager.go:638-661` (`dropUnownedWakes`)
575	- Modify: `internal/host/rollout.go:315-320` (`armRollout`), `internal/host/rolloutdeclined.go:68-84` (`honourDeclinedFailures`)
576	- Test: `internal/rollout/failures_test.go`, new `internal/rollout/intent_test.go`, `internal/host/rollout_test.go`
577
578	Interfaces:
579	- Produces:
580	- `type Intent struct { Account, App, Build, Mode, ID string }` (JSON keys lower-case), stored in `failureFile.Intents map[string]Intent` by host.
581	- `func (b *FailureBook) Begin(host string, in Intent) error`: writes the intent and saves; on a save error the intent is removed from memory and the error returned, and the caller must not execute.
582	- `func (b *FailureBook) End(host, id string) error`: removes the host's intent if its ID matches, and saves.
583	- `func (b *FailureBook) Reconciled() int`: how many intents `OpenFailureBook` turned into records.
584	- `func (b *FailureBook) Flush() error`: saves now (used after reconciliation).
585
586	Rules (Astra R1 #5, R3 §5 item 2):
587	1. Write-ahead. An adoption attempt (`d.Class == classAdopt`) calls `Begin(host, Intent{..., ID: "a" + strconv.FormatUint(d.SchedRev, 10)})` in `execute` after `resolve` and before `Swap`/`executeWorker`. A planned first wake (`plan.FirstWake && plan.Boot.Version != "" && plan.Refuse == ""`) calls `Begin(host, Intent{..., ID: "w" + strconv.FormatUint(tok, 10)})` at the end of `Plan` (no activator lock is held there). A failed `Begin` refuses: the attempt finishes `SwapAbandoned` with `Executed: false` and reason `"failure records could not be written"`; the wake is refused with the approved text `"this box cannot start instances right now: its record of failed builds cannot be read"` (the same string as `host.refuseTexts[rolloutBookUnreadable]`; share it, do not copy-edit it).
588	2. End runs after the outcome's own book write (`Record`/`MarkHoldRan`), never before: attempts in `done()`, wakes on the token's terminal `OnGen` event (`GenSettled`, `GenFailed`, `GenFellBack`, `GenCancelled`), on `dropUnownedWakes` for the dropped entry's token, and in `Begin` itself when a newer wake intent replaces an older wake intent for the host. A `Begin` that finds an attempt intent (`"a..."`) for the host returns an error.
589	3. Reconcile at open. Each intent left in the file becomes a failure record for its build on that host (`Cause` `release`, `Mode` the intent's, `HeldDown` and a hold-ran marker when `Mode == "hold"`, `PromotedAt` and `At` the open time, `Reason` a string `reasonWord` maps to an existing word; check `internal/rollout/events.go`). Intents are then deleted in memory and the book is marked dirty.
590	4. Schema 2. `saveLocked` writes `Schema: 2`; `OpenFailureBook` accepts 1 and 2 and errors on anything else.
591	5. `armRollout`: after `OpenFailureBook`, if `book.Reconciled() > 0`, log the count and `Flush()`; a flush error is `refuseManaged(svc, rolloutBookUnreadable)`. `honourDeclinedFailures`: the reconciled records are honoured; a flush error is logged.
592
593	- [ ] Step 1: Write the failing tests
594
595	```go
596	// internal/rollout/intent_test.go
597	// A save fails by making the data dir read-only (WriteFileAtomic writes its
598	// temp file beside the target). Running as root defeats it: the tests skip.
599
600	// Review Focus 1: B's failure is held only in memory because the save
601	// failed; the agent restarts. The intent written before B ran is still on
602	// disk and comes back as B's failure.
603	func TestAFailedSaveThenRestartKeepsTheFailure(t *testing.T) {
604	dd := t.TempDir()
605	b, _ := OpenFailureBook(dd)
606	if err := b.Begin("h", Intent{Account: "bab", App: "blog", Build: "v2", Mode: "hold", ID: "a7"}); err != nil {
607	t.Fatal(err)
608	}
609	if err := os.Chmod(dd, 0o500); err != nil {
610	t.Fatal(err)
611	}
612	t.Cleanup(func() { os.Chmod(dd, 0o700) })
613	if _, err := b.Record(FailureRecord{Account: "bab", App: "blog", Build: "v2", Host: "h", Mode: "hold", HeldDown: []string{"h"}}); err == nil {
614	t.Skip("the save did not fail (running as root?)")
615	}
616	_ = b.End("h", "a7") // fails too: the intent stays on disk
617	os.Chmod(dd, 0o700)
618	b2, err := OpenFailureBook(dd)
619	if err != nil {
620	t.Fatal(err)
621	}
622	if b2.Reconciled() != 1 \|\| !b2.HeldDown("bab", "blog", "v2", "h") {
623	t.Fatalf("after restart: reconciled %d, held down %v", b2.Reconciled(), b2.HeldDown("bab", "blog", "v2", "h"))
624	}
625	if m, ok := b2.HoldRanOn("h"); !ok \|\| m.Build != "v2" {
626	t.Fatal("a hold intent must leave its marker")
627	}
628	}
629
630	func TestABeginThatCannotSaveRefusesAndLeavesNothing(t *testing.T) {
631	dd := t.TempDir()
632	b, _ := OpenFailureBook(dd)
633	os.Chmod(dd, 0o500)
634	t.Cleanup(func() { os.Chmod(dd, 0o700) })
635	if err := b.Begin("h", Intent{Account: "bab", App: "blog", Build: "v2", Mode: "overlap", ID: "a1"}); err == nil {
636	t.Skip("the save did not fail")
637	}
638	os.Chmod(dd, 0o700)
639	b2, _ := OpenFailureBook(dd)
640	if b2.Reconciled() != 0 {
641	t.Fatal("a refused begin left an intent")
642	}
643	}
644
645	func TestASchema1BookReadsAndIsWrittenAsSchema2(t *testing.T) {
646	dd := t.TempDir()
647	raw := `{"schema":1,"records":[{"account":"bab","app":"blog","build":"v2","host":"h","cause":"release","mode":"overlap","reason":"x","promoted_at":"2026-10-01T00:00:00Z","at":"2026-10-01T00:00:00Z"}]}`
648	os.WriteFile(filepath.Join(dd, FailureFile), []byte(raw), 0o644)
649	b, err := OpenFailureBook(dd)
650	if err != nil \|\| !b.Blocks("bab", "blog", "v2", "x", false) {
651	t.Fatalf("schema 1: %v", err)
652	}
653	if err := b.Flush(); err != nil {
654	t.Fatal(err)
655	}
656	out, _ := os.ReadFile(filepath.Join(dd, FailureFile))
657	if !strings.Contains(string(out), `"schema": 2`) {
658	t.Fatalf("written as %s", out)
659	}
660	}
661
662	// Review Focus 2: what an older agent sees. Its reader refused schema 2.
663	func TestASchema2BookIsUnreadableToASchema1Reader(t *testing.T) {
664	var f struct{ Schema int `json:"schema"` }
665	dd := t.TempDir()
666	b, _ := OpenFailureBook(dd)
667	_ = b.Flush()
668	raw, _ := os.ReadFile(filepath.Join(dd, FailureFile))
669	json.Unmarshal(raw, &f)
670	if f.Schema == 1 {
671	t.Fatal("an old reader would accept the file and drop markers and intents")
672	}
673	}
674	```
675
676	Also add, in `intent_test.go`, Manager-level tests: (a) an adoption whose `Begin` fails never reaches `Act.Swap` (`h.act.snapshot()` shows no call) and is retried after the abandon deferral; (b) a first-wake plan whose `Begin` fails is refused with the shared text; (c) a planned wake's intent is ended by its `GenSettled` and by `dropUnownedWakes`. In `internal/host/rollout_test.go`: an armed box whose book holds an intent reconciles it, and a flush failure refuses managed starts (`rolloutUnavailable() == rolloutBookUnreadable`).
677
678	- [ ] Step 2: Run to verify they fail.
679
680	- [ ] Step 3: Implement the rules. `Begin`/`End` never run under `scheduler.mu` or `activator.Service.mu`. Reword the `dirty` comment (`failures.go:89-92`) and `Record`'s comment (`:155-163`) to say what now protects a failed save: the intent on disk.
681
682	- [ ] Step 4: Run the rollout package, the host tests `-run 'Rollout\|Declin\|Book'` (unsandboxed), and `mutate.sh 300`; paste all three.
683
684	- [ ] Step 5: Commit `zdd-readiness: write-ahead intents and schema 2 keep failures across a failed save and a restart (F5)`
685
686	---
687
688	### Task 6: An uncertain start in hold mode is a failure (ruling 206)
689
690	Corrections (plan review round 1; these override the text below):
691	- Fallback needs termination evidence (plan review #9). A nested start error can wrap both `errStartedThenExited` and a failed termination proof (`backend_exec.go:1709`), and `spawnPlanned` refuses a fallback when `proc == nil` (`activator/wakeplan.go:393`). So in overlap, `errStartedThenExited` is a failure of the boot build (`w.failed`), and the fallback runs only when the generation's termination is proven (no `genStartUnproven` in the error chain, or the activator's own proof succeeded). Otherwise the wake records the failure (`GenFailed`), serves nothing, and keeps the lease/containment as an unproven wake does today. Test both cleanup outcomes.
692
693	Files:
694	- Modify: `internal/activator/backend_exec.go:~1702-1725` (the `markStartUncertain` sites; give the `inv == ""` case its own sentinel `errStartedThenExited`)
695	- Modify: `internal/activator/swaphold.go:~190-205` (`holdResult`)
696	- Modify: `internal/activator/wakeplan.go:~355-390` (`spawnPlanned`)
697	- Test: `internal/activator/swaphold_test.go` or the file with `holdResult` tests; `internal/activator/wakeplan_test.go`
698
699	Interfaces:
700	- Produces: `var errStartedThenExited = errors.New("generation exited before it was observed")` wrapped by `markStartUncertain` for the `inv == ""` case, so `errors.Is` finds both.
701
702	Rules (Astra R3 ruling 206): a nested start that root accepted and whose outcome is uncertain may have run its build.
703	1. Hold swap: `holdResult` returns `SwapDown` (held down by `done()`, cohort halted) when `werr != nil`, the proc is nil and `startMayHaveRun(nil, werr)` is true. It never returns abandoned with `Executed: true` for such a start, so the abandon deferral cannot re-run the build.
704	2. Planned wake: `spawnPlanned` treats an uncertain start as a failure of the boot build (`w.failed`) in hold mode always, and in overlap mode only for `errStartedThenExited`. Other uncertain overlap starts keep today's `GenCancelled{Executed: true}`.
705
706	- [ ] Step 1: Write the failing tests against the nested fakes the existing uncertain-start tests use (search `markStartUncertain` in `internal/activator/*_test.go`): a hold swap whose nested start returns `markStartUncertain(errStartedThenExited)` yields `SwapDown` with `Executed` true; one returning another uncertain error also yields `SwapDown`; a hold planned wake with an uncertain start emits `GenFailed` (not `GenCancelled`); an overlap planned wake with `errStartedThenExited` emits `GenFellBack` when it has a fallback, and with another uncertain error still emits `GenCancelled{Executed: true}`.
707
708	- [ ] Step 2: Run to verify they fail (unsandboxed, `TMPDIR=/tmp/k`, `-run 'Uncertain'`).
709
710	- [ ] Step 3: Implement.
711
712	- [ ] Step 4: Run `-run 'Uncertain\|Hold\|Planned\|Swap'` in activator; paste.
713
714	- [ ] Step 5: Commit `zdd-readiness: an uncertain nested start is a hold failure, never a re-runnable abandon (ruling 206)`
715
716	---
717
718	### Task 7: Execution admission: worker gate, managed revalidation, uncaptured recycle
719
720	Files:
721	- Modify: `internal/activator/backend_exec.go:~2113-2160` (`StartWorkerStaged`), `internal/activator/workerbuild.go:~51-118` (`startWorker`, `runBeforeExec`), `:~160-186` (`recycleWorkerLocked`)
722	- Modify: `internal/rollout/scheduler.go:96-103` (`execute`)
723	- Test: `internal/activator/exec_gate_test.go`, `internal/rollout/manager_test.go`
724
725	Rules (Astra R3 §5 item 4, ruling 206 "park worker launch hook"):
726	1. Worker gate after preparation. `runBeforeExec` keeps only the generations-file publication. The caller's ownership gate rides the context (`withExecGate(ctx, gate)`), and `StartWorkerStaged` spawns with `b.spawnInCgroupGated(rt, build, b.launchGate(ctx, rt.Host))`, so the gate runs after cgroup preparation and before every start attempt, as the instance path does (`cgroup.go:~354/367`).
727	2. Managed revalidation. `execute`: after reading the row, `if !Managed(rt) \|\| release.RouteAccount(rt) != s.key.acct \|\| rt.App != s.key.app` the attempt finishes `SwapAbandoned` with `Executed: false` and reason `"route no longer managed"`, before `resolve`.
728	3. Uncaptured recycle. In `recycleWorkerLocked`, when `a.CoalesceRecycle != nil` (an armed or refusing box) and the worker has no captured build (`e.workerServing.StagedPath == ""`), it never respawns from the live link: with a live process (`stopOld != nil`) the recycle is skipped and the process keeps running; with none, the entry stays down and the agent logs once per host. Today's behaviour stays when `CoalesceRecycle` is nil.
729
730	- [ ] Step 1: Write the failing tests: (a) in `exec_gate_test.go`, modelled on the instance tests at `:419`, a worker start whose gate starts refusing during cgroup preparation never execs (the fake cgroup's prepare hook flips the gate); (b) a worker start retry after a failed PID move re-asks the gate; (c) in `manager_test.go`, a queued host whose row becomes a unit route before dispatch gets no `Swap` call and its outcome is abandoned; (d) an armed box's recycle of a worker with no captured build makes no live-link spawn.
731
732	- [ ] Step 2: Run to verify they fail.
733
734	- [ ] Step 3: Implement.
735
736	- [ ] Step 4: Run activator `-run 'Gate\|Worker\|Recycle'` (unsandboxed), rollout package, `mutate.sh 300`; paste.
737
738	- [ ] Step 5: Commit `zdd-readiness: worker gates after cgroup preparation, managed revalidation at dispatch, no uncaptured live-link recycle`
739
740	---
741
742	### Task 8: Host retries are signed by the console
743
744	Corrections (plan review round 1; these override the text below):
745	- Bind the grant to the restriction's epoch, not to a watermark (plan review #13). Drop `restartMemory.retryAt`. `HostRetryGrant` gains `Epoch uint64 \`json:"epoch"\``. The epoch of a host's restriction is a persisted book sequence number: `FailureRecord` gains `Epochs map[string]uint64 \`json:"epochs,omitempty"\`` (per host: `++seq` each time `Record` changes that host's standing in the record, and the record-wide entry under key `""` each time the record is created or merged), and a marker's epoch is its `Seq` (Task 4). `func (b *FailureBook) RestrictionEpoch(acct, app, build, host string) uint64` returns the larger of the record's `""` and host entries, or the marker's `Seq` when the marker names `build`.
746	- Status publishes it. `status.RolloutHost` gains `HeldEpoch uint64 \`json:"held_epoch,omitempty"\``, set beside `HeldBy` (`internal/rollout/status.go:74-93`). The console's rollout join and `retryHostBuild` (`internal/console/apiretry.go:41-58`) return `(build string, epoch uint64)`, and the door signs both. An absent epoch (an older box) is 0; the console still signs it, and the box refuses a 0 epoch.
747	- Which build a grant clears (plan review #5). `Manager.RetryHost(host, build string, epoch uint64) bool` clears (a) the record of `build` for the host, when one holds the host, and (b) the host's marker, when it names `build`; each only if `RestrictionEpoch(..., build, host) == epoch`. A grant naming the marker's build B while the slot wants A clears the marker and nothing else. Drop the `wantBuild` equality rule in the body below.
748	- The box's verification keeps the 24-hour issue window (`HostRetryGrantTTL = 24 * time.Hour`) as a backstop.
749	- Tests replace the replay test: a grant whose epoch predates a second failure of the same build on the same host clears nothing; reopening the book and replaying a used grant clears nothing new.
750
751	Files:
752	- Create: `internal/release/hostretry.go`, `internal/release/hostretry_test.go`
753	- Modify: `internal/release/restartscope.go:107-110` (`RestartScope.Signed`), `:161-175` (`ValidRestartScope`), `AppendRestartScope` if it compares scopes
754	- Modify: `internal/console/apiretry.go:61-71` (`requestHostRetry`), `:76-122`, and the browser twin `internal/console/browserretry.go`
755	- Modify: `internal/host/restartscope.go:~179-200` (`retryRequests` returns the signed field), `internal/host/restart.go:~838-863` (verify before `retryHost`), `internal/host/agent.go:~2296-2306` (`restartMem.pub`, `retryHost` signature)
756	- Modify: `internal/rollout/manager.go:515-553` (`RetryHost` takes the build)
757	- Test: `internal/console/apiretry_test.go` (or the existing retry test file), `internal/host/restart_test.go` (or the retry tests' file), `internal/rollout/manager_test.go`
758
759	Interfaces:
760	- Produces:
761
762	```go
763	// internal/release/hostretry.go
764	package release
765
766	// HostRetryGrant is the console's signed `carlos instances retry <host>`
767	// (spec §8.4, §11): a box clears a host's failure hold only on a grant the
768	// deployment root signed, naming that account, app, host and build.
769	type HostRetryGrant struct {
770	V int `json:"v"`
771	Account string `json:"account"`
772	App string `json:"app"`
773	Host string `json:"host"`
774	Build string `json:"build"`
775	IssuedAt string `json:"issued_at"` // RFC 3339, UTC, whole seconds
776	}
777
778	// HostRetryGrantTTL is how long a box accepts a grant after it was issued.
779	const HostRetryGrantTTL = time.Hour
780
781	var ErrHostRetryGrant = errors.New("host retry grant: not valid for this request")
782
783	func SignHostRetry(sg Signer, acct, app, host, build string, now time.Time) (string, error) {
784	g := HostRetryGrant{V: 1, Account: acct, App: app, Host: host, Build: build,
785	IssuedAt: now.UTC().Truncate(time.Second).Format(time.RFC3339)}
786	wire, err := SignRootEnvelope(sg, g)
787	return string(wire), err
788	}
789
790	// VerifyHostRetry checks wire against the root key and the entry it rode in:
791	// same account, app and host, issued no later than now plus a minute of skew
792	// and no earlier than now minus HostRetryGrantTTL.
793	func VerifyHostRetry(pub ed25519.PublicKey, wire, acct, app, host string, now time.Time) (HostRetryGrant, error) {
794	var g HostRetryGrant
795	if err := VerifyRootEnvelope(pub, []byte(wire), &g); err != nil {
796	return HostRetryGrant{}, err
797	}
798	at, err := time.Parse(time.RFC3339, g.IssuedAt)
799	if err != nil \|\| g.V != 1 \|\| g.Account != acct \|\| g.App != app \|\| g.Host != host \|\| g.Build == "" \|\|
800	at.After(now.Add(time.Minute)) \|\| at.Before(now.Add(-HostRetryGrantTTL)) {
801	return HostRetryGrant{}, ErrHostRetryGrant
802	}
803	return g, nil
804	}
805	```
806
807	- `RestartScope.Signed string \`json:"signed,omitempty"\``: only a `retry` scope may carry it; `ValidRestartScope` refuses it on other kinds and above `MaxRootEnvelopeBytes`.
808	- `Manager.RetryHost(host, build string) bool`: clears only when `build` is the build the host's queue slot wants (`h.wantBuild`) or, with no slot build, the build a record holds the host on; a mismatch clears nothing (and leaves the marker).
809	- `restartMemory.pub ed25519.PublicKey`; `restartMemory.retryHost func(host, build string) bool`; `restartMemory.retryAt map[string]time.Time` (newest accepted `IssuedAt` per host).
810
811	Rules:
812	1. Console. `requestHostRetry(ctx, acct, app, host, build)` signs with `release.SignHostRetry(release.RootSigner(s.cfg.Key), ...)` and appends `RestartScope{Kind: retry, Value: host, Signed: wire}`. With `s.cfg.Key == nil` the door answers 503 and appends nothing. For its text, reuse the read-only console's existing 503 wording for promote (`internal/console/server.go:~4135`) if it reads right for a retry; otherwise add a draft with `// TODO-COPY(zdd-readiness.retry-no-key)`.
813	2. Box. Every retry entry is verified with `mem.pub` against the entry's account, app and host at the box's clock. Unsigned, unverifiable, expired, not newer than `retryAt[host]`, or a build that does not match: logged (`agent: retry requested for %s, dropped: %s`) and skipped. A box with no public key drops every retry. Verified: `retryAt[host] = IssuedAt`, then `retryHost(host, g.Build)`.
814	3. Old entries stay readable; a new box reading an old console's unsigned entry drops it.
815
816	- [ ] Step 1: Write the failing tests: release (sign/verify round trip; wrong host, wrong app, expired, future, tampered payload, scoped signer refused); console (the API and browser doors append an entry whose `signed` verifies for the held build; a console with no key answers 503 and appends nothing); host (Review Focus 4: an unsigned entry, a replay of the same grant under a new entry id, and a grant for another build each leave `RetryHost` uncalled; a valid grant calls it once with the grant's build); rollout (`RetryHost("h", "v9")` with the slot wanting v2 clears nothing).
817
818	- [ ] Step 2: Run to verify they fail.
819
820	- [ ] Step 3: Implement. Keep the restart-scope id/mark logic as is.
821
822	- [ ] Step 4: Run release, rollout (sandboxed), console and host `-run 'Retry\|RestartScope'` (unsandboxed); paste.
823
824	- [ ] Step 5: Commit `zdd-readiness: host retries are root-signed in the console and verified on the box (Q4, spec §11)`
825
826	---
827
828	### Task 9: Declined boxes gate the actual launch and honour markers
829
830	Corrections (plan review round 1; these override the text below):
831	- Every declined launch mechanism is gated against the identity it actually starts (plan review #6): flat instance (`launchGate`), nested instance (`runExecGate`, `backend_exec.go:1682`), worker (`StartWorkerStaged`, `onto == nil`), and a claims-backed wake (after the claim snapshot resolves the live link, `claim_snapshot.go:87`, before the root request; Task 1 makes a claims box decline, so this is a declined box's path). Each passes the version it will start.
832	- No flat read/exec race. When `LiveExecGate` is set, the flat live-link spawn resolves the link once (`release.ResolveLive`), gates that version, and execs the resolved staged path instead of the link. With the gate nil (every other box) the spawn is byte-for-byte today's. Drop the "Residual" sentence in rule 1.
833	- Markers on a declined box: the installed good build is refused while a marker stands (as in the body); Task 10's pointer pass clears the marker on a signed rollback, as the armed box does (Task 3), so `HeldDownRefusal`'s "roll back or retry" stays true on a declined box.
834
835	Files:
836	- Modify: `internal/activator/genbackend.go:~65-114` (`launchGate`), `internal/activator/backend_exec.go:~1465-1500` (live-link spawn), `:~2113-2150` (`StartWorkerStaged`, `onto == nil`), `internal/activator/activator.go:~2641` (`convergeWorkerLocked`), `internal/activator/workerbuild.go:~160-186` (`recycleWorkerLocked` converge branch)
837	- Modify: `internal/host/rolloutdeclined.go` (`failureHolds`: markers, the gate, activation on markers or records)
838	- Test: `internal/host/rolloutdeclined_test.go`, `internal/activator/exec_gate_test.go`
839
840	Interfaces:
841	- Produces: `ExecBackend.LiveExecGate func(rt registry.Route, build string) error` (nil everywhere except a declined box with records or markers). It is asked (a) in `launchGate` when the context carries no gate, with the version read from the live link immediately before the start; (b) in `StartWorkerStaged` with `onto == nil`, with the resolved `id.Version`, before the spawn; (c) in `convergeWorkerLocked` and the converge branch of `recycleWorkerLocked` with the live link's version, before the old process is stopped: a refusal leaves the worker running its current build (a recycle of a dead worker respawns its captured build when it has one).
842	- `failureHolds.heldBuild(rt)` also answers for markers (rule below), and `failureHolds.gate(rt, build) error` returns `errors.New(rollout.HeldDownRefusal(b))` or `errors.New(refuseNoSwaps)` when `build` is held for `rt`.
843
844	Rules (Astra R1 #6, R2/R3 declined residuals; spec §6.4, §8.4):
845	1. Exec-time check. The plan-time refusal stays; the gate repeats it at the real launch with the build actually being started. Residual, recorded in the ledger: a flat instance execs the live-link path, so a root repoint landing between the gate's read and `exec` is not caught.
846	2. Markers on a declined box. With a marker `m` for the route's account and app: the marked build may start unless held down, or blocked by a record while the route has a good build; any other build a record blocks is refused; the installed good build is refused (a declined box has no pointer read in this path; recovery is a signed retry, Task 10, or a newer build); any other build starts.
847	3. `honourDeclinedFailures` installs the planner and the gate when the book holds records or markers.
848
849	- [ ] Step 1: Write the failing tests: (a) `TestDeclinedBoxRefusesAHeldBuildAtLaunch`: plan with the live link on the good build, repoint to the held build before the start, assert no exec (use `TestDeclinedBoxKeepsAHeldBuildOffARealActivator`, `rolloutdeclined_test.go:314`, as the template); (b) a declined worker whose live link moves to a held build is not stopped by the sweep's converge and keeps its process; (c) a declined worker that crashes with the live link on a held build respawns its captured build, or stays down with none; (d) markers: a declined box with only a marker refuses the installed good build and the marked build when a record blocks it; (e) flag off: `LiveExecGate` is nil and no leg changes (`legsAreTodays`).
850
851	- [ ] Step 2: Run to verify they fail (unsandboxed).
852
853	- [ ] Step 3: Implement. `LiveExecGate` is consulted under the activator's lock in (c); it must only read memory and the registry row and the link, as `heldBuild` does today.
854
855	- [ ] Step 4: Run host `-run 'Declin\|Rollout'` and activator `-run 'Gate\|Worker\|Converge'`; paste.
856
857	- [ ] Step 5: Commit `zdd-readiness: a declined box gates the actual launch, workers included, and honours hold-ran markers`
858
859	---
860
861	### Task 10: Declined boxes process retries
862
863	Corrections (plan review round 1; these override the text below):
864	- A declined box publishes its holds, and the console lets them be retried (plan review #5). `rolloutStatusProbe`'s declined report (`internal/host/rollout.go:481-490`) gains `Hosts` entries for each managed route `failureHolds` holds: `HeldBy` (the held build), `HeldEpoch` (`RestrictionEpoch`, Task 8), `Attempt` `status.AttemptDown` when held down or when a marker refuses the installed build, else `status.AttemptFailed`. `retryHostBuild` (`internal/console/apiretry.go:50`) accepts a report whose `Unavailable` is a decline code (`rolloutwords.UnavailableDeclines`) when the host has a `HeldBy`; it still refuses every other `Unavailable`. Add an API-to-box test: a declined box's held host gets a signed grant from the door, and the declined box's restart leg clears it.
865	- Retry clearing follows Task 8's epoch rule (`retryHost(host, build, epoch)`).
866	- The pointer pass also clears a marker on a signed rollback (Task 3's rule), and runs for apps with markers as well as records.
867
868	Files:
869	- Modify: `internal/host/rolloutdeclined.go` (`failureHolds.retryHost`, `failureHolds.applyPointers`; the comment that says the book is never written)
870	- Modify: `internal/host/agent.go:~2259-2306, ~2390-2396` (`fastConvergeLoop`: set `restartMem.retryHost` for a declined box; run the pointer pass)
871	- Test: `internal/host/rolloutdeclined_test.go`
872
873	Interfaces:
874	- Consumes: Task 8's `restartMemory.retryHost func(host, build string) bool` and signed-entry verification; Task 4's `FailureBook.RetryHost`, `ClearOnPointer`, `ClearHoldRan`.
875	- Produces: `func (h failureHolds) retryHost(host, build string) bool` and `func (h failureHolds) applyPointers(ctx context.Context, read func(ctx context.Context, acct, app, channel string) (*release.ChannelPointer, error), routes []registry.Route)`.
876
877	Rules (Astra R2 ruling 194 "startup-only records"):
878	1. A verified signed retry for a host clears that host from the record for `build` (`RetryHost`) and its marker (`ClearHoldRan`), and saves.
879	2. At most every 30 s (the armed `pointerReadEvery`), for each app with a standing record, the declined box reads each followed channel's signed pointer (`release.Channel` with the agent's store and key, both already open in `fastConvergeLoop`) and clears a record whose build the pointer names with a newer `PromotedAt` (`ClearOnPointer`). A read error is logged and skipped. No store or no key: no pointer pass, logged once.
880	3. Records and markers now change in the running process, so `heldBuild` reads the live book (it does today); no restart is needed.
881
882	- [ ] Step 1: Write the failing tests: a declined box with a hold-down of v2 on h wakes h onto v2 after a verified signed retry for (h, v2) (planner returns today's plan); after a pointer naming v2 with a newer `PromotedAt` the record is gone; an older pointer clears nothing; a retry for another build clears nothing.
883
884	- [ ] Step 2–4: run to fail, implement, run `-run 'Declin\|Retry'` (unsandboxed); paste.
885
886	- [ ] Step 5: Commit `zdd-readiness: declined boxes process signed retries and promote --retry pointers`
887
888	---
889
890	### Task 11: Bounded reads and acknowledgement writes
891
892	Corrections (plan review round 1; these override the text below):
893	- Ack order is fixed at the transition (plan review #10). `configTrack.activate` (called under `m.mu`) hands back the ack together with a per-host sequence taken under `m.mu`; the writer (`Manager.acks`) keeps the newest pending ack per host (latest sequence wins, older ones are dropped, never written after a newer one), submission never blocks, and one drain goroutine writes. When the manager's context ends, the drain finishes the write in progress and stops.
894	- Flag off keeps `configAckFunc` as it is (plan review #11): put the 10 s bound in the armed path only (the writer wraps each write in `context.WithTimeout`), not in `configAckFunc`.
895	- A graceful stop joins the scheduler (Task 5 review; spec §8.6 "Shutdown cancels and joins the whole operation set"). Add `func (m *Manager) Close(ctx context.Context)`: it stops admitting attempts, then waits (bounded by ctx) for every in-flight `execute` goroutine to finish its `done()` (including the book write and the intent `End`) and for every in-flight `OnGen` call to return, then stops the ack writer as above. `internal/host/agent.go` calls it right after `svc.Shutdown(ctx)` (~line 732), with the same 60 s context, when a manager is armed. Test: an attempt whose `Swap` is cancelled by shutdown has its outcome and `End` on disk when `Close` returns (no intent left in the file), so a planned agent restart mid-swap reconciles nothing.
896
897	Files:
898	- Modify: `internal/rollout/policy.go:~60-117` (`policyCache`)
899	- Modify: `internal/rollout/manager.go:418-466, 555-568, 831-887, 1017-1057` (`applyPointers`, `failedUnder`, `resolve`/`stagedDemand`, `activationDone`, `configSlept`), `internal/rollout/scheduler.go:46-50, 113-117`, `internal/rollout/wakeplan.go:254-283`
900	- Modify: `internal/host/agent.go:~1166-1173` (`configAckFunc`)
901	- Test: `internal/rollout/policy_test.go`, `internal/rollout/manager_test.go`
902
903	Interfaces:
904	- Produces: `const readBound = 5 * time.Second` (rollout); `failedUnder(ctx context.Context, rt registry.Route, build string) time.Time`; `Manager.acks` (an ordered asynchronous writer: one drain goroutine, FIFO, no lock held while writing).
905
906	Rules (Astra R1 Flag triage T3/T8/T10, R3 §5 items 6-7):
907	1. Policy cache. Each fill takes a per-key generation before `load`; the result is stored only if no newer fill has stored since. A failed read stamps `failedAt` and is not retried for `policyRetryAfter = 5 * time.Second`; meanwhile `get` answers the last value, or hold for an app never read. Missing (an empty manifest from `appconfig.Get`) is the defaults; an error is never the defaults.
908	2. Bounds. `policies.get`'s load, `Deps.Pointer` and `Deps.Demand` run under `context.WithTimeout(ctx, readBound)`. `failedUnder` takes the caller's context: the wake's in `planRow`, the attempt's in `execute`.
909	3. Acks. `activationDone` and `configSlept` hand `write` to `m.acks` instead of calling it, so neither a scheduler slot (`done()` calls `configActivated` before `finish`) nor the scheduler loop waits on the store. `configAckFunc` bounds its store write with a 10 s timeout.
910
911	- [ ] Step 1: Write the failing tests: (a) an older slow fill finishing after a newer one does not overwrite it (gate the loader on channels); (b) a failing loader is called at most once within `policyRetryAfter`; (c) a blocked ack writer does not stop `done()` from freeing the slot (the next attempt is admitted while the writer blocks); (d) a blocked pointer read in `failedUnder` returns zero once the wake's context is cancelled.
912
913	- [ ] Step 2–4: run to fail, implement, run the rollout package and host `-run 'ConfigAck\|Rollout'`; paste.
914
915	- [ ] Step 5: Commit `zdd-readiness: versioned policy fills, bounded store reads, acks off the scheduler's path`
916
917	---
918
919	### Task 12: Workers record clean one-shots and wait out a fallback; halts are always announced
920
921	Files:
922	- Modify: `internal/activator/workerswap.go:~173-192` (`SwapResult.CleanExit`), `internal/activator/activator.go:~1209-1222, ~2484-2562` (`WakeWorker` budget)
923	- Modify: `internal/rollout/scheduler.go:181-194` (`done`), `internal/rollout/manager.go` (a good write for a clean exit), `internal/rollout/wakeplan.go:469-476` (`buildFailed` with no scheduler)
924	- Test: `internal/activator/workerswap_test.go` (or the worker test file), `internal/rollout/worker_test.go`, `internal/rollout/wakeplan_test.go`
925
926	Interfaces:
927	- Produces: `activator.SwapResult.CleanExit bool`: the replacement's build exited cleanly as a one-shot (spec §6.5 success).
928
929	Rules:
930	1. One-shot good (spec §6.5, §8.5): `done()` for `SwapDone` with `res.CleanExit` writes the good build from `res.Gen` without `acknowledged` (the attempt still holds the host's slot, so no newer generation can have completed). Snapshot rules from Task 4 apply (`d.BookSeq`).
931	2. WakeWorker budget: while the entry is a first wake, `WakeWorker`'s wait extends once to `arrival + firstWakeBudget`, as request waits do (`activator.go:~1209-1222`).
932	3. Halt event: `buildFailed` with no scheduler emits `EvCohortHalted` with the approved text when `Record` reports the record new, as `cohortFailed` does.
933
934	- [ ] Step 1: Write the failing tests for each rule (a parked one-shot replacement records good; a first-wake worker whose fallback takes 40 s returns success to `WakeWorker`; a first-wake failure on a host with no scheduler emits the halt event).
935
936	- [ ] Step 2–4: run to fail, implement, run activator `-run 'Worker'` and the rollout package; paste.
937
938	- [ ] Step 5: Commit `zdd-readiness: clean one-shot replacements record good, WakeWorker waits a fallback out, halts are announced`
939
940	---
941
942	### Task 13: Fallback artifacts stay, and a hold build is published before its incumbent stops
943
944	Corrections (plan review round 1; these override the text below):
945	- Publish outside `a.mu` (plan review #10). `persistGenerations` takes `genMu` then `a.mu` (`generations_file.go:238`), and `swapHold`'s admit body runs under `a.mu` (`swaphold.go:107`). So: register the pending generation under `a.mu`, release it, persist, re-acquire `a.mu`, and re-run the full admission checks (ownership, newest, the entry's state) before `beginStopLocked`. Any check that fails withdraws the pending entry and abandons with `Executed: false`.
946	- The retention change is an authorized flag-off exception (Global Constraints).
947
948	Files:
949	- Modify: `internal/host/retention.go:41-46`
950	- Modify: `internal/activator/swaphold.go:~110-135, ~245-255`, `internal/activator/generations_file.go:~151-181`
951	- Test: `internal/host/retention_test.go`, `internal/activator/swaphold_test.go` (or the hold tests' file)
952
953	Rules (Astra R1 T12, R3 §5 item 9):
954	1. Bucket retention pins `rt.GoodVersion` beside `Version` and `PrevVersion`.
955	2. `swapHold` registers the hold build's generation as pending (as `startWorker` does with `genPending`) and persists the generations file before `beginStopLocked` stops the incumbent; a failed persist abandons the swap (`Executed: false`) with the incumbent still serving. The pending entry is dropped when the hold withdraws.
956
957	- [ ] Step 1: Write the failing tests: retention keeps a good build no route installs; a hold swap whose generations write fails leaves the incumbent running and returns abandoned; the generations file names the hold build before the incumbent's stop begins (record the order in the fakes).
958
959	- [ ] Step 2–4: run to fail, implement, run host `-run Retention` and activator `-run 'Hold\|Generations'`; paste.
960
961	- [ ] Step 5: Commit `zdd-readiness: retention pins the good build; a hold build is published before its incumbent stops`
962
963	---
964
965	### Task 14: The stage relay gives up, and a populated legacy parent is refused
966
967	Flag blocker confirmed on the dev box (#272, 2026-10-11, arm64, kernel 6.12.105, systemd 257, real-unit check (d)): a generation started under a legacy `carlos-exec@<host>` parent that still holds a process starts silently with NO memory limit, while the strict path refuses. The guard must fail closed and be reported: the generation start is refused while the host's parent holds processes directly, with an agent log line naming the host and the pid count. Unit-test it with a fake cgroupfs, and add a kernel-gated test (skipped unless the env the existing kernel tests use, e.g. `CARLOS_CLAIM_SPAWN_KERNEL=1` with `cgroup.Setup`, is present) that puts a process directly in a delegated parent and asserts the generation start is refused and creates no `g*` child.
968
969	Corrections (plan review round 1; these override the text below):
970	- Refuse direct parent members whatever children exist (plan review #12). `ensureParent` (`internal/cgroup/gen.go:127`) returns an error now (its signature changes; update every caller, including `StartChildGated` at `:168`). It reads the parent's own `cgroup.procs` (direct members only, not `cgroup.events`' descendant population) and returns `ErrLegacyParentPopulated` whenever that file lists a process, with or without `g` children. Replace the "one with a `g1` child still passes" test with `TestAPopulatedParentWithAStaleEmptyGenerationIsRefused`; keep a test that an empty parent with live `g` children passes.
971
972	Files:
973	- Modify: `internal/host/stagerelay.go` (`pendingHead` gains `since time.Time`; `relayGiveUp = time.Hour`)
974	- Modify: `internal/cgroup/gen.go:127-146` (`ensureParent`), and its caller's error path in `internal/activator` (abandon before exec)
975	- Test: `internal/host/stagerelay_test.go`, `internal/cgroup/gen_test.go`
976
977	Rules (Astra R1 T15 and T14 L146):
978	1. A relay head that root has neither installed nor refused for `relayGiveUp` is dropped with a log line; the next blocked pass notes it again.
979	2. `ensureParent` reads the parent's own `cgroup.procs`; when it holds a process and no generation child exists, it returns `ErrLegacyParentPopulated` and creates no child. The generation start fails before exec (the swap abandons with `Executed: false`; a wake fails as a start failure before exec).
980
981	- [ ] Step 1: Write the failing tests (a relay head older than `relayGiveUp` stops touching the marker; a fake cgroupfs parent with a pid in `cgroup.procs` and no `g*` child makes `ensureParent` return `ErrLegacyParentPopulated`; one with a `g1` child still passes).
982
983	- [ ] Step 2–4: run to fail, implement, run host `-run Relay`, cgroup `-run Parent` (unsandboxed); paste.
984
985	- [ ] Step 5: Commit `zdd-readiness: the stage relay gives up after an hour; a populated legacy parent refuses generation placement`
986
987	---
988
989	### Task 15: F6 and leftover executable files in /run
990
991	Files:
992	- Modify: `internal/activator/opset.go:~244-285` (`genChildEmpty`)
993	- Modify: `internal/instancectl/gen.go:~259-300` (`sweepGenLeftovers`)
994	- Test: `internal/activator/opset_test.go`, `internal/instancectl/gen_test.go`
995
996	Rules (Astra IMPL-3 #F6 `CODEX-FINDINGS-ZDD-IMPL-3.md:44-49`; IMPL-2-R2 `:45`):
997	1. F6 (flag off): when `KillGen` answers `errNoGenContainment` for a flat exec row's retained failed wake, `genChildEmpty` requires the host cgroup proven empty (`hostKiller.KillHost`) before the wake counts as stopped. Before writing it, establish what else can live in `carlos-exec@<host>` at that moment (co-spawned sidecar members, another live entry). If a live process of another entry can be there, use the non-killing `Cleanup.Populated()` check instead of `KillHost`, so the wake stays retained (and its lease kept) until the cgroup empties on its own. If neither is safe, stop and report BLOCKED with what you found; the item is then listed as owed.
998	2. /run sweep: `sweepGenLeftovers` also walks `ExeRoot` for generation exe directories (`<iid>@<n>`, parsed with the names helpers) that have no drop-in, are not starting (`genStarting`), whose unit systemd reports inactive or failed and whose cgroup is empty, and removes them under `genMu`. Host directories (no `@`) are never touched.
999
1000	- [ ] Step 1: Write the failing tests (F6: a flag-off retained wake whose leader exited is not released while the host cgroup is populated, and is released once it is empty; sweep: an exe-only leftover is removed, a starting one and a host dir are kept).
1001
1002	- [ ] Step 2–4: run to fail, implement, run activator `-run 'Unproven\|Retained\|GenChild'` and instancectl `-run Sweep` (unsandboxed); paste.
1003
1004	- [ ] Step 5: Commit `zdd-readiness: F6 proves the host cgroup empty for a retained flag-off wake; /run exe leftovers are swept`
1005
1006	---
1007
1008	### Task 16: Simulation: new invariants, new mutants, one 3000-seed sweep
1009
1010	Files:
1011	- Modify: `internal/rollout/sim_test.go`, `internal/rollout/testdata/rolloutsim/mutate.sh:58-63`
1012	- Modify: the marked lines in `internal/rollout/wakeplan.go`, `failures.go`, `scheduler.go` (add `// rolloutsim:Mn` markers)
1013
1014	Rules:
1015	1. Add mutants for this plan's invariants, each marked on exactly one line, each with the invariant that must catch it:
1016	- M7 (N7): the marked-build `Blocks` check in `decideHoldRan` disabled (`if false && hasGood && in.Book.Blocks(`). Caught by I1 (no instance boots a build a record holds for it) now that the exemption is gone.
1017	- M8 (N6): `ClearHoldRanBefore`'s sequence test disabled. Needs an invariant that a good write decided before a hold build ran never clears that build's marker; add it to the simulation as I5 if the simulation models good writes racing marks, otherwise make M8's kill a named unit test (rule 2).
1018	- M9 (retry ordering): `preRetryLocked` returns false. Invariant: a retry is never undone by an outcome that began before it (I6).
1019	- M10 (F5): a failed `Begin` does not refuse. Invariant: after an injected save failure and a simulated restart, no failed or held build boots on an instance it failed on (I7). Add a fault-injecting book path to the simulation if needed.
1020	- M11 (plan review #3): `failCohort` halts on a `PreRetry` disposition. Invariant I6 (a retry is never undone) must cover the cohort halt, not only the book.
1021	- M12 (plan review #4): `markLocked` keeps the old `Seq` for a different execution of the same build. Caught by I5 or by the extended N6 unit test (rule 2).
1022	2. If a mutant's invariant cannot be modelled in the simulation, extend `mutate.sh` so a mutant line may name its own test (`-run`) instead of `TestSimInvariants`, keep the integrity rules, and say so in the script's header.
1023	3. Run `uptime` first (back off above load 12), then once: `nice -n 19 internal/rollout/testdata/rolloutsim/mutate.sh 3000` with the sandboxed exports. Want a PASSED baseline and CAUGHT for M1 to M12. Paste the verdict lines.
1024
1025	- [ ] Steps: add invariants and mutants, run `mutate.sh 300` until clean, commit, run the 3000 sweep, commit any fixes and the evidence in the report.
1026	- [ ] Commit `zdd-readiness: simulation invariants and mutants for markers, retries and intents`
1027
1028	---
1029
1030	### Task 17: A host that is starting or stopping is never counted done
1031
1032	Files:
1033	- Modify: `internal/status/snapshot.go` (`RolloutHost.Transient`), `internal/rollout/status.go:~40-95` (`Manager.Status`)
1034	- Modify: the console's rollout view struct and its join (`internal/console/apiinstances.go`, search `apiRollout` / the rollout view the list returns), the CLI's `apiRollout` and `classifyRollout` (`deploywatch.go:~182`)
1035	- Test: `internal/rollout/status_test.go`, `internal/console/rolloutjoin_test.go`, `deploywatch_test.go`
1036
1037	Interfaces:
1038	- Produces: `status.RolloutHost.Transient bool \`json:"transient,omitempty"\``: the activator owns the host (`Owns`) but has no current generation for it (`Current` false), so its process is starting or stopping and the cached serving build says nothing settled. The console view carries it as `transient`.
1039
1040	Rules (Astra R3 §5 item 10; plan review #8): spec §9.1 counts a sleeping instance as "installed, awaiting first wake"; a host caught mid-wake or mid-stop is neither asleep nor awake.
1041	1. `Manager.Status` asks `Owns` and `Current` for each host with no manager or scheduler lock held (the lock order on `Manager`), and for a transient host reports `Transient: true`, `Awake: false`, and no `Serving` from the stale cache.
1042	2. `classifyRollout` returns pending for a transient host, so the watch neither counts it done as asleep nor as swapped.
1043
1044	- [ ] Step 1: Write the failing tests: `Status` reports a host the fake activator owns with no current generation as transient with no serving build; the console list carries `"transient": true`; the watch keeps a transient host pending through a sweep and completes once it reports asleep or swapped.
1045	- [ ] Step 2–4: run to fail, implement, run rollout `-run Status`, console `-run Rollout`, root `-run TestWatch`; paste.
1046	- [ ] Step 5: Commit `zdd-readiness: a host starting or stopping is reported transient and never counted done`
1047
1048	---
1049
1050	### Task 18: Roll notes, the owed list, and the copy index
1051
1052	Files:
1053	- Modify: `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md` ("Before Gate E: owed", the claims reply, a new "Claims-on boxes" note)
1054	- Create: `copy-review/strings.json` (only if any `TODO-COPY` marker exists on the branch)
1055
1056	- [ ] Step 1: `git grep -n 'TODO-COPY'` and build `copy-review/strings.json` in the copy-review skill's format (`id`, `section`, `label`, `text`, `context`, `notes`), one row per marker, the `id` matching the marker. If there are none, write none and say so.
1057	- [ ] Step 2: Update the roll notes: which #272 items this branch closes (task commit refs filled in by the controller), the claims-on limitation and its reason code, the retry-signing roll order (boxes before consoles; unsigned retries are ignored), the schema-2 downgrade note, and the real-box checks unchanged (D1, the kernel suite, real-unit checks, the flagship probe).
1058	- [ ] Step 3: Roll note from #272: `platform-infrastructure/scripts/update-platform-binary.sh` does not restart `carlos-instance-control`; use `carlos system update` (main's system apply restarts it, 0bb57a31), or restart it by hand after the old script.
1059	- [ ] Step 4: Commit `zdd-readiness: roll notes and copy index`
1060
1061	---
1062
1063	### Task 19: A killed cgroup is recreated before anything respawns into it
1064
1065	(Added 2026-10-11 from the dev-box run on #272. Execute it before Task 15, which uses `KillHost`.)
1066
1067	Flag blocker confirmed on the dev box (arm64, kernel 6.12.105, systemd 257): once `cgroup.kill` has been written to a cgroup, every later `clone3(CLONE_INTO_CGROUP)` into that same cgroup is SIGKILLed at once. Recreating the cgroup fixes it. `TestClaimContainmentCleanupRespawn` (`internal/activator/claimspawn_kernel_linux_test.go:23`) reproduces it 3 of 3 on the box. Exposed: `ExecBackend.KillSidecar` (`internal/activator/genbackend.go:340`, the role-sidecar child) and `ExecBackend.KillHost` (`:431`, `carlos-exec@<host>`), which kill a cgroup and keep it. Generation kills remove their cgroup and are safe.
1068
1069	Files:
1070	- Modify: `internal/cgroup/cleanup_linux.go:~282` (`Cleanup.KillAndEmpty`) and/or the cgroup `Manager`'s start path (`internal/cgroup/cgroup.go:~295-370`), `internal/cgroup/cleanup_other.go` if a signature changes
1071	- Modify: `internal/activator/genbackend.go:340-370, 431-449` if the fix lives there
1072	- Test: `internal/cgroup/*_test.go` (fake cgroupfs), the kernel-gated `internal/activator/claimspawn_kernel_linux_test.go`
1073
1074	Rules:
1075	1. After a `cgroup.kill` on a cgroup we keep (the host cgroup, a role child), nothing is ever cloned into that same directory again. Before the next spawn into it, the emptied cgroup (and any empty child directories left under it) is removed and created fresh. Choose the place, removal right after the proven-empty kill or recreation at the next start, and say why. Either way, the recreated cgroup gets the same limits and controller settings it had (the ones the start path applies when it creates the cgroup), and it stays under the same parent, so the parent's accounting is unchanged.
1076	2. Removal must not race a concurrent start into the same cgroup. Use whatever lock the start path already holds for that host, or add one, and document it.
1077	3. A removal that fails (busy, EBUSY on a cgroup that is somehow repopulated) is an error: the kill is not proven, and the next start refuses rather than clone into a killed cgroup.
1078	4. Flag off: `KillHost` is reached with the flag off only through Task 15's F6 path. The new behaviour applies there too; it is an authorized exception (it only replaces a cgroup that was just proven empty).
1079
1080	- [ ] Step 1: Write the failing tests: a fake-cgroupfs test that a kill of a kept cgroup followed by a start creates a fresh directory (new inode, or a recorded remove-then-create) with the limit files written again; a test that a failed removal makes the next start refuse; and make `TestClaimContainmentCleanupRespawn`'s kernel-gated path (unchanged gate, `CARLOS_CLAIM_SPAWN_KERNEL=1`) exercise the real `KillHost`/`KillSidecar` then respawn sequence if it does not already. The kernel test cannot run on this build box; say so, and leave it for the dev-box rerun.
1081	- [ ] Step 2–4: run to fail, implement, run cgroup and activator `-run 'Kill\|Respawn\|Cleanup\|Sidecar'` (unsandboxed); paste.
1082	- [ ] Step 5: Commit `zdd-readiness: a cgroup that took cgroup.kill is recreated before anything respawns into it`
1083