amadan

carlos / platform Public

Clone 
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1# S3 Change Index Implementation Plan
2
3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5**Goal:** Stop every box and the console from re-walking the whole deployment bucket: writers bump a per-scope generation in `control/changes.json`, readers re-read only what moved, a 30-minute backstop walk bounds staleness.
6
7**Architecture:** The bump lives inside the two concrete stores (`S3Store`, `DirStore`), enabled by deployment openers. A process-wide `release.ChangeWatch` polls the index with a conditional GET. `provision.RecordView` keeps the record families the walk reads and exposes immutable snapshots as a read-through `release.Store`, so `Reconcile`, `ReconcileSidecars`, the moves pass, steering and the console sweeps run unchanged over a snapshot. A `release.ReadGate` gates the per-app GET loops on generations plus a maximum age.
8
9**Tech Stack:** Go, AWS SDK v2 S3 (existing client in `internal/release`), `DirStore` for tests, MinIO in the delivered-contract test.
10
11**Spec:** `docs/superpowers/specs/2026-10-21-s3-change-index-design.md` (rev 3, Astra-accepted). Section numbers below (§N) refer to it.
12
13## Global Constraints
14
15- Index key `control/changes.json`, body `{"schema":1,"entries":{"<scope>":<uint64>}}`; generations only increase; entries never removed in v1 (§3.1).
16- Scopes exactly per §3.2: `apps/<a>/<b>` (except data prefixes `logs`, `errors`, `analytics`, `runs`, `metrics`), `accounts/<a>`, `control/certs`, `control/edges`; nothing else.
17- Bump after a successful mutation only; at most 4 attempts with jitter inside a 2s budget detached from the caller's cancellation; a failed bump is logged and never fails the write (§3.3).
18- Backstop interval 30 minutes, jittered ±10% (R3). Gated reads have a maximum successful-read age equal to the backstop (§5).
19- Readers armed only by `CARLOS_CHANGE_INDEX=read` (box `host.env`, console environment). Unset means exactly today's behavior. Writers always bump once shipped (§7).
20- A store without `ConditionalGetter` + `ETagLister`, a denied index read, or a malformed index means "unknown": walk as today, never "empty and unchanged" (§3.4).
21- Core only: no AWS SDK outside `internal/release`'s existing S3 client (§6).
22- Lock order: `w.pass` → `reconcileMu` → `RecordView.mu`; `RecordView.mu` is a leaf (§4.3).
23- Tests: run only touched packages locally with `-run`; CI runs the full suite. Leave `GOCACHE` at its default. The simulation runs niced, one 3000-seed sweep per revision.
24- No tenant names in code, tests or comments (use `shop`, `example.com`).
25- Every new top-level test stands alone: `make test` is about to split `internal/console` and the root package across separate `go test` processes, so no test may depend on another having run first. Root-package e2e tests call `buildCarlos(t)` and never write to the shared read-only binary it returns.
26- `internal/host` tests call `t.Parallel()` unless they set env (`t.Setenv`), cwd, umask, log output or package vars, or reach `edge.SecureUnixListen`. Tasks 8 and 9 set the package vars `armedChangeWatch`/`armedRecordView` (and `CARLOS_CHANGE_INDEX`), so those tests stay serial; tests that only build their own `fastDeps`, `genCache` or gates call `t.Parallel()`.
27- Any user-facing text (release notes for R4) goes through the copy-review skill before it is written.
28
29## Review Focus
30
311. **A deployment with no `control/changes.json` yet** (first arm, fresh bucket): the first `Refresh` must rebuild by walking, and the view must never publish as empty-and-complete. Test in Task 6.
322. **A malformed or future-schema index** (hand edit, newer writer): readers treat it as unusable and walk; writers do not overwrite an unknown schema. Tests in Tasks 1 and 4.
333. **A mixed fleet**: an armed reader while an old binary writes a record without bumping. The record appears after the next `Rebuild`, and no route is retired in between. Tests in Tasks 6 and 7.
344. **An app purge** deleting hundreds of keys under one scope: bumps stay bounded (one per mutation, each a single CAS), the view drops the app once, nothing is retired before the purge's own records are gone. Test in Task 2 (bump count) and Task 6 (view).
355. **Index read denied by IAM** (role lacks `GetObject` on the key): `ChangeWatch.Usable()` stays false, `ReadGate.Due` is always true, every loop behaves as today. Tests in Tasks 4 and 9.
36
37---
38
39### Task 1: Change scope and the index document
40
41**Files:**
42- Create: `internal/release/changeindex.go`
43- Test: `internal/release/changeindex_test.go`
44
45**Interfaces:**
46- Produces:
47 - `const ChangeIndexKey = "control/changes.json"`
48 - `type ChangeIndex struct { Schema int \`json:"schema"\`; Entries map[string]uint64 \`json:"entries"\` }`
49 - `func ChangeScope(key string) (scope string, ok bool)`
50 - `func ReadChangeIndex(ctx context.Context, s Store) (ChangeIndex, string, error)` — absent object returns `ChangeIndex{Schema: 1, Entries: {}}`, etag `""`, nil; schema ≠ 1 returns `ErrChangeIndexSchema`.
51 - `var ErrChangeIndexSchema = errors.New("release: change index schema unsupported")`
52 - `func bumpChangeIndex(ctx context.Context, get func(context.Context, string) ([]byte, string, error), put func(context.Context, string, []byte, string) (string, error), scopes ...string) error` — unexported; takes raw get/put so the store hook (Task 2) can call it without recursion.
53
54- [ ] **Step 1: Write the failing tests**
55
56```go
57func TestChangeScope(t *testing.T) {
58 cases := map[string]string{
59 "apps/bab/shop/instances/a.example.com.json": "apps/bab/shop",
60 "apps/bab/shop/config-gen.json": "apps/bab/shop",
61 "apps/bab/shop/channels/canary/x": "apps/bab/shop",
62 "apps/bab/shop/sidecars.json": "apps/bab/shop",
63 "accounts/bab/account.json": "accounts/bab",
64 "accounts/bab/telemetry.json": "accounts/bab",
65 "control/certs/example.com": "control/certs",
66 "control/edges/box-a.json": "control/edges",
67 }
68 for k, want := range cases {
69 got, ok := ChangeScope(k)
70 if !ok || got != want { t.Errorf("ChangeScope(%q) = %q,%v want %q", k, got, ok, want) }
71 }
72 for _, k := range []string{
73 "apps/bab/shop/logs/box/2026-10-21-10/x", "apps/bab/shop/errors/x",
74 "apps/bab/shop/analytics/x", "apps/bab/shop/runs/r1/request.json",
75 "apps/bab/shop/metrics/x", "apps/bab/shop", "apps/bab/", "apps/",
76 ChangeIndexKey, "control/serial", "control/edge-mesh-key",
77 "control/edge-mesh-key.next", "releases/x", "status/boxes/a.json",
78 "fleets/f/apps/bab/shop/config.json",
79 } {
80 if s, ok := ChangeScope(k); ok { t.Errorf("ChangeScope(%q) = %q, want none", k, s) }
81 }
82}
83
84func TestBumpChangeIndexCreatesThenIncrements(t *testing.T) // DirStore: two bumps of "apps/bab/shop" -> 2; other entry untouched
85func TestBumpChangeIndexRetriesOnConflict(t *testing.T) // put fails with ErrCASConflict twice, then succeeds -> nil, final gen 1
86func TestBumpChangeIndexGivesUpAfterFourAttempts(t *testing.T) // 4 conflicts -> non-nil error, exactly 4 puts
87func TestBumpChangeIndexRefusesUnknownSchema(t *testing.T) // existing {"schema":2,...} -> ErrChangeIndexSchema, no put
88func TestReadChangeIndexAbsentIsEmpty(t *testing.T) // Entries non-nil, len 0, etag ""
89```
90
91- [ ] **Step 2: Run to verify they fail**
92
93Run: `go test ./internal/release/ -run 'TestChangeScope|TestBumpChangeIndex|TestReadChangeIndex' -count=1`
94Expected: FAIL (undefined: ChangeScope)
95
96- [ ] **Step 3: Implement** `ChangeScope`, `ReadChangeIndex`, `bumpChangeIndex` in `internal/release/changeindex.go`. `ChangeScope` requires `apps/<a>/<b>/<rest>` with all three non-empty segments and `<rest>`'s first segment not in the data set; `accounts/<a>/<rest>` with both non-empty. `bumpChangeIndex` applies all scopes in one read-modify-write; conflict is `errors.Is(err, ErrCASConflict)`; jitter 20–120ms between attempts; it does not create its own context (Task 2 passes the detached one).
97
98- [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS
99
100- [ ] **Step 5: Commit** `git commit -m "release: change scopes and the change index document (#336)"`
101
102---
103
104### Task 2: Bump inside the concrete stores
105
106**Files:**
107- Modify: `internal/release/store.go` (`S3Store` struct `:681`, `DirStore` struct `:342`, `Put` `:414/:787`, `PutFrom` `:456/:820`, `Delete` `:660/:970`)
108- Modify: `internal/release/conditional_delete.go` (`DeleteIfMatch` `:63/:86`)
109- Test: `internal/release/changeindex_store_test.go`
110
111**Interfaces:**
112- Consumes: Task 1's `ChangeScope`, `bumpChangeIndex`.
113- Produces:
114 - `func (s *S3Store) EnableChangeIndex()`, `func (d *DirStore) EnableChangeIndex()`
115 - `func NewDeploymentS3Store(ctx context.Context, bucket string) (*S3Store, error)` and `func NewDeploymentS3StoreIn(ctx context.Context, bucket, region string) (*S3Store, error)` — the S3 constructors plus `EnableChangeIndex()`.
116 - `var changeIndexBumpBudget = 2 * time.Second` (var for tests).
117
118- [ ] **Step 1: Write the failing tests** (all on `DirStore` with the index enabled)
119
120```go
121func TestStoreBumpsAfterEveryMutation(t *testing.T) {
122 // Put, PutFrom, Delete, DeleteIfMatch on "apps/bab/shop/instances/h.json"
123 // each raise entries["apps/bab/shop"] by exactly 1.
124}
125func TestStoreDoesNotBumpFailedMutation(t *testing.T) // Put with stale ifMatch -> ErrCASConflict, gen unchanged
126func TestStoreDoesNotBumpUnscopedKeys(t *testing.T) // logs/, control/serial, the index key itself -> no index object
127func TestStoreBumpFailureDoesNotFailWrite(t *testing.T) // index object is a directory (unwritable) -> Put returns nil, record written
128func TestStoreBumpDetachedFromCallerCancel(t *testing.T)// ctx cancelled right after Put returns from the backend -> bump still lands
129func TestStoreWithoutEnableNeverBumps(t *testing.T)
130func TestStoreCapabilitiesPreserved(t *testing.T) {
131 d := NewDirStore(t.TempDir()); d.EnableChangeIndex()
132 var s Store = d
133 _, ok1 := s.(ConditionalDeleter); _, ok2 := s.(PutStreamer); _, ok3 := s.(Opener)
134 _, ok4 := s.(SizeLister); _, ok5 := s.(PageLister); _, ok6 := s.(ModLister)
135 if !(ok1 && ok2 && ok3 && ok4 && ok5 && ok6) { t.Fatal("index-enabled DirStore lost an extension") }
136}
137func TestPurgeBumpsBounded(t *testing.T) // 300 deletes under one app -> gen 300, each bump one CAS (count puts on a counting wrapper of the raw backend)
138```
139
140- [ ] **Step 2: Run to verify they fail**
141
142Run: `go test ./internal/release/ -run 'TestStore(Bumps|DoesNot|BumpFailure|BumpDetached|WithoutEnable|Capabilities)|TestPurgeBumps' -count=1`
143Expected: FAIL (undefined: EnableChangeIndex)
144
145- [ ] **Step 3: Implement.** Add an unexported `changeIndex bool` field to both structs. Split each mutation into the existing body (renamed `putRaw`, `putFromRaw`, `deleteRaw`, `deleteIfMatchRaw`) and a public method that calls the raw one and, on success and when enabled and `ChangeScope` matches, calls `bumpChangeIndex` with `getRaw`/`putRaw` under `context.WithTimeout(context.WithoutCancel(ctx), changeIndexBumpBudget)`, logging a failure as `release: change index bump %s: %v`. The struct literal `&release.S3Store{Client:..., Bucket:...}` elsewhere must still compile (exported fields unchanged).
146
147- [ ] **Step 4: Run to verify they pass** (same command, plus `go test ./internal/release/ -count=1` for regressions). Expected: PASS
148
149- [ ] **Step 5: Commit** `git commit -m "release: concrete stores bump the change index after each mutation (#336)"`
150
151---
152
153### Task 3: Read extensions and their forwarding
154
155**Files:**
156- Modify: `internal/release/store.go` (new interfaces beside `ModLister` `:186-207`; methods on both stores; `prefixStore` forwarding `:113-217`)
157- Modify: `internal/hostops/mutation_boundary.go` (forwarding surface `:108-130`)
158- Modify: the MinIO delivered-contract test (`grep -rn TestDeliveredContractMinio`)
159- Test: `internal/release/readext_test.go`, `internal/hostops/mutation_boundary_test.go`
160
161**Interfaces:**
162- Produces:
163 - `type ConditionalGetter interface { GetIfChanged(ctx context.Context, key, etag string) (body []byte, newETag string, changed bool, err error) }` — `etag == ""` behaves as `Get`; unchanged returns `nil, etag, false, nil`; absent returns an `os.ErrNotExist`-wrapped error.
164 - `type KeyETag struct { Key, ETag string }`
165 - `type ETagLister interface { ListETags(ctx context.Context, prefix string) ([]KeyETag, error) }` — sorted by key; ETags in the same form `Get` returns.
166 - Both on `*S3Store` and `*DirStore`; `prefixStore` and `mutationBoundary` forward both (boundary returns `ErrContract` when the raw store lacks one, matching `ListMod`).
167
168- [ ] **Step 1: Write the failing tests**
169
170```go
171func TestGetIfChangedDirStore(t *testing.T) // put k; GetIfChanged(k,"") -> changed, etag e; GetIfChanged(k,e) -> !changed, body nil; put again; GetIfChanged(k,e) -> changed
172func TestGetIfChangedAbsent(t *testing.T) // errors.Is(err, os.ErrNotExist)
173func TestListETagsMatchesGet(t *testing.T) // for each listed key, ETag == etag from Get
174func TestPrefixStoreForwardsReadExtensions(t *testing.T) // NewPrefixStore(dir,"fleets/f/") answers both, keys stripped
175func TestBoundaryForwardsReadExtensions(t *testing.T) // in hostops
176```
177In the MinIO contract test add: `GetIfChanged` returns `changed=false` for the current ETag (S3 304), and `ListETags` ETags equal `Get` ETags.
178
179- [ ] **Step 2: Run to verify they fail**
180
181Run: `go test ./internal/release/ ./internal/hostops/ -run 'GetIfChanged|ListETags|ForwardsReadExtensions' -count=1`
182Expected: FAIL
183
184- [ ] **Step 3: Implement.** S3: `GetObject` with `IfNoneMatch: etag`; a 304 surfaces from the SDK as a smithy API error with HTTP status 304 — map it to `changed=false`. `ListETags` pages `ListObjectsV2` and keeps `Key`/`ETag` exactly as `Get` reports them (same quoting). DirStore: compute the ETag the way its `Get` does.
185
186- [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS. MinIO contract runs in CI.
187
188- [ ] **Step 5: Commit** `git commit -m "release: conditional GET and ETag listings, forwarded through prefix stores and the hostops boundary (#336)"`
189
190---
191
192### Task 4: ChangeWatch and ReadGate
193
194**Files:**
195- Create: `internal/release/changewatch.go`
196- Test: `internal/release/changewatch_test.go`
197
198**Interfaces:**
199- Consumes: Tasks 1 and 3.
200- Produces:
201 - `const ChangeIndexBackstop = 30 * time.Minute` (R3; here so `provision`, `host`, `edge` and `console` share it without import cycles)
202 - `type ChangeWatch struct` (safe for concurrent use)
203 - `func NewChangeWatch(s Store) *ChangeWatch`
204 - `func (w *ChangeWatch) Poll(ctx context.Context) (moved []string, err error)` — conditional GET of `ChangeIndexKey`; returns scopes whose generation rose (or appeared) since the last successful poll; on 304 returns `nil, nil`; any error, an unknown schema, or a store lacking `ConditionalGetter` marks the watch unusable and returns the error.
205 - `func (w *ChangeWatch) Gen(scope string) (gen uint64, ok bool)` — `ok` false while unusable.
206 - `func (w *ChangeWatch) Observed() (map[string]uint64, bool)` — copy of the last good entries.
207 - `func (w *ChangeWatch) Usable() bool`
208 - `type ReadGate struct`; `func NewReadGate(w *ChangeWatch, maxAge time.Duration) *ReadGate`
209 - `func (g *ReadGate) Due(scope string, now time.Time) (gen uint64, due bool)` — due when the watch is nil or unusable, the scope was never marked, its last mark failed, its generation moved past the marked one, or `now` is at least `maxAge` past the last successful mark.
210 - `func (g *ReadGate) Mark(scope string, gen uint64, now time.Time, ok bool)` — records the generation captured by `Due` before the read.
211
212- [ ] **Step 1: Write the failing tests**
213
214```go
215func TestChangeWatchFirstPollReportsAll(t *testing.T) // index with 2 entries -> moved has both
216func TestChangeWatchUnchangedIs304(t *testing.T) // second poll, no writes -> moved nil, err nil; counting store shows GetIfChanged with etag
217func TestChangeWatchReportsOnlyMoved(t *testing.T)
218func TestChangeWatchAbsentIndexIsUsableEmpty(t *testing.T) // absent -> Usable true, Observed empty
219func TestChangeWatchDeniedIsUnusable(t *testing.T) // store whose Get of the key fails with access denied -> Usable false
220func TestChangeWatchUnknownSchemaIsUnusable(t *testing.T)
221func TestChangeWatchWithoutExtensionIsUnusable(t *testing.T)
222func TestReadGateDue(t *testing.T) // table: unusable->due; never marked->due; marked ok same gen within age->!due; gen moved->due; age>=maxAge->due; last mark failed->due
223```
224
225- [ ] **Step 2: Run to verify they fail**
226
227Run: `go test ./internal/release/ -run 'TestChangeWatch|TestReadGate' -count=1`
228Expected: FAIL
229
230- [ ] **Step 3: Implement** in `changewatch.go`. Absent index is usable with empty entries (Review Focus 1 is handled by `RecordView`, not here).
231
232- [ ] **Step 4: Run to verify they pass.** Expected: PASS
233
234- [ ] **Step 5: Commit** `git commit -m "release: ChangeWatch polls the change index; ReadGate gates reads on generation and age (#336)"`
235
236---
237
238### Task 5: Every deployment opener enables the index
239
240**Files:**
241- Modify: deployment-bucket openers among the `release.NewS3Store`/`NewS3StoreIn` call sites: `cli.go:1957,2067,3591`, `cmd/console/main.go:633` (the deployment bucket; `:333` is the database bucket), `internal/host/agent.go:4368,4462`, `internal/edge/edge.go:3407`, `internal/host/relocatepass.go:251,680`, `internal/host/activation_command_linux.go:67,92,206,283`, `internal/ops/ops.go:339`, `internal/vet/vet.go:226`, `internal/host/seedwiring.go:102`. Classify each by reading it; switch deployment ones to `NewDeploymentS3Store[In]`.
242- Modify: `internal/host/boxpublish.go` only if its test shows the concrete assertion `:259` failing (it should not: the type is still `*release.S3Store`).
243- Create: `internal/release/openers_test.go`
244
245**Interfaces:**
246- Consumes: Task 2's `NewDeploymentS3Store[In]`.
247
248- [ ] **Step 1: Write the failing test** `TestS3StoreOpenersClassified`: parse every non-test `.go` file in the module with `go/parser`, find calls to `release.NewS3Store`, `release.NewS3StoreIn` (and the package-internal `NewS3Store` outside `store.go`), and require each `file:func` to appear in a table `otherBucketOpeners map[string]string` (value: the reason, for example `"database bucket"`, `"hibernate data bucket"`). Deployment openers must call `NewDeploymentS3Store[In]` instead, so they never appear.
249
250- [ ] **Step 2: Run to verify it fails**
251
252Run: `go test ./internal/release/ -run TestS3StoreOpenersClassified -count=1`
253Expected: FAIL listing the unclassified call sites
254
255- [ ] **Step 3: Switch** each deployment opener to `NewDeploymentS3Store[In]` and list the rest in `otherBucketOpeners` with a reason.
256
257- [ ] **Step 4: Box publication witness** `TestBoxPublishThroughDeploymentOpener` in `internal/host`: `openDeployStoreEnv` against a `DirStore`-backed fake is out of reach, so assert at the type level that `NewDeploymentS3Store`'s result satisfies the `*release.S3Store` assertion used at `boxpublish.go:259` (compile-time `var _ *release.S3Store = mustDeploy()` plus a runtime check through the same `opened.(*release.S3Store)` expression).
258
259- [ ] **Step 5: Run** `go test ./internal/release/ ./internal/host/ -run 'TestS3StoreOpenersClassified|TestBoxPublishThroughDeploymentOpener' -count=1` and `go build ./...`. Expected: PASS, build clean.
260
261- [ ] **Step 6: IAM check.** Read `infra/modules/deployment-buckets/main.tf` and `infra/modules/region-host` policies: confirm every writing role (box, console) is allowed `s3:GetObject` and `s3:PutObject` on `control/changes.json` and no bucket-policy Deny covers it. If anything denies it, add an allow and a `tests/*.tftest.hcl` assertion in that module. Record the result (and the same check owed on the Tito infrastructure repo) in the branch's #336 thread.
262
263- [ ] **Step 7: Commit** `git commit -m "Deployment openers enable the change index; opener classification test (#336)"`
264
265---
266
267### Task 6: RecordView and its snapshot store
268
269**Files:**
270- Create: `internal/provision/recordview.go`, `internal/provision/recordview_store.go`
271- Test: `internal/provision/recordview_test.go`
272
273**Interfaces:**
274- Consumes: Tasks 1, 3, 4 (including `release.ChangeIndexBackstop`); existing `knownAccounts` (`reconcile.go:1760`), `listAppSubtree` (`:2022`), `ManifestKey`/`RecordsPrefix` (`provision.go:321`), `SidecarsPrefix` (`sidecar.go:122`), `release.ReadAppTombstone`'s key, `domains.AccountManifestSqid`.
275- Produces:
276 - `type RecordView struct` with `mu sync.Mutex` (leaf lock)
277 - `func NewRecordView(live release.Store, watch *release.ChangeWatch) *RecordView`
278 - `func (v *RecordView) Refresh(ctx context.Context) error` — §4.2 under `mu`: `watch.Poll`, then re-read every scope with `applied < observed` or new; an unusable watch, or a view never built, runs `Rebuild` instead.
279 - `func (v *RecordView) Rebuild(ctx context.Context) error` — §4.4 under `mu`: read `watch.Observed()` (after a `Poll`) first, then walk everything, publish, set `applied` = the pre-walk generations.
280 - `func (v *RecordView) Snapshot() *ViewSnapshot` — atomic load; nil before the first publication.
281 - `type ViewSnapshot struct { Epoch uint64; Complete bool /* plus unexported cached families */ }`
282 - `func (sn *ViewSnapshot) Store(live release.Store) release.Store` — read-through store, below.
283 - `func ScopeApp(scope string) (acct, app string, ok bool)` — parses `apps/<a>/<b>`.
284
285**Cached families** (what a snapshot answers; everything else goes to `live`):
286- `List("accounts/")` and `List("accounts/<a>/")`: complete key listings per account, kept from `ListETags("accounts/<a>/")` on each account-scope re-read.
287- `Get("accounts/<a>/account.json")`: cached body or not-found.
288- `ListDirs("apps/")`, `ListDirs("apps/<a>/")`: accounts and apps that have at least one in-scope app scope; `BucketShapedListDirs()` returns true so `listAppSubtree` walks.
289- `List` of `apps/<a>/<b>/instances/` and `apps/<a>/<b>/sidecars/`: complete, from `ListETags`.
290- `Get` of each record key under those prefixes, of `apps/<a>/<b>/deleted.json` (trash marker), `instances.json` and `sidecars.json`: cached body, cached not-found (an `os.ErrNotExist`-wrapped error), or the cached read error.
291- Every `Put`/`PutFrom`/`Delete`/`DeleteIfMatch` goes to `live`, and the key, and every listing prefix it falls under, is answered from `live` for the rest of that snapshot store's life (read-your-writes and CAS retries).
292- Optional extensions not listed above: forwarded to `live`.
293
294- [ ] **Step 1: Write the failing tests** (DirStore with the index enabled, `ChangeWatch` over it)
295
296```go
297func TestViewMatchesWalk(t *testing.T) // 3 accounts, 5 apps, records/sidecars/manifests/trash: ListAllRecordsStrict over sn.Store(live) == over live
298func TestViewRefreshRereadsOnlyMovedApp(t *testing.T) // counting store: after one record Put in shop, Refresh lists only apps/bab/shop/{instances,sidecars}/ and GETs only the changed key + that app's three manifests/marker
299func TestViewRefreshNoChangeIsOneGet(t *testing.T) // counting store: exactly one GetIfChanged
300func TestViewRetriesFailedScopeOn304(t *testing.T) // inject one List failure for shop after its bump; next Refresh (index 304) re-reads shop; Complete false until then (codex 4)
301func TestViewFirstRefreshWithoutIndexRebuilds(t *testing.T) // no control/changes.json, records exist -> snapshot has them, Complete true (Review Focus 1)
302func TestViewUnusableWatchRebuilds(t *testing.T) // malformed index -> Refresh walks (Review Focus 2)
303func TestViewBypassWriteAppearsAfterRebuild(t *testing.T) // record written with index disabled -> absent after Refresh, present after Rebuild (Review Focus 3)
304func TestViewPurgeDropsApp(t *testing.T) // delete every key under shop -> ListDirs("apps/bab/") no longer has shop (Review Focus 4)
305func TestViewEpochsIncrease(t *testing.T)
306func TestViewBackstopCannotOverwriteNewer(t *testing.T) // codex 2 interleaving: Rebuild blocked mid-walk by a hook; concurrent Refresh must wait on mu; final snapshot has the record
307func TestSnapshotStoreReadYourWrites(t *testing.T) // Put via sn.Store, then Get/List return live state
308func TestSnapshotStoreErrorsMatchLive(t *testing.T) // not-found Get of a cached-family key: errors.Is(err, os.ErrNotExist)
309func TestSnapshotStoreOtherKeysGoLive(t *testing.T) // Get of config.json / List("apps/bab/shop/channels/") hit live
310```
311
312- [ ] **Step 2: Run to verify they fail**
313
314Run: `go test ./internal/provision/ -run 'TestView|TestSnapshotStore' -count=1`
315Expected: FAIL
316
317- [ ] **Step 3: Implement** `recordview.go` (state, `Refresh`, `Rebuild`, publication with `atomic.Pointer[ViewSnapshot]`) and `recordview_store.go` (the read-through store). `Rebuild` reuses `knownAccounts` and `listAppSubtree` over `live` (and `ListETags` to skip unchanged bodies already cached); an app-scope re-read uses `ListETags` on both prefixes and `GetIfChanged` for the three per-app objects. Incomplete stays incomplete until every scope with `applied < observed` succeeds.
318
319- [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS
320
321- [ ] **Step 5: Commit** `git commit -m "provision: RecordView keeps the record families and serves snapshots as a read-through store (#336)"`
322
323---
324
325### Task 7: Invariant simulation and mutation checks
326
327**Files:**
328- Create: `internal/provision/recordview_sim_test.go`
329
330**Interfaces:**
331- Consumes: Task 6.
332
333- [ ] **Step 1: Write the simulation** `TestRecordViewSimulation` (seeded by `-seed` and `-seeds` flags, default 200 in CI, skipped under `-short`): random actors on a `DirStore` — writers that bump, writers that bypass (`EnableChangeIndex` off), writers whose bump fails (injected), incremental `Refresh`, `Rebuild`, and a consumer that takes snapshots under a fake `w.pass` and records epochs. Check after every step: (a) after a `Refresh` that observed generation g for a scope, every bumped write to that scope at ≤ g is in the snapshot; (b) after `Rebuild`, `ListAllRecordsStrict(sn.Store(live))` equals `ListAllRecordsStrict(live)`; (c) snapshot epochs only increase and no scope's contents revert; (d) a snapshot with `Complete == false` never returns a nil error from `ListAllRecordsStrict`; (e) a failed scope is re-read on the next `Refresh` with the index unchanged. The seed set must include the three named interleavings in §8 (codex 1, 2, 4) as fixed seeds or scripted cases.
334
335- [ ] **Step 2: Run** `nice -n 19 go test ./internal/provision/ -run TestRecordViewSimulation -seeds 3000 -count=1`. Expected: PASS
336
337- [ ] **Step 3: Mutation checks** in a scratch copy of the tree under `$TMPDIR` (never in the worktree): apply each mutation, run the sweep, confirm it FAILS, discard the copy. Mutations: bump before write (Task 2); record `applied` for failed scopes (Task 6); skip the pre-walk index read in `Rebuild`; publish the rebuild outside `mu`; consume a handed-over snapshot instead of taking one under `w.pass`. Record the five results in the commit message.
338
339- [ ] **Step 4: Commit** `git commit -m "provision: RecordView invariant simulation; five mutation checks fail as they must (#336)"`
340
341---
342
343### Task 8: Box agent reads through the view
344
345**Files:**
346- Modify: `internal/host/agent.go` (`reconcileSerializedWith` `:1569`, fast-converge deps and `fastTick` `:2537`, `fullPass` `:2787`, `instancesLoop` `:3358`, agent start-up wiring)
347- Modify: `internal/host/movespass.go` (`movesPass` `:1060`), `internal/host/movesoffline.go` (`refreshOfflineView` `:241` callers), `internal/host/steering_loop.go:100`
348- Modify: `internal/appconfig/syncexec.go` (new option beside `:751-810`)
349- Create: `internal/host/changeindex.go`
350- Test: `internal/host/changeindex_test.go`, `internal/appconfig/syncexec_test.go`
351
352**Interfaces:**
353- Consumes: Tasks 4, 6.
354- Produces:
355 - `func changeIndexArmed(getenv func(string) string) bool` — `getenv("CARLOS_CHANGE_INDEX") == "read"`
356 - `var armedChangeWatch atomic.Pointer[release.ChangeWatch]`, `var armedRecordView atomic.Pointer[provision.RecordView]` (set once at start-up when armed; nil means today's behavior everywhere)
357 - `func viewStore(ctx context.Context, live release.Store) (s release.Store, epoch uint64, ok bool)` — `Refresh` then `Snapshot().Store(live)`; `ok` false when disarmed, so callers use `live`.
358 - `func recordViewBackstopLoop(ctx context.Context, v *provision.RecordView)` — `Rebuild` at start and every `release.ChangeIndexBackstop` ±10%, then runs `fullPass` (unrestricted).
359 - `func appconfig.WithRouteFilter(keep func(registry.Route) bool) SyncOption`
360 - `fastDeps.scopedPass func(ctx context.Context, moved []string)`
361 - `movesWorker.lastEpoch uint64`
362
363- [ ] **Step 1: Write the failing tests**
364
365```go
366func TestReconcileUsesViewWhenArmed(t *testing.T) // armed, counting live store: two reconcileSerializedWith calls with no writes -> zero List calls on live
367func TestReconcileWalksWhenDisarmed(t *testing.T) // disarmed -> today's List calls (regression guard)
368func TestMovesPassTakesSnapshotUnderPass(t *testing.T) // codex 1: hook pauses a pass after Reconcile; a second pass runs moves at epoch N+1; first pass's moves then sees epoch >= N+1 (never older)
369func TestMovesPassRefusesOlderEpoch(t *testing.T) // forced older snapshot -> pass skipped with a log line, no move step taken
370func TestFastTickIndexOnlyRunsScopedPass(t *testing.T) // serial unchanged, index moved for shop -> scopedPass called with ["apps/bab/shop"], fullPass not called
371func TestScopedPassTouchesOnlyMovedApp(t *testing.T) // codex 10 witness: registry hosts shop and cafe; bump shop; counting store sees no key under apps/bab/cafe/
372func TestSerialChangeStillRunsFullPass(t *testing.T)
373func TestInstancesLoopFloorIsRefreshWhenArmed(t *testing.T) // nothing changed -> one GetIfChanged on live, no List
374func TestSyncExecWithRouteFilter(t *testing.T) // in appconfig: filter excluding cafe -> no read of cafe's bundle
375```
376
377- [ ] **Step 2: Run to verify they fail**
378
379Run: `go test ./internal/host/ ./internal/appconfig/ -run 'TestReconcileUsesView|TestReconcileWalksWhenDisarmed|TestMovesPass(Takes|Refuses)|TestFastTickIndexOnly|TestScopedPass|TestSerialChangeStill|TestInstancesLoopFloor|TestSyncExecWithRouteFilter' -count=1`
380Expected: FAIL
381
382- [ ] **Step 3: Implement.**
383 - Start-up: when armed, open the watch on the deployment store, build the view, store both pointers, start `recordViewBackstopLoop`. A fleet box (store door) leaves itself disarmed: its watch would be unusable and every `Refresh` would fall back to a full walk anyway.
384 - `reconcileSerializedWith`: inside `reconcileMu`, `s2, _, ok := viewStore(ctx, s)`; pass `s2` (or `s`) to `refreshOfflineView`, `provision.Reconcile` and `provision.ReconcileSidecars`.
385 - `movesPass`: inside `w.pass`, take `viewStore`; refuse and log `agent: moves pass: snapshot epoch %d older than %d consumed` if the epoch is lower than `w.lastEpoch`; set `w.lastEpoch`.
386 - `steeredRoutesFromRecords` callers pass `viewStore`'s store.
387 - `fastTick`: when the watch is armed, `moved, err := watch.Poll(ctx)` every tick; a serial change keeps `fullPass`; otherwise non-empty `moved` (after the 5s clamp) runs `scopedPass(ctx, moved)`.
388 - `scopedPass`: `reconcileSerialized`, `telemetryPass`, `runMovesPass`, `runRunsPass`, aliases as in `fullPass`, then `release.PassFilter` with `extra` keeping routes whose `(Account, App)` is in `ScopeApp(moved)`, then `syncExecSerialized` with `appconfig.WithRouteFilter` on the same set.
389 - `instancesLoop`: unchanged structure; its reconcile now goes through the view, so a quiet floor costs one conditional GET.
390
391- [ ] **Step 4: Run to verify they pass** (same command), then `go test ./internal/host/ -count=1 -run 'Moves|Reconcile|FastTick|Steer'` for regressions. Expected: PASS
392
393- [ ] **Step 5: Commit** `git commit -m "host: reconcile, sidecars, moves and steering read through the record view; index-only changes run a scoped pass (#336)"`
394
395---
396
397### Task 9: Box GET loops gated by generation and age
398
399**Files:**
400- Modify: `internal/host/agent.go` (`genPollPass` `:1720`, `configGenLoop` `:1979`, the `PassFilter` call `:2628` and `fastMembers` `:2214`)
401- Modify: `internal/host/telemetry_pass.go:58`
402- Test: `internal/host/readgate_test.go`
403
404**Interfaces:**
405- Consumes: Task 4's `ReadGate`, Task 8's `armedChangeWatch`.
406- Produces: `genCache` gains a `gate *release.ReadGate`; `fastDeps` gains `canaryGate *release.ReadGate`; `telemetryPass` uses a package `telemetryGate` plus a per-pass account cache. All gates are `NewReadGate(armedChangeWatch.Load(), release.ChangeIndexBackstop)`; with a nil watch every read is due (today's behavior).
407
408- [ ] **Step 1: Write the failing tests**
409
410```go
411func TestConfigGenSkipsUnmovedApp(t *testing.T) // armed, two ticks, no bump -> one GET of config-gen.json (the first)
412func TestConfigGenReadsMovedApp(t *testing.T)
413func TestConfigGenReadsAtMaxAge(t *testing.T) // fake clock past 30m -> GET (codex 9)
414func TestConfigGenRetriesAfterError(t *testing.T)
415func TestCanaryPointerGated(t *testing.T) // canary route stays a member; pointer GET only on move or age
416func TestTelemetryAccountReadOncePerPass(t *testing.T) // 3 apps in one account -> one GET of accounts/<a>/telemetry.json
417func TestTelemetryGatedWithAgeBound(t *testing.T) // codex 9
418func TestGatesDueWhenWatchUnusable(t *testing.T) // Review Focus 5: watch denied -> every read due on every tick
419```
420
421- [ ] **Step 2: Run to verify they fail**
422
423Run: `go test ./internal/host/ -run 'TestConfigGen|TestCanaryPointerGated|TestTelemetry(Account|Gated)|TestGatesDueWhenWatchUnusable' -count=1`
424Expected: FAIL
425
426- [ ] **Step 3: Implement.** Each read calls `gen, due := gate.Due(scope, now)`; skip when not due; after the read `gate.Mark(scope, gen, now, err == nil)`. For canaries, gate the pointer read inside the `PassFilter` membership for `canary/*` routes only; stable routes keep today's path.
427
428- [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS
429
430- [ ] **Step 5: Commit** `git commit -m "host: config-gen, canary pointers and telemetry read only on change or at the backstop age (#336)"`
431
432---
433
434### Task 10: Edge cert mirror and home map
435
436**Files:**
437- Modify: `internal/edge/certsync.go` (`certMirror` struct, `Refresh` `:622`, loop `:411`)
438- Modify: `internal/edge/homemap.go` (`Refresh` `:320`, `refreshHosts` `:264`)
439- Modify: the edge's wiring in `internal/edge/edge.go` to pass the process's watch (nil when disarmed)
440- Test: `internal/edge/certsync_test.go`, `internal/edge/homemap_test.go`
441
442**Interfaces:**
443- Consumes: Tasks 3, 4.
444- Produces: `certMirror.watch *release.ChangeWatch`, `homeMap.watch *release.ChangeWatch`; each keeps the last successful full rebuild time.
445
446- [ ] **Step 1: Write the failing tests**
447
448```go
449func TestCertMirrorSkipsWhenUnmoved(t *testing.T) // armed, two polls, no bump -> second poll: no List, no Get
450func TestCertMirrorFetchesOnlyChanged(t *testing.T) // one cert rewritten -> ListETags + exactly one Get
451func TestCertMirrorDropsRemovedKey(t *testing.T) // deleted challenge token disappears from the snapshot
452func TestCertMirrorFullRebuildAtBackstop(t *testing.T)
453func TestCertMirrorBootRefreshUnchanged(t *testing.T) // boot path still a full read
454func TestHomeMapKeysPolledIndependently(t *testing.T) // codex 8: rotate control/edge-mesh-key, control/edges unchanged -> new key loaded within one poll
455func TestHomeMapHeartbeatOnUnchangedPoll(t *testing.T) // RefreshAttempted advances when hosts are skipped
456```
457
458- [ ] **Step 2: Run to verify they fail**
459
460Run: `go test ./internal/edge/ -run 'TestCertMirror|TestHomeMap(Keys|Heartbeat)' -count=1`
461Expected: FAIL
462
463- [ ] **Step 3: Implement.** Cert mirror: when the watch is usable and `control/certs` has not moved and the last full rebuild is younger than `release.ChangeIndexBackstop`, the poll does nothing; when it moved, `ListETags(certPrefix)`, GET new or changed keys, drop keys absent from the listing; otherwise today's full `Refresh`. Home map: same rule for `refreshHosts` on `control/edges`; `refreshKeys` and `health.RefreshAttempted` run on every poll as today.
464
465- [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS
466
467- [ ] **Step 5: Commit** `git commit -m "edge: cert mirror and home map re-read only what moved; mesh keys keep their own poll (#336)"`
468
469---
470
471### Task 11: Console sweeps
472
473**Files:**
474- Modify: `cmd/console/main.go` (arm watch and view when `CARLOS_CHANGE_INDEX=read`; start the backstop loop)
475- Modify: `internal/console/server.go` (Server fields), `internal/console/runsweep.go` (`:62-114`), `internal/console/seedsweep.go` (`:140-165`), `internal/console/store.go` (new query beside `RecordActivity` `:2591`)
476- Test: `internal/console/runsweep_test.go`, `internal/console/seedsweep_test.go`, `internal/console/store_test.go`
477
478**Interfaces:**
479- Consumes: Tasks 4, 6.
480- Produces:
481 - `Server.recordView *provision.RecordView` (nil when disarmed)
482 - `func (s *Store) AppsWithActionSince(action string, since time.Time) ([]AppRef, error)` with `type AppRef struct{ Sqid, App string }` — distinct `(account sqid, app)` from the `audit` table where `action = ?` and the time column ≥ `since`.
483 - `const orphanCandidateWindow = 48 * time.Hour`, `const orphanFullSweepEvery = 24 * time.Hour`, `Server.lastOrphanFull time.Time`
484
485- [ ] **Step 1: Write the failing tests**
486
487```go
488func TestRunSweepReadsViewWhenArmed(t *testing.T) // counting releases store: no List of apps/ across two sweeps with no writes
489func TestOrphanHousekeepingListsOnlyRecentRunApps(t *testing.T) // shop has run-start 1h ago, cafe none -> ListDirs only under apps/bab/shop/runs/ (codex R2-1)
490func TestOrphanHousekeepingFullSweepDaily(t *testing.T) // fake clock past 24h -> every app with records is swept
491func TestAppsWithActionSince(t *testing.T) // rows older than since excluded; distinct pairs
492func TestSeedSweepSkipsWalkWithoutSnapshots(t *testing.T)// no snapshots -> no strict live walk; abandonment still sees a Seed record via the view
493func TestSeedSweepWalksStrictWithSnapshots(t *testing.T) // snapshots present -> live ListAllRecordsStrict after the snapshot listing, as today
494```
495
496- [ ] **Step 2: Run to verify they fail**
497
498Run: `go test ./internal/console/ -run 'TestRunSweepReadsView|TestOrphanHousekeeping|TestAppsWithActionSince|TestSeedSweep(Skips|Walks)' -count=1`
499Expected: FAIL
500
501- [ ] **Step 3: Implement.** `runSweepPass` takes its record store from `recordView.Refresh` + `Snapshot().Store(live)` when armed. Housekeeping builds its app set from `AppsWithActionSince("run-start", now-48h)` unless `now-lastOrphanFull >= 24h`, in which case it uses every app with records and sets `lastOrphanFull`. Seed sweep: when the snapshot listing is empty and the view is armed, read records from the view snapshot; otherwise keep the strict live walk exactly where it is today (after `seedsPassBetweenListings`).
502
503- [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS
504
505- [ ] **Step 5: Commit** `git commit -m "console: run sweep reads the record view; orphan housekeeping narrows to recent runs; seed sweep skips the walk with nothing to delete (#336)"`
506
507---
508
509### Task 12: Census overlay, release note, rollout notes
510
511**Files:**
512- Create: the plan14 root-reads census overlay for every touched covered package (recipe in the `litestream-284` memory and #280), `census-ruling-s3-change-index`
513- Modify: `README.md` rollout/arming section (where release recipes live), `CHANGELOG` or release-notes location the repo uses
514- Test: the census test package (`internal/provision`'s root-reads census test)
515
516- [ ] **Step 1: Run the census test** `go test ./internal/provision/ -run RootRead -count=1`. Expected: FAIL naming changed packages/units.
517- [ ] **Step 2: Merge main first, then generate the overlay** so it loads after the latest one on main (as of 2026-10-21 the overlays from `ci-speed-activator` and `securelisten-337`; check main's latest when you get there), with the scratch-test recipe (`rootReadProduction`, `loadRootReadInventory`, `rootReadLegacyFingerprint`, `rootReadLegacySeal`), add it, and post the census-ruling request on #280.
518- [ ] **Step 3: Run the census test again.** Expected: PASS
519- [ ] **Step 4: Release note for R4** (old CLIs in direct-bucket mode converge at the 30-minute backstop until updated) and the arming steps from §7: draft the strings, run them through the copy-review skill, apply Paul's edits verbatim.
520- [ ] **Step 5: Full local gate on touched packages** `go test ./internal/release/ ./internal/provision/ ./internal/host/ ./internal/edge/ ./internal/console/ ./internal/appconfig/ ./internal/hostops/ -count=1` (host and console with `-run` filters per the targeted-tests rule if the full packages are too heavy), `go vet ./...`, and the cross-compile check (`GOOS=darwin go build ./...`, `GOOS=windows go build ./...`). Expected: PASS
521- [ ] **Step 5b: Claim invariant.** If any console test added here swaps process-wide state (a package var, log output, a client transport) or uses `t.Setenv` inside a `TestClaimConsoleMutationInvariant` fixture, list it serial (`claimSerialDoors` or the probe's `serial` field) and call `claimMustRunAlone(t, ...)` before the swap. Task 11 adds no mutation doors, so this is a check, not expected work.
522- [ ] **Step 6: Shard weights.** If any new console or root-package test takes several seconds, refresh `.amadan/test-weights.tsv` with `.amadan/test-weights.sh` (CI runs `.amadan/test-gate.sh`, which shards console 8 ways and root 3 ways). Put no CI-only script under `scripts/`: the root-read census counts every `.sh` outside dot directories.
523- [ ] **Step 7: Merge main, commit and push**; wait for CI green on the tip (one `=== <job>` section per job; failures listed under FAILED: at the end).
524