amadan
S3 change index: implementation plan (#336)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+519
−0
docs/superpowers/plans/2026-10-21-s3-change-index.md+519 −0
CI passed — run details
CI log
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok amadan.net/carlos/platform.git/launchctx 0.011s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok github.com/carlosframework/platform 326.619s
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 2.580s
ok github.com/carlosframework/platform/cmd/moveprobe 92.253s
ok github.com/carlosframework/platform/cmd/zddprobe 0.701s
ok github.com/carlosframework/platform/internal/activator 252.047s
ok github.com/carlosframework/platform/internal/agentsock 0.015s
ok github.com/carlosframework/platform/internal/alerting 0.003s
ok github.com/carlosframework/platform/internal/analytics 0.020s
ok github.com/carlosframework/platform/internal/appconfig 2.434s
ok github.com/carlosframework/platform/internal/apperr 0.086s
ok github.com/carlosframework/platform/internal/applog 0.137s
ok github.com/carlosframework/platform/internal/awsinv 0.004s
ok github.com/carlosframework/platform/internal/boxchoice 0.004s
ok github.com/carlosframework/platform/internal/boxid 0.598s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.008s
ok github.com/carlosframework/platform/internal/caps 0.014s
ok github.com/carlosframework/platform/internal/certwarm 0.004s
ok github.com/carlosframework/platform/internal/cgroup 0.693s
ok github.com/carlosframework/platform/internal/changelog 0.353s
ok github.com/carlosframework/platform/internal/claimactivation 0.795s
ok github.com/carlosframework/platform/internal/clamd 0.055s
? github.com/carlosframework/platform/internal/cliflag [no test files]
ok github.com/carlosframework/platform/internal/console 434.576s
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.435s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.008s
ok github.com/carlosframework/platform/internal/dbverify 0.004s
ok github.com/carlosframework/platform/internal/dnsrec 0.005s
ok github.com/carlosframework/platform/internal/docsite 0.006s
ok github.com/carlosframework/platform/internal/domains 2.375s
ok github.com/carlosframework/platform/internal/doorkey 0.006s
ok github.com/carlosframework/platform/internal/economics 0.007s
ok github.com/carlosframework/platform/internal/edge 44.961s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.008s
ok github.com/carlosframework/platform/internal/fleetapi 0.132s
ok github.com/carlosframework/platform/internal/gate 1.656s
ok github.com/carlosframework/platform/internal/genselect 0.021s
ok github.com/carlosframework/platform/internal/geo 0.022s
ok github.com/carlosframework/platform/internal/geoip 0.031s
ok github.com/carlosframework/platform/internal/historyreadback 0.153s
ok github.com/carlosframework/platform/internal/host 191.475s
ok github.com/carlosframework/platform/internal/hostactors 6.096s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.005s
ok github.com/carlosframework/platform/internal/hostbox 2.652s
ok github.com/carlosframework/platform/internal/hostclaim 0.549s
ok github.com/carlosframework/platform/internal/hostenv 0.005s
ok github.com/carlosframework/platform/internal/hostevidence 0.006s
ok github.com/carlosframework/platform/internal/hostexec 4.906s
ok github.com/carlosframework/platform/internal/hostfence 0.035s
ok github.com/carlosframework/platform/internal/hostfiles 10.534s
ok github.com/carlosframework/platform/internal/hostfloor 0.015s
ok github.com/carlosframework/platform/internal/hostguard 1.052s
ok github.com/carlosframework/platform/internal/hostops 1.595s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.002s
ok github.com/carlosframework/platform/internal/hostquarantine 0.002s
ok github.com/carlosframework/platform/internal/hostrelocate 0.481s
ok github.com/carlosframework/platform/internal/hostresolve 0.167s
ok github.com/carlosframework/platform/internal/hostwrite 0.002s
ok github.com/carlosframework/platform/internal/infraview 0.002s
ok github.com/carlosframework/platform/internal/instancectl 5.234s
ok github.com/carlosframework/platform/internal/instanceexec 2.748s
ok github.com/carlosframework/platform/internal/instancefs 0.018s
ok github.com/carlosframework/platform/internal/instancekey 0.015s
ok github.com/carlosframework/platform/internal/instanceupdates 0.019s
ok github.com/carlosframework/platform/internal/layout 1.604s
ok github.com/carlosframework/platform/internal/lbhealth 0.004s
ok github.com/carlosframework/platform/internal/lease 9.927s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.383s
ok github.com/carlosframework/platform/internal/ledger 0.012s
ok github.com/carlosframework/platform/internal/mail 0.176s
ok github.com/carlosframework/platform/internal/maintenance 0.082s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.561s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.078s
ok github.com/carlosframework/platform/internal/metrics 0.005s
ok github.com/carlosframework/platform/internal/moves 21.118s
ok github.com/carlosframework/platform/internal/moves/archive 0.813s
ok github.com/carlosframework/platform/internal/objstore 0.213s
ok github.com/carlosframework/platform/internal/ops 0.183s
ok github.com/carlosframework/platform/internal/park 0.206s
ok github.com/carlosframework/platform/internal/privatefile 0.005s
ok github.com/carlosframework/platform/internal/provision 36.558s
ok github.com/carlosframework/platform/internal/publicrelease 0.007s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.547s
ok github.com/carlosframework/platform/internal/relay 7.227s
ok github.com/carlosframework/platform/internal/release 24.373s
ok github.com/carlosframework/platform/internal/rollout 3.807s
ok github.com/carlosframework/platform/internal/rolloutwords 0.003s
ok github.com/carlosframework/platform/internal/rootaws 0.006s
ok github.com/carlosframework/platform/internal/rootparticipation 0.058s
ok github.com/carlosframework/platform/internal/rootstate 0.027s
ok github.com/carlosframework/platform/internal/runs 0.594s
ok github.com/carlosframework/platform/internal/s3compat 0.005s
ok github.com/carlosframework/platform/internal/safefile 0.004s
ok github.com/carlosframework/platform/internal/schedule 2.551s
ok github.com/carlosframework/platform/internal/seeds 0.012s
ok github.com/carlosframework/platform/internal/sim/hostlife 9.653s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.886s
ok github.com/carlosframework/platform/internal/status 0.057s
ok github.com/carlosframework/platform/internal/steering 0.005s
ok github.com/carlosframework/platform/internal/stopevidence 0.242s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.002s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.005s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.560s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 49807522 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
minio: downloaded, digest verified
mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
mc: downloaded, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.03s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.02s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.034s
=== RUN TestT4StorageProbeMinio
2026/10/21 13:05:11 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E08D35328AF7E7, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.031s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-21-s3-change-index.md b/docs/superpowers/plans/2026-10-21-s3-change-index.md| new file mode 100644 |
| index 000000000..2c392e461 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-21-s3-change-index.md |
| @@ -0,0 +1,519 @@ |
| +# S3 Change Index Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Stop every box and the console from re-walking the whole deployment bucket: writers bump a per-scope generation in `control/changes.json`, readers re-read only what moved, a 30-minute backstop walk bounds staleness. |
| + |
| +**Architecture:** The bump lives inside the two concrete stores (`S3Store`, `DirStore`), enabled by deployment openers. A process-wide `release.ChangeWatch` polls the index with a conditional GET. `provision.RecordView` keeps the record families the walk reads and exposes immutable snapshots as a read-through `release.Store`, so `Reconcile`, `ReconcileSidecars`, the moves pass, steering and the console sweeps run unchanged over a snapshot. A `release.ReadGate` gates the per-app GET loops on generations plus a maximum age. |
| + |
| +**Tech Stack:** Go, AWS SDK v2 S3 (existing client in `internal/release`), `DirStore` for tests, MinIO in the delivered-contract test. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-21-s3-change-index-design.md` (rev 3, Astra-accepted). Section numbers below (§N) refer to it. |
| + |
| +## Global Constraints |
| + |
| +- Index key `control/changes.json`, body `{"schema":1,"entries":{"<scope>":<uint64>}}`; generations only increase; entries never removed in v1 (§3.1). |
| +- Scopes exactly per §3.2: `apps/<a>/<b>` (except data prefixes `logs`, `errors`, `analytics`, `runs`, `metrics`), `accounts/<a>`, `control/certs`, `control/edges`; nothing else. |
| +- Bump after a successful mutation only; at most 4 attempts with jitter inside a 2s budget detached from the caller's cancellation; a failed bump is logged and never fails the write (§3.3). |
| +- Backstop interval 30 minutes, jittered ±10% (R3). Gated reads have a maximum successful-read age equal to the backstop (§5). |
| +- Readers armed only by `CARLOS_CHANGE_INDEX=read` (box `host.env`, console environment). Unset means exactly today's behavior. Writers always bump once shipped (§7). |
| +- A store without `ConditionalGetter` + `ETagLister`, a denied index read, or a malformed index means "unknown": walk as today, never "empty and unchanged" (§3.4). |
| +- Core only: no AWS SDK outside `internal/release`'s existing S3 client (§6). |
| +- Lock order: `w.pass` → `reconcileMu` → `RecordView.mu`; `RecordView.mu` is a leaf (§4.3). |
| +- Tests: run only touched packages locally with `-run`; CI runs the full suite. Leave `GOCACHE` at its default. The simulation runs niced, one 3000-seed sweep per revision. |
| +- No tenant names in code, tests or comments (use `shop`, `example.com`). |
| +- Any user-facing text (release notes for R4) goes through the copy-review skill before it is written. |
| + |
| +## Review Focus |
| + |
| +1. **A deployment with no `control/changes.json` yet** (first arm, fresh bucket): the first `Refresh` must rebuild by walking, and the view must never publish as empty-and-complete. Test in Task 6. |
| +2. **A malformed or future-schema index** (hand edit, newer writer): readers treat it as unusable and walk; writers do not overwrite an unknown schema. Tests in Tasks 1 and 4. |
| +3. **A mixed fleet**: an armed reader while an old binary writes a record without bumping. The record appears after the next `Rebuild`, and no route is retired in between. Tests in Tasks 6 and 7. |
| +4. **An app purge** deleting hundreds of keys under one scope: bumps stay bounded (one per mutation, each a single CAS), the view drops the app once, nothing is retired before the purge's own records are gone. Test in Task 2 (bump count) and Task 6 (view). |
| +5. **Index read denied by IAM** (role lacks `GetObject` on the key): `ChangeWatch.Usable()` stays false, `ReadGate.Due` is always true, every loop behaves as today. Tests in Tasks 4 and 9. |
| + |
| +--- |
| + |
| +### Task 1: Change scope and the index document |
| + |
| +**Files:** |
| +- Create: `internal/release/changeindex.go` |
| +- Test: `internal/release/changeindex_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `const ChangeIndexKey = "control/changes.json"` |
| + - `type ChangeIndex struct { Schema int \`json:"schema"\`; Entries map[string]uint64 \`json:"entries"\` }` |
| + - `func ChangeScope(key string) (scope string, ok bool)` |
| + - `func ReadChangeIndex(ctx context.Context, s Store) (ChangeIndex, string, error)` — absent object returns `ChangeIndex{Schema: 1, Entries: {}}`, etag `""`, nil; schema ≠ 1 returns `ErrChangeIndexSchema`. |
| + - `var ErrChangeIndexSchema = errors.New("release: change index schema unsupported")` |
| + - `func bumpChangeIndex(ctx context.Context, get func(context.Context, string) ([]byte, string, error), put func(context.Context, string, []byte, string) (string, error), scopes ...string) error` — unexported; takes raw get/put so the store hook (Task 2) can call it without recursion. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +func TestChangeScope(t *testing.T) { |
| + cases := map[string]string{ |
| + "apps/bab/shop/instances/a.example.com.json": "apps/bab/shop", |
| + "apps/bab/shop/config-gen.json": "apps/bab/shop", |
| + "apps/bab/shop/channels/canary/x": "apps/bab/shop", |
| + "apps/bab/shop/sidecars.json": "apps/bab/shop", |
| + "accounts/bab/account.json": "accounts/bab", |
| + "accounts/bab/telemetry.json": "accounts/bab", |
| + "control/certs/example.com": "control/certs", |
| + "control/edges/box-a.json": "control/edges", |
| + } |
| + for k, want := range cases { |
| + got, ok := ChangeScope(k) |
| + if !ok || got != want { t.Errorf("ChangeScope(%q) = %q,%v want %q", k, got, ok, want) } |
| + } |
| + for _, k := range []string{ |
| + "apps/bab/shop/logs/box/2026-10-21-10/x", "apps/bab/shop/errors/x", |
| + "apps/bab/shop/analytics/x", "apps/bab/shop/runs/r1/request.json", |
| + "apps/bab/shop/metrics/x", "apps/bab/shop", "apps/bab/", "apps/", |
| + ChangeIndexKey, "control/serial", "control/edge-mesh-key", |
| + "control/edge-mesh-key.next", "releases/x", "status/boxes/a.json", |
| + "fleets/f/apps/bab/shop/config.json", |
| + } { |
| + if s, ok := ChangeScope(k); ok { t.Errorf("ChangeScope(%q) = %q, want none", k, s) } |
| + } |
| +} |
| + |
| +func TestBumpChangeIndexCreatesThenIncrements(t *testing.T) // DirStore: two bumps of "apps/bab/shop" -> 2; other entry untouched |
| +func TestBumpChangeIndexRetriesOnConflict(t *testing.T) // put fails with ErrCASConflict twice, then succeeds -> nil, final gen 1 |
| +func TestBumpChangeIndexGivesUpAfterFourAttempts(t *testing.T) // 4 conflicts -> non-nil error, exactly 4 puts |
| +func TestBumpChangeIndexRefusesUnknownSchema(t *testing.T) // existing {"schema":2,...} -> ErrChangeIndexSchema, no put |
| +func TestReadChangeIndexAbsentIsEmpty(t *testing.T) // Entries non-nil, len 0, etag "" |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify they fail** |
| + |
| +Run: `go test ./internal/release/ -run 'TestChangeScope|TestBumpChangeIndex|TestReadChangeIndex' -count=1` |
| +Expected: FAIL (undefined: ChangeScope) |
| + |
| +- [ ] **Step 3: Implement** `ChangeScope`, `ReadChangeIndex`, `bumpChangeIndex` in `internal/release/changeindex.go`. `ChangeScope` requires `apps/<a>/<b>/<rest>` with all three non-empty segments and `<rest>`'s first segment not in the data set; `accounts/<a>/<rest>` with both non-empty. `bumpChangeIndex` applies all scopes in one read-modify-write; conflict is `errors.Is(err, ErrCASConflict)`; jitter 20–120ms between attempts; it does not create its own context (Task 2 passes the detached one). |
| + |
| +- [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS |
| + |
| +- [ ] **Step 5: Commit** `git commit -m "release: change scopes and the change index document (#336)"` |
| + |
| +--- |
| + |
| +### Task 2: Bump inside the concrete stores |
| + |
| +**Files:** |
| +- Modify: `internal/release/store.go` (`S3Store` struct `:681`, `DirStore` struct `:342`, `Put` `:414/:787`, `PutFrom` `:456/:820`, `Delete` `:660/:970`) |
| +- Modify: `internal/release/conditional_delete.go` (`DeleteIfMatch` `:63/:86`) |
| +- Test: `internal/release/changeindex_store_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1's `ChangeScope`, `bumpChangeIndex`. |
| +- Produces: |
| + - `func (s *S3Store) EnableChangeIndex()`, `func (d *DirStore) EnableChangeIndex()` |
| + - `func NewDeploymentS3Store(ctx context.Context, bucket string) (*S3Store, error)` and `func NewDeploymentS3StoreIn(ctx context.Context, bucket, region string) (*S3Store, error)` — the S3 constructors plus `EnableChangeIndex()`. |
| + - `var changeIndexBumpBudget = 2 * time.Second` (var for tests). |
| + |
| +- [ ] **Step 1: Write the failing tests** (all on `DirStore` with the index enabled) |
| + |
| +```go |
| +func TestStoreBumpsAfterEveryMutation(t *testing.T) { |
| + // Put, PutFrom, Delete, DeleteIfMatch on "apps/bab/shop/instances/h.json" |
| + // each raise entries["apps/bab/shop"] by exactly 1. |
| +} |
| +func TestStoreDoesNotBumpFailedMutation(t *testing.T) // Put with stale ifMatch -> ErrCASConflict, gen unchanged |
| +func TestStoreDoesNotBumpUnscopedKeys(t *testing.T) // logs/, control/serial, the index key itself -> no index object |
| +func TestStoreBumpFailureDoesNotFailWrite(t *testing.T) // index object is a directory (unwritable) -> Put returns nil, record written |
| +func TestStoreBumpDetachedFromCallerCancel(t *testing.T)// ctx cancelled right after Put returns from the backend -> bump still lands |
| +func TestStoreWithoutEnableNeverBumps(t *testing.T) |
| +func TestStoreCapabilitiesPreserved(t *testing.T) { |
| + d := NewDirStore(t.TempDir()); d.EnableChangeIndex() |
| + var s Store = d |
| + _, ok1 := s.(ConditionalDeleter); _, ok2 := s.(PutStreamer); _, ok3 := s.(Opener) |
| + _, ok4 := s.(SizeLister); _, ok5 := s.(PageLister); _, ok6 := s.(ModLister) |
| + if !(ok1 && ok2 && ok3 && ok4 && ok5 && ok6) { t.Fatal("index-enabled DirStore lost an extension") } |
| +} |
| +func TestPurgeBumpsBounded(t *testing.T) // 300 deletes under one app -> gen 300, each bump one CAS (count puts on a counting wrapper of the raw backend) |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify they fail** |
| + |
| +Run: `go test ./internal/release/ -run 'TestStore(Bumps|DoesNot|BumpFailure|BumpDetached|WithoutEnable|Capabilities)|TestPurgeBumps' -count=1` |
| +Expected: FAIL (undefined: EnableChangeIndex) |
| + |
| +- [ ] **Step 3: Implement.** Add an unexported `changeIndex bool` field to both structs. Split each mutation into the existing body (renamed `putRaw`, `putFromRaw`, `deleteRaw`, `deleteIfMatchRaw`) and a public method that calls the raw one and, on success and when enabled and `ChangeScope` matches, calls `bumpChangeIndex` with `getRaw`/`putRaw` under `context.WithTimeout(context.WithoutCancel(ctx), changeIndexBumpBudget)`, logging a failure as `release: change index bump %s: %v`. The struct literal `&release.S3Store{Client:..., Bucket:...}` elsewhere must still compile (exported fields unchanged). |
| + |
| +- [ ] **Step 4: Run to verify they pass** (same command, plus `go test ./internal/release/ -count=1` for regressions). Expected: PASS |
| + |
| +- [ ] **Step 5: Commit** `git commit -m "release: concrete stores bump the change index after each mutation (#336)"` |
| + |
| +--- |
| + |
| +### Task 3: Read extensions and their forwarding |
| + |
| +**Files:** |
| +- Modify: `internal/release/store.go` (new interfaces beside `ModLister` `:186-207`; methods on both stores; `prefixStore` forwarding `:113-217`) |
| +- Modify: `internal/hostops/mutation_boundary.go` (forwarding surface `:108-130`) |
| +- Modify: the MinIO delivered-contract test (`grep -rn TestDeliveredContractMinio`) |
| +- Test: `internal/release/readext_test.go`, `internal/hostops/mutation_boundary_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `type ConditionalGetter interface { GetIfChanged(ctx context.Context, key, etag string) (body []byte, newETag string, changed bool, err error) }` — `etag == ""` behaves as `Get`; unchanged returns `nil, etag, false, nil`; absent returns an `os.ErrNotExist`-wrapped error. |
| + - `type KeyETag struct { Key, ETag string }` |
| + - `type ETagLister interface { ListETags(ctx context.Context, prefix string) ([]KeyETag, error) }` — sorted by key; ETags in the same form `Get` returns. |
| + - Both on `*S3Store` and `*DirStore`; `prefixStore` and `mutationBoundary` forward both (boundary returns `ErrContract` when the raw store lacks one, matching `ListMod`). |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +func TestGetIfChangedDirStore(t *testing.T) // put k; GetIfChanged(k,"") -> changed, etag e; GetIfChanged(k,e) -> !changed, body nil; put again; GetIfChanged(k,e) -> changed |
| +func TestGetIfChangedAbsent(t *testing.T) // errors.Is(err, os.ErrNotExist) |
| +func TestListETagsMatchesGet(t *testing.T) // for each listed key, ETag == etag from Get |
| +func TestPrefixStoreForwardsReadExtensions(t *testing.T) // NewPrefixStore(dir,"fleets/f/") answers both, keys stripped |
| +func TestBoundaryForwardsReadExtensions(t *testing.T) // in hostops |
| +``` |
| +In the MinIO contract test add: `GetIfChanged` returns `changed=false` for the current ETag (S3 304), and `ListETags` ETags equal `Get` ETags. |
| + |
| +- [ ] **Step 2: Run to verify they fail** |
| + |
| +Run: `go test ./internal/release/ ./internal/hostops/ -run 'GetIfChanged|ListETags|ForwardsReadExtensions' -count=1` |
| +Expected: FAIL |
| + |
| +- [ ] **Step 3: Implement.** S3: `GetObject` with `IfNoneMatch: etag`; a 304 surfaces from the SDK as a smithy API error with HTTP status 304 — map it to `changed=false`. `ListETags` pages `ListObjectsV2` and keeps `Key`/`ETag` exactly as `Get` reports them (same quoting). DirStore: compute the ETag the way its `Get` does. |
| + |
| +- [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS. MinIO contract runs in CI. |
| + |
| +- [ ] **Step 5: Commit** `git commit -m "release: conditional GET and ETag listings, forwarded through prefix stores and the hostops boundary (#336)"` |
| + |
| +--- |
| + |
| +### Task 4: ChangeWatch and ReadGate |
| + |
| +**Files:** |
| +- Create: `internal/release/changewatch.go` |
| +- Test: `internal/release/changewatch_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Tasks 1 and 3. |
| +- Produces: |
| + - `const ChangeIndexBackstop = 30 * time.Minute` (R3; here so `provision`, `host`, `edge` and `console` share it without import cycles) |
| + - `type ChangeWatch struct` (safe for concurrent use) |
| + - `func NewChangeWatch(s Store) *ChangeWatch` |
| + - `func (w *ChangeWatch) Poll(ctx context.Context) (moved []string, err error)` — conditional GET of `ChangeIndexKey`; returns scopes whose generation rose (or appeared) since the last successful poll; on 304 returns `nil, nil`; any error, an unknown schema, or a store lacking `ConditionalGetter` marks the watch unusable and returns the error. |
| + - `func (w *ChangeWatch) Gen(scope string) (gen uint64, ok bool)` — `ok` false while unusable. |
| + - `func (w *ChangeWatch) Observed() (map[string]uint64, bool)` — copy of the last good entries. |
| + - `func (w *ChangeWatch) Usable() bool` |
| + - `type ReadGate struct`; `func NewReadGate(w *ChangeWatch, maxAge time.Duration) *ReadGate` |
| + - `func (g *ReadGate) Due(scope string, now time.Time) (gen uint64, due bool)` — due when the watch is nil or unusable, the scope was never marked, its last mark failed, its generation moved past the marked one, or `now` is at least `maxAge` past the last successful mark. |
| + - `func (g *ReadGate) Mark(scope string, gen uint64, now time.Time, ok bool)` — records the generation captured by `Due` before the read. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +func TestChangeWatchFirstPollReportsAll(t *testing.T) // index with 2 entries -> moved has both |
| +func TestChangeWatchUnchangedIs304(t *testing.T) // second poll, no writes -> moved nil, err nil; counting store shows GetIfChanged with etag |
| +func TestChangeWatchReportsOnlyMoved(t *testing.T) |
| +func TestChangeWatchAbsentIndexIsUsableEmpty(t *testing.T) // absent -> Usable true, Observed empty |
| +func TestChangeWatchDeniedIsUnusable(t *testing.T) // store whose Get of the key fails with access denied -> Usable false |
| +func TestChangeWatchUnknownSchemaIsUnusable(t *testing.T) |
| +func TestChangeWatchWithoutExtensionIsUnusable(t *testing.T) |
| +func TestReadGateDue(t *testing.T) // table: unusable->due; never marked->due; marked ok same gen within age->!due; gen moved->due; age>=maxAge->due; last mark failed->due |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify they fail** |
| + |
| +Run: `go test ./internal/release/ -run 'TestChangeWatch|TestReadGate' -count=1` |
| +Expected: FAIL |
| + |
| +- [ ] **Step 3: Implement** in `changewatch.go`. Absent index is usable with empty entries (Review Focus 1 is handled by `RecordView`, not here). |
| + |
| +- [ ] **Step 4: Run to verify they pass.** Expected: PASS |
| + |
| +- [ ] **Step 5: Commit** `git commit -m "release: ChangeWatch polls the change index; ReadGate gates reads on generation and age (#336)"` |
| + |
| +--- |
| + |
| +### Task 5: Every deployment opener enables the index |
| + |
| +**Files:** |
| +- Modify: deployment-bucket openers among the `release.NewS3Store`/`NewS3StoreIn` call sites: `cli.go:1957,2067,3591`, `cmd/console/main.go:633` (the deployment bucket; `:333` is the database bucket), `internal/host/agent.go:4368,4462`, `internal/edge/edge.go:3407`, `internal/host/relocatepass.go:251,680`, `internal/host/activation_command_linux.go:67,92,206,283`, `internal/ops/ops.go:339`, `internal/vet/vet.go:226`, `internal/host/seedwiring.go:102`. Classify each by reading it; switch deployment ones to `NewDeploymentS3Store[In]`. |
| +- Modify: `internal/host/boxpublish.go` only if its test shows the concrete assertion `:259` failing (it should not: the type is still `*release.S3Store`). |
| +- Create: `internal/release/openers_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 2's `NewDeploymentS3Store[In]`. |
| + |
| +- [ ] **Step 1: Write the failing test** `TestS3StoreOpenersClassified`: parse every non-test `.go` file in the module with `go/parser`, find calls to `release.NewS3Store`, `release.NewS3StoreIn` (and the package-internal `NewS3Store` outside `store.go`), and require each `file:func` to appear in a table `otherBucketOpeners map[string]string` (value: the reason, for example `"database bucket"`, `"hibernate data bucket"`). Deployment openers must call `NewDeploymentS3Store[In]` instead, so they never appear. |
| + |
| +- [ ] **Step 2: Run to verify it fails** |
| + |
| +Run: `go test ./internal/release/ -run TestS3StoreOpenersClassified -count=1` |
| +Expected: FAIL listing the unclassified call sites |
| + |
| +- [ ] **Step 3: Switch** each deployment opener to `NewDeploymentS3Store[In]` and list the rest in `otherBucketOpeners` with a reason. |
| + |
| +- [ ] **Step 4: Box publication witness** `TestBoxPublishThroughDeploymentOpener` in `internal/host`: `openDeployStoreEnv` against a `DirStore`-backed fake is out of reach, so assert at the type level that `NewDeploymentS3Store`'s result satisfies the `*release.S3Store` assertion used at `boxpublish.go:259` (compile-time `var _ *release.S3Store = mustDeploy()` plus a runtime check through the same `opened.(*release.S3Store)` expression). |
| + |
| +- [ ] **Step 5: Run** `go test ./internal/release/ ./internal/host/ -run 'TestS3StoreOpenersClassified|TestBoxPublishThroughDeploymentOpener' -count=1` and `go build ./...`. Expected: PASS, build clean. |
| + |
| +- [ ] **Step 6: IAM check.** Read `infra/modules/deployment-buckets/main.tf` and `infra/modules/region-host` policies: confirm every writing role (box, console) is allowed `s3:GetObject` and `s3:PutObject` on `control/changes.json` and no bucket-policy Deny covers it. If anything denies it, add an allow and a `tests/*.tftest.hcl` assertion in that module. Record the result (and the same check owed on the Tito infrastructure repo) in the branch's #336 thread. |
| + |
| +- [ ] **Step 7: Commit** `git commit -m "Deployment openers enable the change index; opener classification test (#336)"` |
| + |
| +--- |
| + |
| +### Task 6: RecordView and its snapshot store |
| + |
| +**Files:** |
| +- Create: `internal/provision/recordview.go`, `internal/provision/recordview_store.go` |
| +- Test: `internal/provision/recordview_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Tasks 1, 3, 4 (including `release.ChangeIndexBackstop`); existing `knownAccounts` (`reconcile.go:1760`), `listAppSubtree` (`:2022`), `ManifestKey`/`RecordsPrefix` (`provision.go:321`), `SidecarsPrefix` (`sidecar.go:122`), `release.ReadAppTombstone`'s key, `domains.AccountManifestSqid`. |
| +- Produces: |
| + - `type RecordView struct` with `mu sync.Mutex` (leaf lock) |
| + - `func NewRecordView(live release.Store, watch *release.ChangeWatch) *RecordView` |
| + - `func (v *RecordView) Refresh(ctx context.Context) error` — §4.2 under `mu`: `watch.Poll`, then re-read every scope with `applied < observed` or new; an unusable watch, or a view never built, runs `Rebuild` instead. |
| + - `func (v *RecordView) Rebuild(ctx context.Context) error` — §4.4 under `mu`: read `watch.Observed()` (after a `Poll`) first, then walk everything, publish, set `applied` = the pre-walk generations. |
| + - `func (v *RecordView) Snapshot() *ViewSnapshot` — atomic load; nil before the first publication. |
| + - `type ViewSnapshot struct { Epoch uint64; Complete bool /* plus unexported cached families */ }` |
| + - `func (sn *ViewSnapshot) Store(live release.Store) release.Store` — read-through store, below. |
| + - `func ScopeApp(scope string) (acct, app string, ok bool)` — parses `apps/<a>/<b>`. |
| + |
| +**Cached families** (what a snapshot answers; everything else goes to `live`): |
| +- `List("accounts/")` and `List("accounts/<a>/")`: complete key listings per account, kept from `ListETags("accounts/<a>/")` on each account-scope re-read. |
| +- `Get("accounts/<a>/account.json")`: cached body or not-found. |
| +- `ListDirs("apps/")`, `ListDirs("apps/<a>/")`: accounts and apps that have at least one in-scope app scope; `BucketShapedListDirs()` returns true so `listAppSubtree` walks. |
| +- `List` of `apps/<a>/<b>/instances/` and `apps/<a>/<b>/sidecars/`: complete, from `ListETags`. |
| +- `Get` of each record key under those prefixes, of `apps/<a>/<b>/deleted.json` (trash marker), `instances.json` and `sidecars.json`: cached body, cached not-found (an `os.ErrNotExist`-wrapped error), or the cached read error. |
| +- Every `Put`/`PutFrom`/`Delete`/`DeleteIfMatch` goes to `live`, and the key, and every listing prefix it falls under, is answered from `live` for the rest of that snapshot store's life (read-your-writes and CAS retries). |
| +- Optional extensions not listed above: forwarded to `live`. |
| + |
| +- [ ] **Step 1: Write the failing tests** (DirStore with the index enabled, `ChangeWatch` over it) |
| + |
| +```go |
| +func TestViewMatchesWalk(t *testing.T) // 3 accounts, 5 apps, records/sidecars/manifests/trash: ListAllRecordsStrict over sn.Store(live) == over live |
| +func TestViewRefreshRereadsOnlyMovedApp(t *testing.T) // counting store: after one record Put in shop, Refresh lists only apps/bab/shop/{instances,sidecars}/ and GETs only the changed key + that app's three manifests/marker |
| +func TestViewRefreshNoChangeIsOneGet(t *testing.T) // counting store: exactly one GetIfChanged |
| +func TestViewRetriesFailedScopeOn304(t *testing.T) // inject one List failure for shop after its bump; next Refresh (index 304) re-reads shop; Complete false until then (codex 4) |
| +func TestViewFirstRefreshWithoutIndexRebuilds(t *testing.T) // no control/changes.json, records exist -> snapshot has them, Complete true (Review Focus 1) |
| +func TestViewUnusableWatchRebuilds(t *testing.T) // malformed index -> Refresh walks (Review Focus 2) |
| +func TestViewBypassWriteAppearsAfterRebuild(t *testing.T) // record written with index disabled -> absent after Refresh, present after Rebuild (Review Focus 3) |
| +func TestViewPurgeDropsApp(t *testing.T) // delete every key under shop -> ListDirs("apps/bab/") no longer has shop (Review Focus 4) |
| +func TestViewEpochsIncrease(t *testing.T) |
| +func TestViewBackstopCannotOverwriteNewer(t *testing.T) // codex 2 interleaving: Rebuild blocked mid-walk by a hook; concurrent Refresh must wait on mu; final snapshot has the record |
| +func TestSnapshotStoreReadYourWrites(t *testing.T) // Put via sn.Store, then Get/List return live state |
| +func TestSnapshotStoreErrorsMatchLive(t *testing.T) // not-found Get of a cached-family key: errors.Is(err, os.ErrNotExist) |
| +func TestSnapshotStoreOtherKeysGoLive(t *testing.T) // Get of config.json / List("apps/bab/shop/channels/") hit live |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify they fail** |
| + |
| +Run: `go test ./internal/provision/ -run 'TestView|TestSnapshotStore' -count=1` |
| +Expected: FAIL |
| + |
| +- [ ] **Step 3: Implement** `recordview.go` (state, `Refresh`, `Rebuild`, publication with `atomic.Pointer[ViewSnapshot]`) and `recordview_store.go` (the read-through store). `Rebuild` reuses `knownAccounts` and `listAppSubtree` over `live` (and `ListETags` to skip unchanged bodies already cached); an app-scope re-read uses `ListETags` on both prefixes and `GetIfChanged` for the three per-app objects. Incomplete stays incomplete until every scope with `applied < observed` succeeds. |
| + |
| +- [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS |
| + |
| +- [ ] **Step 5: Commit** `git commit -m "provision: RecordView keeps the record families and serves snapshots as a read-through store (#336)"` |
| + |
| +--- |
| + |
| +### Task 7: Invariant simulation and mutation checks |
| + |
| +**Files:** |
| +- Create: `internal/provision/recordview_sim_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 6. |
| + |
| +- [ ] **Step 1: Write the simulation** `TestRecordViewSimulation` (seeded by `-seed` and `-seeds` flags, default 200 in CI, skipped under `-short`): random actors on a `DirStore` — writers that bump, writers that bypass (`EnableChangeIndex` off), writers whose bump fails (injected), incremental `Refresh`, `Rebuild`, and a consumer that takes snapshots under a fake `w.pass` and records epochs. Check after every step: (a) after a `Refresh` that observed generation g for a scope, every bumped write to that scope at ≤ g is in the snapshot; (b) after `Rebuild`, `ListAllRecordsStrict(sn.Store(live))` equals `ListAllRecordsStrict(live)`; (c) snapshot epochs only increase and no scope's contents revert; (d) a snapshot with `Complete == false` never returns a nil error from `ListAllRecordsStrict`; (e) a failed scope is re-read on the next `Refresh` with the index unchanged. The seed set must include the three named interleavings in §8 (codex 1, 2, 4) as fixed seeds or scripted cases. |
| + |
| +- [ ] **Step 2: Run** `nice -n 19 go test ./internal/provision/ -run TestRecordViewSimulation -seeds 3000 -count=1`. Expected: PASS |
| + |
| +- [ ] **Step 3: Mutation checks** in a scratch copy of the tree under `$TMPDIR` (never in the worktree): apply each mutation, run the sweep, confirm it FAILS, discard the copy. Mutations: bump before write (Task 2); record `applied` for failed scopes (Task 6); skip the pre-walk index read in `Rebuild`; publish the rebuild outside `mu`; consume a handed-over snapshot instead of taking one under `w.pass`. Record the five results in the commit message. |
| + |
| +- [ ] **Step 4: Commit** `git commit -m "provision: RecordView invariant simulation; five mutation checks fail as they must (#336)"` |
| + |
| +--- |
| + |
| +### Task 8: Box agent reads through the view |
| + |
| +**Files:** |
| +- Modify: `internal/host/agent.go` (`reconcileSerializedWith` `:1569`, fast-converge deps and `fastTick` `:2537`, `fullPass` `:2787`, `instancesLoop` `:3358`, agent start-up wiring) |
| +- Modify: `internal/host/movespass.go` (`movesPass` `:1060`), `internal/host/movesoffline.go` (`refreshOfflineView` `:241` callers), `internal/host/steering_loop.go:100` |
| +- Modify: `internal/appconfig/syncexec.go` (new option beside `:751-810`) |
| +- Create: `internal/host/changeindex.go` |
| +- Test: `internal/host/changeindex_test.go`, `internal/appconfig/syncexec_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Tasks 4, 6. |
| +- Produces: |
| + - `func changeIndexArmed(getenv func(string) string) bool` — `getenv("CARLOS_CHANGE_INDEX") == "read"` |
| + - `var armedChangeWatch atomic.Pointer[release.ChangeWatch]`, `var armedRecordView atomic.Pointer[provision.RecordView]` (set once at start-up when armed; nil means today's behavior everywhere) |
| + - `func viewStore(ctx context.Context, live release.Store) (s release.Store, epoch uint64, ok bool)` — `Refresh` then `Snapshot().Store(live)`; `ok` false when disarmed, so callers use `live`. |
| + - `func recordViewBackstopLoop(ctx context.Context, v *provision.RecordView)` — `Rebuild` at start and every `release.ChangeIndexBackstop` ±10%, then runs `fullPass` (unrestricted). |
| + - `func appconfig.WithRouteFilter(keep func(registry.Route) bool) SyncOption` |
| + - `fastDeps.scopedPass func(ctx context.Context, moved []string)` |
| + - `movesWorker.lastEpoch uint64` |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +func TestReconcileUsesViewWhenArmed(t *testing.T) // armed, counting live store: two reconcileSerializedWith calls with no writes -> zero List calls on live |
| +func TestReconcileWalksWhenDisarmed(t *testing.T) // disarmed -> today's List calls (regression guard) |
| +func TestMovesPassTakesSnapshotUnderPass(t *testing.T) // codex 1: hook pauses a pass after Reconcile; a second pass runs moves at epoch N+1; first pass's moves then sees epoch >= N+1 (never older) |
| +func TestMovesPassRefusesOlderEpoch(t *testing.T) // forced older snapshot -> pass skipped with a log line, no move step taken |
| +func TestFastTickIndexOnlyRunsScopedPass(t *testing.T) // serial unchanged, index moved for shop -> scopedPass called with ["apps/bab/shop"], fullPass not called |
| +func TestScopedPassTouchesOnlyMovedApp(t *testing.T) // codex 10 witness: registry hosts shop and cafe; bump shop; counting store sees no key under apps/bab/cafe/ |
| +func TestSerialChangeStillRunsFullPass(t *testing.T) |
| +func TestInstancesLoopFloorIsRefreshWhenArmed(t *testing.T) // nothing changed -> one GetIfChanged on live, no List |
| +func TestSyncExecWithRouteFilter(t *testing.T) // in appconfig: filter excluding cafe -> no read of cafe's bundle |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify they fail** |
| + |
| +Run: `go test ./internal/host/ ./internal/appconfig/ -run 'TestReconcileUsesView|TestReconcileWalksWhenDisarmed|TestMovesPass(Takes|Refuses)|TestFastTickIndexOnly|TestScopedPass|TestSerialChangeStill|TestInstancesLoopFloor|TestSyncExecWithRouteFilter' -count=1` |
| +Expected: FAIL |
| + |
| +- [ ] **Step 3: Implement.** |
| + - Start-up: when armed, open the watch on the deployment store, build the view, store both pointers, start `recordViewBackstopLoop`. A fleet box (store door) leaves itself disarmed: its watch would be unusable and every `Refresh` would fall back to a full walk anyway. |
| + - `reconcileSerializedWith`: inside `reconcileMu`, `s2, _, ok := viewStore(ctx, s)`; pass `s2` (or `s`) to `refreshOfflineView`, `provision.Reconcile` and `provision.ReconcileSidecars`. |
| + - `movesPass`: inside `w.pass`, take `viewStore`; refuse and log `agent: moves pass: snapshot epoch %d older than %d consumed` if the epoch is lower than `w.lastEpoch`; set `w.lastEpoch`. |
| + - `steeredRoutesFromRecords` callers pass `viewStore`'s store. |
| + - `fastTick`: when the watch is armed, `moved, err := watch.Poll(ctx)` every tick; a serial change keeps `fullPass`; otherwise non-empty `moved` (after the 5s clamp) runs `scopedPass(ctx, moved)`. |
| + - `scopedPass`: `reconcileSerialized`, `telemetryPass`, `runMovesPass`, `runRunsPass`, aliases as in `fullPass`, then `release.PassFilter` with `extra` keeping routes whose `(Account, App)` is in `ScopeApp(moved)`, then `syncExecSerialized` with `appconfig.WithRouteFilter` on the same set. |
| + - `instancesLoop`: unchanged structure; its reconcile now goes through the view, so a quiet floor costs one conditional GET. |
| + |
| +- [ ] **Step 4: Run to verify they pass** (same command), then `go test ./internal/host/ -count=1 -run 'Moves|Reconcile|FastTick|Steer'` for regressions. Expected: PASS |
| + |
| +- [ ] **Step 5: Commit** `git commit -m "host: reconcile, sidecars, moves and steering read through the record view; index-only changes run a scoped pass (#336)"` |
| + |
| +--- |
| + |
| +### Task 9: Box GET loops gated by generation and age |
| + |
| +**Files:** |
| +- Modify: `internal/host/agent.go` (`genPollPass` `:1720`, `configGenLoop` `:1979`, the `PassFilter` call `:2628` and `fastMembers` `:2214`) |
| +- Modify: `internal/host/telemetry_pass.go:58` |
| +- Test: `internal/host/readgate_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 4's `ReadGate`, Task 8's `armedChangeWatch`. |
| +- Produces: `genCache` gains a `gate *release.ReadGate`; `fastDeps` gains `canaryGate *release.ReadGate`; `telemetryPass` uses a package `telemetryGate` plus a per-pass account cache. All gates are `NewReadGate(armedChangeWatch.Load(), release.ChangeIndexBackstop)`; with a nil watch every read is due (today's behavior). |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +func TestConfigGenSkipsUnmovedApp(t *testing.T) // armed, two ticks, no bump -> one GET of config-gen.json (the first) |
| +func TestConfigGenReadsMovedApp(t *testing.T) |
| +func TestConfigGenReadsAtMaxAge(t *testing.T) // fake clock past 30m -> GET (codex 9) |
| +func TestConfigGenRetriesAfterError(t *testing.T) |
| +func TestCanaryPointerGated(t *testing.T) // canary route stays a member; pointer GET only on move or age |
| +func TestTelemetryAccountReadOncePerPass(t *testing.T) // 3 apps in one account -> one GET of accounts/<a>/telemetry.json |
| +func TestTelemetryGatedWithAgeBound(t *testing.T) // codex 9 |
| +func TestGatesDueWhenWatchUnusable(t *testing.T) // Review Focus 5: watch denied -> every read due on every tick |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify they fail** |
| + |
| +Run: `go test ./internal/host/ -run 'TestConfigGen|TestCanaryPointerGated|TestTelemetry(Account|Gated)|TestGatesDueWhenWatchUnusable' -count=1` |
| +Expected: FAIL |
| + |
| +- [ ] **Step 3: Implement.** Each read calls `gen, due := gate.Due(scope, now)`; skip when not due; after the read `gate.Mark(scope, gen, now, err == nil)`. For canaries, gate the pointer read inside the `PassFilter` membership for `canary/*` routes only; stable routes keep today's path. |
| + |
| +- [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS |
| + |
| +- [ ] **Step 5: Commit** `git commit -m "host: config-gen, canary pointers and telemetry read only on change or at the backstop age (#336)"` |
| + |
| +--- |
| + |
| +### Task 10: Edge cert mirror and home map |
| + |
| +**Files:** |
| +- Modify: `internal/edge/certsync.go` (`certMirror` struct, `Refresh` `:622`, loop `:411`) |
| +- Modify: `internal/edge/homemap.go` (`Refresh` `:320`, `refreshHosts` `:264`) |
| +- Modify: the edge's wiring in `internal/edge/edge.go` to pass the process's watch (nil when disarmed) |
| +- Test: `internal/edge/certsync_test.go`, `internal/edge/homemap_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Tasks 3, 4. |
| +- Produces: `certMirror.watch *release.ChangeWatch`, `homeMap.watch *release.ChangeWatch`; each keeps the last successful full rebuild time. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +func TestCertMirrorSkipsWhenUnmoved(t *testing.T) // armed, two polls, no bump -> second poll: no List, no Get |
| +func TestCertMirrorFetchesOnlyChanged(t *testing.T) // one cert rewritten -> ListETags + exactly one Get |
| +func TestCertMirrorDropsRemovedKey(t *testing.T) // deleted challenge token disappears from the snapshot |
| +func TestCertMirrorFullRebuildAtBackstop(t *testing.T) |
| +func TestCertMirrorBootRefreshUnchanged(t *testing.T) // boot path still a full read |
| +func TestHomeMapKeysPolledIndependently(t *testing.T) // codex 8: rotate control/edge-mesh-key, control/edges unchanged -> new key loaded within one poll |
| +func TestHomeMapHeartbeatOnUnchangedPoll(t *testing.T) // RefreshAttempted advances when hosts are skipped |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify they fail** |
| + |
| +Run: `go test ./internal/edge/ -run 'TestCertMirror|TestHomeMap(Keys|Heartbeat)' -count=1` |
| +Expected: FAIL |
| + |
| +- [ ] **Step 3: Implement.** Cert mirror: when the watch is usable and `control/certs` has not moved and the last full rebuild is younger than `release.ChangeIndexBackstop`, the poll does nothing; when it moved, `ListETags(certPrefix)`, GET new or changed keys, drop keys absent from the listing; otherwise today's full `Refresh`. Home map: same rule for `refreshHosts` on `control/edges`; `refreshKeys` and `health.RefreshAttempted` run on every poll as today. |
| + |
| +- [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS |
| + |
| +- [ ] **Step 5: Commit** `git commit -m "edge: cert mirror and home map re-read only what moved; mesh keys keep their own poll (#336)"` |
| + |
| +--- |
| + |
| +### Task 11: Console sweeps |
| + |
| +**Files:** |
| +- Modify: `cmd/console/main.go` (arm watch and view when `CARLOS_CHANGE_INDEX=read`; start the backstop loop) |
| +- Modify: `internal/console/server.go` (Server fields), `internal/console/runsweep.go` (`:62-114`), `internal/console/seedsweep.go` (`:140-165`), `internal/console/store.go` (new query beside `RecordActivity` `:2591`) |
| +- Test: `internal/console/runsweep_test.go`, `internal/console/seedsweep_test.go`, `internal/console/store_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Tasks 4, 6. |
| +- Produces: |
| + - `Server.recordView *provision.RecordView` (nil when disarmed) |
| + - `func (s *Store) AppsWithActionSince(action string, since time.Time) ([]AppRef, error)` with `type AppRef struct{ Sqid, App string }` — distinct `(account sqid, app)` from the `audit` table where `action = ?` and the time column ≥ `since`. |
| + - `const orphanCandidateWindow = 48 * time.Hour`, `const orphanFullSweepEvery = 24 * time.Hour`, `Server.lastOrphanFull time.Time` |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +func TestRunSweepReadsViewWhenArmed(t *testing.T) // counting releases store: no List of apps/ across two sweeps with no writes |
| +func TestOrphanHousekeepingListsOnlyRecentRunApps(t *testing.T) // shop has run-start 1h ago, cafe none -> ListDirs only under apps/bab/shop/runs/ (codex R2-1) |
| +func TestOrphanHousekeepingFullSweepDaily(t *testing.T) // fake clock past 24h -> every app with records is swept |
| +func TestAppsWithActionSince(t *testing.T) // rows older than since excluded; distinct pairs |
| +func TestSeedSweepSkipsWalkWithoutSnapshots(t *testing.T)// no snapshots -> no strict live walk; abandonment still sees a Seed record via the view |
| +func TestSeedSweepWalksStrictWithSnapshots(t *testing.T) // snapshots present -> live ListAllRecordsStrict after the snapshot listing, as today |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify they fail** |
| + |
| +Run: `go test ./internal/console/ -run 'TestRunSweepReadsView|TestOrphanHousekeeping|TestAppsWithActionSince|TestSeedSweep(Skips|Walks)' -count=1` |
| +Expected: FAIL |
| + |
| +- [ ] **Step 3: Implement.** `runSweepPass` takes its record store from `recordView.Refresh` + `Snapshot().Store(live)` when armed. Housekeeping builds its app set from `AppsWithActionSince("run-start", now-48h)` unless `now-lastOrphanFull >= 24h`, in which case it uses every app with records and sets `lastOrphanFull`. Seed sweep: when the snapshot listing is empty and the view is armed, read records from the view snapshot; otherwise keep the strict live walk exactly where it is today (after `seedsPassBetweenListings`). |
| + |
| +- [ ] **Step 4: Run to verify they pass** (same command). Expected: PASS |
| + |
| +- [ ] **Step 5: Commit** `git commit -m "console: run sweep reads the record view; orphan housekeeping narrows to recent runs; seed sweep skips the walk with nothing to delete (#336)"` |
| + |
| +--- |
| + |
| +### Task 12: Census overlay, release note, rollout notes |
| + |
| +**Files:** |
| +- Create: the plan14 root-reads census overlay for every touched covered package (recipe in the `litestream-284` memory and #280), `census-ruling-s3-change-index` |
| +- Modify: `README.md` rollout/arming section (where release recipes live), `CHANGELOG` or release-notes location the repo uses |
| +- Test: the census test package (`internal/provision`'s root-reads census test) |
| + |
| +- [ ] **Step 1: Run the census test** `go test ./internal/provision/ -run RootRead -count=1`. Expected: FAIL naming changed packages/units. |
| +- [ ] **Step 2: Generate the overlay** with the scratch-test recipe (`rootReadProduction`, `loadRootReadInventory`, `rootReadLegacyFingerprint`, `rootReadLegacySeal`), add it, and post the census-ruling request on #280. |
| +- [ ] **Step 3: Run the census test again.** Expected: PASS |
| +- [ ] **Step 4: Release note for R4** (old CLIs in direct-bucket mode converge at the 30-minute backstop until updated) and the arming steps from §7: draft the strings, run them through the copy-review skill, apply Paul's edits verbatim. |
| +- [ ] **Step 5: Full local gate on touched packages** `go test ./internal/release/ ./internal/provision/ ./internal/host/ ./internal/edge/ ./internal/console/ ./internal/appconfig/ ./internal/hostops/ -count=1` (host and console with `-run` filters per the targeted-tests rule if the full packages are too heavy), `go vet ./...`, and the cross-compile check (`GOOS=darwin go build ./...`, `GOOS=windows go build ./...`). Expected: PASS |
| +- [ ] **Step 6: Commit and push**; wait for CI green on the tip. |