| 1 | # S3 change index: stop re-walking the deployment bucket |
| 2 | |
| 3 | Amadan discussion: carlos/platform #336. Branch `s3-polling-336`. |
| 4 | Status: rev 3, 2026-10-21. Astra accepted rev 2 at spec level (round 2); |
| 5 | rev 3 closes round 2's two cost gaps, marked **(codex R2-N)**. Round 1 |
| 6 | findings are marked **(codex N)**; all were accepted. |
| 7 | |
| 8 | ## 0. Problem and evidence |
| 9 | |
| 10 | Flagship S3 cost $211 for October by the 20th and runs at ~$16/day, up |
| 11 | from ~$2/day in early September. S3 server access logs (2026-10-20 21:14Z |
| 12 | to 2026-10-21 11:00Z, 5.5M requests, matching Cost Explorer) put nearly all |
| 13 | of it on the deployment bucket, made by the box roles: |
| 14 | |
| 15 | | Family (per day, whole fleet) | Count | ≈ /month | |
| 16 | |---|---|---| |
| 17 | | LIST `apps/<acct>/<app>/instances/` | 1.17M | $175 | |
| 18 | | LIST `apps/<acct>/` (delimiter) | 0.49M | $73 | |
| 19 | | LIST `apps/<acct>/<app>/sidecars/` | 0.34M | $52 | |
| 20 | | GET `control/certs/*` | 1.55M | $19 | |
| 21 | | GET `accounts/<sqid>/*` | 1.41M | $17 | |
| 22 | | GET `apps/*/*/config-gen.json` | 1.23M | $15 | |
| 23 | | GET `apps/*/*/channels/canary` | 0.60M | $7 | |
| 24 | |
| 25 | The cost scales with apps × boxes × passes, not with use: about $3 per app |
| 26 | per month. The causes, all on main at 3d444a1b: |
| 27 | |
| 28 | - `provision.listAppSubtree` (`internal/provision/reconcile.go:2022`) lists |
| 29 | every account and every app in the bucket; `knownAccounts` (`:1760`) |
| 30 | re-GETs every `account.json`. |
| 31 | - Every box runs that walk three times per full pass: `provision.Reconcile` |
| 32 | and `ReconcileSidecars` under `reconcileMu` |
| 33 | (`internal/host/agent.go:1542,1570`), then `ListAllRecordsStrict` in the |
| 34 | moves pass (`internal/host/movespass.go:1079`). `refreshOfflineView` can |
| 35 | add a fourth. |
| 36 | - Full passes fire on any change to the global `control/serial` (any |
| 37 | tenant's console action, read every 2s by `fastTick`) and on the ungated |
| 38 | 5–6 minute `instancesLoop` (`agent.go:3358`). Fitted: ~570 passes per box |
| 39 | per day. |
| 40 | - The console's `StartRunSweep` (`internal/console/runsweep.go:17`) walks |
| 41 | every app every 30s. |
| 42 | - `configGenLoop` (`agent.go:1979`) GETs each hosted app's |
| 43 | `config-gen.json` every 3–4s; `PassFilter` (`agent.go:2628`) GETs every |
| 44 | canary channel pointer on every 2s tick; the edge cert mirror |
| 45 | (`internal/edge/certsync.go:411`) GETs every cert every 30s. |
| 46 | |
| 47 | ## 1. Rulings (Paul, 2026-10-21) |
| 48 | |
| 49 | - **R1.** The bucket stays the store of record. No database, cache tier or |
| 50 | DynamoDB in this work (weighed on #336: DynamoDB is ~20x cheaper per |
| 51 | read but is a second required component and a large migration; a cache |
| 52 | needs the same change signal this spec adds). |
| 53 | - **R2.** Fix the change detection: writers record which app changed in |
| 54 | the bucket, readers re-read only what changed, a slow full walk remains |
| 55 | as the backstop. |
| 56 | - **R3.** The backstop interval is 30 minutes (answer to §10 Q1). |
| 57 | - **R4.** Old CLIs writing straight to the bucket converge at the backstop |
| 58 | until operators update; the release notes say so (answer to §10 Q2). |
| 59 | |
| 60 | ## 2. Goals and non-goals |
| 61 | |
| 62 | Goals: |
| 63 | - Steady-state request rate independent of the number of apps when |
| 64 | nothing changes; work proportional to what changed when something does. |
| 65 | - No loss of correctness: every reader converges to the bucket's state; |
| 66 | the bound on staleness for writers that bypass the index is the backstop |
| 67 | interval. |
| 68 | - Provider-neutral: plain S3 API (conditional PUT, conditional GET, |
| 69 | ListObjectsV2), works on MinIO and `DirStore`. |
| 70 | - Target: flagship S3 below ~$100/month, roughly storage, litestream and a |
| 71 | polling floor. |
| 72 | |
| 73 | Non-goals (follow-ups, named so they are not lost): `domains.Reconcile`'s |
| 74 | flat `List("apps/")` (~60 pages every 5–6 min on the housekeeper box, ~$3 |
| 75 | a month), console `materialize`/`settle` `List("apps/")`, the litestream |
| 76 | data bucket, and the `control/serial` mechanism itself, which stays. |
| 77 | |
| 78 | ## 3. The change index |
| 79 | |
| 80 | ### 3.1 Object |
| 81 | |
| 82 | One object per deployment bucket: `control/changes.json`. |
| 83 | |
| 84 | ```json |
| 85 | {"schema": 1, |
| 86 | "entries": {"apps/bab/shop": 41, "accounts/bab": 3, |
| 87 | "control/certs": 977, "control/edges": 12}} |
| 88 | ``` |
| 89 | |
| 90 | Each entry is a **scope** and a generation, a positive integer that only |
| 91 | increases. The object is unsigned (like `control/serial`), carries no |
| 92 | authority and is only ever a hint to re-read: a forged or rolled-back index |
| 93 | can delay convergence until the backstop, never change what a reader |
| 94 | believes about a record (the reader always re-reads the records |
| 95 | themselves). Size is ~40 bytes per app; entries are not removed in v1. |
| 96 | |
| 97 | ### 3.2 Which writes bump which scope |
| 98 | |
| 99 | A pure function `ChangeScope(key) (scope string, ok bool)` in |
| 100 | `internal/release`: |
| 101 | |
| 102 | | Key | Scope | |
| 103 | |---|---| |
| 104 | | `apps/<a>/<b>/<rest>` where the first segment of `<rest>` is not a data prefix | `apps/<a>/<b>` | |
| 105 | | `apps/<a>/<b>/{logs,errors,analytics,runs,metrics,schedule-status}/…` | none | |
| 106 | | `accounts/<a>/…` | `accounts/<a>` | |
| 107 | | `control/certs/…` | `control/certs` | |
| 108 | | `control/edges/…` | `control/edges` | |
| 109 | | everything else (including `control/changes.json`, `control/serial` and `control/edge-mesh-key*`) | none | |
| 110 | |
| 111 | The data-prefix list is a denylist on purpose: a missing exclusion costs |
| 112 | an extra re-read, a missing inclusion costs staleness until the backstop. |
| 113 | `runs/` is excluded because run state that readers act on lives in the |
| 114 | instance record (`ActiveRun`), which is in scope. The exact data-prefix set |
| 115 | is fixed in the plan from the writer map (#336 thread) and pinned by a |
| 116 | table test. Keys outside every scope keep the readers they have today |
| 117 | (§5 for the mesh keys). |
| 118 | |
| 119 | ### 3.3 Write path |
| 120 | |
| 121 | **The bump lives inside the two concrete stores**, not in a decorator |
| 122 | **(codex 3, 6, 11)**. `S3Store` and `DirStore` gain an opt-in |
| 123 | `EnableChangeIndex()`; once enabled, each of their four mutation methods, |
| 124 | `Put`, `PutFrom`, `Delete` and `DeleteIfMatch`, bumps the key's scope after |
| 125 | the mutation succeeds. Because the hook is in the concrete type: |
| 126 | |
| 127 | - every optional extension (`ConditionalDeleter`, `PutStreamer`, `Opener`, |
| 128 | `SizeLister`, `PageLister`, `BucketListDirs`, `ModLister`) is untouched, |
| 129 | and every mutation path, including `release.DeleteIfMatch` |
| 130 | (`internal/release/conditional_delete.go:29`) and streaming `PutFrom`, |
| 131 | bumps; |
| 132 | - callers that type-assert `*release.S3Store`, such as box publication |
| 133 | (`internal/host/boxpublish.go:259`), still match; |
| 134 | - outer wrappers (`NewPrefixStore`, the hostops mutation boundary, |
| 135 | `internal/hostops/mutation_boundary.go:108,188`) sit above the hook, so |
| 136 | their writes bump, and the index write itself goes straight to the |
| 137 | bucket from inside the store, never through the boundary's classifier |
| 138 | (`internal/hostops/mutation_classifier.go:47,168`). That is deliberate: |
| 139 | the index carries no authority, so the boundary has nothing to decide |
| 140 | about it. |
| 141 | |
| 142 | The bump: |
| 143 | |
| 144 | 1. `Get control/changes.json` (absent = empty, schema 1). |
| 145 | 2. Set `entries[scope] = old + 1` (or 1). |
| 146 | 3. `Put` with `If-Match` on the ETag read, or create-only if absent. |
| 147 | 4. On a precondition failure, retry from 1, at most 4 attempts with |
| 148 | jitter, all inside a 2s budget detached from the caller's cancellation |
| 149 | (the `bumpSerial` pattern, `internal/console/serialbump.go:31`). |
| 150 | |
| 151 | Properties: |
| 152 | - **Bump after write, never before.** A reader that sees the new |
| 153 | generation and re-lists is guaranteed (S3 strong read-after-write) to see |
| 154 | the write. Bumping first would let a reader re-list before the write and |
| 155 | record the new generation, missing the write until the backstop. |
| 156 | - **Best effort.** A failed bump is logged and never fails the write. The |
| 157 | write is then visible at the next backstop walk, or sooner if anything |
| 158 | else bumps the same scope. |
| 159 | - **Coalescing** (plan detail): a process may batch several scopes into one |
| 160 | read-modify-write, provided each scope's bump still follows its write. |
| 161 | Bursts (a purge deleting hundreds of keys under one app, run churn) are |
| 162 | expected to coalesce to one bump per scope per batch; contention on the |
| 163 | single object is otherwise the same class as `control/serial` today. |
| 164 | |
| 165 | Every opener of the deployment store enables the index: the console |
| 166 | (`cmd/console/main.go`), the box agent and edge (`internal/host/agent.go`, |
| 167 | `internal/edge/edge.go`), the CLI's direct-bucket `openStore()` |
| 168 | (`cli.go:2065`) and every other `release.NewS3Store` call site that opens |
| 169 | the deployment bucket. A test walks the AST for `release.NewS3Store`/ |
| 170 | `NewS3StoreIn` call sites and requires each to be listed as either |
| 171 | "deployment, index enabled" or "other bucket" with a reason, so a new |
| 172 | opener cannot silently skip the index. |
| 173 | |
| 174 | IAM: every role that writes the deployment bucket needs `s3:PutObject` and |
| 175 | `s3:GetObject` on `control/changes.json`. The plan checks the |
| 176 | `deployment-buckets` and `region-host` module policies and the Tito |
| 177 | deployment's equivalents; a denied bump is a failed bump (logged, covered |
| 178 | by the backstop), never a failed write. |
| 179 | |
| 180 | Writes that still bypass the index: binaries built before this change |
| 181 | (old CLIs in direct-bucket mode, boxes mid-roll), and any tool writing the |
| 182 | bucket directly. The backstop (§4.4) covers them. |
| 183 | |
| 184 | ### 3.4 Read path |
| 185 | |
| 186 | Two new optional read extensions, implemented by both concrete stores: |
| 187 | |
| 188 | - `ConditionalGetter.GetIfChanged(ctx, key, etag)` returns |
| 189 | `(body, etag, changed, err)`; S3 answers 304 Not Modified for an |
| 190 | unchanged object. A 304 costs one GET ($0.40 per million), so a 2s poll |
| 191 | is ~43k GETs per process per day. |
| 192 | - `ETagLister.ListETags(ctx, prefix)` returns each key with its ETag. S3 |
| 193 | returns the ETag with every ListObjectsV2 entry; `DirStore` returns the |
| 194 | same ETag its `Get` returns. ETags avoid the one-second LastModified |
| 195 | resolution trap `ClaimCache` needs a settle window for. |
| 196 | |
| 197 | The hostops mutation boundary and `NewPrefixStore` forward both read |
| 198 | extensions **(codex 6)**. |
| 199 | |
| 200 | **Capability fallback (codex 7).** A reader uses the index only when its |
| 201 | store answers both extensions and the index read succeeds. A store without |
| 202 | them (the fleet store door, `internal/edge/storedoor.go:149,431`, and test |
| 203 | fakes), a denied read, or any index error means "unknown", never "empty |
| 204 | and unchanged": that pass walks as today. Customer fleet boxes therefore |
| 205 | keep today's polling; a tenant-filtered index for the door is a follow-up, |
| 206 | because exposing the global index would disclose other tenants' account |
| 207 | and app names. |
| 208 | |
| 209 | ## 4. The record view (box agent and console) |
| 210 | |
| 211 | ### 4.1 State |
| 212 | |
| 213 | A `RecordView` in `internal/provision` holds, for the whole deployment: |
| 214 | |
| 215 | - per account: known/soft-deleted, from `account.json`, with the |
| 216 | generation it was read at; |
| 217 | - per app: the `instances/` and `sidecars/` key listings with each key's |
| 218 | ETag; the parsed record bodies; the trash marker; the instances manifest |
| 219 | `instances.json` (`ManifestKey`, `internal/provision/provision.go:321`; |
| 220 | seeds and update policy, `reconcile.go:995`) and the sidecars manifest |
| 221 | `sidecars.json` (`internal/provision/sidecar.go:122`; placement, |
| 222 | `sidecar_reconcile.go:328`), each through its existing parser and |
| 223 | keeping its distinct missing and error semantics **(codex 5)**; and the |
| 224 | app's applied generation; |
| 225 | - `complete` and `unreadable` flags with today's meanings (strict |
| 226 | consumers fail on an incomplete view exactly as they fail on a partial |
| 227 | walk today). |
| 228 | |
| 229 | It produces the same `[]Record` / sidecar slices and the same |
| 230 | `known`/`readFailed` results the walk produces today, so `Reconcile`, |
| 231 | `ReconcileSidecars`, the moves pass, `steeringLoop` and the console sweeps |
| 232 | consume it without changing their logic. |
| 233 | |
| 234 | ### 4.2 Refresh |
| 235 | |
| 236 | The view keeps two generation maps **(codex 4)**: `observed` (the latest |
| 237 | index it has read) and `applied` (per scope, the generation its current |
| 238 | contents reflect). `Refresh(ctx)`: |
| 239 | |
| 240 | 1. Conditional GET of the index. Changed: update `observed`. Unchanged |
| 241 | (304): keep `observed`. |
| 242 | 2. For every scope where `applied < observed`, or that is new: re-read that |
| 243 | scope. Step 2 runs on a 304 too, so a scope whose earlier re-read failed |
| 244 | is retried on the next refresh. An app scope re-lists `instances/` and |
| 245 | `sidecars/` with `ListETags`, GETs only keys that are new or whose ETag |
| 246 | changed, and re-reads the trash marker and both manifests. An account |
| 247 | scope re-reads `account.json`. |
| 248 | 3. Set `applied[scope]` to the observed generation only for scopes re-read |
| 249 | successfully. The view is incomplete while any scope has |
| 250 | `applied < observed`. |
| 251 | |
| 252 | ### 4.3 Publication, snapshots and lock order |
| 253 | |
| 254 | **(codex 1, 2)** |
| 255 | |
| 256 | - **One refresh lock.** `RecordView.mu` serializes every mutation of the |
| 257 | view: incremental `Refresh` and the backstop rebuild (§4.4) alike. A |
| 258 | publication therefore always reflects reads that started after the |
| 259 | previous publication's reads, so the view never moves backwards, and a |
| 260 | slow backstop cannot publish over a newer incremental read. This is the |
| 261 | same serialization the walk has today, where Reconcile walks inside |
| 262 | `reconcileMu` (`internal/host/agent.go:1529-1542,1570`). |
| 263 | - **Immutable snapshots.** Each publication produces an immutable |
| 264 | snapshot with a monotonically increasing epoch, swapped in atomically. |
| 265 | Consumers read snapshots; they never hold `mu`. |
| 266 | - **Reconcile and sidecars.** Inside `reconcileMu`, the pass calls |
| 267 | `Refresh` and then takes the current snapshot, so a pass's snapshot is |
| 268 | never older than the previous pass's. |
| 269 | - **Moves.** Today the moves pass walks inside `w.pass` |
| 270 | (`internal/host/movespass.go:1061,1079`). It keeps that shape, and acts |
| 271 | only on live data: inside `w.pass` it calls `Refresh` and takes the |
| 272 | current snapshot, but uses it only to decide whether there is any move |
| 273 | work at all. When the snapshot has a record with a move (or one it cannot |
| 274 | decode), the last walk found one, or the worker holds any move state of |
| 275 | its own (a wake fence, the strict set, the hold or offline keep view |
| 276 | unbuilt or naming a host, archive verdicts, a landing lease, an op |
| 277 | journal or page outbox entry on disk), the pass walks the bucket and runs |
| 278 | exactly as before the index; otherwise it skips the walk. A snapshot can |
| 279 | be older than a walk a pass made while the view could not vouch (a bump |
| 280 | that failed), and move steps have side effects in both directions, so |
| 281 | they never act on it (final review R1, finding 1). The offline keep view |
| 282 | is always read from the bucket. A move started by a write whose bump |
| 283 | failed waits for the backstop rebuild. The pass never takes a copy |
| 284 | handed over from the Reconcile leg. |
| 285 | - **Lock order:** `reconcileMu` → `RecordView.mu`, and `w.pass` → |
| 286 | `reconcileMu` → `RecordView.mu`. `RecordView.mu` is a leaf: nothing is |
| 287 | acquired while holding it. Move operations that take `reconcileMu` |
| 288 | inside `w.pass` (`internal/host/movesoffline.go:1065`) keep their |
| 289 | existing order, and nothing holds `reconcileMu` across the moves pass. |
| 290 | |
| 291 | ### 4.4 Backstop walk |
| 292 | |
| 293 | A full strict walk (today's code path, with `ListETags` reuse for bodies) |
| 294 | rebuilds the view every `changeIndexBackstop` (30 minutes, R3, jittered |
| 295 | ±10%) per process, and at start. Under `RecordView.mu` it reads the index |
| 296 | first, walks, publishes the rebuilt view, and sets `observed` and |
| 297 | `applied` to the index generations read **before** the walk began, so a |
| 298 | bump that lands during the walk triggers a re-read afterwards, never a |
| 299 | skip. Holding `mu` for the walk blocks incremental refreshes for its |
| 300 | duration, which is no worse than today, where every Reconcile pass walks |
| 301 | inside `reconcileMu`. |
| 302 | |
| 303 | The backstop bounds staleness for writers that bypass the index. Today |
| 304 | those writers (box run/seed/move writes, direct-bucket CLI writes) rely on |
| 305 | the 5-minute floor; with the index they converge in seconds, and only |
| 306 | failures and old binaries wait for the backstop. |
| 307 | |
| 308 | **Removals are decided by the bucket's walk.** The same writers mean |
| 309 | absence in a snapshot is not the bucket's word: a record written without |
| 310 | its bump is missing from a Complete snapshot until the next rebuild. So a |
| 311 | `Reconcile` or `ReconcileSidecars` pass over a snapshot that reaches a |
| 312 | removal candidate proved by absence (record gone, a sidecar gone or no |
| 313 | longer placeable, the trash marker) runs the record walk once more over the |
| 314 | live bucket (`scanRecords`, `scanSidecars` over `freshStore`) and decides |
| 315 | every removal in that pass from what that walk read, by the walk's own |
| 316 | rules: invalid shapes and empty channels protected before scope, hosts |
| 317 | matched across every key (a host re-homed to another key is present), full |
| 318 | sidecar eligibility (region, pool, placement from the record and the live |
| 319 | sidecars manifest, against this box's inbound), and the live trash marker. |
| 320 | A walk that cannot read cleanly keeps every row that pass, as the walk |
| 321 | always has. A snapshot pass with no removal candidate reads nothing more. |
| 322 | Over any store that is not a snapshot the pass's own walk already is the |
| 323 | bucket's, and nothing is read twice (final review R1, findings 2 and 4, |
| 324 | amadan #336). |
| 325 | |
| 326 | ### 4.5 What triggers a pass, and what it reads |
| 327 | |
| 328 | - `fastTick` keeps reading `control/serial` every 2s as today and also |
| 329 | polls the index (§3.4). The 5s clamp stays. |
| 330 | - **A serial change** triggers today's full pass, with the record walks |
| 331 | replaced by `Refresh`. Its adopt and config-sync legs keep their current |
| 332 | scope (hosted routes), because the serial is also bumped for things |
| 333 | outside the index's scopes. |
| 334 | - **An index-only change** triggers a **scoped pass** **(codex 10)**: |
| 335 | `Refresh`, Reconcile and ReconcileSidecars over the snapshot, the moves |
| 336 | pass, and then adopt (`release.Pass`, `internal/release/adopt.go:682`) |
| 337 | and config sync (`internal/appconfig/syncexec.go:174`) restricted to |
| 338 | hosted routes whose app scope moved. Changing one app re-reads that app |
| 339 | only. A witness test asserts that an index bump for one app causes no |
| 340 | read under any other app's prefix. |
| 341 | - `instancesLoop`'s 5–6 minute floor stays, now as a `Refresh` (one |
| 342 | conditional GET when nothing changed) followed by a scoped pass over |
| 343 | whatever moved. |
| 344 | - The backstop (§4.4) ends with today's unrestricted full pass, so every |
| 345 | leg is re-run over everything at least every 30 minutes. |
| 346 | |
| 347 | **Known limit (codex R2-10).** Serial-triggered passes keep adopt and |
| 348 | config sync over every hosted route, because the serial also signals |
| 349 | changes outside the index's scopes and there is no inventory of those yet. |
| 350 | That fan-out is bounded by the routes a box hosts, not by every app in the |
| 351 | deployment, and runs at the serial's rate (about once a minute on |
| 352 | flagship): roughly 0.35M GETs a day across the fleet, ~$4 a month. Scoping |
| 353 | those passes needs the console to mark serial bumps whose effects the |
| 354 | index fully covers; that is a follow-up, measured after rollout before it |
| 355 | is designed. |
| 356 | |
| 357 | ### 4.6 Console sweeps |
| 358 | |
| 359 | - **Run sweep** (30s): keeps its interval and reads the console's own |
| 360 | `RecordView` after a `Refresh`. Run state the boxes write |
| 361 | (`SetRecordRun`) is an instance-record write and bumps the app's scope |
| 362 | once the boxes run this code. Minutes-stale is already safe here |
| 363 | (`KeepFor` 24h, CAS on every change). |
| 364 | - **Orphan-run housekeeping** (every twentieth sweep, |
| 365 | `internal/console/runsweep.go:39,110,232`) today lists `runs/` for every |
| 366 | app with an instance record. It now lists only apps with a `run-start` |
| 367 | activity row (`internal/console/apiruns.go:315`) in the last 48 hours, |
| 368 | the window in which a run's objects can still be live or awaiting |
| 369 | cleanup, and does today's all-app sweep once a day as the backstop. |
| 370 | Orphans are still found by time (a stray's newest object a day old), not |
| 371 | by index movement, since `runs/` is outside every scope |
| 372 | **(codex R2-1)**. |
| 373 | - **Seed sweep** (5 min): keeps its strict walk, because its snapshot |
| 374 | deletes are proved by absence and must see records listed after the |
| 375 | snapshot listing (`seedsweep.go:126-131`). It now skips the walk |
| 376 | entirely when the snapshot listing is empty, since there is nothing to |
| 377 | delete; stale-seed abandonment reads the view. |
| 378 | |
| 379 | ## 5. GET loops |
| 380 | |
| 381 | Every gated read below also has a **maximum successful-read age equal to |
| 382 | the backstop interval**, a read on first use, and a retry on error, |
| 383 | independent of index movement **(codex 9)**. |
| 384 | |
| 385 | - **`configGenLoop`**: GETs an app's `config-gen.json` only when the app's |
| 386 | scope generation moved, or when its last successful read is older than |
| 387 | the backstop interval. `genCache` (`agent.go:1641`) stays as the ETag |
| 388 | comparison. |
| 389 | - **Canary pointers** (`fastMembers` → `PassFilter`): canary routes stay |
| 390 | members of every tick, but `PassFilter` reads a channel pointer only when |
| 391 | its app's generation moved or its last successful read is older than the |
| 392 | backstop. Direct-bucket promotes from new CLIs bump the app scope, so |
| 393 | canaries keep adopting in seconds; old CLIs fall back to the backstop |
| 394 | (R4). |
| 395 | - **`telemetryPass`** (`internal/host/telemetry_pass.go:35`): app and |
| 396 | account `telemetry.json` reads are gated on their scope generations plus |
| 397 | the age rule, and the account file is read once per pass, not once per |
| 398 | app. |
| 399 | - **Cert mirror** (`certsync.go:627`): polls the index; when |
| 400 | `control/certs` moves, `ListETags` and GET only new or changed certs; a |
| 401 | full rebuild, the one that drops spent challenge tokens, runs at the |
| 402 | backstop and whenever a key disappears from the listing. The boot |
| 403 | refresh before the listeners bind is unchanged. |
| 404 | - **Home map** (`internal/edge/homemap.go:320`): only the hosts half is |
| 405 | gated on `control/edges`. The mesh keys (`control/edge-mesh-key` and |
| 406 | `.next`, `internal/edgeproof/edgeproof.go:58`) keep their independent |
| 407 | 30s poll (two GETs per box per 30s), and `RefreshAttempted` is still |
| 408 | recorded on every poll, unchanged ones included, so the five-minute |
| 409 | refresh heartbeat (`internal/lbhealth/lbhealth.go:36,174`) holds |
| 410 | **(codex 8)**. |
| 411 | |
| 412 | ## 6. Provider boundary |
| 413 | |
| 414 | Everything is core: conditional PUT and GET and ListObjectsV2 ETags are |
| 415 | plain S3 API, supported by MinIO and implemented by `DirStore`. No AWS SDK |
| 416 | outside `internal/release`'s existing S3 client. A deployment that never |
| 417 | arms the readers behaves exactly as today, and so does any store that |
| 418 | lacks the read extensions (§3.4). `TestDeliveredContractMinio` gains |
| 419 | coverage for `GetIfChanged` and `ListETags`. |
| 420 | |
| 421 | ## 7. Rollout |
| 422 | |
| 423 | Writers ship first and always bump (cost: ~one extra GET and PUT per |
| 424 | relevant write, ~$3/month). Readers are armed per process by |
| 425 | `CARLOS_CHANGE_INDEX=read` (box `host.env`, console environment), off by |
| 426 | default, so a mixed fleet behaves as today. Writers that bump never change |
| 427 | any read path, so shipping them is safe with readers disarmed; this |
| 428 | includes box publication, which keeps its concrete `*release.S3Store` |
| 429 | (§3.3). |
| 430 | |
| 431 | 0. Check IAM (§3.3) on dev, flagship and Tito before the release. |
| 432 | 1. Release with the writers and dormant readers. Roll dev boxes, dev |
| 433 | console, flagship boxes, flagship console (roll order rule). |
| 434 | 2. Arm dev (boxes then console). Check: passes still converge (instance |
| 435 | create, delete, config set, promote, a run) within seconds; the |
| 436 | backstop log line appears every ~30 minutes; request rate on dev's |
| 437 | bucket drops (Cost Explorer daily, usage type by operation). |
| 438 | 3. Arm flagship boxes, then the console. Measure Cost Explorer daily |
| 439 | ListBucket/GetObject for three days against the 2026-10-19 baseline |
| 440 | (2.26M LIST, 7.5M GET). |
| 441 | 4. Disarm is removing the variable and restarting: readers fall back to |
| 442 | walking, writers keep bumping harmlessly. |
| 443 | |
| 444 | The census overlay for the touched packages is owed per the plan14 |
| 445 | process (#280). Release notes tell operators to update the CLI (R4). |
| 446 | |
| 447 | ## 8. Testing |
| 448 | |
| 449 | - **Production-wired witnesses**: the write hook is exercised through |
| 450 | the real openers (console, agent, CLI direct mode) and through the final |
| 451 | claim-enabled store handles (hostops boundary, prefix stores) against |
| 452 | `DirStore`, not a fake **(codex 6)**; box publication runs through the |
| 453 | index-enabled opener **(codex 11)**; the AST opener test from §3.3. |
| 454 | - **Capability preservation**: every optional extension the concrete |
| 455 | stores implement today is still implemented with the index enabled, and |
| 456 | every mutation method bumps **(codex 3)**. |
| 457 | - **Invariant simulation**: random interleavings of writers (some |
| 458 | bumping, some bypassing, some failing their bump), incremental |
| 459 | refreshes, backstop walks and consumers, on `DirStore`. Invariants: |
| 460 | (a) after a `Refresh` that observed generation g for a scope, every write |
| 461 | to that scope that bumped to ≤ g is visible in the view; (b) after a |
| 462 | backstop walk, the view equals a fresh strict walk; (c) published epochs |
| 463 | only increase and no scope's contents ever revert to an older read; |
| 464 | (d) a strict consumer never receives an incomplete view as complete; |
| 465 | (e) a scope whose re-read failed is retried on the next refresh even |
| 466 | when the index is unchanged. Named interleavings that must be in the |
| 467 | seed set: the backstop lists an app before a record appears while an |
| 468 | incremental refresh sees and publishes it **(codex 2)**; a pass captures |
| 469 | a snapshot, stalls, and another pass consumes a newer one before the |
| 470 | first reaches `w.pass` **(codex 1)**; an app refresh fails after the |
| 471 | index moved and the next index read is a 304 **(codex 4)**. One |
| 472 | 3000-seed sweep per revision, niced, per the CPU budget. Mutation |
| 473 | checks: bump before write; record generations for failed scopes; |
| 474 | skip the before-walk index read in §4.4; publish the backstop outside |
| 475 | `mu`; consume a handed-over snapshot in the moves pass. Each must fail |
| 476 | the simulation. |
| 477 | - **Scoped-pass witness**: one app's bump causes no reads under any other |
| 478 | app's prefix **(codex 10)**. |
| 479 | - **Gated-read ages**: telemetry, config-gen, canary pointers and certs |
| 480 | are re-read at the age bound with the index unchanged **(codex 9)**; |
| 481 | mesh-key rotation reaches the home map with `control/edges` unchanged, |
| 482 | and the health heartbeat survives unchanged polls **(codex 8)**. |
| 483 | - **Fallback**: a store without the read extensions, and a denied index |
| 484 | read, both produce today's walk, never an empty view **(codex 7)**. |
| 485 | - Table tests for `ChangeScope`; unit tests for `GetIfChanged` and |
| 486 | `ListETags` on both stores, plus MinIO in the delivered-contract test. |
| 487 | |
| 488 | ## 9. Expected effect |
| 489 | |
| 490 | | Family (per day) | Today | After (estimate) | |
| 491 | |---|---|---| |
| 492 | | Record walks (LIST) | ~2.0M | ~70k (backstop 48/day/process + changed apps) | |
| 493 | | Index + serial polls (GET) | 0.13M | ~0.35M | |
| 494 | | Record and manifest bodies (GET) | ~2M | changed only | |
| 495 | | config-gen + canary + certs + accounts (GET) | ~4.8M | changed only + age-bound re-reads | |
| 496 | |
| 497 | Expected flagship S3: ~$50–90/month (storage $16, transfer ~$25, |
| 498 | litestream ~$26, requests ~$10–25), flat as apps are added. |
| 499 | |
| 500 | ## 10. Questions for Paul (answered 2026-10-21, see R3 and R4) |
| 501 | |
| 502 | 1. **Backstop interval.** 30 minutes bounds staleness for old binaries and |
| 503 | failed bumps at ~$5/month. 15 minutes doubles that cost; 60 halves it. |
| 504 | Recommendation: 30. |
| 505 | 2. **Old CLIs in direct-bucket mode** (promote, env set, trash): their |
| 506 | writes converge at the backstop (up to 30 minutes) instead of within |
| 507 | seconds as today (`configGenLoop`'s 3s poll, canary pointers read every |
| 508 | tick) until operators update the CLI. Console and new-CLI writes are |
| 509 | unaffected. |
| 510 | |
| 511 | ## 11. Review log |
| 512 | |
| 513 | - Round 1 (Astra, bbae135c): reject, 3 P1 and 8 P2. All accepted: |
| 514 | codex 1 (moves snapshot ordering, §4.3), 2 (backstop publication |
| 515 | ordering, §4.3–4.4), 3 (capabilities, §3.3), 4 (304 retry, §4.2), |
| 516 | 5 (manifests, §4.1), 6 (hostops order, §3.3–3.4), 7 (fleet door, |
| 517 | §3.4), 8 (mesh keys and heartbeat, §5), 9 (telemetry age, §5), |
| 518 | 10 (per-app fan-out, §4.5), 11 (box publication, §3.3, §7). |
| 519 | - Round 2 (Astra, ca129623): accept at spec level. Round 1 dispositions: |
| 520 | ten resolved, 10 partly (serial-triggered fan-out). New: R2-1 (orphan |
| 521 | housekeeping). Rev 3 closes R2-1 (§4.6) and records R2-10 as a measured |
| 522 | known limit (§4.5). |
| 523 | |
amadan