amadan
docs: record the currency question and the cutover blocker
Two things the follow-up run established that were not in the design note. §5b, the second question: the drills only ever asked "does this replica restore?", and a dead replicator leaves a replica that answers yes forever while the database moves on without it. The 2026-08-11 flagship incident is the worked example — caught by a human reading a journal, which is not a control. Records the change-counter signal and the two one-sided choices that keep it from crying wolf. §5c, why the generator has not taken over: the blocker is data, not code. The kass route names a pre-migration database path that does not exist, so the generated config covers 2 databases where the hand-written one covers 5 — demonstrated by a read-only preview on the box, and refused by the shrink guard rather than discovered afterwards. Also records the conclusion that the litestream 0.5 upgrade and the takeover want to be ONE cutover, because the upgrade's payoff only arrives with the config that states the retention.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 file changed,
+77
−1
docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md+77 −1
diff --git a/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md b/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md| index 420b7844c..ae01d56c0 100644 |
| --- a/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md |
| +++ b/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md |
| @@ -203,11 +203,87 @@ The general lesson, worth carrying: **files a unit stages are charged to its |
| cgroup — tmpfs fatally, page cache survivably.** The fix for a drill that OOMs |
| is to get the bytes onto disk; the fix for one that crawls is a bigger ceiling. |
| +## 5b. The second question: is the replica CURRENT? (2026-08-16) |
| + |
| +Everything above answers "does this replica restore?". Working the follow-ups |
| +turned up a second question the drills had never asked, with a blind spot |
| +shaped exactly like the failure it exists to catch. |
| + |
| +**A replicator that dies leaves a replica that passes every check.** It is |
| +valid SQLite, it restores, `integrity_check` says ok — forever, while the |
| +database it mirrors moves on without it. The drill stays green and the backup |
| +quietly becomes a snapshot of the day replication stopped. |
| + |
| +This is not hypothetical. On 2026-08-11 the flagship's litestream exited with |
| +`permission denied` writing its shadow WAL into a DynamicUser tenant's state |
| +directory. A root drop-in fixed it the same minute and today's replicas are |
| +current (S3 timestamps checked) — but nothing in the platform would have said |
| +a word if it had not been fixed. A human reading a journal was the detection. |
| + |
| +The signal is SQLite's own **file-change counter** (header bytes 24–27), which |
| +advances on every write transaction reaching the main database file. Compare |
| +the source's counter with the restored copy's: a replica that stopped tracking |
| +is a number that stopped moving. No S3 layout knowledge, no litestream version |
| +dependency, no cross-machine clock — two files, read moments apart, by the |
| +same process. |
| + |
| +Two deliberately **one-sided** choices, because a durability check should |
| +under-claim rather than cry wolf: |
| + |
| +- WAL mode advances the main file's counter only at a checkpoint, so a source |
| + with a hot WAL reads *lower* than reality. That can hide a stale replica (a |
| + missed alarm); it can never invent one (a false alarm). |
| +- A source written within `staleGrace` (15m) is never called stale however far |
| + behind its replica is. That is ordinary lag, and calling it a fault would put |
| + the drill permanently red on the busiest — most valuable — databases on the |
| + box, which is how a check stops being read. |
| + |
| +Both drills ask it now, and a stale replica FAILS: what comes back is not the |
| +database. "Source is not on disk" fails too — a config naming a database that |
| +no longer exists has a replica that can never advance again. |
| + |
| +## 5c. Why the generator has not taken over the boxes yet |
| + |
| +`carlos ops litestream` now has everything it needs — unit-tenant DB trees, a |
| +refusal to shrink the replication set, per-path replica keys, and a filter that |
| +replicates only what a route claims. The remaining blocker is **data, not |
| +code**, and it is recorded in issue #216. |
| + |
| +A read-only preview on the flagship (a build staged in /tmp, generating to a |
| +temp path, nothing touched) shows the whole problem in one table: |
| + |
| +| hand-written config today (5) | what the generator produces (2) | |
| +| --- | --- | |
| +| 4 × the kass tree | — | |
| +| `/data/registry.db` | `/data/registry.db` | |
| +| — | `/data/console/console.db` ✅ | |
| + |
| +The kass route's `db_path` names `/data/home.kass.training.db`, which does not |
| +exist: it is the pre-migration path from the app-capabilities design §5, and |
| +the §5 migration was never performed (#58). `stampDBPaths` is deliberately |
| +conservative about overwriting an operator's `-db`, which is the right rule — |
| +it simply assumes the pre-migration file exists, and here neither file does. |
| + |
| +Two things follow, and both are decisions rather than work: |
| + |
| +1. A tenant whose data is a TREE needs more than one `db_path` column. The |
| + manifest already carries `data.dbs` as a list; only the stamp is |
| + single-valued (`d.DBs[0]`). |
| +2. **The litestream 0.3 → 0.5 upgrade and the generator takeover want to be |
| + ONE cutover.** Each starts a fresh replica lineage on its own, and the |
| + upgrade's whole payoff — the 7-day window in §3 — only arrives with the |
| + generated config, because that is what states the `snapshot:` stanza. |
| + Upgrading the binary alone just swaps litestream's 24h default for its own. |
| + |
| +Until then the guard is what keeps the status quo safe: an accidental cutover |
| +is refused, by name, rather than silently dropping four databases. |
| + |
| ## 6. Known follow-ups |
| - **Console surface.** `control/restore-verify.json` is read from the bucket |
| today, like `control/registry-snapshot.json`. A console/API view of "last |
| - proven restorable" per instance is the obvious next step. |
| + proven restorable" per instance is the obvious next step — and now wants a |
| + second column, "last proven CURRENT", since §5b made those different facts. |
| - **Fleet-box publication.** `hibernate-verify` resolves the deployment store |
| from `CARLOS_DEPLOYMENT_BUCKET`/`_DIR` only. A fleet box that reaches its |
| deployment through the console door publishes nothing — and says so loudly |