amadan
Merge pull request #217 from carlosframework/corruption-doc-update
docs: record the currency question and the cutover blocker
1 file changed,
+77
−1
docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md+77 −1
diff --git a/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md b/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md| index 420b7844c..ae01d56c0 100644 |
| --- a/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md |
| +++ b/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md |
| @@ -203,11 +203,87 @@ The general lesson, worth carrying: **files a unit stages are charged to its |
| cgroup — tmpfs fatally, page cache survivably.** The fix for a drill that OOMs |
| is to get the bytes onto disk; the fix for one that crawls is a bigger ceiling. |
| +## 5b. The second question: is the replica CURRENT? (2026-08-16) |
| + |
| +Everything above answers "does this replica restore?". Working the follow-ups |
| +turned up a second question the drills had never asked, with a blind spot |
| +shaped exactly like the failure it exists to catch. |
| + |
| +**A replicator that dies leaves a replica that passes every check.** It is |
| +valid SQLite, it restores, `integrity_check` says ok — forever, while the |
| +database it mirrors moves on without it. The drill stays green and the backup |
| +quietly becomes a snapshot of the day replication stopped. |
| + |
| +This is not hypothetical. On 2026-08-11 the flagship's litestream exited with |
| +`permission denied` writing its shadow WAL into a DynamicUser tenant's state |
| +directory. A root drop-in fixed it the same minute and today's replicas are |
| +current (S3 timestamps checked) — but nothing in the platform would have said |
| +a word if it had not been fixed. A human reading a journal was the detection. |
| + |
| +The signal is SQLite's own **file-change counter** (header bytes 24–27), which |
| +advances on every write transaction reaching the main database file. Compare |
| +the source's counter with the restored copy's: a replica that stopped tracking |
| +is a number that stopped moving. No S3 layout knowledge, no litestream version |
| +dependency, no cross-machine clock — two files, read moments apart, by the |
| +same process. |
| + |
| +Two deliberately **one-sided** choices, because a durability check should |
| +under-claim rather than cry wolf: |
| + |
| +- WAL mode advances the main file's counter only at a checkpoint, so a source |
| + with a hot WAL reads *lower* than reality. That can hide a stale replica (a |
| + missed alarm); it can never invent one (a false alarm). |
| +- A source written within `staleGrace` (15m) is never called stale however far |
| + behind its replica is. That is ordinary lag, and calling it a fault would put |
| + the drill permanently red on the busiest — most valuable — databases on the |
| + box, which is how a check stops being read. |
| + |
| +Both drills ask it now, and a stale replica FAILS: what comes back is not the |
| +database. "Source is not on disk" fails too — a config naming a database that |
| +no longer exists has a replica that can never advance again. |
| + |
| +## 5c. Why the generator has not taken over the boxes yet |
| + |
| +`carlos ops litestream` now has everything it needs — unit-tenant DB trees, a |
| +refusal to shrink the replication set, per-path replica keys, and a filter that |
| +replicates only what a route claims. The remaining blocker is **data, not |
| +code**, and it is recorded in issue #216. |
| + |
| +A read-only preview on the flagship (a build staged in /tmp, generating to a |
| +temp path, nothing touched) shows the whole problem in one table: |
| + |
| +| hand-written config today (5) | what the generator produces (2) | |
| +| --- | --- | |
| +| 4 × the kass tree | — | |
| +| `/data/registry.db` | `/data/registry.db` | |
| +| — | `/data/console/console.db` ✅ | |
| + |
| +The kass route's `db_path` names `/data/home.kass.training.db`, which does not |
| +exist: it is the pre-migration path from the app-capabilities design §5, and |
| +the §5 migration was never performed (#58). `stampDBPaths` is deliberately |
| +conservative about overwriting an operator's `-db`, which is the right rule — |
| +it simply assumes the pre-migration file exists, and here neither file does. |
| + |
| +Two things follow, and both are decisions rather than work: |
| + |
| +1. A tenant whose data is a TREE needs more than one `db_path` column. The |
| + manifest already carries `data.dbs` as a list; only the stamp is |
| + single-valued (`d.DBs[0]`). |
| +2. **The litestream 0.3 → 0.5 upgrade and the generator takeover want to be |
| + ONE cutover.** Each starts a fresh replica lineage on its own, and the |
| + upgrade's whole payoff — the 7-day window in §3 — only arrives with the |
| + generated config, because that is what states the `snapshot:` stanza. |
| + Upgrading the binary alone just swaps litestream's 24h default for its own. |
| + |
| +Until then the guard is what keeps the status quo safe: an accidental cutover |
| +is refused, by name, rather than silently dropping four databases. |
| + |
| ## 6. Known follow-ups |
| - **Console surface.** `control/restore-verify.json` is read from the bucket |
| today, like `control/registry-snapshot.json`. A console/API view of "last |
| - proven restorable" per instance is the obvious next step. |
| + proven restorable" per instance is the obvious next step — and now wants a |
| + second column, "last proven CURRENT", since §5b made those different facts. |
| - **Fleet-box publication.** `hibernate-verify` resolves the deployment store |
| from `CARLOS_DEPLOYMENT_BUCKET`/`_DIR` only. A fleet box that reaches its |
| deployment through the console door publishes nothing — and says so loudly |