amadan
restore-verify: hold one restored database at a time, not all of them
Found by arming the drill on carlos-tito-ie and watching it: Result=oom-kill, "Consumed 19.963s CPU time, 256.2M memory peak", after ~40 of 54 replicas had restored CLEAN. The unit runs PrivateTmp=yes, so its temp dir is a tmpfs and every restored byte is charged to the unit's own cgroup against MemoryMax=256M. The restored files ARE the memory. RestoreVerify kept every copy until the run ended, so the drill's peak was the SUM of the fleet's databases (247M on tito) — a check that grew its own failure as the fleet grew, and would have started failing silently at whatever size crossed the line. Each copy is now deleted as soon as it is checked, on every path including the failure and retry ones, so the peak is the LARGEST SINGLE database. MemoryMax goes to 512M with the arithmetic written down beside it. carlos-hibernate-verify had the property already — lease.VerifyReplica deletes its throwaway copy in a defer — and gets the same ceiling and note. Also adds the flagship litestream 0.3.13 → 0.5.15 upgrade plan. The blocker is that 0.5 cannot read a 0.3 replica (the flagship's objects are the generations/*.snapshot.lz4 layout; 0.5 writes LTX), so it is not a binary swap but a new lineage under a new prefix, keeping the old binary and objects until the new lineage has proven itself. Notably the box ALREADY runs both formats — the activator's embedded 0.5.14 writes LTX for hibernating instances — so LTX is proven there; the upgrade makes one replicator match the other.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
5 files changed,
+222
−9
docs/superpowers/plans/2026-08-15-flagship-litestream-0.5-upgrade.md+120 −0internal/host/units/carlos-hibernate-verify.service+5 −1internal/host/units/carlos-restore-verify.service+10 −1internal/ops/litestream.go+37 −7internal/ops/ops_test.go+50 −0
diff --git a/docs/superpowers/plans/2026-08-15-flagship-litestream-0.5-upgrade.md b/docs/superpowers/plans/2026-08-15-flagship-litestream-0.5-upgrade.md| new file mode 100644 |
| index 000000000..6f84d3c59 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-15-flagship-litestream-0.5-upgrade.md |
| @@ -0,0 +1,120 @@ |
| +# Flagship litestream 0.3.13 → 0.5.15 — upgrade plan |
| + |
| +Status: plan only, nothing applied. Written 2026-08-15 after the corruption- |
| +defence audit (`docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md`). |
| + |
| +## Why |
| + |
| +`carlos-flagship-ie` runs **litestream v0.3.13, dated October 2023**, while the |
| +repo pins `litestreamVersion = "0.5.15"` and vendors the 0.5.14 library. Three |
| +things follow: |
| + |
| +1. The snapshot/retention policy `carlos ops litestream` now writes is a 0.5 |
| + feature. 0.3.13 parses it, exits 0, and ignores it — verified on the box. |
| + The flagship's real recovery window stays at litestream's 24h default no |
| + matter what the config says. (`ops.SnapshotPolicySupported` now says so out |
| + loud, but saying it is not fixing it.) |
| +2. `carlos ops restore-verify` shells out to whatever `litestream` is on PATH, |
| + so the daily drill proves the *0.3* replicas restore — with a *0.3* binary. |
| + The moment the binary is replaced, that drill is testing something new. |
| +3. The box already runs **both formats**. The activator's embedded replication |
| + (hibernating instances → the instances bucket) is the vendored **0.5.14** |
| + library writing LTX; only the box-wide replicator is 0.3. So LTX is already |
| + proven in production on this box — the upgrade is not a leap into an |
| + untested format, it is making one replicator match the other. |
| + |
| +## The one hard constraint |
| + |
| +**0.5 cannot read a 0.3 replica.** The flagship's objects are the 0.3 layout: |
| + |
| + db/registry.db/generations/4e15a5dfb57c9e89/snapshots/00000661.snapshot.lz4 |
| + db/registry.db/generations/4e15a5dfb57c9e89/wal/00000661_00001038.wal.lz4 |
| + |
| +0.5 writes LTX instead. So this is not a binary swap — it is **starting a new |
| +replica lineage**, with a window during which the only restorable history of |
| +anything written before the cutover lives in objects the new binary cannot |
| +read. Every step below exists to keep that window honest. |
| + |
| +Do **not** point 0.5 at the existing prefix. Mixing formats under one prefix is |
| +how a replica set becomes unrestorable by either binary. |
| + |
| +## Sizing (measured 2026-08-15) |
| + |
| +| | flagship | |
| +| --- | --- | |
| +| disk | 20 G, 11 G free | |
| +| RAM | 1858 MB total, 871 MB available | |
| +| `/tmp` (tmpfs) | 930 M | |
| +| databases in the config | 5, all small (kass instances total 324 K) | |
| + |
| +The tito OOM (54 databases, 247 M, `MemoryMax=256M`) does **not** threaten this |
| +box — its database set is tiny. The `MemoryMax=512M` and delete-as-you-go fix |
| +still apply; they are just not urgent here. |
| + |
| +## Steps |
| + |
| +**0. Precondition.** PR #200 merged and rolled to the flagship, so the box has |
| +the delete-as-you-go `restore-verify`, the version warning, and |
| +`hibernate-verify`. Confirm `carlos version` on the box reports the new build |
| +before starting. |
| + |
| +**1. Keep the old binary and name it.** Copy the current binary aside as |
| +`/usr/local/bin/litestream-0.3` before anything replaces `/usr/local/bin/litestream`. |
| +It is the ONLY thing that can read the pre-cutover objects; losing it turns the |
| +old lineage into landfill. Prove it still runs (`litestream-0.3 version`). |
| + |
| +**2. Prove a restore from the OLD lineage, with the OLD binary, and keep it.** |
| +Restore every configured database to a scratch path with `litestream-0.3` and |
| +`ops.CheckRestoredDB`-equivalent checks (`carlos ops restore-verify |
| +-litestream-bin /usr/local/bin/litestream-0.3` does exactly this). Green here |
| +is the baseline the whole cutover is measured against — do not proceed on a red |
| +or an unread result. |
| + |
| +**3. Cut over under a NEW prefix.** Regenerate the config with a prefix that |
| +has never held 0.3 objects (e.g. `-prefix db-ltx` beside today's `db`), install |
| +0.5.15 as `/usr/local/bin/litestream`, then restart `carlos-litestream.service`. |
| +The new lineage starts empty and fills from the live databases; the old prefix |
| +is untouched and still readable by `litestream-0.3`. |
| + |
| +**4. Prove the NEW lineage restores.** `carlos ops restore-verify` (now the 0.5 |
| +binary, new config). Not the same day's first pass — wait for at least one |
| +snapshot interval so the drill is exercising a real snapshot rather than the |
| +initial sync. The version warning from step 0 must have disappeared: its |
| +absence is the evidence that the stated 7-day window is now enforced. |
| + |
| +**5. Let both lineages run until the old one is past useful.** The old prefix |
| +stops advancing at cutover, so its usable window closes on its own. Keep it — |
| +and `litestream-0.3` — for at least as long as the retention you would want to |
| +restore from (the new 7-day policy is the natural minimum; longer is cheap, |
| +these objects are small). Only then delete the old prefix, and delete the |
| +binary last. |
| + |
| +## Verification |
| + |
| +- `litestream version` reports 0.5.15 and `carlos ops litestream` no longer |
| + warns — the two together are what prove the recovery window is real rather |
| + than merely written down. |
| +- `carlos ops restore-verify` green against the new lineage on two consecutive |
| + days (one pass can pass on the initial sync alone). |
| +- `carlos ops hibernate-verify` unaffected throughout: it uses the embedded |
| + library and the instances bucket, neither of which this touches. Confirm it |
| + is still green after the cutover as a control. |
| +- The edge keeps serving throughout: this changes replication only, and the |
| + restart in step 3 is the replicator, not the edge. |
| + |
| +## Rollback |
| + |
| +Any failure at or before step 4: reinstall `litestream-0.3` as |
| +`/usr/local/bin/litestream`, regenerate the config with the original prefix, |
| +restart `carlos-litestream.service`. The old lineage never stopped being valid, |
| +which is the entire reason for the new-prefix rule. |
| + |
| +## Related |
| + |
| +- tito already runs 0.5.15 with a 0.5-format replica set, generated by the |
| + legacy `titogo ops litestream`. Bringing tito's config under `carlos ops |
| + litestream` is a separate piece of work with no binary change in it, and is |
| + what would deliver the retention policy and `console.db` there. |
| +- **`console.db` is unreplicated on both boxes today** and neither this plan |
| + nor merging #200 changes that on its own — it lands when `carlos ops |
| + litestream` becomes the generator on each box. |
diff --git a/internal/host/units/carlos-hibernate-verify.service b/internal/host/units/carlos-hibernate-verify.service| index dbd37a716..2b9f1e26a 100644 |
| --- a/internal/host/units/carlos-hibernate-verify.service |
| +++ b/internal/host/units/carlos-hibernate-verify.service |
| @@ -43,4 +43,8 @@ PrivateTmp=yes |
| PrivateDevices=yes |
| RestrictAddressFamilies=AF_INET AF_INET6 |
| MemoryDenyWriteExecute=yes |
| -MemoryMax=256M |
| +# Same tmpfs-is-memory arithmetic as carlos-restore-verify.service (read its |
| +# note): PrivateTmp charges every restored byte to this cgroup. This drill is |
| +# one-at-a-time by construction — lease.VerifyReplica deletes each throwaway |
| +# copy in a defer — so the peak is one database, and the same 512M applies. |
| +MemoryMax=512M |
diff --git a/internal/host/units/carlos-restore-verify.service b/internal/host/units/carlos-restore-verify.service| index 4a893062a..fe90f9df4 100644 |
| --- a/internal/host/units/carlos-restore-verify.service |
| +++ b/internal/host/units/carlos-restore-verify.service |
| @@ -23,4 +23,13 @@ PrivateTmp=yes |
| PrivateDevices=yes |
| RestrictAddressFamilies=AF_INET AF_INET6 |
| MemoryDenyWriteExecute=yes |
| -MemoryMax=256M |
| +# PrivateTmp=yes makes this unit's /tmp a tmpfs, so every byte the drill |
| +# restores is charged to THIS cgroup against the limit below — the restored |
| +# files are the memory. carlos-tito-ie, 2026-08-15: 54 databases totalling |
| +# 247M were OOM-killed at a "256.2M memory peak" after ~40 clean restores, |
| +# because copies accumulated. RestoreVerify now deletes each copy once it is |
| +# checked, so the peak is the LARGEST SINGLE database rather than their sum; |
| +# 512M leaves room for a database several times bigger than today's largest |
| +# (149M) plus litestream's own working set. Raise it if a single tenant |
| +# database ever approaches this, and see ops.RestoreVerify for the rest. |
| +MemoryMax=512M |
diff --git a/internal/ops/litestream.go b/internal/ops/litestream.go| index bc703c486..718b42938 100644 |
| --- a/internal/ops/litestream.go |
| +++ b/internal/ops/litestream.go |
| @@ -264,9 +264,23 @@ func FleetDBs(reg *registry.Registry, dataDir string) ([]string, error) { |
| return dbs, nil |
| } |
| +// restoreAndCheckFn is the per-database restore behind a package var, so the |
| +// one-at-a-time property below is testable without S3 or a litestream binary. |
| +var restoreAndCheckFn = restoreAndCheck |
| + |
| // RestoreVerify restores every database in the generated config from S3 into |
| // tmpDir and proves each opens clean. Returns an error on any failure so the |
| // caller (a oneshot unit) can exit non-zero where the fleet sweeps see it. |
| +// |
| +// Exactly ONE restored copy exists at a time: each is deleted as soon as it |
| +// has been checked. This is a memory constraint, not housekeeping — |
| +// carlos-restore-verify.service runs PrivateTmp=yes, so tmpDir is a tmpfs and |
| +// every byte written there is charged to the unit's own cgroup against |
| +// MemoryMax. Keeping all of them meant the drill's peak was the SUM of the |
| +// fleet's databases: measured on carlos-tito-ie 2026-08-15, 54 databases |
| +// totalling 247M under MemoryMax=256M, OOM-killed at a "256.2M memory peak" |
| +// after ~40 clean restores — a drill that grew its own failure as the fleet |
| +// grew. The peak is now the largest single database. |
| func RestoreVerify(litestreamBin, cfgPath, tmpDir string) error { |
| raw, err := os.ReadFile(cfgPath) |
| if err != nil { |
| @@ -279,16 +293,22 @@ func RestoreVerify(litestreamBin, cfgPath, tmpDir string) error { |
| failures := 0 |
| for _, db := range dbs { |
| - if err := restoreAndCheck(litestreamBin, cfgPath, db, tmpDir); err != nil { |
| + // Sweep this database's copy however the check ends — a failure must |
| + // not leave the wreckage behind for the next restore to be charged for. |
| + out := filepath.Join(tmpDir, filepath.Base(db)) |
| + err := restoreAndCheckFn(litestreamBin, cfgPath, db, tmpDir) |
| + if err != nil { |
| + removeRestored(out) |
| // One retry after a pause: a database created moments ago may not |
| // have finished its first replication pass yet. |
| time.Sleep(retryDelay) |
| - err = restoreAndCheck(litestreamBin, cfgPath, db, tmpDir) |
| - if err != nil { |
| - failures++ |
| - fmt.Printf("FAIL %s: %v\n", db, err) |
| - continue |
| - } |
| + err = restoreAndCheckFn(litestreamBin, cfgPath, db, tmpDir) |
| + } |
| + removeRestored(out) // every path, success included |
| + if err != nil { |
| + failures++ |
| + fmt.Printf("FAIL %s: %v\n", db, err) |
| + continue |
| } |
| fmt.Printf("ok %s\n", db) |
| } |
| @@ -299,6 +319,16 @@ func RestoreVerify(litestreamBin, cfgPath, tmpDir string) error { |
| return nil |
| } |
| +// removeRestored deletes one restored copy and the sidecars an open may have |
| +// attached beside it (CheckRestoredDB's read-only DSN does not stop the driver |
| +// creating a -wal). Errors are dropped: this is reclamation, and a file that |
| +// is already gone is the outcome wanted. |
| +func removeRestored(path string) { |
| + for _, p := range []string{path, path + "-wal", path + "-shm"} { |
| + _ = os.Remove(p) |
| + } |
| +} |
| + |
| // restoreAndCheck pulls one database back from its replica and runs the same |
| // checks a disaster recovery would lean on: the file opens, integrity_check |
| // says ok, and the schema is non-empty. |
diff --git a/internal/ops/ops_test.go b/internal/ops/ops_test.go| index 1a07842e5..27babd0c8 100644 |
| --- a/internal/ops/ops_test.go |
| +++ b/internal/ops/ops_test.go |
| @@ -337,3 +337,53 @@ func TestSnapshotPolicySupported(t *testing.T) { |
| } |
| } |
| } |
| + |
| +// The drill must hold ONE restored database at a time, not all of them. |
| +// |
| +// This is not tidiness. carlos-restore-verify.service runs PrivateTmp=yes, so |
| +// its temp dir is a tmpfs whose pages are charged to the unit's own cgroup — |
| +// accumulating every restored copy there spends MemoryMax on files. Measured |
| +// on carlos-tito-ie 2026-08-15: 54 databases totalling 247M against |
| +// MemoryMax=256M, killed by the OOM killer at "256.2M memory peak" after ~40 |
| +// clean restores. Deleting each copy once it has been checked bounds the peak |
| +// at the LARGEST SINGLE database instead of their sum. |
| +func TestRestoreVerifyHoldsOneRestoreAtATime(t *testing.T) { |
| + tmp := t.TempDir() |
| + cfgPath := filepath.Join(t.TempDir(), "litestream.yml") |
| + dbs := []string{"/data/a.db", "/data/b.db", "/data/c.db"} |
| + if err := os.WriteFile(cfgPath, []byte(LitestreamConfig(dbs, "b", "p", "eu-west-1")), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + |
| + maxSeen := 0 |
| + prev := restoreAndCheckFn |
| + restoreAndCheckFn = func(_, _, dbPath, tmpDir string) error { |
| + // Stand in for a restore: leave a file behind, like the real one does. |
| + if err := os.WriteFile(filepath.Join(tmpDir, filepath.Base(dbPath)), []byte("restored"), 0o600); err != nil { |
| + return err |
| + } |
| + ents, err := os.ReadDir(tmpDir) |
| + if err != nil { |
| + return err |
| + } |
| + if len(ents) > maxSeen { |
| + maxSeen = len(ents) |
| + } |
| + return nil |
| + } |
| + t.Cleanup(func() { restoreAndCheckFn = prev }) |
| + |
| + if err := RestoreVerify("litestream", cfgPath, tmp); err != nil { |
| + t.Fatalf("RestoreVerify: %v", err) |
| + } |
| + if maxSeen > 1 { |
| + t.Fatalf("as many as %d restored copies were held at once — the drill spends MemoryMax on tmpfs files", maxSeen) |
| + } |
| + left, err := os.ReadDir(tmp) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(left) != 0 { |
| + t.Fatalf("RestoreVerify left %d file(s) behind", len(left)) |
| + } |
| +} |