amadan
verify drills: correct the tmpfs rationale with what the box measured
PR #204 claimed that moving restores off the PrivateTmp tmpfs would make MemoryMax bound litestream's working set rather than tenant size. That is wrong, and the first live run said so: page cache is charged to the cgroup just as tmpfs is, so the peak barely moved (485.6M → 494.4M on the flagship, same 412M tenant; tito's box-wide drill did fall, 194.9M → 141.3M). The change is still right, for a different reason. What differs is not the number but what the ceiling DOES when the restore does not fit: tmpfs pages are unevictable -> hitting MemoryMax KILLS the drill page cache is reclaimable -> hitting MemoryMax THROTTLES it A/B on the flagship, same 412M database, MemoryMax=128M: on tmpfs, oom-killed after 8.3s; on disk, still running after ten minutes under reclaim, stopped by hand. So a tenant outgrowing the ceiling now degrades instead of failing. Which also means size is NOT irrelevant, as the old comment implied: a badly undersized ceiling makes the drill crawl. MemoryMax must still clear the largest single database, and the comments now say so and carry the watch item (seapointish.bag.oncarlos.com.db was 412M against 512M on 2026-08-16). Comments and the design note only — no behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 files changed,
+65
−30
docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md+26 −6internal/host/units/carlos-hibernate-verify.service+9 −8internal/host/units/carlos-restore-verify.service+30 −16
diff --git a/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md b/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md| index f1a8332ce..420b7844c 100644 |
| --- a/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md |
| +++ b/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md |
| @@ -172,16 +172,36 @@ largest single database — but on tmpfs that still means **the ceiling has to |
| exceed the biggest tenant**, so the drill would begin failing on exactly the |
| databases it exists to vouch for, about 30M of growth away. |
| -The fix is to stop restoring into RAM: both drill units now carry |
| -`CacheDirectory=carlos-verify` and `Environment=TMPDIR=%C/carlos-verify`, so |
| -restores land on disk and `MemoryMax` bounds litestream's working set (tens of |
| -MB) rather than tenant size. Pinned by |
| +Both drill units now carry `CacheDirectory=carlos-verify` and |
| +`Environment=TMPDIR=%C/carlos-verify`, so restores land on disk. Pinned by |
| `TestVerifyDrillsRestoreToDiskNotTmpfs`, because it is a one-line property |
| that regresses silently — the drill keeps passing right up until a tenant |
| grows. |
| -The general lesson, worth carrying: **under `PrivateTmp=yes`, temp files are |
| -memory.** Any unit that stages data through /tmp is charged for it. |
| +**What that actually buys, stated correctly.** The first framing of this fix |
| +(in PR #204) claimed the ceiling would then bound litestream's working set |
| +rather than tenant size. That is WRONG, and the box said so: page cache is |
| +charged to the cgroup just as tmpfs is, so the reported peak barely moved |
| +(485.6M on tmpfs → 494.4M on disk, same 412M tenant; tito's box-wide drill did |
| +fall, 194.9M → 141.3M). What changes is not the number but **what the ceiling |
| +does when the restore does not fit**: |
| + |
| +| | hitting `MemoryMax` | |
| +| --- | --- | |
| +| tmpfs (unevictable) | **kills** the drill | |
| +| page cache (reclaimable) | **throttles** it | |
| + |
| +Measured on the flagship, same 412M database, `MemoryMax=128M`: on tmpfs, |
| +oom-killed after 8.3s; on disk, still running after ten minutes under reclaim, |
| +stopped by hand. So a tenant outgrowing the ceiling now degrades instead of |
| +failing — but size is not irrelevant, because a badly undersized ceiling makes |
| +the drill crawl. `MemoryMax` must still sit comfortably above the largest |
| +single database. (Watch item: `seapointish.bag.oncarlos.com.db` was 412M |
| +against a 512M ceiling on 2026-08-16.) |
| + |
| +The general lesson, worth carrying: **files a unit stages are charged to its |
| +cgroup — tmpfs fatally, page cache survivably.** The fix for a drill that OOMs |
| +is to get the bytes onto disk; the fix for one that crawls is a bigger ceiling. |
| ## 6. Known follow-ups |
diff --git a/internal/host/units/carlos-hibernate-verify.service b/internal/host/units/carlos-hibernate-verify.service| index e491c837f..3d0880649 100644 |
| --- a/internal/host/units/carlos-hibernate-verify.service |
| +++ b/internal/host/units/carlos-hibernate-verify.service |
| @@ -29,11 +29,12 @@ Type=oneshot |
| # wrong store. |
| EnvironmentFile=-/etc/carlos/host.env |
| ExecStart=/opt/carlos/carlos ops hibernate-verify |
| -# Disk, not the PrivateTmp tmpfs — see carlos-restore-verify.service's note. |
| -# This drill hit it hardest: the flagship's largest hibernating tenant is 412M |
| -# and the first live run peaked at 485.6M against a 512M ceiling, i.e. the |
| -# drill was ~30M of tenant growth away from failing on the very databases it |
| -# is there to vouch for. |
| +# Disk, not the PrivateTmp tmpfs — see carlos-restore-verify.service's note |
| +# for the measurements. This drill is where it mattered most: the flagship's |
| +# largest hibernating tenant is 412M and the first live run peaked at 485.6M |
| +# against a 512M ceiling. On tmpfs that gap was ~30M of tenant growth away |
| +# from an OOM kill on the very databases the drill is there to vouch for; on |
| +# disk the same overrun only slows it down. |
| CacheDirectory=carlos-verify |
| Environment=TMPDIR=%C/carlos-verify |
| User=carlos |
| @@ -50,7 +51,7 @@ PrivateTmp=yes |
| PrivateDevices=yes |
| RestrictAddressFamilies=AF_INET AF_INET6 |
| MemoryDenyWriteExecute=yes |
| -# Bounds the working set, not the databases (TMPDIR is on disk above). This |
| -# drill is one-at-a-time by construction — lease.VerifyReplica deletes each |
| -# throwaway copy in a defer — so the same generous 512M ceiling applies. |
| +# One-at-a-time by construction — lease.VerifyReplica deletes each throwaway |
| +# copy in a defer — so this needs to clear the largest single tenant, not the |
| +# fleet. Same ceiling and the same WATCH note as the sibling unit. |
| MemoryMax=512M |
diff --git a/internal/host/units/carlos-restore-verify.service b/internal/host/units/carlos-restore-verify.service| index 6fb57944e..c5ebeed2f 100644 |
| --- a/internal/host/units/carlos-restore-verify.service |
| +++ b/internal/host/units/carlos-restore-verify.service |
| @@ -13,14 +13,25 @@ Wants=network-online.target |
| Type=oneshot |
| ExecStart=/opt/carlos/carlos ops restore-verify -config /etc/litestream.yml |
| EnvironmentFile=/data/litestream.env |
| -# Restore into a DISK-backed directory, not the PrivateTmp tmpfs. A restored |
| -# database on tmpfs is charged to this cgroup as memory, so the drill's peak |
| -# tracked the size of the fleet's largest tenant and would OOM the moment one |
| -# outgrew MemoryMax — measured on the flagship 2026-08-15, a 412M tenant |
| -# produced a 485.6M peak against a 512M ceiling. On disk, the peak is |
| -# litestream's own working set (tens of MB) and the drill stops competing with |
| -# the tenants it exists to protect. systemd creates and cleans %C/carlos-verify |
| -# and binds it writable despite ProtectSystem=strict. |
| +# Restore into a DISK-backed directory, not the PrivateTmp tmpfs. Both are |
| +# charged to this cgroup — page cache is accounted just like tmpfs, so the |
| +# reported peak barely moves (flagship: 485.6M on tmpfs, 494.4M on disk, for a |
| +# 412M tenant). What changes is what the ceiling DOES when the restore does not |
| +# fit, and that is the whole point: |
| +# |
| +# tmpfs pages are unevictable -> hitting MemoryMax KILLS the drill |
| +# page cache is reclaimable -> hitting MemoryMax THROTTLES it |
| +# |
| +# Measured on the flagship 2026-08-16, same 412M database, MemoryMax=128M: |
| +# on tmpfs, oom-killed after 8.3s; on disk, still running after 10 minutes |
| +# under reclaim (stopped by hand). So the restored file no longer has to FIT |
| +# in the cgroup's memory — a tenant outgrowing the ceiling degrades instead of |
| +# failing. |
| +# |
| +# It does NOT make size irrelevant: a badly undersized ceiling makes the drill |
| +# crawl. Keep MemoryMax comfortably above the largest tenant. systemd creates |
| +# and cleans %C/carlos-verify and binds it writable despite |
| +# ProtectSystem=strict. |
| CacheDirectory=carlos-verify |
| Environment=TMPDIR=%C/carlos-verify |
| User=carlos |
| @@ -33,12 +44,15 @@ PrivateTmp=yes |
| PrivateDevices=yes |
| RestrictAddressFamilies=AF_INET AF_INET6 |
| MemoryDenyWriteExecute=yes |
| -# With TMPDIR on disk (above) this bounds litestream's WORKING SET, not the |
| -# databases — measured 194.9M for tito's 27 replicas and 485.6M on the |
| -# flagship back when a 412M tenant still landed in tmpfs. A ceiling costs |
| -# nothing unused, so it stays generous. Two earlier lessons are kept here |
| -# because both are load-bearing: tito was OOM-killed at "256.2M memory peak" |
| -# on 2026-08-15 when restored copies ACCUMULATED (RestoreVerify now deletes |
| -# each once checked — see its comment), and tmpfs is why the files counted as |
| -# memory at all. |
| +# Sized to sit ABOVE the largest single database, not to bound the process: |
| +# restored bytes are charged here either way (see the TMPDIR note). Measured |
| +# after the move to disk: tito's 27 replicas 141.3M (was 194.9M on tmpfs), the |
| +# flagship's 412M tenant 494.4M. Two earlier lessons are load-bearing and kept: |
| +# tito was OOM-killed at a "256.2M memory peak" on 2026-08-15 when restored |
| +# copies ACCUMULATED (RestoreVerify now deletes each once checked — see its |
| +# comment), and tmpfs is why unfitting files were fatal rather than slow. |
| +# |
| +# WATCH: seapointish.bag.oncarlos.com.db was 412M against this 512M on |
| +# 2026-08-16. It will throttle, not fail, when it crosses — but raise this |
| +# then. |
| MemoryMax=512M |