amadan
Merge pull request #205 from carlosframework/verify-tmp-rationale
verify drills: correct the tmpfs rationale with what the box measured
3 files changed,
+65
−30
docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md+26 −6internal/host/units/carlos-hibernate-verify.service+9 −8internal/host/units/carlos-restore-verify.service+30 −16
diff --git a/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md b/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md| index f1a8332ce..420b7844c 100644 |
| --- a/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md |
| +++ b/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md |
| @@ -172,16 +172,36 @@ largest single database — but on tmpfs that still means **the ceiling has to |
| exceed the biggest tenant**, so the drill would begin failing on exactly the |
| databases it exists to vouch for, about 30M of growth away. |
| -The fix is to stop restoring into RAM: both drill units now carry |
| -`CacheDirectory=carlos-verify` and `Environment=TMPDIR=%C/carlos-verify`, so |
| -restores land on disk and `MemoryMax` bounds litestream's working set (tens of |
| -MB) rather than tenant size. Pinned by |
| +Both drill units now carry `CacheDirectory=carlos-verify` and |
| +`Environment=TMPDIR=%C/carlos-verify`, so restores land on disk. Pinned by |
| `TestVerifyDrillsRestoreToDiskNotTmpfs`, because it is a one-line property |
| that regresses silently — the drill keeps passing right up until a tenant |
| grows. |
| -The general lesson, worth carrying: **under `PrivateTmp=yes`, temp files are |
| -memory.** Any unit that stages data through /tmp is charged for it. |
| +**What that actually buys, stated correctly.** The first framing of this fix |
| +(in PR #204) claimed the ceiling would then bound litestream's working set |
| +rather than tenant size. That is WRONG, and the box said so: page cache is |
| +charged to the cgroup just as tmpfs is, so the reported peak barely moved |
| +(485.6M on tmpfs → 494.4M on disk, same 412M tenant; tito's box-wide drill did |
| +fall, 194.9M → 141.3M). What changes is not the number but **what the ceiling |
| +does when the restore does not fit**: |
| + |
| +| | hitting `MemoryMax` | |
| +| --- | --- | |
| +| tmpfs (unevictable) | **kills** the drill | |
| +| page cache (reclaimable) | **throttles** it | |
| + |
| +Measured on the flagship, same 412M database, `MemoryMax=128M`: on tmpfs, |
| +oom-killed after 8.3s; on disk, still running after ten minutes under reclaim, |
| +stopped by hand. So a tenant outgrowing the ceiling now degrades instead of |
| +failing — but size is not irrelevant, because a badly undersized ceiling makes |
| +the drill crawl. `MemoryMax` must still sit comfortably above the largest |
| +single database. (Watch item: `seapointish.bag.oncarlos.com.db` was 412M |
| +against a 512M ceiling on 2026-08-16.) |
| + |
| +The general lesson, worth carrying: **files a unit stages are charged to its |
| +cgroup — tmpfs fatally, page cache survivably.** The fix for a drill that OOMs |
| +is to get the bytes onto disk; the fix for one that crawls is a bigger ceiling. |
| ## 6. Known follow-ups |
diff --git a/internal/host/units/carlos-hibernate-verify.service b/internal/host/units/carlos-hibernate-verify.service| index e491c837f..3d0880649 100644 |
| --- a/internal/host/units/carlos-hibernate-verify.service |
| +++ b/internal/host/units/carlos-hibernate-verify.service |
| @@ -29,11 +29,12 @@ Type=oneshot |
| # wrong store. |
| EnvironmentFile=-/etc/carlos/host.env |
| ExecStart=/opt/carlos/carlos ops hibernate-verify |
| -# Disk, not the PrivateTmp tmpfs — see carlos-restore-verify.service's note. |
| -# This drill hit it hardest: the flagship's largest hibernating tenant is 412M |
| -# and the first live run peaked at 485.6M against a 512M ceiling, i.e. the |
| -# drill was ~30M of tenant growth away from failing on the very databases it |
| -# is there to vouch for. |
| +# Disk, not the PrivateTmp tmpfs — see carlos-restore-verify.service's note |
| +# for the measurements. This drill is where it mattered most: the flagship's |
| +# largest hibernating tenant is 412M and the first live run peaked at 485.6M |
| +# against a 512M ceiling. On tmpfs that gap was ~30M of tenant growth away |
| +# from an OOM kill on the very databases the drill is there to vouch for; on |
| +# disk the same overrun only slows it down. |
| CacheDirectory=carlos-verify |
| Environment=TMPDIR=%C/carlos-verify |
| User=carlos |
| @@ -50,7 +51,7 @@ PrivateTmp=yes |
| PrivateDevices=yes |
| RestrictAddressFamilies=AF_INET AF_INET6 |
| MemoryDenyWriteExecute=yes |
| -# Bounds the working set, not the databases (TMPDIR is on disk above). This |
| -# drill is one-at-a-time by construction — lease.VerifyReplica deletes each |
| -# throwaway copy in a defer — so the same generous 512M ceiling applies. |
| +# One-at-a-time by construction — lease.VerifyReplica deletes each throwaway |
| +# copy in a defer — so this needs to clear the largest single tenant, not the |
| +# fleet. Same ceiling and the same WATCH note as the sibling unit. |
| MemoryMax=512M |
diff --git a/internal/host/units/carlos-restore-verify.service b/internal/host/units/carlos-restore-verify.service| index 6fb57944e..c5ebeed2f 100644 |
| --- a/internal/host/units/carlos-restore-verify.service |
| +++ b/internal/host/units/carlos-restore-verify.service |
| @@ -13,14 +13,25 @@ Wants=network-online.target |
| Type=oneshot |
| ExecStart=/opt/carlos/carlos ops restore-verify -config /etc/litestream.yml |
| EnvironmentFile=/data/litestream.env |
| -# Restore into a DISK-backed directory, not the PrivateTmp tmpfs. A restored |
| -# database on tmpfs is charged to this cgroup as memory, so the drill's peak |
| -# tracked the size of the fleet's largest tenant and would OOM the moment one |
| -# outgrew MemoryMax — measured on the flagship 2026-08-15, a 412M tenant |
| -# produced a 485.6M peak against a 512M ceiling. On disk, the peak is |
| -# litestream's own working set (tens of MB) and the drill stops competing with |
| -# the tenants it exists to protect. systemd creates and cleans %C/carlos-verify |
| -# and binds it writable despite ProtectSystem=strict. |
| +# Restore into a DISK-backed directory, not the PrivateTmp tmpfs. Both are |
| +# charged to this cgroup — page cache is accounted just like tmpfs, so the |
| +# reported peak barely moves (flagship: 485.6M on tmpfs, 494.4M on disk, for a |
| +# 412M tenant). What changes is what the ceiling DOES when the restore does not |
| +# fit, and that is the whole point: |
| +# |
| +# tmpfs pages are unevictable -> hitting MemoryMax KILLS the drill |
| +# page cache is reclaimable -> hitting MemoryMax THROTTLES it |
| +# |
| +# Measured on the flagship 2026-08-16, same 412M database, MemoryMax=128M: |
| +# on tmpfs, oom-killed after 8.3s; on disk, still running after 10 minutes |
| +# under reclaim (stopped by hand). So the restored file no longer has to FIT |
| +# in the cgroup's memory — a tenant outgrowing the ceiling degrades instead of |
| +# failing. |
| +# |
| +# It does NOT make size irrelevant: a badly undersized ceiling makes the drill |
| +# crawl. Keep MemoryMax comfortably above the largest tenant. systemd creates |
| +# and cleans %C/carlos-verify and binds it writable despite |
| +# ProtectSystem=strict. |
| CacheDirectory=carlos-verify |
| Environment=TMPDIR=%C/carlos-verify |
| User=carlos |
| @@ -33,12 +44,15 @@ PrivateTmp=yes |
| PrivateDevices=yes |
| RestrictAddressFamilies=AF_INET AF_INET6 |
| MemoryDenyWriteExecute=yes |
| -# With TMPDIR on disk (above) this bounds litestream's WORKING SET, not the |
| -# databases — measured 194.9M for tito's 27 replicas and 485.6M on the |
| -# flagship back when a 412M tenant still landed in tmpfs. A ceiling costs |
| -# nothing unused, so it stays generous. Two earlier lessons are kept here |
| -# because both are load-bearing: tito was OOM-killed at "256.2M memory peak" |
| -# on 2026-08-15 when restored copies ACCUMULATED (RestoreVerify now deletes |
| -# each once checked — see its comment), and tmpfs is why the files counted as |
| -# memory at all. |
| +# Sized to sit ABOVE the largest single database, not to bound the process: |
| +# restored bytes are charged here either way (see the TMPDIR note). Measured |
| +# after the move to disk: tito's 27 replicas 141.3M (was 194.9M on tmpfs), the |
| +# flagship's 412M tenant 494.4M. Two earlier lessons are load-bearing and kept: |
| +# tito was OOM-killed at a "256.2M memory peak" on 2026-08-15 when restored |
| +# copies ACCUMULATED (RestoreVerify now deletes each once checked — see its |
| +# comment), and tmpfs is why unfitting files were fatal rather than slow. |
| +# |
| +# WATCH: seapointish.bag.oncarlos.com.db was 412M against this 512M on |
| +# 2026-08-16. It will throttle, not fail, when it crosses — but raise this |
| +# then. |
| MemoryMax=512M |