carlos-litestream is OOM-killed in a loop at the 256M default with 100 databases, and nothing reports a stalled replica Issueopen
Claimed by claude~paul@keymail.dev1 week ago
claude agent, operated by paul@keymail.devdriven by paul@keymail.devsession d2936192 · claude-code/2.1.292 · DOCS-T1
On carlos-tito-ie (i-025a0b70f3d93304d) carlos-litestream.service was OOM-killed in a restart loop from about 12:00 UTC on 2026-10-13 until it was given more memory at 14:06 UTC. Nothing reported it; it was found by a person asking whether backups were running.
What happened
The unit ships with MemoryMax=256M. It replicates 100 databases from /etc/litestream.yml (litestream 0.5.15, 1.3 GB of SQLite in total, the largest 236 MB).
Kernel log, repeated every ~6s: Memory cgroup out of memory: Killed process … (litestream) … anon-rss:251140kB. The process lived about 4 seconds per start; RestartSec=2.
95 databases still kept up, because their changes fit in a 4-second life. The five largest (72–236 MB, all import mirrors) stopped shipping entirely: about 250 uploads an hour between them before, 69 in the 12:00 hour, 0 in the 13:00 hour. They were 1.3–2.2 hours behind when measured at 13:45.
With the cap raised to 1G the unit has run without a kill: anon sits at 229–247 MB, the rest is page cache (memory.events: max 2645, oom_kill 0). All five caught up within 90 seconds. So steady-state anonymous memory for 100 databases is already at the old cap, and this was growth past a limit, not a one-off spike. It will recur on any box as the database count rises.
What is on the box now
A drop-in, /etc/systemd/system/carlos-litestream.service.d/memory-max.conf, setting MemoryMax=1G (Paul, 2026-10-13). It is a workaround and should go when the unit's own default is right.
Asks
The unit default. 256M does not hold 100 databases. Either raise it, or size it from the number of databases in the generated config.
A freshness signal.carlos-restore-verify passed at 00:13 the same day: it proves a replica restores, not that it is recent. Something should compare each database's newest replica object with its local position and go red past a threshold, for both replicators (this unit, and the agent's in-process replication to instances/<host>/). A unit sitting in activating (auto-restart) with NRestarts in the thousands should also be surfaced by itself.
Somewhere to see it. A replication column on the console's instance list (last replicated, lag) would have answered the original question without SSM. It needs to push an alert as well; a page nobody is looking at would not have caught this either.
The log flood erased the evidence. Each restart logs two lines per database, so 900+ restarts pushed the unit's journal retention down to under two hours and the start of the incident is gone. The start time above is inferred from S3 upload timestamps.
Not verified
Three platform-run import mirrors (apropos, productatheart, fronteers-conference) show a replica 5–27 hours older than the database file's mtime. That may only be checkpoint churn on a sleeping instance; transaction ids were not compared.
claude agent, operated by paul@keymail.devoperated by paul@keymail.devsession c4a04fdc · claude-code/2.1.292 · DOCS-T1
Claimed. Work is on branch litestream-oom-284; its tasks carry the plan. State as of 2026-10-13 23:50 IST:
Built and pushed (516ffbd2)
Ask 1, the cap. carlos ops litestream now writes a systemd drop-in (carlos-litestream.service.d/50-memory.conf) with MemoryMax sized from the number of databases in the config it just generated: 256M + 8M a database, floor 512M, so 100 databases get 1056M. It also sets GOMEMLIMIT to three quarters of the cap, because Go does not read the cgroup limit and otherwise lets the heap double before collecting. The unit's own default rises from 256M to 512M for boxes whose config is not generated. The formula is fitted to this thread's one data point (1G held 100 databases). A memory.peak reading from the box after a restart would let it be tightened.
Ask 4, the flood. RestartSteps=5 and RestartMaxDelaySec=60 on the unit: a looping replicator restarts once a minute instead of every 6 seconds. 60s is the ceiling because the unit being down is a backup gap.
Ask 2, second half. A restarts alert rule: the agent reads NRestarts for every non-template carlos-*.service each tick and fires when one unit restarts more than 3 times in 10 minutes. It clears after 10 minutes with none.
The hand-made memory-max.conf drop-in on the affected box sorts after 50-memory.conf, so it keeps winning until someone removes it.
A correction to the opening post.carlos-restore-verify does check currency, not only that the replica restores: internal/ops/freshness.go compares SQLite's change counter in the restored copy with the live file and fails a replica that trails a source quiet for 15 minutes. It did not catch this incident for two reasons: it runs once a day (it passed 12 hours before the loop began), and a database that is written continuously never goes quiet for 15 minutes, so the busiest databases are never judged stale. The import mirrors were exactly that case.
Still to do on the branch
The freshness signal proper, for both replicators, and the console's replication column (design first).
Alert wording for the new rules, through copy review.
A census overlay for the root-read changes in internal/ops and the unit file (the plan14 census test fails until then).
Nothing here is in v0.27.0.
claude agent, operated by paul@keymail.devdriven by paul@keymail.devsession c4a04fdc · claude-code/2.1.292 · DOCS-T1
State as of 2026-10-14.
Slice A is built and reviewed on litestream-oom-284 (fa63c1ea). Astra reviewed the code five times; the fifth pass says MERGE-READY. What changed from the first post here:
The restarts alert is per unit (restarts@<unit>). A second unit that starts looping while one alert is up is its own page. The watch counts dated rises of systemd's counter, so a manual restart (which zeroes it) no longer hides a loop; it is saved to <data>/monitor/restarts.json and sampled at agent start, so the agent's own restart loop is counted; a gap in sampling longer than the window, or a clock stepped backwards, starts the count again instead of dating old restarts to now. An alert clears only after ten quiet minutes with the unit not failed.
The memory cap is written atomically, under a lock on the drop-in's directory, and sized from the config on disk at that moment. A reload systemd did not take is remembered in a marker file and retried by the next run. systemctl calls have deadlines.
Limit, stated plainly: an agent that dies within seconds of starting cannot report its own loop from the inside.
Not merged. It waits on Paul's copy review of ten strings, a census ruling asked on #280, and his go-ahead. Nothing here is in v0.27.0.
Slice B (asks 2 and 3: freshness and the console column) is not built. Two designs were rejected in review; the reasoning and a proposed change of mechanism are on #290, waiting on Paul. The short version: sampling litestream's own bookkeeping from outside cannot prove a replica is current, and for the box-wide binary there is no outside evidence of capture at all. Work so far is parked on replica-freshness-284.
Until slice B exists, what tells you a replica has stalled is: the restarts alert (a crash loop, within about a minute), and the daily restore-verify, with the blind spot noted above for databases that are never quiet.
amadan