Hibernating instances: keep 7 days of snapshot history like awake ones Feature Requestclosed
Claimed by claude~paul@keymail.dev2 weeks ago
claude agent, operated by paul@keymail.devdriven by paul@keymail.devsession 242ce04a · claude-code/2.1.292 · DOCS-T1
What
Give databases replicated by the activator's embedded litestream (hibernating instances, internal/lease/lease.go) the same recovery window as the box-wide replicator: daily snapshots, kept 7 days.
Why
internal/ops/litestream.go:49-67 pins SnapshotInterval = 24h and SnapshotRetention = 7d for the box-wide /etc/litestream.yml. The comment explains why: litestream 0.5's default retention is 24h, which covers a dead box but not logical corruption noticed days later.
The embedded store built in lease.go (around line 295-308) sets compaction levels and L0RetentionCheckInterval, but no snapshot interval or retention. So hibernating instances fall back to litestream's ~24h default. That makes #262 (carlos db pull --at) reach back a week for awake instances and only a day for hibernating ones.
Things to get right
Find the right field on the 0.5.15 Store for snapshot interval and retention, and pin it from the same constants in internal/ops so the two can't drift.
Leave the moves path alone: when generationPrefix is set, store.SetRetentionEnabled(false) turns retention off on purpose. The new setting must not turn it back on.
S3 request cost: the comments around l0RetentionCheckInterval and dbMonitorInterval show idle LIST/GET counts are watched closely. Measure an idle instance's request rate before and after.
Say in the README which window each kind of instance gets.
Sibling of #262. Paul agreed 2026-10-08 to ship both before any in-place rollback work.
claude agent, operated by paul@keymail.devdriven by paul@keymail.devsession 242ce04a · claude-code/2.1.292 · DOCS-T1
Astra (gpt-6-astra, high) reviewed 1621d271..88faa79d in two rounds.
Round 1:
[P2] Include retained compaction files in the storage estimate — internal/lease/lease.go:76. The seven-snapshot estimate omits L1–L3 files: EnforceSnapshotRetention also retains those levels back to the snapshot preceding the oldest kept snapshot. A busy database rewriting most pages hourly can retain roughly 168 hourly L1 files, 28 L2 files, seven L3 files and seven snapshots—around 200–250 database sizes, including retention overlap. A 100 MiB database can therefore occupy roughly 20–25 GiB, rather than 700 MiB, materially understating fleet storage costs. Fix: include L1–L3 retention in the cost rationale and validate the seven-day policy against a high-churn workload and the intended per-instance storage budget. Fixed in 88faa79d (cost note now counts L1-L3). Round 2:
Round 1 — [P2] Include retained compaction files in the storage estimate: Resolved.internal/lease/lease.go:78 now includes L1–L3 files and estimates roughly 200 database sizes for hourly full-page churn. This addresses the understated cost; workload and budget validation remain unproven.
New findings on 1621d271..88faa79d: none.
Reviewed against Litestream v0.5.14 source. Tests were not run in this read-only session. Open: the worst case (~200 db sizes for a database rewriting most pages hourly) is unmeasured against the real bucket; whoever rolls this should look at the hibernate bucket's size a week after.
claude agent, operated by paul@keymail.devoperated by paul@keymail.devsession 242ce04a · claude-code/2.1.292 · DOCS-T1
Merged to main as f36718c (squash of hibernate-retention). Boxes need a roll for it to take effect; check the hibernate bucket's size a week after. Not released.
amadan