amadan
server: cut litestream's idle S3 chatter ~10x (both replication roles) (#747)
Litestream's defaults are tuned for hot failover; both of our roles are gentler than that, yet the fleet burned ~113k S3 Tier-1 requests in ~14h (~$0.56) with near-zero writes — dominated by per-database retention/ compaction LISTs on 15s/30s timers (v0.5.14: the L0 retention monitor LISTs L1+L0 per DB every 15s; each compaction-level monitor costs up to two LISTs per DB per tick, L1 every 30s). Box-wide backup (deploy/fleet-backup.sh — generated /etc/litestream.yml): this is a backup, not failover, so recency trades for request volume: sync-interval 60s (RPO ~= one minute of a space's ciphertext; idle DBs upload nothing at any interval), compaction ladder 5m/30m/3h (same geometric shape as the 30s/5m/1h defaults), snapshots pinned daily with 72h retention (v0.5 enforces snapshot retention on the snapshot tick — there is no separate retention-check knob), and L0 retention checks hourly instead of every 15s (L0 files only become reclaimable after an L1 compaction, so hourly sweeps delete exactly the same files). Re-run the script on the box to regenerate the config and restart the unit. Embedded activator (internal/activator/replica.go): hibernating instances checkpoint deterministically at sleep via SyncAndWait, which calls db.Sync + Replica.Sync directly — independent of every interval here — so background compaction promptness buys nothing. Custom CompactionLevels L1 5m / L2 30m / L3 3h replace DefaultCompactionLevels, DB.MonitorInterval 1s -> 10s (local WAL poll, no S3), and the store's L0RetentionCheckInterval 15s -> 5m (matching L1 — checking more often than L1 compacts is pure waste). Replica.SyncInterval deliberately stays 1s: idle replica ticks make zero S3 calls, and while awake it bounds crash-loss to ~1s against the 30s lease TTL. Every compaction monitor still runs an immediate pass at store.Open, so each wake compacts whatever a previous short awake window left pending. Requests/day, current shape (6 always-on DBs + mostly-asleep fleet): before ~750/h per always-on DB idle (L0 retention 240 ticks x 2 LISTs + L1 120 x 2 + L2/L3) ~= 108k/day for the box backup alone, ~194k/day observed fleet-wide. After: ~31/h per fleet-backup DB (~4.5k/day for 6) and ~53/h per awake activator store; mostly-asleep instances pay a handful of requests per wake/sleep cycle. Fleet total lands in the mid-single-digit thousands/day — a 20-40x cut on the backup role and >10x fleet-wide (~$0.97/day -> ~$0.03/day). Verified: full go test ./... green; live suites against LCHAT_TEST_S3_BUCKET (hibernation cycle, mesh drain failover, mesh hard-kill failover, lease mutual exclusion) all pass with the new cadences — logs confirm the monitors start at 5m/30m/3h/24h and the sleep-time drain still ships the final TXID before the lease releases.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2 files changed,
+79
−1
deploy/fleet-backup.sh+37 −0internal/activator/replica.go+42 −1
diff --git a/deploy/fleet-backup.sh b/deploy/fleet-backup.sh| index b7dfa62..0cad248 100755 |
| --- a/deploy/fleet-backup.sh |
| +++ b/deploy/fleet-backup.sh |
| @@ -72,6 +72,43 @@ fi |
| { |
| echo "# generated by deploy/fleet-backup.sh — re-run the script, don't edit" |
| + # Cadence: this replicator is a BACKUP, not failover (the activator owns |
| + # failover for hibernating hosts), so every timer below trades recency for |
| + # S3 request volume. Litestream's defaults (sync 1s, L1 compaction 30s, L0 |
| + # retention LISTs every 15s) burned ~113k Tier-1 requests in 14h across the |
| + # fleet with near-zero writes — the retention/compaction monitors LIST per |
| + # database per tick whether or not anything changed. |
| + cat <<'YAML' |
| +# RPO: at most ~60s of a space's ciphertext is lost if the box dies — for a |
| +# backup of family-chat instances that is plenty; idle databases upload |
| +# nothing regardless (an idle sync tick makes no S3 calls). |
| +sync-interval: 60s |
| +# Compaction ladder, ~10x the 30s/5m/1h defaults with the same geometric |
| +# shape (restore read-amplification stays bounded). L1's tick is the request |
| +# driver — each level's monitor costs up to two LISTs per database per tick. |
| +levels: |
| + - interval: 5m |
| + - interval: 30m |
| + - interval: 3h |
| +# Snapshots stay daily — the gentle direction here is LESS often, not the |
| +# hourly one might reach for: each snapshot uploads a full database image. |
| +# Retention 72h keeps ~3 daily snapshots so one bad newest snapshot never |
| +# strands the restore chain. v0.5 enforces snapshot retention on the snapshot |
| +# tick itself (there is no separate retention-check knob), so retention |
| +# sweeps are daily too. |
| +snapshot: |
| + interval: 24h |
| + retention: 72h |
| +# The L0 retention monitor is the worst idle offender: every tick LISTs L1 |
| +# (and usually L0) per database even with zero writes, and its 15s default |
| +# exists for litestream's VFS read replicas, which we don't run. L0 files |
| +# only become reclaimable after an L1 compaction, so hourly sweeps delete |
| +# exactly the same files — just 240x fewer LISTs. l0-retention is the minimum |
| +# age before a compacted L0 file may go; 15m costs pennies of storage and |
| +# keeps recent increments around for point-in-time restores. |
| +l0-retention: 15m |
| +l0-retention-check-interval: 1h |
| +YAML |
| echo "dbs:" |
| for f in "${dbs[@]}"; do |
| [ -f "$f" ] || { echo "warning: $f missing, skipped" >&2; continue; } |
diff --git a/internal/activator/replica.go b/internal/activator/replica.go| index 29d2737..2a59389 100644 |
| --- a/internal/activator/replica.go |
| +++ b/internal/activator/replica.go |
| @@ -28,6 +28,45 @@ import ( |
| // ~30s of its last renewal — the failover window of the whole design. |
| const LeaseTTL = 30 * time.Second |
| +// compactionLevels replaces litestream.DefaultCompactionLevels (L1 30s / |
| +// L2 5m / L3 1h) with a ~10× gentler ladder. The defaults are tuned for a |
| +// long-lived replicator where background compaction promptness matters; a |
| +// hibernating instance instead checkpoints deterministically at sleep via |
| +// SyncAndWait (which calls db.Sync + Replica.Sync directly, independent of |
| +// every interval here), so aggressive background compaction buys nothing — |
| +// it only burns S3 Tier-1 LISTs: each level's monitor tick costs up to two |
| +// LISTs per DB, and L1 at 30s dominated the fleet's idle request bill. |
| +// Correctness is untouched — compaction is purely a restore-time |
| +// read-amplification optimisation (restore replays snapshot + whatever L0 |
| +// files exist), and each level's monitor still runs one immediate pass at |
| +// store.Open, so every wake compacts anything a previous short awake window |
| +// left pending. Same three-level geometric shape, so restore reads stay |
| +// bounded. |
| +var compactionLevels = litestream.CompactionLevels{ |
| + {Level: 0, Interval: 0}, |
| + {Level: 1, Interval: 5 * time.Minute}, |
| + {Level: 2, Interval: 30 * time.Minute}, |
| + {Level: 3, Interval: 3 * time.Hour}, |
| +} |
| + |
| +// dbMonitorInterval slows litestream's DB monitor (default 1s) — the local |
| +// WAL poll that shovels new frames into shadow LTX. It makes no S3 calls, |
| +// but 1s polling per awake instance is pointless churn when the sleep-time |
| +// SyncAndWait ships everything regardless; 10s keeps an awake instance's |
| +// background replication lag negligible against the 30s lease TTL. |
| +// (Replica.SyncInterval stays at its 1s default on purpose: idle replica |
| +// ticks make zero S3 calls — the upload loop no-ops when the local TXID |
| +// hasn't moved — and while awake it bounds crash-loss to ~1s.) |
| +const dbMonitorInterval = 10 * time.Second |
| + |
| +// l0RetentionCheckInterval slows the store's L0 retention monitor (default |
| +// 15s — tuned for litestream's VFS read replicas, which we don't run). Every |
| +// tick LISTs L1 and usually L0 per DB even with zero writes — the single |
| +// biggest idle-request source. L0 files only become reclaimable after an L1 |
| +// compaction, so checking more often than the L1 interval (5m above) is pure |
| +// waste; matching it cuts the LISTs 20× with no change to what gets deleted. |
| +const l0RetentionCheckInterval = 5 * time.Minute |
| + |
| // renewer keeps a held lease alive on a ticker. A failed renewal is recorded |
| // and renewal stops — leadership is lost and the owner must stand down; |
| // retrying could fence with a new claimant who took over after expiry. |
| @@ -165,13 +204,15 @@ func (h *HeldLease) Replicate(ctx context.Context, dbPath string) (*LeasedReplic |
| // error loop until their meta dirs were cleared by hand. A distinct |
| // namespace ends the class. |
| db.SetMetaPath(filepath.Join(filepath.Dir(dbPath), ".activator-"+filepath.Base(dbPath)+"-litestream")) |
| + db.MonitorInterval = dbMonitorInterval |
| client := lss3.NewReplicaClient() |
| client.Bucket = h.leaser.Bucket |
| client.Path = h.leaser.Path |
| client.Region = h.region |
| db.Replica = litestream.NewReplicaWithClient(db, client) |
| - store := litestream.NewStore([]*litestream.DB{db}, litestream.DefaultCompactionLevels) |
| + store := litestream.NewStore([]*litestream.DB{db}, compactionLevels) |
| + store.L0RetentionCheckInterval = l0RetentionCheckInterval |
| if err := store.Open(ctx); err != nil { |
| return nil, fmt.Errorf("open litestream store: %w", err) |
| } |