amadan
server: stretch S3 control-plane cadences — lease TTL 15m, mesh beat 60s, gentler compaction (#836)
The fleet's S3 bill is ~100% Tier-1 requests, not data: at 20 instances it ran ~$0.50/day (~$18/yr per always-awake instance vs €5/y plans). ~70% was the 30s lease TTL's 10s renewal PUTs, ~12% the 15s mesh heartbeat, ~13% the compaction/retention monitor LISTs. Data volume: 67 MB (~$0.0003/day). - LeaseTTL 30s → 15m (renew 5m): 8,640 → 288 PUTs/day per awake instance. Failover/crash-rewake window widens to ~15m, but only for HARD crashes — graceful sleep/deploy/shutdown release the lease explicitly. - meshHeartbeatInterval 15s → 60s: 11.5k → 2.9k req/day/box; dead-box detection 45s → 3m, still well inside the lease expiry that gates takeover. - Compaction ladder 5m/30m/3h → 1h/6h/24h and L0 retention check 5m → 1h: ~1.3k → ~110 LISTs/day per awake instance; compaction remains purely a restore-time read-amplification optimisation. Always-awake instance: ~$18/yr → ~$0.72/yr. Asleep instances stay zero.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
3 files changed,
+52
−29
docs/planet-scale.md+14 −7internal/activator/replica.go+28 −17mesh.go+10 −5
diff --git a/docs/planet-scale.md b/docs/planet-scale.md| index 95d7e8d..432827c 100644 |
| --- a/docs/planet-scale.md |
| +++ b/docs/planet-scale.md |
| @@ -123,9 +123,14 @@ correction the plan earned by testing (recorded on #550): the issue assumed |
| the lease was CLI config, but **the lease exists only in the library API — the |
| CLI silently absorbs a second replicator**, orphaning the loser's writes with |
| no error signal. Embedding it turned that from a footgun into the mechanism. |
| -TTL is 30 s, renewed every 10 s; a failed renewal means leadership is lost and |
| -the holder stands down — a wedged process is claimable by a healthy peer |
| -within ~30 s, which is the failover window of the whole design. Restores run |
| +TTL is 15 m, renewed every 5 m (each renewal is one S3 Tier-1 PUT — at the |
| +original 30 s/10 s cadence the lease heartbeat alone was ~70% of the fleet's |
| +S3 request bill, ~$16/yr per always-awake instance against €5/y plans); a |
| +failed renewal means leadership is lost and the holder stands down. The |
| +failover window follows the TTL: a wedged or hard-killed process is claimable |
| +by a healthy peer within ~15 m of its last renewal — every graceful path |
| +(sleep, deploy, shutdown) releases explicitly and hands over in seconds, so |
| +the window prices only real crashes. Restores run |
| *under* the held lease (restoring outside it could resurrect a stale copy that |
| shadows the true holder's newer state). |
| @@ -249,10 +254,11 @@ The multi-box mechanism from Part 4 v2 shipped as PR #678, gated on |
| codepaths, and the current single-box deployment runs exactly as before. |
| - **Membership is S3 heartbeats**, not gossip: each box PUTs one object per |
| - 15 seconds under the hibernation prefix's `mesh/members/`, and the object |
| + 60 seconds under the hibernation prefix's `mesh/members/`, and the object |
| *key* is the record (base64url of the peer URL) so a single LIST yields the |
| whole membership. Liveness reads S3's own `LastModified`, so a box with a |
| - wrong clock can't poison the staleness judgment. Stale after 45s. |
| + wrong clock can't poison the staleness judgment. Stale after 3 m (three |
| + missed beats — well inside the lease expiry that actually gates takeover). |
| - **Rendezvous (HRW) hashing is the placement hint** — `owner(host)` is a |
| pure function of the live member set; adding a box moves only the hosts |
| that land on it. **The lease stays the truth**: a box that wrongly believes |
| @@ -265,8 +271,9 @@ codepaths, and the current single-box deployment runs exactly as before. |
| - **Drain deletes the heartbeat first**, then checkpoints and releases as |
| usual — so deploys, scale-down, and box death are the same code path. The |
| kill-a-box property is a live test, both ways: a drained owner's instances |
| - are serving on the survivor ~2 seconds later; a SIGKILLed owner's within |
| - ~30 (the lease TTL), data intact in both. |
| + are serving on the survivor ~2 seconds later; a SIGKILLed owner's once its |
| + leases expire (the TTL — proven live at the original 30 s TTL, since raised |
| + to 15 m for S3 request cost), data intact in both. |
| Hardening that same week (PR #681, both lessons from the first real |
| production flip): flipping a host refuses while the box-wide backup |
diff --git a/internal/activator/replica.go b/internal/activator/replica.go| index 2a59389..b9b8faf 100644 |
| --- a/internal/activator/replica.go |
| +++ b/internal/activator/replica.go |
| @@ -23,37 +23,46 @@ import ( |
| lss3 "github.com/benbjohnson/litestream/s3" |
| ) |
| -// LeaseTTL matches litestream's own default. Renewal runs at TTL/3, so a |
| -// wedged process loses the lease (and a healthy peer can claim it) within |
| -// ~30s of its last renewal — the failover window of the whole design. |
| -const LeaseTTL = 30 * time.Second |
| +// LeaseTTL is deliberately long — 30× litestream's 30s default. Renewal runs |
| +// at TTL/3 and each renewal is one S3 Tier-1 PUT, so at the old 30s TTL the |
| +// lease heartbeat alone was 8,640 PUTs/day per awake instance (~$16/yr — |
| +// ~70% of the fleet's entire S3 bill, and more than a €5/y space's revenue); |
| +// at 15m it's 288/day (~$0.50/yr). The price is the failover window: a |
| +// HARD-crashed holder (kill -9, box death) strands the lease until expiry, |
| +// so a peer's takeover — or a re-wake on the same box — can stall up to |
| +// ~15m. Every graceful path (sleep, deploy, shutdown) releases the lease |
| +// explicitly and is unaffected. Shorten this only if fast crash-failover |
| +// ever outranks per-instance cost. |
| +const LeaseTTL = 15 * time.Minute |
| // compactionLevels replaces litestream.DefaultCompactionLevels (L1 30s / |
| -// L2 5m / L3 1h) with a ~10× gentler ladder. The defaults are tuned for a |
| +// L2 5m / L3 1h) with a far gentler ladder. The defaults are tuned for a |
| // long-lived replicator where background compaction promptness matters; a |
| // hibernating instance instead checkpoints deterministically at sleep via |
| // SyncAndWait (which calls db.Sync + Replica.Sync directly, independent of |
| // every interval here), so aggressive background compaction buys nothing — |
| // it only burns S3 Tier-1 LISTs: each level's monitor tick costs up to two |
| -// LISTs per DB, and L1 at 30s dominated the fleet's idle request bill. |
| +// LISTs per DB even with zero writes — L1 at 30s dominated the fleet's idle |
| +// request bill, and the first fix (5m/30m/3h) still left ~1.3k LISTs/day per |
| +// awake instance once the lease PUTs were tamed. |
| // Correctness is untouched — compaction is purely a restore-time |
| // read-amplification optimisation (restore replays snapshot + whatever L0 |
| // files exist), and each level's monitor still runs one immediate pass at |
| // store.Open, so every wake compacts anything a previous short awake window |
| // left pending. Same three-level geometric shape, so restore reads stay |
| -// bounded. |
| +// bounded: worst case a hard crash replays up to ~an hour of L0 files. |
| var compactionLevels = litestream.CompactionLevels{ |
| {Level: 0, Interval: 0}, |
| - {Level: 1, Interval: 5 * time.Minute}, |
| - {Level: 2, Interval: 30 * time.Minute}, |
| - {Level: 3, Interval: 3 * time.Hour}, |
| + {Level: 1, Interval: 1 * time.Hour}, |
| + {Level: 2, Interval: 6 * time.Hour}, |
| + {Level: 3, Interval: 24 * time.Hour}, |
| } |
| // dbMonitorInterval slows litestream's DB monitor (default 1s) — the local |
| // WAL poll that shovels new frames into shadow LTX. It makes no S3 calls, |
| // but 1s polling per awake instance is pointless churn when the sleep-time |
| -// SyncAndWait ships everything regardless; 10s keeps an awake instance's |
| -// background replication lag negligible against the 30s lease TTL. |
| +// SyncAndWait ships everything regardless; 10s bounds an awake instance's |
| +// crash-loss window to ~10s, plenty tight for chat. |
| // (Replica.SyncInterval stays at its 1s default on purpose: idle replica |
| // ticks make zero S3 calls — the upload loop no-ops when the local TXID |
| // hasn't moved — and while awake it bounds crash-loss to ~1s.) |
| @@ -63,9 +72,10 @@ const dbMonitorInterval = 10 * time.Second |
| // 15s — tuned for litestream's VFS read replicas, which we don't run). Every |
| // tick LISTs L1 and usually L0 per DB even with zero writes — the single |
| // biggest idle-request source. L0 files only become reclaimable after an L1 |
| -// compaction, so checking more often than the L1 interval (5m above) is pure |
| -// waste; matching it cuts the LISTs 20× with no change to what gets deleted. |
| -const l0RetentionCheckInterval = 5 * time.Minute |
| +// compaction, so checking more often than the L1 interval (1h above) is pure |
| +// waste; matching it cuts the LISTs 240× vs the default with no change to |
| +// what gets deleted. |
| +const l0RetentionCheckInterval = 1 * time.Hour |
| // renewer keeps a held lease alive on a ticker. A failed renewal is recorded |
| // and renewal stops — leadership is lost and the owner must stand down; |
| @@ -293,8 +303,9 @@ func (r *LeasedReplicator) Close(ctx context.Context) error { |
| if lease != nil { |
| // The release gets its OWN budget, never the caller's: a wedged final |
| // sync exhausts ctx, and releasing under a dead context leaves the |
| - // lease held until TTL — the next wake then 503s for up to 30s over |
| - // nothing (hit live on the first real instance flip). The release is |
| + // lease held until TTL — the next wake then 503s for up to the full |
| + // TTL over nothing (hit live on the first real instance flip). The |
| + // release is |
| // one small S3 DELETE; five seconds is generous. |
| relCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) |
| defer cancel() |
diff --git a/mesh.go b/mesh.go| index 12197c8..adb0c9f 100644 |
| --- a/mesh.go |
| +++ b/mesh.go |
| @@ -75,14 +75,19 @@ const meshForwardedHeader = "X-Lchat-Mesh-Forwarded" |
| const ( |
| // meshHeartbeatInterval paces the PUT + LIST beat. Membership changes are |
| - // rare events; this is liveness, not coordination. |
| - meshHeartbeatInterval = 15 * time.Second |
| + // rare events; this is liveness, not coordination — and each beat is two |
| + // S3 Tier-1 requests, so 15s cost ~11.5k requests/day per box (~$21/yr) |
| + // for nothing membership-shaped in return. Slower still would work, but |
| + // staleness (3 beats) rides on this, and 3m dead-box detection already |
| + // sits well inside the lease expiry that actually gates takeover. |
| + meshHeartbeatInterval = 60 * time.Second |
| // meshStaleAfter is how old a heartbeat may be before its box is presumed |
| // dead — three missed beats. It's judged against the reader's clock but |
| // the timestamp is S3's LastModified (the bucket's clock), so only |
| - // reader-vs-S3 skew matters, tiny against 45s. Together with the ~30s |
| - // lease TTL this bounds failover: a hard-killed box's hosts wake elsewhere |
| - // once its heartbeat is stale AND its leases expire. |
| + // reader-vs-S3 skew matters, tiny against 3m. Together with the lease |
| + // TTL (activator.LeaseTTL, 15m) this bounds failover: a hard-killed box's |
| + // hosts wake elsewhere once its heartbeat is stale AND its leases expire — |
| + // lease-dominated, so ~15m. |
| meshStaleAfter = 3 * meshHeartbeatInterval |
| ) |