eleven / ocho Public

docs: the durability ladder — ops guide, tier economics, honest backup copy (#870)

- planet-scale guide: replication-classes section on the hibernation page
  (the three rungs, the two invariants graceful-flush can't weaken, the
  lchat replication CLI, billing-driven classing), reference entries for the
  command/env/glossary, and the stale pre-#836 lease numbers fixed (30s → 15m)
  on the hibernation and mesh pages.
- tier-economics.md: durability-per-tier table with per-class S3 request
  costs, the replication line item in the cost model, and the decision
  record: ONE Business plan with best-available durability built in — no
  Business Plus (a €2–20/yr resilience delta can't carry a second paid tier);
  contractual RPO/RTO if ever demanded is an Enterprise conversation, not a
  checkbox.
- changelog: member-facing entry with the honest framing — backed up every
  time a space goes quiet + at least nightly; a sudden server loss can cost
  recent hours on Free/Family, minutes on Community, seconds on Business.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Paul Campbell 3 months ago ea50bbe8df56a974b68859f9fff375e892f91344 parent f38421c

5 files changed, +91 −7

changelog/2026-07-14-backup-ladder.md +12 −0
docs/planet-scale-guide/hibernation.html +23 −3
docs/planet-scale-guide/mesh.html +1 −1
docs/planet-scale-guide/reference.html +7 −2
docs/tier-economics.md +48 −1

diff --git a/changelog/2026-07-14-backup-ladder.md b/changelog/2026-07-14-backup-ladder.md

new file mode 100644

index 0000000..d8dc8e8

--- /dev/null

+++ b/changelog/2026-07-14-backup-ladder.md

@@ -0,0 +1,12 @@

+🤖 For spaces hosted with us, backups now match your plan — and we're being

+precise about what each plan promises. Every space is backed up in full **every time it goes

+quiet**, and again whenever we update or restart a server, so day to day

+your history is safe on every plan. The difference is what happens if a

+server is lost *suddenly* — a hardware failure with no warning. Business

+spaces stream every change off the server within seconds, so a sudden

+failure loses almost nothing. Community spaces sync every few minutes.

+Free and Family spaces sync when they go quiet and at least once a day —

+so in that rare worst case, the most recent hours of activity could be

+lost, and everything older is safe. Nothing changes about encryption:

+backups are the same locked data the server already holds, and no backup

+can be read without your keys.

diff --git a/docs/planet-scale-guide/hibernation.html b/docs/planet-scale-guide/hibernation.html

index fd63283..146824a 100644

--- a/docs/planet-scale-guide/hibernation.html

+++ b/docs/planet-scale-guide/hibernation.html

@@ -51,14 +51,14 @@ lchat instances # the HIBERNATE column shows who is

<h2 id="the-lease">The lease: one writer, enforced by the bucket itself</h2>

<p>The central safety problem of hibernation is: <em>what guarantees that two processes never write the same space’s database at once?</em> (Two boxes might both think they should wake a space; a wedged old process might still be running when a new one starts.) The answer is <strong>the lease</strong> — a small lock object in the bucket, at <code><prefix>/<host>/lock.json</code>, acquired using S3’s conditional writes (an operation that atomically fails if the object already exists). Whoever holds a space’s lease is the one writer; everyone else fails loudly and immediately.</p>

<ul>

- <li>The lease has a <strong>30-second lifetime</strong> and its holder renews it roughly every <strong>10 seconds</strong>. A process that stops renewing — crashed, wedged, unplugged — loses the lease within about 30 seconds, and a healthy process elsewhere can then claim it. <strong>That ~30 seconds is the failover window of the entire design.</strong></li>

+ <li>The lease has a <strong>fifteen-minute lifetime</strong> and its holder renews it roughly every <strong>five minutes</strong> (stretched from the original 30s/10s in #836: each renewal is one S3 Tier-1 PUT, and at the old cadence the lease heartbeat alone was ~70% of the fleet’s entire S3 bill). A process that stops renewing — crashed, wedged, unplugged — loses the lease within about fifteen minutes, and a healthy process elsewhere can then claim it. <strong>That ~15 minutes is the <em>hard-crash</em> failover window of the design</strong>; every graceful path (sleep, deploy, shutdown) releases the lease explicitly and is unaffected.</li>

<li>A failed renewal is terminal on purpose: the holder <em>stands down</em> (checkpoints nothing further, stops serving) rather than retrying, because a retry could collide with a new holder that legitimately took over after expiry.</li>

<li>Restoring a database from the bucket happens only <em>while holding the lease</em>. Restoring outside it could resurrect a stale copy that shadows the true holder’s newer state.</li>

</ul>

<p>Replication itself is performed by <a href="reference.html#gloss-litestream">Litestream</a> — a well-known tool that continuously streams SQLite changes to object storage — embedded <em>as a library</em> inside the router. The embedding is not an implementation detail but a hard-won correction: the Litestream command-line tool, run twice against one database, <em>silently</em> absorbs the second copy and orphans one side’s writes with no error at all; only the library interface exposes the lease that turns a second writer into a loud, typed failure. The build chose the loud failure.</p>

<span class="label">In plain words</span>

- <p>Every sleeping space has a key hanging on a hook in the shared storage room, and the rule is absolute: you may only touch a space’s data while you are holding its key, and the key dissolves in your hand if you stop checking in every few seconds. Two people can never hold it at once, and a collapsed keyholder drops it within half a minute.</p>

+ <p>Every sleeping space has a key hanging on a hook in the shared storage room, and the rule is absolute: you may only touch a space’s data while you are holding its key, and the key dissolves in your hand if you stop checking in every few minutes. Two people can never hold it at once, and a collapsed keyholder drops it within a quarter of an hour.</p>

</div>

<h2 id="lifecycle">The life of a hibernating space</h2>

@@ -91,7 +91,7 @@ lchat instances # the HIBERNATE column shows who is

<li>The activator <strong>claims the lease</strong> at <code><prefix>/<host>/lock.json</code>. If another holder has it (for example, another box in a <a href="mesh.html">mesh</a>), the request is answered <code>503</code> with a <code>Retry-After</code> header — deliberately: “sleeping spaces are slow to wake” must degrade to a polite retry, never a conflict.</li>

<li>It <strong>restores the database from the bucket only if the local file is missing</strong>. A database already on disk is trusted as-is — on a single box, under the lease, the local copy can only be this box’s own last checkpoint, so most wakes skip the download entirely. This is the “warm cache”, and it is why routine wakes are fast.</li>

<li>It <strong>starts the instance as a direct child process</strong> of the router — the same versioned binary its systemd unit would run (<code>/opt/lchat/live/<host></code>), with the same environment file (<code>/data/<host>.env</code>) — and waits up to 30 seconds for the socket to answer.</li>

- <li>It <strong>starts replication under the held lease</strong>, so every subsequent write streams to the bucket within about a second, and finally spawns the bot sidecar if one is staged (<a href="#sidecar-co-spawn">below</a>). Then the waiting requests proceed.</li>

+ <li>It <strong>starts replication under the held lease</strong> — at the host’s <a href="#replication-classes">replication class</a>’s cadence: within about a second on the default <code>stream</code> class, every five minutes on <code>batched</code>, daily on <code>daily</code> — and finally spawns the bot sidecar if one is staged (<a href="#sidecar-co-spawn">below</a>). Then the waiting requests proceed.</li>

</ol>

<p>The entire wake has a five-minute budget; a queued request waits at most 30 seconds before giving up with a 503. A <em>failed</em> wake discards all bookkeeping so the next request retries from scratch — one bad wake never becomes a stuck host.</p>

<h3 id="sleep">Sleeping: what happens when a space goes idle</h3>

@@ -106,6 +106,26 @@ lchat instances # the HIBERNATE column shows who is

<h3 id="failure">Failure is just the wake path again</h3>

<p>Every failure mode collapses into “stand down; the next request re-wakes”: a child that dies unexpectedly, a lease lost to a renewal failure, a checkpoint that errors. And an S3 outage while spaces are <em>awake</em> is survivable by design — the local SQLite file is authoritative while the lease is held, replication queues and drains when the bucket returns, and only <em>cold</em> wakes (no local file) actually require the bucket to answer.</p>

+<h2 id="replication-classes">Replication classes: the per-plan durability ladder</h2>

+<p>While a space is awake, how often its writes reach the bucket is a <strong>per-host class</strong> on the registry row (<code>routes.replication</code>), because the change-driven sync is the one S3 cost that scales with how busy a space is — pennies a year on a quiet space, potentially ~€145/year on one that writes every second around the clock. Three rungs:</p>

+<div class="tablewrap">

+<table>

+ <tr><th>Class</th><th>Awake sync cadence</th><th>What a hard box crash can lose</th><th>Who gets it</th></tr>

+ <tr><td><code>stream</code> (default, and every pre-ladder host)</td><td>change-driven, ~1s</td><td>seconds</td><td>Business; operator-run hosts unless set otherwise</td></tr>

+ <tr><td><code>batched</code></td><td>at most every 5 minutes</td><td>minutes</td><td>Community & Friends</td></tr>

+ <tr><td><code>daily</code></td><td>at wake, then at most every 24 hours</td><td>up to a day</td><td>Free and Family</td></tr>

+</table>

+</div>

+<p>Two invariants the class can never weaken, worth internalising before touching it: <strong>every graceful path still flushes in full</strong> — idle sleep, deploys, restarts and shutdowns all checkpoint the space to the bucket regardless of class — and therefore <strong>a sleeping space’s parked copy is always exactly current</strong>. The class bounds only what a <em>sudden</em> box loss can take, and only from the current awake window. (Corollary: a space with a bot sidecar never sleeps, so on the <code>daily</code> rung its real cadence is the daily tick — the one profile where “up to a day” is the everyday exposure, not a rare worst case.)</p>

+<figure class="cmd">

+<figcaption>Run on the box, as root — the class applies at the host’s next wake</figcaption>

+<pre><code>lchat replication chat-b.example.com # read the current class

+lchat replication chat-b.example.com daily # set it (stream | batched | daily) + reload the router

+lchat instances # the REPLICATION column shows every host’s rung

+lchat add-instance test-x.11ch.at --hibernate --replication daily # born on a rung</code></pre>

+</figure>

+<p>On provisioned fleets the class is <strong>billing-driven, not hand-set</strong>: with <code>LCHAT_CLOUD_REPLICATION_CLASSES</code> configured on the cloud service (<code>free=daily,family=daily,community=batched,business=stream</code>), spaces are born on the free rung, Stripe webhook events move them on every upgrade or lapse, and the meter sweep reconciles drift — the same event-driven-plus-sweep shape as storage caps. controld records the class and reloads the router only on an actual change; nothing restarts, no sleeping space wakes, and an awake instance keeps its current cadence until it next sleeps. A misconfigured <em>paid</em> tier deliberately maps to <code>stream</code> — config typos fail toward durability, never away from what a member paid for. See <code>docs/cloud-provisioner.md</code> for the config and <code>docs/tier-economics.md</code> for the cost table behind the rungs.</p>

<h2 id="due-time-wakes">Scheduled deliveries wake sleeping spaces on time</h2>

<p>Spaces can hold scheduled events — a reminder due at 9:00, a message scheduled for tomorrow. A naive hibernation design would sleep through them. Instead, at sleep time the space reports its earliest pending delivery, the activator persists that timestamp in the registry (so it survives router restarts), and the same background sweep wakes any sleeping host whose time has arrived — no member request needed. Worst-case lateness is about one sweep interval (thirty seconds) plus a wake. Bot schedules created by members (“send me a news summary every morning”) live encrypted in the sidecar, which the server cannot read — so the sidecar itself reports its earliest scheduled run to the instance (one bare timestamp crosses; the schedule stays ciphertext), and the instance folds it into the same answer.</p>

diff --git a/docs/planet-scale-guide/mesh.html b/docs/planet-scale-guide/mesh.html

index 8ecf8b9..e33626c 100644

--- a/docs/planet-scale-guide/mesh.html

+++ b/docs/planet-scale-guide/mesh.html

@@ -101,7 +101,7 @@ LCHAT_MESH_DOMAINS=11ch.example.com

<figcaption>Run on the box being drained</figcaption>

<pre><code>systemctl stop lchat-router # heartbeat deleted, spaces checkpointed, leases released</code></pre>

</figure>

-<p>The recovery numbers, verified by live tests in both directions: after a <em>graceful</em> drain, the survivor is serving the drained box’s spaces about <strong>two seconds</strong> later; after a <em>hard kill</em> (no drain, heartbeat left to rot), within about <strong>thirty seconds</strong> — the lease lifetime — with data intact in both cases, because every write had been streaming to the bucket all along.</p>

+<p>The recovery numbers, verified by live tests in both directions: after a <em>graceful</em> drain, the survivor is serving the drained box’s spaces about <strong>two seconds</strong> later; after a <em>hard kill</em> (no drain, heartbeat left to rot), within about <strong>the lease lifetime</strong> — thirty seconds when those tests ran, <strong>fifteen minutes</strong> since #836 stretched the lease to tame its S3 request bill — with data intact in both cases (up to the space’s <a href="hibernation.html#replication-classes">replication class</a>’s window), because writes had been syncing to the bucket all along.</p>

<h2 id="mesh-limits">The honest limits: what stands between here and a true multi-box fleet</h2>

<p>Three gaps, in decreasing order of consequence. None of them is speculative — each corresponds to a concrete piece of code that exists, is reserved, or was withdrawn:</p>

diff --git a/docs/planet-scale-guide/reference.html b/docs/planet-scale-guide/reference.html

index aa2e3c9..c4253c4 100644

--- a/docs/planet-scale-guide/reference.html

+++ b/docs/planet-scale-guide/reference.html

@@ -39,8 +39,10 @@

<dd>Deletes the host’s registry row and reloads the router (traffic stops); with <code>--stop</code> also stops and disables the instance and sidecar units. <strong>Never touches data.</strong> Explained in <a href="spaces.html#removing">part 05</a>.</dd>

<dt><code>lchat hibernate <host> --on | --off</code></dt>

<dd>Moves one host between systemd management and activator management, in either direction, safely ordered so the host is never unserveable. Requires the router to be running with a hibernation bucket for <code>--on</code>. Explained in <a href="hibernation.html#host-switch">part 06</a>.</dd>

+ <dt><code>lchat replication <host> [stream | batched | daily]</code></dt>

+ <dd>Reads (no argument) or sets a hibernating host’s <a href="hibernation.html#replication-classes">replication class</a> — how often awake writes sync to the bucket. A set reloads the router and applies at the host’s next wake; on provisioned fleets billing drives this instead. Explained in <a href="hibernation.html#replication-classes">part 06</a>.</dd>

<dt><code>lchat instances</code></dt>

- <dd>Prints the registry: HOST, KIND, STATUS, VERSION, HIBERNATE, SOCKET, plus a <code>last error</code> line for any failed row. The first stop for “what does this box think it is serving?”.</dd>

+ <dd>Prints the registry: HOST, KIND, STATUS, VERSION, HIBERNATE, REPLICATION, SOCKET, plus a <code>last error</code> line for any failed row. The first stop for “what does this box think it is serving?”.</dd>

<dt><code>lchat admin-link <host></code> / <code>lchat setup-invite <host></code></dt>

<dd>Recover the admin link (from the environment file — the only plaintext copy of the token) / mint a fresh one-time setup invite.</dd>

</dl>

@@ -130,6 +132,7 @@

<tr><td><code>LCHAT_CLOUD_REAP_AFTER_DAYS</code></td><td>0 (reaper off)</td><td>Age before a never-claimed space is reclaimable. <a href="spaces.html#reaper">Part 05</a>.</td></tr>

<tr><td><code>LCHAT_CLOUD_REAP_INTERVAL_HOURS</code> / <code>LCHAT_CLOUD_REAP_MAX_PER_SWEEP</code> / <code>LCHAT_CLOUD_REAP_DRY_RUN</code></td><td>6 / 5 / off</td><td>Reaper sweep cadence / per-sweep cap / log-only mode.</td></tr>

<tr><td><code>LCHAT_CLOUD_STORAGE_CAPS</code>, <code>LCHAT_STRIPE_*</code></td><td>unset</td><td>Per-tier storage caps and the dormant billing rails. <span class="status partial">partially built</span></td></tr>

+ <tr><td><code>LCHAT_CLOUD_REPLICATION_CLASSES</code></td><td>unset (every space streams)</td><td>Per-tier durability rungs, e.g. <code>free=daily,family=daily,community=batched,business=stream</code>. <a href="hibernation.html#replication-classes">Part 06</a>.</td></tr>

</table>

</div>

@@ -216,7 +219,9 @@

<dt id="gloss-litestream">Litestream</dt>

<dd>A tool that continuously streams a SQLite database’s changes into object storage, allowing restore to any recent moment. Used twice in this system, in two forms: a stock copy for <a href="operations.html#backups">fleet backups</a>, and embedded inside the router for <a href="hibernation.html#the-lease">hibernation</a>.</dd>

<dt id="gloss-lease">the lease</dt>

- <dd>The per-space lock object (<code>lock.json</code>) in the hibernation bucket, held while a process serves the space and renewed every few seconds. The system’s single source of truth about who may write a space’s data. <a href="hibernation.html#the-lease">Part 06</a>.</dd>

+ <dd>The per-space lock object (<code>lock.json</code>) in the hibernation bucket, held while a process serves the space and renewed every few minutes. The system’s single source of truth about who may write a space’s data. <a href="hibernation.html#the-lease">Part 06</a>.</dd>

+ <dt id="gloss-replication-class">replication class</dt>

+ <dd>A hibernating host’s durability rung (<code>stream</code> / <code>batched</code> / <code>daily</code>): how often its awake writes sync to the bucket, set per plan by billing or per host by <code>lchat replication</code>. Graceful sleeps and deploys always flush in full regardless. <a href="hibernation.html#replication-classes">Part 06</a>.</dd>

<dt id="gloss-activator">the activator</dt>

<dd>The component inside the router that wakes sleeping spaces on demand and puts idle ones to sleep. Exists only when <code>LCHAT_HIBERNATE_BUCKET</code> is set. <a href="hibernation.html">Part 06</a>.</dd>

<dt id="gloss-hrw">rendezvous hashing</dt>

diff --git a/docs/tier-economics.md b/docs/tier-economics.md

index 8d710dd..a798b9d 100644

--- a/docs/tier-economics.md

+++ b/docs/tier-economics.md

@@ -79,6 +79,51 @@ until its flip. One deliberate permanent exception: **a space with a bot

sidecar never sleeps** (the sidecar's WebSocket holds it awake) — one more

reason AI usage carries its own meter instead of hiding in a flat price.

+## Durability per tier — the replication ladder (decided 2026-07-14)

+The fleet's S3 bill is ~100% Tier-1 *requests*, not data (#836), and the one

+request stream that scales with activity is litestream's change-driven WAL

+sync — worth pennies a year on a quiet space and up to ~€145/year on a

+worst-case space that writes every second around the clock. That cost profile

+is why durability is now a **per-plan replication class** on the activator's

+replica (`routes.replication`; mechanism #866, billing push #869):

+|---|---|---|---|---|

+Three things the ladder deliberately does **not** change:

+- **Every graceful path flushes in full** — idle sleep, deploy, restart,

+ shutdown all checkpoint the space to S3 regardless of class. The class only

+ bounds what a *sudden* box loss can take, and only while awake.

+- **Hibernation's parked copy is always exactly current** — sleeping on a

+ lower rung is not a weaker backup; the rung applies to awake windows only.

+- **Nothing about confidentiality** — the replica is the same ciphertext the

+ server already holds.

+The honest member-facing framing (held in the changelog and any plan copy):

+*backed up every time your space goes quiet, and at least nightly; a

+catastrophic server failure may cost the most recent hours on Free/Family,

+minutes on Community, seconds on Business.* One caveat priced in above: a

+space with a bot sidecar never sleeps, so for those the "at least nightly"

+floor is the real day-to-day cadence on the daily rung.

+**One Business plan, not a "Business Plus" (decided 2026-07-14).** The full

+resilience stack — streaming replication, the retention ladder behind

+point-in-time restore, room to tighten the crash-failover window — costs

+€2–20/year on realistic Business spaces against €999–1,188/year of revenue.

+A resilience delta that small can't carry a second paid tier: splitting it

+would add a Stripe price pair, picker copy, and a matrix cell to explain,

+to sell something we can afford to include. So Business simply *includes*

+best-available durability, and the ladder stays three legible rungs priced

+by what they actually cost. If a segment ever demands contractual RPO/RTO,

+multi-region replicas, or compliance-grade restore drills, that's a

+custom/Enterprise conversation — negotiated, not a checkbox tier — and the

+books will show its costs separately when it exists.

## The model

For a tier with gross annual revenue `G` across `n` charges/year:

@@ -87,7 +132,9 @@ For a tier with gross annual revenue `G` across `n` charges/year:

net = G − (0.25 × n) − (0.015 × G) # Stripe fees out

cost = GB_stored × storage_rate × 12 # media, the only real dial

+ slot_rate × 12 × awake_fraction # compute while awake

- + ~€0.10 # wakes, requests, push (noise)

+ + repl_requests # by replication class (see Durability):

+ # daily/batched ≈ €0; stream ≈ €2–20

+ + ~€0.10 # wakes, other requests, push (noise)

profit = net − cost

break_even_GB = net ÷ (storage_rate × 12) # where a space stops covering itself

```