Snapshots — Design

Status: designed, not yet reviewed, not yet a plan. Extends: docs/superpowers/specs/2026-08-30-docs-design.md §4.1 (the log is bounded), §5.1/§5.2 (what the core does, and why the sandbox is load-bearing). Follows: 2026-09-01-live-sync-design.md, whose cursor rules this must not disturb.

1. Why now, and it is not latency

The obvious reason to want this is speed: DEBOUNCE_MS is 2000ms because every accepted update is one doc_updates row, and a shorter debounce multiplies rows. That is real, and it is the second reason.

The first is a cliff already in production. internal/core/core.go's maxRetainedUpdates is 4096, and once a document's retained log reaches it, Handle.Apply refuses every further update — ErrUpdateLogFull, surfaced to the browser as a 400 that the save queue treats as permanent. The document stops accepting edits. That constant's own comment is explicit that this is a deliberate, visible failure standing in for snapshot support, chosen because silently dropping old entries would make a later rebuild replay an incomplete history and produce quietly wrong content.

At today's 2s debounce that is roughly 2h16m of continuous typing on one document. A person writing all day reaches it. Nothing warns them first; the document simply starts refusing.

So this work is not a performance optimisation with a latency bonus. It is the thing that stops a document going read-only, and it happens to be what makes near-realtime affordable afterwards.

2. Scope

Snapshots only. Paul's ruling, 2026-09-01: nothing is deleted.

§4.1 names three mechanisms — coalescing before persistence, a snapshot cadence, and a compaction horizon that drops superseded updates. This slice builds the middle one. The horizon is deferred because deletion is irreversible and forecloses any future history feature over the window it eats; coalescing is deferred because it belongs with the transport work that makes a short debounce worth having.

The consequence worth stating plainly: this slice makes storage grow faster, not slower. The update log still grows, and snapshots are added on top of it — a full copy of the document every 256 updates, all retained under "nothing is deleted", all replicated by Litestream. For a large document the snapshots outgrow the log they accelerate. This slice bounds replay cost and removes the cap; bounding storage is what the horizon is for.

One correction to that, though, which review surfaced and which does not need the horizon ruling: a superseded snapshot is derived data, not history. It is reproducible from the log at any time, so deleting it deletes nothing a reader could ever have asked for — unlike an update row, whose loss is irreversible. This design therefore rules that only the newest snapshot per document is retained, and older ones are deleted as they are superseded. That keeps the added storage at one document-sized blob rather than one per 256 updates, and it is not the compaction horizon in disguise: no update row is touched, and the "nothing is deleted" ruling is about history, which this is not.

3. What a snapshot is

One Yjs update encoding the document's entire state at a point in the log — exactly what yrs produces internally today and what core/src/abi.rs does not yet expose. Applying it to an empty document yields the same document the whole log yields.

The call is ReadTxn::encode_state_as_update_v1(&StateVector::default()), and which one is not a detail. Against the empty state vector it encodes all blocks, the entire delete set, and — through merge_pending_v1 — the pending queue, meaning updates whose dependencies never arrived and which apply_update accepted and parked. That is Yjs's own persistence mechanism, and it is merge-equivalent to the log rather than merely render-equivalent: every clock range and every tombstone survives, so a later concurrent merge integrates against a snapshot exactly as it would against the full history.

Two neighbours in the same API are traps, and an implementer reaching for the nearest-looking name will find them first:

• encode_diff_v1(&StateVector::default()) looks identical and silently drops the pending queue (its own doc comment says so). A log holding an update whose dependency arrives later heals under full replay and does not heal under a diff-built snapshot. That is precisely the failure this design most fears — renders identically today, breaks a merge tomorrow — and §D5's comparison cannot see it, because pending content is invisible in both documents.
• yrs's own Snapshot type and encode_state_from_snapshot are a name collision with this spec's vocabulary and a different feature entirely (historical views, requiring skip_gc). Not this.

The fixed guid and client id play no part in the encoded state, so nothing here touches the no-entropy rule; the "prosemirror" root travels inside the update; and GC settings match on both sides, so tombstone handling is symmetric with log replay.

That single property is what makes everything below cheap: a snapshot is not a new kind of thing, it is an update. Every path that already knows how to apply concatenated updates — Store.replay, and the browser's own applyServerLog — handles one with no changes at all.

4. Decisions

D1 — The ABI gains a state export, v3 → v4

docs_core_snapshot_len / docs_core_snapshot_read, mirroring the existing html_len/html_read pair. internal/core's version check is exact-match, so the wasm artifact and the host move together.

This is the export core.go's maxRetainedUpdates comment already specifies: "an ABI export that encodes a document's current state as one update… so rebuild could restore a compacted snapshot plus only the updates since it".

D2 — doc_snapshots, and it is never authoritative

New table (migration 0004), extending §4's data model with the columns that failure handling actually needs: doc_id, seq, state, abi_version, created_at.

seq names the last update the snapshot includes. abi_version is what makes the failure table's "from an older ABI → discarded" implementable at all; without it the row records nothing about what produced it, and the rule is unenforceable prose.

Two schema decisions the implementer would otherwise have to invent, settled here. (doc_id, seq) is not unique: under §D3's before-apply placement two concurrent boundary POSTs can legitimately produce snapshots at the same seq, and they are equivalent by construction, so a collision is not an error to raise. "Newest usable" therefore means highest seq, then highest id, so a duplicate resolves deterministically to the last one written.

The migration adds a table and touches no existing row, which is the whole of its effect on documents that already exist — migration 0003's own comment sets the precedent that this is spec-level material rather than an implementation detail.

state is plaintext for now, exactly as doc_updates is; sealing (§2) is its own work. One honest note it does change, though: a doc_updates row is a fragment of an edit, while a snapshot row is the entire document in a single plaintext blob. That is a materially fatter target in the same threat model, and worth a sentence in the sealing work rather than a redesign here.

A snapshot is an accelerator, never the source of truth. The log remains complete and authoritative. Anything unreadable, corrupt, or produced by an older ABI is discarded and the full log replayed instead — slower, always correct. This is the property that makes the whole slice safe to ship: the worst failure is a performance regression, never wrong content.

D3 — Taken from the replayed state, before the new update is applied

Store.ApplyUpdate replays the log into a Handle, applies the incoming update, then persists it. The snapshot is taken from the replayed state, before Apply, at seq = the highest seq that replay actually read.

The obvious placement — after a successful write, snapshotting the handle that just accepted the update — is wrong twice, and both ways were found in review rather than in production:

It can record a state that is missing a committed row. The replay runs outside ApplyUpdate's transaction; only MAX(seq)+INSERT is serialised by the single-connection write pool. Two concurrent POSTs to one document — the flush timer racing the beforeunload beacon, a pair the live-sync design already names as real — both replay rows 1..10; A commits seq 11, B commits seq 12. B's handle holds 1..10 plus its own update and not row 11. A snapshot written there claims seq 12 while missing row 11's content, and §D5 cannot catch it, because it compares the candidate against the same stale handle that produced it. Every later render and every §D7 response would then be missing that content, with Docs-Seq: 12 genuinely lying about the bytes in hand — the live-sync design's D2 failure re-entering through a seq column instead of a MAX(seq) query. It would heal at the next cadence boundary 256 updates later, and never if the document went idle first.

And it never rescues a document that is already at the cap — the case §1 opens with, and the one that may exist in production now. A capped document replays to exactly maxRetainedUpdates, so Handle.Apply returns ErrUpdateLogFull before any update succeeds. Snapshot-after-success is never reached, and the document 400s forever.

Taking it from the replayed state fixes both by construction. The replayed prefix is complete — seq is gapless by MAX+1, and replay read every row that existed when it ran — so the seq column is unconditionally truthful. And the read happens against a fully usable handle whether or not the incoming update will be accepted, so the first POST to a capped document writes a snapshot at seq 4096, still fails once with a 400, and the client's own retry replays snapshot plus an empty tail and succeeds. One visible failure, then healed.

Still on the write path, and still never on a timer. §7.3's rule is that a hibernating instance runs no timers, so a background snapshotter stops working exactly when a document goes idle enough to be worth snapshotting. A request means the instance is awake by definition.

Two costs to state. The boundary POST pays a state export, §D5's verification and one INSERT, all bounded by the existing guest-call timeout — which now indirectly bounds snapshot size: a document whose export or re-apply exceeds that timeout traps every attempt, never gets a snapshot, and pays a rebuild penalty per attempt. WithGuestCallTimeout is the existing knob if that ever bites. And once the distance reaches the cadence, every subsequent POST retries until one succeeds — not "the next Nth write".

D4 — Cadence of 256 updates

Replay is then at most one snapshot plus 256 updates, and the retained-update count in a Handle stays around 16× clear of maxRetainedUpdates.

Conditionally, and the condition belongs in writing. That bound holds only while snapshot writes keep succeeding. Roughly fifteen consecutive cadence failures — each logged at Error, so noisy rather than silent — walk the tail back to the cliff. Before D8 that is where this section ended: the next replay past that point met ErrUpdateLogFull inside Store.replay itself, for a read exactly as for a write, and the failure table's "correct, possibly slower" bar was not met. D8 closes that: a tail replay that meets the cap re-seeds a fresh handle from its own state, in memory, and carries on, so reaching the cliff this way no longer ends a read, or the write riding the same replay. What is left of the condition is cost, not failure — a re-seed on every replay until a persisted snapshot finally succeeds and this section's 16× bound is restored. The cliff is removed in every ordinary case and merely deferred to a per-replay cost in a persistently failing one, which is a weaker claim than "retired" and the true one.

The cadence distance is read from the snapshot row §D6's replay already loads, not from a separate query — the same discipline as the live-sync design's D2, for the same reason.

A byte-size trigger (§4.1's "or M bytes, whichever first") is deliberately not in this slice: with no coalescing yet, one update is one debounced burst of typing, so update count and bytes track each other closely enough that a second trigger would add a knob nobody can yet set from evidence.

D5 — A snapshot is verified before it is trusted, inside the core

Before persisting, the candidate is applied to a fresh document inside the Rust core — no second wasm instance, no second host-side handle — and three things are compared against the document it was taken from:

• state vector equality, as a Rust struct comparison;
• delete set equality, likewise;
• fragment_json equality, not rendered text.

Rust-side struct comparison avoids depending on encoding determinism, which does not hold: update encoding is not canonical, so no cheap byte-level proof exists and the spec should stop wishing for one. Comparing structure rather than text also catches formatting-only divergence, which a text comparison misses at identical cost.

This is not a proof, and it is the right place to stop. What survives it is "yrs corrupted content while preserving every clock range, every deletion, and the full structural render", which is a narrow enough residual to accept.

What it can never catch is a snapshot taken from a stale state, because that compares a candidate against the very document that produced it and they agree perfectly. That failure is closed by §D3's placement, not here — a comparator cannot detect a truthful copy of the wrong thing.

D6 — Store.replay reads snapshot-then-tail

Load the newest usable snapshot, apply it, then apply only seq > snapshot.seq. Falls back to a full replay when there is no snapshot or it fails D2's usability check.

D7 — GET /d/{sqid}/updates serves snapshot-then-tail

When the requesting cursor is behind the newest snapshot, the response is that snapshot followed by the updates after it, instead of every row since the cursor.

This needs no client change. applyServerLog already loops Y.readUpdate over a concatenated body, a snapshot is an update, and Yjs integration is idempotent — a client that asks ?since=5 and receives a snapshot covering through 256 has lost nothing, because the snapshot contains rows 6-256.

Two things it fixes beyond tidiness: initial load stops being O(all history), and maxLogBytes (8MB) stops being a second cliff a long-lived document eventually hits.

Docs-Seq keeps its meaning and its rule. It is still the highest seq the returned bytes actually reach, still derived from what is in hand and never from a separate MAX(seq) query — the snapshot's own seq column is a property of the row being sent, not a second question asked of the database. The live-sync design's D2 (empty slice echoes since) and D3 (cursor+1) are untouched.

The endpoint's own doc comment must stop describing the response as the raw log, because it will no longer be one.

Select the snapshot row first and the tail as seq > snapshot.seq. The reverse order is also loss-free, since overlap is idempotent, but one order should be stated so two readers do not each pick their own.

The one place §D2's safety story does not reach, stated rather than papered over. Store.replay can discard a bad snapshot because it applies it and sees the failure. updatesGet cannot: it serves the blob without a core call, so a row corrupted on disk after §D5 verified it goes straight to the browser, where Y.readUpdate throws mid-loop and the resync controller refetches the same bytes forever. Server-side rendering keeps working; the editor becomes permanently unloadable for that document. That contradicts both "the worst failure is a performance regression" and "this needs no client change", so this design pays for it: ?raw=1 forces the pre-snapshot behaviour, returning only doc_updates rows, and applyServerLog refetches with it once when a decode throws. Small, and it makes the log's authority reachable from the one reader for whom it matters most.

D8 — A tail replay past the cap re-seeds in memory rather than failing

Found after this design was reviewed, in a task review of its implementation: what happens when Store.replay has no usable snapshot to start from — D2's ordinary "discarded, full replay" story for a corrupt row, a stale one, or (the one that will actually happen) every row at once on the deploy that ships the next ABI bump, since newestSnapshot's exact-ABI match invalidates them all together — and the log it falls back to is longer than core.MaxRetainedUpdates? Full replay is precisely what §1 opens by naming as already unsafe past that count: it is the cliff Handle.Apply was built to stop at, met this time from inside Store.replay's own tail loop instead of from a client's POST, and reached with no snapshot in hand to shorten the log that triggers it.

§3's property closes it the same way it closes everything else here: a snapshot is nothing but a handle's own state, exported, and applying it to an empty document reproduces the handle. Store.replay's tail loop uses that on itself. When applying a row meets core.ErrUpdateLogFull, it takes that same export of the handle it already has — mid-loop, via Handle.Snapshot — opens a fresh handle from the document's own stored GUID and client id (never a new identity; see Document), applies the export, and continues from the row that was refused, which is re-applied into the fresh handle and not dropped. reseedForReplay in internal/docs/store.go is the mechanism; its call site is the one place in the tail loop that catches core.ErrUpdateLogFull rather than surfacing it. A log of any length gets through in as many re-seeds as it needs, and what a replay retains at once stays bounded by the cap regardless — the same bound §D4 relies on, just re-established mid-replay instead of assumed for it.

It writes nothing, deliberately, and this is the one place that needs saying twice: Store.replay is the read path (Text, HTML and JSON all reach it), and a read that persisted a row would be a materially different change — the one place today's design allows a snapshot to originate is a request that has already proved the instance awake (D3), which a plain read has not. The persisted snapshot this document is still owed arrives the ordinary way, on its next ApplyUpdate: crossing the cap here means the gap already exceeds snapshotCadence many times over, so that write's own cadence check (D3, D4) passes trivially and it snapshots at the current ABI — which is exactly what an ABI bump needs repaired, and the reason this stays self-healing rather than becoming a second, competing writer to reason about.

It logs at Warn, once per re-seed — not Error, and not silent either, and both halves of that were argued from evidence rather than assumed. Not Error, because nothing is wrong with the document: it is complete and correct, only slower to read, exactly the distinction D4 draws between a cliff and a cost. Not silent, which is what this was at first, on the reasoning that the document's next write repairs it — reasoning that assumes a write arrives. Reads write nothing, so a long document that is finished and only ever read again, which is most of what an ABI bump invalidates at once, would re-seed on every single read, forever, with degraded latency as the only trace. One line per read is the price of that not being invisible to an operator.

5. What does not change

The wire format (a snapshot is an ordinary Yjs update); the browser bundle; the cursor protocol; the stream and its hub; ownership scoping; doc_updates itself, which keeps every row it has.

6. Failure modes

| Failure | Behaviour | |---|---| | No snapshot exists | Full replay, as today | | Snapshot row unreadable or from an older ABI | Discarded, full replay, logged | | D5's verification fails | Snapshot not written, log untouched, logged at Error | | Snapshot write fails after the update committed | Update stands, no snapshot, next Nth write tries again | | Core call traps producing the snapshot | Same as above; the update is already durable | | A full replay (no usable snapshot) crosses core.MaxRetainedUpdates mid-tail | Re-seeds a fresh handle from its own state, in memory, and continues; writes nothing, logged at Warn per re-seed (D8) |

Every row of that table ends in "correct, possibly slower", which is the bar this design sets for itself.

7. Non-goals

The compaction horizon and any deletion of rows; coalescing before persistence; sealing state; named versions (§14), which will need to pin points a future horizon must not collapse; the WebSocket transport; presence.

8. Testing

• Equivalence is the central test. For a document with more updates than the cadence, the text and HTML produced through snapshot-then-tail must equal what a full replay produces, byte for byte. Property-shaped rather than one fixture: several documents, several cadences, including a snapshot at the exact boundary.
• The cap stops being reachable. Apply more than maxRetainedUpdates updates to one document and assert it still accepts writes — the regression test for the cliff this exists to remove. It should fail against main today.
• A document that is ALREADY at the cap recovers, which the test above does not cover: driven through the new path a document never reaches the cap, so the production-shaped case needs its own fixture — a log of maxRetainedUpdates rows and no snapshot, asserting one 400 followed by a successful retry. This is §D3's second defect, and without this test the fix for it is unverified.
• The concurrent boundary race (§D3's first defect): two POSTs racing at the cadence boundary must never produce a snapshot whose seq outruns its content. Assert against the rendered document, not against the snapshot row, since the row is exactly what a stale snapshot would make look right.
• ?raw=1 returns only doc_updates rows, and a client that hits a decode failure recovers through it.
• Fallbacks are exercised, not assumed: a corrupted snapshot row, a snapshot from a wrong ABI, and a missing snapshot each still produce correct content.
• D5's verifier rejects a bad snapshot — constructed by hand, since the core will not produce one on request.
• The GET's cursor contract survives: ?since= behind, at, and ahead of the snapshot; Docs-Seq still derived from bytes in hand; the live-sync suite green unchanged.
• Wire fixtures unchanged. They are the contract, not an output.

9. Risks

1. The ABI bump is a coordination point. internal/core's check is exact-match, so a stale wasm artifact fails at load. Anyone rebasing across this rebuilds the artifact.
2. A wrong snapshot is the worst bug this repo could ship — silently wrong content for every future reader, with the log that would prove it still sitting there unread. D2's never-authoritative rule and D5's verification exist for that one risk, and the review should be hardest on exactly that.
3. Storage still grows, and this slice adds to it. §2 states the shape and the supersession rule that keeps it to one blob per document. The log itself is untouched and unbounded until the horizon lands.
4. maxLogBytes is bounded against history, not against size. Snapshot-then- tail means a long-lived document no longer sends its whole past, but a document whose current state exceeds 8MB still fails the GET — with the snapshot in the body. Different cliff, further away, not removed.

10. Review record

Reviewed by Fable on 2026-09-01, at the spec stage, per AGENTS.md. Verdict: approve with changes; all folded in above.

Two were correctness defects with the same root — where the snapshot is taken from. Snapshotting the handle that just accepted an update can record a state missing a concurrently-committed row (§D3), and can never rescue a document already at the cap, which is the case §1 opens with. Taking it from the replayed state before Apply closes both by construction.

The rest: the exact yrs call named along with the two neighbours that look right and are not (§3); §D5 moved inside the core and strengthened from a text comparison to state-vector, delete-set and structural equality, with its honest limit stated; the abi_version column without which the failure table's own rule is unenforceable (§D2); the ?raw=1 escape hatch for a corrupt snapshot reaching a browser that has no way to refuse it (§D7); the cap's retirement restated as conditional on snapshot writes succeeding (§D4); and the supersession ruling that keeps this from making storage worse than the problem it solves (§2).

The review also confirmed what §3 rests on: against an empty state vector, encode_state_as_update_v1 is merge-equivalent to the log and not merely render-equivalent, so a snapshot is safe to build future concurrent merges on.

§D8 was added after this review, not folded into it: a finding from a later task review of the implementation, that a full replay falling back to no usable snapshot on a log already past core.MaxRetainedUpdates met ErrUpdateLogFull inside Store.replay itself with no way out, which is the ABI-bump case this same design's D2 and §9's first risk both name as the thing that will actually happen. The fix — re-seeding a fresh handle from the replay's own state, in memory, mid-tail — was implemented and tested before this section existed; §D8 is this spec catching up to what the code already does, not a proposal.