amadan
Spec: snapshots, so a document stops going read-only
maxRetainedUpdates is a cliff rather than a budget: at 4096 updates Apply refuses everything further and the document stops accepting edits, which at today's debounce is about two hours of continuous typing. Snapshots remove that, and separately make a shorter debounce affordable — but the cliff is the reason, not the latency. A snapshot is an ordinary Yjs update encoding the whole state, so Store.replay and the browser's applyServerLog already know how to apply one. It is never authoritative: the log stays complete, and anything unreadable is discarded in favour of a full replay, so every failure mode ends in correct-but-slower. Reviewed by Fable at the spec stage, which found two correctness defects sharing one root. Snapshotting the handle that just accepted an update can record a state missing a row a concurrent POST committed — the replay runs outside the transaction — and the verification cannot catch it, because it compares the candidate against the same stale handle. The same placement also never rescues a document already at the cap, since Apply refuses before any update succeeds, so the case the spec opens with would have kept failing forever. Taking the snapshot from the replayed state before Apply closes both by construction. Also folded in: the exact yrs call, since the two neighbours that look right silently drop the pending queue or are a different feature entirely; verification moved inside the core and strengthened past a text comparison; the abi_version column without which the spec's own fallback rule is unenforceable; a raw escape hatch for a corrupt snapshot reaching a browser with no way to refuse it; and a supersession ruling, because retaining every snapshot would have made storage worse than the problem being solved.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 file changed,
+382
−0
docs/superpowers/specs/2026-09-01-snapshots-design.md+382 −0
CI passed — run details
Intent
Attributed to this commit: 73.7 g CO₂e (9468994 tokens). This is an attribution, not a measurement — usage counted from this commit's own marker forward to the next one on worktree-snapshots, never backward from it. A turn that produced more than one commit books all of its tokens to the last one, and a concurrent session on this branch can add to the same figure. Some of it was priced using the shared default rate, because no model-specific factor row exists yet for at least one model involved. The Carbon view has the repo total, which is never derived from this attribution.
diff --git a/docs/superpowers/specs/2026-09-01-snapshots-design.md b/docs/superpowers/specs/2026-09-01-snapshots-design.md| new file mode 100644 |
| index 0000000..3ad05ef |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-09-01-snapshots-design.md |
| @@ -0,0 +1,382 @@ |
| +# Snapshots — Design |
| + |
| +**Status:** designed, not yet reviewed, not yet a plan. |
| +**Extends:** `docs/superpowers/specs/2026-08-30-docs-design.md` §4.1 (the log is |
| +bounded), §5.1/§5.2 (what the core does, and why the sandbox is load-bearing). |
| +**Follows:** `2026-09-01-live-sync-design.md`, whose cursor rules this must not |
| +disturb. |
| + |
| +## 1. Why now, and it is not latency |
| + |
| +The obvious reason to want this is speed: `DEBOUNCE_MS` is 2000ms because every |
| +accepted update is one `doc_updates` row, and a shorter debounce multiplies rows. |
| +That is real, and it is the second reason. |
| + |
| +The first is a cliff already in production. `internal/core/core.go`'s |
| +`maxRetainedUpdates` is 4096, and once a document's retained log reaches it, |
| +`Handle.Apply` refuses every further update — `ErrUpdateLogFull`, surfaced to the |
| +browser as a 400 that the save queue treats as permanent. **The document stops |
| +accepting edits.** That constant's own comment is explicit that this is a |
| +deliberate, visible failure standing in for snapshot support, chosen because |
| +silently dropping old entries would make a later rebuild replay an incomplete |
| +history and produce quietly wrong content. |
| + |
| +At today's 2s debounce that is roughly 2h16m of continuous typing on one |
| +document. A person writing all day reaches it. Nothing warns them first; the |
| +document simply starts refusing. |
| + |
| +So this work is not a performance optimisation with a latency bonus. It is the |
| +thing that stops a document going read-only, and it happens to be what makes |
| +near-realtime affordable afterwards. |
| + |
| +## 2. Scope |
| + |
| +**Snapshots only.** Paul's ruling, 2026-09-01: nothing is deleted. |
| + |
| +§4.1 names three mechanisms — coalescing before persistence, a snapshot cadence, |
| +and a compaction horizon that drops superseded updates. This slice builds the |
| +middle one. The horizon is deferred because deletion is irreversible and |
| +forecloses any future history feature over the window it eats; coalescing is |
| +deferred because it belongs with the transport work that makes a short debounce |
| +worth having. |
| + |
| +The consequence worth stating plainly: **this slice makes storage grow faster, |
| +not slower.** The update log still grows, and snapshots are added on top of it — |
| +a full copy of the document every 256 updates, all retained under "nothing is |
| +deleted", all replicated by Litestream. For a large document the snapshots |
| +outgrow the log they accelerate. This slice bounds *replay cost* and removes the |
| +cap; bounding storage is what the horizon is for. |
| + |
| +One correction to that, though, which review surfaced and which does not need the |
| +horizon ruling: **a superseded snapshot is derived data, not history.** It is |
| +reproducible from the log at any time, so deleting it deletes nothing a reader |
| +could ever have asked for — unlike an update row, whose loss is irreversible. |
| +This design therefore rules that **only the newest snapshot per document is |
| +retained**, and older ones are deleted as they are superseded. That keeps the |
| +added storage at one document-sized blob rather than one per 256 updates, and it |
| +is not the compaction horizon in disguise: no update row is touched, and the |
| +"nothing is deleted" ruling is about history, which this is not. |
| + |
| +## 3. What a snapshot is |
| + |
| +One Yjs update encoding the document's entire state at a point in the log — |
| +exactly what `yrs` produces internally today and what `core/src/abi.rs` does not |
| +yet expose. Applying it to an empty document yields the same document the whole |
| +log yields. |
| + |
| +**The call is `ReadTxn::encode_state_as_update_v1(&StateVector::default())`, and |
| +which one is not a detail.** Against the empty state vector it encodes all |
| +blocks, the entire delete set, and — through `merge_pending_v1` — the pending |
| +queue, meaning updates whose dependencies never arrived and which `apply_update` |
| +accepted and parked. That is Yjs's own persistence mechanism, and it is |
| +merge-equivalent to the log rather than merely render-equivalent: every clock |
| +range and every tombstone survives, so a later concurrent merge integrates |
| +against a snapshot exactly as it would against the full history. |
| + |
| +Two neighbours in the same API are traps, and an implementer reaching for the |
| +nearest-looking name will find them first: |
| + |
| +- **`encode_diff_v1(&StateVector::default())`** looks identical and silently |
| + drops the pending queue (its own doc comment says so). A log holding an update |
| + whose dependency arrives later heals under full replay and does not heal under |
| + a diff-built snapshot. That is precisely the failure this design most fears — |
| + renders identically today, breaks a merge tomorrow — and §D5's comparison |
| + cannot see it, because pending content is invisible in both documents. |
| +- **yrs's own `Snapshot` type** and `encode_state_from_snapshot` are a name |
| + collision with this spec's vocabulary and a different feature entirely |
| + (historical views, requiring `skip_gc`). Not this. |
| + |
| +The fixed guid and client id play no part in the encoded state, so nothing here |
| +touches the no-entropy rule; the `"prosemirror"` root travels inside the update; |
| +and GC settings match on both sides, so tombstone handling is symmetric with log |
| +replay. |
| + |
| +That single property is what makes everything below cheap: a snapshot is not a |
| +new kind of thing, it is *an update*. Every path that already knows how to apply |
| +concatenated updates — `Store.replay`, and the browser's own `applyServerLog` — |
| +handles one with no changes at all. |
| + |
| +## 4. Decisions |
| + |
| +### D1 — The ABI gains a state export, v3 → v4 |
| + |
| +`docs_core_snapshot_len` / `docs_core_snapshot_read`, mirroring the existing |
| +`html_len`/`html_read` pair. `internal/core`'s version check is exact-match, so |
| +the wasm artifact and the host move together. |
| + |
| +This is the export `core.go`'s `maxRetainedUpdates` comment already specifies: |
| +"an ABI export that encodes a document's current state as one update… so rebuild |
| +could restore a compacted snapshot plus only the updates since it". |
| + |
| +### D2 — `doc_snapshots`, and it is never authoritative |
| + |
| +New table (migration `0004`), extending §4's data model with the columns that |
| +failure handling actually needs: `doc_id`, `seq`, `state`, **`abi_version`**, |
| +`created_at`. |
| + |
| +`seq` names the last update the snapshot includes. `abi_version` is what makes |
| +the failure table's "from an older ABI → discarded" implementable at all; without |
| +it the row records nothing about what produced it, and the rule is unenforceable |
| +prose. |
| + |
| +Two schema decisions the implementer would otherwise have to invent, settled |
| +here. `(doc_id, seq)` is **not** unique: under §D3's before-apply placement two |
| +concurrent boundary POSTs can legitimately produce snapshots at the same seq, and |
| +they are equivalent by construction, so a collision is not an error to raise. |
| +"Newest usable" therefore means **highest `seq`, then highest `id`**, so a |
| +duplicate resolves deterministically to the last one written. |
| + |
| +The migration adds a table and touches no existing row, which is the whole of its |
| +effect on documents that already exist — migration `0003`'s own comment sets the |
| +precedent that this is spec-level material rather than an implementation detail. |
| + |
| +`state` is plaintext for now, exactly as `doc_updates` is; sealing (§2) is its own |
| +work. One honest note it does change, though: a `doc_updates` row is a fragment of |
| +an edit, while a snapshot row is **the entire document in a single plaintext |
| +blob**. That is a materially fatter target in the same threat model, and worth a |
| +sentence in the sealing work rather than a redesign here. |
| + |
| +**A snapshot is an accelerator, never the source of truth.** The log remains |
| +complete and authoritative. Anything unreadable, corrupt, or produced by an older |
| +ABI is discarded and the full log replayed instead — slower, always correct. This |
| +is the property that makes the whole slice safe to ship: the worst failure is a |
| +performance regression, never wrong content. |
| + |
| +### D3 — Taken from the replayed state, before the new update is applied |
| + |
| +`Store.ApplyUpdate` replays the log into a `Handle`, applies the incoming update, |
| +then persists it. The snapshot is taken from **the replayed state, before |
| +`Apply`**, at `seq` = the highest seq that replay actually read. |
| + |
| +The obvious placement — after a successful write, snapshotting the handle that |
| +just accepted the update — is wrong twice, and both ways were found in review |
| +rather than in production: |
| + |
| +**It can record a state that is missing a committed row.** The replay runs |
| +*outside* `ApplyUpdate`'s transaction; only MAX(seq)+INSERT is serialised by the |
| +single-connection write pool. Two concurrent POSTs to one document — the flush |
| +timer racing the `beforeunload` beacon, a pair the live-sync design already names |
| +as real — both replay rows 1..10; A commits seq 11, B commits seq 12. B's handle |
| +holds 1..10 plus its own update and *not row 11*. A snapshot written there claims |
| +seq 12 while missing row 11's content, and §D5 cannot catch it, because it |
| +compares the candidate against the same stale handle that produced it. Every |
| +later render and every §D7 response would then be missing that content, with |
| +`Docs-Seq: 12` genuinely lying about the bytes in hand — the live-sync design's |
| +D2 failure re-entering through a seq column instead of a `MAX(seq)` query. It |
| +would heal at the next cadence boundary 256 updates later, and never if the |
| +document went idle first. |
| + |
| +**And it never rescues a document that is already at the cap** — the case §1 |
| +opens with, and the one that may exist in production now. A capped document |
| +replays to exactly `maxRetainedUpdates`, so `Handle.Apply` returns |
| +`ErrUpdateLogFull` *before* any update succeeds. Snapshot-after-success is never |
| +reached, and the document 400s forever. |
| + |
| +Taking it from the replayed state fixes both by construction. The replayed prefix |
| +is complete — seq is gapless by MAX+1, and replay read every row that existed |
| +when it ran — so the seq column is unconditionally truthful. And the read happens |
| +against a fully usable handle whether or not the incoming update will be |
| +accepted, so the first POST to a capped document writes a snapshot at seq 4096, |
| +still fails once with a 400, and the client's own retry replays snapshot plus an |
| +empty tail and succeeds. One visible failure, then healed. |
| + |
| +**Still on the write path, and still never on a timer.** §7.3's rule is that a |
| +hibernating instance runs no timers, so a background snapshotter stops working |
| +exactly when a document goes idle enough to be worth snapshotting. A request |
| +means the instance is awake by definition. |
| + |
| +Two costs to state. The boundary POST pays a state export, §D5's verification and |
| +one INSERT, all bounded by the existing guest-call timeout — which now indirectly |
| +bounds snapshot size: a document whose export or re-apply exceeds that timeout |
| +traps every attempt, never gets a snapshot, and pays a rebuild penalty per |
| +attempt. `WithGuestCallTimeout` is the existing knob if that ever bites. And once |
| +the distance reaches the cadence, *every* subsequent POST retries until one |
| +succeeds — not "the next Nth write". |
| + |
| +### D4 — Cadence of 256 updates |
| + |
| +Replay is then at most one snapshot plus 256 updates, and the retained-update |
| +count in a `Handle` stays around 16× clear of `maxRetainedUpdates`. |
| + |
| +**Conditionally, and the condition belongs in writing.** That bound holds only |
| +while snapshot writes keep succeeding. Roughly fifteen consecutive cadence |
| +failures — each logged at Error, so noisy rather than silent — walk the tail back |
| +to the cliff, and at that point the failure table's "correct, possibly slower" |
| +bar is no longer met: it ends in `ErrUpdateLogFull` again. The cliff is removed in |
| +every ordinary case and merely deferred in a persistently failing one, which is a |
| +weaker claim than "retired" and the true one. |
| + |
| +The cadence distance is read from the snapshot row §D6's replay already loads, |
| +not from a separate query — the same discipline as the live-sync design's D2, for |
| +the same reason. |
| + |
| +A byte-size trigger (§4.1's "or M bytes, whichever first") is deliberately not in |
| +this slice: with no coalescing yet, one update is one debounced burst of typing, |
| +so update count and bytes track each other closely enough that a second trigger |
| +would add a knob nobody can yet set from evidence. |
| + |
| +### D5 — A snapshot is verified before it is trusted, inside the core |
| + |
| +Before persisting, the candidate is applied to a fresh document *inside the Rust |
| +core* — no second wasm instance, no second host-side handle — and three things |
| +are compared against the document it was taken from: |
| + |
| +- **state vector equality**, as a Rust struct comparison; |
| +- **delete set equality**, likewise; |
| +- **`fragment_json` equality**, not rendered text. |
| + |
| +Rust-side struct comparison avoids depending on encoding determinism, which does |
| +not hold: update encoding is not canonical, so no cheap byte-level proof exists |
| +and the spec should stop wishing for one. Comparing structure rather than text |
| +also catches formatting-only divergence, which a text comparison misses at |
| +identical cost. |
| + |
| +**This is not a proof, and it is the right place to stop.** What survives it is |
| +"yrs corrupted content while preserving every clock range, every deletion, and |
| +the full structural render", which is a narrow enough residual to accept. |
| + |
| +What it can never catch is a snapshot taken from a stale state, because that |
| +compares a candidate against the very document that produced it and they agree |
| +perfectly. That failure is closed by §D3's placement, not here — a comparator |
| +cannot detect a truthful copy of the wrong thing. |
| + |
| +### D6 — `Store.replay` reads snapshot-then-tail |
| + |
| +Load the newest usable snapshot, apply it, then apply only `seq > snapshot.seq`. |
| +Falls back to a full replay when there is no snapshot or it fails D2's usability |
| +check. |
| + |
| +### D7 — `GET /d/{sqid}/updates` serves snapshot-then-tail |
| + |
| +When the requesting cursor is behind the newest snapshot, the response is that |
| +snapshot followed by the updates after it, instead of every row since the cursor. |
| + |
| +This needs **no client change**. `applyServerLog` already loops `Y.readUpdate` |
| +over a concatenated body, a snapshot is an update, and Yjs integration is |
| +idempotent — a client that asks `?since=5` and receives a snapshot covering |
| +through 256 has lost nothing, because the snapshot contains rows 6-256. |
| + |
| +Two things it fixes beyond tidiness: initial load stops being O(all history), and |
| +`maxLogBytes` (8MB) stops being a second cliff a long-lived document eventually |
| +hits. |
| + |
| +**`Docs-Seq` keeps its meaning and its rule.** It is still the highest seq the |
| +returned bytes actually reach, still derived from what is in hand and never from |
| +a separate `MAX(seq)` query — the snapshot's own `seq` column is a property of the |
| +row being sent, not a second question asked of the database. The live-sync |
| +design's D2 (empty slice echoes `since`) and D3 (cursor+1) are untouched. |
| + |
| +The endpoint's own doc comment must stop describing the response as the raw log, |
| +because it will no longer be one. |
| + |
| +Select the snapshot row first and the tail as `seq > snapshot.seq`. The reverse |
| +order is also loss-free, since overlap is idempotent, but one order should be |
| +stated so two readers do not each pick their own. |
| + |
| +**The one place §D2's safety story does not reach, stated rather than papered |
| +over.** `Store.replay` can discard a bad snapshot because it applies it and sees |
| +the failure. `updatesGet` cannot: it serves the blob without a core call, so a row |
| +corrupted on disk *after* §D5 verified it goes straight to the browser, where |
| +`Y.readUpdate` throws mid-loop and the resync controller refetches the same bytes |
| +forever. Server-side rendering keeps working; the editor becomes permanently |
| +unloadable for that document. That contradicts both "the worst failure is a |
| +performance regression" and "this needs no client change", so this design pays for |
| +it: **`?raw=1` forces the pre-snapshot behaviour**, returning only `doc_updates` |
| +rows, and `applyServerLog` refetches with it once when a decode throws. Small, and |
| +it makes the log's authority reachable from the one reader for whom it matters |
| +most. |
| + |
| +## 5. What does not change |
| + |
| +The wire format (a snapshot is an ordinary Yjs update); the browser bundle; the |
| +cursor protocol; the stream and its hub; ownership scoping; `doc_updates` itself, |
| +which keeps every row it has. |
| + |
| +## 6. Failure modes |
| + |
| +| Failure | Behaviour | |
| +|---|---| |
| +| No snapshot exists | Full replay, as today | |
| +| Snapshot row unreadable or from an older ABI | Discarded, full replay, logged | |
| +| D5's verification fails | Snapshot not written, log untouched, logged at Error | |
| +| Snapshot write fails after the update committed | Update stands, no snapshot, next Nth write tries again | |
| +| Core call traps producing the snapshot | Same as above; the update is already durable | |
| + |
| +Every row of that table ends in "correct, possibly slower", which is the bar this |
| +design sets for itself. |
| + |
| +## 7. Non-goals |
| + |
| +The compaction horizon and any deletion of rows; coalescing before persistence; |
| +sealing `state`; named versions (§14), which will need to pin points a future |
| +horizon must not collapse; the WebSocket transport; presence. |
| + |
| +## 8. Testing |
| + |
| +- **Equivalence is the central test.** For a document with more updates than the |
| + cadence, the text and HTML produced through snapshot-then-tail must equal what a |
| + full replay produces, byte for byte. Property-shaped rather than one fixture: |
| + several documents, several cadences, including a snapshot at the exact boundary. |
| +- **The cap stops being reachable.** Apply more than `maxRetainedUpdates` updates |
| + to one document and assert it still accepts writes — the regression test for the |
| + cliff this exists to remove. It should fail against `main` today. |
| +- **A document that is ALREADY at the cap recovers**, which the test above does |
| + not cover: driven through the new path a document never reaches the cap, so the |
| + production-shaped case needs its own fixture — a log of `maxRetainedUpdates` |
| + rows and no snapshot, asserting one 400 followed by a successful retry. This is |
| + §D3's second defect, and without this test the fix for it is unverified. |
| +- **The concurrent boundary race** (§D3's first defect): two POSTs racing at the |
| + cadence boundary must never produce a snapshot whose seq outruns its content. |
| + Assert against the rendered document, not against the snapshot row, since the |
| + row is exactly what a stale snapshot would make look right. |
| +- **`?raw=1` returns only `doc_updates` rows**, and a client that hits a decode |
| + failure recovers through it. |
| +- **Fallbacks are exercised, not assumed:** a corrupted snapshot row, a snapshot |
| + from a wrong ABI, and a missing snapshot each still produce correct content. |
| +- **D5's verifier rejects a bad snapshot** — constructed by hand, since the core |
| + will not produce one on request. |
| +- **The GET's cursor contract survives:** `?since=` behind, at, and ahead of the |
| + snapshot; `Docs-Seq` still derived from bytes in hand; the live-sync suite green |
| + unchanged. |
| +- **Wire fixtures unchanged.** They are the contract, not an output. |
| + |
| +## 9. Risks |
| + |
| +1. **The ABI bump is a coordination point.** `internal/core`'s check is |
| + exact-match, so a stale wasm artifact fails at load. Anyone rebasing across |
| + this rebuilds the artifact. |
| +2. **A wrong snapshot is the worst bug this repo could ship** — silently wrong |
| + content for every future reader, with the log that would prove it still sitting |
| + there unread. D2's never-authoritative rule and D5's verification exist for |
| + that one risk, and the review should be hardest on exactly that. |
| +3. **Storage still grows, and this slice adds to it.** §2 states the shape and the |
| + supersession rule that keeps it to one blob per document. The log itself is |
| + untouched and unbounded until the horizon lands. |
| +4. **`maxLogBytes` is bounded against history, not against size.** Snapshot-then- |
| + tail means a long-lived document no longer sends its whole past, but a document |
| + whose *current state* exceeds 8MB still fails the GET — with the snapshot in |
| + the body. Different cliff, further away, not removed. |
| + |
| +## 10. Review record |
| + |
| +Reviewed by **Fable** on 2026-09-01, at the spec stage, per AGENTS.md. Verdict: |
| +approve with changes; all folded in above. |
| + |
| +Two were correctness defects with the same root — where the snapshot is taken |
| +from. Snapshotting the handle that just accepted an update can record a state |
| +missing a concurrently-committed row (§D3), and can never rescue a document |
| +already at the cap, which is the case §1 opens with. Taking it from the replayed |
| +state before `Apply` closes both by construction. |
| + |
| +The rest: the exact `yrs` call named along with the two neighbours that look |
| +right and are not (§3); §D5 moved inside the core and strengthened from a text |
| +comparison to state-vector, delete-set and structural equality, with its honest |
| +limit stated; the `abi_version` column without which the failure table's own rule |
| +is unenforceable (§D2); the `?raw=1` escape hatch for a corrupt snapshot reaching |
| +a browser that has no way to refuse it (§D7); the cap's retirement restated as |
| +conditional on snapshot writes succeeding (§D4); and the supersession ruling that |
| +keeps this from making storage worse than the problem it solves (§2). |
| + |
| +The review also confirmed what §3 rests on: against an empty state vector, |
| +`encode_state_as_update_v1` is merge-equivalent to the log and not merely |
| +render-equivalent, so a snapshot is safe to build future concurrent merges on. |