oficina / docs Public

branch:

Clone

git clone https://amadan.net/oficina/docs

Plain git — no account needed to clone.

Download

Download this file

1	# Snapshots Implementation Plan
2
3	> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5	Goal: A document that has been edited past `maxRetainedUpdates` keeps accepting edits, and replaying one costs a snapshot plus a short tail instead of its whole history.
6
7	Architecture: The Rust core gains a state export — one Yjs update encoding the entire document — and a verifier that checks a candidate against the document it came from without leaving the core. `Store.ApplyUpdate` takes that snapshot from the replayed state before applying the incoming update, verifies it, and writes it to a new `doc_snapshots` table every 256 updates, deleting the one it supersedes. `Store.replay` and `GET /d/{sqid}/updates` both read snapshot-then-tail; a snapshot is never authoritative, so anything unreadable falls back to a full replay of the log.
8
9	Tech Stack: Rust `yrs` 0.23.5 compiled to wasm32 and run under wazero, Go 1.26, SQLite through GORM, JS `yjs` 13.6.27 unchanged.
10
11	Spec: `docs/superpowers/specs/2026-09-01-snapshots-design.md`. Read it before Task 1 — every decision below argues from one of its D-numbers, and the two most dangerous mistakes in this work (where the snapshot is taken from, and which `yrs` call encodes it) are both spec decisions rather than implementation choices.
12
13	## Global Constraints
14
15	- `CGO_ENABLED=0`. One static binary. No new Go module requirements.
16	- No entropy from the CRDT library. `Doc::new`, `Options::with_client_id` and friends are banned by clippy `disallowed-methods`. A new `Doc` takes both a guid and a client id from its caller — including the throwaway document Task 2's verifier builds.
17	- The core module must have zero host imports (`core/tests/no_host_imports.rs`).
18	- `internal/core`'s ABI check is exact-match: the wasm artifact and the host move together, and a stale artifact fails at load. Rebuild it after any Rust change.
19	- Never call `sessions.UserID`; never use `scope.Owned`. The AST guard in `internal/lint` fails the build.
20	- `Store.ApplyUpdate`'s owner re-derivation and its MAX(seq)+INSERT transaction are not to be weakened, reordered, or moved.
21	- The fixtures in `fixtures/` are the contract, not an output.
22	- Run `.amadan/ci.d/10-gate` before reporting any task complete. Exit 0 is green.
23	- Environment: `GOCACHE` must point somewhere writable (`$TMPDIR/go-build`); the default is read-only under the agent sandbox and fails as "setup failed", which reads like a build break and is not one. The Rust toolchain is at `~/.cargo/bin`, not on a default PATH. A fresh worktree needs `npm ci` in `fixtures/`.
24
25	## File Structure
26
27	\| File \| Responsibility \|
28	\|---\|---\|
29	\| `core/src/snapshot.rs` (new) \| Encode a document's whole state as one update; verify a candidate against a live document. The only place that knows either operation. \|
30	\| `core/src/lib.rs` \| Re-export `snapshot_bytes` / `verify_snapshot`. \|
31	\| `core/src/abi.rs` \| `docs_core_snapshot_len` / `_read`, `docs_core_verify_snapshot`; ABI 3 → 4. \|
32	\| `core/tests/snapshot.rs` (new) \| Round-trip, merge-equivalence, and verifier tests in Rust. \|
33	\| `internal/core/core.go` \| `abiVersion` 3 → 4; `(Handle).Snapshot`, `(Handle).VerifySnapshot`. \|
34	\| `internal/docs/migrations/0004_doc_snapshots.sql` (new) \| The table. \|
35	\| `internal/docs/models.go` \| `DocSnapshot`. \|
36	\| `internal/docs/snapshots.go` (new) \| Read the newest usable snapshot; write one and supersede its predecessor. The only file that knows the cadence. \|
37	\| `internal/docs/store.go` \| `replay` reads snapshot-then-tail; `ApplyUpdate` takes and writes the snapshot. \|
38	\| `internal/docs/handlers.go` \| `updatesGet` serves snapshot-then-tail, and `?raw=1`. \|
39	\| `fixtures/live-updates.mjs` \| One retry through `?raw=1` when applying fetched bytes throws. \|
40	\| `fixtures/editor-entry.js` \| Pass the raw flag through; same fallback on the initial load. \|
41
42	---
43
44	### Task 1: The core can encode a document's whole state
45
46	Without this there is nothing to store. The danger here is picking the wrong `yrs` call: two neighbours look right and are not (spec §3).
47
48	Files:
49	- Create: `core/src/snapshot.rs`, `core/tests/snapshot.rs`
50	- Modify: `core/src/lib.rs`, `core/src/abi.rs`
51
52	Interfaces:
53	- Produces: `pub fn snapshot_bytes(doc: &Doc) -> Vec<u8>` — the whole state as one update.
54	- Produces ABI: `docs_core_snapshot_len(handle) -> i32`, `docs_core_snapshot_read(handle, out_ptr, out_len) -> i32`, same convention and error codes as the existing `text`/`html`/`json` pairs.
55
56	- [ ] Step 1: Write the failing test
57
58	`core/tests/snapshot.rs`:
59
60	```rust
61	use yrs::{Doc, Options, ReadTxn, Transact, XmlFragmentPrelim, GetString};
62	use docs_core::{snapshot_bytes, fragment_text};
63
64	fn doc_with(client_id: u64) -> Doc {
65	Doc::with_options(Options::with_guid_and_client_id("test-guid".into(), client_id))
66	}
67
68	#[test]
69	fn a_snapshot_replayed_into_an_empty_doc_reproduces_the_document() {
70	let source = doc_with(1);
71	let update = std::fs::read("../fixtures/vectors/fragment/simple.bin").unwrap();
72	source.transact_mut().apply_update(yrs::Update::decode_v1(&update).unwrap()).unwrap();
73
74	let snap = snapshot_bytes(&source);
75
76	let rebuilt = doc_with(2);
77	rebuilt.transact_mut().apply_update(yrs::Update::decode_v1(&snap).unwrap()).unwrap();
78
79	assert_eq!(fragment_text(&rebuilt), fragment_text(&source));
80	assert!(!fragment_text(&rebuilt).is_empty(), "fixture produced no text");
81	}
82	```
83
84	- [ ] Step 2: Run it and watch it fail
85
86	Run: `PATH="$HOME/.cargo/bin:$PATH" cargo test --manifest-path core/Cargo.toml --test snapshot`
87	Expected: FAIL — `snapshot_bytes` is not defined.
88
89	- [ ] Step 3: Implement `snapshot_bytes`
90
91	`core/src/snapshot.rs`:
92
93	```rust
94	//! Encoding a whole document as one update, and checking that the
95	//! result is the document it claims to be.
96	//!
97	//! The call below is `encode_state_as_update_v1` against an EMPTY state
98	//! vector, and which call this is matters more than anything else in
99	//! this file. Against the empty vector it encodes every block, the
100	//! entire delete set, and — through yrs's own `merge_pending_v1` — the
101	//! pending queue, meaning updates whose dependencies never arrived and
102	//! which `apply_update` accepted and parked. That makes the result
103	//! merge-equivalent to the log it replaces, not merely
104	//! render-equivalent: every clock range and every tombstone survives,
105	//! so a later concurrent merge integrates against a snapshot exactly as
106	//! it would against the full history.
107	//!
108	//! Two neighbours are traps, and both look like this one:
109	//!
110	//! - `encode_diff_v1(&StateVector::default())` silently drops the
111	//! pending queue (its own doc comment says so). A document holding an
112	//! update whose dependency arrives later heals under a full replay and
113	//! does NOT heal under a diff-built snapshot — and no render
114	//! comparison can see the difference, because pending content is
115	//! invisible in both documents.
116	//! - yrs's own `Snapshot` type and `encode_state_from_snapshot` are a
117	//! different feature (historical views, requiring `skip_gc`) that
118	//! happens to share this module's name.
119	use yrs::{Doc, ReadTxn, StateVector, Transact};
120
121	/// The whole document, as one update.
122	pub fn snapshot_bytes(doc: &Doc) -> Vec<u8> {
123	doc.transact().encode_state_as_update_v1(&StateVector::default())
124	}
125	```
126
127	Add to `core/src/lib.rs`:
128
129	```rust
130	mod snapshot;
131	pub use snapshot::snapshot_bytes;
132	```
133
134	- [ ] Step 4: Run the test and watch it pass
135
136	Run: `PATH="$HOME/.cargo/bin:$PATH" cargo test --manifest-path core/Cargo.toml --test snapshot`
137	Expected: PASS.
138
139	- [ ] Step 5: Add the ABI pair
140
141	In `core/src/abi.rs`, beside `json_string`:
142
143	```rust
144	/// The whole document as one update. Bytes, not a string — the host
145	/// reads it through the same len/read convention as the three
146	/// renderers, which is why this is shaped like them.
147	pub fn snapshot_vec(handle: u32) -> Option<Vec<u8>> {
148	get_doc(handle).map(\|doc\| snapshot_bytes(&doc))
149	}
150	```
151
152	and the two exports, copying the `docs_core_json_len` / `docs_core_json_read` bodies exactly, substituting `snapshot_vec` for `json_string` and `s.as_ptr()` reading from the `Vec<u8>`:
153
154	```rust
155	#[no_mangle]
156	pub extern "C" fn docs_core_snapshot_len(handle: u32) -> i32 {
157	match snapshot_vec(handle) {
158	Some(v) => v.len().try_into().unwrap_or(i32::MAX),
159	None => ERR_NO_DOC,
160	}
161	}
162
163	/// # Safety
164	///
165	/// `out_ptr` and `out_len` must describe a byte range that lies entirely
166	/// within this module's linear memory and is valid for writes for the
167	/// duration of this call.
168	#[no_mangle]
169	pub unsafe extern "C" fn docs_core_snapshot_read(handle: u32, out_ptr: u32, out_len: u32) -> i32 {
170	let v = match snapshot_vec(handle) {
171	Some(v) => v,
172	None => return ERR_NO_DOC,
173	};
174	if v.len() > out_len as usize {
175	return ERR_BUFFER_TOO_SMALL;
176	}
177	if !in_bounds(out_ptr, out_len) {
178	return ERR_INVALID_LENGTH;
179	}
180	std::ptr::copy_nonoverlapping(v.as_ptr(), out_ptr as *mut u8, v.len());
181	v.len().try_into().unwrap_or(i32::MAX)
182	}
183	```
184
185	- [ ] Step 6: Bump the ABI version to 4 in `core/src/abi.rs`, and leave the existing exports untouched.
186
187	- [ ] Step 7: Build the artifact and run the Rust suite
188
189	Run: `PATH="$HOME/.cargo/bin:$PATH" ./scripts/build-wasm.sh && PATH="$HOME/.cargo/bin:$PATH" DOCS_REQUIRE_WASM=1 cargo test --manifest-path core/Cargo.toml`
190	Expected: PASS, including `no_host_imports`.
191
192	- [ ] Step 8: Commit — `git add core/ && git commit`
193
194	---
195
196	### Task 2: The core can tell a good snapshot from a bad one
197
198	Spec D5. This runs inside the core so the comparison is against Rust structs rather than encoded bytes, whose encoding is not canonical.
199
200	Files:
201	- Modify: `core/src/snapshot.rs`, `core/src/abi.rs`, `core/src/lib.rs`
202	- Test: `core/tests/snapshot.rs`
203
204	Interfaces:
205	- Produces: `pub fn verify_snapshot(doc: &Doc, candidate: &[u8], guid: &str, client_id: u64) -> bool`
206	- Produces ABI: `docs_core_verify_snapshot(handle, ptr, len) -> i32` — `0` match, `1` mismatch, `ERR_DECODE` undecodable, `ERR_NO_DOC` unknown handle.
207
208	- [ ] Step 1: Write the failing tests
209
210	Append to `core/tests/snapshot.rs`:
211
212	```rust
213	#[test]
214	fn a_snapshot_of_this_document_verifies() {
215	let source = doc_with(1);
216	let update = std::fs::read("../fixtures/vectors/fragment/simple.bin").unwrap();
217	source.transact_mut().apply_update(yrs::Update::decode_v1(&update).unwrap()).unwrap();
218	let snap = snapshot_bytes(&source);
219	assert!(docs_core::verify_snapshot(&source, &snap, "verify-guid", 99));
220	}
221
222	#[test]
223	fn a_snapshot_of_a_different_document_does_not_verify() {
224	let source = doc_with(1);
225	source.transact_mut()
226	.apply_update(yrs::Update::decode_v1(
227	&std::fs::read("../fixtures/vectors/fragment/simple.bin").unwrap()).unwrap()).unwrap();
228
229	let other = doc_with(1);
230	other.transact_mut()
231	.apply_update(yrs::Update::decode_v1(
232	&std::fs::read("../fixtures/vectors/fragment/marks.bin").unwrap()).unwrap()).unwrap();
233	let wrong = snapshot_bytes(&other);
234
235	assert!(!docs_core::verify_snapshot(&source, &wrong, "verify-guid", 99));
236	}
237
238	#[test]
239	fn garbage_does_not_verify() {
240	let source = doc_with(1);
241	assert!(!docs_core::verify_snapshot(&source, &[0xff, 0x00, 0x13, 0x37], "verify-guid", 99));
242	}
243	```
244
245	- [ ] Step 2: Run and watch all three fail (`verify_snapshot` undefined).
246
247	- [ ] Step 3: Implement it
248
249	Append to `core/src/snapshot.rs`:
250
251	```rust
252	/// True when `candidate`, replayed into an empty document, is the same
253	/// document as `doc`.
254	///
255	/// Three comparisons, none of them on encoded bytes: update encoding is
256	/// not canonical (merge boundaries vary), so byte equality would report
257	/// false mismatches and no cheap byte-level proof exists. State vector
258	/// equality says both documents saw the same clock ranges from the same
259	/// clients; delete set equality says they agree on every deletion;
260	/// `fragment_json` equality says they render to the same structure,
261	/// which catches formatting-only divergence that a text comparison
262	/// misses at identical cost.
263	///
264	/// This is deliberately not a proof. What survives all three is "yrs
265	/// corrupted content while preserving every clock range, every deletion
266	/// and the full structural render", and that is a narrow enough residual
267	/// to accept.
268	///
269	/// What it CANNOT catch, stated because the temptation is to believe it
270	/// can: a snapshot taken from a stale state verifies perfectly, because
271	/// it is compared against the very document that produced it. That
272	/// failure is closed by taking the snapshot from the replayed state
273	/// (the spec's D3), not here.
274	///
275	/// guid/client_id are the caller's, never generated here — the clippy
276	/// disallowed-methods rule against drawing entropy from the CRDT library
277	/// applies to this throwaway document exactly as it does to a real one.
278	pub fn verify_snapshot(doc: &Doc, candidate: &[u8], guid: &str, client_id: u64) -> bool {
279	let update = match yrs::Update::decode_v1(candidate) {
280	Ok(u) => u,
281	Err(_) => return false,
282	};
283	let rebuilt = Doc::with_options(Options::with_guid_and_client_id(guid.into(), client_id));
284	{
285	let mut txn = rebuilt.transact_mut();
286	if txn.apply_update(update).is_err() {
287	return false;
288	}
289	}
290	let (a, b) = (doc.transact(), rebuilt.transact());
291	a.state_vector() == b.state_vector()
292	&& a.delete_set() == b.delete_set()
293	&& crate::fragment_json(doc) == crate::fragment_json(&rebuilt)
294	}
295	```
296
297	Export it from `lib.rs` (`pub use snapshot::{snapshot_bytes, verify_snapshot};`).
298
299	- [ ] Step 4: Run the tests and watch them pass.
300
301	- [ ] Step 5: Add the ABI export
302
303	```rust
304	/// 0 = the candidate is this document, 1 = it is not, ERR_DECODE = the
305	/// bytes are not an update at all, ERR_NO_DOC = no such handle.
306	///
307	/// # Safety
308	///
309	/// `ptr`/`len` must describe a readable byte range inside this module's
310	/// linear memory.
311	#[no_mangle]
312	pub unsafe extern "C" fn docs_core_verify_snapshot(handle: u32, ptr: u32, len: u32) -> i32 {
313	if !in_bounds(ptr, len) {
314	return ERR_INVALID_LENGTH;
315	}
316	let candidate = std::slice::from_raw_parts(ptr as *const u8, len as usize);
317	match get_doc(handle) {
318	None => ERR_NO_DOC,
319	Some(doc) => {
320	// The verifier's throwaway document takes a fixed guid and
321	// client id rather than generating any: entropy inside the
322	// core is banned (clippy disallowed-methods), and this
323	// document is discarded before the call returns, so a fixed
324	// pair is not merely acceptable but correct.
325	if verify_snapshot(&doc, candidate, "docs-core-verify", 1) { 0 } else { 1 }
326	}
327	}
328	}
329	```
330
331	- [ ] Step 6: Rebuild the artifact, run the Rust suite, commit.
332
333	---
334
335	### Task 3: The Go host can take and check a snapshot
336
337	Files:
338	- Modify: `internal/core/core.go`
339	- Test: `internal/core/core_test.go`
340
341	Interfaces:
342	- Produces: `func (h *Handle) Snapshot(ctx context.Context) ([]byte, error)`
343	- Produces: `func (h *Handle) VerifySnapshot(ctx context.Context, candidate []byte) (bool, error)`
344
345	- [ ] Step 1: Write the failing test
346
347	```go
348	func TestSnapshotRoundTripsThroughAFreshHandle(t *testing.T) {
349	rt := newTestRuntime(t)
350	ctx := context.Background()
351
352	h, err := rt.Open(ctx, "guid-a", 1)
353	if err != nil {
354	t.Fatalf("Open: %v", err)
355	}
356	defer h.Close(ctx)
357	if err := h.Apply(ctx, fragmentFixture(t, "simple")); err != nil {
358	t.Fatalf("Apply: %v", err)
359	}
360	want, err := h.Text(ctx)
361	if err != nil {
362	t.Fatalf("Text: %v", err)
363	}
364
365	snap, err := h.Snapshot(ctx)
366	if err != nil {
367	t.Fatalf("Snapshot: %v", err)
368	}
369	if len(snap) == 0 {
370	t.Fatal("Snapshot returned no bytes")
371	}
372
373	fresh, err := rt.Open(ctx, "guid-b", 2)
374	if err != nil {
375	t.Fatalf("Open fresh: %v", err)
376	}
377	defer fresh.Close(ctx)
378	if err := fresh.Apply(ctx, snap); err != nil {
379	t.Fatalf("Apply snapshot: %v", err)
380	}
381	got, err := fresh.Text(ctx)
382	if err != nil {
383	t.Fatalf("Text: %v", err)
384	}
385	if got != want {
386	t.Errorf("snapshot rebuilt %q, want %q", got, want)
387	}
388	}
389
390	func TestVerifySnapshotAcceptsItsOwnAndRejectsAnother(t *testing.T) {
391	rt := newTestRuntime(t)
392	ctx := context.Background()
393	h, _ := rt.Open(ctx, "guid-a", 1)
394	defer h.Close(ctx)
395	if err := h.Apply(ctx, fragmentFixture(t, "simple")); err != nil {
396	t.Fatalf("Apply: %v", err)
397	}
398	snap, err := h.Snapshot(ctx)
399	if err != nil {
400	t.Fatalf("Snapshot: %v", err)
401	}
402
403	ok, err := h.VerifySnapshot(ctx, snap)
404	if err != nil {
405	t.Fatalf("VerifySnapshot: %v", err)
406	}
407	if !ok {
408	t.Error("a document refused its own snapshot")
409	}
410
411	if ok, err := h.VerifySnapshot(ctx, []byte{0xff, 0x00}); err != nil \|\| ok {
412	t.Errorf("garbage verified: ok=%v err=%v", ok, err)
413	}
414	}
415	```
416
417	Match the existing helpers in `core_test.go` for building a runtime and reading a fragment fixture; if they are named differently, use the existing names rather than adding new ones.
418
419	- [ ] Step 2: Run and watch it fail.
420
421	- [ ] Step 3: Bump `abiVersion` to 4 and implement
422
423	```go
424	// Snapshot returns the document's entire state as one Yjs update — what
425	// doc_snapshots stores so a later replay can start from it instead of
426	// from the whole log (the snapshots design, D1).
427	//
428	// Bytes rather than a string, but it rides the same len/read machinery
429	// as Text/HTML/JSON, including their trap-and-retry: a Go string is a
430	// byte sequence and nothing here interprets it, so "snapshot" is simply
431	// a fourth renderer name to readOn.
432	func (h *Handle) Snapshot(ctx context.Context) ([]byte, error) {
433	s, err := h.read(ctx, "snapshot")
434	if err != nil {
435	return nil, err
436	}
437	return []byte(s), nil
438	}
439
440	// VerifySnapshot asks the core whether candidate, replayed into an empty
441	// document, is this document — state vector, delete set and structural
442	// render (the snapshots design, D5). It is a smoke test rather than a
443	// proof, and it cannot detect a snapshot taken from a stale state; see
444	// core/src/snapshot.rs for both limits.
445	func (h *Handle) VerifySnapshot(ctx context.Context, candidate []byte) (bool, error) {
446	// ... allocate candidate into guest memory exactly as Apply does,
447	// call docs_core_verify_snapshot, free, and map: 0 -> true, 1 ->
448	// false, ERR_DECODE -> false with no error (bad bytes are an answer,
449	// not a failure), anything else -> error.
450	}
451	```
452
453	Write `VerifySnapshot`'s body by copying `Apply`'s allocate/call/free structure — it is the only existing export that takes a byte buffer, and its guest-memory handling, trap taxonomy and retry are what this needs too.
454
455	- [ ] Step 4: Rebuild the artifact, run `go test ./internal/core/`, commit.
456
457	---
458
459	### Task 4: The table, and reading the newest usable snapshot
460
461	Files:
462	- Create: `internal/docs/migrations/0004_doc_snapshots.sql`, `internal/docs/snapshots.go`
463	- Modify: `internal/docs/models.go`
464	- Test: `internal/docs/snapshots_test.go` (new)
465
466	Interfaces:
467	- Produces: `type DocSnapshot struct { ID, DocID, Seq int64; State []byte; ABIVersion uint32; CreatedAt time.Time }`
468	- Produces: `func (s Store) newestSnapshot(ctx context.Context, docID int64) (DocSnapshot, error)` — nil, nil when there is none usable.
469	- Produces: `const snapshotCadence = 256`
470
471	- [ ] Step 1: Write the migration
472
473	`0004_doc_snapshots.sql`:
474
475	```sql
476	-- doc_snapshots holds one Yjs update per row encoding a document's
477	-- entire state as of `seq` (the snapshots design, D2). It is an
478	-- accelerator, never the source of truth: doc_updates remains complete
479	-- and authoritative, and a row here that cannot be read is discarded in
480	-- favour of replaying the log.
481	--
482	-- (doc_id, seq) is deliberately NOT unique. Two concurrent POSTs at the
483	-- cadence boundary can both snapshot the same replayed prefix, and those
484	-- rows are equivalent by construction, so a collision is not an error to
485	-- raise. "Newest usable" resolves as highest seq, then highest id.
486	--
487	-- abi_version records what produced the row, which is what makes
488	-- discarding a snapshot from an older core implementable rather than
489	-- merely promised.
490	--
491	-- Additive: creates a table, touches no existing row.
492	CREATE TABLE IF NOT EXISTS doc_snapshots (
493	id INTEGER PRIMARY KEY AUTOINCREMENT,
494	doc_id INTEGER NOT NULL,
495	seq INTEGER NOT NULL,
496	state BLOB NOT NULL,
497	abi_version INTEGER NOT NULL,
498	created_at DATETIME NOT NULL
499	);
500
501	CREATE INDEX IF NOT EXISTS idx_doc_snapshots_doc_seq
502	ON doc_snapshots (doc_id, seq DESC, id DESC);
503	```
504
505	- [ ] Step 2: Write the failing test
506
507	`internal/docs/snapshots_test.go`:
508
509	```go
510	func TestNewestSnapshotPrefersHighestSeqThenHighestID(t *testing.T) {
511	app, err := NewForTest(t)
512	if err != nil {
513	t.Fatalf("NewForTest: %v", err)
514	}
515	ctx := context.Background()
516	owner := app.TestPerson(t, "owner@example.com")
517	doc, _ := app.Store().Create(ctx, owner, "Snapshots")
518
519	write := func(seq int64, state string, abi uint32) {
520	row := &DocSnapshot{DocID: doc.ID, Seq: seq, State: []byte(state), ABIVersion: abi, CreatedAt: time.Now().UTC()}
521	if err := app.Store().db.WithContext(ctx).Create(row).Error; err != nil {
522	t.Fatalf("insert snapshot: %v", err)
523	}
524	}
525	write(10, "older", coreABIVersion)
526	write(20, "first-at-20", coreABIVersion)
527	write(20, "second-at-20", coreABIVersion)
528
529	got, err := app.Store().newestSnapshot(ctx, doc.ID)
530	if err != nil {
531	t.Fatalf("newestSnapshot: %v", err)
532	}
533	if got == nil \|\| string(got.State) != "second-at-20" {
534	t.Fatalf("newestSnapshot returned %+v, want the second row at seq 20", got)
535	}
536	}
537
538	func TestNewestSnapshotIgnoresAnotherABIsRows(t *testing.T) {
539	app, _ := NewForTest(t)
540	ctx := context.Background()
541	owner := app.TestPerson(t, "owner@example.com")
542	doc, _ := app.Store().Create(ctx, owner, "Old ABI")
543
544	row := &DocSnapshot{DocID: doc.ID, Seq: 5, State: []byte("from an older core"), ABIVersion: coreABIVersion - 1, CreatedAt: time.Now().UTC()}
545	if err := app.Store().db.WithContext(ctx).Create(row).Error; err != nil {
546	t.Fatalf("insert: %v", err)
547	}
548
549	got, err := app.Store().newestSnapshot(ctx, doc.ID)
550	if err != nil {
551	t.Fatalf("newestSnapshot: %v", err)
552	}
553	if got != nil {
554	t.Error("a snapshot from another ABI was treated as usable")
555	}
556	}
557
558	func TestNewestSnapshotIsNilWhenThereAreNone(t *testing.T) {
559	app, _ := NewForTest(t)
560	ctx := context.Background()
561	owner := app.TestPerson(t, "owner@example.com")
562	doc, _ := app.Store().Create(ctx, owner, "None")
563
564	got, err := app.Store().newestSnapshot(ctx, doc.ID)
565	if err != nil \|\| got != nil {
566	t.Errorf("newestSnapshot = (%v, %v), want (nil, nil)", got, err)
567	}
568	}
569	```
570
571	- [ ] Step 3: Run and watch them fail.
572
573	- [ ] Step 4: Add the model and the reader
574
575	`models.go`:
576
577	```go
578	// DocSnapshot is one document's entire state encoded as a single Yjs
579	// update, as of Seq. It is an accelerator for replay and for the update
580	// endpoint, never the source of truth — doc_updates keeps every row, and
581	// a snapshot that cannot be read is discarded in favour of replaying
582	// them (the snapshots design, D2).
583	//
584	// State is plaintext in this table, exactly as DocUpdate.UpdateBin is.
585	// Worth one caveat the sealing work should carry: a doc_updates row is a
586	// fragment of an edit, while this is the entire document in one blob.
587	//
588	// ABIVersion records the core that produced State. Without it, "discard
589	// a snapshot from an older ABI" is unenforceable prose.
590	type DocSnapshot struct {
591	ID int64 `gorm:"primaryKey"`
592	DocID int64 `gorm:"column:doc_id;not null;index"`
593	Seq int64 `gorm:"column:seq;not null"`
594
595	State []byte `gorm:"column:state;not null"`
596	ABIVersion uint32 `gorm:"column:abi_version;not null"`
597
598	CreatedAt time.Time `gorm:"column:created_at;not null"`
599	}
600	```
601
602	`snapshots.go`:
603
604	```go
605	package docs
606
607	// snapshotCadence is how many updates may accumulate past a snapshot
608	// before the next one is taken. 256 keeps a replay to a snapshot plus a
609	// short tail, which is what holds the retained-update count in a
610	// core.Handle clear of maxRetainedUpdates (4096) — about 16x of headroom.
611	//
612	// That headroom is conditional, and the condition is worth knowing:
613	// it holds while snapshot writes keep succeeding. Roughly fifteen
614	// consecutive failures — each logged at Error, so noisy rather than
615	// silent — walk the tail back to the cap.
616	const snapshotCadence = 256
617
618	// newestSnapshot returns the most recent usable snapshot for docID, or
619	// nil when there is none.
620	//
621	// "Usable" excludes rows written by a different core ABI: the state
622	// encoding is that core's business, and a row from another one is
623	// discarded rather than trusted. Ordering is seq DESC, id DESC — two
624	// concurrent POSTs at the cadence boundary may legitimately write
625	// equivalent snapshots at the same seq, so the tie resolves to the last
626	// row written rather than being treated as an error.
627	func (s Store) newestSnapshot(ctx context.Context, docID int64) (DocSnapshot, error) {
628	var row DocSnapshot
629	err := s.db.WithContext(ctx).
630	Where("doc_id = ? AND abi_version = ?", docID, coreABIVersion).
631	Order("seq DESC, id DESC").
632	First(&row).Error
633	if errors.Is(err, gorm.ErrRecordNotFound) {
634	return nil, nil
635	}
636	if err != nil {
637	return nil, fmt.Errorf("docs: newest snapshot for document %d: %w", docID, err)
638	}
639	return &row, nil
640	}
641	```
642
643	Export the ABI version from `internal/core` as `core.ABIVersion` and alias it here as `coreABIVersion`, rather than duplicating the number.
644
645	- [ ] Step 5: Register the migration in `models.go`'s `Schema` — it is `migrate.MustFromFS(migrationFS, "docs")` over `migrations/*.sql`, so the new file is picked up by its number with no code change. Confirm by running the tests.
646
647	- [ ] Step 6: Run tests, then commit.
648
649	---
650
651	### Task 5: Replay reads snapshot-then-tail, and writes one at the cadence
652
653	The heart of it, and the task carrying both defects the spec review found.
654
655	Files:
656	- Modify: `internal/docs/store.go`, `internal/docs/snapshots.go`
657	- Test: `internal/docs/snapshots_test.go`
658
659	Interfaces:
660	- Produces: `func (s Store) writeSnapshot(ctx context.Context, doc Document, h *core.Handle, seq int64) error`
661	- Changes: `replay` starts from the newest usable snapshot; `ApplyUpdate` takes the snapshot from the replayed state before applying the incoming update.
662
663	- [ ] Step 1: Write the failing tests — equivalence, the race, and the capped document
664
665	```go
666	// The central property: a document replayed through snapshot-then-tail
667	// must render identically to one replayed from the whole log.
668	func TestSnapshotThenTailRendersIdenticallyToAFullReplay(t *testing.T) {
669	app, _ := NewForTest(t)
670	ctx := context.Background()
671	owner := app.TestPerson(t, "owner@example.com")
672	doc, _ := app.Store().Create(ctx, owner, "Equivalence")
673
674	update := validUpdate(t)
675	for i := 0; i < snapshotCadence+5; i++ {
676	if _, err := app.Store().ApplyUpdate(ctx, doc, update); err != nil {
677	t.Fatalf("ApplyUpdate %d: %v", i, err)
678	}
679	}
680	withSnapshot, err := app.Store().HTML(ctx, doc)
681	if err != nil {
682	t.Fatalf("HTML: %v", err)
683	}
684
685	// Delete every snapshot and read again: same document, full replay.
686	if err := app.Store().db.WithContext(ctx).
687	Where("doc_id = ?", doc.ID).Delete(&DocSnapshot{}).Error; err != nil {
688	t.Fatalf("delete snapshots: %v", err)
689	}
690	fullReplay, err := app.Store().HTML(ctx, doc)
691	if err != nil {
692	t.Fatalf("HTML after deleting snapshots: %v", err)
693	}
694	if withSnapshot != fullReplay {
695	t.Error("snapshot-then-tail and full replay disagree about the document")
696	}
697	}
698
699	// The spec's D3, first defect: a snapshot must never claim a seq whose
700	// content it does not contain. Asserted against the rendered document,
701	// never against the snapshot row — the row is exactly what a stale
702	// snapshot makes look right.
703	func TestConcurrentPostsAtTheCadenceBoundaryDoNotStrandContent(t *testing.T) {
704	app, _ := NewForTest(t)
705	ctx := context.Background()
706	owner := app.TestPerson(t, "owner@example.com")
707	doc, _ := app.Store().Create(ctx, owner, "Boundary")
708
709	for i := 0; i < snapshotCadence-1; i++ {
710	if _, err := app.Store().ApplyUpdate(ctx, doc, validUpdate(t)); err != nil {
711	t.Fatalf("warmup %d: %v", i, err)
712	}
713	}
714
715	var wg sync.WaitGroup
716	for i := 0; i < 8; i++ {
717	wg.Add(1)
718	go func() {
719	defer wg.Done()
720	_, _ = app.Store().ApplyUpdate(ctx, doc, validUpdate(t))
721	}()
722	}
723	wg.Wait()
724
725	withSnapshot, err := app.Store().HTML(ctx, doc)
726	if err != nil {
727	t.Fatalf("HTML: %v", err)
728	}
729	if err := app.Store().db.WithContext(ctx).
730	Where("doc_id = ?", doc.ID).Delete(&DocSnapshot{}).Error; err != nil {
731	t.Fatalf("delete snapshots: %v", err)
732	}
733	fullReplay, err := app.Store().HTML(ctx, doc)
734	if err != nil {
735	t.Fatalf("HTML: %v", err)
736	}
737	if withSnapshot != fullReplay {
738	t.Error("a snapshot written during concurrent writes stranded content the log still holds")
739	}
740	}
741
742	// The spec's D3, second defect, and the case section 1 opens with: a
743	// document that is ALREADY at the cap when this ships. Driven through
744	// the new path a document never reaches the cap, so this builds the
745	// production shape directly — a full log and no snapshot — and asserts
746	// it recovers.
747	func TestADocumentAlreadyAtTheCapRecovers(t *testing.T) {
748	app, _ := NewForTest(t)
749	ctx := context.Background()
750	owner := app.TestPerson(t, "owner@example.com")
751	doc, _ := app.Store().Create(ctx, owner, "At the cap")
752
753	// Write rows straight into the log, bypassing ApplyUpdate, so no
754	// snapshot is ever taken — exactly the state a document edited
755	// before this feature shipped is in.
756	update := validUpdate(t)
757	rows := make([]DocUpdate, 0, core.MaxRetainedUpdates)
758	for i := 1; i <= core.MaxRetainedUpdates; i++ {
759	rows = append(rows, DocUpdate{DocID: doc.ID, Seq: int64(i), UpdateBin: update, CreatedAt: time.Now().UTC()})
760	}
761	if err := app.Store().db.WithContext(ctx).CreateInBatches(rows, 500).Error; err != nil {
762	t.Fatalf("seed a full log: %v", err)
763	}
764
765	// The first write still fails — the handle is full before the
766	// snapshot can help — but it must leave a snapshot behind.
767	if _, err := app.Store().ApplyUpdate(ctx, doc, update); err == nil {
768	t.Log("first write succeeded; acceptable if the cap was not reached")
769	}
770	snap, err := app.Store().newestSnapshot(ctx, doc.ID)
771	if err != nil {
772	t.Fatalf("newestSnapshot: %v", err)
773	}
774	if snap == nil {
775	t.Fatal("a capped document got no snapshot, so it can never recover")
776	}
777
778	// The retry replays snapshot + empty tail and succeeds.
779	if _, err := app.Store().ApplyUpdate(ctx, doc, update); err != nil {
780	t.Errorf("retry after a snapshot existed still failed: %v", err)
781	}
782	}
783	```
784
785	```go
786	// A corrupted snapshot row must not corrupt the document: the log is
787	// authoritative, so an unreadable snapshot is discarded and the whole
788	// log replayed instead. Correct, and merely slower.
789	func TestACorruptSnapshotFallsBackToTheLog(t *testing.T) {
790	app, _ := NewForTest(t)
791	ctx := context.Background()
792	owner := app.TestPerson(t, "owner@example.com")
793	doc, _ := app.Store().Create(ctx, owner, "Corrupt")
794	for i := 0; i < snapshotCadence+2; i++ {
795	if _, err := app.Store().ApplyUpdate(ctx, doc, validUpdate(t)); err != nil {
796	t.Fatalf("ApplyUpdate %d: %v", i, err)
797	}
798	}
799	want, err := app.Store().HTML(ctx, doc)
800	if err != nil {
801	t.Fatalf("HTML: %v", err)
802	}
803
804	if err := app.Store().db.WithContext(ctx).Model(&DocSnapshot{}).
805	Where("doc_id = ?", doc.ID).
806	Update("state", []byte{0xff, 0x00, 0x13, 0x37}).Error; err != nil {
807	t.Fatalf("corrupt the snapshot: %v", err)
808	}
809
810	got, err := app.Store().HTML(ctx, doc)
811	if err != nil {
812	t.Fatalf("HTML with a corrupt snapshot: %v", err)
813	}
814	if got != want {
815	t.Error("a corrupt snapshot changed the document instead of being discarded")
816	}
817	}
818
819	// The cliff itself, driven the long way round: more updates than
820	// maxRetainedUpdates through the real path, which must keep accepting
821	// them. Slow — thousands of wasm applies — so it is skipped under
822	// -short, but it is the one test that proves the headline claim rather
823	// than a proxy for it.
824	func TestADocumentPastTheRetainedCapKeepsAcceptingWrites(t *testing.T) {
825	if testing.Short() {
826	t.Skip("thousands of updates; run without -short")
827	}
828	app, _ := NewForTest(t)
829	ctx := context.Background()
830	owner := app.TestPerson(t, "owner@example.com")
831	doc, _ := app.Store().Create(ctx, owner, "Past the cap")
832
833	for i := 0; i < core.MaxRetainedUpdates+50; i++ {
834	if _, err := app.Store().ApplyUpdate(ctx, doc, validUpdate(t)); err != nil {
835	t.Fatalf("ApplyUpdate %d of %d: %v", i, core.MaxRetainedUpdates+50, err)
836	}
837	}
838	}
839	```
840
841	`core.MaxRetainedUpdates` and `core.ABIVersion` must be exported from `internal/core` for these tests and for `snapshots.go`; export them rather than duplicating 4096 and the version number.
842
843	`replay`'s signature changes to return what the caller needs rather than making it ask again: `func (s Store) replay(ctx context.Context, doc Document) (h *core.Handle, replayedSeq int64, lastSnapshotSeq int64, err error)`. Both seqs come from rows it has already read — the same no-second-query discipline the live-sync design's D2 sets, for the same reason.
844
845	- [ ] Step 2: Run all three and watch them fail.
846
847	- [ ] Step 3: Make `replay` read snapshot-then-tail
848
849	In `store.go`'s `replay`, before loading updates: read `newestSnapshot`; if there is one, `h.Apply(ctx, snap.State)` and load only `seq > snap.Seq`. If applying it fails, log at Error, discard it, and fall back to loading the whole log — a snapshot is never authoritative.
850
851	- [ ] Step 4: Take and write the snapshot in `ApplyUpdate`, before `Apply`
852
853	After `replay` returns the handle and before `h.Apply(ctx, update)`:
854
855	```go
856	// The snapshot is taken HERE — from the replayed state, before the
857	// incoming update is applied — and the placement is the whole of its
858	// correctness (the snapshots design, D3).
859	//
860	// Taking it after a successful Apply is wrong twice. The replay above
861	// runs outside the transaction below, so a concurrent POST can commit
862	// a row this handle never saw; a snapshot taken from that handle
863	// would claim a seq whose content it does not hold, and D5's
864	// verification cannot notice, because it compares the candidate
865	// against this same stale handle. And a document already at the cap
866	// never reaches an after-success hook at all — Apply refuses first —
867	// so the one case this feature exists for would stay broken forever.
868	//
869	// The replayed prefix is complete by construction: seq is gapless
870	// (MAX+1 under a single-writer pool) and replay read every row that
871	// existed when it ran. So replayedSeq is unconditionally truthful.
872	if replayedSeq-lastSnapshotSeq >= snapshotCadence {
873	if err := s.writeSnapshot(ctx, doc, h, replayedSeq); err != nil {
874	// Never fatal: the update below still stands, the log is
875	// untouched, and the next POST tries again.
876	s.logger.Error("docs: write snapshot", "doc_id", doc.ID, "seq", replayedSeq, "err", err)
877	}
878	}
879	```
880
881	`replay` returns `replayedSeq` and `lastSnapshotSeq` alongside the handle — read from the snapshot row it already loaded, never a second query.
882
883	- [ ] Step 5: Implement `writeSnapshot`
884
885	Take `h.Snapshot(ctx)`, `h.VerifySnapshot(ctx, candidate)`, refuse to write on a false verdict (log at Error and return an error), otherwise insert the row with `core.ABIVersion` and delete every older snapshot for that document — superseded snapshots are derived data, reproducible from the log, so removing them deletes no history and keeps this feature from costing one document-sized blob per 256 updates.
886
887	- [ ] Step 6: Run the three tests, plus the whole `internal/docs` suite, and watch them pass.
888
889	- [ ] Step 7: Commit.
890
891	---
892
893	### Task 6: The update endpoint serves snapshot-then-tail, and `?raw=1`
894
895	Files:
896	- Modify: `internal/docs/handlers.go`
897	- Test: `internal/docs/cursor_test.go`
898
899	- [ ] Step 1: Write the failing tests
900
901	```go
902	// pastCadence builds a document with more updates than the cadence, so a
903	// snapshot exists, and returns it.
904	func pastCadence(t testing.T, app App, title string) *Document {
905	t.Helper()
906	ctx := context.Background()
907	owner := app.TestPerson(t, "owner@example.com")
908	doc, err := app.Store().Create(ctx, owner, title)
909	if err != nil {
910	t.Fatalf("Create: %v", err)
911	}
912	for i := 0; i < snapshotCadence+3; i++ {
913	if _, err := app.Store().ApplyUpdate(ctx, doc, validUpdate(t)); err != nil {
914	t.Fatalf("ApplyUpdate %d: %v", i, err)
915	}
916	}
917	return doc
918	}
919
920	// A client far behind the snapshot receives the snapshot plus the tail
921	// rather than the whole log, and the header still names the highest seq
922	// the bytes actually reach.
923	func TestUpdatesGetServesSnapshotThenTail(t *testing.T) {
924	app, err := NewForTest(t)
925	if err != nil {
926	t.Fatalf("NewForTest: %v", err)
927	}
928	doc := pastCadence(t, app, "Snapshot then tail")
929
930	snap := serve(app, app.TestRequest(t, "owner@example.com", http.MethodGet,
931	"/d/"+doc.Sqid+"/updates?since=1", nil))
932	if snap.Code != http.StatusOK {
933	t.Fatalf("GET = %d, want 200", snap.Code)
934	}
935	raw := serve(app, app.TestRequest(t, "owner@example.com", http.MethodGet,
936	"/d/"+doc.Sqid+"/updates?since=1&raw=1", nil))
937
938	if snap.Body.Len() >= raw.Body.Len() {
939	t.Errorf("snapshot response %d bytes, raw log %d — the snapshot bought nothing",
940	snap.Body.Len(), raw.Body.Len())
941	}
942	if got := docsSeq(t, snap.Result()); got != int64(snapshotCadence+3) {
943	t.Errorf("Docs-Seq = %d, want %d", got, snapshotCadence+3)
944	}
945	}
946
947	// raw=1 is the escape hatch a browser needs when a snapshot will not
948	// decode: doc_updates rows only, which is where the log's authority is
949	// actually reachable from.
950	func TestRawParameterReturnsOnlyLogRows(t *testing.T) {
951	app, err := NewForTest(t)
952	if err != nil {
953	t.Fatalf("NewForTest: %v", err)
954	}
955	ctx := context.Background()
956	doc := pastCadence(t, app, "Raw")
957
958	raw := serve(app, app.TestRequest(t, "owner@example.com", http.MethodGet,
959	"/d/"+doc.Sqid+"/updates?raw=1", nil))
960	if raw.Code != http.StatusOK {
961	t.Fatalf("GET raw = %d, want 200", raw.Code)
962	}
963
964	var rows []DocUpdate
965	if err := app.Store().db.WithContext(ctx).
966	Where("doc_id = ?", doc.ID).Order("seq ASC").Find(&rows).Error; err != nil {
967	t.Fatalf("load log: %v", err)
968	}
969	var want []byte
970	for _, r := range rows {
971	want = append(want, r.UpdateBin...)
972	}
973	if !bytes.Equal(raw.Body.Bytes(), want) {
974	t.Error("raw=1 did not return exactly the concatenated log")
975	}
976	}
977
978	// The live-sync design's D2 survives snapshots: the header is still
979	// derived from the bytes in hand.
980	func TestDocsSeqStillDerivedWithSnapshots(t *testing.T) {
981	app, err := NewForTest(t)
982	if err != nil {
983	t.Fatalf("NewForTest: %v", err)
984	}
985	doc := pastCadence(t, app, "Header")
986	top := int64(snapshotCadence + 3)
987
988	behind := serve(app, app.TestRequest(t, "owner@example.com", http.MethodGet,
989	"/d/"+doc.Sqid+"/updates?since=2", nil))
990	if got := docsSeq(t, behind.Result()); got != top {
991	t.Errorf("Docs-Seq behind the snapshot = %d, want %d", got, top)
992	}
993
994	ahead := serve(app, app.TestRequest(t, "owner@example.com", http.MethodGet,
995	"/d/"+doc.Sqid+"/updates?since="+strconv.FormatInt(top+5, 10), nil))
996	if ahead.Body.Len() != 0 {
997	t.Errorf("a cursor ahead of the log returned %d bytes, want none", ahead.Body.Len())
998	}
999	if got := docsSeq(t, ahead.Result()); got != top+5 {
1000	t.Errorf("Docs-Seq ahead of the log = %d, want %d (an echo of since)", got, top+5)
1001	}
1002	}
1003	```
1004
1005	- [ ] Step 2: Run and watch them fail.
1006
1007	- [ ] Step 3: Implement. In `updatesGet`: parse `raw`; when not raw, read `newestSnapshot`, and if it exists and `snapshot.Seq > since`, write the snapshot's `State` first and then rows with `seq > snapshot.Seq`; otherwise behave exactly as today. `Docs-Seq` remains `max(since, highest seq in the bytes actually written)` — the snapshot's own `seq` column counts, because it travels with bytes that contain its whole prefix. Update the handler's doc comment: it no longer returns the raw log.
1008
1009	- [ ] Step 4: Run the full suite. Commit.
1010
1011	---
1012
1013	### Task 7: The browser can escape a corrupt snapshot
1014
1015	Files:
1016	- Modify: `fixtures/live-updates.mjs`, `fixtures/live-updates.test.mjs`, `fixtures/editor-entry.js`
1017	- Rebuild: `internal/docs/static/editor.<hash>.js`
1018
1019	- [ ] Step 1: Write the failing test
1020
1021	```js
1022	test('a decode failure retries once through the raw log, then gives up', async () => {
1023	const { live, stream, fetches, applied } = build({
1024	onUpdates: () => { throw new Error('bad update') },
1025	})
1026	live.start()
1027	stream.emit(1)
1028
1029	fetches.calls[0].settle({ bytes: new Uint8Array([1]), seq: 1 })
1030	await Promise.resolve(); await Promise.resolve()
1031
1032	assert.equal(fetches.calls.length, 2, 'no retry after a decode failure')
1033	assert.equal(fetches.calls[1].raw, true, 'the retry did not ask for the raw log')
1034	assert.equal(live.cursor, 0, 'the cursor advanced over bytes that never applied')
1035	})
1036	```
1037
1038	`fakeFetches` must record the `raw` option; extend it rather than adding a second fake.
1039
1040	- [ ] Step 2: Run and watch it fail.
1041
1042	- [ ] Step 3: Implement. `fetchSince(since, { raw })`; in `drain`, wrap the `onUpdates` call so a throw retries the same cursor once with `raw: true`, and a second throw reports through `onError` and leaves the cursor alone. Document why: a snapshot corrupted on disk after it was verified is invisible to the server, which serves the blob without a core call, so the browser is the only reader that can detect it and the only one that needs a way out.
1043
1044	- [ ] Step 4: Wire the same fallback into `editor-entry.js`'s initial load, which fetches before any subscriber exists.
1045
1046	- [ ] Step 5: Rebuild the bundle — `cd fixtures && node build-bundles.mjs` — and commit the result. Do not hand-edit it.
1047
1048	- [ ] Step 6: Run `node --test live-updates.test.mjs`, then the gate. Commit.
1049
1050	---
1051
1052	### Task 8: It actually works
1053
1054	- [ ] Step 1: `.amadan/ci.d/10-gate`, exit 0.
1055	- [ ] Step 2: A document past the cadence still edits in a real browser. Drive the editor, type past a snapshot boundary, reload, and confirm the content survives — the one check that exercises snapshot, tail, and the client together.
1056	- [ ] Step 3: Measure what this bought. Time a replay of a document with several thousand updates before and after. The number belongs in the branch discussion; it is the evidence the cadence is right.
1057