amadan
Plan 6: snapshots
Eight tasks, each test-first. The Rust core learns to encode a whole document as one update and to check a candidate against the document it came from; the store takes that snapshot from the replayed state before applying an incoming update, verifies it, and writes it every 256 updates, deleting the one it supersedes; replay and the update endpoint both read snapshot-then-tail; and the browser gets one escape hatch for a snapshot that will not decode. The plan spells out the two mistakes the spec review caught, in the tasks where they would be made rather than only in the spec. Task 1 names the exact yrs call and both neighbours that look right and are not. Task 5 carries the placement argument inline, because "snapshot after a successful apply" is the natural thing to write and it is wrong twice: it can record a state missing a concurrently committed row, and it never rescues a document already at the cap. Self-review against the spec found three gaps in the first draft, now closed: the corrupt-snapshot fallback had no test, the cap regression had no test that drove the real path past 4096, and Task 6's tests were sketched rather than written.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 file changed,
+1056
−0
docs/superpowers/plans/2026-09-01-snapshots.md+1056 −0
CI passed — run details
diff --git a/docs/superpowers/plans/2026-09-01-snapshots.md b/docs/superpowers/plans/2026-09-01-snapshots.md| new file mode 100644 |
| index 0000000..2851f6b |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-09-01-snapshots.md |
| @@ -0,0 +1,1056 @@ |
| +# Snapshots Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** A document that has been edited past `maxRetainedUpdates` keeps accepting edits, and replaying one costs a snapshot plus a short tail instead of its whole history. |
| + |
| +**Architecture:** The Rust core gains a state export — one Yjs update encoding the entire document — and a verifier that checks a candidate against the document it came from without leaving the core. `Store.ApplyUpdate` takes that snapshot from the *replayed* state before applying the incoming update, verifies it, and writes it to a new `doc_snapshots` table every 256 updates, deleting the one it supersedes. `Store.replay` and `GET /d/{sqid}/updates` both read snapshot-then-tail; a snapshot is never authoritative, so anything unreadable falls back to a full replay of the log. |
| + |
| +**Tech Stack:** Rust `yrs` 0.23.5 compiled to wasm32 and run under wazero, Go 1.26, SQLite through GORM, JS `yjs` 13.6.27 unchanged. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-09-01-snapshots-design.md`. Read it before Task 1 — every decision below argues from one of its D-numbers, and the two most dangerous mistakes in this work (where the snapshot is taken from, and which `yrs` call encodes it) are both spec decisions rather than implementation choices. |
| + |
| +## Global Constraints |
| + |
| +- `CGO_ENABLED=0`. One static binary. No new Go module requirements. |
| +- **No entropy from the CRDT library.** `Doc::new`, `Options::with_client_id` and friends are banned by clippy `disallowed-methods`. A new `Doc` takes both a guid and a client id from its caller — including the throwaway document Task 2's verifier builds. |
| +- The core module must have **zero host imports** (`core/tests/no_host_imports.rs`). |
| +- `internal/core`'s ABI check is exact-match: the wasm artifact and the host move together, and a stale artifact fails at load. Rebuild it after any Rust change. |
| +- **Never call `sessions.UserID`; never use `scope.Owned`.** The AST guard in `internal/lint` fails the build. |
| +- `Store.ApplyUpdate`'s owner re-derivation and its MAX(seq)+INSERT transaction are not to be weakened, reordered, or moved. |
| +- The fixtures in `fixtures/` are the contract, not an output. |
| +- Run `.amadan/ci.d/10-gate` before reporting any task complete. Exit 0 is green. |
| +- **Environment:** `GOCACHE` must point somewhere writable (`$TMPDIR/go-build`); the default is read-only under the agent sandbox and fails as "setup failed", which reads like a build break and is not one. The Rust toolchain is at `~/.cargo/bin`, not on a default PATH. A fresh worktree needs `npm ci` in `fixtures/`. |
| + |
| +## File Structure |
| + |
| +| File | Responsibility | |
| +|---|---| |
| +| `core/src/snapshot.rs` *(new)* | Encode a document's whole state as one update; verify a candidate against a live document. The only place that knows either operation. | |
| +| `core/src/lib.rs` | Re-export `snapshot_bytes` / `verify_snapshot`. | |
| +| `core/src/abi.rs` | `docs_core_snapshot_len` / `_read`, `docs_core_verify_snapshot`; ABI 3 → 4. | |
| +| `core/tests/snapshot.rs` *(new)* | Round-trip, merge-equivalence, and verifier tests in Rust. | |
| +| `internal/core/core.go` | `abiVersion` 3 → 4; `(*Handle).Snapshot`, `(*Handle).VerifySnapshot`. | |
| +| `internal/docs/migrations/0004_doc_snapshots.sql` *(new)* | The table. | |
| +| `internal/docs/models.go` | `DocSnapshot`. | |
| +| `internal/docs/snapshots.go` *(new)* | Read the newest usable snapshot; write one and supersede its predecessor. The only file that knows the cadence. | |
| +| `internal/docs/store.go` | `replay` reads snapshot-then-tail; `ApplyUpdate` takes and writes the snapshot. | |
| +| `internal/docs/handlers.go` | `updatesGet` serves snapshot-then-tail, and `?raw=1`. | |
| +| `fixtures/live-updates.mjs` | One retry through `?raw=1` when applying fetched bytes throws. | |
| +| `fixtures/editor-entry.js` | Pass the raw flag through; same fallback on the initial load. | |
| + |
| +--- |
| + |
| +### Task 1: The core can encode a document's whole state |
| + |
| +Without this there is nothing to store. The danger here is picking the wrong `yrs` call: two neighbours look right and are not (spec §3). |
| + |
| +**Files:** |
| +- Create: `core/src/snapshot.rs`, `core/tests/snapshot.rs` |
| +- Modify: `core/src/lib.rs`, `core/src/abi.rs` |
| + |
| +**Interfaces:** |
| +- Produces: `pub fn snapshot_bytes(doc: &Doc) -> Vec<u8>` — the whole state as one update. |
| +- Produces ABI: `docs_core_snapshot_len(handle) -> i32`, `docs_core_snapshot_read(handle, out_ptr, out_len) -> i32`, same convention and error codes as the existing `text`/`html`/`json` pairs. |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +`core/tests/snapshot.rs`: |
| + |
| +```rust |
| +use yrs::{Doc, Options, ReadTxn, Transact, XmlFragmentPrelim, GetString}; |
| +use docs_core::{snapshot_bytes, fragment_text}; |
| + |
| +fn doc_with(client_id: u64) -> Doc { |
| + Doc::with_options(Options::with_guid_and_client_id("test-guid".into(), client_id)) |
| +} |
| + |
| +#[test] |
| +fn a_snapshot_replayed_into_an_empty_doc_reproduces_the_document() { |
| + let source = doc_with(1); |
| + let update = std::fs::read("../fixtures/vectors/fragment/simple.bin").unwrap(); |
| + source.transact_mut().apply_update(yrs::Update::decode_v1(&update).unwrap()).unwrap(); |
| + |
| + let snap = snapshot_bytes(&source); |
| + |
| + let rebuilt = doc_with(2); |
| + rebuilt.transact_mut().apply_update(yrs::Update::decode_v1(&snap).unwrap()).unwrap(); |
| + |
| + assert_eq!(fragment_text(&rebuilt), fragment_text(&source)); |
| + assert!(!fragment_text(&rebuilt).is_empty(), "fixture produced no text"); |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run it and watch it fail** |
| + |
| +Run: `PATH="$HOME/.cargo/bin:$PATH" cargo test --manifest-path core/Cargo.toml --test snapshot` |
| +Expected: FAIL — `snapshot_bytes` is not defined. |
| + |
| +- [ ] **Step 3: Implement `snapshot_bytes`** |
| + |
| +`core/src/snapshot.rs`: |
| + |
| +```rust |
| +//! Encoding a whole document as one update, and checking that the |
| +//! result is the document it claims to be. |
| +//! |
| +//! The call below is `encode_state_as_update_v1` against an EMPTY state |
| +//! vector, and which call this is matters more than anything else in |
| +//! this file. Against the empty vector it encodes every block, the |
| +//! entire delete set, and — through yrs's own `merge_pending_v1` — the |
| +//! pending queue, meaning updates whose dependencies never arrived and |
| +//! which `apply_update` accepted and parked. That makes the result |
| +//! merge-equivalent to the log it replaces, not merely |
| +//! render-equivalent: every clock range and every tombstone survives, |
| +//! so a later concurrent merge integrates against a snapshot exactly as |
| +//! it would against the full history. |
| +//! |
| +//! Two neighbours are traps, and both look like this one: |
| +//! |
| +//! - `encode_diff_v1(&StateVector::default())` silently drops the |
| +//! pending queue (its own doc comment says so). A document holding an |
| +//! update whose dependency arrives later heals under a full replay and |
| +//! does NOT heal under a diff-built snapshot — and no render |
| +//! comparison can see the difference, because pending content is |
| +//! invisible in both documents. |
| +//! - yrs's own `Snapshot` type and `encode_state_from_snapshot` are a |
| +//! different feature (historical views, requiring `skip_gc`) that |
| +//! happens to share this module's name. |
| +use yrs::{Doc, ReadTxn, StateVector, Transact}; |
| + |
| +/// The whole document, as one update. |
| +pub fn snapshot_bytes(doc: &Doc) -> Vec<u8> { |
| + doc.transact().encode_state_as_update_v1(&StateVector::default()) |
| +} |
| +``` |
| + |
| +Add to `core/src/lib.rs`: |
| + |
| +```rust |
| +mod snapshot; |
| +pub use snapshot::snapshot_bytes; |
| +``` |
| + |
| +- [ ] **Step 4: Run the test and watch it pass** |
| + |
| +Run: `PATH="$HOME/.cargo/bin:$PATH" cargo test --manifest-path core/Cargo.toml --test snapshot` |
| +Expected: PASS. |
| + |
| +- [ ] **Step 5: Add the ABI pair** |
| + |
| +In `core/src/abi.rs`, beside `json_string`: |
| + |
| +```rust |
| +/// The whole document as one update. Bytes, not a string — the host |
| +/// reads it through the same len/read convention as the three |
| +/// renderers, which is why this is shaped like them. |
| +pub fn snapshot_vec(handle: u32) -> Option<Vec<u8>> { |
| + get_doc(handle).map(|doc| snapshot_bytes(&doc)) |
| +} |
| +``` |
| + |
| +and the two exports, copying the `docs_core_json_len` / `docs_core_json_read` bodies exactly, substituting `snapshot_vec` for `json_string` and `s.as_ptr()` reading from the `Vec<u8>`: |
| + |
| +```rust |
| +#[no_mangle] |
| +pub extern "C" fn docs_core_snapshot_len(handle: u32) -> i32 { |
| + match snapshot_vec(handle) { |
| + Some(v) => v.len().try_into().unwrap_or(i32::MAX), |
| + None => ERR_NO_DOC, |
| + } |
| +} |
| + |
| +/// # Safety |
| +/// |
| +/// `out_ptr` and `out_len` must describe a byte range that lies entirely |
| +/// within this module's linear memory and is valid for writes for the |
| +/// duration of this call. |
| +#[no_mangle] |
| +pub unsafe extern "C" fn docs_core_snapshot_read(handle: u32, out_ptr: u32, out_len: u32) -> i32 { |
| + let v = match snapshot_vec(handle) { |
| + Some(v) => v, |
| + None => return ERR_NO_DOC, |
| + }; |
| + if v.len() > out_len as usize { |
| + return ERR_BUFFER_TOO_SMALL; |
| + } |
| + if !in_bounds(out_ptr, out_len) { |
| + return ERR_INVALID_LENGTH; |
| + } |
| + std::ptr::copy_nonoverlapping(v.as_ptr(), out_ptr as *mut u8, v.len()); |
| + v.len().try_into().unwrap_or(i32::MAX) |
| +} |
| +``` |
| + |
| +- [ ] **Step 6: Bump the ABI version to 4** in `core/src/abi.rs`, and leave the existing exports untouched. |
| + |
| +- [ ] **Step 7: Build the artifact and run the Rust suite** |
| + |
| +Run: `PATH="$HOME/.cargo/bin:$PATH" ./scripts/build-wasm.sh && PATH="$HOME/.cargo/bin:$PATH" DOCS_REQUIRE_WASM=1 cargo test --manifest-path core/Cargo.toml` |
| +Expected: PASS, including `no_host_imports`. |
| + |
| +- [ ] **Step 8: Commit** — `git add core/ && git commit` |
| + |
| +--- |
| + |
| +### Task 2: The core can tell a good snapshot from a bad one |
| + |
| +Spec D5. This runs inside the core so the comparison is against Rust structs rather than encoded bytes, whose encoding is not canonical. |
| + |
| +**Files:** |
| +- Modify: `core/src/snapshot.rs`, `core/src/abi.rs`, `core/src/lib.rs` |
| +- Test: `core/tests/snapshot.rs` |
| + |
| +**Interfaces:** |
| +- Produces: `pub fn verify_snapshot(doc: &Doc, candidate: &[u8], guid: &str, client_id: u64) -> bool` |
| +- Produces ABI: `docs_core_verify_snapshot(handle, ptr, len) -> i32` — `0` match, `1` mismatch, `ERR_DECODE` undecodable, `ERR_NO_DOC` unknown handle. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +Append to `core/tests/snapshot.rs`: |
| + |
| +```rust |
| +#[test] |
| +fn a_snapshot_of_this_document_verifies() { |
| + let source = doc_with(1); |
| + let update = std::fs::read("../fixtures/vectors/fragment/simple.bin").unwrap(); |
| + source.transact_mut().apply_update(yrs::Update::decode_v1(&update).unwrap()).unwrap(); |
| + let snap = snapshot_bytes(&source); |
| + assert!(docs_core::verify_snapshot(&source, &snap, "verify-guid", 99)); |
| +} |
| + |
| +#[test] |
| +fn a_snapshot_of_a_different_document_does_not_verify() { |
| + let source = doc_with(1); |
| + source.transact_mut() |
| + .apply_update(yrs::Update::decode_v1( |
| + &std::fs::read("../fixtures/vectors/fragment/simple.bin").unwrap()).unwrap()).unwrap(); |
| + |
| + let other = doc_with(1); |
| + other.transact_mut() |
| + .apply_update(yrs::Update::decode_v1( |
| + &std::fs::read("../fixtures/vectors/fragment/marks.bin").unwrap()).unwrap()).unwrap(); |
| + let wrong = snapshot_bytes(&other); |
| + |
| + assert!(!docs_core::verify_snapshot(&source, &wrong, "verify-guid", 99)); |
| +} |
| + |
| +#[test] |
| +fn garbage_does_not_verify() { |
| + let source = doc_with(1); |
| + assert!(!docs_core::verify_snapshot(&source, &[0xff, 0x00, 0x13, 0x37], "verify-guid", 99)); |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run and watch all three fail** (`verify_snapshot` undefined). |
| + |
| +- [ ] **Step 3: Implement it** |
| + |
| +Append to `core/src/snapshot.rs`: |
| + |
| +```rust |
| +/// True when `candidate`, replayed into an empty document, is the same |
| +/// document as `doc`. |
| +/// |
| +/// Three comparisons, none of them on encoded bytes: update encoding is |
| +/// not canonical (merge boundaries vary), so byte equality would report |
| +/// false mismatches and no cheap byte-level proof exists. State vector |
| +/// equality says both documents saw the same clock ranges from the same |
| +/// clients; delete set equality says they agree on every deletion; |
| +/// `fragment_json` equality says they render to the same structure, |
| +/// which catches formatting-only divergence that a text comparison |
| +/// misses at identical cost. |
| +/// |
| +/// This is deliberately not a proof. What survives all three is "yrs |
| +/// corrupted content while preserving every clock range, every deletion |
| +/// and the full structural render", and that is a narrow enough residual |
| +/// to accept. |
| +/// |
| +/// What it CANNOT catch, stated because the temptation is to believe it |
| +/// can: a snapshot taken from a stale state verifies perfectly, because |
| +/// it is compared against the very document that produced it. That |
| +/// failure is closed by taking the snapshot from the replayed state |
| +/// (the spec's D3), not here. |
| +/// |
| +/// guid/client_id are the caller's, never generated here — the clippy |
| +/// disallowed-methods rule against drawing entropy from the CRDT library |
| +/// applies to this throwaway document exactly as it does to a real one. |
| +pub fn verify_snapshot(doc: &Doc, candidate: &[u8], guid: &str, client_id: u64) -> bool { |
| + let update = match yrs::Update::decode_v1(candidate) { |
| + Ok(u) => u, |
| + Err(_) => return false, |
| + }; |
| + let rebuilt = Doc::with_options(Options::with_guid_and_client_id(guid.into(), client_id)); |
| + { |
| + let mut txn = rebuilt.transact_mut(); |
| + if txn.apply_update(update).is_err() { |
| + return false; |
| + } |
| + } |
| + let (a, b) = (doc.transact(), rebuilt.transact()); |
| + a.state_vector() == b.state_vector() |
| + && a.delete_set() == b.delete_set() |
| + && crate::fragment_json(doc) == crate::fragment_json(&rebuilt) |
| +} |
| +``` |
| + |
| +Export it from `lib.rs` (`pub use snapshot::{snapshot_bytes, verify_snapshot};`). |
| + |
| +- [ ] **Step 4: Run the tests and watch them pass.** |
| + |
| +- [ ] **Step 5: Add the ABI export** |
| + |
| +```rust |
| +/// 0 = the candidate is this document, 1 = it is not, ERR_DECODE = the |
| +/// bytes are not an update at all, ERR_NO_DOC = no such handle. |
| +/// |
| +/// # Safety |
| +/// |
| +/// `ptr`/`len` must describe a readable byte range inside this module's |
| +/// linear memory. |
| +#[no_mangle] |
| +pub unsafe extern "C" fn docs_core_verify_snapshot(handle: u32, ptr: u32, len: u32) -> i32 { |
| + if !in_bounds(ptr, len) { |
| + return ERR_INVALID_LENGTH; |
| + } |
| + let candidate = std::slice::from_raw_parts(ptr as *const u8, len as usize); |
| + match get_doc(handle) { |
| + None => ERR_NO_DOC, |
| + Some(doc) => { |
| + // The verifier's throwaway document takes a fixed guid and |
| + // client id rather than generating any: entropy inside the |
| + // core is banned (clippy disallowed-methods), and this |
| + // document is discarded before the call returns, so a fixed |
| + // pair is not merely acceptable but correct. |
| + if verify_snapshot(&doc, candidate, "docs-core-verify", 1) { 0 } else { 1 } |
| + } |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 6: Rebuild the artifact, run the Rust suite, commit.** |
| + |
| +--- |
| + |
| +### Task 3: The Go host can take and check a snapshot |
| + |
| +**Files:** |
| +- Modify: `internal/core/core.go` |
| +- Test: `internal/core/core_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `func (h *Handle) Snapshot(ctx context.Context) ([]byte, error)` |
| +- Produces: `func (h *Handle) VerifySnapshot(ctx context.Context, candidate []byte) (bool, error)` |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +```go |
| +func TestSnapshotRoundTripsThroughAFreshHandle(t *testing.T) { |
| + rt := newTestRuntime(t) |
| + ctx := context.Background() |
| + |
| + h, err := rt.Open(ctx, "guid-a", 1) |
| + if err != nil { |
| + t.Fatalf("Open: %v", err) |
| + } |
| + defer h.Close(ctx) |
| + if err := h.Apply(ctx, fragmentFixture(t, "simple")); err != nil { |
| + t.Fatalf("Apply: %v", err) |
| + } |
| + want, err := h.Text(ctx) |
| + if err != nil { |
| + t.Fatalf("Text: %v", err) |
| + } |
| + |
| + snap, err := h.Snapshot(ctx) |
| + if err != nil { |
| + t.Fatalf("Snapshot: %v", err) |
| + } |
| + if len(snap) == 0 { |
| + t.Fatal("Snapshot returned no bytes") |
| + } |
| + |
| + fresh, err := rt.Open(ctx, "guid-b", 2) |
| + if err != nil { |
| + t.Fatalf("Open fresh: %v", err) |
| + } |
| + defer fresh.Close(ctx) |
| + if err := fresh.Apply(ctx, snap); err != nil { |
| + t.Fatalf("Apply snapshot: %v", err) |
| + } |
| + got, err := fresh.Text(ctx) |
| + if err != nil { |
| + t.Fatalf("Text: %v", err) |
| + } |
| + if got != want { |
| + t.Errorf("snapshot rebuilt %q, want %q", got, want) |
| + } |
| +} |
| + |
| +func TestVerifySnapshotAcceptsItsOwnAndRejectsAnother(t *testing.T) { |
| + rt := newTestRuntime(t) |
| + ctx := context.Background() |
| + h, _ := rt.Open(ctx, "guid-a", 1) |
| + defer h.Close(ctx) |
| + if err := h.Apply(ctx, fragmentFixture(t, "simple")); err != nil { |
| + t.Fatalf("Apply: %v", err) |
| + } |
| + snap, err := h.Snapshot(ctx) |
| + if err != nil { |
| + t.Fatalf("Snapshot: %v", err) |
| + } |
| + |
| + ok, err := h.VerifySnapshot(ctx, snap) |
| + if err != nil { |
| + t.Fatalf("VerifySnapshot: %v", err) |
| + } |
| + if !ok { |
| + t.Error("a document refused its own snapshot") |
| + } |
| + |
| + if ok, err := h.VerifySnapshot(ctx, []byte{0xff, 0x00}); err != nil || ok { |
| + t.Errorf("garbage verified: ok=%v err=%v", ok, err) |
| + } |
| +} |
| +``` |
| + |
| +Match the existing helpers in `core_test.go` for building a runtime and reading a fragment fixture; if they are named differently, use the existing names rather than adding new ones. |
| + |
| +- [ ] **Step 2: Run and watch it fail.** |
| + |
| +- [ ] **Step 3: Bump `abiVersion` to 4 and implement** |
| + |
| +```go |
| +// Snapshot returns the document's entire state as one Yjs update — what |
| +// doc_snapshots stores so a later replay can start from it instead of |
| +// from the whole log (the snapshots design, D1). |
| +// |
| +// Bytes rather than a string, but it rides the same len/read machinery |
| +// as Text/HTML/JSON, including their trap-and-retry: a Go string is a |
| +// byte sequence and nothing here interprets it, so "snapshot" is simply |
| +// a fourth renderer name to readOn. |
| +func (h *Handle) Snapshot(ctx context.Context) ([]byte, error) { |
| + s, err := h.read(ctx, "snapshot") |
| + if err != nil { |
| + return nil, err |
| + } |
| + return []byte(s), nil |
| +} |
| + |
| +// VerifySnapshot asks the core whether candidate, replayed into an empty |
| +// document, is this document — state vector, delete set and structural |
| +// render (the snapshots design, D5). It is a smoke test rather than a |
| +// proof, and it cannot detect a snapshot taken from a stale state; see |
| +// core/src/snapshot.rs for both limits. |
| +func (h *Handle) VerifySnapshot(ctx context.Context, candidate []byte) (bool, error) { |
| + // ... allocate candidate into guest memory exactly as Apply does, |
| + // call docs_core_verify_snapshot, free, and map: 0 -> true, 1 -> |
| + // false, ERR_DECODE -> false with no error (bad bytes are an answer, |
| + // not a failure), anything else -> error. |
| +} |
| +``` |
| + |
| +Write `VerifySnapshot`'s body by copying `Apply`'s allocate/call/free structure — it is the only existing export that takes a byte buffer, and its guest-memory handling, trap taxonomy and retry are what this needs too. |
| + |
| +- [ ] **Step 4: Rebuild the artifact, run `go test ./internal/core/`, commit.** |
| + |
| +--- |
| + |
| +### Task 4: The table, and reading the newest usable snapshot |
| + |
| +**Files:** |
| +- Create: `internal/docs/migrations/0004_doc_snapshots.sql`, `internal/docs/snapshots.go` |
| +- Modify: `internal/docs/models.go` |
| +- Test: `internal/docs/snapshots_test.go` *(new)* |
| + |
| +**Interfaces:** |
| +- Produces: `type DocSnapshot struct { ID, DocID, Seq int64; State []byte; ABIVersion uint32; CreatedAt time.Time }` |
| +- Produces: `func (s *Store) newestSnapshot(ctx context.Context, docID int64) (*DocSnapshot, error)` — nil, nil when there is none usable. |
| +- Produces: `const snapshotCadence = 256` |
| + |
| +- [ ] **Step 1: Write the migration** |
| + |
| +`0004_doc_snapshots.sql`: |
| + |
| +```sql |
| +-- doc_snapshots holds one Yjs update per row encoding a document's |
| +-- entire state as of `seq` (the snapshots design, D2). It is an |
| +-- accelerator, never the source of truth: doc_updates remains complete |
| +-- and authoritative, and a row here that cannot be read is discarded in |
| +-- favour of replaying the log. |
| +-- |
| +-- (doc_id, seq) is deliberately NOT unique. Two concurrent POSTs at the |
| +-- cadence boundary can both snapshot the same replayed prefix, and those |
| +-- rows are equivalent by construction, so a collision is not an error to |
| +-- raise. "Newest usable" resolves as highest seq, then highest id. |
| +-- |
| +-- abi_version records what produced the row, which is what makes |
| +-- discarding a snapshot from an older core implementable rather than |
| +-- merely promised. |
| +-- |
| +-- Additive: creates a table, touches no existing row. |
| +CREATE TABLE IF NOT EXISTS doc_snapshots ( |
| + id INTEGER PRIMARY KEY AUTOINCREMENT, |
| + doc_id INTEGER NOT NULL, |
| + seq INTEGER NOT NULL, |
| + state BLOB NOT NULL, |
| + abi_version INTEGER NOT NULL, |
| + created_at DATETIME NOT NULL |
| +); |
| + |
| +CREATE INDEX IF NOT EXISTS idx_doc_snapshots_doc_seq |
| + ON doc_snapshots (doc_id, seq DESC, id DESC); |
| +``` |
| + |
| +- [ ] **Step 2: Write the failing test** |
| + |
| +`internal/docs/snapshots_test.go`: |
| + |
| +```go |
| +func TestNewestSnapshotPrefersHighestSeqThenHighestID(t *testing.T) { |
| + app, err := NewForTest(t) |
| + if err != nil { |
| + t.Fatalf("NewForTest: %v", err) |
| + } |
| + ctx := context.Background() |
| + owner := app.TestPerson(t, "owner@example.com") |
| + doc, _ := app.Store().Create(ctx, owner, "Snapshots") |
| + |
| + write := func(seq int64, state string, abi uint32) { |
| + row := &DocSnapshot{DocID: doc.ID, Seq: seq, State: []byte(state), ABIVersion: abi, CreatedAt: time.Now().UTC()} |
| + if err := app.Store().db.WithContext(ctx).Create(row).Error; err != nil { |
| + t.Fatalf("insert snapshot: %v", err) |
| + } |
| + } |
| + write(10, "older", coreABIVersion) |
| + write(20, "first-at-20", coreABIVersion) |
| + write(20, "second-at-20", coreABIVersion) |
| + |
| + got, err := app.Store().newestSnapshot(ctx, doc.ID) |
| + if err != nil { |
| + t.Fatalf("newestSnapshot: %v", err) |
| + } |
| + if got == nil || string(got.State) != "second-at-20" { |
| + t.Fatalf("newestSnapshot returned %+v, want the second row at seq 20", got) |
| + } |
| +} |
| + |
| +func TestNewestSnapshotIgnoresAnotherABIsRows(t *testing.T) { |
| + app, _ := NewForTest(t) |
| + ctx := context.Background() |
| + owner := app.TestPerson(t, "owner@example.com") |
| + doc, _ := app.Store().Create(ctx, owner, "Old ABI") |
| + |
| + row := &DocSnapshot{DocID: doc.ID, Seq: 5, State: []byte("from an older core"), ABIVersion: coreABIVersion - 1, CreatedAt: time.Now().UTC()} |
| + if err := app.Store().db.WithContext(ctx).Create(row).Error; err != nil { |
| + t.Fatalf("insert: %v", err) |
| + } |
| + |
| + got, err := app.Store().newestSnapshot(ctx, doc.ID) |
| + if err != nil { |
| + t.Fatalf("newestSnapshot: %v", err) |
| + } |
| + if got != nil { |
| + t.Error("a snapshot from another ABI was treated as usable") |
| + } |
| +} |
| + |
| +func TestNewestSnapshotIsNilWhenThereAreNone(t *testing.T) { |
| + app, _ := NewForTest(t) |
| + ctx := context.Background() |
| + owner := app.TestPerson(t, "owner@example.com") |
| + doc, _ := app.Store().Create(ctx, owner, "None") |
| + |
| + got, err := app.Store().newestSnapshot(ctx, doc.ID) |
| + if err != nil || got != nil { |
| + t.Errorf("newestSnapshot = (%v, %v), want (nil, nil)", got, err) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 3: Run and watch them fail.** |
| + |
| +- [ ] **Step 4: Add the model and the reader** |
| + |
| +`models.go`: |
| + |
| +```go |
| +// DocSnapshot is one document's entire state encoded as a single Yjs |
| +// update, as of Seq. It is an accelerator for replay and for the update |
| +// endpoint, never the source of truth — doc_updates keeps every row, and |
| +// a snapshot that cannot be read is discarded in favour of replaying |
| +// them (the snapshots design, D2). |
| +// |
| +// State is plaintext in this table, exactly as DocUpdate.UpdateBin is. |
| +// Worth one caveat the sealing work should carry: a doc_updates row is a |
| +// fragment of an edit, while this is the entire document in one blob. |
| +// |
| +// ABIVersion records the core that produced State. Without it, "discard |
| +// a snapshot from an older ABI" is unenforceable prose. |
| +type DocSnapshot struct { |
| + ID int64 `gorm:"primaryKey"` |
| + DocID int64 `gorm:"column:doc_id;not null;index"` |
| + Seq int64 `gorm:"column:seq;not null"` |
| + |
| + State []byte `gorm:"column:state;not null"` |
| + ABIVersion uint32 `gorm:"column:abi_version;not null"` |
| + |
| + CreatedAt time.Time `gorm:"column:created_at;not null"` |
| +} |
| +``` |
| + |
| +`snapshots.go`: |
| + |
| +```go |
| +package docs |
| + |
| +// snapshotCadence is how many updates may accumulate past a snapshot |
| +// before the next one is taken. 256 keeps a replay to a snapshot plus a |
| +// short tail, which is what holds the retained-update count in a |
| +// core.Handle clear of maxRetainedUpdates (4096) — about 16x of headroom. |
| +// |
| +// That headroom is conditional, and the condition is worth knowing: |
| +// it holds while snapshot writes keep succeeding. Roughly fifteen |
| +// consecutive failures — each logged at Error, so noisy rather than |
| +// silent — walk the tail back to the cap. |
| +const snapshotCadence = 256 |
| + |
| +// newestSnapshot returns the most recent usable snapshot for docID, or |
| +// nil when there is none. |
| +// |
| +// "Usable" excludes rows written by a different core ABI: the state |
| +// encoding is that core's business, and a row from another one is |
| +// discarded rather than trusted. Ordering is seq DESC, id DESC — two |
| +// concurrent POSTs at the cadence boundary may legitimately write |
| +// equivalent snapshots at the same seq, so the tie resolves to the last |
| +// row written rather than being treated as an error. |
| +func (s *Store) newestSnapshot(ctx context.Context, docID int64) (*DocSnapshot, error) { |
| + var row DocSnapshot |
| + err := s.db.WithContext(ctx). |
| + Where("doc_id = ? AND abi_version = ?", docID, coreABIVersion). |
| + Order("seq DESC, id DESC"). |
| + First(&row).Error |
| + if errors.Is(err, gorm.ErrRecordNotFound) { |
| + return nil, nil |
| + } |
| + if err != nil { |
| + return nil, fmt.Errorf("docs: newest snapshot for document %d: %w", docID, err) |
| + } |
| + return &row, nil |
| +} |
| +``` |
| + |
| +Export the ABI version from `internal/core` as `core.ABIVersion` and alias it here as `coreABIVersion`, rather than duplicating the number. |
| + |
| +- [ ] **Step 5: Register the migration** in `models.go`'s `Schema` — it is `migrate.MustFromFS(migrationFS, "docs")` over `migrations/*.sql`, so the new file is picked up by its number with no code change. Confirm by running the tests. |
| + |
| +- [ ] **Step 6: Run tests, then commit.** |
| + |
| +--- |
| + |
| +### Task 5: Replay reads snapshot-then-tail, and writes one at the cadence |
| + |
| +The heart of it, and the task carrying both defects the spec review found. |
| + |
| +**Files:** |
| +- Modify: `internal/docs/store.go`, `internal/docs/snapshots.go` |
| +- Test: `internal/docs/snapshots_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `func (s *Store) writeSnapshot(ctx context.Context, doc *Document, h *core.Handle, seq int64) error` |
| +- Changes: `replay` starts from the newest usable snapshot; `ApplyUpdate` takes the snapshot from the replayed state **before** applying the incoming update. |
| + |
| +- [ ] **Step 1: Write the failing tests — equivalence, the race, and the capped document** |
| + |
| +```go |
| +// The central property: a document replayed through snapshot-then-tail |
| +// must render identically to one replayed from the whole log. |
| +func TestSnapshotThenTailRendersIdenticallyToAFullReplay(t *testing.T) { |
| + app, _ := NewForTest(t) |
| + ctx := context.Background() |
| + owner := app.TestPerson(t, "owner@example.com") |
| + doc, _ := app.Store().Create(ctx, owner, "Equivalence") |
| + |
| + update := validUpdate(t) |
| + for i := 0; i < snapshotCadence+5; i++ { |
| + if _, err := app.Store().ApplyUpdate(ctx, doc, update); err != nil { |
| + t.Fatalf("ApplyUpdate %d: %v", i, err) |
| + } |
| + } |
| + withSnapshot, err := app.Store().HTML(ctx, doc) |
| + if err != nil { |
| + t.Fatalf("HTML: %v", err) |
| + } |
| + |
| + // Delete every snapshot and read again: same document, full replay. |
| + if err := app.Store().db.WithContext(ctx). |
| + Where("doc_id = ?", doc.ID).Delete(&DocSnapshot{}).Error; err != nil { |
| + t.Fatalf("delete snapshots: %v", err) |
| + } |
| + fullReplay, err := app.Store().HTML(ctx, doc) |
| + if err != nil { |
| + t.Fatalf("HTML after deleting snapshots: %v", err) |
| + } |
| + if withSnapshot != fullReplay { |
| + t.Error("snapshot-then-tail and full replay disagree about the document") |
| + } |
| +} |
| + |
| +// The spec's D3, first defect: a snapshot must never claim a seq whose |
| +// content it does not contain. Asserted against the rendered document, |
| +// never against the snapshot row — the row is exactly what a stale |
| +// snapshot makes look right. |
| +func TestConcurrentPostsAtTheCadenceBoundaryDoNotStrandContent(t *testing.T) { |
| + app, _ := NewForTest(t) |
| + ctx := context.Background() |
| + owner := app.TestPerson(t, "owner@example.com") |
| + doc, _ := app.Store().Create(ctx, owner, "Boundary") |
| + |
| + for i := 0; i < snapshotCadence-1; i++ { |
| + if _, err := app.Store().ApplyUpdate(ctx, doc, validUpdate(t)); err != nil { |
| + t.Fatalf("warmup %d: %v", i, err) |
| + } |
| + } |
| + |
| + var wg sync.WaitGroup |
| + for i := 0; i < 8; i++ { |
| + wg.Add(1) |
| + go func() { |
| + defer wg.Done() |
| + _, _ = app.Store().ApplyUpdate(ctx, doc, validUpdate(t)) |
| + }() |
| + } |
| + wg.Wait() |
| + |
| + withSnapshot, err := app.Store().HTML(ctx, doc) |
| + if err != nil { |
| + t.Fatalf("HTML: %v", err) |
| + } |
| + if err := app.Store().db.WithContext(ctx). |
| + Where("doc_id = ?", doc.ID).Delete(&DocSnapshot{}).Error; err != nil { |
| + t.Fatalf("delete snapshots: %v", err) |
| + } |
| + fullReplay, err := app.Store().HTML(ctx, doc) |
| + if err != nil { |
| + t.Fatalf("HTML: %v", err) |
| + } |
| + if withSnapshot != fullReplay { |
| + t.Error("a snapshot written during concurrent writes stranded content the log still holds") |
| + } |
| +} |
| + |
| +// The spec's D3, second defect, and the case section 1 opens with: a |
| +// document that is ALREADY at the cap when this ships. Driven through |
| +// the new path a document never reaches the cap, so this builds the |
| +// production shape directly — a full log and no snapshot — and asserts |
| +// it recovers. |
| +func TestADocumentAlreadyAtTheCapRecovers(t *testing.T) { |
| + app, _ := NewForTest(t) |
| + ctx := context.Background() |
| + owner := app.TestPerson(t, "owner@example.com") |
| + doc, _ := app.Store().Create(ctx, owner, "At the cap") |
| + |
| + // Write rows straight into the log, bypassing ApplyUpdate, so no |
| + // snapshot is ever taken — exactly the state a document edited |
| + // before this feature shipped is in. |
| + update := validUpdate(t) |
| + rows := make([]DocUpdate, 0, core.MaxRetainedUpdates) |
| + for i := 1; i <= core.MaxRetainedUpdates; i++ { |
| + rows = append(rows, DocUpdate{DocID: doc.ID, Seq: int64(i), UpdateBin: update, CreatedAt: time.Now().UTC()}) |
| + } |
| + if err := app.Store().db.WithContext(ctx).CreateInBatches(rows, 500).Error; err != nil { |
| + t.Fatalf("seed a full log: %v", err) |
| + } |
| + |
| + // The first write still fails — the handle is full before the |
| + // snapshot can help — but it must leave a snapshot behind. |
| + if _, err := app.Store().ApplyUpdate(ctx, doc, update); err == nil { |
| + t.Log("first write succeeded; acceptable if the cap was not reached") |
| + } |
| + snap, err := app.Store().newestSnapshot(ctx, doc.ID) |
| + if err != nil { |
| + t.Fatalf("newestSnapshot: %v", err) |
| + } |
| + if snap == nil { |
| + t.Fatal("a capped document got no snapshot, so it can never recover") |
| + } |
| + |
| + // The retry replays snapshot + empty tail and succeeds. |
| + if _, err := app.Store().ApplyUpdate(ctx, doc, update); err != nil { |
| + t.Errorf("retry after a snapshot existed still failed: %v", err) |
| + } |
| +} |
| +``` |
| + |
| +```go |
| +// A corrupted snapshot row must not corrupt the document: the log is |
| +// authoritative, so an unreadable snapshot is discarded and the whole |
| +// log replayed instead. Correct, and merely slower. |
| +func TestACorruptSnapshotFallsBackToTheLog(t *testing.T) { |
| + app, _ := NewForTest(t) |
| + ctx := context.Background() |
| + owner := app.TestPerson(t, "owner@example.com") |
| + doc, _ := app.Store().Create(ctx, owner, "Corrupt") |
| + for i := 0; i < snapshotCadence+2; i++ { |
| + if _, err := app.Store().ApplyUpdate(ctx, doc, validUpdate(t)); err != nil { |
| + t.Fatalf("ApplyUpdate %d: %v", i, err) |
| + } |
| + } |
| + want, err := app.Store().HTML(ctx, doc) |
| + if err != nil { |
| + t.Fatalf("HTML: %v", err) |
| + } |
| + |
| + if err := app.Store().db.WithContext(ctx).Model(&DocSnapshot{}). |
| + Where("doc_id = ?", doc.ID). |
| + Update("state", []byte{0xff, 0x00, 0x13, 0x37}).Error; err != nil { |
| + t.Fatalf("corrupt the snapshot: %v", err) |
| + } |
| + |
| + got, err := app.Store().HTML(ctx, doc) |
| + if err != nil { |
| + t.Fatalf("HTML with a corrupt snapshot: %v", err) |
| + } |
| + if got != want { |
| + t.Error("a corrupt snapshot changed the document instead of being discarded") |
| + } |
| +} |
| + |
| +// The cliff itself, driven the long way round: more updates than |
| +// maxRetainedUpdates through the real path, which must keep accepting |
| +// them. Slow — thousands of wasm applies — so it is skipped under |
| +// -short, but it is the one test that proves the headline claim rather |
| +// than a proxy for it. |
| +func TestADocumentPastTheRetainedCapKeepsAcceptingWrites(t *testing.T) { |
| + if testing.Short() { |
| + t.Skip("thousands of updates; run without -short") |
| + } |
| + app, _ := NewForTest(t) |
| + ctx := context.Background() |
| + owner := app.TestPerson(t, "owner@example.com") |
| + doc, _ := app.Store().Create(ctx, owner, "Past the cap") |
| + |
| + for i := 0; i < core.MaxRetainedUpdates+50; i++ { |
| + if _, err := app.Store().ApplyUpdate(ctx, doc, validUpdate(t)); err != nil { |
| + t.Fatalf("ApplyUpdate %d of %d: %v", i, core.MaxRetainedUpdates+50, err) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +`core.MaxRetainedUpdates` and `core.ABIVersion` must be exported from `internal/core` for these tests and for `snapshots.go`; export them rather than duplicating 4096 and the version number. |
| + |
| +`replay`'s signature changes to return what the caller needs rather than making it ask again: `func (s *Store) replay(ctx context.Context, doc *Document) (h *core.Handle, replayedSeq int64, lastSnapshotSeq int64, err error)`. Both seqs come from rows it has already read — the same no-second-query discipline the live-sync design's D2 sets, for the same reason. |
| + |
| +- [ ] **Step 2: Run all three and watch them fail.** |
| + |
| +- [ ] **Step 3: Make `replay` read snapshot-then-tail** |
| + |
| +In `store.go`'s `replay`, before loading updates: read `newestSnapshot`; if there is one, `h.Apply(ctx, snap.State)` and load only `seq > snap.Seq`. If applying it fails, log at Error, discard it, and fall back to loading the whole log — a snapshot is never authoritative. |
| + |
| +- [ ] **Step 4: Take and write the snapshot in `ApplyUpdate`, before `Apply`** |
| + |
| +After `replay` returns the handle and before `h.Apply(ctx, update)`: |
| + |
| +```go |
| + // The snapshot is taken HERE — from the replayed state, before the |
| + // incoming update is applied — and the placement is the whole of its |
| + // correctness (the snapshots design, D3). |
| + // |
| + // Taking it after a successful Apply is wrong twice. The replay above |
| + // runs outside the transaction below, so a concurrent POST can commit |
| + // a row this handle never saw; a snapshot taken from that handle |
| + // would claim a seq whose content it does not hold, and D5's |
| + // verification cannot notice, because it compares the candidate |
| + // against this same stale handle. And a document already at the cap |
| + // never reaches an after-success hook at all — Apply refuses first — |
| + // so the one case this feature exists for would stay broken forever. |
| + // |
| + // The replayed prefix is complete by construction: seq is gapless |
| + // (MAX+1 under a single-writer pool) and replay read every row that |
| + // existed when it ran. So replayedSeq is unconditionally truthful. |
| + if replayedSeq-lastSnapshotSeq >= snapshotCadence { |
| + if err := s.writeSnapshot(ctx, doc, h, replayedSeq); err != nil { |
| + // Never fatal: the update below still stands, the log is |
| + // untouched, and the next POST tries again. |
| + s.logger.Error("docs: write snapshot", "doc_id", doc.ID, "seq", replayedSeq, "err", err) |
| + } |
| + } |
| +``` |
| + |
| +`replay` returns `replayedSeq` and `lastSnapshotSeq` alongside the handle — read from the snapshot row it already loaded, never a second query. |
| + |
| +- [ ] **Step 5: Implement `writeSnapshot`** |
| + |
| +Take `h.Snapshot(ctx)`, `h.VerifySnapshot(ctx, candidate)`, refuse to write on a false verdict (log at Error and return an error), otherwise insert the row with `core.ABIVersion` and delete every older snapshot for that document — superseded snapshots are derived data, reproducible from the log, so removing them deletes no history and keeps this feature from costing one document-sized blob per 256 updates. |
| + |
| +- [ ] **Step 6: Run the three tests, plus the whole `internal/docs` suite, and watch them pass.** |
| + |
| +- [ ] **Step 7: Commit.** |
| + |
| +--- |
| + |
| +### Task 6: The update endpoint serves snapshot-then-tail, and `?raw=1` |
| + |
| +**Files:** |
| +- Modify: `internal/docs/handlers.go` |
| +- Test: `internal/docs/cursor_test.go` |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +```go |
| +// pastCadence builds a document with more updates than the cadence, so a |
| +// snapshot exists, and returns it. |
| +func pastCadence(t *testing.T, app *App, title string) *Document { |
| + t.Helper() |
| + ctx := context.Background() |
| + owner := app.TestPerson(t, "owner@example.com") |
| + doc, err := app.Store().Create(ctx, owner, title) |
| + if err != nil { |
| + t.Fatalf("Create: %v", err) |
| + } |
| + for i := 0; i < snapshotCadence+3; i++ { |
| + if _, err := app.Store().ApplyUpdate(ctx, doc, validUpdate(t)); err != nil { |
| + t.Fatalf("ApplyUpdate %d: %v", i, err) |
| + } |
| + } |
| + return doc |
| +} |
| + |
| +// A client far behind the snapshot receives the snapshot plus the tail |
| +// rather than the whole log, and the header still names the highest seq |
| +// the bytes actually reach. |
| +func TestUpdatesGetServesSnapshotThenTail(t *testing.T) { |
| + app, err := NewForTest(t) |
| + if err != nil { |
| + t.Fatalf("NewForTest: %v", err) |
| + } |
| + doc := pastCadence(t, app, "Snapshot then tail") |
| + |
| + snap := serve(app, app.TestRequest(t, "owner@example.com", http.MethodGet, |
| + "/d/"+doc.Sqid+"/updates?since=1", nil)) |
| + if snap.Code != http.StatusOK { |
| + t.Fatalf("GET = %d, want 200", snap.Code) |
| + } |
| + raw := serve(app, app.TestRequest(t, "owner@example.com", http.MethodGet, |
| + "/d/"+doc.Sqid+"/updates?since=1&raw=1", nil)) |
| + |
| + if snap.Body.Len() >= raw.Body.Len() { |
| + t.Errorf("snapshot response %d bytes, raw log %d — the snapshot bought nothing", |
| + snap.Body.Len(), raw.Body.Len()) |
| + } |
| + if got := docsSeq(t, snap.Result()); got != int64(snapshotCadence+3) { |
| + t.Errorf("Docs-Seq = %d, want %d", got, snapshotCadence+3) |
| + } |
| +} |
| + |
| +// raw=1 is the escape hatch a browser needs when a snapshot will not |
| +// decode: doc_updates rows only, which is where the log's authority is |
| +// actually reachable from. |
| +func TestRawParameterReturnsOnlyLogRows(t *testing.T) { |
| + app, err := NewForTest(t) |
| + if err != nil { |
| + t.Fatalf("NewForTest: %v", err) |
| + } |
| + ctx := context.Background() |
| + doc := pastCadence(t, app, "Raw") |
| + |
| + raw := serve(app, app.TestRequest(t, "owner@example.com", http.MethodGet, |
| + "/d/"+doc.Sqid+"/updates?raw=1", nil)) |
| + if raw.Code != http.StatusOK { |
| + t.Fatalf("GET raw = %d, want 200", raw.Code) |
| + } |
| + |
| + var rows []DocUpdate |
| + if err := app.Store().db.WithContext(ctx). |
| + Where("doc_id = ?", doc.ID).Order("seq ASC").Find(&rows).Error; err != nil { |
| + t.Fatalf("load log: %v", err) |
| + } |
| + var want []byte |
| + for _, r := range rows { |
| + want = append(want, r.UpdateBin...) |
| + } |
| + if !bytes.Equal(raw.Body.Bytes(), want) { |
| + t.Error("raw=1 did not return exactly the concatenated log") |
| + } |
| +} |
| + |
| +// The live-sync design's D2 survives snapshots: the header is still |
| +// derived from the bytes in hand. |
| +func TestDocsSeqStillDerivedWithSnapshots(t *testing.T) { |
| + app, err := NewForTest(t) |
| + if err != nil { |
| + t.Fatalf("NewForTest: %v", err) |
| + } |
| + doc := pastCadence(t, app, "Header") |
| + top := int64(snapshotCadence + 3) |
| + |
| + behind := serve(app, app.TestRequest(t, "owner@example.com", http.MethodGet, |
| + "/d/"+doc.Sqid+"/updates?since=2", nil)) |
| + if got := docsSeq(t, behind.Result()); got != top { |
| + t.Errorf("Docs-Seq behind the snapshot = %d, want %d", got, top) |
| + } |
| + |
| + ahead := serve(app, app.TestRequest(t, "owner@example.com", http.MethodGet, |
| + "/d/"+doc.Sqid+"/updates?since="+strconv.FormatInt(top+5, 10), nil)) |
| + if ahead.Body.Len() != 0 { |
| + t.Errorf("a cursor ahead of the log returned %d bytes, want none", ahead.Body.Len()) |
| + } |
| + if got := docsSeq(t, ahead.Result()); got != top+5 { |
| + t.Errorf("Docs-Seq ahead of the log = %d, want %d (an echo of since)", got, top+5) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run and watch them fail.** |
| + |
| +- [ ] **Step 3: Implement.** In `updatesGet`: parse `raw`; when not raw, read `newestSnapshot`, and if it exists and `snapshot.Seq > since`, write the snapshot's `State` first and then rows with `seq > snapshot.Seq`; otherwise behave exactly as today. `Docs-Seq` remains `max(since, highest seq in the bytes actually written)` — the snapshot's own `seq` column counts, because it travels with bytes that contain its whole prefix. Update the handler's doc comment: it no longer returns the raw log. |
| + |
| +- [ ] **Step 4: Run the full suite. Commit.** |
| + |
| +--- |
| + |
| +### Task 7: The browser can escape a corrupt snapshot |
| + |
| +**Files:** |
| +- Modify: `fixtures/live-updates.mjs`, `fixtures/live-updates.test.mjs`, `fixtures/editor-entry.js` |
| +- Rebuild: `internal/docs/static/editor.<hash>.js` |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +```js |
| +test('a decode failure retries once through the raw log, then gives up', async () => { |
| + const { live, stream, fetches, applied } = build({ |
| + onUpdates: () => { throw new Error('bad update') }, |
| + }) |
| + live.start() |
| + stream.emit(1) |
| + |
| + fetches.calls[0].settle({ bytes: new Uint8Array([1]), seq: 1 }) |
| + await Promise.resolve(); await Promise.resolve() |
| + |
| + assert.equal(fetches.calls.length, 2, 'no retry after a decode failure') |
| + assert.equal(fetches.calls[1].raw, true, 'the retry did not ask for the raw log') |
| + assert.equal(live.cursor, 0, 'the cursor advanced over bytes that never applied') |
| +}) |
| +``` |
| + |
| +`fakeFetches` must record the `raw` option; extend it rather than adding a second fake. |
| + |
| +- [ ] **Step 2: Run and watch it fail.** |
| + |
| +- [ ] **Step 3: Implement.** `fetchSince(since, { raw })`; in `drain`, wrap the `onUpdates` call so a throw retries the same cursor once with `raw: true`, and a second throw reports through `onError` and leaves the cursor alone. Document why: a snapshot corrupted on disk after it was verified is invisible to the server, which serves the blob without a core call, so the browser is the only reader that can detect it and the only one that needs a way out. |
| + |
| +- [ ] **Step 4: Wire the same fallback into `editor-entry.js`'s initial load**, which fetches before any subscriber exists. |
| + |
| +- [ ] **Step 5: Rebuild the bundle** — `cd fixtures && node build-bundles.mjs` — and commit the result. Do not hand-edit it. |
| + |
| +- [ ] **Step 6: Run `node --test live-updates.test.mjs`, then the gate. Commit.** |
| + |
| +--- |
| + |
| +### Task 8: It actually works |
| + |
| +- [ ] **Step 1: `.amadan/ci.d/10-gate`, exit 0.** |
| +- [ ] **Step 2: A document past the cadence still edits in a real browser.** Drive the editor, type past a snapshot boundary, reload, and confirm the content survives — the one check that exercises snapshot, tail, and the client together. |
| +- [ ] **Step 3: Measure what this bought.** Time a replay of a document with several thousand updates before and after. The number belongs in the branch discussion; it is the evidence the cadence is right. |