oficina / docs Public

Clone
git clone https://amadan.net/oficina/docs

Plain git — no account needed to clone.

Download

Download this file

1# Live Updates Across Tabs — Design
2
3**Status:** designed, reviewed by Fable (2026-09-01), one ruling open for Paul (§9).
4**Spec it extends:** `docs/superpowers/specs/2026-08-30-docs-design.md` — §4 (data
5model), §4.1 (the log is bounded), §4.3 (rejected `eventlog`), §7.1 (merge and
6transport), §7.3 (presence and hibernation).
7**Plan it produces:** `docs/superpowers/plans/2026-09-01-live-sync.md`.
8
9## 1. What this is
10
11Open a document in two tabs, type in one, and the other updates without a reload.
12Same person, two tabs, two browsers, or two devices — anywhere a document you own is
13open.
14
15This is the content half of §7's collaboration story, and only that half. It is the
16"later plan" the editing plan's D1 promised: *"Live collaboration, presence, and the
17socket's idle timeout arrive together in a later plan, and the update-log format this
18plan establishes is what that plan will stream."* Presence and the socket do not
19arrive here — see §8 for why they are separable, and §10 for what is deliberately
20left out.
21
22## 2. Where the code is today
23
24Editing is plain HTTP. The browser fetches `GET /d/{sqid}/updates`
25(`internal/docs/handlers.go:311`), which returns every row of the document's update
26log concatenated in seq order, and replays it into a `Y.Doc`. Edits produce Yjs
27updates which are debounced (2000ms as this design shipped; 250ms since — §7.5),
28merged, and POSTed back
29(`internal/docs/handlers.go:359`), validated through the Rust core and appended to
30`doc_updates`. Both routes are owner-scoped through one helper
31(`getDocumentForUpdates`, `handlers.go:277`) and sit inside the `RequireSession` group
32under `csrf.Protect` (`app.go:435–455`).
33
34Nothing pushes to an open editor. A second tab sees changes only on reload. That is
35the whole gap this design closes.
36
37Two existing mechanisms this design leans on rather than replaces:
38
39- **`doc_updates.seq`** is allocated MAX+INSERT inside a transaction
40 (`store.go:539`). The long comment above it explains why that is safe rather than
41 racy: rastrillo/db caps the write pool at exactly one connection, so a second
42 concurrent `ApplyUpdate` cannot begin its transaction until this one commits.
43- **The client's resync path** (`attemptResync` in `fixtures/editor-entry.js`, driven
44 by `resync-controller.mjs`) already rebuilds the `Y.Doc`, the save queue and the
45 `EditorView` wholesale from the server's log. Live updates hang off that same
46 lifecycle rather than inventing a second one.
47
48## 3. Transport: SSE advisory, not a WebSocket
49
50§7.1 specifies a WebSocket with an SSE fallback. This ships an SSE-shaped transport
51first, deliberately.
52
53**What a WebSocket would cost here.** rastrillo v0.22.0 has no WebSocket helper — a
54new dependency in a deliberately lean `go.mod`. The y-protocols sync handshake would
55be hand-rolled in Go, needing state-vector and diff entry points the Rust core does
56not export; that is an ABI bump, which is exactly the work the editing plan's D2
57avoided by having the browser replay raw update bytes. `y-websocket` would need
58vendoring into a bundle this app builds by hand. `Origin` would need checking
59manually, since AGENTS.md's `Sec-Fetch-Site` reasoning covers ordinary requests and
60not a WebSocket handshake. And §7.3's hibernation problem would get worse rather than
61deferred. Most of that expense buys awareness, which §8 removes from this slice.
62
63**What this costs instead.** One extra round trip per change, and a transport that is
64not yet what §7.1 describes.
65
66**Rejected alternatives.** Polling `?since=` on a timer: simplest, but "live" becomes
67seconds late and a parked tab polls forever, which pins a hibernating instance harder
68than a stream that can close. A same-browser `BroadcastChannel`: nearly free, but only
69syncs one browser's own tabs, which is not what was asked.
70
71### 3.1 The stream carries no document bytes
72
73`jobs/events.go` in rastrillo — the only `text/event-stream` producer in the
74framework — carries no HTML, on the reasoning that keeping one rendering path is worth
75more than saving a round trip. The same reasoning applies to bytes: the stream carries
76a seq number, and the client fetches the actual updates through the same binary
77endpoint it already uses at mount, through the same `applyServerLog`.
78
79This also sidesteps SSE being a text protocol. Yjs updates are opaque binary; putting
80them on the stream would mean base64 and roughly 33% overhead on every keystroke
81batch, plus a second code path for document bytes that would need its own tests and
82could drift from the first.
83
84## 4. Protocol
85
86### D1 — `seq` becomes a wire value
87
88`doc_updates.seq` already exists and is already allocated safely. This promotes it
89from an internal ordering column to part of the client-facing contract.
90
91- `Store.ApplyUpdate` returns `(int64, error)` — the assigned seq — instead of `error`.
92 **The seq is meaningful only when `err == nil`.**
93- `GET /d/{sqid}/updates` accepts an optional `?since=N` and returns only rows with
94 `seq > N`. `maxLogBytes` still applies to the slice, still checked before the first
95 byte is written.
96- Both `GET` and `POST /d/{sqid}/updates` carry a `Docs-Seq` response header.
97- A malformed `since` (non-numeric, negative) is a 400.
98
99### D2 — `Docs-Seq` is derived, never queried
100
101**`Docs-Seq` = `max(since, highest seq in the bytes actually returned)`.**
102
103An earlier draft said "the highest seq included, or the document's current max when
104the slice is empty". Fable's review found that this loses an update, silently and
105permanently. The second clause needs a separate `MAX(seq)` query, and reads go to the
106read replicas while writes go to the single-connection write pool — nothing serializes
107the two:
108
109> Tab A is at cursor 45 and fetches `?since=45`. The slice query returns empty. Tab B's
110> POST commits row 46 and broadcasts. A's `MAX(seq)` query then runs and returns 46.
111> A receives an empty body and `Docs-Seq: 46`, and advances its cursor to 46 having
112> never applied row 46. The broadcast that would have rescued it arrived while A's
113> fetch was in flight and was coalesced into a follow-up `?since=46` — which is also
114> empty. Row 46 stays invisible in tab A until a manual reload.
115
116Deriving the header from the bytes in hand makes that interleaving unrepresentable:
117an empty slice echoes `since` back, so a cursor never moves over a row the client did
118not receive. The no-`since` initial GET on an empty document derives 0, which seeds
119the cursor correctly.
120
121**The client half of the same rule: a cursor only ever moves forward.**
122`cursor = max(cursor, header)`, never assignment, so a response that arrives out of
123order cannot drag a cursor backwards.
124
125### D3 — the cursor+1 rule for a tab's own POST
126
127A tab that POSTs and gets `Docs-Seq: 43` back may advance its cursor to 43 **only if
128its cursor was 42** — only if `seq == cursor + 1`.
129
130Otherwise another tab wrote rows in between that this tab has not applied, and
131adopting 43 would skip them permanently. When the rule does not hold the cursor stays
132put, the tab is "behind", and an ordinary `?since=` fetch — triggered by the stream,
133or by the connect-time event on the next reopen — heals it.
134
135The rule earns its place by removing the echo round trip: in the common case (one tab
136typing) every POST advances the cursor without a fetch, and the broadcast for that seq
137is a no-op when it arrives.
138
139Fable constructed the adversarial interleavings — two tabs racing, a `beforeunload`
140beacon racing a flush including the case where the beacon fires but the page survives,
141a 5xx retry over an already-committed row, two concurrent flushes from one tab with
142responses arriving inverted, and a resync mid-flight — and none of them loses an
143update, duplicates one with effect, or wedges a cursor permanently. A cursor that
144fails to advance is always healed; that is the property to preserve if this rule is
145ever changed.
146
147### D4 — broadcast on success only
148
149The handler broadcasts after `ApplyUpdate` returns a nil error, never before and never
150on an error.
151
152This matters because `ApplyUpdate` commits the `doc_updates` row in its transaction
153and then bumps `documents.updated_at` *outside* it. If that bump fails, the method
154returns an error over a row that is already durable. The handler 500s, sends no
155`Docs-Seq`, and broadcasts nothing; the client's save queue treats a 5xx as retryable
156and re-POSTs the same bytes, which lands a **duplicate row**. That is accepted: Yjs
157integration is idempotent so the document converges, the cursor+1 rule keeps the
158client honest (its cursor never moved), and the cost is one wasted row against
159`maxRetainedUpdates`. The same is true of a connection dropped after commit.
160
161The alternative — broadcasting from inside `Store.ApplyUpdate` — is rejected in D5.
162
163### D5 — the broadcast seam is the handler, not `Store`
164
165`Store` stays a persistence type that knows nothing about transports, and
166`doc_updates` stays "a plain append-only table" as §4.3 insisted when it rejected
167`eventlog` for carrying JSON payloads and a redundant Lamport order.
168
169The accepted consequence: a writer that does not go through `updatesPost` does not
170notify. Today the only other writer is sample seeding (`seedIfEnabled` → `Seed` →
171`ApplyUpdate`), which runs when a person's row is first created — before any editor of
172theirs can be open — so the consequence has no reachable case. A future importer would
173need to notify explicitly, and this paragraph is the note that says so.
174
175## 5. Server design
176
177### 5.1 The hub — `internal/docs/stream.go` (new)
178
179A `map[int64]map[chan int64]struct{}` behind a mutex: subscribe by document id and get
180a one-slot channel, unsubscribe on stream close. Sends are non-blocking — a full slot
181means a notification is already pending, and since every client fetch is
182`since=cursor`, coalescing a burst into one wake-up loses nothing. The seq on the
183channel is advisory: it tells a client *that* there is something new, and the fetch
184determines *what*.
185
186One instance per team, one process — the docs spec's opening paragraph ("One instance
187per team") and its §3, System shape ("One SQLite file per instance") — so there is
188nothing to coordinate across machines and no shared-broker-shaped hole here.
189
190### 5.2 The stream — `GET /d/{sqid}/stream`
191
192Owner-scoped through the existing `getDocumentForUpdates` helper, inheriting its
193404-not-403 behaviour unchanged (a stranger must not learn the document exists).
194Registered in the same `RequireSession` group as the update routes; a GET, so
195`csrf.Protect` passes it through untouched.
196
197The body follows `jobs/events.go`: an `http.Flusher` check, `http.NewResponseController`,
198`text/event-stream` with `Cache-Control: no-store`, a 200 and a flush before the first
199event, a 30-second write deadline re-armed before every write, a `: ping` comment
200heartbeat every 15 seconds, a 5-minute TTL, and a `select` that includes
201`r.Context().Done()` so a graceful drain is never held open.
202
203Each piece of that recipe is carried for its own reason, not by imitation:
204
205- **The write deadline** is what lets a stream coexist with `serve`'s zero
206 `WriteTimeout` default (`serve.go`'s own Timeouts recipe). `cmd/docs/main.go` goes
207 through `rastrillo.Resolve` with the defaults, so that zero applies here.
208- **The heartbeat** is what makes the write deadline ever fire on a quiet stream. It
209 is what bounds discovery of a dead client to about 45 seconds instead of never.
210- **The TTL** is the hard bound on any one connection, and half of the hibernation
211 story (§6).
212- **The 1-second database tick is dropped.** `jobs/events.go` polls because, in its own
213 words, it had no broadcast primitive to wait on. This has one, so the handler blocks
214 on the hub channel and never polls. One thing the tick also did is not replaced: it
215 re-checked the entity's existence every second and could emit `gone`. There is no
216 delete route yet, and the TTL bounds the exposure to five minutes; when deletion
217 lands, this needs revisiting.
218
219Events are `event: update` with `data: <seq>`. **One is sent immediately on connect,
220carrying the highest seq the server currently has.** That event is load-bearing rather
221than an optimisation: it is what heals a tab whose cursor legitimately failed to
222advance (D3), and what covers the gap between the page rendering and the stream
223opening. It gets its own test.
224
225### 5.3 Deliberately omitted: a subscriber cap
226
227Every route here is behind `RequireSession`, a document has exactly one owner today,
228and a subscriber costs a goroutine and a one-slot channel. Revisit when sharing lands
229and a document can have many viewers.
230
231## 6. Hibernation (§7.3)
232
233§7.3's requirement: *"The editor socket has an idle timeout: no edits and no presence
234activity for a set period closes it, and any activity reconnects. A parked tab stops
235holding the instance open."*
236
237**Server.** The 5-minute TTL is the hard bound. No timer exists outside a live
238connection, so nothing here is a `time.Ticker` that dies at hibernation — the trap
239§4.1 and §7.3 both name.
240
241**Client.** Three rules, and the first is the one an implementer will get wrong:
242
2431. **The module owns every open and every close.** `EventSource` reconnects natively
244 on any server close, which would defeat every policy decision below — including the
245 idle-close the module just made. So native reconnection is never relied on: every
246 close is an explicit `es.close()`, and every open is a deliberate decision. This is
247 the one piece of `jobs/events.go`'s recipe whose reason does *not* survive the move:
248 that handler relies on native auto-reconnect precisely because it has no close
249 policy of its own. Miss this and a parked tab's browser reopens the stream forever,
250 which is §7.3's exact failure.
2512. **Close when hidden, or when idle.** A hidden tab closes immediately; a visible one
252 closes after an idle period with no local edits and nothing received.
2533. **"Any activity" means any activity.** Visibility, focus and local edits are not
254 enough: a tab that is already visible and already focused while its person *reads* a
255 long document fires none of them, so the idle timer would close the stream and
256 nothing would ever reopen it — and the watching half of "type in one, watch the
257 other" would go permanently stale. Passive `pointermove`, `scroll` and `keydown`
258 listeners count as activity, both deferring the idle close and reopening a closed
259 stream.
260
261A visible, active tab reconnecting every five minutes does hold its instance awake.
262That is intended: someone is there.
263
264## 7. Client design
265
266### 7.1 A new module, `fixtures/live-updates.mjs`
267
268Follows the convention `save-queue.mjs` and `resync-controller.mjs` set — dependencies
269injected, so it runs under plain Node with no DOM, no `EventSource` and no real
270`Y.Doc`, and its own `.test.mjs` goes in the gate. It owns:
271
272- the cursor, and D2's monotonic rule;
273- the `EventSource` lifecycle and every policy in §6;
274- fetch serialization: one in-flight `?since=` request at a time, with a pending flag,
275 so a burst of notifications cannot interleave two applies;
276- delivering fetched bytes to its caller.
277
278Two rules its tests pin, because both are currently true by accident rather than by
279construction: a stopped instance drops a late-resolving fetch rather than delivering
280it, and a stream event is a fetch trigger, never a cursor assignment.
281
282### 7.2 Wiring, confined to `attemptResync`
283
284`attemptResync` already rebuilds the `Y.Doc`, save queue and `EditorView` wholesale, and
285is used both for the first mount and for every recovery. Live updates join that
286lifecycle: a `currentLive` alongside `currentQueue`/`currentView`, stopped and replaced
287exactly as those are. The initial GET reads `Docs-Seq` to seed the cursor. Fetched
288bytes go through the existing `applyServerLog`.
289
290Nothing in `save-queue.mjs` changes. The `post` closure that `editor-entry.js` already
291supplies becomes an async closure that reads `Docs-Seq` on a 2xx and returns the
292response unchanged; the queue consumes only `r.ok` and `r.status`. The
293`beforeunload` beacon bypasses `post` and reads no response at all, which is harmless:
294it moves no cursor.
295
296### 7.3 Echo, divergence, and undo
297
298**Echo** needs no new mechanism. `applyServerLog` already tags replayed bytes with
299`Y.readUpdate(decoder, ydoc, 'server')`, and the `ydoc.on('update')` handler already
300returns early on `origin === 'server'`. ySyncPlugin does not re-emit remote-origin
301changes as new Yjs updates either, so there is no ProseMirror-level echo.
302
303**Divergence.** While `resyncController.diverged` is true the local document is known
304poisoned and a rebuild is coming, so the subscriber stops applying. `attemptResync`
305then replaces it wholesale. No new state machine.
306
307**Undo.** `yUndoPlugin` already does the right thing: its `Y.UndoManager` tracks only
308ySyncPlugin's own origin, so `'server'`-origin updates are never captured and Mod-z
309after a remote edit undoes your own last edit rather than someone else's.
310**Do not add `'server'` to `trackedOrigins`.** This sentence exists because that
311change looks like a fix and is a bug.
312
313Two known cosmetic behaviours, neither data loss: a remote transaction landing
314mid-IME-composition can cancel the composition, and undo history resets on a resync
315because the whole document is rebuilt.
316
317### 7.4 A dead stream is a freshness failure, not a save failure
318
319`[docs-save-status]` means "your edit did not save". A stream failure must never light
320it: POST is unaffected and edits are still saving. Repeated `EventSource` failures stop
321the stream and leave the tab to heal on the next activity-triggered reopen. Logged to
322the console; nothing user-visible in this slice, and therefore no new catalog keys.
323
324### 7.5 The debounce — 2000ms at the time of writing, 250ms since
325
326`DEBOUNCE_MS` sets the floor on how live this can feel: the other tab cannot see an
327edit before the originating tab sends it.
328
329**As this design shipped, it was 2000ms, and that was not a decision about feel.**
330Every accepted POST is one `doc_updates` row, and `internal/core/core.go`'s
331`maxRetainedUpdates` (4096, no compaction at the time) capped a document at that many
332before it refused edits permanently. So the constant was headroom against a cliff —
3334096 × 2s is about 2h16m of continuous typing, 4096 × 0.8s about 55 minutes — and
334**"live" therefore meant "within about two seconds of the typist pausing"**, stated
335plainly rather than papered over.
336
337**Superseded 2026-09-01 by the snapshots work** (`2026-09-01-snapshots-design.md`,
338landed as `6bb0a03`). Replay now starts from a snapshot plus a bounded tail, the cliff
339is gone, and the constant became free to be about perception. It is **250ms** — under
340the roughly 300ms at which a delay stops reading as a response and starts reading as a
341wait. "Live" now means within about a quarter-second of the typist pausing, plus a
342round trip.
343
344What that costs is stated where it belongs, in `DEBOUNCE_MS`'s own comment: eight times
345the row rate, bounded for replay by snapshots and **not** bounded for storage, because
346§4.1's compaction horizon is still deferred. The mechanism that would return the rows
347without returning the latency is §4.1's coalescing at the storage boundary, which
348belongs with the WebSocket transport.
349
350## 8. Why presence is separable
351
352§7.3 ties presence to the socket, and this design has no socket. That is not a
353deferral of something half-built: presence needs ephemeral, high-frequency,
354upstream-and-downstream messages (awareness), which is exactly what a WebSocket is for
355and exactly what an SSE-plus-POST pair is bad at. Building presence on this transport
356would mean building the wrong thing twice.
357
358It is also unreachable today. Documents are owner-scoped — `internal/docs/models.go`
359has `Document`, `Person` and `DocUpdate` and no `grants` table — so the only "other
360viewer" is the same person's other tab, and a presence stack that shows you yourself
361is worse than nothing. Presence arrives with sharing, on a socket, as §7 always
362intended.
363
364## 9. Ruling, made 2026-09-01
365
366**§7.1 should be amended rather than annotated, and the wording matters.**
367
368§7.1's fallback describes "the client POSTs batched updates with a monotonic sequence
369number **per connection**, and the server acknowledges the high-water mark". What this
370design ships is not that: `Docs-Seq` is the document's global log seq, durable in
371`doc_updates`, meaningful across connections and across tabs. It is a different
372protocol and a better one — a per-connection counter cannot heal a tab that reconnects,
373which is most of what §6 needs.
374
375So "the fallback shipped first" would be a false description of the code.
376
377**Paul ruled on 2026-09-01: amend it.** The docs design spec's §7.1 is now revision 3
378and describes the doc-seq protocol as the transport, with the WebSocket as the upgrade
379that arrives with presence. That edit is part of this branch, and it records the
380correction rather than quietly overwriting the old wording — a per-connection counter
381was a decision, and one that could not have survived §7.3's idle timeout.
382
383The same ruling accepted §6's connection policy as it stands, including its one known
384consequence: **a hands-free watcher goes stale at the first server TTL.** Receiving
385events defers the idle close but cannot reopen after the server cuts the stream at five
386minutes, because the client treats every `EventSource` error — an expected TTL close
387included — as "drop it and wait for activity". A wall display nobody touches shows five
388minutes of updates and then freezes. That is the honest reading of §6.3's "any activity
389reconnects": *some* activity is still required. If it ever needs to change,
390`EventSource.readyState` at error time distinguishes a close the browser means to retry
391from a fatal one, which would mean relaxing §6.1's blanket rule — deliberately stronger
392than it needs to be, since `es.close()` never triggers a reconnection and only
393server-side closes were ever at issue.
394
395## 10. Non-goals
396
397Presence, remote cursors, sharing and grants, WebSocket transport, offline
398(`y-indexeddb`), snapshots and compaction. Each belongs to a spec section that already
399describes it; none is blocked by anything here.
400
401## 11. Testing
402
403- **Go, `internal/docs/stream_test.go` (new).** `httptest.NewRecorder` cannot express a
404 long-lived stream — it cannot observe bytes before the handler returns — so these use
405 `httptest.NewServer` and read the body incrementally, with short injected durations
406 (mirroring `jobs/events.go`'s injectable `tick`/`ttl`/`heartbeat`). Cases: a POST from
407 another request reaches a subscriber as `event: update` with the right seq; the
408 connect-time event carries the current max; a stranger's document 404s; the TTL closes
409 cleanly; a heartbeat appears on a quiet stream; a cancelled request context ends the
410 handler.
411- **Go, `internal/docs/updates_test.go` (extend).** `?since=` slicing; `Docs-Seq` on both
412 GET and POST; **D2's derivation — an empty slice echoes `since` and never reports a
413 higher number**; a malformed `since` is a 400; `maxLogBytes` still enforced on a slice.
414- **Node, `fixtures/live-updates.test.mjs` (new).** D3 including the case where the
415 cursor must *not* advance; D2's monotonic cursor under an out-of-order response;
416 fetch serialization under a burst; notification coalescing; the §6 close/reopen
417 policy including the visible-and-idle case; a stopped instance dropping a late fetch.
418- **Acceptance.** `./scripts/run-dev.sh`, one document open in two tabs, type in one,
419 watch the other. That is the ask, and it belongs in the plan as a step.
420- `.amadan/ci.d/10-gate` green, with the rebuilt editor bundle committed.
421
422## 12. Review record
423
424Reviewed by **Fable** on 2026-09-01, per AGENTS.md's highest-available-capability rule,
425at the design stage rather than after implementation. Verdict: approve with changes.
426Two defects found and folded in above — the `Docs-Seq` TOCTOU (D2) and the
427visible-idle-tab staleness (§6.3) — along with the `EventSource` auto-reconnect rule
428(§6.1), the broadcast-on-success contract (D4), the `trackedOrigins` warning (§7.3), and
429the §7.1 wording correction (§9). The review verified all three of the design's claims
430about existing code, and confirmed rastrillo v0.22.0 offers no WebSocket, pubsub or
431streaming helper beyond `jobs/events.go`.
432