amadan
Design: importing DOCX, Markdown and PDF into a new document
Paul's ask overnight: drag a Google Docs / Word / PDF file onto the document list and get something you can edit. §11 of the main design has always promised "DOCX, PDF and Markdown in and out"; out shipped, and this is in. Four claims in here came from throwaway spikes rather than from reading library docs, and two of them were surprises worth the hour: - Text inserted at the right edge of a bold run inherits the bold, so every run has to state every mark explicitly, including the off ones. - heading.level is a *string* on this app's wire -- writing the "correct" number would diverge from every document the editor has ever written. The spikes also proved the part the whole design rests on: an update built by the Rust core is accepted by upstream yjs and converts, through the editor's own schema, into exactly the intended ProseMirror tree. A server-built document the browser cannot open was the risk; it does not happen. Found on the way and reported rather than fixed here: fragment_text renders marks as literal <strong> tags. Latent today (no seed document carries a mark, Store.Text has no non-test caller) and decidedly not latent once import exists, so the fix is task 1 of the plan.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RMNxUbTqqxehd9JdtkYfbe
1 file changed,
+426
−0
docs/superpowers/specs/2026-09-01-import-design.md+426 −0
CI cancelled — superseded by a newer push — run details
Intent
Attributed to this commit: 103 g CO₂e (10445607 tokens). This is an attribution, not a measurement — usage counted from this commit's own marker forward to the next one on worktree-import-documents, never backward from it. A turn that produced more than one commit books all of its tokens to the last one, and a concurrent session on this branch can add to the same figure. Some of it was priced using the shared default rate, because no model-specific factor row exists yet for at least one model involved. The Carbon view has the repo total, which is never derived from this attribution.
diff --git a/docs/superpowers/specs/2026-09-01-import-design.md b/docs/superpowers/specs/2026-09-01-import-design.md| new file mode 100644 |
| index 0000000..0ec518b |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-09-01-import-design.md |
| @@ -0,0 +1,426 @@ |
| +# 🤖 Docs — import |
| + |
| +Design, 2026-09-01. Written from Paul's ask: *"can we do e.g. Google Docs / |
| +PDF / Microsoft Word import? Drag and drop into the docs list and the |
| +document becomes available to edit."* |
| + |
| +This is the other half of §11 of the [main design](2026-08-30-docs-design.md), |
| +which has always said Docs is an editor "with DOCX, PDF and Markdown **in and |
| +out**". Out shipped (`internal/export`, `internal/pdf`). In is this document. |
| + |
| +Four load-bearing claims here were established by throwaway spikes before |
| +anything was written down, not reasoned from the library docs. They are in |
| +§4, and the two that surprised me are the reason this spec exists in its |
| +current shape rather than a worse one. |
| + |
| +--- |
| + |
| +## 1. The scope, and what "Google Docs" actually means |
| + |
| +**A person drags one or more files onto their document list. Each becomes a |
| +new document, theirs, immediately editable, at its own URL.** |
| + |
| +The formats, in the order they earn their place: |
| + |
| +| Format | Route in | Fidelity | |
| +|---|---|---| |
| +| `.docx` | Word, Google Docs "Download → Word", LibreOffice, Pages | Structural: headings, lists, blockquotes, bold/italic/code | |
| +| `.md` | Anything | Structural, within Markdown's own subset | |
| +| `.txt` | Anything | Paragraphs | |
| +| `.pdf` | Anything | **Text only, best effort.** §7 | |
| + |
| +**There is no Google Drive integration in this slice, and that is a |
| +decision rather than an omission.** "Google Docs import" reaches us as the |
| +`.docx` (or `.odt`) a person downloads, because the alternative — OAuth |
| +against Drive, then pulling document bodies through Google's API — ships |
| +customer documents through a third party and holds a token that can read |
| +every file in someone's Drive. §2 of the main spec rules that out for |
| +conversion APIs already, and the same reasoning applies to the source. |
| +A Drive picker is a real feature, and it belongs in §14 behind its own |
| +consent design, not smuggled in under "drag and drop". |
| + |
| +`.odt` is deliberately *not* in the table above. It is the same |
| +zip-plus-XML shape as `.docx` and would likely fall out of that work in an |
| +afternoon, but "likely" is how scope grows overnight; it is listed in §12 |
| +as the first thing to add next. |
| + |
| +--- |
| + |
| +## 2. The shape: parse in Go, build the CRDT in Rust |
| + |
| +Import is the mirror of export, and it is worth stating why that symmetry |
| +is not just tidiness. |
| + |
| +``` |
| + ┌─────────────── export (shipped) ───────────────┐ |
| + document ──▶ core: fragment_json ──▶ ProseMirror JSON ──▶ internal/export ──▶ .md / .pdf |
| + ▲ |
| + the interchange |
| + │ |
| + .docx / .md / .pdf ──▶ internal/importer ──▶ ProseMirror JSON ──▶ core: build ──▶ Yjs update ──▶ document |
| + └─────────────── import (this spec) ──────────────┘ |
| +``` |
| + |
| +**Go parses the files.** `archive/zip` and `encoding/xml` are in the |
| +standard library and a `.docx` is exactly those two things. Go is also |
| +where the HTTP surface, the size limits and the ownership scoping already |
| +live. |
| + |
| +**Rust builds the CRDT.** Nothing outside the core has ever encoded a Yjs |
| +update and nothing outside the core should start. The wire format is the |
| +one thing in this system with two implementations and a fixture gate |
| +holding them honest (§0 of the main spec); a third encoder, in Go, written |
| +for import, would be a fourth thing to keep in step and the least tested of |
| +them. `internal/docs/seed.go` already says this out loud — it has no Go-side |
| +encoder either, which is why the sample documents are committed `.bin` files |
| +generated by a Node script. |
| + |
| +**And that seed pipeline is the honest precedent for the whole design.** |
| +Seeding already needs "turn authored content into a Yjs update"; it solves |
| +it by generating the update offline, at build time, because there was no way |
| +to do it at runtime. Import is the same need with a file a person supplies |
| +thirty seconds ago. Building the runtime path retires the offline hack: once |
| +the core can build an update from a document tree, `generate-seed.mjs` and |
| +its six committed `.update.bin` files are redundant. Retiring them is §12, |
| +not this slice, but the direction matters — this is not a parallel mechanism |
| +beside the seed one, it is the general form of it. |
| + |
| +--- |
| + |
| +## 3. The interchange: ProseMirror JSON, now in both directions |
| + |
| +`core/src/fragment.rs` already emits ProseMirror-shaped JSON |
| +(`fragment_json`, ABI 3) so `internal/export` can ask "where does this list |
| +end, which run is bold". That JSON becomes the import contract too, read in |
| +the other direction. |
| + |
| +One shape, both ways, which buys a gate that is otherwise hard to get: |
| + |
| +> **Round-trip:** for any tree the importers can produce, |
| +> `fragment_json(build(tree))` must equal `tree`. |
| + |
| +That is a real property test over a corpus, and it catches the whole class |
| +of "the importer emitted something the core silently dropped" — the failure |
| +mode that turns into a person's missing paragraph. |
| + |
| +The vocabulary is what the editor's schema already speaks and what |
| +`fragment.rs` already renders, no more: |
| + |
| +- **Blocks:** `paragraph`, `heading` (`level` 1–6), `bullet_list`, |
| + `ordered_list`, `list_item`, `blockquote` |
| +- **Marks:** `strong`, `em`, `code`, and the font/size span marks |
| + (`font_sans`, `font_mono`, `size_sm`, `size_lg`, `size_xl`) |
| + |
| +Import emits no node and no mark outside that set. Anything a source file |
| +carries that does not map onto it is a line in the import report (§8), not |
| +an invented node type — the core carries unknown nodes through on purpose, |
| +and inventing one here would put a node in a document that no renderer, |
| +exporter or editor understands. |
| + |
| +--- |
| + |
| +## 4. What the spikes proved |
| + |
| +Four findings, from running the code rather than reading about it. Two were |
| +surprises, and both would have been expensive to meet during implementation. |
| + |
| +**F1 — the core can build an update, and the readers accept it.** A yrs |
| +`XmlFragmentRef` built with `XmlElementPrelim`/`XmlTextPrelim` and encoded |
| +with `encode_state_as_update_v1` applies cleanly to a fresh document, and |
| +`fragment_html`/`fragment_text`/`fragment_json` all render it as expected. |
| +The mechanism is real; no new dependency. |
| + |
| +**F2 — real Yjs and y-prosemirror accept it too.** The update, applied by |
| +upstream `yjs` 13.6.27 and converted through `yXmlFragmentToProseMirrorRootNode` |
| +against the *actual editor schema*, produced exactly the intended |
| +ProseMirror tree. This is the check that mattered: a server-built document |
| +that the browser editor cannot open is the failure this whole design risks, |
| +and it does not happen. |
| + |
| +**F3 — every text run must state every mark explicitly, including the ones |
| +that are off.** Text inserted at the right edge of a bold run silently |
| +inherits the bold: building `"plain "` + bold `"bold"` + `" and"` yields |
| +`<strong>bold and </strong>`. Y.Text formatting is boundary-inheriting, so |
| +each run is inserted with the full mark set and `Any::Null` for every mark |
| +that should not apply. Found by an assertion failing, not by reasoning — |
| +and an importer that got this wrong would produce documents where a single |
| +bold word bleeds across the rest of the paragraph. |
| + |
| +**F4 — `heading.level` is a *string* on this app's wire, and import must |
| +match.** Writing it as a number is the obvious "correct" choice and is |
| +wrong: every document the editor has ever written stores `level: "2"`, |
| +because that is what y-prosemirror produces, and `fragment.rs` reads it back |
| +with `.parse::<u8>()`. Import writes the string. |
| + |
| + Noted while establishing this, not a blocker and not this slice's to fix: |
| + `prosemirror-model` 1.25 validates `level` as a number, so |
| + `node.check()` rejects every heading in the app today. Nothing calls |
| + `check()`, so nothing is broken — but it is a live tripwire for any future |
| + code that validates a document, and it is written down here because it |
| + cost an hour to understand and would cost it again. |
| + |
| +**And one pre-existing bug found on the way, reported not fixed:** |
| +`fragment_text` renders marks as literal markup — a bold word comes back as |
| +`plain and <strong>bold</strong>`, because yrs's `XmlTextRef::get_string` |
| +serialises formatting as tags. No seed document carries a mark and |
| +`Store.Text` currently has no non-test caller, so it is latent today. It |
| +stops being latent the moment import exists, because imported documents are |
| +full of marks. **Fixing it is a task in this plan** (§10), since import is |
| +what makes it reachable, but it is a bug in the renderer rather than a bug |
| +in import. |
| + |
| +--- |
| + |
| +## 5. The new core surface |
| + |
| +Deliberately one new function, because the machinery to get bytes back out |
| +already exists. |
| + |
| +```rust |
| +// core/src/build.rs |
| +pub fn build_fragment(doc: &Doc, tree_json: &str) -> Result<(), BuildError> |
| +``` |
| + |
| +```c |
| +// core/src/abi.rs — ABI 4 → 5 |
| +docs_core_build(handle: u32, json_ptr: u32, json_len: u32) -> i32 |
| +``` |
| + |
| +It populates the document behind `handle` from the tree. The host then reads |
| +the result through the **existing** `docs_core_snapshot_len` / |
| +`docs_core_snapshot_read` pair, which already means "this document's whole |
| +state as one update". So the sequence, host-side, is: |
| + |
| +1. `Store.Create` — a row, with its app-assigned GUID and client id |
| +2. open a scratch core document on that same GUID and client id |
| +3. `docs_core_build` with the tree |
| +4. `Snapshot` — the update bytes |
| +5. `Store.ApplyUpdate` — the ordinary persistence path, byte-identical in |
| + shape to what seeding and a person's own typing already do |
| + |
| +Step 5 is the point of steps 2–4. Import writes nothing to the database that |
| +an edit does not; there is no second write path to keep correct, and an |
| +imported document is indistinguishable from a typed one the moment it lands. |
| + |
| +**The JSON parser is hand-written**, matching the choice `fragment_json` |
| +already made and documents ("the crate has no serialisation dependency |
| +outside dev-dependencies"). It parses one closed shape and **refuses |
| +everything else** — unknown node type, unknown mark, missing field, depth |
| +past a fixed limit, a `level` outside 1–6. This inverts the reader's |
| +carry-unknowns-through rule on purpose: a renderer meeting an unknown node |
| +should preserve the text, but a *builder* meeting one has been handed |
| +something its only caller could not have meant, and continuing would write |
| +the mistake into a document permanently. |
| + |
| +`build_fragment` is also the natural fuzz target, and gets one. |
| + |
| +--- |
| + |
| +## 6. The importers |
| + |
| +`internal/importer`, one file per format behind a small interface. Each |
| +returns a tree and a report; neither the HTTP layer nor the core knows |
| +which format it came from. |
| + |
| +```go |
| +type Importer interface { |
| + Import(r io.Reader, size int64) (Tree, Report, error) |
| +} |
| +``` |
| + |
| +**DOCX** (`docx.go`) — `archive/zip` for the container, `encoding/xml` for |
| +`word/document.xml`. Paragraphs from `w:p`, style from `w:pStyle` |
| +(`Heading1`–`Heading6`, `Title`, `Quote`), runs from `w:r`/`w:t` with |
| +`w:b`/`w:i`, monospace runs mapped to `code` by font name, lists from |
| +`w:numPr` with `w:ilvl` for depth and `word/numbering.xml` for |
| +bullet-vs-ordered. This is the format that answers both "Microsoft Word" and |
| +"Google Docs" in the ask, and it gets the most care. |
| + |
| +**Markdown** (`markdown.go`) — a small parser for the subset |
| +`internal/export/markdown.go` already writes, which makes export→import a |
| +round trip we can test rather than assert. |
| + |
| +**Plain text** (`text.go`) — blank-line-separated paragraphs. Twelve lines, |
| +and it is the fallback for anything textual we cannot place. |
| + |
| +**PDF** (`pdf.go`) — §7. |
| + |
| +--- |
| + |
| +## 7. PDF, honestly |
| + |
| +A PDF is a description of marks on a page. It does not contain paragraphs, |
| +headings, lists, or reading order — those were discarded when it was made. |
| +Anything that claims to import a PDF "as a document" is reconstructing, |
| +and reconstruction is sometimes wrong. |
| + |
| +So: **PDF import extracts text and produces paragraphs, and the import |
| +report says exactly that, every time, on every PDF.** No heading detection |
| +by font size, no list detection by bullet glyph, in this slice. Those |
| +heuristics are individually plausible and collectively produce a document |
| +that is subtly wrong in ways a person has to hunt for, which is worse than |
| +one that is plainly plain. |
| + |
| +Two behaviours that are not negotiable: |
| + |
| +- **A PDF with no text layer** — a scan, a photo — imports *nothing*. It |
| + fails with "this PDF has no text in it; it looks like a scan", and creates |
| + no document. Silently creating an empty document is the worst available |
| + outcome: it looks like the feature worked. |
| +- **Extraction is sandboxed the same as everything else** (§9). PDF parsers |
| + are a classic source of panics on malformed input, and this one is reading |
| + files from the internet. |
| + |
| +Text extraction needs machinery Docs does not have: `internal/pdf` *writes* |
| +PDFs, and the reader in `internal/export/pdfread_test.go` is a test fixture |
| +that walks files we generated ourselves. Reading arbitrary PDFs means |
| +xref tables, object streams, font encodings and ToUnicode CMaps. |
| +**Recommendation: take a dependency** rather than write that. Pandoc and |
| +conversion APIs stay rejected (§11.2 of the main spec — a second runtime, |
| +and third-party data egress); a pure-Go, in-process library is neither. |
| +The choice of library, with its licence and its failure behaviour on |
| +malformed input, is a task in the plan rather than a decision made here on |
| +no evidence. |
| + |
| +--- |
| + |
| +## 8. The report — degrading loudly |
| + |
| +§11.2 of the main spec requires it: "shows a report naming exactly what it |
| +could not bring in, and where". |
| + |
| +```go |
| +type Report struct { |
| + Format string // "docx" |
| + Warnings []Warning // Kind, Where ("page 3", "paragraph 41"), Detail |
| +} |
| +``` |
| + |
| +What produces a warning: a table (dropped, its cell text kept as |
| +paragraphs), an image, a footnote, a comment, a tracked change, a link |
| +(its text is kept, the target is not), a font with no mapping, a heading |
| +level past 6, and — always, once — a PDF. |
| + |
| +The report is shown after import, on the document, and dismissible. A |
| +document that came in clean says so briefly rather than showing an empty |
| +report. What it must never do is stay silent about a table it threw away. |
| + |
| +--- |
| + |
| +## 9. The HTTP surface, and the drop zone |
| + |
| +**`POST /documents/import`**, inside the existing `RequireSession` group, |
| +next to `POST /documents`. Multipart, one or more files. |
| + |
| +**It is a real form first, and drag-and-drop second.** `list.html` gets a |
| +file input in the page header beside the create form; it works with no |
| +JavaScript, and posts to the same endpoint. The drop zone is progressive |
| +enhancement over it: `dragover` on the list reveals a target, `drop` posts |
| +the same multipart body through `fetch`. This is the same shape §6.2 of the |
| +main spec already requires of reading a document ("works with no script") |
| +and the same shape the create form already has. |
| + |
| +Responses: a no-JS post of a single file redirects `303` to the new |
| +document, exactly as `POST /documents` does. The `fetch` path gets JSON — |
| +per-file status, so three files where the middle one fails report as such |
| +rather than as one failure — and the page inserts the new rows. |
| + |
| +**Ownership and scoping are not novel here.** The owner comes from |
| +`h.owner(r)` like every other handler; `Store.Create` takes it; nothing in |
| +the multipart body names an owner or a document id. This is called out only |
| +because "a new endpoint that creates documents" is exactly where a |
| +mass-assignment mistake would live. |
| + |
| +--- |
| + |
| +## 10. Untrusted files |
| + |
| +Everything in this feature parses a file a person can put anything into, on |
| +a public instance. The limits are part of the design, not hardening to add |
| +later: |
| + |
| +- **25 MB per file**, enforced by `http.MaxBytesReader` before the body is |
| + read into memory, exactly as `maxUpdateBytes` already does for updates |
| +- **Zip:** a cap on entry count, a cap on total *uncompressed* bytes, and a |
| + refusal of any entry whose declared size exceeds it — a `.docx` is a zip, |
| + and a zip bomb is a 40 KB file that becomes 4 GB |
| +- **XML:** Go's `encoding/xml` does not expand external entities, which |
| + removes the classic attack; a nesting-depth cap covers the rest |
| +- **A wall-clock deadline** on the whole import, and the parse runs where a |
| + panic is recovered and becomes a failed import rather than a dead process |
| +- **The core's existing containment** covers `docs_core_build`: a trap |
| + discards and rebuilds the instance (`internal/core`), and the JSON it is |
| + fed came from our own encoder, not from the file |
| + |
| +Also in this section because it belongs to no other: the tree the importers |
| +build is capped in node count and depth before it reaches the core. A |
| +50 MB text file is a legitimate `.txt` and an illegitimate document. |
| + |
| +--- |
| + |
| +## 11. Titles |
| + |
| +In order, first that yields something: the document's own title metadata |
| +(`docProps/core.xml`'s `dc:title` for DOCX, PDF's `/Title`), then the first |
| +heading in the imported content, then the filename with its extension |
| +stripped. Never empty. |
| + |
| +--- |
| + |
| +## 12. Gates |
| + |
| +On top of §12 of the main spec, which all still applies: |
| + |
| +- **Round-trip:** `fragment_json(build(tree)) == tree` over a corpus, in |
| + Rust |
| +- **Cross-implementation:** a Rust-built update applied by upstream `yjs` |
| + and converted with the editor's real schema produces the expected |
| + ProseMirror tree — F2's spike, promoted into `fixtures/` beside the |
| + existing wire-compat fixtures. This is the gate that would catch a |
| + server-built document the editor cannot open. |
| +- **Export→import→export** for Markdown, which must be identity within the |
| + subset |
| +- **Real files, committed:** a `.docx` from Word, one from Google Docs, one |
| + from LibreOffice — they differ, and a fixture from only one of them tests |
| + only one of them |
| +- **Malformed input:** a truncated zip, a `.docx` with no |
| + `word/document.xml`, a zip bomb, a text-free PDF — each asserts a clean |
| + failure, not a panic and not an empty document |
| +- **Fuzz** `build_fragment` |
| + |
| +--- |
| + |
| +## 13. Cut from this slice |
| + |
| +Each with what would bring it back: |
| + |
| +| Cut | Trigger to build it | |
| +|---|---| |
| +| Google Drive picker / OAuth | Its own consent and token-scope design (§14 of the main spec) | |
| +| `.odt` | First — it is the same shape as `.docx` | |
| +| Tables, images | The schema does not have them; they arrive with the editor's, not the importer's | |
| +| PDF heading/list heuristics | Evidence from real imported PDFs that the plain version is not enough | |
| +| Footnotes, tracked changes | Already §14 of the main spec | |
| +| Retiring `generate-seed.mjs` in favour of the runtime builder | This lands and is trusted (§2) | |
| +| Import into an *existing* document | Someone asks; "replace what I am editing" is a different and more dangerous feature than "make me a new document" | |
| + |
| +--- |
| + |
| +## 14. Sequence |
| + |
| +The order is chosen so the thing Paul actually asked for works end to end as |
| +early as possible, and each step leaves the tree green. |
| + |
| +1. **Core: `build_fragment`, `docs_core_build`, ABI 5**, round-trip test, |
| + fuzz target — plus the `fragment_text` mark-leak fix (§4) |
| +2. **Go: `internal/core.Handle.Build`, `Store.Import`** — the four-step |
| + sequence in §5, tested against the real wasm |
| +3. **Text and Markdown importers** — the smallest real end-to-end path |
| +4. **`POST /documents/import`, the form, the drop zone, the report** — at |
| + the end of this step the feature is usable |
| +5. **DOCX** — the format the ask is actually about, and the most work |
| +6. **PDF** — dependency choice first, then extraction, then the honest |
| + report |
| + |
| +Steps 1–5 are the feature. Step 6 is the one whose scope is least certain |
| +and it is last for that reason. |