amadan

oficina / docs Public

The core can build a document, and the editor can open it

build_fragment turns canonical ProseMirror JSON -- the same shape
fragment_json already emits for export -- into a Y.XmlFragment. This is
the mechanism the whole import feature hangs off, and it is deliberately
the only new thing here: no ABI export, no Go caller, no behaviour change
to any existing document. What it buys is that the risky half of the
design is now proven rather than argued.

Two properties, and the second is the one that matters.

The round trip -- fragment_json(build(tree)) structurally equal to tree
-- catches "the importer emitted something the core silently dropped",
which is how a person loses a paragraph. Structural, via
serde_json::Value, never string equality, because fragment_json writes
JSON by hand.

But the round trip alone would pass a document the editor cannot open:
fragment_json will happily render a text node directly under doc, so a
broken importer could round-trip perfectly and still break the editor.
So build_fragment enforces the editor's own content model, and
scripts/wire-gate.sh gains a check that asks the only question that
settles it -- emit_import builds a document in Rust, and verify-import
has upstream yjs, y-prosemirror and the real editor schema turn the
bytes back into a ProseMirror tree. It reconstructs exactly what was
built, nested list and all. Every Rust test reads the document back
through the same yrs that wrote it, so Rust agrees with itself even
where both halves are wrong about the browser; this is the one check
that does not have that blind spot.

One finding worth the reader's time: heading level is written as a
STRING, and that is forced rather than chosen. The editor writes a
number, but yrs 0.23.5 has no public API that can store a non-string XML
attribute -- insert_attribute is typed V: Into<String>, and the typed
path goes through MapRef, which wraps a private BranchPtr. set_level
documents what that costs: it matches this repo's existing fixtures, the
editor opens it, and prosemirror-model's validate:"number" would reject
it if anything ever called node.check() -- which is already true of every
heading in the fixtures, not just imported ones.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RMNxUbTqqxehd9JdtkYfbe
Paul Campbell pushed by paul@keymail.dev ed1a16bd0fdc2fc244ab0d1afa72f5bfa5e84624 parent 167cf27
7 files changed, +1522 −0
  • core/src/bin/emit_import.rs +62 −0
  • core/src/build.rs +766 −0
  • core/src/lib.rs +2 −0
  • core/tests/build.rs +516 −0
  • fixtures/verify-import.mjs +167 −0
  • internal/core/testdata/docs_core.wasm binary
  • scripts/wire-gate.sh +9 −0

CI cancelled — superseded by a newer push — run details

Intent

full ledger: worktree-import-documents

diff --git a/core/src/bin/emit_import.rs b/core/src/bin/emit_import.rs
new file mode 100644
index 0000000..401e1e7
--- /dev/null
+++ b/core/src/bin/emit_import.rs
@@ -0,0 +1,62 @@
+//! Builds a document through `build_fragment` and writes a v1 update to
+//! stdout, so the JS side can check that what the *importer* produces is
+//! something the real editor can open.
+//!
+//! This is `emit_update`'s sibling and exists for a sharper reason. That
+//! one proves JS can read a Y.Text Rust wrote. This one proves the whole
+//! import path: a tree goes in as JSON, comes out as CRDT bytes, and
+//! upstream `yjs` plus `y-prosemirror` plus the editor's own schema turn
+//! it back into the document it started as. A server-built document the
+//! browser cannot open is the single failure this design risks, and it
+//! is invisible to every Rust test — `fragment_json` reads through the
+//! same yrs that wrote it, so the two agree even when both are wrong
+//! about what the browser expects.
+//!
+//! Paired with `fixtures/verify-import.mjs`; run by `scripts/wire-gate.sh`.
+
+use docs_core::build_fragment;
+use std::io::Write;
+use yrs::{Doc, Options, ReadTxn, Transact};
+
+/// The tree the JS side expects to get back. Exercises every part of the
+/// vocabulary whose CRDT representation could plausibly differ between
+/// the two implementations: a heading's attribute, a run with one mark,
+/// a run with two, an unmarked run following a marked one (the
+/// mark-inheritance hazard), both list kinds, a nested list, and a
+/// blockquote.
+const TREE: &str = concat!(
+ r#"{"type":"doc","content":["#,
+ r#"{"type":"heading","attrs":{"level":2},"content":[{"type":"text","text":"Quarterly Report"}]},"#,
+ r#"{"type":"paragraph","content":["#,
+ r#"{"type":"text","text":"plain and "},"#,
+ r#"{"type":"text","text":"bold","marks":[{"type":"strong"}]},"#,
+ r#"{"type":"text","text":" and "},"#,
+ r#"{"type":"text","text":"both","marks":[{"type":"font_sans"},{"type":"strong"}]}]},"#,
+ r#"{"type":"bullet_list","content":["#,
+ r#"{"type":"list_item","content":["#,
+ r#"{"type":"paragraph","content":[{"type":"text","text":"Outer"}]},"#,
+ r#"{"type":"bullet_list","content":[{"type":"list_item","content":["#,
+ r#"{"type":"paragraph","content":[{"type":"text","text":"Inner"}]}]}]}]}]},"#,
+ r#"{"type":"ordered_list","content":[{"type":"list_item","content":["#,
+ r#"{"type":"paragraph","content":[{"type":"text","text":"First"}]}]}]},"#,
+ r#"{"type":"blockquote","content":[{"type":"paragraph","content":["#,
+ r#"{"type":"text","text":"Quoted","marks":[{"type":"em"}]}]}]}]}"#
+);
+
+fn main() {
+ // Both values fixed here, and never drawn from yrs (the same Global
+ // Constraint emit_update records: see core/src/lib.rs and
+ // clippy.toml). A real import draws its client id in Go, freshly per
+ // import and inside [0, 2^32) -- see the import design §4 D1 for why
+ // the row's own 63-bit ClientID must never be used. 7002 is inside
+ // that range and is this binary's own fixed id, one past
+ // emit_update's 7001.
+ let guid: yrs::Uuid = "emit-import-guid".into();
+ let doc = Doc::with_options(Options::with_guid_and_client_id(guid, 7002));
+
+ build_fragment(&doc, TREE).expect("building the fixture tree must succeed");
+
+ let txn = doc.transact();
+ let update = txn.encode_state_as_update_v1(&yrs::StateVector::default());
+ std::io::stdout().write_all(&update).unwrap();
+}
diff --git a/core/src/build.rs b/core/src/build.rs
new file mode 100644
index 0000000..527c07f
--- /dev/null
+++ b/core/src/build.rs
@@ -0,0 +1,766 @@
+//! Building a ProseMirror-shaped `Y.XmlFragment` from canonical JSON —
+//! the import half of the interchange `fragment_json` already writes
+//! (the import design, §3 and §5).
+//!
+//! # Why this is strict where the renderers are lenient
+//!
+//! `fragment.rs` carries unknown nodes and unknown marks through on
+//! purpose: an older binary must not lose a newer editor's text, so
+//! silence there would be data loss. This module inverts that rule. Its
+//! only caller is `internal/importer`, so an unknown node here is not a
+//! newer editor being generous — it is a bug in the one program allowed
+//! to call this, and building it would write that bug into a document
+//! permanently. Everything unrecognised is refused.
+//!
+//! # The content model, and why the round trip is not enough on its own
+//!
+//! `fragment_json` will happily render a `text` node directly under
+//! `doc`, or a `list_item` outside a list. So a builder that only had to
+//! satisfy `fragment_json(build(tree)) == tree` could produce a document
+//! the editor cannot open, which is exactly the failure the
+//! cross-implementation gate exists to prevent. `validate` below
+//! enforces the editor's own content model (`fixtures/editor-entry.js`'s
+//! schema: `prosemirror-schema-basic` plus `addListNodes(…, 'paragraph
+//! block*', 'block')`) before anything is written.
+//!
+//! # One `XmlText` per block
+//!
+//! y-prosemirror groups every adjacent ProseMirror text node in a block
+//! into a single `Y.XmlText` (`sync-plugin.js`'s `normalizePNodeContent`).
+//! A builder that gave each run its own `XmlText` would still round-trip
+//! — `fragment_json` emits one `text_json` call per `XmlText` child and
+//! never merges across siblings — and the editor would rewrite the whole
+//! block on the first edit, because the CRDT shape would differ from
+//! what the browser writes. So runs go into one `XmlText` per block.
+
+use crate::fragment::ROOT;
+use std::collections::HashMap;
+use std::sync::Arc;
+use yrs::types::xml::XmlFragment;
+use yrs::types::Attrs;
+use yrs::{
+ Any, Doc, Text, Transact, TransactionMut, Xml, XmlElementPrelim, XmlElementRef, XmlTextPrelim,
+ XmlTextRef,
+};
+
+/// Why a build was refused.
+#[derive(Debug)]
+pub enum BuildError {
+ /// The bytes were not JSON, or not the closed shape this module
+ /// parses (a missing field, a value of the wrong type).
+ Json(String),
+ /// The tree parsed but is not one the editor could open: an unknown
+ /// node or mark, a `level` outside 1–6, or a content-model
+ /// violation.
+ Schema(String),
+ /// Nesting past `MAX_DEPTH`. Checked while parsing, before the
+ /// recursion that would otherwise overflow the stack — which is not
+ /// a catchable error in Rust and would abort the whole instance.
+ TooDeep,
+ /// The document already has content. This builds documents; it does
+ /// not append to them.
+ RootNotEmpty,
+}
+
+impl std::fmt::Display for BuildError {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match self {
+ BuildError::Json(m) => write!(f, "json: {m}"),
+ BuildError::Schema(m) => write!(f, "schema: {m}"),
+ BuildError::TooDeep => write!(f, "tree nested deeper than {MAX_DEPTH}"),
+ BuildError::RootNotEmpty => write!(f, "document already has content"),
+ }
+ }
+}
+
+/// Maximum JSON nesting. A document tree costs roughly two JSON levels
+/// per block level (the object, then its `content` array), so this
+/// allows block nesting far past anything a real document has while
+/// staying well inside the stack a wasm instance is given.
+const MAX_DEPTH: usize = 128;
+
+/// The marks this build knows. Deliberately the same list as
+/// `fragment.rs`'s `MARKS`, and a mark outside it is refused rather than
+/// carried — see the module comment.
+const MARKS: &[&str] = &[
+ "font_sans",
+ "font_mono",
+ "size_sm",
+ "size_lg",
+ "size_xl",
+ "strong",
+ "em",
+ "code",
+];
+
+// --- a minimal JSON reader -------------------------------------------
+//
+// Hand-written, matching the choice `fragment_json` already made and
+// documents: the crate has no serialisation dependency outside
+// dev-dependencies. This reads the general grammar rather than the
+// document shape, so malformed input fails as `Json` with a position
+// instead of being mistaken for a schema problem.
+
+enum Json {
+ Null,
+ /// Legal JSON, never legal in a document tree -- carried so a
+ /// boolean where a node belongs fails as a shape error with a
+ /// position, not as a parse error.
+ Bool,
+ Num(f64),
+ Str(String),
+ Arr(Vec<Json>),
+ Obj(Vec<(String, Json)>),
+}
+
+impl Json {
+ fn kind(&self) -> &'static str {
+ match self {
+ Json::Null => "null",
+ Json::Bool => "bool",
+ Json::Num(_) => "number",
+ Json::Str(_) => "string",
+ Json::Arr(_) => "array",
+ Json::Obj(_) => "object",
+ }
+ }
+
+ /// The value of `key`, if this is an object that has it.
+ fn get(&self, key: &str) -> Option<&Json> {
+ match self {
+ Json::Obj(pairs) => pairs.iter().find(|(k, _)| k == key).map(|(_, v)| v),
+ _ => None,
+ }
+ }
+}
+
+struct Parser<'a> {
+ b: &'a [u8],
+ i: usize,
+ depth: usize,
+}
+
+type JsonResult<T> = Result<T, BuildError>;
+
+fn json_err(msg: impl Into<String>) -> BuildError {
+ BuildError::Json(msg.into())
+}
+
+impl<'a> Parser<'a> {
+ fn new(s: &'a str) -> Self {
+ Parser {
+ b: s.as_bytes(),
+ i: 0,
+ depth: 0,
+ }
+ }
+
+ fn ws(&mut self) {
+ while let Some(c) = self.b.get(self.i) {
+ match c {
+ b' ' | b'\t' | b'\n' | b'\r' => self.i += 1,
+ _ => break,
+ }
+ }
+ }
+
+ fn peek(&mut self) -> JsonResult<u8> {
+ self.ws();
+ self.b
+ .get(self.i)
+ .copied()
+ .ok_or_else(|| json_err("unexpected end of input"))
+ }
+
+ fn eat(&mut self, c: u8) -> JsonResult<()> {
+ if self.peek()? == c {
+ self.i += 1;
+ Ok(())
+ } else {
+ Err(json_err(format!(
+ "expected {:?} at byte {}",
+ c as char, self.i
+ )))
+ }
+ }
+
+ fn value(&mut self) -> JsonResult<Json> {
+ match self.peek()? {
+ b'{' => self.object(),
+ b'[' => self.array(),
+ b'"' => Ok(Json::Str(self.string()?)),
+ b't' => self.literal("true", Json::Bool),
+ b'f' => self.literal("false", Json::Bool),
+ b'n' => self.literal("null", Json::Null),
+ _ => self.number(),
+ }
+ }
+
+ fn literal(&mut self, word: &str, v: Json) -> JsonResult<Json> {
+ if self.b[self.i..].starts_with(word.as_bytes()) {
+ self.i += word.len();
+ Ok(v)
+ } else {
+ Err(json_err(format!("bad literal at byte {}", self.i)))
+ }
+ }
+
+ /// Enters one nesting level, refusing past `MAX_DEPTH`. This is the
+ /// check that keeps a hostile tree from overflowing the stack: it
+ /// fires on the way *in*, before the recursive call.
+ fn enter(&mut self) -> JsonResult<()> {
+ self.depth += 1;
+ if self.depth > MAX_DEPTH {
+ return Err(BuildError::TooDeep);
+ }
+ Ok(())
+ }
+
+ fn array(&mut self) -> JsonResult<Json> {
+ self.enter()?;
+ self.eat(b'[')?;
+ let mut out = Vec::new();
+ if self.peek()? == b']' {
+ self.i += 1;
+ self.depth -= 1;
+ return Ok(Json::Arr(out));
+ }
+ loop {
+ out.push(self.value()?);
+ match self.peek()? {
+ b',' => self.i += 1,
+ b']' => {
+ self.i += 1;
+ break;
+ }
+ c => {
+ return Err(json_err(format!(
+ "expected ',' or ']' at byte {}, got {:?}",
+ self.i, c as char
+ )))
+ }
+ }
+ }
+ self.depth -= 1;
+ Ok(Json::Arr(out))
+ }
+
+ fn object(&mut self) -> JsonResult<Json> {
+ self.enter()?;
+ self.eat(b'{')?;
+ let mut out = Vec::new();
+ if self.peek()? == b'}' {
+ self.i += 1;
+ self.depth -= 1;
+ return Ok(Json::Obj(out));
+ }
+ loop {
+ let key = self.string()?;
+ self.eat(b':')?;
+ let value = self.value()?;
+ out.push((key, value));
+ match self.peek()? {
+ b',' => self.i += 1,
+ b'}' => {
+ self.i += 1;
+ break;
+ }
+ c => {
+ return Err(json_err(format!(
+ "expected ',' or '}}' at byte {}, got {:?}",
+ self.i, c as char
+ )))
+ }
+ }
+ }
+ self.depth -= 1;
+ Ok(Json::Obj(out))
+ }
+
+ fn number(&mut self) -> JsonResult<Json> {
+ let start = self.i;
+ if self.b.get(self.i) == Some(&b'-') {
+ self.i += 1;
+ }
+ while let Some(c) = self.b.get(self.i) {
+ match c {
+ b'0'..=b'9' | b'.' | b'e' | b'E' | b'+' | b'-' => self.i += 1,
+ _ => break,
+ }
+ }
+ let s = std::str::from_utf8(&self.b[start..self.i])
+ .map_err(|_| json_err(format!("invalid utf-8 in number at byte {start}")))?;
+ s.parse::<f64>()
+ .map(Json::Num)
+ .map_err(|_| json_err(format!("bad number {s:?} at byte {start}")))
+ }
+
+ fn string(&mut self) -> JsonResult<String> {
+ self.eat(b'"')?;
+ let mut out = String::new();
+ loop {
+ let c = *self
+ .b
+ .get(self.i)
+ .ok_or_else(|| json_err("unterminated string"))?;
+ self.i += 1;
+ match c {
+ b'"' => break,
+ b'\\' => {
+ let e = *self
+ .b
+ .get(self.i)
+ .ok_or_else(|| json_err("unterminated escape"))?;
+ self.i += 1;
+ match e {
+ b'"' => out.push('"'),
+ b'\\' => out.push('\\'),
+ b'/' => out.push('/'),
+ b'b' => out.push('\u{8}'),
+ b'f' => out.push('\u{c}'),
+ b'n' => out.push('\n'),
+ b'r' => out.push('\r'),
+ b't' => out.push('\t'),
+ b'u' => out.push(self.unicode_escape()?),
+ other => {
+ return Err(json_err(format!("bad escape \\{:?}", other as char)));
+ }
+ }
+ }
+ // A raw control character is not legal JSON. Refusing it
+ // keeps this reader honest rather than lenient, which is
+ // the whole posture of this module.
+ 0x00..=0x1F => {
+ return Err(json_err(format!(
+ "raw control character {c:#04x} in string at byte {}",
+ self.i - 1
+ )));
+ }
+ _ => {
+ // Multi-byte UTF-8 passes through untouched; the
+ // input was a &str, so it is already valid.
+ let start = self.i - 1;
+ let len = utf8_len(c);
+ let end = start + len;
+ if end > self.b.len() {
+ return Err(json_err("truncated utf-8 sequence"));
+ }
+ let s = std::str::from_utf8(&self.b[start..end])
+ .map_err(|_| json_err("invalid utf-8 in string"))?;
+ out.push_str(s);
+ self.i = end;
+ }
+ }
+ }
+ Ok(out)
+ }
+
+ /// A `\uXXXX` escape, joining a surrogate pair when one follows.
+ fn unicode_escape(&mut self) -> JsonResult<char> {
+ let hi = self.hex4()?;
+ if (0xD800..0xDC00).contains(&hi) {
+ // High surrogate: a low one must follow.
+ if self.b.get(self.i) == Some(&b'\\') && self.b.get(self.i + 1) == Some(&b'u') {
+ self.i += 2;
+ let lo = self.hex4()?;
+ if (0xDC00..0xE000).contains(&lo) {
+ let c = 0x10000 + (((hi - 0xD800) as u32) << 10) + (lo - 0xDC00) as u32;
+ return char::from_u32(c).ok_or_else(|| json_err("bad surrogate pair"));
+ }
+ return Err(json_err("unpaired high surrogate"));
+ }
+ return Err(json_err("unpaired high surrogate"));
+ }
+ char::from_u32(hi as u32).ok_or_else(|| json_err("bad \\u escape"))
+ }
+
+ fn hex4(&mut self) -> JsonResult<u16> {
+ let end = self.i + 4;
+ if end > self.b.len() {
+ return Err(json_err("truncated \\u escape"));
+ }
+ let s = std::str::from_utf8(&self.b[self.i..end])
+ .map_err(|_| json_err("invalid utf-8 in \\u escape"))?;
+ let v =
+ u16::from_str_radix(s, 16).map_err(|_| json_err(format!("bad \\u escape {s:?}")))?;
+ self.i = end;
+ Ok(v)
+ }
+
+ /// Refuses trailing content after the top-level value: `{}{}` is not
+ /// one document.
+ fn finish(&mut self) -> JsonResult<()> {
+ self.ws();
+ if self.i == self.b.len() {
+ Ok(())
+ } else {
+ Err(json_err(format!("trailing input at byte {}", self.i)))
+ }
+ }
+}
+
+fn utf8_len(b: u8) -> usize {
+ match b {
+ 0x00..=0x7F => 1,
+ 0xC0..=0xDF => 2,
+ 0xE0..=0xEF => 3,
+ _ => 4,
+ }
+}
+
+// --- the tree ---------------------------------------------------------
+
+enum Node {
+ /// A block or `list_item` element, with its children.
+ Element {
+ tag: String,
+ level: Option<u8>,
+ content: Vec<Node>,
+ },
+ /// One text run and the marks on it, in the order given.
+ Text { text: String, marks: Vec<String> },
+}
+
+fn schema_err(msg: impl Into<String>) -> BuildError {
+ BuildError::Schema(msg.into())
+}
+
+/// Reads one node object. Shape errors are `Json`; vocabulary and value
+/// errors are `Schema`.
+fn read_node(v: &Json) -> Result<Node, BuildError> {
+ let ty = match v.get("type") {
+ Some(Json::Str(s)) => s.as_str(),
+ Some(other) => return Err(json_err(format!("\"type\" is a {}", other.kind()))),
+ None => return Err(json_err("node has no \"type\"")),
+ };
+
+ if ty == "text" {
+ let text = match v.get("text") {
+ Some(Json::Str(s)) => s.clone(),
+ Some(other) => return Err(json_err(format!("\"text\" is a {}", other.kind()))),
+ None => return Err(json_err("text node has no \"text\"")),
+ };
+ // Canonical form has no empty runs: fragment_json drops them
+ // (fragment.rs:302), so one here means the producer is not
+ // emitting canonical trees and the round trip would not hold.
+ if text.is_empty() {
+ return Err(schema_err("empty text run"));
+ }
+ let mut marks = Vec::new();
+ if let Some(m) = v.get("marks") {
+ let Json::Arr(items) = m else {
+ return Err(json_err(format!("\"marks\" is a {}", m.kind())));
+ };
+ // Canonical form omits "marks" entirely when empty.
+ if items.is_empty() {
+ return Err(schema_err("empty \"marks\" array"));
+ }
+ for item in items {
+ let name = match item.get("type") {
+ Some(Json::Str(s)) => s.clone(),
+ Some(other) => {
+ return Err(json_err(format!("mark \"type\" is a {}", other.kind())))
+ }
+ None => return Err(json_err("mark has no \"type\"")),
+ };
+ if !MARKS.contains(&name.as_str()) {
+ return Err(schema_err(format!("unknown mark {name:?}")));
+ }
+ if marks.contains(&name) {
+ return Err(schema_err(format!("mark {name:?} repeated on one run")));
+ }
+ marks.push(name);
+ }
+ }
+ return Ok(Node::Text { text, marks });
+ }
+
+ let level = if ty == "heading" {
+ match v.get("attrs").and_then(|a| a.get("level")) {
+ Some(Json::Num(n)) => {
+ let n = *n;
+ if n.fract() != 0.0 || !(1.0..=6.0).contains(&n) {
+ return Err(schema_err(format!("heading level {n} outside 1–6")));
+ }
+ Some(n as u8)
+ }
+ Some(other) => return Err(schema_err(format!("heading level is a {}", other.kind()))),
+ None => return Err(schema_err("heading has no level")),
+ }
+ } else {
+ if v.get("attrs").is_some() {
+ return Err(schema_err(format!("{ty} carries attrs")));
+ }
+ None
+ };
+
+ let mut content = Vec::new();
+ match v.get("content") {
+ Some(Json::Arr(items)) => {
+ for item in items {
+ content.push(read_node(item)?);
+ }
+ }
+ Some(other) => return Err(json_err(format!("\"content\" is a {}", other.kind()))),
+ None => return Err(json_err(format!("{ty} has no \"content\""))),
+ }
+
+ Ok(Node::Element {
+ tag: ty.to_string(),
+ level,
+ content,
+ })
+}
+
+fn is_block(tag: &str) -> bool {
+ matches!(
+ tag,
+ "paragraph" | "heading" | "bullet_list" | "ordered_list" | "blockquote"
+ )
+}
+
+fn tag_of(n: &Node) -> &str {
+ match n {
+ Node::Element { tag, .. } => tag,
+ Node::Text { .. } => "text",
+ }
+}
+
+/// The editor's content model (`fixtures/editor-entry.js`'s schema).
+/// See the module comment for why the round trip does not cover this.
+fn validate(n: &Node) -> Result<(), BuildError> {
+ let Node::Element { tag, content, .. } = n else {
+ return Ok(());
+ };
+ match tag.as_str() {
+ // paragraph | heading -> text* (a subset of the schema's
+ // inline*, since import emits no other inline node).
+ "paragraph" | "heading" => {
+ for c in content {
+ if !matches!(c, Node::Text { .. }) {
+ return Err(schema_err(format!("{tag} contains a {}", tag_of(c))));
+ }
+ }
+ }
+ // bullet_list | ordered_list -> list_item+
+ "bullet_list" | "ordered_list" => {
+ if content.is_empty() {
+ return Err(schema_err(format!("{tag} is empty")));
+ }
+ for c in content {
+ if tag_of(c) != "list_item" {
+ return Err(schema_err(format!("{tag} contains a {}", tag_of(c))));
+ }
+ }
+ }
+ // list_item -> paragraph block*
+ "list_item" => {
+ match content.first() {
+ Some(first) if tag_of(first) == "paragraph" => {}
+ Some(other) => {
+ return Err(schema_err(format!(
+ "list_item starts with a {}, not a paragraph",
+ tag_of(other)
+ )))
+ }
+ None => return Err(schema_err("list_item is empty")),
+ }
+ for c in &content[1..] {
+ if !is_block(tag_of(c)) {
+ return Err(schema_err(format!("list_item contains a {}", tag_of(c))));
+ }
+ }
+ }
+ // blockquote -> block+
+ "blockquote" => {
+ if content.is_empty() {
+ return Err(schema_err("blockquote is empty"));
+ }
+ for c in content {
+ if !is_block(tag_of(c)) {
+ return Err(schema_err(format!("blockquote contains a {}", tag_of(c))));
+ }
+ }
+ }
+ other => return Err(schema_err(format!("unknown node {other:?}"))),
+ }
+ for c in content {
+ validate(c)?;
+ }
+ Ok(())
+}
+
+// --- building ---------------------------------------------------------
+
+/// The `Attrs` for one run: every mark it carries, as an empty map.
+///
+/// An empty map, not `true`: y-prosemirror writes a mark as
+/// `{ name: mark.attrs }` and every mark this build knows is
+/// attribute-free, so `{}` is what the browser puts on the wire. A `true`
+/// renders identically through `fragment_json` and differs in the CRDT,
+/// which makes y-prosemirror rewrite the run on the first edit.
+fn run_attrs(marks: &[String]) -> Attrs {
+ let mut attrs = Attrs::new();
+ for m in marks {
+ attrs.insert(
+ Arc::from(m.as_str()),
+ Any::Map(Arc::new(HashMap::<String, Any>::new())),
+ );
+ }
+ attrs
+}
+
+/// Inserts a block's runs into one `XmlText`, each through
+/// `insert_with_attributes`.
+///
+/// Never plain `insert`: `insert` does no attribute handling
+/// (`text.rs:212`), so a run following a bold one inherits the bold.
+/// `insert_with_attributes` unsets whatever the caller omitted
+/// (`text.rs:707`'s `unset_missing`), which is the same step Yjs's own
+/// `insertText` performs — so an unmarked run passes `Attrs::new()` and
+/// comes out unmarked.
+fn insert_runs(txn: &mut TransactionMut, text: &XmlTextRef, runs: &[(&str, &[String])]) {
+ let mut at = 0;
+ for (s, marks) in runs {
+ text.insert_with_attributes(txn, at, s, run_attrs(marks));
+ at = text.len(txn);
+ }
+}
+
+fn build_children(
+ txn: &mut TransactionMut,
+ parent: &XmlElementRef,
+ children: &[Node],
+) -> Result<(), BuildError> {
+ // A block's text runs share one XmlText (see the module comment), so
+ // gather them before touching the CRDT.
+ let runs: Vec<(&str, &[String])> = children
+ .iter()
+ .filter_map(|c| match c {
+ Node::Text { text, marks } => Some((text.as_str(), marks.as_slice())),
+ _ => None,
+ })
+ .collect();
+ if !runs.is_empty() {
+ let text = parent.push_back(txn, XmlTextPrelim::new(""));
+ insert_runs(txn, &text, &runs);
+ }
+
+ for child in children {
+ if let Node::Element { .. } = child {
+ build_element(txn, parent, child)?;
+ }
+ }
+ Ok(())
+}
+
+fn build_element(
+ txn: &mut TransactionMut,
+ parent: &XmlElementRef,
+ node: &Node,
+) -> Result<(), BuildError> {
+ let Node::Element {
+ tag,
+ level,
+ content,
+ } = node
+ else {
+ return Ok(());
+ };
+ let el = parent.push_back(txn, XmlElementPrelim::empty(tag.as_str()));
+ if let Some(level) = level {
+ set_level(txn, &el, *level);
+ }
+ build_children(txn, &el, content)
+}
+
+/// Builds `tree_json` into `doc`'s ProseMirror root.
+///
+/// The document must be empty: this builds documents rather than
+/// appending to them, and the one caller only ever builds a new one.
+pub fn build_fragment(doc: &Doc, tree_json: &str) -> Result<(), BuildError> {
+ let mut p = Parser::new(tree_json);
+ let root_json = p.value()?;
+ p.finish()?;
+
+ if !matches!(root_json.get("type"), Some(Json::Str(s)) if s == "doc") {
+ return Err(schema_err("root node is not a doc"));
+ }
+ let Some(Json::Arr(items)) = root_json.get("content") else {
+ return Err(json_err("doc has no \"content\" array"));
+ };
+ // doc -> block+
+ if items.is_empty() {
+ return Err(schema_err("document has no blocks"));
+ }
+
+ let mut blocks = Vec::with_capacity(items.len());
+ for item in items {
+ let node = read_node(item)?;
+ if !is_block(tag_of(&node)) {
+ return Err(schema_err(format!("doc contains a {}", tag_of(&node))));
+ }
+ validate(&node)?;
+ blocks.push(node);
+ }
+
+ let root = doc.get_or_insert_xml_fragment(ROOT);
+ let mut txn = doc.transact_mut();
+ if root.len(&txn) > 0 {
+ return Err(BuildError::RootNotEmpty);
+ }
+ for block in &blocks {
+ let Node::Element {
+ tag,
+ level,
+ content,
+ } = block
+ else {
+ continue;
+ };
+ let el = root.push_back(&mut txn, XmlElementPrelim::empty(tag.as_str()));
+ if let Some(level) = level {
+ set_level(&mut txn, &el, *level);
+ }
+ build_children(&mut txn, &el, content)?;
+ }
+ Ok(())
+}
+
+/// Sets a heading's `level` — as a **string**, which is not a choice.
+///
+/// The editor writes a number: `fixtures/toolbar.mjs:167` computes
+/// `Number(value.split('-')[1])` and y-prosemirror passes the attribute
+/// straight through (`sync-plugin.js:874`). Matching that from Rust
+/// would be better, and **yrs 0.23.5's public API cannot do it**. The
+/// only attribute setter the `Xml` trait exposes is `insert_attribute`,
+/// typed `V: Into<String>` (`xml.rs:1002`), so a string is the only
+/// value it can store. The typed path — an `XmlElement`'s attributes
+/// are the same branch map a `Y.Map` uses, so a `MapRef` over it would
+/// take an `Any` — is closed: `MapRef` wraps `BranchPtr`
+/// (`map.rs:62, :512`) and `BranchPtr` is private to the crate. There
+/// is no third way short of forking yrs, and the version is pinned
+/// (`=0.23.5`) because it is half of a wire contract.
+///
+/// What this costs, stated plainly rather than discovered later:
+///
+/// - It is what this repo's own fixtures already contain
+/// (`fixtures/generate-fragment.mjs:50` writes `{ level: '2' }`), and
+/// the editor opens such a document — `yXmlFragmentToProseMirrorRootNode`
+/// against the real schema produces the right tree, verified by spike.
+/// - `fragment_json` renders it identically either way: it reads through
+/// `get_attribute`, which stringifies whatever is stored
+/// (`xml.rs:1026`), then parses. So the round trip cannot see this and
+/// neither can any Rust test — see `heading_level_survives_as_a_level`.
+/// - It differs from what the browser would have written, so
+/// y-prosemirror's attribute comparison may rewrite an imported
+/// heading on its first edit. That is CRDT churn, not lost text.
+/// - `prosemirror-model` 1.25 declares `level` as `validate: "number"`,
+/// so an explicit `node.check()` rejects a string level. Nothing in
+/// the editor calls `check()` today, which is why this works — but it
+/// is a live tripwire for any future code that validates a document,
+/// and it applies to every heading in this repo's fixtures too, not
+/// only to imported ones.
+fn set_level(txn: &mut TransactionMut, el: &XmlElementRef, level: u8) {
+ el.insert_attribute(txn, "level", level.to_string());
+}
diff --git a/core/src/lib.rs b/core/src/lib.rs
index e5fb37e..449df8e 100644
--- a/core/src/lib.rs
+++ b/core/src/lib.rs
@@ -54,8 +54,10 @@
//! above) rather than relying on `fastrand` to fail loudly when it can't —
//! it won't.
pub mod abi;
+mod build;
mod fragment;
mod snapshot;
+pub use build::{build_fragment, BuildError};
pub use fragment::{fragment_html, fragment_json, fragment_text, ROOT as FRAGMENT_ROOT};
pub use snapshot::{snapshot_bytes, verify_snapshot};
diff --git a/core/tests/build.rs b/core/tests/build.rs
new file mode 100644
index 0000000..aa87abb
--- /dev/null
+++ b/core/tests/build.rs
@@ -0,0 +1,516 @@
+//! `build_fragment` — the import half of the interchange (the import
+//! design, §3 and §5).
+//!
+//! The headline property is the round trip: for any canonical tree,
+//! `fragment_json(build(tree))` is structurally equal to the tree. That
+//! is what makes "the importer emitted something the core silently
+//! dropped" a test failure rather than a person's missing paragraph.
+//!
+//! Structurally equal, via `serde_json::Value`, never string equality:
+//! `fragment_json` writes JSON by hand and a Go `encoding/json` encoding
+//! of the same tree differs in escaping and key order without differing
+//! in meaning.
+
+use docs_core::{build_fragment, fragment_json, BuildError};
+use serde_json::Value;
+use yrs::types::xml::XmlFragment;
+use yrs::{Doc, Options, Text, Transact};
+
+fn doc() -> Doc {
+ Doc::with_options(Options::with_guid_and_client_id("build-test".into(), 1))
+}
+
+/// Builds `json` and asserts the core renders exactly it back.
+fn round_trip(json: &str) {
+ let d = doc();
+ build_fragment(&d, json).expect("build must succeed");
+ let got: Value = serde_json::from_str(&fragment_json(&d)).expect("rendered json parses");
+ let want: Value = serde_json::from_str(json).expect("input json parses");
+ assert_eq!(got, want, "round trip");
+}
+
+fn err(json: &str) -> BuildError {
+ build_fragment(&doc(), json).expect_err("build must be refused")
+}
+
+// --- the round trip ---
+
+#[test]
+fn round_trips_a_paragraph() {
+ round_trip(
+ r#"{"type":"doc","content":[{"type":"paragraph","content":[{"type":"text","text":"Hello."}]}]}"#,
+ );
+}
+
+#[test]
+fn round_trips_an_empty_paragraph() {
+ round_trip(r#"{"type":"doc","content":[{"type":"paragraph","content":[]}]}"#);
+}
+
+#[test]
+fn round_trips_headings_at_every_level() {
+ for level in 1..=6 {
+ round_trip(&format!(
+ r#"{{"type":"doc","content":[{{"type":"heading","attrs":{{"level":{level}}},"content":[{{"type":"text","text":"H"}}]}}]}}"#
+ ));
+ }
+}
+
+#[test]
+fn round_trips_marked_runs() {
+ round_trip(concat!(
+ r#"{"type":"doc","content":[{"type":"paragraph","content":["#,
+ r#"{"type":"text","text":"plain and "},"#,
+ r#"{"type":"text","text":"bold","marks":[{"type":"strong"}]},"#,
+ r#"{"type":"text","text":" and "},"#,
+ r#"{"type":"text","text":"italic","marks":[{"type":"em"}]}]}]}"#
+ ));
+}
+
+#[test]
+fn round_trips_every_mark_this_build_knows() {
+ for mark in [
+ "font_sans",
+ "font_mono",
+ "size_sm",
+ "size_lg",
+ "size_xl",
+ "strong",
+ "em",
+ "code",
+ ] {
+ round_trip(&format!(
+ r#"{{"type":"doc","content":[{{"type":"paragraph","content":[{{"type":"text","text":"x","marks":[{{"type":"{mark}"}}]}}]}}]}}"#
+ ));
+ }
+}
+
+// Marks come back in MARKS order (fragment.rs's mark_names), so a run
+// carrying two of them round-trips only if the builder does not depend
+// on the order it was given.
+#[test]
+fn round_trips_a_run_with_two_marks() {
+ round_trip(concat!(
+ r#"{"type":"doc","content":[{"type":"paragraph","content":["#,
+ r#"{"type":"text","text":"both","marks":[{"type":"font_sans"},{"type":"strong"}]}]}]}"#
+ ));
+}
+
+#[test]
+fn round_trips_lists_and_blockquotes() {
+ round_trip(concat!(
+ r#"{"type":"doc","content":[{"type":"bullet_list","content":["#,
+ r#"{"type":"list_item","content":[{"type":"paragraph","content":[{"type":"text","text":"One"}]}]},"#,
+ r#"{"type":"list_item","content":[{"type":"paragraph","content":[{"type":"text","text":"Two"}]}]}]},"#,
+ r#"{"type":"ordered_list","content":["#,
+ r#"{"type":"list_item","content":[{"type":"paragraph","content":[{"type":"text","text":"First"}]}]}]},"#,
+ r#"{"type":"blockquote","content":[{"type":"paragraph","content":[{"type":"text","text":"Quoted"}]}]}]}"#
+ ));
+}
+
+// A list nested inside a list item -- the deepest structure the editor's
+// own schema permits (list_item -> paragraph block*).
+#[test]
+fn round_trips_a_nested_list() {
+ round_trip(concat!(
+ r#"{"type":"doc","content":[{"type":"bullet_list","content":["#,
+ r#"{"type":"list_item","content":["#,
+ r#"{"type":"paragraph","content":[{"type":"text","text":"Outer"}]},"#,
+ r#"{"type":"bullet_list","content":[{"type":"list_item","content":["#,
+ r#"{"type":"paragraph","content":[{"type":"text","text":"Inner"}]}]}]}]}]}]}"#
+ ));
+}
+
+// Text a document can really contain: quotes and backslashes (which the
+// JSON parser must unescape), astral-plane characters (surrogate pairs
+// if the producer escaped them), and control characters.
+#[test]
+fn round_trips_text_needing_escapes() {
+ round_trip(concat!(
+ r#"{"type":"doc","content":[{"type":"paragraph","content":["#,
+ r#"{"type":"text","text":"she said \"hi\" \\ then\tstopped"}]}]}"#
+ ));
+}
+
+#[test]
+fn round_trips_escaped_astral_characters() {
+ let d = doc();
+ build_fragment(
+ &d,
+ r#"{"type":"doc","content":[{"type":"paragraph","content":[{"type":"text","text":"😀 ok"}]}]}"#,
+ )
+ .expect("build must succeed");
+ let json = fragment_json(&d);
+ assert!(json.contains('\u{1F600}'), "got {json}");
+}
+
+// --- the F3 regression: runs must not inherit the previous run's marks ---
+//
+// Text inserted with plain `insert` at the right edge of a bold run
+// inherits the bold. The builder therefore inserts every run through
+// insert_with_attributes, with an empty Attrs for an unmarked run, which
+// unsets whatever was active. Without that, the second run below comes
+// back carrying "strong".
+#[test]
+fn an_unmarked_run_after_a_marked_one_carries_no_mark() {
+ let d = doc();
+ build_fragment(
+ &d,
+ concat!(
+ r#"{"type":"doc","content":[{"type":"paragraph","content":["#,
+ r#"{"type":"text","text":"bold","marks":[{"type":"strong"}]},"#,
+ r#"{"type":"text","text":" plain"}]}]}"#
+ ),
+ )
+ .expect("build must succeed");
+ assert_eq!(
+ fragment_json(&d),
+ concat!(
+ r#"{"type":"doc","content":[{"type":"paragraph","content":["#,
+ r#"{"type":"text","text":"bold","marks":[{"type":"strong"}]},"#,
+ r#"{"type":"text","text":" plain"}]}]}"#
+ )
+ );
+}
+
+// The same hazard in the other direction: two differently-marked runs
+// must not merge their marks.
+#[test]
+fn adjacent_runs_keep_their_own_marks() {
+ let d = doc();
+ build_fragment(
+ &d,
+ concat!(
+ r#"{"type":"doc","content":[{"type":"paragraph","content":["#,
+ r#"{"type":"text","text":"a","marks":[{"type":"strong"}]},"#,
+ r#"{"type":"text","text":"b","marks":[{"type":"em"}]}]}]}"#
+ ),
+ )
+ .expect("build must succeed");
+ let json = fragment_json(&d);
+ assert!(
+ json.contains(r#"{"type":"text","text":"a","marks":[{"type":"strong"}]}"#),
+ "got {json}"
+ );
+ assert!(
+ json.contains(r#"{"type":"text","text":"b","marks":[{"type":"em"}]}"#),
+ "got {json}"
+ );
+}
+
+// --- wire shape: what the browser will compare against ---
+//
+// y-prosemirror writes a mark as `{ name: mark.attrs }`, i.e. an empty
+// map for every attribute-free mark (sync-plugin.js:856), and the repo's
+// own fixtures agree (generate-fragment.mjs's `{ strong: {} }`). A
+// builder writing `true` renders identically through fragment_json but
+// differs from what the browser writes, so y-prosemirror's attribute
+// comparison rewrites the run on the first edit. fragment_json cannot
+// see the difference, so this test reads the CRDT directly.
+#[test]
+fn a_mark_is_stored_as_an_empty_map_not_a_boolean() {
+ let d = doc();
+ build_fragment(
+ &d,
+ r#"{"type":"doc","content":[{"type":"paragraph","content":[{"type":"text","text":"x","marks":[{"type":"strong"}]}]}]}"#,
+ )
+ .expect("build must succeed");
+
+ let frag = d.get_or_insert_xml_fragment("prosemirror");
+ let txn = d.transact();
+ let para = match frag.get(&txn, 0).expect("one paragraph") {
+ yrs::types::xml::XmlOut::Element(e) => e,
+ other => panic!("expected an element, got {other:?}"),
+ };
+ let text = match para.get(&txn, 0).expect("one text node") {
+ yrs::types::xml::XmlOut::Text(t) => t,
+ other => panic!("expected a text node, got {other:?}"),
+ };
+ let chunks = text.diff(&txn, yrs::types::text::YChange::identity);
+ let attrs = chunks[0]
+ .attributes
+ .as_ref()
+ .expect("the run carries marks");
+ match attrs.get("strong").expect("strong is present") {
+ yrs::Any::Map(m) => assert!(m.is_empty(), "strong must be an empty map, got {m:?}"),
+ other => panic!("strong must be a map like y-prosemirror writes, got {other:?}"),
+ }
+}
+
+// y-prosemirror groups all adjacent ProseMirror text nodes in a block
+// into ONE Y.XmlText (sync-plugin.js's normalizePNodeContent), and
+// fragment_json emits one `text_json` call per XmlText child, never
+// merging across two siblings. A builder that gave each run its own
+// XmlText would therefore still round-trip -- and would be rewritten by
+// the editor on the first edit, because the CRDT shape differs from
+// what the browser would have written. Runs are inserted into a single
+// XmlText per block.
+#[test]
+fn a_block_holds_exactly_one_xml_text_however_many_runs() {
+ let d = doc();
+ build_fragment(
+ &d,
+ concat!(
+ r#"{"type":"doc","content":[{"type":"paragraph","content":["#,
+ r#"{"type":"text","text":"a","marks":[{"type":"strong"}]},"#,
+ r#"{"type":"text","text":"b"},"#,
+ r#"{"type":"text","text":"c","marks":[{"type":"em"}]}]}]}"#
+ ),
+ )
+ .expect("build must succeed");
+
+ let frag = d.get_or_insert_xml_fragment("prosemirror");
+ let txn = d.transact();
+ let para = match frag.get(&txn, 0).expect("one paragraph") {
+ yrs::types::xml::XmlOut::Element(e) => e,
+ other => panic!("expected an element, got {other:?}"),
+ };
+ assert_eq!(
+ para.len(&txn),
+ 1,
+ "a block's runs share one XmlText, as y-prosemirror writes them"
+ );
+}
+
+// A heading's level survives the build and comes back as the level it
+// went in as.
+//
+// This asserts only what Rust can see, and that is less than it looks:
+// `fragment_json` reads the attribute through `get_attribute`, which
+// stringifies whatever is stored (xml.rs:1026), then parses and writes
+// it unquoted. So this test passes whether the attribute is stored as a
+// string, a number or a bigint, and no Rust test can tell them apart.
+//
+// It is stored as a string, because yrs 0.23.5 cannot store anything
+// else -- see set_level's own comment for the full reasoning and what
+// it costs. The distinction is only observable from the browser, so the
+// assertion that belongs to it lives in the cross-implementation gate
+// (the import design §13), where `typeof` is meaningful.
+#[test]
+fn heading_level_survives_as_a_level() {
+ let d = doc();
+ build_fragment(
+ &d,
+ r#"{"type":"doc","content":[{"type":"heading","attrs":{"level":3},"content":[{"type":"text","text":"H"}]}]}"#,
+ )
+ .expect("build must succeed");
+ let json = fragment_json(&d);
+ assert!(json.contains(r#""attrs":{"level":3}"#), "got {json}");
+ assert!(
+ !json.contains(r#""level":"3""#),
+ "level must not be a string, got {json}"
+ );
+}
+
+// --- refusals: the builder is strict where the readers are lenient ---
+//
+// The renderers carry unknown nodes and marks through on purpose (an
+// older binary must not lose a newer editor's text). A builder meeting
+// one has been handed something its only caller could not have meant,
+// and building it would write the mistake into a document permanently.
+
+#[test]
+fn refuses_an_unknown_node() {
+ assert!(matches!(
+ err(r#"{"type":"doc","content":[{"type":"table","content":[]}]}"#),
+ BuildError::Schema(_)
+ ));
+}
+
+#[test]
+fn refuses_an_unknown_mark() {
+ assert!(matches!(
+ err(
+ r#"{"type":"doc","content":[{"type":"paragraph","content":[{"type":"text","text":"x","marks":[{"type":"link"}]}]}]}"#
+ ),
+ BuildError::Schema(_)
+ ));
+}
+
+#[test]
+fn refuses_a_root_that_is_not_a_doc() {
+ assert!(matches!(
+ err(r#"{"type":"paragraph","content":[]}"#),
+ BuildError::Schema(_)
+ ));
+}
+
+#[test]
+fn refuses_a_level_outside_one_to_six() {
+ for level in ["0", "7", "99", "-1"] {
+ assert!(
+ matches!(
+ err(&format!(
+ r#"{{"type":"doc","content":[{{"type":"heading","attrs":{{"level":{level}}},"content":[]}}]}}"#
+ )),
+ BuildError::Schema(_)
+ ),
+ "level {level} must be refused"
+ );
+ }
+}
+
+// A heading with no level at all: fragment_json always emits one, so an
+// absent level means the producer is not emitting canonical trees.
+#[test]
+fn refuses_a_heading_with_no_level() {
+ assert!(matches!(
+ err(r#"{"type":"doc","content":[{"type":"heading","content":[]}]}"#),
+ BuildError::Schema(_)
+ ));
+}
+
+// --- the content model (the import design §3) ---
+//
+// The round trip alone does not prove a tree is one the editor can
+// open: fragment_json will happily render a text node directly under
+// doc, so a broken importer would pass the round trip and break the
+// editor. These are the rules that close that gap.
+
+#[test]
+fn refuses_text_directly_under_doc() {
+ assert!(matches!(
+ err(r#"{"type":"doc","content":[{"type":"text","text":"loose"}]}"#),
+ BuildError::Schema(_)
+ ));
+}
+
+#[test]
+fn refuses_a_list_item_outside_a_list() {
+ assert!(matches!(
+ err(
+ r#"{"type":"doc","content":[{"type":"list_item","content":[{"type":"paragraph","content":[]}]}]}"#
+ ),
+ BuildError::Schema(_)
+ ));
+}
+
+#[test]
+fn refuses_a_block_inside_a_paragraph() {
+ assert!(matches!(
+ err(
+ r#"{"type":"doc","content":[{"type":"paragraph","content":[{"type":"paragraph","content":[]}]}]}"#
+ ),
+ BuildError::Schema(_)
+ ));
+}
+
+#[test]
+fn refuses_a_list_with_a_non_list_item_child() {
+ assert!(matches!(
+ err(
+ r#"{"type":"doc","content":[{"type":"bullet_list","content":[{"type":"paragraph","content":[]}]}]}"#
+ ),
+ BuildError::Schema(_)
+ ));
+}
+
+#[test]
+fn refuses_an_empty_list() {
+ assert!(matches!(
+ err(r#"{"type":"doc","content":[{"type":"bullet_list","content":[]}]}"#),
+ BuildError::Schema(_)
+ ));
+}
+
+// list_item -> paragraph block*: the first child must be a paragraph,
+// which is what prosemirror-schema-list's addListNodes(…, 'paragraph
+// block*', 'block') means (fixtures/editor-entry.js).
+#[test]
+fn refuses_a_list_item_not_starting_with_a_paragraph() {
+ assert!(matches!(
+ err(concat!(
+ r#"{"type":"doc","content":[{"type":"bullet_list","content":[{"type":"list_item","content":["#,
+ r#"{"type":"bullet_list","content":[{"type":"list_item","content":[{"type":"paragraph","content":[]}]}]}]}]}]}"#
+ )),
+ BuildError::Schema(_)
+ ));
+}
+
+#[test]
+fn refuses_an_empty_document() {
+ assert!(matches!(
+ err(r#"{"type":"doc","content":[]}"#),
+ BuildError::Schema(_)
+ ));
+}
+
+#[test]
+fn refuses_text_in_a_blockquote_without_a_paragraph() {
+ assert!(matches!(
+ err(
+ r#"{"type":"doc","content":[{"type":"blockquote","content":[{"type":"text","text":"x"}]}]}"#
+ ),
+ BuildError::Schema(_)
+ ));
+}
+
+// --- malformed input ---
+
+#[test]
+fn refuses_malformed_json() {
+ for bad in [
+ "",
+ "{",
+ "not json",
+ r#"{"type":"doc","content":}"#,
+ r#"{"type":"doc","content":[}]}"#,
+ "\u{0}",
+ ] {
+ assert!(
+ matches!(err(bad), BuildError::Json(_)),
+ "{bad:?} must be refused as malformed JSON"
+ );
+ }
+}
+
+#[test]
+fn refuses_a_missing_text_field() {
+ assert!(matches!(
+ err(r#"{"type":"doc","content":[{"type":"paragraph","content":[{"type":"text"}]}]}"#),
+ BuildError::Json(_) | BuildError::Schema(_)
+ ));
+}
+
+// Deep nesting is how a hostile tree turns a recursive descent into a
+// stack overflow, which is not a catchable error in Rust -- it aborts
+// the whole wasm instance. The cap is checked while parsing, before any
+// recursion goes deep.
+#[test]
+fn refuses_a_tree_deeper_than_the_cap() {
+ let mut json = String::from(r#"{"type":"doc","content":["#);
+ let depth = 500;
+ for _ in 0..depth {
+ json.push_str(r#"{"type":"blockquote","content":["#);
+ }
+ json.push_str(r#"{"type":"paragraph","content":[]}"#);
+ for _ in 0..depth {
+ json.push_str("]}");
+ }
+ json.push_str("]}");
+ assert!(matches!(err(&json), BuildError::TooDeep));
+}
+
+// The same shape, but as unclosed brackets: the parser must hit the cap
+// rather than recursing on the way in.
+#[test]
+fn refuses_deeply_nested_arrays_while_parsing() {
+ let json = format!("{}{}", r#"{"type":"doc","content":"#, "[".repeat(5000));
+ assert!(matches!(
+ err(&json),
+ BuildError::TooDeep | BuildError::Json(_)
+ ));
+}
+
+// --- building is for new documents only ---
+
+#[test]
+fn refuses_a_root_that_already_has_content() {
+ let d = doc();
+ let tree = r#"{"type":"doc","content":[{"type":"paragraph","content":[{"type":"text","text":"first"}]}]}"#;
+ build_fragment(&d, tree).expect("the first build must succeed");
+ assert!(matches!(
+ build_fragment(&d, tree).expect_err("the second must be refused"),
+ BuildError::RootNotEmpty
+ ));
+}
diff --git a/fixtures/verify-import.mjs b/fixtures/verify-import.mjs
new file mode 100644
index 0000000..ca59e3f
--- /dev/null
+++ b/fixtures/verify-import.mjs
@@ -0,0 +1,167 @@
+// Reads the update core/src/bin/emit_import.rs writes on stdout and
+// checks that the real editor can open it.
+//
+// This is the gate the whole import design rests on. Every Rust test of
+// the builder reads the document back through the same yrs that wrote
+// it, so Rust and Rust agree even where both are wrong about what the
+// browser expects. Only this script asks the question that matters: does
+// upstream yjs, plus y-prosemirror, plus the editor's own schema, turn
+// these bytes back into the document they started as?
+//
+// Paired with fixtures/verify-rust.mjs, which asks the same question of
+// a bare Y.Text; run by scripts/wire-gate.sh.
+
+import * as Y from 'yjs'
+import { Schema } from 'prosemirror-model'
+import { schema as basic } from 'prosemirror-schema-basic'
+import { addListNodes } from 'prosemirror-schema-list'
+import { yXmlFragmentToProseMirrorRootNode } from 'y-prosemirror'
+
+// ---------------------------------------------------------------------
+// MIRRORS fixtures/editor-entry.js's schema. The bundle builds its
+// schema inline, and this file cannot import it without turning the
+// editor's entry point into a module and rebuilding the committed,
+// content-hashed bundle -- so the two definitions are duplicated for
+// now, and a drift between them would make this gate quietly test the
+// wrong schema. Extracting a shared fixtures/editor-schema.mjs that both
+// import is a task in the import plan; until it lands, a change to the
+// editor's schema must be made here too.
+// ---------------------------------------------------------------------
+function spanMark(cls, group) {
+ return {
+ group,
+ excludes: group,
+ parseDOM: [{ tag: 'span.' + cls }],
+ toDOM: () => ['span', { class: cls }, 0],
+ }
+}
+const marks = basic.spec.marks
+ .addToStart('size_xl', spanMark('docs-size-xl', 'fontSize'))
+ .addToStart('size_lg', spanMark('docs-size-lg', 'fontSize'))
+ .addToStart('size_sm', spanMark('docs-size-sm', 'fontSize'))
+ .addToStart('font_mono', spanMark('docs-font-mono', 'fontFamily'))
+ .addToStart('font_sans', spanMark('docs-font-sans', 'fontFamily'))
+const schema = new Schema({
+ nodes: addListNodes(basic.spec.nodes, 'paragraph block*', 'block'),
+ marks,
+})
+
+// The document emit_import.rs built, as ProseMirror's own JSON.
+//
+// heading level is the STRING "2", not the number 2, and that is the
+// current, deliberate behaviour rather than a bug in this expectation:
+// yrs 0.23.5 has no public API that can store a non-string XML
+// attribute (its only setter is typed `V: Into<String>`, and the typed
+// path goes through a private BranchPtr), so a string is the only thing
+// the Rust builder can write. core/src/build.rs's set_level carries the
+// full reasoning and what it costs. If this ever becomes the number 2 --
+// a yrs upgrade, or a different write path -- that is a real change and
+// this line is where it announces itself.
+const EXPECTED = {
+ type: 'doc',
+ content: [
+ { type: 'heading', attrs: { level: '2' }, content: [{ type: 'text', text: 'Quarterly Report' }] },
+ {
+ type: 'paragraph',
+ content: [
+ { type: 'text', text: 'plain and ' },
+ { type: 'text', marks: [{ type: 'strong' }], text: 'bold' },
+ { type: 'text', text: ' and ' },
+ { type: 'text', marks: [{ type: 'font_sans' }, { type: 'strong' }], text: 'both' },
+ ],
+ },
+ {
+ type: 'bullet_list',
+ content: [
+ {
+ type: 'list_item',
+ content: [
+ { type: 'paragraph', content: [{ type: 'text', text: 'Outer' }] },
+ {
+ type: 'bullet_list',
+ content: [
+ { type: 'list_item', content: [{ type: 'paragraph', content: [{ type: 'text', text: 'Inner' }] }] },
+ ],
+ },
+ ],
+ },
+ ],
+ },
+ {
+ type: 'ordered_list',
+ attrs: { order: 1 },
+ content: [{ type: 'list_item', content: [{ type: 'paragraph', content: [{ type: 'text', text: 'First' }] }] }],
+ },
+ {
+ type: 'blockquote',
+ content: [{ type: 'paragraph', content: [{ type: 'text', marks: [{ type: 'em' }], text: 'Quoted' }] }],
+ },
+ ],
+}
+
+function readStdin() {
+ return new Promise((resolve, reject) => {
+ const chunks = []
+ process.stdin.on('data', (c) => chunks.push(c))
+ process.stdin.on('end', () => resolve(Buffer.concat(chunks)))
+ process.stdin.on('error', reject)
+ })
+}
+
+function fail(msg) {
+ console.error('verify-import: ' + msg)
+ process.exit(1)
+}
+
+const update = await readStdin()
+if (update.length === 0) fail('no update on stdin')
+
+const doc = new Y.Doc()
+try {
+ Y.applyUpdate(doc, update)
+} catch (e) {
+ fail('yjs could not apply the update Rust built: ' + e.message)
+}
+
+const frag = doc.getXmlFragment('prosemirror')
+
+let node
+try {
+ node = yXmlFragmentToProseMirrorRootNode(frag, schema)
+} catch (e) {
+ fail('y-prosemirror could not convert the fragment against the editor schema: ' + e.message)
+}
+
+const got = node.toJSON()
+if (JSON.stringify(got) !== JSON.stringify(EXPECTED)) {
+ console.error('verify-import: the editor would open a different document than was built.')
+ console.error('--- got ---')
+ console.error(JSON.stringify(got, null, 1))
+ console.error('--- want ---')
+ console.error(JSON.stringify(EXPECTED, null, 1))
+ process.exit(1)
+}
+
+// The structure is right; now check it would actually render. A heading
+// whose level is a string still has to reach <h2>, since that is what a
+// reader sees. toDOM is what the editor calls.
+const heading = node.child(0)
+const dom = heading.type.spec.toDOM(heading)
+if (dom[0] !== 'h2') fail(`a level-2 heading renders as ${JSON.stringify(dom[0])}, not "h2"`)
+
+// Deliberately NOT node.check(): prosemirror-model 1.25 declares
+// heading.level as validate:"number", so check() rejects a string level
+// -- for every heading in this repo's fixtures too, not only imported
+// ones. Nothing in the editor calls check(), which is why documents
+// work today. Asserting the real behaviour here, and saying why, is
+// honest; calling check() and skipping the test would not be.
+try {
+ node.check()
+ console.error('verify-import: NOTE — node.check() now passes. That means heading levels')
+ console.error(' became numbers somewhere. Good news, but update this script and')
+ console.error(' core/src/build.rs\'s set_level, which documents the opposite.')
+} catch {
+ // Expected today. See above.
+}
+
+console.log(`verify-import: ok — ${update.length} bytes, ${node.childCount} blocks, opened by yjs ${Y.default?.version ?? ''}`.trim())
diff --git a/internal/core/testdata/docs_core.wasm b/internal/core/testdata/docs_core.wasm
index 61a292a..7efd897 100755
Binary files a/internal/core/testdata/docs_core.wasm and b/internal/core/testdata/docs_core.wasm differ
diff --git a/scripts/wire-gate.sh b/scripts/wire-gate.sh
index 4b41870..636efe6 100755
--- a/scripts/wire-gate.sh
+++ b/scripts/wire-gate.sh
@@ -20,4 +20,13 @@ echo "--- Rust reads what JS wrote"
echo "--- JS reads what Rust wrote"
(cd core && cargo run --quiet --locked --bin emit_update) | node fixtures/verify-rust.mjs
+# The same direction, one level up: not "can JS read a Y.Text Rust
+# wrote" but "can the real editor open a document the importer built".
+# Every Rust test of build_fragment reads the document back through the
+# same yrs that wrote it, so Rust agrees with itself even where both
+# halves are wrong about what the browser expects. This is the only
+# check that asks yjs, y-prosemirror and the editor's own schema.
+echo "--- the editor opens what the importer built"
+(cd core && cargo run --quiet --locked --bin emit_import) | node fixtures/verify-import.mjs
+
echo "wire-gate: ok"