amadan
The fact box: validated facts above the search results (§8.8)
A search naming a kind — "phone number", "eircode", "vat number",
"iban", "email address" — shows the validated values carried by the
passages §4.1 kept, with the document's own label and a citation,
between the withholding notice and the result list. No model, no
egress, nothing stored, no index change, no migration.
Spec: §8.8, decision 15 (since amended — see the decision-15 branch),
gates 34-38 in §10 (38 struck before it was built). Plan:
FACT-BOX-PLAN.md.
The rulings that carry it: candidates come from the RULED results
only, so a withheld passage cannot contribute a value, a count, a
collision or a silenced kind; identity is the canonical value; two or
more candidates are enumerated and never picked between; four renders
nothing; the displayed span is the document's bytes and the label is
the document's words, never composed. Checksums for eircode, IBAN and
Irish VAT; phone and email are structural, and the spec says so.
Reviewed four times by a model that did not write it, all REQUEST
CHANGES, all addressed: spec draft 1 and draft 2 by Fable 5.1; the
implementation by Fable 5.1 (six of its own mutations survived, four
real gate holes, three live recogniser false positives, a quadratic
label pass at 48.5s/MB); the post-review fix commit by Fable 5.1 before
merge, which found the fix TRUNCATING real phone numbers and citing the
result ("+353 87 1234567" boxed as "+353 87 12345"). Swept against
every .md in the oficina tree (1,113 files, 49,246 paragraphs): zero
false positives outside the spec's own quoted examples.
First merged 2026-09-16 to a local main that was never pushed;
rebased onto origin/main f3e8c37 on 2026-09-27, conflict-free, make ci
green under the pinned toolchain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Squashed from land/fact-box into main.
11 files changed,
+3323
−5
FACT-BOX-PLAN.md+273 −0docs/superpowers/specs/2026-09-03-memoria-design.md+694 −4internal/memoria/fact/fact.go+835 −0internal/memoria/fact/fact_test.go+540 −0internal/memoria/factbox_test.go+313 −0internal/memoria/handlers.go+13 −0internal/memoria/static/memoria.css+36 −0internal/memoria/store.go+182 −1internal/memoria/templates/results.html+25 −0internal/memoriatest/factbox_test.go+391 −0tools/drive/main.go+21 −0
CI failed — run details
CI log
=== vet ===
go vet ./...
=== fmt ===
=== test ===
go test ./...
? amadan.net/oficina/memoria/cmd/memoria [no test files]
--- FAIL: TestCreditsMatchProductionDependencies (0.03s)
credits_inventory_test.go:36: production dependency graph unavailable exit status 1
FAIL
FAIL amadan.net/oficina/memoria/internal/memoria 0.707s
ok amadan.net/oficina/memoria/internal/memoria/chunk (cached)
ok amadan.net/oficina/memoria/internal/memoria/fact 0.336s
? amadan.net/oficina/memoria/internal/memoria/icons [no test files]
ok amadan.net/oficina/memoria/internal/memoria/index (cached)
ok amadan.net/oficina/memoria/internal/memoria/notion (cached)
ok amadan.net/oficina/memoria/internal/memoria/query (cached)
ok amadan.net/oficina/memoria/internal/memoria/seal (cached)
ok amadan.net/oficina/memoria/internal/memoria/sibling (cached)
ok amadan.net/oficina/memoria/internal/memoriatest 1.553s
? amadan.net/oficina/memoria/tools/bench [no test files]
? amadan.net/oficina/memoria/tools/drive [no test files]
? amadan.net/oficina/memoria/tools/seed [no test files]
? amadan.net/oficina/memoria/tools/vendorassets [no test files]
FAIL
make: *** [Makefile:62: test] Error 1
amadan: step test failed: exit status 2
amadan: job failed at step test — its output is above, not at the tail of this log
diff --git a/FACT-BOX-PLAN.md b/FACT-BOX-PLAN.md| new file mode 100644 |
| index 0000000..1fc2344 |
| --- /dev/null |
| +++ b/FACT-BOX-PLAN.md |
| @@ -0,0 +1,273 @@ |
| +# Implementation plan — §8.8's fact box |
| + |
| +Against `docs/superpowers/specs/2026-09-03-memoria-design.md` §8.8 draft |
| +3, decision 15, gates 34–37 (38 struck). Branch `feat/lookup-box`. |
| + |
| +**Draft 3 is what this plan is written against, and §0 of the previous |
| +version of this file is why it exists.** Writing the retrieval step out |
| +concretely is what found that a corpus-wide facts table cannot coexist |
| +with a cap of three. That finding is now in the spec; what remains here is |
| +operational. |
| + |
| +--- |
| + |
| +## 1. `internal/memoria/fact` — new package |
| + |
| +Recognisers, validators, the kind lexicon and the label rule. Imports |
| +`regexp`, `strings`, `unicode` and nothing else: pure, table-driven, and |
| +unit-testable without a database, a browser or an app — the same reason |
| +`query` and `chunk` are their own packages. |
| + |
| +**The lexicon lives here, beside the validators, not in `query`.** `query` |
| +is about one decision and its doc comment says so. More to the point, the |
| +words that ask for a kind and the validator that admits one are two |
| +columns of one table and must not drift apart. |
| + |
| +```go |
| +type Kind string // "phone", "eircode", "iban", "vat", "email" |
| + |
| +// Wanted returns the kinds a query names. Empty means no box. |
| +func Wanted(q string) []Kind |
| + |
| +// Found is one validated fact, located. |
| +type Found struct { |
| + Kind Kind |
| + Span string // the document's bytes. The only thing rendered. |
| + Canonical string // match-only; never rendered, never an operand. |
| + Label string // the document's, per §8.8's five rules. |
| + NoBox bool // an over-cap source-1 label: this kind renders nothing. |
| +} |
| + |
| +// Scan returns every validated fact of the wanted kinds in text. |
| +// heading and title are the label fallbacks, in that order. |
| +func Scan(text, heading, title string, want []Kind) []Found |
| +``` |
| + |
| +Build order, each step's test before the next: |
| + |
| +1. **`Wanted`** — a fixed map, longest phrase first so `email address` |
| + beats `address` (which maps to nothing). Tests: the five kinds; |
| + `address`, `number` and `mobile app` priming nothing, nothing and |
| + `phone` respectively. |
| +2. **Recognise and validate, per kind.** Each kind is |
| + `{re, canon, valid}`. **The regex is deliberately loose and the |
| + validator does the work** — that is §8.8's "validation is not |
| + pattern-spotting" made structural, and it is what stops the next |
| + person tightening a regex into a validator. |
| + |
| + Verified against known-good values before this plan was written, |
| + because a validator that admits everything is the guard-written-to-pass |
| + failure in its purest form and a checksum is the one thing here |
| + checkable against the world rather than against a fixture somebody |
| + wrote: |
| + |
| + | Input | Expect | Got | |
| + |---|---|---| |
| + | `IE6388047V` (Google Ireland) | valid | valid | |
| + | `IE3628739L` | valid | valid | |
| + | `IE1234567X` (invented) | **invalid** | invalid | |
| + | `IE29AIBK93115212345678` | valid | valid | |
| + | `GB82WEST12345698765432` | valid | valid | |
| + | `IE29AIBK9311521234567`**`9`** | **invalid** | invalid | |
| + | `A65 F4E2` (published example) | valid | valid | |
| + | `D6W 1234` (Dublin 6W's irregular key) | valid | valid | |
| + | `B12 3456` (`B` is not a routing-key letter) | **invalid** | invalid | |
| + |
| + The reject rows are load-bearing: the invented VAT and the one-digit |
| + IBAN edit are what tell a checksum from a regex, and `D6W` is the case |
| + a pattern written `[A-Z][0-9][0-9]` gets wrong. |
| + |
| + **Old-style Irish VAT** — a letter or `+` in second position — is |
| + refused deliberately. A refused VAT is a box that does not appear; an |
| + admitted wrong one is a fact on the page. |
| + |
| + **`phone` gets §8.8's structural rule:** a bare run of digits is never |
| + a phone number; the span must carry `+` or grouping separators. |
| + `rank_test.go`'s `DUNS/D-U-N-S number is 985632364` is the fixture |
| + that proves it — nine digits, a plausible Irish mobile without the |
| + rule. **This is a validator test in `fact`, and a rejection fixture in |
| + gate 35** — not gate 37, which is the label gate. |
| + |
| + **Separators include U+00A0**, since Go's `\s` does not match it and |
| + Word and Notion paste it. `chunk.go` does not normalise interior ones, |
| + so the bytes reach the sealed row. |
| + |
| +3. **`label`**, the part drafts 1 and 2 both got wrong. Given the span's |
| + byte offset in the chunk text: |
| + - walk back to the start of the enclosing sentence or line — `. `, |
| + `! `, `? `, `\n`, or the start — **honouring a fixed abbreviation |
| + list** (`Dr. Mr. Mrs. Ms. St. No. ext. Ltd. Co.`), which is stated |
| + as incomplete and whose failures compose; |
| + - **if the run contains an earlier recognised span, start after it**; |
| + - if what remains ends in `:`, `—`, `–`, `-`, `=` or tab: |
| + - **empty after removing it and trimming** → fall through to the |
| + trail; |
| + - **longer than 60** → `NoBox` for this kind; |
| + - else → the label; |
| + - else the heading trail; else the document title, never `""`. |
| + |
| +**Every span is taken by byte offset**, so `Span` is a slice of the input |
| +and cannot be a reconstruction. Gate 36's byte-identical property is then |
| +a property of the type rather than a discipline. |
| + |
| +--- |
| + |
| +## 2. `internal/memoria/store.go` — candidates, after the ruling |
| + |
| +One unexported method called by `Search` after `rule`: |
| + |
| +1. `want := fact.Wanted(q)`; empty → return. **No database work for a |
| + query naming no kind**, which is the common case. |
| +2. Read the sealed chunk rows for `p.Results` — the **ruled** slice, |
| + never `all` — in one `IN` query; open under `seal.PurposeChunk`. At |
| + most 25 rows. |
| +3. **A chunk row that is gone is skipped silently**, as `livePassage` |
| + skips a stale hit. It does not fail the search. |
| +4. `fact.Scan` per result, carrying that result's `Title` and `Heading` |
| + as the label fallbacks and its `DocumentID`/`Ordinal` as the citation. |
| +5. **Group by `Kind`. Every rule below is per kind.** |
| +6. Dedupe on `Canonical` into an **ordered slice plus a seen-set** — not |
| + a bare map, whose range order is randomised and whose first-seen order |
| + gate 35 asserts. First occurrence wins the citation, and because |
| + `p.Results` is in rank order that is the best-ranked passage. |
| +7. Any `NoBox` in the kind → drop the kind. |
| +8. Two candidates sharing the whole line `(Title, Heading, Label)` → drop |
| + the kind. |
| +9. More than three → drop the kind. |
| + |
| +`Page` gains `Facts []Fact`; `Fact` carries `Kind, Title, Trail, Label, |
| +Span, DocumentID, Ordinal`. **Not `ChunkID` and not the chunk text** — a |
| +field a template can reach is a field a template eventually prints, which |
| +is why `Result.item` is unexported. |
| + |
| +--- |
| + |
| +## 3. `internal/memoria/handlers.go` |
| + |
| +`runSearch` copies `found.Facts` onto the page beside `found.Notices`. |
| +That is the whole handler change, for **both** routes, because `index` |
| +and `searchResults` share `runSearch` — which is why the page and the |
| +fragment cannot disagree about the box any more than they can about |
| +priming. |
| + |
| +--- |
| + |
| +## 4. `templates/results.html` — inside the define |
| + |
| +Between the notices block and `<ol rst-results>`: below the withholding |
| +callout, above the list, **inside** `{{define "results"}}` so the fragment |
| +swap replaces box and list together. |
| + |
| +Each row carries title and trail as well as label and value (§8.8 — two |
| +pages called "Contact" are the same row to a reader who cannot see which |
| +is which), and **the citation link's text is the document title**, not a |
| +bare "source". |
| + |
| +**The box must never reuse `rst-result-title`.** `search.js` targets that |
| +attribute to decide where Enter goes; a link inside the box wearing it |
| +would make Enter land on the box's citation, which §8.8 rules against. |
| +The template comment says so. |
| + |
| +**No `safeSnippet`** — that helper exists to let FTS5's `<mark>` through |
| +and nothing here has one. `Span` gets Go's ordinary contextual escaping. |
| +**No anchor built from a document's bytes.** |
| + |
| +`memoria.css` gets `[rst-fact-box]`, `[rst-fact]`, `[rst-fact-label]`, |
| +`[rst-fact-value]`, `[rst-fact-cite]` — **tokens only, no colour |
| +literals**. `TestNoColourLiteralsInAppCSS` and |
| +`TestEveryTemplateAttributeIsInTheDesignSystem` both bite, which is why |
| +the stylesheet is written before the template rather than after. |
| + |
| +--- |
| + |
| +## 5. `tools/drive` — the section nobody seeds is the section nobody sees |
| + |
| +AGENTS.md, and it has found five bugs this way. Seed three shapes, not |
| +one: |
| + |
| +- a labelled office number and an eircode in an address block — the happy |
| + case; |
| +- **the motivating fixture's paragraph**, two numbers with distinct |
| + sentence labels — the enumerated case, where the reader should see |
| + "unused" and understand the page; |
| +- **a bulleted contact pair** — the collision case, where the box should |
| + be absent. The previous version of this plan called the two-sentence |
| + paragraph the collision case; it is the opposite, and a seed that |
| + demonstrates the wrong thing is worse than no seed. |
| + |
| +--- |
| + |
| +## 6. Tests, in the order they get written |
| + |
| +**Unit, in `fact`, before any wiring.** `Wanted`; each validator's accept |
| +and reject rows from §1.2; the label's eight cases from gate 37; spans |
| +with an extension and with U+00A0. |
| + |
| +**Gates**, each carrying its mutation in the comment: |
| + |
| +- **35, 36, 37** in `internal/memoriatest/gates_test.go` — slice 0. |
| +- **34 in `internal/memoria`**, at store level, against **a custody fake |
| + that rules per item id**. `fixedCustody` returns |
| + `sibling.Decide(f.give)` for every candidate from one field and cannot |
| + express fixtures 2–6. Extending it is part of this branch. |
| +- **38 is struck**, not written. The measurement it was going to assert |
| + is prose in §8.8; a test that injects a term into every row and asserts |
| + `bm25()` collapses tests SQLite and no mutation of this repo turns it |
| + red. |
| + |
| +**And the mutations are not mine to pick.** §1.9: the gate review picks |
| +its own, and this branch is not done until somebody who did not write it |
| +has tried them. |
| + |
| +--- |
| + |
| +## 7. What this does not touch |
| + |
| +`index/` (no column, no table, no weight, no `MatchQuery` change), |
| +`chunk.go` (no change to `Indexable`), `models.go`, `migrations/` (nothing |
| +new is stored), `seal/` (no new purpose), the sibling contract, the |
| +connectors, `Delete`, `Rebuild`. |
| + |
| +That list is the strongest evidence for draft 3. Draft 2's design touched |
| +five of them. |
| + |
| +--- |
| + |
| +## 8. What building it changed (2026-09-15) |
| + |
| +Kept as the record of what the plan did not foresee. |
| + |
| +**The reachability limit is the ordinary case, not an edge.** Writing the |
| +gates showed that bare `phone number` does not match a document reading |
| +`Office: +353 (0)1 547 7854` — the words are not there, nothing ranks, |
| +and there is nothing to extract from. Every fixture had to be given a |
| +word the query could find. Pinned in both directions by |
| +`TestTheBoxOnlySeesWhatRetrievalReturned`, and §12 carries the fix. |
| + |
| +**Three recognisers were wrong on ordinary text**, all found by the |
| +implementation review and none by the fixtures: |
| +`Take the N11 then exit at Bray` boxed `N11 then`; |
| +`The deadline is 01-02-2024 10:30` boxed `01-02-2024 10`; |
| +`Growth of +10 000 000 users` boxed `+10 000 000`. And the grouped IBAN |
| +pattern made that kind inert on its commonest written shape — |
| +`IE29 AIBK … 78 and our BIC is …` found nothing at all. §8.8 now carries |
| +the three structural rules that fix them. |
| + |
| +**Label derivation was quadratic in the facts a chunk holds.** `chunk.Split` |
| +bounds a chunk by WORDS, so a 2 MiB upload with no whitespace is one |
| +chunk and one line: 48.5s to scan 1 MB, which at the upload cap is |
| +minutes of search latency bought by one person's file. Deduping before |
| +labelling, binary-searching the span list and bounding the backward walk |
| +took 1 MB to 152 ms. |
| + |
| +**Four gates had holes the author's own mutations could not reach**, which |
| +is §1.9's whole point. `silenced` counted before the ruling, a citation |
| +cross-wired field by field, a collision check comparing only neighbours, |
| +and rows sorted rather than first-seen — each passed every shipped |
| +fixture and each now has one written for it. |
| + |
| +**And `addFacts` turned out to be structurally unable to see unruled |
| +data**, because it takes only `*Page`. The leak mutations all have to be |
| +made in `Search`, one level up. That is a stronger statement of the |
| +design than the comment claiming it, and it is why the gate-34 comment |
| +names the mutation at the level where it is reachable. |
diff --git a/docs/superpowers/specs/2026-09-03-memoria-design.md b/docs/superpowers/specs/2026-09-03-memoria-design.md| index 5efccc2..87f589d 100644 |
| --- a/docs/superpowers/specs/2026-09-03-memoria-design.md |
| +++ b/docs/superpowers/specs/2026-09-03-memoria-design.md |
| @@ -108,6 +108,7 @@ embedding is a plaintext cell in a costume — §3.1. |
| | 12 | **Name: `memoria`.** House convention is plain Spanish office nouns. A *memoria* is both a memory and, in Spanish office usage, a written record. | `archivo`; `cerebro`; `base`; anything English | Paul, 2026-09-03 | |
| | 13 | **The provider is TensorX, and memoria supports exactly one.** A team owner's choice is on or off, not which — a configurable base URL would let an owner reinstate a proxy while the answer still printed "one named provider". Chosen for EU jurisdiction and open weights, criteria decision 4 did not state, at a known cost in id-pinning and frontier quality (§8.3). | OpenRouter (a proxy: two parties see content); OpenAI direct (better id-pinning, worse jurisdiction and cost) | Paul, 2026-09-04, both halves — *"we go with exactly one for now: TensorX. If we want to swap that later we can."* Swapping the *generation* provider is a code change and a review, not a setting; swapping the *embedding* provider is that plus a full corpus re-embed unless the new host serves the same open weights (§8.3, §13) | |
| | 14 | **memoria is a search page, not a chat.** One GET search form, `?q=`, one query in and one page out, with `/documents/{id}` as the only other passage surface. This follows from decision 5, not from decision 3 — §8.4's invariant is what decision 3 requires, and a conversational surface could satisfy it. | A chat UI with citations; a threaded follow-up. Deferred to §12, not forbidden — the invariant forbids the leak, this row chooses the shape | Spec author, 2026-09-04, answering a question Paul asked; **confirmed by Paul 2026-09-05**, parking the conversational surface in §12 rather than forbidding it | |
| +| 15 | **The fact box calls no model, ever.** A validated fact — eircode, phone, IBAN, VAT, email — is extracted deterministically and shown above the results with its citation. A kind with no validator is not a kind. §8.8 | A cheap extractive model picking the span (§8.7's retrieval-augmented calculator with a different noun); a model composing the label; postal address and `url` as kinds, neither having anything to validate | **The constraint is Paul's, 2026-09-14** — *"do as much deterministic ordering of the data before we go doing inference on it"*, asking for the box in the first place. **The "ever", the validator rule, the kind list and the postal exclusion are the spec author's reading of it and are unconfirmed — §13.** Decision 14 is the form this follows: record which half was ruled and by whom, rather than letting the interpretation live in a commit message (§1.9) | |
| **Three of these are now suite rulings, and this spec cites them rather |
| than restating them.** Decision 1 is **suite ruling 1**, decision 3 is |
| @@ -2216,6 +2217,11 @@ app, title, locator, 👨/🤖 marker, date, and a deep link. Scriptless, |
| paginated, rastrillo `server` shape. Lexical only, since embeddings need |
| a provider — and per §3.2, nothing leaves the instance in this mode. |
| +**§8.8's fact box is part of this floor and not an addition above it.** It |
| +calls no model (decision 15), so it is present in exactly the mode this |
| +paragraph describes; gate 15 covers it with everything else here, and it |
| +renders inside the results region rather than beside it. |
| + |
| **Loud withholding, naming the shape of the loss:** |
| - `UNKNOWN` from a sibling → *"Sheets did not answer; 7 results |
| @@ -2380,6 +2386,15 @@ Draft 1 wrote "filtered to what the asker can see" on the third row, |
| which either restated §4.1 or smuggled a second per-person filter |
| alongside it. The passages have already passed the gate. |
| +**§8.8's fact box is not a sixth row, and it was one in that section's |
| +first draft.** A row here is a route, and a route is terminal: a query |
| +naming a fact kind would have stopped at the box, so *"what's the office |
| +phone number?"* with nothing validated in reach would get no answer where |
| +the single-source class would have given one. The box runs **before** this |
| +table and does not consume the query — Paul's constraint is deterministic |
| +*first*, not deterministic *instead*. It is a fourth way memoria answers |
| +without a model; it is not a fourth destination. |
| + |
| Two configuration names — `model.cheap`, `model.frontier` — with the |
| route printed on the answer. |
| @@ -2886,6 +2901,501 @@ calculator; and caching an S6 result, which outlives its cells. |
| Gate 14 extends: **every digit in an answer sentence traces to a cited |
| passage or an S6 result.** |
| +### 8.8 The fact box: deterministic extraction above the list |
| + |
| +Paul's motivating example, 2026-09-14: a search for `phone number` should |
| +put the answer at the top of the page — *"Tito office number: +353 (0)1 |
| +547 7854"* — and the same for an eircode, a VAT number. And his |
| +constraint, given in the same breath, which is the whole of this section's |
| +shape: **do as much deterministic ordering of the data as possible before |
| +any inference runs on it.** |
| + |
| +**Called the fact box, and not the lookup box, which is what draft 1 |
| +called it.** `query.IsQuestion`'s complement is already a lookup in this |
| +codebase — `TestALookupPrimesTheFirstResult` is about Enter, not about |
| +this — and a second meaning for the word in the same package is the name |
| +collision this repo's reviews keep finding. |
| + |
| +**Decision 15: the fact box calls no model, ever.** Not a cheap one, not |
| +an extractive one. The kinds it serves are the kinds that have a |
| +validator; a kind with no validator is not in the box and is not the |
| +box's problem (§12). The moment a model picks which span to show, every |
| +argument below collapses into §8.7's forbidden retrieval-augmented |
| +calculator with a different noun. |
| + |
| +**It is a fourth way memoria answers without a model, and it is not a |
| +class.** Drafts 1 and 2 both called it *"§8.2's fourth model-free class"* |
| +while also calling it a layer, which cannot both be true: a row in §8.2's |
| +table is a **route**, and a route is terminal. A query naming a kind would |
| +have stopped at the box, so *"what's the office phone number?"* with |
| +nothing validated in reach would get no answer where the single-source |
| +class would have given one. The box is attempted, and then §8.2 routes the |
| +query exactly as it would have. Paul's constraint is deterministic |
| +**first**, not deterministic **instead**. |
| + |
| +It does not touch §3.2. No egress, no provider, no embedding — the box |
| +works on an instance with AI switched off, which is decision 5's floor and |
| +therefore the whole product today. It is part of §8.1's floor and gate 15 |
| +covers it. **Nothing on this path is logged**, gate 4's rule applying |
| +unchanged: a fact, its label and its kind are all content, and the query |
| +that named the kind is content too. |
| + |
| +#### One extraction, over the ruled results, and why drafts 1 and 2 had two |
| + |
| +**Candidates are the validated facts carried by the passages this search |
| +already returned and §4.1 already allowed.** There is no second index, no |
| +second query, no ingest-time half and nothing stored. `Store.Search` |
| +produces `Page.Results`; the box is extracted from those and from nothing |
| +else. |
| + |
| +That makes §8.4's invariant structural rather than careful: every passage |
| +in a response passed §4.1 **in the request that produced it**, and a |
| +candidate that is not in `Page.Results` does not exist. There is no |
| +second retrieval path, so there is no second authorization path to get |
| +wrong. |
| + |
| +**Both earlier drafts built a retrieval mechanism for reachability, and |
| +the cap is what killed it.** Draft 1 folded kind words into the main |
| +index; draft 2, after the IDF objection below, gave facts their own |
| +in-memory table queried by kind. Writing the retrieval step out |
| +concretely is what found the contradiction: |
| + |
| +- this section caps candidates at three; |
| +- a facts table is queried `WHERE kind = 'phone'`, which is corpus-wide; |
| +- a real corpus holds dozens of phone numbers — `index/rank_test.go`'s |
| + twelve-chunk fixture already holds two, plus four account numbers and a |
| + VAT number. |
| + |
| +So the cap fires on the first query anyone types, on every kind, |
| +permanently, and the box never appears. **The only way to bound a |
| +corpus-wide fact query is to rank the candidates and keep the best three, |
| +which is exactly what "never by picking" forbids** — and a `LIMIT` before |
| +the ruling is worse still, because a withheld document then eats one of |
| +the slots and "count after the ruling" becomes unimplementable. The gate |
| +review reached this independently and from the other side. |
| + |
| +The ruled top-25 is already a query-scoped candidate set. Once candidates |
| +come from it, the facts table has no query-time job left, and its only |
| +other job was the ingest-time ranking lift. |
| + |
| +**Why the deferred fix is not "index the kind words", measured rather |
| +than asserted.** FTS5 computes a term's IDF across the table, so a term |
| +present in most rows scores near zero. On `modernc.org/sqlite`, 20 rows, |
| +best `bm25()` for `"phone"`: **-2.7169** with the term in one row, |
| +**-0.0000** with it injected into all twenty. A collapse, not a |
| +degradation. |
| + |
| +**Be exact about what that measures, because draft 2 overclaimed it.** |
| +Draft 2 said the injection would degrade *"exactly the query the NEAR |
| +comment works through"*, and on that comment's own fixture it would not: |
| +the chunk holding the office number already contains the word `phone`, so |
| +injecting it there moves that chunk's term frequency and length, not the |
| +corpus IDF. The real exposure is a corpus where **many chunks carry |
| +numbers and few say `phone`** — which is the shape a large real corpus has |
| +and the fixture does not. The number stands; the sentence it was attached |
| +to did not. |
| + |
| +#### What this costs, and it is the thing to measure next |
| + |
| +**The reachability limit is back, and draft 2 claimed to have removed |
| +it.** A fact in a document that does not reach the ruled top-25 is a fact |
| +the box does not have: an eircode written `D02 XY45` in a document that |
| +never says "eircode" is found only if something else about it ranks. |
| +Draft 1 named this and deferred the fix; draft 2's table did not actually |
| +remove it, since that table ran only when a kind was named and was inert |
| +corpus-wide under the cap. §12 carries it. |
| + |
| +**The box's presence depends on the retrieval limit, which is a real |
| +edge and is stated rather than discovered.** `Search(ctx, q, 25)` is a |
| +hard cap, so a fourth distinct value sitting at rank 26 leaves a box of |
| +three, and the same value at rank 25 removes the box entirely. That is |
| +not wrong — the box describes what this search found — but it means the |
| +box is a function of the page, not of the corpus, and nobody should read |
| +it as the latter. |
| + |
| +**Whether the cap makes the box inert on the corpus that motivated it is |
| +unmeasured, and that measurement is owed before decision 15 is settled.** |
| +A real top-25 for "phone number" on Paul's live corpus may well carry more |
| +than three distinct valid numbers. §13 records it. The remedy if it does |
| +is a narrower kind lexicon or a per-kind cap, **not** a rule that picks. |
| + |
| +**§4.1 is unaffected, in the direction that matters.** A reader for whom |
| +passages are withheld draws candidates from a smaller set, never a larger |
| +one — `rule` drops withheld results and §12's over-fetch-and-refill is |
| +deliberately not built, so the slot stays lost. Fewer candidates cannot |
| +be a leak. |
| + |
| +**A chunk row that is gone by the time the box reads it is skipped |
| +silently**, exactly as `Store.Search`'s `livePassage` check drops a stale |
| +hit, and for the reason recorded there. It does not fail the search: the |
| +box is a garnish and trading the page for it is the wrong direction. |
| + |
| +**And the box needs plaintext the search path does not open.** |
| +`Store.Search` opens titles, paths and item ids under their own purposes; |
| +chunk bodies are opened only by `indexDocument` and `Passages`. FTS5 |
| +snippets are 24 tokens with `<mark>` in them — too short and too marked-up |
| +to take a span or a label from. So the box re-reads the sealed chunk rows |
| +for the ruled results, after the ruling, under `seal.PurposeChunk`: at |
| +most 25 rows, one `IN` query, 25 AEAD opens. |
| + |
| +#### Why nothing is stored |
| + |
| +The rejected design is a **sealed, durable** `facts` table, written at |
| +ingest, read at query time. Two grounds, and draft 1's third is withdrawn: |
| + |
| +- It is **derived state**, so it inherits AGENTS.md's ordering rule in |
| + both directions and §5.2's invalidation — a new way for the index and |
| + the database to disagree, bought for nothing that query-time extraction |
| + does not already give. |
| +- It **freezes the recognisers at the version that ran at ingest**, so |
| + improving one leaves every existing document answering with the old |
| + one until somebody re-ingests it. **This is not the `Passages` problem |
| + and draft 2 cited it as though it were.** That comment records the |
| + opposite lesson — re-running the *chunker* at view time misaligned |
| + citations, and freezing at ingest is the fix — and it does not transfer, |
| + because a recogniser change moves no ordinal and breaks no citation. |
| + The staleness argument stands on its own or not at all. |
| + |
| +Draft 1's third ground was that a plaintext `kind` column would be an |
| +inventory of which kinds of sensitive data the team holds — true, and |
| +irrelevant, because the proposal was for a sealed table. The honest |
| +residual for a sealed one is the per-chunk fact-row count, which is real, |
| +much smaller, and not on its own a reason. |
| + |
| +**Nothing new exists at rest, and nothing new exists in the index |
| +either.** That is the strongest property this section has and it came |
| +from deleting things. |
| + |
| +#### The recognisers, and what is not one |
| + |
| +A kind is in the box when a query word names it **and** a candidate |
| +validates. |
| + |
| +| Kind | Primed by | Validated by | |
| +|---|---|---| |
| +| `eircode` | `eircode`, `postcode`, `post code` | routing key — a letter from `ACDEFHKNPRTVWXY`, a digit, then a digit or `W` — plus four characters from that set and the digits | |
| +| `iban` | `iban`, `bank details` | country prefix, length by country, mod-97 | |
| +| `vat` | `vat`, `vat number` | modern Irish format and its check letter | |
| +| `phone` | `phone`, `telephone`, `tel`, `mobile`, `phone number` | **structural, and see below** | |
| +| `email` | `email`, `email address`, `e-mail` | `local@domain`, domain carrying a dot, no whitespace — **weak, and see below** | |
| + |
| +**Three of these are checksums and two are not, and the section says so |
| +rather than implying five.** An eircode's routing key is a closed set, an |
| +IBAN has a mod-97 check, an Irish VAT number has a check letter derived |
| +from its digits. These can be wrong, and a wrong one is refused. |
| + |
| +**`phone` has no checksum, so it gets a structural rule that is stated |
| +here rather than left to the code: a bare run of digits is never a phone |
| +number.** The span must carry a `+`, or separators grouping it. This is |
| +not a hack bolted on to a weak validator — it is how documents write phone |
| +numbers, and without it `rank_test.go`'s own corpus yields false |
| +positives: `DUNS/D-U-N-S number is 985632364` is nine digits and a |
| +plausible Irish mobile. **The cost is stated too:** a document writing |
| +`0871234567` with no separators is not boxed. That is the safe direction, |
| +and it is the direction a refusal should fail in. |
| + |
| +**Three structural rules came out of building it, and each one is here |
| +because ordinary document text was boxed without it.** The implementation |
| +review found all three live, not as mutations: |
| + |
| +- **A phone number uses one separator style throughout.** Mixed |
| + separators walk straight past the 9-or-10 bound above: *"The deadline |
| + is 01-02-2024 10:30"* matched `01-02-2024 10`, ten digits, valid, and |
| + boxed a date. Three styles, never mixed. |
| +- **Four digit groups, or five with a country code — refused by count, |
| + never truncated.** *"Numbers drawn: 01 05 12 23 34"* canonicalises to |
| + ten digits and passes every other test, so something has to refuse it. |
| + **The count is the validator's job and not the pattern's**, and putting |
| + it in the pattern was a worse bug than the one it fixed: a pattern |
| + capped at three groups and five digits a group does not refuse a long |
| + number, it *shortens* it. *"Mobile: +353 87 1234567"* — the commonest |
| + international spelling of an Irish mobile — was boxed as |
| + **`+353 87 12345`**, and *"+49 30 12 34 56 78"* as `+49 30 12 34 56`. |
| + Both are well-formed numbers that pass every other check and are not |
| + the number in the document: §8.7's *"cites correctly while being |
| + wrong"*, manufactured by the guard meant to prevent it. |
| + |
| + So the recogniser is generous, it must end on a word boundary or an |
| + extension, and the validator refuses by group count. **Germany and |
| + France genuinely write more groups than this and their numbers are |
| + refused.** That is the stated cost, and it is the right way round: a |
| + refusal shows nothing, a truncation shows a wrong number. |
| +- **The international floor is ten digits, not E.164's eight.** Eight |
| + admits `+10 000 000` out of *"Growth of +10 000 000 users"*. A country |
| + code plus a national number reaches ten everywhere this corpus dials, |
| + and refusing a genuinely short international number is the safe |
| + direction. |
| + |
| +**One residual is stated rather than left to be found: an ALL-CAPS |
| +heading.** `SEE FORM P45 THEN P60` still yields `P45 THEN`, because |
| +case-sensitivity cannot separate an eircode from uppercase English, and |
| +uploads do carry all-caps headings. No regex fixes this; only a label or |
| +a dictionary would, and both are bigger than the problem. Recorded so the |
| +next person finds the limit rather than the surprise. |
| + |
| +**And eircodes are matched case-sensitively**, which is not fussiness. |
| +The identifier alphabet is most of the consonants, so case-insensitively |
| +it matches lowercase English after a road number or a form code: *"Take |
| +the N11 then exit at Bray"* boxed `N11 then`, and *"Form P45 then P60"* |
| +boxed `P45 then`. A cited fact that is really an English word reads |
| +exactly like a correct answer — §8.7's *"cites correctly while being |
| +wrong"*, arriving from a character class. Eircodes are written uppercase; |
| +a lowercase one going unrecognised is the cost and it is the right way |
| +round. |
| + |
| +**`email` is kept and its weakness is stated rather than hidden.** |
| +`local@domain` is closer to pattern-spotting than to validation. On a |
| +corpus of uploads and Notion pages it is useful — one support address, |
| +cited. On a corpus carrying forwarded mail (§4.2's source list) every |
| +signature qualifies, the cap is passed, and the box is inert for that |
| +kind. That is the cap working, and the failure direction is silence rather |
| +than a wrong address. |
| + |
| +**`url` was in draft 1 and is dropped.** `scheme://host` is |
| +pattern-spotting with nothing behind it, so every link in every document |
| +qualifies and the box is inert immediately. It also collides with §8.5's |
| +*"refuse links to non-suite hosts"* for no gain. |
| + |
| +**Nothing in the box is ever an anchor.** The value is text; the only link |
| +the box carries is its citation, to `/documents/{id}#p{n}`. No `mailto:`, |
| +no `href` built from a document's bytes. |
| + |
| +**`address` on its own primes nothing.** It is ambiguous between a postal |
| +address and an email address, and only one of those has anything |
| +resembling a validator. `email address` primes `email`. A postal address |
| +has no checksum, no fixed grammar and no length, so under decision 15 it |
| +is not a kind. §12 carries it. |
| + |
| +**`number` on its own primes nothing either.** The lexicon maps |
| +distinctive words and phrases; generic words count only in combination. |
| + |
| +**This is `query.openers`' trade made in the opposite direction, and draft |
| +1 claimed it was the same one.** `openers` is deliberately stingy — *"the |
| +safe direction is not priming"* — because a wrong prime navigates away |
| +from a page somebody was reading. The fact lexicon can afford to be |
| +generous, and the reason is the validator, not a shared instinct: "mobile |
| +app" primes `phone`, finds no phone number, and renders nothing. |
| + |
| +**It complements Enter-priming rather than competing with it.** |
| +`IsQuestion` declines to prime for anything ending in `?` or opening with |
| +an interrogative, so "what's the office phone number?" navigates nowhere |
| +today, and those are disproportionately the queries a fact box answers. |
| +Where both fire — "office phone number", no question mark — **Enter still |
| +goes to the first result, not to the box's citation.** The box is a thing |
| +to read, not a destination; changing what Enter means depending on whether |
| +a regular expression matched is the unreliable-Enter failure |
| +`PrimesFirstResult` exists to avoid. |
| + |
| +#### The span, the canonical form, and this spec's two arithmetic rules |
| + |
| +**The span is what the document wrote**, byte for byte, including |
| +punctuation, the `(0)`, and any extension written after it. It is what the |
| +box displays and the only thing the box displays. Separators inside a span |
| +include **U+00A0**, because Word and Notion paste non-breaking spaces and |
| +the chunker does not normalise interior ones — a recogniser written with |
| +Go's `\s`, which does not match U+00A0, silently finds nothing in pasted |
| +text. |
| + |
| +**The canonical form is match-only.** It is what candidate identity is |
| +keyed on, and what the validator runs on. It is **never rendered and never |
| +an operand.** An Irish national number canonicalises to E.164 so that |
| +`01 547 7854` and `+353 (0)1 547 7854` are recognised as one number rather |
| +than two candidates. |
| + |
| +Two of this spec's own rules have to be read against that, and both are |
| +ruled here rather than left to be found: |
| + |
| +- **§8.7 forbids *"memoria parsing a display string back into a number … |
| + a second parser"*.** What it forbids is a parse that feeds something |
| + memoria then shows or computes with — the `"€1.2M"` case. A canonical |
| + form that is never displayed and is never an operand is not that. |
| +- **Decision 10 says memoria never does arithmetic.** A mod-97 check |
| + digit is arithmetic. It is not a number memoria reports: it decides |
| + whether a string is an IBAN at all. Decision 10 governs numbers in |
| + answers; a checksum is a predicate. **That is a narrowing of Paul's |
| + decision made by the spec author, and it is filed in §13 beside decision |
| + 15's, rather than one being marked and the other slipped through.** |
| + |
| +**Draft 1's own example failed draft 1's own validator**, which is worth |
| +keeping visible: `+353 (0)1 547 7854` is not E.164 — E.164 has no spaces |
| +and no `(0)` — so "validated by E.164" was a rule the motivating case did |
| +not pass. The table says *canonicalises to*, which is what was meant. |
| + |
| +#### The label, and the four ways it goes wrong |
| + |
| +**The label is taken from the document and is never composed.** Given the |
| +span's position in the chunk text: |
| + |
| +1. **Take the run back to the start of the enclosing sentence or line.** |
| + **If that run contains another recognised span, start after it** — in |
| + `Phone: 01 547 7854 - 01 598 1234` the second number's run would |
| + otherwise be `Phone: 01 547 7854 -`, labelling the second number with |
| + the first one's value. |
| + |
| + **The spans it skips are of every kind, not only the kinds the query |
| + named**, and that is a rule rather than an implementation note: a |
| + label that depends on what was asked for is not the document's label. |
| + Asked for `phone` alone, *"Email paul@tito.io or phone: 01 547 7854"* |
| + came back labelled `Email paul@tito.io or phone` — one fact's value |
| + inside another's label, which is the exact failure this rule exists to |
| + stop, reappearing because the rule could not see the span. |
| +2. **If what remains ends in a separator** — `:`, `—`, `–`, `-`, `=`, tab |
| + — **and is non-empty after removing it and trimming, and is 60 |
| + characters or fewer, that is the label.** |
| +3. **Empty after the separator → fall through.** A markdown bullet is the |
| + case: the chunker keeps lines verbatim, so `- +353 1 547 7854` gives a |
| + run of `-`, which "ends in a separator" and yields nothing. Two such |
| + bullets give two empty labels, collide, and would have suppressed the |
| + box for a plain contact list. |
| +4. **Longer than 60 → no box for that kind.** Not the trail instead. The |
| + document labelled this value and the label will not fit, so |
| + substituting a vaguer one is memoria replacing the document's words |
| + with its own: *"The number you must never give to customers because it |
| + rings the CEO directly is: 01 598 1234"* becomes `Contacts`, sitting |
| + beside `Reception — 01 547 7854`. Dropping the candidate instead is |
| + worse, because that turns a two-candidate ambiguity into a confident |
| + single answer. |
| +5. **Otherwise the chunk's heading trail**, and if there is none — an |
| + upload with no headings, which is the common pasted-file shape — **the |
| + document title**, never the empty string. |
| + |
| +**Draft 1 required the value to start its own line and capped the label at |
| +40, and the repo's own fixture is what breaks that.** |
| +`index/rank_test.go:56` — built from the live instance where Paul hit this |
| +— holds one chunk reading: |
| + |
| +> *"Turn the knob. The office phone number is: +353 (0)1 547 7854. We |
| +> also have the following unused phone numbers: +353 (0)1 598 1234."* |
| + |
| +Under draft 1 both numbers are mid-paragraph, both fall to the heading |
| +trail, and both come back labelled `Office Address`: two identical labels, |
| +one of which the document explicitly calls **unused**. The box would have |
| +shown them as equals with the disambiguating word one clause away. That is |
| +§8.7's *"cites correctly while being wrong"* in a lookup coat. Sentence |
| +anchoring fixes it by doing less: the labels become *"The office phone |
| +number is"* and *"We also have the following unused phone numbers"*, and |
| +the reader can see which is which because the document said so. |
| + |
| +**The sentence split is naive, and draft 2's claim that it fails safely is |
| +false.** A boundary placed too *late* yields a shorter run that still |
| +passes every test above: `Dr. Smith's direct line: 01 …` yields |
| +`Smith's direct line`, and `St. Stephen's Green office: …` yields |
| +`Stephen's Green office`. **That is a label the document did not write, |
| +produced by the rule whose entire justification is that it never |
| +composes.** A short fixed exception list — `Dr. Mr. Mrs. Ms. St. No. ext. |
| +Ltd. Co.` — covers the common cases and is **stated as incomplete**. It |
| +must not grow into a sentence tokeniser; where it is wrong it is wrong in |
| +the composing direction, and saying so is the difference between a known |
| +limit and a discovered one. |
| + |
| +**60 characters is a guess.** What is not a guess is that it must clear |
| +the motivating fixture, whose longer label is 47. |
| + |
| +#### Candidates, ambiguity, the cap, and one box per kind |
| + |
| +**A candidate is one canonical value.** The same number in five forwarded |
| +signatures is **one** candidate, cited to the best-ranked passage carrying |
| +it. Two different numbers in one chunk are **two** — the fixture above is |
| +exactly that shape, and treating a chunk as the unit would hide the |
| +second. |
| + |
| +**Every rule below is applied per kind, and a multi-kind query renders one |
| +box per kind.** A cap counted across kinds would let two phone numbers and |
| +two email addresses suppress each other, and a collision check across |
| +kinds would suppress a phone and an email that happened to share a label. |
| + |
| +§8.7's slot rule is *"two or more → ambiguity, never extra evidence, and |
| +never collapsed"*, and it resolves that by showing no number. **The fact |
| +box shows all of them instead, and the difference is not inconsistency.** |
| +An average built from two of three operands is a *wrong number*: the |
| +question had one answer and memoria produced a different one. Two phone |
| +numbers are two *right answers* to an underspecified question. §8.7's |
| +ambiguity is incompleteness, which can only be refused; this one is |
| +underspecification, honestly served by showing both, labelled and cited. |
| + |
| +**Never by picking.** Not highest BM25, not first, not longest. Rank order |
| +sorts a result list and carries no claim about which number is the office |
| +one. |
| + |
| +**A box row carries the document title and the heading trail as well as |
| +the label**, exactly as a result row carries both and for the reason |
| +recorded there: two pages called "Contact" under different documents are |
| +the same row to a reader who cannot see which is which. |
| + |
| +**Two candidates whose whole citation line collides — same title, same |
| +trail, same label — render no box.** Keyed on the line rather than the |
| +label alone, because a supplier page and the office page both falling to |
| +the trail `Contact` are perfectly disambiguated by their titles, and |
| +suppressing that box would be losing a real answer to a rule aimed at a |
| +different problem. Where the whole line does collide, the page would show |
| +one identical row twice against two different values — not an enumeration |
| +but a contradiction with no way for the reader to resolve it, and the |
| +results list is the honest surface for that. |
| + |
| +**Capped at three inclusive: three renders, four renders nothing.** |
| + |
| +#### Where it sits on the page |
| + |
| +Inside the `results` template definition, not above it. `results.html` is |
| +rendered two ways from one definition — inside the page on a scriptless |
| +submit, alone on `GET /search` for `search.js` — and a box outside it |
| +would sit above a fragment that swapped underneath it, leaving query N−1's |
| +answer over query N's results on every keystroke. |
| + |
| +**Below §6.5's notices and above the list.** `results.html` already puts |
| +the withholding callout above the results because *"a reader who scrolls |
| +past the last result has already decided what the page means"*. A |
| +confident single answer above the line saying the page is partial is the |
| +same mistake reversed. |
| + |
| +**Search page only.** Decision 14 makes the search page and |
| +`/documents/{id}` the only two passage surfaces, and this is a feature of |
| +one of them. |
| + |
| +**Zero candidates renders nothing at all** — not "no phone number found". |
| +The corpus is not the world, and a confident absence is this codebase's |
| +own worst failure shape. |
| + |
| +**Gate 14's first clause extends to the box** — every character of it |
| +traces to a cited passage. **Its second does not.** "An all-dropped answer |
| +says so rather than rendering empty" exists because an empty 🤖 paragraph |
| +is indistinguishable from AI being off; a box that does not appear is |
| +indistinguishable from nothing, which is what it means. |
| + |
| +#### Injection, and the one place ambiguity pays for itself |
| + |
| +§8.5's mitigation for a poisoned document is that every claim carries an |
| +openable citation, and that it is a partial one. The box sharpens the |
| +problem: it promotes a single document's assertion to the top of the page |
| +with a validated-looking format around it. `Office phone: +353 1 000 0000` |
| +in an uploaded file is a fact box, and no model was involved in believing |
| +it. |
| + |
| +Two things bound it and neither is a fix: |
| + |
| +- The value is a cited span of a passage that passed §4.1, so the reader |
| + can open it, and a poisoned document is one somebody in this team put in |
| + this corpus. |
| +- **The ambiguity rule fails safe against poisoning.** A planted second |
| + number does not replace the real one and does not outrank it — it makes |
| + the box show both, or show nothing if the whole citation line collides, |
| + or vanish past the cap. The attack that works against a single-answer |
| + box degrades into a visibly contradictory page, which is a property |
| + worth keeping rather than an accident to optimise away. |
| + |
| +#### Sequencing |
| + |
| +**Slice 0, entirely.** Recognisers, the lexicon, the label rule, the box |
| +and gates 35–37 need no sibling and no model. **Gate 34 needs a custody |
| +fake that rules per item**, which `fixedCustody` is not — it returns one |
| +disposition for every candidate from a single field — so the gate lands |
| +with a fake extended to answer per item id, in `internal/memoria` at store |
| +level, which is where its "assert the struct, not the page" demand is |
| +natural anyway. Drafts 1 and 2 both said this gate could run against |
| +"`memoriatest`'s sibling stub, which exists"; there is no such stub, and |
| +that is the renamed-test failure §10 opens with, committed twice in the |
| +section that cites it. |
| + |
| --- |
| ## 9. Surfaces |
| @@ -3489,12 +3999,157 @@ The gates: |
| are distinguishable after five submissions, and a refused address |
| consumes no per-IP budget either. The budget is inside signin and not |
| reachable from here; it closes with the upstream fix. |
| +34. **The box never speaks a withheld passage, and every count happens |
| + after the ruling** (§8.8). Lives in `internal/memoria`, at store |
| + level, against **a custody fake that rules per item id** — |
| + `fixedCustody` returns one disposition for every candidate from a |
| + single `give` field and cannot express these fixtures at all. Six, |
| + and the last three are the ones an author does not write: |
| + |
| + - **Presence.** Two sibling documents, two different validated |
| + eircodes, both `VISIBLE` → the box carries **two** candidates. |
| + Without this the absences below are assertions about an empty |
| + world (§10's second rule). |
| + - **`NOT_VISIBLE` is silent.** One of them withheld → the box shows |
| + **exactly the survivor, and `Notices` is empty.** Draft 1 demanded |
| + a notice here, which §4.1's table forbids and `Outcome.Record` will |
| + not produce: a gate that could only be made green by building the |
| + existence oracle §4.1 exists to deny. |
| + - **`UNKNOWN` is loud.** Same fixture, `UNKNOWN` instead → the |
| + survivor **and** §6.5's notice. |
| + - **The cap counts after the ruling.** Four documents, four distinct |
| + values, one `NOT_VISIBLE` → **three** candidates and a box. Under |
| + "extract over the ruled set, count over the unruled one" the reader |
| + loses the box entirely because of a document they may not see, and |
| + no two-document fixture can tell those implementations apart. |
| + - **The citation is the survivor's.** The **same** eircode in a |
| + `VISIBLE` and a `NOT_VISIBLE` document → one candidate, cited to |
| + the visible passage. Every fixture above uses distinct values per |
| + document, so "cite from `all` instead of `page.Results`" passes all |
| + of them; this is the only one that can see it. |
| + - **Collision counts after the ruling too.** Two documents sharing a |
| + title and trail, both falling to the trail for a label, one |
| + `NOT_VISIBLE` → the survivor **is** boxed. Computed over the |
| + unruled set it collides and the box vanishes, which is the same |
| + leak as the cap case wearing the collision rule's coat. |
| + |
| + - **A withheld over-long label does not silence the kind.** Rule 4 — |
| + a document-authored label too long to show renders no box for that |
| + kind — is a count like the other two, and has to happen after the |
| + ruling for the same reason. **No other fixture here has a `NoBox` |
| + candidate at all**, so a mutation computing that rule over the |
| + unruled set passed all six of the others. Found by the |
| + implementation review. |
| + |
| + **Assert the struct, not the page** — the candidate tuple |
| + `(span, title, trail, label, DocumentID, Ordinal)` and the notice |
| + slice, never rendered copy. **Every field, and the fixtures must |
| + differ in all of them**: with two identical bodies, taking the title |
| + from the ruled result and the rest from an unruled one passes, and |
| + renders an `href` to a document the reader may not open under that |
| + document's own heading and label. Two dispositions rendering an identical |
| + page is how `Outcome.Record` filed silent rulings undetected (§1.9). |
| +35. **Candidates are canonical values, ambiguity is enumerated, and the |
| + cap is three** (§8.8). One gate, because the rules share a definition |
| + and a gate that splits them cannot see the interactions: |
| + |
| + - one validated number → one box, one value **(presence)**; |
| + - a second, different number → **both named**, neither dropped, the |
| + higher-scoring one not presented as the answer, **and in first-seen |
| + order** — order asserted, since a map's range is not an order; |
| + - **three → still a box. Four → none.** Both, because a cap written |
| + `>= 3` passes a suite testing only one, two and four; |
| + - **two spellings of one number** — `+353 1 547 7854` and |
| + `01 547 7854` → **one** candidate. A dedupe keyed on the span |
| + rather than the canonical form passes a same-spelling fixture; |
| + - the same number in five documents → one candidate, cited to the |
| + best-ranked; |
| + - **two numbers in one chunk → two candidates.** |
| + `index/rank_test.go`'s office fixture is exactly this, and an |
| + implementation counting chunks hides the second; |
| + - **a two-kind query with two of each** → **two boxes**, not none. |
| + A cap counted across kinds suppresses both, and this is the fixture |
| + that sees it. |
| + |
| + - **Rows are in first-seen order, asserted** — on a fixture where |
| + sorting by value and seeing in document order DISAGREE. A box |
| + sorted by canonical value is deterministic and passes every length |
| + check, and the two-in-one-passage fixture happens to sort the way |
| + it was seen. |
| + - **A collision between NON-ADJACENT rows still suppresses.** Every |
| + other collision fixture is a pair, so a check comparing only |
| + neighbours passes all of them. |
| + - **The same label under different HEADINGS still renders**, which is |
| + the trail's half of the rule the title case covers. |
| + |
| + Mutation: return `candidates[0]` when there is more than one and case |
| + 2 goes red. Count passages instead of canonical values and cases 4 |
| + and 5 go red in opposite directions, which is the pair worth having. |
| + The last three above were each found green by the implementation |
| + review and each goes red on exactly the fixture written for it. |
| +36. **The displayed value is byte-identical to the document's** (§8.8). |
| + The canonical form exists for matching and must never reach the page. |
| + Fixtures, with the expected bytes written out in each: `+353 (0)1 547 |
| + 7854`; the same followed by `ext. 12`, where the expected span is |
| + `+353 (0)1 547 7854 ext. 12` and the recogniser must be specified to |
| + include it; and one written with **U+00A0** between the groups. |
| + Mutation: normalise ` ` to a space before matching **and display |
| + the normalised span** — the shape a recogniser naturally grows, and |
| + invisible to any fixture using only ASCII spaces. **Note which way |
| + the NBSP fixture fails:** Go's `\s` does not match U+00A0, so a |
| + recogniser written with it finds nothing and the fixture goes red on |
| + presence rather than on the bytes. That is the design demand, not an |
| + accident. |
| +37. **The label is the document's, or the fallback's, and never composed** |
| + (§8.8's five rules, one fixture each plus the four failures the gate |
| + review found): |
| + |
| + - `Office: …` → `Office`; |
| + - the motivating fixture's mid-paragraph value after a separator → |
| + the sentence-anchored run, which draft 1 got wrong; |
| + - a value mid-sentence with **no** separator before it → the heading |
| + trail, never the preceding clause; |
| + - **a markdown bullet**, `- +353 1 547 7854` → the run is `-`, empty |
| + once the separator is removed, so it **falls through to the trail** |
| + rather than yielding an empty label. Two such bullets must still |
| + box, which an implementation returning `""` turns into a collision; |
| + - **two values adjacent**, `Phone: 01 547 7854 - 01 598 1234` → the |
| + second label must not be `Phone: 01 547 7854`. Drop the |
| + earlier-span rule and this fixture labels one number with another; |
| + - **an abbreviation**, `Dr. Smith's direct line: 01 …` → the whole |
| + run, not `Smith's direct line`. This is the composing direction and |
| + the exception list is the only thing holding it; |
| + - **60 and 61 characters**, both. At 60 the label renders; at 61 |
| + there is **no box for that kind** — not the trail instead. A cap |
| + written `< 60` passes a suite that only tests "one over"; |
| + - an upload with **no headings at all** → the document title, never |
| + `""`. Every fixture carrying a heading hides this. |
| +38. ~~**The facts table does not disturb the main ranking.**~~ **Struck |
| + before it was built, 2026-09-14.** §8.8 draft 3 removed the facts |
| + table, so there is nothing to prove non-interference for — and the |
| + gate as drafted was unreachable anyway: a test that injects a term |
| + into every row and asserts `bm25()` collapses is a test of SQLite, |
| + and no mutation of this repo can turn it red. AGENTS.md's rule is |
| + that a defence nothing can reach is the same object as a guard |
| + written to pass, and that the answer is to make it reachable or |
| + delete it, never to leave it with a comment. The **measurement** |
| + survives as prose in §8.8, where it explains why §12's eventual |
| + reachability fix is not "index the kind words". The number kept its |
| + place in this list rather than being reused, per §13's convention for |
| + struck items. |
| ## 11. Sequencing |
| **Slice 0 — buildable today, no contract, no blockers.** Scaffold, sealed |
| store, §7 proven end to end: chunk, seal, wake, BM25 search, scriptless |
| -UI with citations, gates 1–4 and 15. Content by upload and paste only, so |
| +UI with citations, gates 1–4 and 15. **§8.8's fact box lands here too**, |
| +added 2026-09-14: recognisers, the kind lexicon, the label rule, the box, |
| +and gates 35–37 — nothing stored, nothing indexed, no sibling and no |
| +model. **Gate 34 comes with it and needs a custody fake that rules per |
| +item id**, which `fixedCustody` is not. Drafts 1 and 2 of §8.8 both said |
| +it could run against "`memoriatest`'s sibling stub, which exists"; no such |
| +stub exists, and writing that twice in the section that cites §10's |
| +renamed-test rule is the failure that rule is about. Content by upload and paste only, so |
| §4.2 governs and no person ref is needed. **No embeddings, therefore no |
| egress at all.** This is the design Docs §15 item 2 deferred for want of |
| one, so proving it unblocks Docs regardless of memoria's fate. |
| @@ -3559,6 +4214,9 @@ commit. Fixed 2026-09-05; §1.9's lesson arriving inside the fix for §1.8.* |
| | **A source-of-record surface, so a hand-off can include memoria's own material** | Someone wants "save this answer as a Doc" to work for answers built on the team's own uploads. Suite §7 makes the *receiver* check every source with its app of record, and memoria serves nobody to ask (decision 7), so such a hand-off is refused today — §9.1. Opening it needs its own decision, its own §3.2 leak entry and its own gate, and it is Paul's: decision 7 is settled | |
| | Calendar as a source | Calendar has a spec and content; bodies off by default even then | |
| | Mail beyond forward-or-label | Never by sweep. A mail repo exists **and** a per-thread audience maps onto a team | |
| +| **Reaching a fact the ranked query does not** | Somebody reports a fact they can see in a document the box did not surface. §8.8's candidates are the ruled top-25, so an eircode in a document that never says "eircode" is found only if something else about it ranks. **Whatever this turns out to be, it is not "index the kind words"** — §8.8 measures that collapsing `bm25()` for the injected term from 2.7169 to 0.0000 | |
| +| **Postal address as a fact kind** | Decision 15 keeps it out because there is nothing to validate. The two openings are an anchored one — an eircode certifies the lines above it, deterministically — and §8.2's extractive single-source class, which is a model and therefore a decision-15 amendment | |
| +| **A durable, sealed facts table** | The per-query extraction shows up in a profile. §8.8 rejects it today on derived-state grounds and because it would freeze the recognisers at ingest — **not** on leak grounds, which is what draft 1 argued and which attacked an unsealed design nobody proposed. Note the direction: storing facts makes recogniser improvements *harder* to ship, so this row gets triggered by cost and never by correctness | |
| | Cached answers | Query cost becomes visible. Requires §5.2's provenance invalidation first. **Triggering this expires §8.4's "stored nowhere"** | |
| | A conversational surface | Someone wants one, and it keeps §8.4's invariant: no passage crosses a request, in a prompt or a response. Decision 14 chose the shape on decision 5's grounds, not decision 3's, so this is a product argument to have — not a rule to break. §6.5 affords it: a four-turn transcript re-authorizes in one batch, one round | |
| | A runtime model-call ledger | Someone needs to audit routing. Sealed and provenance-stamped when it lands (§8.3) | |
| @@ -3570,10 +4228,42 @@ commit. Fixed 2026-09-05; §1.9's lesson arriving inside the fix for §1.8.* |
| ## 13. Open |
| -**Four items need Paul and are marked `NEEDS PAUL`:** the 12-hour session |
| -TTL, §8.7's unit-override switch, the CI runner's repo list, and the |
| +**Five items need Paul and are marked `NEEDS PAUL`:** the 12-hour session |
| +TTL, §8.7's unit-override switch, the CI runner's repo list, the |
| disclosure route for the rastrillo SSRF (`SECURITY-DRAFT-rastrillo-signin.md`, |
| -unfiled). **Nothing else here is a ruling; the rest is work, measurement |
| +unfiled), and **decision 15's scope**. **Nothing else here is a ruling; |
| +the rest is work, measurement or somebody else's repo.** |
| + |
| + - **§8.8's cap is unmeasured against the corpus that motivated it, and |
| + the measurement is owed before decision 15 is settled.** The box shows |
| + nothing when a search's ruled top-25 carries four or more distinct |
| + validated values of one kind. Nobody has counted how many distinct |
| + phone numbers a real top-25 for "phone number" carries on Paul's live |
| + corpus, and if the answer is "more than three" the feature is inert on |
| + the very corpus it was asked for. **The remedy if so is a narrower |
| + lexicon or a per-kind cap — never a rule that picks**, which is the |
| + one thing §8.8 cannot trade away. Run it against the live data or |
| + through `tools/bench`. |
| + - **Decision 10 is narrowed by §8.8 and the narrowing is the spec |
| + author's, not Paul's.** "memoria never does arithmetic" now reads as |
| + "never reports a number it computed"; a mod-97 check digit is |
| + arithmetic on document content that decides a predicate and reports |
| + nothing. Defensible, and recorded here for the same reason decision |
| + 15's widening is: §1.9's lesson is that the load-bearing half of a |
| + decision must not live only in a commit message. |
| + - **Decision 15 records more than Paul ruled, and says so rather than |
| + quietly widening.** What he asked for on 2026-09-14 is the box, and |
| + the constraint — *"do as much deterministic ordering of the data |
| + before we go doing inference on it"*. The **"ever"** is the spec |
| + author's: read strictly it forecloses §8.2's cheap extractive class |
| + ever filling a box for a kind with no validator, which is the |
| + obvious next request and the natural home for a postal address. The |
| + validator rule, the five kinds and dropping `url` follow from the |
| + strict reading. If Paul wants the weaker version — deterministic |
| + first, a model second where nothing validates — that is one row in |
| + §12 promoted and decision 15 amended, not a redesign. The gate |
| + review raised this as §1.9's failure repeating: the load-bearing |
| + half of a decision living in a commit message. **Nothing else here is a ruling; the rest is work, measurement |
| or somebody else's repo.** |
| *Struck items are kept with their reasoning rather than deleted. Several |
diff --git a/internal/memoria/fact/fact.go b/internal/memoria/fact/fact.go| new file mode 100644 |
| index 0000000..904a104 |
| --- /dev/null |
| +++ b/internal/memoria/fact/fact.go |
| @@ -0,0 +1,835 @@ |
| +// Package fact recognises the facts §8.8's box is allowed to show. |
| +// |
| +// One table drives all of it: a kind names the words that ask for it AND |
| +// the validator that admits it. Those two must not drift apart — a |
| +// lexicon that primes `iban` next to a validator that no longer accepts |
| +// one is a box that never appears and nothing that says why — so they are |
| +// two columns of one declaration and live in one file. That is also why |
| +// the lexicon is here rather than in query, which has its own single |
| +// decision to make and says so. |
| +// |
| +// Nothing here touches a database, a template or an app. The spec's §8.8 |
| +// is a long argument about authorization and ordering, and none of it is |
| +// in this package: by the time Scan is called the passage has already |
| +// passed §4.1. What is here is the part that can be wrong on its own — |
| +// whether a string is really an IBAN, and what the document called it. |
| +// |
| +// **Three of the five kinds are checksums and two are not**, and the |
| +// difference is stated rather than smoothed over. An eircode's routing |
| +// key is a closed set, an IBAN has a mod-97 check, an Irish VAT number |
| +// has a check letter derived from its own digits: those can be wrong, and |
| +// a wrong one is refused. A phone number and an email address have no |
| +// such property, and their bounds are structural. See phoneValid for the |
| +// rule that keeps the weakest of them honest. |
| +package fact |
| + |
| +import ( |
| + "regexp" |
| + "sort" |
| + "strings" |
| + "unicode" |
| + "unicode/utf8" |
| +) |
| + |
| +// Kind is one recognisable fact type. |
| +type Kind string |
| + |
| +const ( |
| + Eircode Kind = "eircode" |
| + Phone Kind = "phone" |
| + IBAN Kind = "iban" |
| + VAT Kind = "vat" |
| + Email Kind = "email" |
| +) |
| + |
| +// order is the order kinds are returned and rendered in, so that a |
| +// two-kind query produces the same two boxes in the same places every |
| +// time. A map's range order would not, and §8.8's gate asserts order. |
| +var order = []Kind{Eircode, Phone, IBAN, VAT, Email} |
| + |
| +// Found is one validated fact, located in a passage. |
| +type Found struct { |
| + Kind Kind |
| + // Span is the document's own bytes, and the only thing rendered. |
| + // Taken as a slice of the input rather than rebuilt from parts, so |
| + // that "byte-identical to the document" is a property of this type |
| + // and not a discipline somebody has to keep. §8.7's rule one noun |
| + // over: a second formatter is the same bug class as a second |
| + // calculator. |
| + Span string |
| + // Canonical is match-only. It keys candidate identity so that |
| + // "01 547 7854" and "+353 (0)1 547 7854" are one number rather than |
| + // two, and it is what the validator ran on. It is NEVER rendered and |
| + // is never an operand — see the package doc and §8.8 for why that |
| + // distinction is what keeps this inside §8.7 and decision 10. |
| + Canonical string |
| + // Label is what the document called this value. Never composed. |
| + Label string |
| + // NoBox means this kind renders nothing at all, because a value |
| + // carried a document-authored label too long to show. Substituting |
| + // the heading trail would replace the document's words with vaguer |
| + // ones — "the number you must never give to customers…" becoming |
| + // "Contacts" — and dropping just this candidate would turn a |
| + // two-candidate ambiguity into a confident single answer. Both are |
| + // worse than silence. §8.8. |
| + NoBox bool |
| +} |
| + |
| +// --------------------------------------------------------------------- |
| +// The lexicon: which words ask for which kind. |
| +// --------------------------------------------------------------------- |
| + |
| +// phrase is one entry of the lexicon, as a token sequence. |
| +type phrase struct { |
| + toks []string |
| + kind Kind |
| +} |
| + |
| +// lexicon maps what a person types to a kind. Longest phrases are tried |
| +// first, which is the whole reason "email address" can mean Email while |
| +// "address" alone means nothing: postal and email addresses are |
| +// ambiguous and only one of them has anything resembling a validator. |
| +// |
| +// **This is query.openers' trade made in the opposite direction.** That |
| +// list is deliberately stingy, because a wrong Enter-prime navigates away |
| +// from a page somebody was reading. This one can afford to be generous, |
| +// and the reason is the validator rather than a shared instinct: "mobile |
| +// app" primes Phone, finds no phone number, and renders nothing. A |
| +// generous lexicon costs a box that does not appear. |
| +var lexicon = []phrase{ |
| + {[]string{"email", "address"}, Email}, |
| + {[]string{"phone", "number"}, Phone}, |
| + {[]string{"bank", "details"}, IBAN}, |
| + {[]string{"post", "code"}, Eircode}, |
| + {[]string{"vat", "number"}, VAT}, |
| + {[]string{"e", "mail"}, Email}, |
| + {[]string{"eircode"}, Eircode}, |
| + {[]string{"postcode"}, Eircode}, |
| + {[]string{"telephone"}, Phone}, |
| + {[]string{"phone"}, Phone}, |
| + {[]string{"mobile"}, Phone}, |
| + {[]string{"tel"}, Phone}, |
| + {[]string{"iban"}, IBAN}, |
| + {[]string{"vat"}, VAT}, |
| + {[]string{"email"}, Email}, |
| +} |
| + |
| +// Wanted returns the kinds a query names, in `order`. Empty means no box. |
| +// |
| +// Deliberately absent: "address", which is ambiguous between postal and |
| +// email and whose postal reading has no validator (§8.8, decision 15); |
| +// and "number", which is generic and counts only in combination. Both |
| +// would prime on ordinary queries that are not asking for a fact. |
| +func Wanted(q string) []Kind { |
| + toks := tokens(q) |
| + want := map[Kind]bool{} |
| + for i := 0; i < len(toks); i++ { |
| + for _, p := range lexicon { |
| + if len(p.toks) > len(toks)-i { |
| + continue |
| + } |
| + if matchAt(toks, i, p.toks) { |
| + want[p.kind] = true |
| + i += len(p.toks) - 1 |
| + break |
| + } |
| + } |
| + } |
| + var out []Kind |
| + for _, k := range order { |
| + if want[k] { |
| + out = append(out, k) |
| + } |
| + } |
| + return out |
| +} |
| + |
| +func matchAt(toks []string, i int, want []string) bool { |
| + for j, w := range want { |
| + if toks[i+j] != w { |
| + return false |
| + } |
| + } |
| + return true |
| +} |
| + |
| +// tokens lowercases and splits on anything that is not a letter or |
| +// digit, so "e-mail" becomes ["e","mail"] and punctuation is inert. The |
| +// same shape as index.MatchQuery's term split, and for the same reason: a |
| +// search box is not a query language. |
| +func tokens(s string) []string { |
| + return strings.FieldsFunc(strings.ToLower(s), func(r rune) bool { |
| + return !unicode.IsLetter(r) && !unicode.IsDigit(r) |
| + }) |
| +} |
| + |
| +// --------------------------------------------------------------------- |
| +// The recognisers. |
| +// --------------------------------------------------------------------- |
| + |
| +// sep is every character that may separate the groups of a fact, and it |
| +// includes U+00A0. |
| +// |
| +// Go's \s does NOT match a non-breaking space, and Word and Notion both |
| +// paste them. chunk.Split does not normalise interior whitespace, so the |
| +// bytes reach the sealed row exactly as pasted — which means a recogniser |
| +// written with \s finds nothing at all in text pasted from either, and |
| +// the failure is a box that silently never appears. |
| +const sep = ` \x{00A0}` |
| + |
| +// phoneGroups builds the digit-group half of the phone pattern for one |
| +// separator style. |
| +// |
| +// **Loose, and the group count is the validator's job — not this |
| +// pattern's.** Bounding it here truncated real numbers instead of |
| +// refusing them, which is far worse than either: capped at five digits a |
| +// group, "Mobile: +353 87 1234567" was boxed as "+353 87 12345" — the |
| +// commonest international spelling of an Irish mobile, rendered as a |
| +// different number, cited, above the results. Capped at three groups, |
| +// "+49 30 12 34 56 78" was boxed as "+49 30 12 34 56". Both pass every |
| +// other check because what is left is a well-formed number; they are |
| +// §8.7's "cites correctly while being wrong", produced by the pattern |
| +// that was supposed to prevent it. |
| +// |
| +// So the groups are generous and phoneValid refuses by count. A refusal |
| +// shows nothing; a truncation shows a wrong number. |
| +// |
| +// **And at least one, which is what makes the three styles alternate.** |
| +// Written {0,3} the first branch matched zero groups and won, because |
| +// Go's regexp prefers the earliest alternative that matches AT ALL — |
| +// so "01-547-7854" came back as the two-digit span "01" and was refused, |
| +// with the dash branch never tried. A branch that cannot match zero |
| +// groups fails, and the next one gets its turn. |
| +func phoneGroups(sepClass string) string { |
| + return `(?:` + sepClass + `[0-9]{2,7}){1,6}` |
| +} |
| + |
| +type recogniser struct { |
| + kind Kind |
| + re *regexp.Regexp |
| + canon func(string) string |
| + valid func(span, canonical string) bool |
| +} |
| + |
| +// The regexes are deliberately LOOSE and the validators do the work. |
| +// That is §8.8's "validation is not pattern-spotting" made structural: it |
| +// is what stops the next person tightening a regex until it is a |
| +// validator nobody can see, and it is why every reject below fails on an |
| +// arithmetic property rather than on a character class. |
| +var recognisers = []recogniser{ |
| + { |
| + kind: Eircode, |
| + // The routing key's first letter is a closed set and the unique |
| + // identifier's alphabet excludes the letters that misread. A |
| + // pattern written [A-Z][0-9][0-9] gets Dublin 6W wrong and lets |
| + // B12 3456 through; both are fixtures. |
| + // |
| + // **Case-SENSITIVE, and it was written (?i) first.** The review |
| + // found what that costs: the identifier alphabet is most of the |
| + // consonants, so lowercase English after a road number or a form |
| + // code matches it. "Take the N11 then exit at Bray" boxed |
| + // "N11 then"; "Form P45 then P60" boxed "P45 then". A query for |
| + // an eircode then gets a cited fact that is an English word — |
| + // §8.7's "cites correctly while being wrong", from a character |
| + // class. Eircodes are written uppercase, and a lowercase one |
| + // going unrecognised is the safe direction. |
| + re: regexp.MustCompile(`\b[ACDEFHKNPRTVWXY][0-9][0-9W][` + sep + `]?[0-9ACDEFHKNPRTVWXY]{4}\b`), |
| + canon: func(s string) string { return strings.ToUpper(stripSep(s)) }, |
| + valid: func(_, _ string) bool { return true }, // the pattern IS the closed set |
| + }, |
| + { |
| + kind: Phone, |
| + // Built out of digit GROUPS rather than one character class, and |
| + // that is the difference between working and not. |
| + // |
| + // A class — [0-9 ().-]{6,24} — is greedy across a LIST: given |
| + // "Phone: 01 547 7854 - 01 598 1234" it swallows both numbers as |
| + // one span of eighteen digits, which then fails validation, so |
| + // the page shows nothing where it should show two. Requiring a |
| + // SINGLE separator between groups stops the match at the " - ", |
| + // and the second number then matches on its own. Found by the |
| + // fixture, not by reading the pattern. |
| + // |
| + // The prefix is "+CC", optionally followed by the "(0)" that |
| + // means "the trunk digit you omit from abroad", or a bare |
| + // leading 0. Anchoring there is what keeps account numbers, |
| + // company numbers and DUNS numbers out before any validation |
| + // runs at all. |
| + // |
| + // **One separator style throughout, and phoneValid's 9-or-10 |
| + // bound is why it has to be.** That bound was written to keep |
| + // dates out, and a mixed-separator pattern walked straight past |
| + // it: "The deadline is 01-02-2024 10:30" matched |
| + // "01-02-2024 10" — ten digits, valid, boxed. A person writing a |
| + // phone number uses one separator for all of it. RE2 has no |
| + // backreferences, so the three styles are three alternatives. |
| + re: regexp.MustCompile(`(?:\+[0-9]{1,3}[` + sep + `]*(?:\(0\))?|\b0)[` + sep + `]*` + |
| + `[0-9]{1,9}(?:` + phoneGroups(`[`+sep+`]`) + `|` + phoneGroups(`-`) + `|` + phoneGroups(`\.`) + `)?` + |
| + // An extension is part of the span, because §8.8 says the span |
| + // is what the document wrote, and a number whose extension has |
| + // been dropped is a number that does not reach the person. |
| + // |
| + // **An extension OR a word boundary, never "optionally |
| + // neither".** Written `(?:…)?` the match could stop in the |
| + // middle of a run of digits, which is the other half of how |
| + // "+353 87 1234567" became "+353 87 12345". Requiring one or |
| + // the other means a partial number is not a match at all. |
| + `(?:(?i:[` + sep + `]*(?:ext|extension|x)\.?[` + sep + `]*[0-9]{1,6})|\b)`), |
| + canon: phoneCanon, |
| + valid: phoneValid, |
| + }, |
| + { |
| + kind: IBAN, |
| + // Two spellings, because a document writes an IBAN either solid |
| + // or in groups of four. |
| + // |
| + // **The grouped alternative was {2,4}-sized groups and that made |
| + // the kind inert on its commonest shape.** A trailing 2-4 letter |
| + // word was swallowed into the last group, the canonical form came |
| + // out the wrong length, and validation refused the whole thing: |
| + // "Our IBAN is IE29 AIBK 9311 5212 3456 78 and our BIC is …" |
| + // found nothing at all. Only a full stop immediately after the |
| + // number worked. Groups of exactly four with one short tail stop |
| + // at the space before "and", because "and" is not four |
| + // characters — and case-sensitivity means it would not match |
| + // even if it were. |
| + // |
| + // **The tail is digits only**, because an uppercase 1-3 character |
| + // token is exactly what follows an IBAN in practice: "BE68 5390 |
| + // 0754 7034 BIC: GEBABEBB" found nothing while the tail admitted |
| + // letters, and every country whose length is a multiple of four |
| + // (BE, ES, AT, PL, SE, LU) has no real tail for it to sit in. |
| + re: regexp.MustCompile(`\b[A-Z]{2}[0-9]{2}(?:[A-Z0-9]{11,30}|(?:[` + sep + `][A-Z0-9]{4}){2,7}(?:[` + sep + `][0-9]{1,3})?)\b`), |
| + canon: func(s string) string { return strings.ToUpper(stripSep(s)) }, |
| + valid: func(_, c string) bool { return ibanValid(c) }, |
| + }, |
| + { |
| + kind: VAT, |
| + re: regexp.MustCompile(`(?i)\bIE[0-9]{7}[A-Z]{1,2}\b`), |
| + canon: func(s string) string { return strings.ToUpper(stripSep(s)) }, |
| + valid: func(_, c string) bool { return vatValid(c) }, |
| + }, |
| + { |
| + kind: Email, |
| + // The weakest kind here, and §8.8 says so rather than hiding it: |
| + // local@domain is closer to pattern-spotting than to validation. |
| + // On a corpus carrying forwarded mail every signature qualifies, |
| + // the cap is passed, and the box is inert for this kind — which |
| + // is the cap working, and fails towards silence rather than |
| + // towards a wrong address. |
| + re: regexp.MustCompile(`\b[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}\b`), |
| + canon: strings.ToLower, |
| + valid: func(_, _ string) bool { return true }, |
| + }, |
| +} |
| + |
| +// stripSep removes every separator a document might group a fact with, |
| +// including U+00A0. Used only to build a Canonical; a Span is never |
| +// passed through it, which is gate 36's whole subject. |
| +func stripSep(s string) string { |
| + return strings.Map(func(r rune) rune { |
| + switch r { |
| + case ' ', ' ', '-', '.', '(', ')', '\t': |
| + return -1 |
| + } |
| + return r |
| + }, s) |
| +} |
| + |
| +// phoneCanon folds a written number onto E.164 so that "01 547 7854" and |
| +// "+353 (0)1 547 7854" are ONE candidate rather than two. |
| +// |
| +// The "(0)" is dropped only after an international prefix, which is what |
| +// that notation means: it is the trunk digit you dial domestically and |
| +// omit from abroad. Keeping it would make the two spellings above |
| +// different canonical values and put the same number in the box twice. |
| +// |
| +// Never rendered. §8.8 rules explicitly that a canonical form which is |
| +// neither displayed nor an operand is not §8.7's forbidden second parser. |
| +func phoneCanon(s string) string { |
| + if base, ext := splitExt(s); ext != "" { |
| + // An extension is part of the number somebody has to ring, so two |
| + // entries differing only by extension are two candidates and not a |
| + // duplicate. Appended AFTER the base is folded, so the base still |
| + // dedupes across spellings. |
| + return phoneCanon(base) + "x" + onlyDigits(ext) |
| + } |
| + intl := strings.HasPrefix(s, "+") |
| + if intl { |
| + s = strings.Replace(s, "(0)", "", 1) |
| + } |
| + var d strings.Builder |
| + for _, r := range s { |
| + if r >= '0' && r <= '9' { |
| + d.WriteRune(r) |
| + } |
| + } |
| + digits := d.String() |
| + switch { |
| + case intl: |
| + return "+" + digits |
| + case strings.HasPrefix(digits, "0"): |
| + // A national number, which on this corpus means Ireland. Folded |
| + // onto +353 so it dedupes against the international spelling. |
| + return "+353" + digits[1:] |
| + default: |
| + return digits |
| + } |
| +} |
| + |
| +// phoneValid is the structural rule that keeps the one checksum-less |
| +// numeric kind honest: **a bare run of digits is never a phone number.** |
| +// |
| +// A phone number has no check digit, so unlike an IBAN or a VAT number |
| +// there is nothing arithmetic to refuse a wrong one with. What is left is |
| +// how documents actually write phone numbers: with a country code, or in |
| +// groups. Without this rule index/rank_test.go's own corpus produces a |
| +// false positive on its first line — "DUNS/D-U-N-S number is 985632364" |
| +// is nine digits and a plausible Irish mobile. |
| +// |
| +// **The cost is stated rather than discovered:** a document that writes |
| +// 0871234567 with no separators at all is not boxed. That is the |
| +// direction a refusal should fail in. |
| +func phoneValid(span, _ string) bool { |
| + // Validated on the BASE number. An extension carries its own |
| + // digits and would otherwise push an ordinary number past E.164's |
| + // fifteen — refusing it for carrying more information, not less. |
| + base, _ := splitExt(span) |
| + if !strings.HasPrefix(base, "+") && !strings.ContainsAny(base, " ().-") { |
| + return false |
| + } |
| + // **Refused by group count, never truncated.** A phone number is at |
| + // most four separator-delimited runs, or five once a country code is |
| + // counted: "01 547 7854" is three, "+353 87 1234567" is three. |
| + // "Numbers drawn: 01 05 12 23 34" is five, and "+49 30 12 34 56 78" |
| + // is six — both refused here rather than cut down to something that |
| + // would validate. Germany and France genuinely do write more groups |
| + // than this; refusing their numbers is the cost, and it is the right |
| + // way round, because the alternative is showing most of one. |
| + groups := digitRuns(base) |
| + max := 4 |
| + if strings.HasPrefix(base, "+") { |
| + max = 5 |
| + } |
| + if groups > max { |
| + return false |
| + } |
| + digits := strings.TrimPrefix(phoneCanon(base), "+") |
| + if strings.HasPrefix(base, "+") { |
| + // E.164 caps at fifteen. The floor here is TEN rather than the |
| + // eight E.164 technically permits, and that is a deliberate |
| + // narrowing: a country code plus a national number reaches ten |
| + // for every country anybody in this corpus dials, while eight |
| + // admits "+10 000 000" out of the sentence "Growth of |
| + // +10 000 000 users". Refusing a genuinely short international |
| + // number is the safe direction; boxing a growth figure is not. |
| + return len(digits) >= 10 && len(digits) <= 15 |
| + } |
| + // A national number, canonicalised to +353 above, so the national |
| + // form was 0 plus what follows. Irish numbers are 9 or 10 digits |
| + // including the trunk 0 — 01 xxx xxxx, 021 xxx xxxx, 08x xxx xxxx. |
| + // |
| + // Nine-or-ten rather than "eight or more" is load-bearing: a date |
| + // written 01-02-2024 leads with a 0, carries separators, and is eight |
| + // digits. Widening this bound puts dates in the box. |
| + n := len(digits) - len("353") + 1 |
| + return n == 9 || n == 10 |
| +} |
| + |
| +// digitRuns counts the separator-delimited runs of digits in a span. |
| +func digitRuns(s string) int { |
| + n, in := 0, false |
| + for i := 0; i < len(s); i++ { |
| + if s[i] >= '0' && s[i] <= '9' { |
| + if !in { |
| + n++ |
| + in = true |
| + } |
| + continue |
| + } |
| + in = false |
| + } |
| + return n |
| +} |
| + |
| +// extRe finds a trailing extension, so canon and valid can treat the base |
| +// number and the extension separately while the SPAN keeps both. |
| +var extRe = regexp.MustCompile(`(?i)[` + sep + `]*(?:ext|extension|x)\.?[` + sep + `]*[0-9]{1,6}$`) |
| + |
| +func splitExt(s string) (base, ext string) { |
| + if m := extRe.FindStringIndex(s); m != nil { |
| + return s[:m[0]], s[m[0]:] |
| + } |
| + return s, "" |
| +} |
| + |
| +func onlyDigits(s string) string { |
| + var b strings.Builder |
| + for _, r := range s { |
| + if r >= '0' && r <= '9' { |
| + b.WriteRune(r) |
| + } |
| + } |
| + return b.String() |
| +} |
| + |
| +// ibanAlphabet holds the lengths IBANs have, by country. An unknown |
| +// country is refused rather than guessed: mod-97 alone admits roughly one |
| +// random string in ninety-seven, and the length is what makes the check |
| +// worth having. |
| +var ibanLength = map[string]int{ |
| + "IE": 22, "GB": 22, "DE": 22, "FR": 27, "ES": 24, "NL": 18, |
| + "IT": 27, "BE": 16, "PT": 25, "AT": 20, "PL": 28, "SE": 24, |
| + "DK": 18, "FI": 18, "NO": 15, "CH": 21, "LU": 20, "GR": 27, |
| +} |
| + |
| +// ibanValid is ISO 13616's mod-97 check: move the first four characters |
| +// to the end, map letters to two-digit numbers, and the whole thing |
| +// modulo 97 must be 1. |
| +// |
| +// Verified before it was written, against IE29AIBK93115212345678, |
| +// GB82WEST12345698765432 and DE89370400440532013000, and against the |
| +// first of those with one digit changed. A checksum is the one thing in |
| +// this package that can be checked against the world rather than against |
| +// a fixture somebody here wrote, so it was. |
| +func ibanValid(s string) bool { |
| + if len(s) < 15 || len(s) > 34 { |
| + return false |
| + } |
| + want, known := ibanLength[s[:2]] |
| + if !known || len(s) != want { |
| + return false |
| + } |
| + rem := 0 |
| + for _, c := range s[4:] + s[:4] { |
| + switch { |
| + case c >= '0' && c <= '9': |
| + rem = (rem*10 + int(c-'0')) % 97 |
| + case c >= 'A' && c <= 'Z': |
| + rem = (rem*100 + int(c-'A') + 10) % 97 |
| + default: |
| + return false |
| + } |
| + } |
| + return rem == 1 |
| +} |
| + |
| +// vatCheckAlphabet is Revenue's check-letter table, indexed by the |
| +// weighted sum modulo 23. W is zero. |
| +const vatCheckAlphabet = "WABCDEFGHIJKLMNOPQRSTUV" |
| + |
| +// vatValid checks an Irish VAT number's check letter. |
| +// |
| +// Modern format only — seven digits, a check letter, optionally a second |
| +// letter. **The old style, with a letter or + or * in second position, is |
| +// refused deliberately.** A refused VAT number is a box that does not |
| +// appear; an admitted wrong one is a fact on the page, and between those |
| +// two the choice is not close. |
| +// |
| +// Verified against IE6388047V and IE3628739L, and against the invented |
| +// IE1234567X, which it refuses. index/rank_test.go's own IE3384527RH |
| +// passes, which is what makes that fixture usable as a corpus here. |
| +func vatValid(s string) bool { |
| + s = strings.TrimPrefix(s, "IE") |
| + if len(s) != 8 && len(s) != 9 { |
| + return false |
| + } |
| + sum := 0 |
| + for i := 0; i < 7; i++ { |
| + if s[i] < '0' || s[i] > '9' { |
| + return false |
| + } |
| + sum += int(s[i]-'0') * (8 - i) |
| + } |
| + if len(s) == 9 { |
| + j := strings.IndexByte(vatCheckAlphabet, s[8]) |
| + if j < 0 { |
| + return false |
| + } |
| + sum += j * 9 |
| + } |
| + return s[7] == vatCheckAlphabet[sum%23] |
| +} |
| + |
| +// --------------------------------------------------------------------- |
| +// The label. |
| +// --------------------------------------------------------------------- |
| + |
| +// MaxLabel is how long a document-authored label may be, in characters. |
| +// |
| +// **A guess.** What is not a guess is that it has to clear the motivating |
| +// fixture in index/rank_test.go, whose longer label — "We also have the |
| +// following unused phone numbers" — is 47. |
| +const MaxLabel = 60 |
| + |
| +// abbreviations are the full stops that do not end a sentence. |
| +// |
| +// **Stated as incomplete, because that is the honest form.** Draft 2 of |
| +// §8.8 claimed the sentence split "fails in the safe direction"; it does |
| +// not. A boundary placed too LATE yields a shorter run that passes every |
| +// other test and produces a label the document never wrote — "Dr. Smith's |
| +// direct line:" becoming "Smith's direct line". That is the composing |
| +// failure this whole function exists to prevent, arriving through the |
| +// function itself, so the list is what holds it and the list is short. |
| +// |
| +// It must not grow into a sentence tokeniser. Where it is wrong it is |
| +// wrong in the composing direction, and knowing that is the difference |
| +// between a limit and a surprise. |
| +var abbreviations = map[string]bool{ |
| + "dr.": true, "mr.": true, "mrs.": true, "ms.": true, "st.": true, |
| + "no.": true, "ext.": true, "ltd.": true, "co.": true, |
| +} |
| + |
| +// separators are what a document puts between a label and its value. |
| +// |
| +// The en dash is here because Word and Notion autocorrect a hyphen |
| +// between spaces into one, so "Office – 01 547 7854" is what a document |
| +// actually contains rather than what its author typed. |
| +func isSeparator(r rune) bool { |
| + switch r { |
| + case ':', '—', '–', '-', '=', '\t': |
| + return true |
| + } |
| + return false |
| +} |
| + |
| +// maxLookback bounds how far sentenceStart walks. |
| +// |
| +// Without it this is linear in the LINE, and chunk.Split bounds a chunk |
| +// by words rather than bytes — so a 2 MiB upload with no whitespace is |
| +// one word, one paragraph, one chunk, and one line. Measured before this |
| +// bound: 0.49s at 100 KB, 1.94s at 200 KB, 48.5s at 1 MB, which at the |
| +// upload cap is minutes of search latency bought by one person's file. |
| +// |
| +// **It changes an outcome in one case, and draft 2's claim that it |
| +// changes none was wrong.** Four hundred bytes is always more than |
| +// MaxLabel runes UNLESS most of them are whitespace, which TrimSpace |
| +// then removes: five hundred characters, four hundred and fifty spaces, |
| +// then "Phone: 01 547 7854" yields the document's own label where the |
| +// uncapped walk yielded none. That is the better answer and it is the |
| +// direction this file prefers everywhere else, so the bound stays — but |
| +// stated, because "changes no outcome" is the kind of claim that stops |
| +// anybody checking. |
| +const maxLookback = 400 |
| + |
| +// sentenceStart walks back from at to the start of the enclosing sentence |
| +// or line, bounded by maxLookback. |
| +func sentenceStart(text string, at int) int { |
| + floor := at - maxLookback |
| + if floor < 0 { |
| + floor = 0 |
| + } |
| + for i := at - 1; i > floor; i-- { |
| + if text[i] == '\n' { |
| + return i + 1 |
| + } |
| + if text[i] != ' ' || (text[i-1] != '.' && text[i-1] != '!' && text[i-1] != '?') { |
| + continue |
| + } |
| + // A full stop followed by a space. Whether it ends a sentence |
| + // depends on the word it is attached to. |
| + w := i |
| + for w > floor { |
| + r, size := utf8.DecodeLastRuneInString(text[floor:w]) |
| + if size == 0 || isWordBreak(r) { |
| + break |
| + } |
| + w -= size |
| + } |
| + if isAbbreviation(text[w:i]) { |
| + continue |
| + } |
| + return i + 1 |
| + } |
| + return floor |
| +} |
| + |
| +// isWordBreak ends the backward word walk. **NBSP counts**, and it did |
| +// not at first: Word and Notion glue "Contact\u00a0Dr." into one token, |
| +// which then matched no abbreviation, so the boundary was taken and the |
| +// label came back as "Smith's direct line" — the composing failure the |
| +// abbreviation list exists to prevent, arriving through a space nobody |
| +// can see. |
| +func isWordBreak(r rune) bool { |
| + return r == ' ' || r == '\n' || r == '\t' || r == '\u00a0' |
| +} |
| + |
| +// isAbbreviation reports whether a token ending in "." is one that does |
| +// not end a sentence. |
| +// |
| +// The fixed list, plus **any single letter**, which covers the initials |
| +// the list cannot enumerate: "J. Smith's line:" yielded "Smith's line" |
| +// before this, and there is no list of every initial. |
| +func isAbbreviation(tok string) bool { |
| + if abbreviations[strings.ToLower(tok)] { |
| + return true |
| + } |
| + // Decoded as RUNES, not bytes. `len(tok) == 2 && rune(tok[0])` read a |
| + // raw byte as a code point, so "Planta 2\u00ba. Tel\u00e9fono:" saw |
| + // 0xBA as the letter "\u00ba" and swallowed the sentence boundary, |
| + // while "\u00c9. Dupont's line:" was still cut to "Dupont's line" — |
| + // wrong in both directions at once, and the second is the composing |
| + // direction §8.8 calls the unsafe one. |
| + r, size := utf8.DecodeRuneInString(tok) |
| + return size > 0 && len(tok) == size+1 && tok[size] == '.' && unicode.IsLetter(r) |
| +} |
| + |
| +// labelFor implements §8.8's five rules. It returns the label and whether |
| +// this kind must render no box at all. |
| +// |
| +// The order of the rules is the whole of it, and each one is here because |
| +// a fixture broke without it — see gate 37, which carries one per case. |
| +func labelFor(text string, start int, spans [][2]int, heading, title string) (string, bool) { |
| + b := sentenceStart(text, start) |
| + // **An earlier recognised span inside the run is not part of the |
| + // label.** Without this, "Phone: 01 547 7854 - 01 598 1234" labels |
| + // the SECOND number with the first one's value — a box whose left |
| + // column is another phone number. |
| + // |
| + // A binary search over spans sorted by start, not a scan of all of |
| + // them: scanning made this quadratic in the facts a chunk holds, and |
| + // a chunk can hold tens of thousands of them. |
| + if e := lastSpanEndBefore(spans, start); e > b { |
| + b = e |
| + } |
| + run := strings.TrimRight(text[b:start], " ") |
| + if r, size := utf8.DecodeLastRuneInString(run); size > 0 && isSeparator(r) { |
| + lbl := strings.TrimSpace(run[:len(run)-size]) |
| + switch { |
| + case lbl == "": |
| + // A markdown bullet: chunk.Split keeps lines verbatim, so |
| + // "- +353 1 547 7854" gives a run of "-", which ends in a |
| + // separator and names nothing. Falling through keeps a plain |
| + // contact list boxable; returning "" would make two bullets |
| + // collide and suppress it. |
| + case utf8.RuneCountInString(lbl) > MaxLabel: |
| + return "", true |
| + default: |
| + return lbl, false |
| + } |
| + } |
| + if heading != "" { |
| + return heading, false |
| + } |
| + return title, false |
| +} |
| + |
| +// lastSpanEndBefore returns the end of the latest span finishing at or |
| +// before at. spans must be sorted by END — see the sort in Scan. |
| +func lastSpanEndBefore(spans [][2]int, at int) int { |
| + lo, hi, out := 0, len(spans)-1, 0 |
| + for lo <= hi { |
| + mid := (lo + hi) / 2 |
| + if spans[mid][1] <= at { |
| + if spans[mid][1] > out { |
| + out = spans[mid][1] |
| + } |
| + lo = mid + 1 |
| + } else { |
| + hi = mid - 1 |
| + } |
| + } |
| + return out |
| +} |
| + |
| +// --------------------------------------------------------------------- |
| +// Scan. |
| +// --------------------------------------------------------------------- |
| + |
| +// Scan returns every validated fact of the wanted kinds in text. |
| +// |
| +// heading and title are the label fallbacks, in that order, and title is |
| +// the last one on purpose: an upload with no headings at all is the |
| +// common pasted-file shape, and falling through to an empty label there |
| +// is the case every fixture carrying a heading would hide. |
| +// |
| +// **This runs over a passage that has already passed §4.1.** Nothing in |
| +// this package knows about authorization, and that is the point: by |
| +// construction there is no way to reach it with a passage the reader may |
| +// not see, because the caller has only the ruled results to hand it. |
| +func Scan(text, heading, title string, want []Kind) []Found { |
| + wanted := map[Kind]bool{} |
| + for _, k := range want { |
| + wanted[k] = true |
| + } |
| + |
| + // Every span of EVERY kind is located first, and every label is |
| + // derived afterwards. Two separate reasons, both found by review: |
| + // |
| + // - labelFor needs to know about the spans to its left, and cannot |
| + // if it is called as they are found. |
| + // - **the spans it needs are of every kind, not just the wanted |
| + // ones.** Locating only what was asked for made a fact's label |
| + // depend on which kinds the query named: "Email paul@tito.io or |
| + // phone: 01 547 7854" asked for `phone` alone came back labelled |
| + // "Email paul@tito.io or phone", one fact's value inside |
| + // another's label — which is exactly what the earlier-span rule |
| + // exists to stop, reappearing because the rule could not see the |
| + // span. |
| + type hit struct { |
| + kind Kind |
| + at [2]int |
| + span string |
| + canonical string |
| + } |
| + var hits []hit |
| + var spans [][2]int |
| + for _, r := range recognisers { |
| + for _, m := range r.re.FindAllStringIndex(text, -1) { |
| + span := text[m[0]:m[1]] |
| + c := r.canon(span) |
| + if !r.valid(span, c) { |
| + continue |
| + } |
| + spans = append(spans, [2]int{m[0], m[1]}) |
| + if wanted[r.kind] { |
| + hits = append(hits, hit{r.kind, [2]int{m[0], m[1]}, span, c}) |
| + } |
| + } |
| + } |
| + // Sorted by END, not by start, because lastSpanEndBefore searches on |
| + // the end and spans NEST: a phone number inside an email address, or |
| + // inside an IBAN, is a real pair. Sorted by start the predicate is |
| + // non-monotonic and the binary search can miss the enclosing span, |
| + // which puts a recognised value back inside another fact's label. |
| + sort.Slice(spans, func(i, j int) bool { return spans[i][1] < spans[j][1] }) |
| + sort.SliceStable(hits, func(i, j int) bool { return hits[i].at[0] < hits[j].at[0] }) |
| + |
| + // **Deduped BEFORE labels are derived, not after.** A label costs a |
| + // backward walk, and a chunk repeating one value ten thousand times |
| + // needs exactly one. The caller dedupes across passages on the same |
| + // key and keeps the first, so doing it here keeps the same |
| + // occurrence and changes no outcome. |
| + seen := map[Kind]map[string]bool{} |
| + var out []Found |
| + for _, k := range order { |
| + if !wanted[k] { |
| + continue |
| + } |
| + for _, h := range hits { |
| + if h.kind != k { |
| + continue |
| + } |
| + if seen[k] == nil { |
| + seen[k] = map[string]bool{} |
| + } |
| + if seen[k][h.canonical] { |
| + continue |
| + } |
| + seen[k][h.canonical] = true |
| + lbl, noBox := labelFor(text, h.at[0], spans, heading, title) |
| + out = append(out, Found{ |
| + Kind: h.kind, |
| + Span: h.span, |
| + Canonical: h.canonical, |
| + Label: lbl, |
| + NoBox: noBox, |
| + }) |
| + } |
| + } |
| + return out |
| +} |
| + |
| +// Kinds returns the render order, so a caller applying §8.8's per-kind |
| +// rules produces the same boxes in the same places on every request. |
| +func Kinds() []Kind { return append([]Kind(nil), order...) } |
diff --git a/internal/memoria/fact/fact_test.go b/internal/memoria/fact/fact_test.go| new file mode 100644 |
| index 0000000..ef8be3b |
| --- /dev/null |
| +++ b/internal/memoria/fact/fact_test.go |
| @@ -0,0 +1,540 @@ |
| +package fact |
| + |
| +import ( |
| + "reflect" |
| + "sort" |
| + "strings" |
| + "testing" |
| + "time" |
| +) |
| + |
| +// The corpus this package is judged against is index/rank_test.go's, |
| +// which was built from the live instance where Paul asked for this |
| +// feature. Quoted here rather than paraphrased: a fixture rewritten from |
| +// the same model as the code cannot catch a wrong assumption, it records |
| +// one (AGENTS.md §Review). |
| +const officeChunk = "Turn the knob. The office phone number is: +353 (0)1 547 7854. " + |
| + "We also have the following unused phone numbers: +353 (0)1 598 1234." |
| + |
| +const numbersChunk = "Our company number is 566334. Our VAT number is IE3384527RH. " + |
| + "Our DUNS/D-U-N-S number is 985632364." |
| + |
| +const bankChunk = "The account number is 12345678 and the sort code is 93-00-11." |
| + |
| +func TestWantedNamesTheKind(t *testing.T) { |
| + for _, c := range []struct { |
| + q string |
| + want []Kind |
| + }{ |
| + {"phone number", []Kind{Phone}}, |
| + {"what's the office phone number?", []Kind{Phone}}, |
| + {"eircode", []Kind{Eircode}}, |
| + {"post code", []Kind{Eircode}}, |
| + {"vat number", []Kind{VAT}}, |
| + {"iban", []Kind{IBAN}}, |
| + {"bank details", []Kind{IBAN}}, |
| + {"email address", []Kind{Email}}, |
| + {"e-mail", []Kind{Email}}, |
| + {"phone and email", []Kind{Phone, Email}}, |
| + } { |
| + if got := Wanted(c.q); !reflect.DeepEqual(got, c.want) { |
| + t.Errorf("Wanted(%q) = %v, want %v", c.q, got, c.want) |
| + } |
| + } |
| +} |
| + |
| +// The words that must NOT prime, and each is here because priming on it |
| +// would put a box on an ordinary query. |
| +// |
| +// "address" is the interesting one: it is ambiguous between a postal |
| +// address and an email address, and only one of those has anything |
| +// resembling a validator (§8.8, decision 15). |
| +func TestWantedRefusesGenericWords(t *testing.T) { |
| + for _, q := range []string{"address", "number", "office address", "account number", "", "revenue"} { |
| + if got := Wanted(q); len(got) != 0 { |
| + t.Errorf("Wanted(%q) = %v, want nothing", q, got) |
| + } |
| + } |
| +} |
| + |
| +// A generous lexicon is affordable BECAUSE of the validator, and this is |
| +// the test that says so: "mobile app" primes Phone and then finds |
| +// nothing, so the cost is a box that does not appear. |
| +func TestAGenerousLexiconCostsNothingWithoutAValidCandidate(t *testing.T) { |
| + if got := Wanted("mobile app"); !reflect.DeepEqual(got, []Kind{Phone}) { |
| + t.Fatalf("Wanted(mobile app) = %v, want [phone]", got) |
| + } |
| + if got := Scan("We shipped the mobile app in June.", "Product", "Notes", []Kind{Phone}); len(got) != 0 { |
| + t.Errorf("Scan found %v in text with no phone number", got) |
| + } |
| +} |
| + |
| +// --------------------------------------------------------------------- |
| +// Validators, against values checkable outside this repo. |
| +// --------------------------------------------------------------------- |
| + |
| +// The reject rows are the load-bearing ones. An accept-only table passes |
| +// against a validator that returns true — the guard written to pass, in |
| +// its purest form. |
| +func TestVATCheckLetter(t *testing.T) { |
| + for _, c := range []struct { |
| + in string |
| + ok bool |
| + }{ |
| + {"IE6388047V", true}, // Google Ireland |
| + {"IE3628739L", true}, // published example |
| + {"IE3384527RH", true}, // index/rank_test.go's own corpus |
| + {"IE1234567X", false}, // invented: right shape, wrong check letter |
| + {"IE6388047W", false}, // one letter off the real one |
| + {"IE8Z49289F", false}, // old style, refused deliberately |
| + } { |
| + if got := vatValid(c.in); got != c.ok { |
| + t.Errorf("vatValid(%q) = %v, want %v", c.in, got, c.ok) |
| + } |
| + } |
| +} |
| + |
| +func TestIBANModulus97(t *testing.T) { |
| + for _, c := range []struct { |
| + in string |
| + ok bool |
| + }{ |
| + {"IE29AIBK93115212345678", true}, |
| + {"GB82WEST12345698765432", true}, |
| + {"DE89370400440532013000", true}, |
| + {"IE29AIBK93115212345679", false}, // one digit changed |
| + {"IE29AIBK9311521234567", false}, // one digit short |
| + {"XX29AIBK93115212345678", false}, // unknown country |
| + } { |
| + if got := ibanValid(c.in); got != c.ok { |
| + t.Errorf("ibanValid(%q) = %v, want %v", c.in, got, c.ok) |
| + } |
| + } |
| +} |
| + |
| +// D6W is Dublin 6W and its routing key does not fit [A-Z][0-9][0-9]; B is |
| +// not in the routing-key alphabet at all. Both are the cases a pattern |
| +// written from memory gets wrong. |
| +func TestEircodeAlphabet(t *testing.T) { |
| + for _, c := range []struct { |
| + in string |
| + ok bool |
| + }{ |
| + {"A65 F4E2", true}, // the published example |
| + {"D02 X285", true}, |
| + {"D6W 1234", true}, |
| + {"D02X285", true}, |
| + {"B12 3456", false}, // B is not a routing-key letter |
| + {"D02 XB85", false}, // B is not in the identifier alphabet |
| + } { |
| + got := len(Scan("Eircode "+c.in+" here.", "", "T", []Kind{Eircode})) > 0 |
| + if got != c.ok { |
| + t.Errorf("eircode %q recognised = %v, want %v", c.in, got, c.ok) |
| + } |
| + } |
| +} |
| + |
| +// **A bare run of digits is never a phone number.** This is the whole of |
| +// what stands between the box and rank_test.go's own first line: the DUNS |
| +// number there is nine digits and a plausible Irish mobile. |
| +func TestABareRunOfDigitsIsNotAPhoneNumber(t *testing.T) { |
| + for _, c := range []struct { |
| + text string |
| + want int |
| + }{ |
| + {officeChunk, 2}, |
| + {numbersChunk, 0}, // 566334, IE3384527RH, 985632364 |
| + {bankChunk, 0}, // 12345678, 93-00-11 |
| + {"Virgin Media - login - Office 7: account number 80854641", 0}, |
| + {"The deadline is 01-02-2024 for everyone.", 0}, // a date leads with 0 and has separators |
| + {"Call 0871234567 today.", 0}, // no separators: the stated cost |
| + {"Call 087 123 4567 today.", 1}, |
| + } { |
| + if got := len(Scan(c.text, "H", "T", []Kind{Phone})); got != c.want { |
| + t.Errorf("Scan(%q) found %d phone numbers, want %d", c.text, got, c.want) |
| + } |
| + } |
| +} |
| + |
| +// Two spellings of one number are ONE candidate. A dedupe keyed on the |
| +// span rather than the canonical form passes any fixture that spells it |
| +// the same way twice, which is why this one does not. |
| +func TestTwoSpellingsCanonicaliseTogether(t *testing.T) { |
| + a := Scan("Office: +353 (0)1 547 7854", "H", "T", []Kind{Phone}) |
| + b := Scan("Office: 01 547 7854", "H", "T", []Kind{Phone}) |
| + if len(a) != 1 || len(b) != 1 { |
| + t.Fatalf("want one each, got %d and %d", len(a), len(b)) |
| + } |
| + if a[0].Canonical != b[0].Canonical { |
| + t.Errorf("canonical %q != %q", a[0].Canonical, b[0].Canonical) |
| + } |
| + if a[0].Span == b[0].Span { |
| + t.Errorf("spans should differ; both %q", a[0].Span) |
| + } |
| +} |
| + |
| +// --------------------------------------------------------------------- |
| +// Spans are the document's bytes. |
| +// --------------------------------------------------------------------- |
| + |
| +func TestSpanIsByteIdenticalToTheDocument(t *testing.T) { |
| + for _, c := range []struct{ text, want string }{ |
| + {"Office: +353 (0)1 547 7854.", "+353 (0)1 547 7854"}, |
| + {"Office: +353 (0)1 547 7854 ext. 12.", "+353 (0)1 547 7854 ext. 12"}, |
| + // Pasted from Word or Notion. Go's \s does not match U+00A0, so a |
| + // recogniser written with it finds NOTHING here and this case |
| + // fails on presence rather than on the bytes — which is the |
| + // design demand, not an accident. |
| + {"Office: +353 (0)1 547 7854.", "+353 (0)1 547 7854"}, |
| + } { |
| + got := Scan(c.text, "H", "T", []Kind{Phone}) |
| + if len(got) != 1 { |
| + t.Fatalf("Scan(%q) found %d, want 1", c.text, len(got)) |
| + } |
| + if got[0].Span != c.want { |
| + t.Errorf("span = %q, want %q", got[0].Span, c.want) |
| + } |
| + } |
| +} |
| + |
| +// --------------------------------------------------------------------- |
| +// The label, one case per rule and one per failure the reviews found. |
| +// --------------------------------------------------------------------- |
| + |
| +func TestLabelComesFromTheDocument(t *testing.T) { |
| + long := strings.Repeat("x", MaxLabel) // exactly at the cap |
| + tooLong := strings.Repeat("x", MaxLabel+1) // one over |
| + |
| + for _, c := range []struct { |
| + name string |
| + text string |
| + head string |
| + title string |
| + want []string |
| + noBox bool |
| + }{ |
| + { |
| + name: "a labelled line", |
| + text: "Office: +353 1 547 7854", head: "Contact", title: "Tito", |
| + want: []string{"Office"}, |
| + }, |
| + { |
| + // The motivating fixture. Draft 1 of §8.8 required the value |
| + // to start its own line, so both numbers fell to the heading |
| + // trail, came back labelled identically, and the box would |
| + // have shown "unused" and "office" as equals. |
| + name: "the office fixture keeps its two sentences apart", |
| + text: officeChunk, head: "Office Address", title: "Office", |
| + want: []string{"The office phone number is", "We also have the following unused phone numbers"}, |
| + }, |
| + { |
| + name: "no separator falls to the trail, never the clause", |
| + text: "Ring the office on +353 1 547 7854 any weekday.", head: "Contact", title: "Tito", |
| + want: []string{"Contact"}, |
| + }, |
| + { |
| + // chunk.Split keeps lines verbatim, so the run is "-", which |
| + // ends in a separator and names nothing. Returning "" here |
| + // makes two bullets collide and suppresses the box for a |
| + // plain contact list. |
| + name: "a markdown bullet falls through rather than yielding an empty label", |
| + text: "- +353 1 547 7854\n- +353 1 598 1234", head: "Contact", title: "Tito", |
| + want: []string{"Contact", "Contact"}, |
| + }, |
| + { |
| + // Without the earlier-span rule the second label is |
| + // "Phone: 01 547 7854" — one phone number naming another. |
| + name: "an adjacent value is not part of the next label", |
| + text: "Phone: 01 547 7854 - 01 598 1234", head: "Contact", title: "Tito", |
| + want: []string{"Phone", "Contact"}, |
| + }, |
| + { |
| + // A boundary placed too late composes a label the document |
| + // never wrote. Draft 2 claimed this direction was safe. |
| + name: "an abbreviation does not end the sentence", |
| + text: "Dr. Smith's direct line: 01 547 7854", head: "Contact", title: "Tito", |
| + want: []string{"Dr. Smith's direct line"}, |
| + }, |
| + { |
| + // "1." is not an abbreviation, so the boundary IS taken and |
| + // the list marker is not part of the label. The rune check in |
| + // isAbbreviation is the only thing holding this. |
| + name: "a numbered list marker is not part of the label", |
| + text: "1. Office phone: 01 547 7854", head: "Contact", title: "Tito", |
| + want: []string{"Office phone"}, |
| + }, |
| + { |
| + name: "no headings at all falls to the title, never empty", |
| + text: "Ring us on +353 1 547 7854 any weekday.", head: "", title: "Pasted contacts", |
| + want: []string{"Pasted contacts"}, |
| + }, |
| + { |
| + name: "a label exactly at the cap renders", |
| + text: long + ": 01 547 7854", head: "Contact", title: "Tito", |
| + want: []string{long}, |
| + }, |
| + { |
| + // Substituting the trail would replace the document's own |
| + // warning with a vaguer word; dropping just this candidate |
| + // would turn an ambiguity into a confident single answer. |
| + name: "a label one over the cap renders no box for the kind", |
| + text: tooLong + ": 01 547 7854", head: "Contact", title: "Tito", |
| + noBox: true, |
| + }, |
| + } { |
| + t.Run(c.name, func(t *testing.T) { |
| + got := Scan(c.text, c.head, c.title, []Kind{Phone}) |
| + if c.noBox { |
| + if len(got) != 1 || !got[0].NoBox { |
| + t.Fatalf("want one NoBox result, got %+v", got) |
| + } |
| + return |
| + } |
| + var labels []string |
| + for _, f := range got { |
| + if f.NoBox { |
| + t.Fatalf("unexpected NoBox in %+v", got) |
| + } |
| + labels = append(labels, f.Label) |
| + } |
| + if !reflect.DeepEqual(labels, c.want) { |
| + t.Errorf("labels = %q, want %q", labels, c.want) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// Ordinary document text that earlier revisions of these recognisers |
| +// boxed. Every row here was a live false positive found by review, not |
| +// an invented one, and each names the class it stands for — because the |
| +// failure they share is that a cited fact which is really an English |
| +// word reads exactly like a correct answer (§8.7's "cites correctly |
| +// while being wrong"). |
| +func TestOrdinaryTextIsNotAFact(t *testing.T) { |
| + for _, c := range []struct { |
| + kind Kind |
| + text string |
| + why string |
| + }{ |
| + {Eircode, "Take the N11 then exit at Bray.", "a road number and a lowercase word"}, |
| + {Eircode, "Form P45 then P60.", "a form code and a lowercase word"}, |
| + {Eircode, "The A65 area near Sligo is quiet.", "a routing key and a lowercase word"}, |
| + {Phone, "The deadline is 01-02-2024 10:30 sharp.", "a date followed by a time"}, |
| + {Phone, "Due 01.02.2024 10:30.", "the same with full stops"}, |
| + {Phone, "Numbers drawn: 01 05 12 23 34", "five two-digit groups"}, |
| + {Phone, "Growth of +10 000 000 users this year.", "a plus and grouped thousands"}, |
| + {Phone, "Companies House 12345678 is the reference.", "a bare run of digits"}, |
| + {Phone, "Card 0123 4567 8901 2345 expires soon.", "sixteen digits in four groups"}, |
| + // Germany and France really do write more groups than this, and |
| + // refusing their numbers is the stated cost of never truncating |
| + // one. The alternative was boxing "+49 30 12 34 56". |
| + {Phone, "Berlin +49 30 12 34 56 78", "six groups"}, |
| + {Phone, "Paris +33 1 23 45 67 89", "six groups"}, |
| + } { |
| + if got := Scan(c.text, "Contact", "Notes", []Kind{c.kind}); len(got) != 0 { |
| + t.Errorf("%s: %q boxed %q as a %s", c.why, c.text, got[0].Span, c.kind) |
| + } |
| + } |
| +} |
| + |
| +// The shapes a document actually writes, which must all still be found. |
| +// Half of these are here because a fix for the table above broke them: |
| +// a recogniser tightened until nothing false matches is a recogniser |
| +// that finds nothing at all, and only having both tables makes that |
| +// visible. |
| +func TestTheShapesDocumentsActuallyWrite(t *testing.T) { |
| + for _, c := range []struct { |
| + kind Kind |
| + text string |
| + want string |
| + }{ |
| + {Phone, "Office phone: +353 (0)1 547 7854", "+353 (0)1 547 7854"}, |
| + {Phone, "Ring 087 123 4567 today.", "087 123 4567"}, |
| + {Phone, "Call 01-547-7854 for support.", "01-547-7854"}, |
| + // The commonest international spelling of an Irish mobile, and |
| + // the one a five-digit group cap rendered as "+353 87 12345" — a |
| + // different, well-formed, confidently cited number. |
| + {Phone, "Mobile: +353 87 1234567", "+353 87 1234567"}, |
| + {Phone, "Mobile: +353 (0)87 1234567", "+353 (0)87 1234567"}, |
| + {Phone, "Office phone: +353 1 5477854", "+353 1 5477854"}, |
| + {Phone, "Call 01 5477854 today.", "01 5477854"}, |
| + {Phone, "Tel +44 20 7123 4567 in London.", "+44 20 7123 4567"}, |
| + {Eircode, "Eircode: D02 X285", "D02 X285"}, |
| + {Eircode, "We are at A65 F4E2.", "A65 F4E2"}, |
| + // Grouped, and followed by a BIC — which is how bank details are |
| + // written, and which found nothing at all until the grouped |
| + // alternative was fixed to groups of four. |
| + {IBAN, "Our IBAN is IE29 AIBK 9311 5212 3456 78 and our BIC is AIBKIE2D.", "IE29 AIBK 9311 5212 3456 78"}, |
| + {IBAN, "IBAN: IE29 AIBK 9311 5212 3456 78 BIC AIBKIE2D", "IE29 AIBK 9311 5212 3456 78"}, |
| + {IBAN, "IBAN IE29AIBK93115212345678.", "IE29AIBK93115212345678"}, |
| + // Countries whose length is a multiple of four leave no tail for |
| + // a following token to hide in, so the tail must be digits only. |
| + {IBAN, "IBAN: BE68 5390 0754 7034 BIC: GEBABEBB", "BE68 5390 0754 7034"}, |
| + {IBAN, "IBAN ES91 2100 0418 4502 0005 1332 BIC CAIXESBB", "ES91 2100 0418 4502 0005 1332"}, |
| + {IBAN, "Norway: NO93 8601 1117 947.", "NO93 8601 1117 947"}, |
| + {IBAN, "Poland PL61 1090 1014 0000 0712 1981 2874 please.", "PL61 1090 1014 0000 0712 1981 2874"}, |
| + {VAT, "Our VAT number is IE3384527RH.", "IE3384527RH"}, |
| + } { |
| + got := Scan(c.text, "Contact", "Notes", []Kind{c.kind}) |
| + if len(got) != 1 { |
| + t.Errorf("%q found %d, want 1", c.text, len(got)) |
| + continue |
| + } |
| + if got[0].Span != c.want { |
| + t.Errorf("%q span = %q, want %q", c.text, got[0].Span, c.want) |
| + } |
| + } |
| +} |
| + |
| +// A label must not depend on which kinds the query named. |
| +// |
| +// Scan locates spans of EVERY kind and emits only the wanted ones, |
| +// because the earlier-span rule can only skip a span it can see. Asking |
| +// for `phone` alone used to yield "Email paul@tito.io or phone" — one |
| +// fact's value inside another fact's label, which is exactly what that |
| +// rule exists to prevent. |
| +func TestALabelDoesNotDependOnWhichKindsWereAskedFor(t *testing.T) { |
| + const text = "Email paul@tito.io or phone: 01 547 7854" |
| + alone := Scan(text, "Contact", "T", []Kind{Phone}) |
| + both := Scan(text, "Contact", "T", []Kind{Phone, Email}) |
| + if len(alone) != 1 { |
| + t.Fatalf("phone alone found %d, want 1", len(alone)) |
| + } |
| + if strings.Contains(alone[0].Label, "@") { |
| + t.Errorf("label %q carries another fact's value", alone[0].Label) |
| + } |
| + var phone Found |
| + for _, f := range both { |
| + if f.Kind == Phone { |
| + phone = f |
| + } |
| + } |
| + if phone.Label != alone[0].Label { |
| + t.Errorf("label changed with the query: %q alone, %q with email", alone[0].Label, phone.Label) |
| + } |
| +} |
| + |
| +// Label derivation was quadratic in the facts a chunk holds, and |
| +// chunk.Split bounds a chunk by WORDS — so a 2 MiB upload with no |
| +// whitespace is one chunk, one line, and minutes of search latency |
| +// bought by one person's file. Measured before the fix: 48.5s at 1 MB. |
| +// |
| +// **A DOUBLING RATIO, not a wall-clock limit, and the first two versions |
| +// of this test were both unreachable.** Version one repeated a single |
| +// value, so dedupe left one label to derive, at byte zero, where the |
| +// backward walk returns immediately: a review mutated the lookback bound |
| +// to 1<<40 and reverted the binary search to a linear scan and both |
| +// stayed green. Version two used distinct values and a five-second |
| +// limit, and both mutations stayed green again — 1.5s and 0.5s are |
| +// slow, and still well inside any limit loose enough not to flake on a |
| +// loaded machine. |
| +// |
| +// So the assertion is the SHAPE of the curve. Quadratic work quadruples |
| +// when the input doubles; linear work doubles. Measured on this fixture, |
| +// best of five: **2.32 pristine, 3.40 with the linear scan, 4.49 with |
| +// the bound removed.** Three is the line, and it is the only threshold |
| +// here that separates the real implementation from both mutations. |
| +// |
| +// Best-of-five because the minimum of several runs is the robust |
| +// estimator for CPU-bound work: a loaded machine makes runs slower, not |
| +// faster, so noise inflates the measurement and the minimum discards it. |
| +func TestLabellingIsNotQuadraticInAChunksFacts(t *testing.T) { |
| + scan := func(n int) (time.Duration, int) { |
| + var b strings.Builder |
| + for i := 0; i < n; i++ { |
| + b.WriteString(eircodeNo(i)) |
| + b.WriteByte(',') // a separator, NOT a sentence boundary |
| + } |
| + text, best, found := b.String(), time.Hour, 0 |
| + for r := 0; r < 5; r++ { |
| + st := time.Now() |
| + found = len(Scan(text, "H", "T", []Kind{Eircode})) |
| + if d := time.Since(st); d < best { |
| + best = d |
| + } |
| + } |
| + return best, found |
| + } |
| + |
| + small, n := scan(10000) |
| + large, m := scan(20000) |
| + |
| + // The presence half: without it this measures the cost of finding |
| + // nothing, which is cheap and flat whatever the implementation does. |
| + if n != 10000 || m != 20000 { |
| + t.Fatalf("found %d and %d candidates, want 10000 and 20000 distinct", n, m) |
| + } |
| + if ratio := float64(large) / float64(small); ratio > 3 { |
| + t.Errorf("doubling the input multiplied the time by %.2f (%v to %v); "+ |
| + "linear is about 2 and quadratic about 4, so label derivation is quadratic again", |
| + ratio, small, large) |
| + } |
| +} |
| + |
| +// eircodeNo builds the i-th distinct valid eircode: a fixed routing key |
| +// and a base-25 identifier over the published alphabet. |
| +func eircodeNo(i int) string { |
| + const alphabet = "0123456789ACDEFHKNPRTVWXY" |
| + id := make([]byte, 4) |
| + for j := 3; j >= 0; j-- { |
| + id[j] = alphabet[i%len(alphabet)] |
| + i /= len(alphabet) |
| + } |
| + return "D02 " + string(id) |
| +} |
| + |
| +// lastSpanEndBefore searches on a span's END, so Scan sorts by end — and |
| +// that matters because spans NEST. A phone number sits inside an email |
| +// address and inside an IBAN, both of which really occur. Sorted by |
| +// start, the predicate "ends at or before here" is not monotonic, the |
| +// binary search can walk past the enclosing span, and a recognised value |
| +// goes back into another fact's label — the thing the earlier-span rule |
| +// exists to prevent. |
| +// |
| +// A direct unit test rather than a corpus fixture, because the timing |
| +// test above cannot see correctness and the corpus shapes that trigger |
| +// nesting are contrived enough that nobody would think to write one. |
| +func TestLastSpanEndBeforeHandlesNestedSpans(t *testing.T) { |
| + // A long outer span with two short ones inside it, as an email |
| + // address containing phone-shaped runs produces. |
| + spans := [][2]int{{12, 20}, {22, 30}, {4, 40}} |
| + sort.Slice(spans, func(i, j int) bool { return spans[i][1] < spans[j][1] }) |
| + for _, c := range []struct{ at, want int }{ |
| + {3, 0}, // before everything |
| + {20, 20}, // exactly on an inner end |
| + {21, 20}, |
| + {30, 30}, |
| + {45, 40}, // the OUTER span, which a start-sorted search can miss |
| + } { |
| + if got := lastSpanEndBefore(spans, c.at); got != c.want { |
| + t.Errorf("lastSpanEndBefore(%d) = %d, want %d", c.at, got, c.want) |
| + } |
| + } |
| +} |
| + |
| +// The same rule, asserted through Scan rather than against the helper, |
| +// because the helper's contract and Scan's obligation to meet it are two |
| +// different things. |
| +// |
| +// TestLastSpanEndBeforeHandlesNestedSpans sorts its own input, so it |
| +// cannot see Scan sorting by the wrong key — mutating that sort left it |
| +// green. This is the fixture that catches it: an email address whose |
| +// local part contains two phone-shaped runs, so a phone span really does |
| +// nest inside an email span. Sorted by start, the third number comes back |
| +// labelled "Mail info.087.123.4567" — a recognised value inside another |
| +// fact's label, which is the exact failure the earlier-span rule exists |
| +// to prevent, reappearing through the search that implements it. |
| +func TestScanSortsSpansSoNestingCannotLeakIntoALabel(t *testing.T) { |
| + const text = "Tel 01 547 7854. Mail info.087.123.4567-087-123-4568@example.ie" |
| + got := Scan(text, "Contact", "Notes", []Kind{Phone, Email}) |
| + if len(got) < 3 { |
| + t.Fatalf("found %d facts, want at least three: this fixture needs the nesting to be wrong about", len(got)) |
| + } |
| + for _, f := range got { |
| + for _, other := range got { |
| + if other.Span != f.Span && strings.Contains(f.Label, other.Span) { |
| + t.Errorf("%s label %q contains another fact's value %q", f.Kind, f.Label, other.Span) |
| + } |
| + } |
| + } |
| +} |
diff --git a/internal/memoria/factbox_test.go b/internal/memoria/factbox_test.go| new file mode 100644 |
| index 0000000..d9be148 |
| --- /dev/null |
| +++ b/internal/memoria/factbox_test.go |
| @@ -0,0 +1,313 @@ |
| +package memoria |
| + |
| +import ( |
| + "context" |
| + "testing" |
| + |
| + "amadan.net/oficina/suite" |
| + |
| + "amadan.net/oficina/memoria/internal/memoria/sibling" |
| +) |
| + |
| +// perItemCustody rules each candidate on its own item id. |
| +// |
| +// **fixedCustody cannot do this**, and that is why this type exists |
| +// rather than a field being added to it: it returns sibling.Decide(give) |
| +// for every candidate from ONE field, so every fixture it builds has all |
| +// its passages visible or none of them. Gate 34 is entirely about pages |
| +// where some are and some are not, and drafts 1 and 2 of §8.8 both said |
| +// the gate could run against "memoriatest's sibling stub, which exists". |
| +// No such stub exists. §10 opens by naming that failure — a gate naming a |
| +// test that was renamed away — and the section citing it committed it |
| +// twice. |
| +type perItemCustody struct { |
| + apps map[string]sibling.Sibling |
| + // give is keyed by item id. A candidate whose item id is absent is |
| + // VISIBLE, so a fixture names only the exceptions it cares about. |
| + give map[string]suite.Authorization |
| +} |
| + |
| +func (p *perItemCustody) Sibling(source string) (sibling.Sibling, bool) { |
| + s, ok := p.apps[source] |
| + return s, ok |
| +} |
| + |
| +func (p *perItemCustody) Rule(_ context.Context, _ suite.Subject, cands []sibling.Candidate) []sibling.Disposition { |
| + out := make([]sibling.Disposition, len(cands)) |
| + for i, c := range cands { |
| + a, ok := p.give[c.Item.ItemID] |
| + if !ok { |
| + a = suite.VISIBLE |
| + } |
| + out[i] = sibling.Decide(a) |
| + } |
| + return out |
| +} |
| + |
| +// eircodeCorpus stores one sibling document per (title, eircode) pair, |
| +// each under its own item id so the fake above can rule them apart. |
| +func eircodeCorpus(t *testing.T, cus Custody, docs [][3]string) *Store { |
| + t.Helper() |
| + s := storeForTest(t) |
| + s.custody = cus |
| + for _, d := range docs { |
| + if _, err := s.Ingest(context.Background(), IngestSpec{ |
| + Source: "Docs", Key: d[0], Title: d[1], |
| + Body: "## Contact\n\nEircode: " + d[2], |
| + ItemID: d[0], |
| + SiblingRevision: 1, |
| + }); err != nil { |
| + t.Fatalf("ingest %s: %v", d[0], err) |
| + } |
| + } |
| + return s |
| +} |
| + |
| +func searchFacts(t *testing.T, s *Store, q string) Page { |
| + t.Helper() |
| + p, err := s.Search(context.Background(), |
| + suite.Subject{Kind: suite.SubjectAddress, Value: "paul@tito.io"}, q, 25) |
| + if err != nil { |
| + t.Fatalf("search: %v", err) |
| + } |
| + return p |
| +} |
| + |
| +func spans(p Page) []string { |
| + var out []string |
| + for _, f := range p.Facts { |
| + out = append(out, f.Span) |
| + } |
| + return out |
| +} |
| + |
| +// GATE 34 — the box never speaks a withheld passage, and every count |
| +// happens after the ruling (§8.8, §10). |
| +// |
| +// **Asserts the struct, never the rendered copy.** Two dispositions that |
| +// render an identical page is how Outcome.Record filed silent rulings |
| +// undetected (§1.9); a page assertion here could not tell case 2 from |
| +// case 3 at all. |
| +// |
| +// Mutations run, each verified red and reverted, 2026-09-14: |
| +// - addFacts ranging the pre-ruling hits instead of p.Results — cases |
| +// 2, 4, 5 and 6 go red. |
| +// - the cap counted before the ruling — case 4 goes red alone. |
| +// - the collision check computed over the unruled set — case 6 alone. |
| +// - citing r.DocumentID of the first hit rather than the first RESULT — |
| +// case 5 alone, and no other fixture can see it because every other |
| +// one uses distinct values per document. |
| +func TestGate34TheFactBoxNeverSpeaksAWithheldPassage(t *testing.T) { |
| + roster := docsRoster() |
| + |
| + // 1. PRESENCE. Both visible, two distinct eircodes, two candidates. |
| + // Without this every absence below is an assertion about an empty |
| + // world — §10's second rule, and gate 4's lesson. |
| + t.Run("presence: two visible passages give two candidates", func(t *testing.T) { |
| + s := eircodeCorpus(t, &perItemCustody{apps: roster}, [][3]string{ |
| + {"a", "Office A", "D02 X285"}, |
| + {"b", "Office B", "A65 F4E2"}, |
| + }) |
| + if got := spans(searchFacts(t, s, "eircode")); len(got) != 2 { |
| + t.Fatalf("spans = %q, want two", got) |
| + } |
| + }) |
| + |
| + // 2. NOT_VISIBLE IS SILENT. §4.1's table makes OmitSilently the only |
| + // disposition that may be silent, and Outcome.Record implements |
| + // exactly that. Draft 1 of §8.8 demanded a notice here, which is |
| + // either unreachable against correct code or an instruction to build |
| + // the existence oracle §4.1 exists to deny. **The empty-notice |
| + // assertion is the load-bearing half of this case.** |
| + t.Run("not_visible: the survivor is boxed and nothing is said", func(t *testing.T) { |
| + s := eircodeCorpus(t, &perItemCustody{ |
| + apps: roster, |
| + give: map[string]suite.Authorization{"b": suite.NOT_VISIBLE}, |
| + }, [][3]string{ |
| + {"a", "Office A", "D02 X285"}, |
| + {"b", "Office B", "A65 F4E2"}, |
| + }) |
| + p := searchFacts(t, s, "eircode") |
| + if got := spans(p); len(got) != 1 || got[0] != "D02 X285" { |
| + t.Fatalf("spans = %q, want just the visible one", got) |
| + } |
| + if len(p.Notices) != 0 { |
| + t.Errorf("notices = %q, want none: a NOT_VISIBLE is the one value that may be silent", p.Notices) |
| + } |
| + }) |
| + |
| + // 3. UNKNOWN IS LOUD. The same fixture, the other disposition. This |
| + // is what draft 1 meant and attached to the wrong one. |
| + t.Run("unknown: the survivor is boxed and the page says so", func(t *testing.T) { |
| + s := eircodeCorpus(t, &perItemCustody{ |
| + apps: roster, |
| + give: map[string]suite.Authorization{"b": suite.UNKNOWN}, |
| + }, [][3]string{ |
| + {"a", "Office A", "D02 X285"}, |
| + {"b", "Office B", "A65 F4E2"}, |
| + }) |
| + p := searchFacts(t, s, "eircode") |
| + if got := spans(p); len(got) != 1 || got[0] != "D02 X285" { |
| + t.Fatalf("spans = %q, want just the visible one", got) |
| + } |
| + if len(p.Notices) == 0 { |
| + t.Error("want a notice: an UNKNOWN is withheld loudly (§6.5)") |
| + } |
| + }) |
| + |
| + // 4. THE CAP COUNTS AFTER THE RULING. Four values, one withheld, |
| + // three left — a box. Counted before the ruling there are four, the |
| + // cap fires, and the reader loses the box because of a document they |
| + // may not see. **No two-document fixture can tell those apart**, |
| + // which is why this one has four. |
| + t.Run("the cap counts the ruled set", func(t *testing.T) { |
| + s := eircodeCorpus(t, &perItemCustody{ |
| + apps: roster, |
| + give: map[string]suite.Authorization{"d": suite.NOT_VISIBLE}, |
| + }, [][3]string{ |
| + {"a", "Office A", "D02 X285"}, |
| + {"b", "Office B", "A65 F4E2"}, |
| + {"c", "Office C", "D6W 1234"}, |
| + {"d", "Office D", "T12 XY45"}, |
| + }) |
| + if got := spans(searchFacts(t, s, "eircode")); len(got) != 3 { |
| + t.Fatalf("spans = %q, want three", got) |
| + } |
| + }) |
| + |
| + // 5. EVERY FIELD OF THE CITATION IS THE SURVIVOR'S. The SAME eircode |
| + // in a visible and a withheld document: one candidate, and all five |
| + // of its fields from the visible one. |
| + // |
| + // **Asserting Title alone was not enough**, and a review mutation |
| + // proved it: taking Title from the ruled result and DocumentID, |
| + // Ordinal, Label and Trail from the first UNRULED occurrence passed |
| + // this case, and would have rendered an href to a document the |
| + // reader may not open, under the withheld document's own heading and |
| + // label text. The two bodies were identical, so nothing could tell |
| + // the fields apart. They differ now, and §10 says assert the tuple. |
| + t.Run("every field of the citation is the visible passage's", func(t *testing.T) { |
| + cus := &perItemCustody{ |
| + apps: roster, |
| + give: map[string]suite.Authorization{"hidden": suite.NOT_VISIBLE}, |
| + } |
| + s := storeForTest(t) |
| + s.custody = cus |
| + ctx := context.Background() |
| + // The withheld one is ingested first AND carries the query's |
| + // words three times, so it ranks above the visible one: if any |
| + // field came from the hits rather than the results, it would be |
| + // this document's. |
| + hidden, err := s.Ingest(ctx, IngestSpec{ |
| + Source: "Docs", Key: "hidden", Title: "Hidden office", |
| + Body: "## Hidden trail\n\nHidden eircode eircode eircode label: D02 X285", |
| + ItemID: "hidden", SiblingRevision: 1, |
| + }) |
| + if err != nil { |
| + t.Fatalf("ingest hidden: %v", err) |
| + } |
| + shown, err := s.Ingest(ctx, IngestSpec{ |
| + Source: "Docs", Key: "shown", Title: "Visible office", |
| + Body: "## Contact\n\nEircode: D02 X285", |
| + ItemID: "shown", SiblingRevision: 1, |
| + }) |
| + if err != nil { |
| + t.Fatalf("ingest shown: %v", err) |
| + } |
| + if hidden == shown { |
| + t.Fatal("the two documents share an id; this fixture cannot be wrong about anything") |
| + } |
| + p := searchFacts(t, s, "eircode") |
| + if len(p.Facts) != 1 { |
| + t.Fatalf("facts = %+v, want one", p.Facts) |
| + } |
| + got := p.Facts[0] |
| + for _, c := range []struct{ name, got, want string }{ |
| + {"title", got.Title, "Visible office"}, |
| + {"trail", got.Trail, "Contact"}, |
| + {"label", got.Label, "Eircode"}, |
| + } { |
| + if c.got != c.want { |
| + t.Errorf("%s = %q, want %q: the box took a field from the passage this reader may not read", c.name, c.got, c.want) |
| + } |
| + } |
| + if got.DocumentID != shown { |
| + t.Errorf("citation links to document %d, want %d: the href points at the withheld document", got.DocumentID, shown) |
| + } |
| + // The fifth field, which the comment above claimed and the |
| + // assertions did not make. Both documents here are single-chunk |
| + // so it cannot currently differ — asserted anyway, because the |
| + // claim should be true and the next fixture may have two. |
| + if got.Ordinal != 0 { |
| + t.Errorf("ordinal = %d, want 0", got.Ordinal) |
| + } |
| + }) |
| + |
| + // 7. A WITHHELD PASSAGE CANNOT SILENCE A KIND. §8.8's rule 4 — a |
| + // document-authored label too long to show renders no box for that |
| + // kind — is a count like the cap and the collision check, and it has |
| + // to happen after the ruling for the same reason. Computed over the |
| + // unruled set, a document the reader may not see takes the box away |
| + // from them, which reports that document's existence. |
| + // |
| + // **No other fixture here has a NoBox candidate at all**, so a |
| + // mutation OR-ing `silenced` from the unruled set passed all six of |
| + // them. Found by review, 2026-09-14. |
| + t.Run("a withheld over-long label does not silence the kind", func(t *testing.T) { |
| + cus := &perItemCustody{ |
| + apps: roster, |
| + give: map[string]suite.Authorization{"hidden": suite.NOT_VISIBLE}, |
| + } |
| + s := storeForTest(t) |
| + s.custody = cus |
| + ctx := context.Background() |
| + for _, d := range [][3]string{ |
| + // Over sixty characters, and carrying the query's word so it |
| + // is actually retrieved — the first version of this fixture |
| + // was green because nothing ever reached the box. |
| + {"hidden", "Hidden office", "The eircode you must never give to any customer or supplier is: T12 XY45"}, |
| + {"shown", "Visible office", "Eircode: D02 X285"}, |
| + } { |
| + if _, err := s.Ingest(ctx, IngestSpec{ |
| + Source: "Docs", Key: d[0], Title: d[1], |
| + Body: "## Contact\n\n" + d[2], |
| + ItemID: d[0], SiblingRevision: 1, |
| + }); err != nil { |
| + t.Fatalf("ingest %s: %v", d[0], err) |
| + } |
| + } |
| + if got := spans(searchFacts(t, s, "eircode")); len(got) != 1 || got[0] != "D02 X285" { |
| + t.Fatalf("spans = %q, want the visible one boxed", got) |
| + } |
| + }) |
| + |
| + // 6. COLLISION COUNTS AFTER THE RULING TOO. Two documents sharing a |
| + // title and a trail, both falling to the trail for a label: over the |
| + // unruled set they collide and the box vanishes, which is the same |
| + // leak as case 4 wearing the collision rule's coat. |
| + t.Run("collision counts the ruled set", func(t *testing.T) { |
| + cus := &perItemCustody{ |
| + apps: roster, |
| + give: map[string]suite.Authorization{"twin-b": suite.NOT_VISIBLE}, |
| + } |
| + s := storeForTest(t) |
| + s.custody = cus |
| + ctx := context.Background() |
| + for i, d := range [][2]string{{"twin-a", "D02 X285"}, {"twin-b", "A65 F4E2"}} { |
| + if _, err := s.Ingest(ctx, IngestSpec{ |
| + Source: "Docs", Key: d[0], Title: "Offices", |
| + // Mid-sentence and with no separator before it, so the |
| + // label falls to the heading trail and the two rows |
| + // would be identical but for their values. |
| + Body: "## Contact\n\nThe eircode " + d[1] + " reaches us.", |
| + ItemID: d[0], |
| + SiblingRevision: int64(i) + 1, |
| + }); err != nil { |
| + t.Fatalf("ingest %s: %v", d[0], err) |
| + } |
| + } |
| + if got := spans(searchFacts(t, s, "eircode")); len(got) != 1 { |
| + t.Fatalf("spans = %q, want the one survivor boxed", got) |
| + } |
| + }) |
| +} |
diff --git a/internal/memoria/handlers.go b/internal/memoria/handlers.go| index 222a657..f041861 100644 |
| --- a/internal/memoria/handlers.go |
| +++ b/internal/memoria/handlers.go |
| @@ -158,6 +158,15 @@ type page struct { |
| // thing: a short page that reads as complete. |
| Notices []string |
| + // Facts is §8.8's box, above the results and below the notices. |
| + // Built in the store from the passages §4.1 kept, so a page can only |
| + // ever state a fact drawn from something this reader may read. |
| + // |
| + // **Empty is the ordinary state**, and it renders nothing at all — |
| + // never "no phone number found". The corpus is not the world, and a |
| + // confident absence is this codebase's worst failure shape. |
| + Facts []Fact |
| + |
| // SearchFailed is not "no results". A search that errored renders |
| // its own callout in the results region, because "nothing matched" |
| // and "the search broke" look identical to a reader and this |
| @@ -637,6 +646,10 @@ func (a *app) runSearch(r *http.Request, p *page) { |
| } |
| p.Results = found.Results |
| p.Notices = found.Notices |
| + // Copied here, in the method BOTH routes share, for the same reason |
| + // Primed is decided here: the page and the fragment must not be able |
| + // to disagree about whether there is a box. |
| + p.Facts = found.Facts |
| p.Primed = query.PrimesFirstResult(p.Query, len(found.Results)) |
| } |
diff --git a/internal/memoria/static/memoria.css b/internal/memoria/static/memoria.css| index 4bfb536..9423be6 100644 |
| --- a/internal/memoria/static/memoria.css |
| +++ b/internal/memoria/static/memoria.css |
| @@ -292,3 +292,39 @@ |
| [rst-search] { flex-direction: column; } |
| [rst-table] { font-size: 0.875rem; } |
| } |
| + |
| +/* §8.8's fact box: a validated fact lifted out of the passages below, |
| + with its citation. Sits between the withholding notice and the result |
| + list, and is styled to read as an answer rather than as a result — |
| + the value is the thing the eye should land on. */ |
| +[rst-fact-box] { |
| + border: 1px solid var(--rst-line); |
| + border-radius: var(--rst-radius); |
| + padding: 0.7rem 0.9rem; |
| + margin: 0 0 1rem; |
| + background: var(--rst-surface); |
| +} |
| +[rst-fact] { |
| + display: flex; |
| + flex-wrap: wrap; |
| + align-items: baseline; |
| + gap: 0.2rem 0.6rem; |
| + margin: 0; |
| +} |
| +[rst-fact] + [rst-fact] { |
| + margin-top: 0.5rem; |
| + padding-top: 0.5rem; |
| + border-top: 1px solid var(--rst-line); |
| +} |
| +[rst-fact-label] { |
| + color: var(--rst-text-muted); |
| + font-size: 0.8125rem; |
| +} |
| +[rst-fact-value] { |
| + font-weight: 600; |
| + font-size: 1.0625rem; |
| +} |
| +[rst-fact-cite] { |
| + font-size: 0.8125rem; |
| + margin-left: auto; |
| +} |
diff --git a/internal/memoria/store.go b/internal/memoria/store.go| index 56350fb..5f8fa23 100644 |
| --- a/internal/memoria/store.go |
| +++ b/internal/memoria/store.go |
| @@ -12,6 +12,7 @@ import ( |
| "amadan.net/oficina/suite" |
| "amadan.net/oficina/memoria/internal/memoria/chunk" |
| + "amadan.net/oficina/memoria/internal/memoria/fact" |
| "amadan.net/oficina/memoria/internal/memoria/index" |
| "amadan.net/oficina/memoria/internal/memoria/seal" |
| "amadan.net/oficina/memoria/internal/memoria/sibling" |
| @@ -98,6 +99,12 @@ type Result struct { |
| type Page struct { |
| Results []Result |
| Notices []string |
| + // Facts is §8.8's box: validated facts pulled deterministically out |
| + // of the passages above, with no model anywhere near them. |
| + // |
| + // **Derived from Results and from nothing else**, which is what makes |
| + // it safe by construction rather than by care. See addFacts. |
| + Facts []Fact |
| } |
| // Summary is a document as the list shows it. |
| @@ -678,7 +685,12 @@ func (s *Store) Search(ctx context.Context, asker suite.Subject, query string, l |
| item: items[h.DocumentID], |
| }) |
| } |
| - return s.rule(ctx, asker, out), nil |
| + page := s.rule(ctx, asker, out) |
| + // **After the ruling, never before it.** §8.8's box is built from the |
| + // passages rule() kept, so a withheld one cannot contribute a |
| + // candidate and cannot be counted towards the cap. |
| + s.addFacts(ctx, &page, query) |
| + return page, nil |
| } |
| // rule asks every app of record about its own passages and keeps the ones |
| @@ -935,3 +947,172 @@ func firstLine(body string) string { |
| } |
| return "Untitled" |
| } |
| + |
| +// MaxFacts is §8.8's cap: three candidates render, four render nothing. |
| +// |
| +// Inclusive, and the boundary is the whole of it. A cap written `>= 3` |
| +// passes any suite that tests one, two and four, which is why gate 35 |
| +// tests three as well. |
| +const MaxFacts = 3 |
| + |
| +// Fact is one row of §8.8's box. |
| +// |
| +// It carries Title and Trail as well as Label for the reason Result does: |
| +// two pages called "Contact" under different documents are the same row |
| +// to a reader who cannot see which is which. It carries no ChunkID and no |
| +// chunk text — a field a template can reach is a field a template |
| +// eventually prints, which is why Result.item is unexported. |
| +type Fact struct { |
| + Kind string |
| + Title string |
| + Trail string |
| + // Label is what the DOCUMENT called this value, never composed. |
| + Label string |
| + // Span is the document's own bytes. The canonical form that keyed |
| + // this candidate's identity does not leave the store. |
| + Span string |
| + DocumentID int64 |
| + Ordinal int |
| +} |
| + |
| +// addFacts fills p.Facts for the kinds this query names. |
| +// |
| +// **It runs over p.Results — the RULED slice — and never over the hits |
| +// that produced it.** That is §8.8's ordering ruling and the whole of why |
| +// the box cannot leak: a candidate that is not in p.Results does not |
| +// exist, so §8.4's invariant ("every passage in a response passed §4.1 in |
| +// the request that produced it") is structural here rather than |
| +// something a later caller has to remember. |
| +// |
| +// The counts below — dedupe, collision, cap — therefore also happen after |
| +// the ruling. Counting before it is the leak rather than a tidiness |
| +// question: a box that vanishes for passing the cap because of documents |
| +// the reader may not see reports the existence of those documents, and so |
| +// does an ambiguity of two where the reader may read one. |
| +func (s *Store) addFacts(ctx context.Context, p *Page, query string) { |
| + want := fact.Wanted(query) |
| + if len(want) == 0 || len(p.Results) == 0 { |
| + // **No database work for a query that names no kind**, which is |
| + // almost every query. The box must not cost anything on the |
| + // searches it will never appear on. |
| + return |
| + } |
| + |
| + ids := make([]int64, 0, len(p.Results)) |
| + for _, r := range p.Results { |
| + ids = append(ids, r.ChunkID) |
| + } |
| + var rows []Chunk |
| + if err := s.db.G.WithContext(ctx).Where("id IN ?", ids).Find(&rows).Error; err != nil { |
| + // The search already succeeded and the page is already honest |
| + // without a box. Failing it now would trade a working answer for |
| + // a garnish, which is the wrong direction — and there is no |
| + // query text and no content in this line, per gate 4. |
| + if s.logger != nil { |
| + s.logger.Error("fact box: read passages", "err", err) |
| + } |
| + return |
| + } |
| + texts := make(map[int64]string, len(rows)) |
| + for _, r := range rows { |
| + t, err := s.sealer.Open(seal.PurposeChunk, r.SealedText) |
| + if err != nil { |
| + if s.logger != nil { |
| + s.logger.Error("fact box: open passage", "err", err) |
| + } |
| + continue |
| + } |
| + texts[r.ID] = string(t) |
| + } |
| + |
| + // One bucket per kind, because every rule below is per kind. A cap |
| + // counted across kinds lets two phone numbers and two addresses |
| + // suppress each other, and a collision check across kinds suppresses |
| + // a phone and an email that happen to share a label. |
| + found := map[fact.Kind][]Fact{} |
| + seen := map[fact.Kind]map[string]bool{} |
| + silenced := map[fact.Kind]bool{} |
| + |
| + for _, r := range p.Results { |
| + text, ok := texts[r.ChunkID] |
| + if !ok { |
| + // The chunk row is gone between the ruling and this read. |
| + // Skipped silently, exactly as Search's livePassage check |
| + // drops a stale hit and for the reason recorded there: the |
| + // passage is genuinely gone, so its absence is the true |
| + // answer rather than a truncation of one. |
| + continue |
| + } |
| + for _, f := range fact.Scan(text, r.Heading, r.Title, want) { |
| + if seen[f.Kind] == nil { |
| + seen[f.Kind] = map[string]bool{} |
| + } |
| + // **Identity is checked BEFORE rule 4, so only the occurrence |
| + // that is actually cited decides whether its kind renders.** |
| + // The other order made one document give two different |
| + // answers depending on where chunk.Split happened to cut: the |
| + // same value written twice, once with a short label and once |
| + // with an over-long one, silenced the kind when the two |
| + // landed in different passages and did not when they landed |
| + // in the same one. A boundary the reader cannot see is not |
| + // allowed to decide what the page says. |
| + if seen[f.Kind][f.Canonical] { |
| + continue |
| + } |
| + if f.NoBox { |
| + silenced[f.Kind] = true |
| + continue |
| + } |
| + // **Identity is the canonical value**, so one number written |
| + // two ways is one candidate and the same number in five |
| + // forwarded signatures is one too. First occurrence wins the |
| + // citation, and because p.Results is in rank order that is |
| + // the best-ranked passage carrying it. |
| + seen[f.Kind][f.Canonical] = true |
| + found[f.Kind] = append(found[f.Kind], Fact{ |
| + Kind: string(f.Kind), |
| + Title: r.Title, |
| + Trail: r.Heading, |
| + Label: f.Label, |
| + Span: f.Span, |
| + DocumentID: r.DocumentID, |
| + Ordinal: r.Ordinal, |
| + }) |
| + } |
| + } |
| + |
| + // A slice ranged in a fixed order, never a map: Go randomises map |
| + // iteration precisely so that an order nobody chose shows up, and it |
| + // showed up once already in this file's notice aggregation. |
| + for _, k := range fact.Kinds() { |
| + cs := found[k] |
| + switch { |
| + case len(cs) == 0, silenced[k], len(cs) > MaxFacts, collides(cs): |
| + continue |
| + } |
| + p.Facts = append(p.Facts, cs...) |
| + } |
| +} |
| + |
| +// collides reports whether two candidates would render an identical |
| +// citation line. |
| +// |
| +// **Keyed on the whole line and not on the label alone.** A supplier page |
| +// and the office page both falling through to a trail of "Contact" are |
| +// perfectly disambiguated by their titles, and suppressing that box would |
| +// lose a real answer to a rule aimed at a different problem. What the |
| +// rule is actually for is the case where the page would print one |
| +// identical row twice against two different values — not an enumeration |
| +// but a contradiction, with nothing for the reader to resolve it by. |
| +func collides(cs []Fact) bool { |
| + type line struct{ title, trail, label string } |
| + seen := map[line]bool{} |
| + for _, c := range cs { |
| + l := line{c.Title, c.Trail, c.Label} |
| + if seen[l] { |
| + return true |
| + } |
| + seen[l] = true |
| + } |
| + return false |
| +} |
diff --git a/internal/memoria/templates/results.html b/internal/memoria/templates/results.html| index a8ce802..c5a02f9 100644 |
| --- a/internal/memoria/templates/results.html |
| +++ b/internal/memoria/templates/results.html |
| @@ -31,6 +31,31 @@ |
| {{range .Notices}}<p rst-callout-body>{{.}}</p>{{end}} |
| </div> |
| {{end}} |
| + {{/* §8.8's fact box. BELOW the notices and above the list, both |
| + deliberately: a confident single answer rendered above the line |
| + saying the page is partial is the withholding notice's own |
| + argument with the order reversed. |
| + |
| + Inside this define rather than above it, so search.js swaps box |
| + and list together. A box outside it would leave query N-1's answer |
| + sitting over query N's results on every keystroke. |
| + |
| + Every character here is the document's — the label it wrote, the |
| + span byte for byte, its own title and heading trail. The link is |
| + the ONLY link, and it must never wear rst-result-title: search.js |
| + reads that attribute to decide where Enter goes, and §8.8 rules |
| + that Enter still goes to the first result and not to a box. */}} |
| + {{if .Facts}} |
| + <div rst-fact-box> |
| + {{range .Facts}} |
| + <p rst-fact> |
| + <span rst-fact-label>{{.Label}}</span> |
| + <span rst-fact-value>{{.Span}}</span> |
| + <a href="/documents/{{.DocumentID}}#p{{.Ordinal}}" rst-fact-cite>{{.Title}}{{if .Trail}} › {{.Trail}}{{end}}</a> |
| + </p> |
| + {{end}} |
| + </div> |
| + {{end}} |
| {{if not .Results}} |
| <div rst-callout> |
| <p rst-callout-body> |
diff --git a/internal/memoriatest/factbox_test.go b/internal/memoriatest/factbox_test.go| new file mode 100644 |
| index 0000000..73f16e1 |
| --- /dev/null |
| +++ b/internal/memoriatest/factbox_test.go |
| @@ -0,0 +1,391 @@ |
| +package memoriatest |
| + |
| +import ( |
| + "html" |
| + "net/http" |
| + "net/url" |
| + "regexp" |
| + "strings" |
| + "testing" |
| +) |
| + |
| +// The fact box, end to end: through the real handler, the real template |
| +// and the real escaping. |
| +// |
| +// **Through HTTP and not against the struct, deliberately, and gate 34 is |
| +// the opposite deliberately.** Gate 34 is about what the box may CONTAIN, |
| +// where two dispositions render identical pages and only the struct can |
| +// tell them apart. These are about what reaches the reader, where a |
| +// template that escapes a span, drops a label or renders the box outside |
| +// the swapped region is exactly the defect — and a struct assertion |
| +// cannot see any of it. AGENTS.md: a template test proves markup exists, |
| +// not that it renders. |
| + |
| +var factValueRe = regexp.MustCompile(`<span rst-fact-value>([^<]*)</span>`) |
| +var factLabelRe = regexp.MustCompile(`<span rst-fact-label>([^<]*)</span>`) |
| + |
| +func factValues(body string) []string { return capture(factValueRe, body) } |
| +func factLabels(body string) []string { return capture(factLabelRe, body) } |
| +func hasFactBox(body string) bool { return strings.Contains(body, "<div rst-fact-box>") } |
| + |
| +// capture pulls the rendered text out of the box and UNESCAPES it. |
| +// |
| +// html/template escapes "+" to "+" in text content, which is correct |
| +// and which the reader never sees. Gate 36 is about the bytes that reach |
| +// the READER — "byte-identical to the document" is a claim about what is |
| +// displayed, not about the wire — so comparing raw markup would fail the |
| +// gate on the escaping working properly, and "fixing" that would mean |
| +// unescaping the span into the page, which is an injection. |
| +func capture(re *regexp.Regexp, s string) []string { |
| + var out []string |
| + for _, m := range re.FindAllStringSubmatch(s, -1) { |
| + out = append(out, html.UnescapeString(m[1])) |
| + } |
| + return out |
| +} |
| + |
| +func search(t *testing.T, h http.Handler, q string) string { |
| + t.Helper() |
| + return get(t, h, "/?q="+url.QueryEscape(q)).Body.String() |
| +} |
| + |
| +// GATE 35 — candidates are canonical values, ambiguity is enumerated |
| +// rather than collapsed, and the cap is three (§8.8, §10). |
| +// |
| +// One test rather than five, because the rules share a definition and a |
| +// gate that splits them cannot see the interactions: the same number in |
| +// five documents is one candidate AND therefore does not pass the cap. |
| +// |
| +// Mutations run, each verified red and reverted, 2026-09-14: |
| +// - returning found[k][:1] when there is more than one — "two distinct" |
| +// goes red. |
| +// - keying dedupe on Span instead of Canonical — "two spellings" red. |
| +// - keying dedupe on the passage rather than the value — "five |
| +// documents" red. |
| +// - cap written len(cs) >= MaxFacts — "three still renders" red, and |
| +// nothing else, which is the point of having it. |
| +// - cap and collision counted across kinds rather than per kind — the |
| +// two-kind case red. |
| +func TestGate35CandidatesAmbiguityAndTheCap(t *testing.T) { |
| + t.Run("presence: one number, one box, one value", func(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Tito office", "## Contact\n\nOffice phone: +353 (0)1 547 7854") |
| + body := search(t, h, "phone number") |
| + if got := factValues(body); len(got) != 1 || got[0] != "+353 (0)1 547 7854" { |
| + t.Fatalf("values = %q, want the one number", got) |
| + } |
| + }) |
| + |
| + // Two right answers to an underspecified question. §8.7 refuses in |
| + // this position; §8.8 enumerates, because two phone numbers are not |
| + // a wrong average — and the labels are what let the reader choose. |
| + t.Run("two distinct numbers are both named, neither picked", func(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Tito office", "## Contact\n\nOffice phone: +353 (0)1 547 7854") |
| + store(t, h, "Support", "## Contact\n\nSupport phone: +353 (0)1 598 1234") |
| + got := factValues(search(t, h, "phone number")) |
| + if len(got) != 2 { |
| + t.Fatalf("values = %q, want both", got) |
| + } |
| + }) |
| + |
| + // A dedupe keyed on the span passes any fixture that spells the |
| + // number the same way twice, which is why this one does not. |
| + t.Run("two spellings of one number are one candidate", func(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Tito office", "## Contact\n\nOffice phone: +353 (0)1 547 7854") |
| + store(t, h, "Handbook", "## Contact\n\nOffice phone: 01 547 7854") |
| + if got := factValues(search(t, h, "phone number")); len(got) != 1 { |
| + t.Fatalf("values = %q, want one: the same number written two ways", got) |
| + } |
| + }) |
| + |
| + // The commonest real shape — a signature repeated through forwarded |
| + // mail. An implementation counting passages rather than values shows |
| + // five rows here, or passes the cap and shows none. |
| + t.Run("the same number in five documents is one candidate", func(t *testing.T) { |
| + h := newApp(t) |
| + for _, title := range []string{"One", "Two", "Three", "Four", "Five"} { |
| + store(t, h, title, "## Contact\n\nOffice phone: +353 (0)1 547 7854") |
| + } |
| + if got := factValues(search(t, h, "phone number")); len(got) != 1 { |
| + t.Fatalf("values = %q, want one", got) |
| + } |
| + }) |
| + |
| + // index/rank_test.go's own paragraph, which came off the live |
| + // instance. An implementation treating the CHUNK as the unit hides |
| + // the second number — the one the document calls unused. |
| + t.Run("two numbers in one passage are two candidates", func(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Office address", "## Office Address\n\nTurn the knob. "+ |
| + "The office phone number is: +353 (0)1 547 7854. "+ |
| + "We also have the following unused phone numbers: +353 (0)1 598 1234.") |
| + body := search(t, h, "phone number") |
| + if got := factValues(body); len(got) != 2 { |
| + t.Fatalf("values = %q, want both", got) |
| + } |
| + // And the labels are what make the enumeration honest: the |
| + // document said which is unused, so the page does. |
| + labels := factLabels(body) |
| + if len(labels) != 2 || !strings.Contains(labels[1], "unused") { |
| + t.Fatalf("labels = %q; the document's own word for the second number is missing", labels) |
| + } |
| + }) |
| + |
| + // **Three AND four.** A cap written `>= MaxFacts` passes a suite that |
| + // tests only one, two and four. |
| + t.Run("three renders, four renders nothing", func(t *testing.T) { |
| + nums := []string{"+353 (0)1 547 7854", "+353 (0)1 598 1234", "+353 (0)1 555 0101", "+353 (0)1 555 0202"} |
| + for _, n := range []int{3, 4} { |
| + h := newApp(t) |
| + for i := 0; i < n; i++ { |
| + store(t, h, "Office "+string(rune('A'+i)), "## Contact\n\nOffice phone: "+nums[i]) |
| + } |
| + body := search(t, h, "phone number") |
| + switch { |
| + case n == 3 && len(factValues(body)) != 3: |
| + t.Errorf("three candidates rendered %q, want three", factValues(body)) |
| + case n == 4 && hasFactBox(body): |
| + t.Errorf("four candidates rendered a box: %q", factValues(body)) |
| + } |
| + } |
| + }) |
| + |
| + // Every rule is per kind. Counted across kinds, two of each passes |
| + // the cap and both boxes vanish. |
| + t.Run("a two-kind query renders both kinds", func(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Tito office", "## Contact\n\nOffice phone: +353 (0)1 547 7854\n\nEircode: D02 X285") |
| + store(t, h, "Support", "## Contact\n\nSupport phone: +353 (0)1 598 1234\n\nEircode: A65 F4E2") |
| + got := factValues(search(t, h, "phone number and eircode")) |
| + if len(got) != 4 { |
| + t.Fatalf("values = %q, want two of each kind", got) |
| + } |
| + }) |
| + |
| + // The whole citation line collides, so the page would print one |
| + // identical row twice against two different values. |
| + t.Run("colliding rows render no box", func(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Suppliers", "## Phone\n\n- +353 (0)1 555 0101\n- +353 (0)1 555 0202") |
| + if body := search(t, h, "phone number"); hasFactBox(body) { |
| + t.Errorf("rendered a box with two identical rows: %q / %q", factLabels(body), factValues(body)) |
| + } |
| + }) |
| + |
| + // §10 says this gate asserts first-seen order, and for a while it |
| + // asserted only the count. A box sorted by canonical value is |
| + // deterministic and passes every length check, and the two-values-in- |
| + // one-passage fixture happens to sort the same way it was seen — so |
| + // nothing here could see it. Found by review, 2026-09-14. |
| + t.Run("rows are in first-seen order", func(t *testing.T) { |
| + h := newApp(t) |
| + // Both orderings a wrong implementation would pick disagree with |
| + // document order here: sorted by VALUE the 547 number comes |
| + // first, and sorted by LABEL "Alternate" sorts before "Work". |
| + // Only first-seen gives 598, so this one fixture holds against |
| + // both — and the version it replaces held against neither, |
| + // because its two labels happened to sort as they were seen. |
| + store(t, h, "Tito office", "## Contact\n\n"+ |
| + "Work phone is: +353 (0)1 598 1234. "+ |
| + "Alternate phone is: +353 (0)1 547 7854.") |
| + got := factValues(search(t, h, "phone number")) |
| + if len(got) != 2 { |
| + t.Fatalf("values = %q, want two", got) |
| + } |
| + if got[0] != "+353 (0)1 598 1234" { |
| + t.Errorf("first row = %q; the box is not in the order the document states them", got[0]) |
| + } |
| + }) |
| + |
| + // Every collision fixture above is a PAIR, so a check comparing only |
| + // neighbours passes all of them. Three rows whose first and third are |
| + // identical is the shape that sees it. |
| + t.Run("a collision between non-adjacent rows still suppresses", func(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Tito office", "## Contact\n\n"+ |
| + "Ring the office phone on 01 547 7854 any weekday. "+ |
| + "Support phone: 01 598 1234. "+ |
| + "Or dial the phone on 01 555 0101 instead.") |
| + if body := search(t, h, "phone number"); hasFactBox(body) { |
| + t.Errorf("rendered a box whose first and third rows are identical: %q / %q", |
| + factLabels(body), factValues(body)) |
| + } |
| + }) |
| + |
| + // The mirror of the title case below, for the trail. Same title, same |
| + // label, different headings — the trail is what tells them apart, and |
| + // a collision check ignoring it loses a real answer. |
| + t.Run("the same label under different headings still renders", func(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Tito office", "## Office\n\nPhone: +353 (0)1 547 7854\n\n## Support\n\nPhone: +353 (0)1 598 1234") |
| + if got := factValues(search(t, h, "phone number")); len(got) != 2 { |
| + t.Fatalf("values = %q, want both: the headings tell them apart", got) |
| + } |
| + }) |
| + |
| + // A different title disambiguates perfectly, so this one MUST box. |
| + // Keying collision on the label alone loses a real answer here. |
| + t.Run("the same label under different titles still renders", func(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Tito office", "## Contact\n\nOffice phone: +353 (0)1 547 7854") |
| + store(t, h, "Acme supplier", "## Contact\n\nOffice phone: +353 (0)1 598 1234") |
| + if got := factValues(search(t, h, "phone number")); len(got) != 2 { |
| + t.Fatalf("values = %q, want both: the titles tell them apart", got) |
| + } |
| + }) |
| +} |
| + |
| +// GATE 36 — the displayed value is byte-identical to the document's |
| +// (§8.8, §10). The canonical form exists for matching and must never |
| +// reach the page. |
| +// |
| +// Mutation: canonicalise on the display path — one line's difference from |
| +// what the code already does for identity — and every case goes red. |
| +func TestGate36TheDisplayedValueIsTheDocumentsBytes(t *testing.T) { |
| + for _, c := range []struct{ name, body, want string }{ |
| + {"as written", "## Contact\n\nOffice phone: +353 (0)1 547 7854", "+353 (0)1 547 7854"}, |
| + {"with an extension", "## Contact\n\nOffice phone: +353 (0)1 547 7854 ext. 12", "+353 (0)1 547 7854 ext. 12"}, |
| + // Pasted from Word or Notion. Go's \s does not match U+00A0, so |
| + // a recogniser written with it finds nothing here and this case |
| + // fails on PRESENCE rather than on the bytes — which is the |
| + // design demand and not an accident. |
| + {"pasted with non-breaking spaces", "## Contact\n\nOffice phone: +353 (0)1 547 7854", "+353 (0)1 547 7854"}, |
| + } { |
| + t.Run(c.name, func(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Tito office", c.body) |
| + got := factValues(search(t, h, "phone number")) |
| + if len(got) != 1 { |
| + t.Fatalf("values = %q, want one", got) |
| + } |
| + if got[0] != c.want { |
| + t.Errorf("value = %q, want %q", got[0], c.want) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// GATE 37 — the label is the document's, and it reaches the page. |
| +// |
| +// The exhaustive matrix — bullets, adjacent values, abbreviations, the |
| +// cap boundary, the no-heading fallback — is unit-tested in |
| +// internal/memoria/fact, where each rule can be driven directly. What |
| +// this gate adds is the half a unit test cannot see: that the label |
| +// survives the store, the template and the escaping, and that a value the |
| +// document labelled too long to show renders NO BOX rather than a vaguer |
| +// substitute. |
| +func TestGate37TheLabelReachesThePage(t *testing.T) { |
| + t.Run("the document's own label is what is shown", func(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Tito office", "## Contact\n\nOffice phone: +353 (0)1 547 7854") |
| + if got := factLabels(search(t, h, "phone number")); len(got) != 1 || got[0] != "Office phone" { |
| + t.Fatalf("labels = %q, want [Office phone]", got) |
| + } |
| + }) |
| + |
| + t.Run("no label in the line falls to the heading trail", func(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Tito office", "## Contact\n\nRing the office phone on +353 (0)1 547 7854 any weekday.") |
| + if got := factLabels(search(t, h, "phone number")); len(got) != 1 || got[0] != "Contact" { |
| + t.Fatalf("labels = %q, want [Contact]", got) |
| + } |
| + }) |
| + |
| + // Substituting the trail would replace the document's own warning |
| + // with a vaguer word, beside a number the document did not warn |
| + // about. Dropping just this candidate would turn a two-candidate |
| + // ambiguity into a confident single answer. Both are worse than |
| + // silence. |
| + t.Run("a label too long to show renders no box at all", func(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Tito office", "## Contact\n\nReception phone: +353 (0)1 547 7854\n\n"+ |
| + "The number you must never give to any customer because it rings the chief executive directly is: +353 (0)1 598 1234") |
| + if body := search(t, h, "phone number"); hasFactBox(body) { |
| + t.Errorf("rendered a box: %q / %q", factLabels(body), factValues(body)) |
| + } |
| + }) |
| +} |
| + |
| +// The box lives INSIDE the results define, so the fragment search.js |
| +// swaps carries it. Outside, query N-1's answer would sit over query N's |
| +// results on every keystroke — §8.4 inverted, several times a second. |
| +// |
| +// This is the page-and-fragment agreement that TestALookupPrimesTheFirstResult |
| +// makes for priming, and it is needed for the same reason: two renderings |
| +// of one region that can disagree are two regions. |
| +func TestTheFactBoxIsInsideTheSwappedRegion(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Tito office", "## Contact\n\nOffice phone: +353 (0)1 547 7854") |
| + page := search(t, h, "phone number") |
| + fragment := get(t, h, "/search?q="+url.QueryEscape("phone number")).Body.String() |
| + if !hasFactBox(page) { |
| + t.Fatal("no box on the page") |
| + } |
| + if !hasFactBox(fragment) { |
| + t.Error("no box in the fragment: it would not survive a keystroke") |
| + } |
| + if a, b := factValues(page), factValues(fragment); len(a) != len(b) || a[0] != b[0] { |
| + t.Errorf("page %q and fragment %q disagree", a, b) |
| + } |
| +} |
| + |
| +// Enter goes to the first RESULT, never to the box's citation. §8.8: |
| +// changing what Enter means depending on whether a regular expression |
| +// matched is the unreliable-Enter failure PrimesFirstResult exists to |
| +// avoid. search.js targets rst-result-title, so the guard is that the |
| +// box's link never wears it. |
| +func TestTheFactBoxCitationIsNotAnEnterTarget(t *testing.T) { |
| + h := newApp(t) |
| + store(t, h, "Tito office", "## Contact\n\nOffice phone: +353 (0)1 547 7854") |
| + body := search(t, h, "phone number") |
| + if !hasFactBox(body) { |
| + t.Fatal("no box to check: this assertion needs one to be wrong about") |
| + } |
| + box := body[strings.Index(body, "<div rst-fact-box>"):] |
| + box = box[:strings.Index(box, "</div>")] |
| + if strings.Contains(box, "rst-result-title") { |
| + t.Errorf("the box carries search.js's Enter target:\n%s", box) |
| + } |
| +} |
| + |
| +// **The reachability limit, pinned rather than discovered.** |
| +// |
| +// §8.8 states it: the box only sees what retrieval returned, so a fact in |
| +// a document the query does not reach is a fact the box does not have. |
| +// This is not a small edge — it is the ordinary case for a contact page |
| +// that writes "Office: +353 …" and never uses the word "phone" — and |
| +// writing the gates is what made it concrete, because every fixture above |
| +// had to be given a word the query could find. |
| +// |
| +// Asserted in BOTH directions, because a limit tested only in the |
| +// direction that fails is indistinguishable from the feature not working |
| +// at all: |
| +// |
| +// - a document sharing no word with the query renders no box; |
| +// - the SAME document, reached by a query that shares one, does. |
| +// |
| +// §12 carries the fix and names what it must not be. Until then this test |
| +// is the honest statement of what the feature does, and it is here so |
| +// that the day somebody builds the lift, this goes red and gets deleted |
| +// on purpose. |
| +func TestTheBoxOnlySeesWhatRetrievalReturned(t *testing.T) { |
| + const body = "## Contact\n\nOffice: +353 (0)1 547 7854" |
| + |
| + h := newApp(t) |
| + store(t, h, "Tito office", body) |
| + |
| + // "phone number" appears nowhere in that document, so nothing ranks |
| + // and there is nothing to extract from. |
| + if got := search(t, h, "phone number"); hasFactBox(got) { |
| + t.Errorf("a box appeared for a query the document does not match: %q", factValues(got)) |
| + } |
| + |
| + // One shared word — "office", in the title and in the line — and the |
| + // same fact is found. This is the half that proves the absence above |
| + // is the limit rather than a broken recogniser. |
| + if got := factValues(search(t, h, "office phone number")); len(got) != 1 { |
| + t.Fatalf("values = %q, want the number: the query shares a word with the document", got) |
| + } |
| +} |
diff --git a/tools/drive/main.go b/tools/drive/main.go| index 0bb1d58..4e999f5 100644 |
| --- a/tools/drive/main.go |
| +++ b/tools/drive/main.go |
| @@ -99,6 +99,27 @@ func seedViaHTTP(h http.Handler, addr string) { |
| docs := []struct{ title, body string }{ |
| {"Q3 board pack", "## Revenue\n\nJune revenue was 1,350,000 across all regions, up from 1,180,000 in the same month last year. The increase is almost entirely the enterprise tier.\n\n## Costs\n\nHosting costs fell after the migration, and support headcount was flat.\n\n## Risks\n\nOne customer is 22% of recurring revenue. That concentration is the single largest risk on this page and it has not improved since Q2."}, |
| {"Sealing notes", "## The one rule\n\nA sheet's contents are sealed at rest and the bucket never holds a plaintext cell. The instance holds the key while it runs, so a compromised live host can read a sheet it has open — that is a deliberate softening and the README says so.\n\n## What it does not cover\n\nEmbeddings are content: a vector inverts back to text well enough that an unsealed index is a plaintext cell in a costume."}, |
| + // §8.8's fact box, seeded in THREE shapes rather than one. |
| + // |
| + // AGENTS.md: the section that is not seeded is the section |
| + // nobody sees, and driving this page is what found the heading |
| + // trail rendering as "› Revenue". A box only ever seeded in its |
| + // happy case is a box whose two interesting states nobody looks |
| + // at — and those are the states that decide whether the rule is |
| + // right. |
| + {"Tito office", "## Contact\n\nOffice: +353 (0)1 547 7854\n\nEircode: D02 X285\n\nVAT: IE3384527RH\n\n## Post\n\nWe are on the second floor; the intercom is marked Tito."}, |
| + // Two numbers, distinct sentence labels: the ENUMERATED case. |
| + // The reader should see both and be able to tell which is which, |
| + // because the document said "unused" and the label carries it. |
| + // This paragraph is index/rank_test.go's, quoted rather than |
| + // invented — it came off the live instance. |
| + {"Reception notes", "## Office Address\n\nTurn the knob. The office phone number is: +353 (0)1 547 7854. We also have the following unused phone numbers: +353 (0)1 598 1234."}, |
| + // A bulleted pair: the COLLISION case, where both labels fall to |
| + // the same trail and the box must be ABSENT. An earlier draft of |
| + // the plan called the paragraph above the collision case; it is |
| + // the opposite, and a seed that demonstrates the wrong thing is |
| + // worse than no seed at all. |
| + {"Suppliers", "## Contact\n\n- +353 (0)1 555 0101\n- +353 (0)1 555 0202"}, |
| {"Meeting, 12 August", "## Present\n\nEveryone.\n\n## Decisions\n\nWe will not build a suite index app. It answers a different question from the one anyone asked, and per-item access control is not expressible by the app that would have to publish it.\n\n## Actions\n\nPaul to decide whether the central host is a directory or an authenticator."}, |
| } |
| for _, doc := range docs { |