| 1 | # Implementation plan — §8.8's fact box |
| 2 | |
| 3 | Against `docs/superpowers/specs/2026-09-03-memoria-design.md` §8.8 draft |
| 4 | 3, decision 15, gates 34–37 (38 struck). Branch `feat/lookup-box`. |
| 5 | |
| 6 | **Draft 3 is what this plan is written against, and §0 of the previous |
| 7 | version of this file is why it exists.** Writing the retrieval step out |
| 8 | concretely is what found that a corpus-wide facts table cannot coexist |
| 9 | with a cap of three. That finding is now in the spec; what remains here is |
| 10 | operational. |
| 11 | |
| 12 | --- |
| 13 | |
| 14 | ## 1. `internal/memoria/fact` — new package |
| 15 | |
| 16 | Recognisers, validators, the kind lexicon and the label rule. Imports |
| 17 | `regexp`, `strings`, `unicode` and nothing else: pure, table-driven, and |
| 18 | unit-testable without a database, a browser or an app — the same reason |
| 19 | `query` and `chunk` are their own packages. |
| 20 | |
| 21 | **The lexicon lives here, beside the validators, not in `query`.** `query` |
| 22 | is about one decision and its doc comment says so. More to the point, the |
| 23 | words that ask for a kind and the validator that admits one are two |
| 24 | columns of one table and must not drift apart. |
| 25 | |
| 26 | ```go |
| 27 | type Kind string // "phone", "eircode", "iban", "vat", "email" |
| 28 | |
| 29 | // Wanted returns the kinds a query names. Empty means no box. |
| 30 | func Wanted(q string) []Kind |
| 31 | |
| 32 | // Found is one validated fact, located. |
| 33 | type Found struct { |
| 34 | Kind Kind |
| 35 | Span string // the document's bytes. The only thing rendered. |
| 36 | Canonical string // match-only; never rendered, never an operand. |
| 37 | Label string // the document's, per §8.8's five rules. |
| 38 | NoBox bool // an over-cap source-1 label: this kind renders nothing. |
| 39 | } |
| 40 | |
| 41 | // Scan returns every validated fact of the wanted kinds in text. |
| 42 | // heading and title are the label fallbacks, in that order. |
| 43 | func Scan(text, heading, title string, want []Kind) []Found |
| 44 | ``` |
| 45 | |
| 46 | Build order, each step's test before the next: |
| 47 | |
| 48 | 1. **`Wanted`** — a fixed map, longest phrase first so `email address` |
| 49 | beats `address` (which maps to nothing). Tests: the five kinds; |
| 50 | `address`, `number` and `mobile app` priming nothing, nothing and |
| 51 | `phone` respectively. |
| 52 | 2. **Recognise and validate, per kind.** Each kind is |
| 53 | `{re, canon, valid}`. **The regex is deliberately loose and the |
| 54 | validator does the work** — that is §8.8's "validation is not |
| 55 | pattern-spotting" made structural, and it is what stops the next |
| 56 | person tightening a regex into a validator. |
| 57 | |
| 58 | Verified against known-good values before this plan was written, |
| 59 | because a validator that admits everything is the guard-written-to-pass |
| 60 | failure in its purest form and a checksum is the one thing here |
| 61 | checkable against the world rather than against a fixture somebody |
| 62 | wrote: |
| 63 | |
| 64 | | Input | Expect | Got | |
| 65 | |---|---|---| |
| 66 | | `IE6388047V` (Google Ireland) | valid | valid | |
| 67 | | `IE3628739L` | valid | valid | |
| 68 | | `IE1234567X` (invented) | **invalid** | invalid | |
| 69 | | `IE29AIBK93115212345678` | valid | valid | |
| 70 | | `GB82WEST12345698765432` | valid | valid | |
| 71 | | `IE29AIBK9311521234567`**`9`** | **invalid** | invalid | |
| 72 | | `A65 F4E2` (published example) | valid | valid | |
| 73 | | `D6W 1234` (Dublin 6W's irregular key) | valid | valid | |
| 74 | | `B12 3456` (`B` is not a routing-key letter) | **invalid** | invalid | |
| 75 | |
| 76 | The reject rows are load-bearing: the invented VAT and the one-digit |
| 77 | IBAN edit are what tell a checksum from a regex, and `D6W` is the case |
| 78 | a pattern written `[A-Z][0-9][0-9]` gets wrong. |
| 79 | |
| 80 | **Old-style Irish VAT** — a letter or `+` in second position — is |
| 81 | refused deliberately. A refused VAT is a box that does not appear; an |
| 82 | admitted wrong one is a fact on the page. |
| 83 | |
| 84 | **`phone` gets §8.8's structural rule:** a bare run of digits is never |
| 85 | a phone number; the span must carry `+` or grouping separators. |
| 86 | `rank_test.go`'s `DUNS/D-U-N-S number is 985632364` is the fixture |
| 87 | that proves it — nine digits, a plausible Irish mobile without the |
| 88 | rule. **This is a validator test in `fact`, and a rejection fixture in |
| 89 | gate 35** — not gate 37, which is the label gate. |
| 90 | |
| 91 | **Separators include U+00A0**, since Go's `\s` does not match it and |
| 92 | Word and Notion paste it. `chunk.go` does not normalise interior ones, |
| 93 | so the bytes reach the sealed row. |
| 94 | |
| 95 | 3. **`label`**, the part drafts 1 and 2 both got wrong. Given the span's |
| 96 | byte offset in the chunk text: |
| 97 | - walk back to the start of the enclosing sentence or line — `. `, |
| 98 | `! `, `? `, `\n`, or the start — **honouring a fixed abbreviation |
| 99 | list** (`Dr. Mr. Mrs. Ms. St. No. ext. Ltd. Co.`), which is stated |
| 100 | as incomplete and whose failures compose; |
| 101 | - **if the run contains an earlier recognised span, start after it**; |
| 102 | - if what remains ends in `:`, `—`, `–`, `-`, `=` or tab: |
| 103 | - **empty after removing it and trimming** → fall through to the |
| 104 | trail; |
| 105 | - **longer than 60** → `NoBox` for this kind; |
| 106 | - else → the label; |
| 107 | - else the heading trail; else the document title, never `""`. |
| 108 | |
| 109 | **Every span is taken by byte offset**, so `Span` is a slice of the input |
| 110 | and cannot be a reconstruction. Gate 36's byte-identical property is then |
| 111 | a property of the type rather than a discipline. |
| 112 | |
| 113 | --- |
| 114 | |
| 115 | ## 2. `internal/memoria/store.go` — candidates, after the ruling |
| 116 | |
| 117 | One unexported method called by `Search` after `rule`: |
| 118 | |
| 119 | 1. `want := fact.Wanted(q)`; empty → return. **No database work for a |
| 120 | query naming no kind**, which is the common case. |
| 121 | 2. Read the sealed chunk rows for `p.Results` — the **ruled** slice, |
| 122 | never `all` — in one `IN` query; open under `seal.PurposeChunk`. At |
| 123 | most 25 rows. |
| 124 | 3. **A chunk row that is gone is skipped silently**, as `livePassage` |
| 125 | skips a stale hit. It does not fail the search. |
| 126 | 4. `fact.Scan` per result, carrying that result's `Title` and `Heading` |
| 127 | as the label fallbacks and its `DocumentID`/`Ordinal` as the citation. |
| 128 | 5. **Group by `Kind`. Every rule below is per kind.** |
| 129 | 6. Dedupe on `Canonical` into an **ordered slice plus a seen-set** — not |
| 130 | a bare map, whose range order is randomised and whose first-seen order |
| 131 | gate 35 asserts. First occurrence wins the citation, and because |
| 132 | `p.Results` is in rank order that is the best-ranked passage. |
| 133 | 7. Any `NoBox` in the kind → drop the kind. |
| 134 | 8. Two candidates sharing the whole line `(Title, Heading, Label)` → drop |
| 135 | the kind. |
| 136 | 9. More than three → drop the kind. |
| 137 | |
| 138 | `Page` gains `Facts []Fact`; `Fact` carries `Kind, Title, Trail, Label, |
| 139 | Span, DocumentID, Ordinal`. **Not `ChunkID` and not the chunk text** — a |
| 140 | field a template can reach is a field a template eventually prints, which |
| 141 | is why `Result.item` is unexported. |
| 142 | |
| 143 | --- |
| 144 | |
| 145 | ## 3. `internal/memoria/handlers.go` |
| 146 | |
| 147 | `runSearch` copies `found.Facts` onto the page beside `found.Notices`. |
| 148 | That is the whole handler change, for **both** routes, because `index` |
| 149 | and `searchResults` share `runSearch` — which is why the page and the |
| 150 | fragment cannot disagree about the box any more than they can about |
| 151 | priming. |
| 152 | |
| 153 | --- |
| 154 | |
| 155 | ## 4. `templates/results.html` — inside the define |
| 156 | |
| 157 | Between the notices block and `<ol rst-results>`: below the withholding |
| 158 | callout, above the list, **inside** `{{define "results"}}` so the fragment |
| 159 | swap replaces box and list together. |
| 160 | |
| 161 | Each row carries title and trail as well as label and value (§8.8 — two |
| 162 | pages called "Contact" are the same row to a reader who cannot see which |
| 163 | is which), and **the citation link's text is the document title**, not a |
| 164 | bare "source". |
| 165 | |
| 166 | **The box must never reuse `rst-result-title`.** `search.js` targets that |
| 167 | attribute to decide where Enter goes; a link inside the box wearing it |
| 168 | would make Enter land on the box's citation, which §8.8 rules against. |
| 169 | The template comment says so. |
| 170 | |
| 171 | **No `safeSnippet`** — that helper exists to let FTS5's `<mark>` through |
| 172 | and nothing here has one. `Span` gets Go's ordinary contextual escaping. |
| 173 | **No anchor built from a document's bytes.** |
| 174 | |
| 175 | `memoria.css` gets `[rst-fact-box]`, `[rst-fact]`, `[rst-fact-label]`, |
| 176 | `[rst-fact-value]`, `[rst-fact-cite]` — **tokens only, no colour |
| 177 | literals**. `TestNoColourLiteralsInAppCSS` and |
| 178 | `TestEveryTemplateAttributeIsInTheDesignSystem` both bite, which is why |
| 179 | the stylesheet is written before the template rather than after. |
| 180 | |
| 181 | --- |
| 182 | |
| 183 | ## 5. `tools/drive` — the section nobody seeds is the section nobody sees |
| 184 | |
| 185 | AGENTS.md, and it has found five bugs this way. Seed three shapes, not |
| 186 | one: |
| 187 | |
| 188 | - a labelled office number and an eircode in an address block — the happy |
| 189 | case; |
| 190 | - **the motivating fixture's paragraph**, two numbers with distinct |
| 191 | sentence labels — the enumerated case, where the reader should see |
| 192 | "unused" and understand the page; |
| 193 | - **a bulleted contact pair** — the collision case, where the box should |
| 194 | be absent. The previous version of this plan called the two-sentence |
| 195 | paragraph the collision case; it is the opposite, and a seed that |
| 196 | demonstrates the wrong thing is worse than no seed. |
| 197 | |
| 198 | --- |
| 199 | |
| 200 | ## 6. Tests, in the order they get written |
| 201 | |
| 202 | **Unit, in `fact`, before any wiring.** `Wanted`; each validator's accept |
| 203 | and reject rows from §1.2; the label's eight cases from gate 37; spans |
| 204 | with an extension and with U+00A0. |
| 205 | |
| 206 | **Gates**, each carrying its mutation in the comment: |
| 207 | |
| 208 | - **35, 36, 37** in `internal/memoriatest/gates_test.go` — slice 0. |
| 209 | - **34 in `internal/memoria`**, at store level, against **a custody fake |
| 210 | that rules per item id**. `fixedCustody` returns |
| 211 | `sibling.Decide(f.give)` for every candidate from one field and cannot |
| 212 | express fixtures 2–6. Extending it is part of this branch. |
| 213 | - **38 is struck**, not written. The measurement it was going to assert |
| 214 | is prose in §8.8; a test that injects a term into every row and asserts |
| 215 | `bm25()` collapses tests SQLite and no mutation of this repo turns it |
| 216 | red. |
| 217 | |
| 218 | **And the mutations are not mine to pick.** §1.9: the gate review picks |
| 219 | its own, and this branch is not done until somebody who did not write it |
| 220 | has tried them. |
| 221 | |
| 222 | --- |
| 223 | |
| 224 | ## 7. What this does not touch |
| 225 | |
| 226 | `index/` (no column, no table, no weight, no `MatchQuery` change), |
| 227 | `chunk.go` (no change to `Indexable`), `models.go`, `migrations/` (nothing |
| 228 | new is stored), `seal/` (no new purpose), the sibling contract, the |
| 229 | connectors, `Delete`, `Rebuild`. |
| 230 | |
| 231 | That list is the strongest evidence for draft 3. Draft 2's design touched |
| 232 | five of them. |
| 233 | |
| 234 | --- |
| 235 | |
| 236 | ## 8. What building it changed (2026-09-15) |
| 237 | |
| 238 | Kept as the record of what the plan did not foresee. |
| 239 | |
| 240 | **The reachability limit is the ordinary case, not an edge.** Writing the |
| 241 | gates showed that bare `phone number` does not match a document reading |
| 242 | `Office: +353 (0)1 547 7854` — the words are not there, nothing ranks, |
| 243 | and there is nothing to extract from. Every fixture had to be given a |
| 244 | word the query could find. Pinned in both directions by |
| 245 | `TestTheBoxOnlySeesWhatRetrievalReturned`, and §12 carries the fix. |
| 246 | |
| 247 | **Three recognisers were wrong on ordinary text**, all found by the |
| 248 | implementation review and none by the fixtures: |
| 249 | `Take the N11 then exit at Bray` boxed `N11 then`; |
| 250 | `The deadline is 01-02-2024 10:30` boxed `01-02-2024 10`; |
| 251 | `Growth of +10 000 000 users` boxed `+10 000 000`. And the grouped IBAN |
| 252 | pattern made that kind inert on its commonest written shape — |
| 253 | `IE29 AIBK … 78 and our BIC is …` found nothing at all. §8.8 now carries |
| 254 | the three structural rules that fix them. |
| 255 | |
| 256 | **Label derivation was quadratic in the facts a chunk holds.** `chunk.Split` |
| 257 | bounds a chunk by WORDS, so a 2 MiB upload with no whitespace is one |
| 258 | chunk and one line: 48.5s to scan 1 MB, which at the upload cap is |
| 259 | minutes of search latency bought by one person's file. Deduping before |
| 260 | labelling, binary-searching the span list and bounding the backward walk |
| 261 | took 1 MB to 152 ms. |
| 262 | |
| 263 | **Four gates had holes the author's own mutations could not reach**, which |
| 264 | is §1.9's whole point. `silenced` counted before the ruling, a citation |
| 265 | cross-wired field by field, a collision check comparing only neighbours, |
| 266 | and rows sorted rather than first-seen — each passed every shipped |
| 267 | fixture and each now has one written for it. |
| 268 | |
| 269 | **And `addFacts` turned out to be structurally unable to see unruled |
| 270 | data**, because it takes only `*Page`. The leak mutations all have to be |
| 271 | made in `Search`, one level up. That is a stronger statement of the |
| 272 | design than the comment claiming it, and it is why the gate-34 comment |
| 273 | names the mutation at the level where it is reachable. |
| 274 | |
amadan