amadan
fix(index): rank adjacent terms, so "phone number" is not just "phone"
Paul searched the live corpus for "phone number" and got a passage of Virgin Media ACCOUNT numbers above the passage that states the office phone number. The keyboard prime shipped the same day then sent Enter to the wrong document. The root cause is not the weights, which was the first guess and wrong. Measured on the fixture in rank_test.go: "number" alone → every score 0.0000 "phone" alone → chunk 1 at 2.5607, chunk 2 at 1.8670 "phone" OR "number" → identical to "phone" alone FTS5 clamps IDF to ~1e-6 for a term that appears in most documents, so "number" contributed NOTHING to the ranking and a two-word query ranked exactly as its rarest word. "phone number" could not be told from "phone". The wrong passage then won on "Phone" sitting in a short, 3x-weighted heading, while the right one had the words in its body. AND does not fix it: it filters, it does not rank, and it put the same passage first. The phrase alone does fix the order and returns ONE result where there were twenty-five — narrowing recall to fix an ordering is the plausible-empty-list failure with a better score. So the whole query is added as one adjacent phrase, as a further OR operand. FTS5 sums per-phrase contributions, so a passage carrying the words together earns a third contribution and rises, one carrying them scattered is untouched, and everything is still returned: the right passage 1.8670 -> 4.3786, the wrong one unchanged, all fifteen back. TestOrRecallStillRanksBestMatchFirst has said since slice 0 that OR keeps recall and ranking does the discriminating. This is the half of that sentence that was never implemented. Two gates, four mutations, all red then reverted: the operand removed, the phrase ANDed rather than ORed, and the phrase added to single-term queries. The adjacency gate was written twice. The first version could not fail — two documents, no filler, so both terms appeared in every document, both IDFs clamped, every score collapsed to 0 and the order fell back to rowid, which happened to be the order it asserted. Removing the fix entirely left it green. It now runs on the filler corpus, and its scattered passage is winning beforehand: a test whose right answer is already first proves nothing about what put it there. Also records GET /search in spec 9.1, where gate 17's literal route list is read from, along with the older gap that 9.1 has always specified GET /query?q= for a page the app serves at GET /?q=. 🤖 Written by Claude Opus 5, reviewed by Paul.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4 files changed,
+219
−5
docs/superpowers/specs/2026-09-03-memoria-design.md+18 −0internal/memoria/index/index.go+31 −2internal/memoria/index/index_test.go+12 −3internal/memoria/index/rank_test.go+158 −0
diff --git a/docs/superpowers/specs/2026-09-03-memoria-design.md b/docs/superpowers/specs/2026-09-03-memoria-design.md| index 71f54a0..3b5db14 100644 |
| --- a/docs/superpowers/specs/2026-09-03-memoria-design.md |
| +++ b/docs/superpowers/specs/2026-09-03-memoria-design.md |
| @@ -2011,6 +2011,24 @@ memoria exposes exactly two surfaces: |
| path**, and a sibling presenting a connector token is refused: the two |
| credential sets are disjoint and separately scoped. |
| +**What the app actually serves, as of 2026-09-05**, because gate 17's |
| +list is literal and this is where it is read from: |
| + |
| +`GET /`, `GET /search`, `GET /documents/{id}`, `POST /documents`, |
| +`POST /documents/{id}/delete`, the connector routes under |
| +`/connectors/…`, and the auth routes. |
| + |
| +Two of those need saying out loud. **`GET /query?q=` above is not served |
| +and never has been** — the search page is `GET /?q=`, and the paragraph |
| +above has specified a path the app does not have since draft 1. |
| +**`GET /search` is new on 2026-09-05**: it renders the results region |
| +alone, from the same template `GET /?q=` renders it inside a page, so |
| +that live search can replace that region on a keystroke. It is a |
| +narrower surface than the page, under the same session gate, and it adds |
| +no passage the page did not already speak. It is not a second query |
| +surface and not a step toward one; delete the script and nothing calls |
| +it. |
| + |
| **No feed. Nothing consumes memoria.** If something ever does, that is a |
| new decision with its own leak-list entry, not an endpoint left lying |
| around for it. |
diff --git a/internal/memoria/index/index.go b/internal/memoria/index/index.go| index 07a2dc9..3db1e36 100644 |
| --- a/internal/memoria/index/index.go |
| +++ b/internal/memoria/index/index.go |
| @@ -279,7 +279,7 @@ func MatchQuery(raw string) string { |
| if cur.Len() == 0 { |
| return |
| } |
| - terms = append(terms, `"`+cur.String()+`"`) |
| + terms = append(terms, cur.String()) |
| cur.Reset() |
| } |
| for _, r := range raw { |
| @@ -299,7 +299,36 @@ func MatchQuery(raw string) string { |
| if len(terms) == 0 { |
| return "" |
| } |
| - return strings.Join(terms, " OR ") |
| + quoted := make([]string, 0, len(terms)+1) |
| + for _, t := range terms { |
| + quoted = append(quoted, `"`+t+`"`) |
| + } |
| + // The whole query as one adjacent phrase, added as a further OR |
| + // operand rather than replacing anything. |
| + // |
| + // This is the ranking half of the OR decision, and it was missing. |
| + // OR keeps recall, and the comment on TestOrRecallStillRanksBestMatchFirst |
| + // says the discriminating is then done by ranking — but BM25 alone |
| + // cannot do it when one of the words is common. FTS5 clamps IDF to |
| + // ~1e-6 for a term in most documents, so "number" contributed |
| + // nothing and "phone number" ranked exactly as "phone" did. A |
| + // passage with "Phone" in a short 3×-weighted heading beat the |
| + // passage that actually stated the office phone number. |
| + // |
| + // FTS5 scores each phrase in the expression and sums them, so a |
| + // passage carrying the words ADJACENTLY earns a third contribution |
| + // and rises; one carrying them scattered is untouched. Measured on |
| + // the fixture in rank_test.go: the right passage 1.8670 → 4.3786, |
| + // the wrong one unchanged at 2.5607, all fifteen still returned. |
| + // |
| + // An OR operand and not AND, and not the phrase alone: both of those |
| + // return only passages carrying the phrase — one result where there |
| + // were twenty-five. Narrowing recall to fix an ordering is the |
| + // plausible-empty-list failure with a better score. |
| + if len(terms) > 1 { |
| + quoted = append(quoted, `"`+strings.Join(terms, " ")+`"`) |
| + } |
| + return strings.Join(quoted, " OR ") |
| } |
| // randomName gives each in-memory index its own shared-cache key. Random |
diff --git a/internal/memoria/index/index_test.go b/internal/memoria/index/index_test.go| index a9103b5..20a81b6 100644 |
| --- a/internal/memoria/index/index_test.go |
| +++ b/internal/memoria/index/index_test.go |
| @@ -123,10 +123,19 @@ func TestRemoveDocumentDropsItsChunks(t *testing.T) { |
| } |
| func TestMatchQueryQuotesAndOrs(t *testing.T) { |
| + // The trailing operand is the whole query as one adjacent phrase. It |
| + // is a RANKING signal: FTS5 sums per-phrase contributions, so a |
| + // passage carrying the words together scores above one carrying them |
| + // scattered, while both are still returned. See MatchQuery. |
| + // |
| + // "what's up" is the interesting row. The apostrophe splits the query |
| + // into three terms, and the phrase reassembles them as "what s up" — |
| + // which matches, because unicode61 tokenises the apostrophe out of a |
| + // document's "what's up" in exactly the same way. |
| for in, want := range map[string]string{ |
| - "june revenue": `"june" OR "revenue"`, |
| - "what's up": `"what" OR "s" OR "up"`, |
| - " spaced out ": `"spaced" OR "out"`, |
| + "june revenue": `"june" OR "revenue" OR "june revenue"`, |
| + "what's up": `"what" OR "s" OR "up" OR "what s up"`, |
| + " spaced out ": `"spaced" OR "out" OR "spaced out"`, |
| "!!!": ``, |
| "café": `"café"`, |
| } { |
diff --git a/internal/memoria/index/rank_test.go b/internal/memoria/index/rank_test.go| new file mode 100644 |
| index 0000000..9f7d24f |
| --- /dev/null |
| +++ b/internal/memoria/index/rank_test.go |
| @@ -0,0 +1,158 @@ |
| +package index |
| + |
| +import "testing" |
| + |
| +// The query that found this: "phone number" on the real corpus put a |
| +// passage of Virgin Media ACCOUNT numbers above the passage that states |
| +// the office phone number. Paul hit it on the live instance, and the |
| +// keyboard prime shipped the same day sent Enter to the wrong document. |
| +// |
| +// The corpus below is what makes the test able to fail, and it took two |
| +// attempts to build. A two-document version scored every hit at exactly |
| +// 0.0000: FTS5's bm25 clamps IDF to ~1e-6 for a term that appears in |
| +// every document, so the scores collapse and the order is rowid. "number" |
| +// has to be COMMON here and "phone" RARE, as on the real corpus, or this |
| +// measures nothing. |
| +// |
| +// Measured on that corpus before the fix: |
| +// |
| +// "phone" OR "number" → chunk 1 at 2.5607, chunk 2 at 1.8670 |
| +// "number" alone → every score 0.0000 |
| +// "phone" alone → identical to "phone" OR "number" |
| +// |
| +// The second line is the finding. A common term contributes nothing to |
| +// ranking, so a two-word query ranked exactly as its rarest word alone, |
| +// and "phone number" could not be told from "phone". |
| +func rankCorpus(t *testing.T) *Index { |
| + t.Helper() |
| + ix, err := Open() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { ix.Close() }) |
| + |
| + var entries []Entry |
| + add := func(heading, title, text string) { |
| + entries = append(entries, Entry{ |
| + ChunkID: int64(len(entries) + 1), |
| + DocumentID: int64(len(entries) + 1), |
| + Ordinal: len(entries), |
| + Title: title, |
| + Path: "Team Tito Limited", |
| + Heading: heading, |
| + Text: text, |
| + }) |
| + } |
| + |
| + // Chunk 1: "Phone" in a short, 3×-weighted heading, and no phone |
| + // number in the body. Chunk 2: the answer, in the body, twice. |
| + add("Utilities › Internet & Phone", "Office", |
| + "Virgin Media - login - Office 7: account number 80854641 - Office 2: account number 80753900") |
| + add("Office Address", "Office", |
| + "Turn the knob. The office phone number is: +353 (0)1 547 7854. We also have the following unused phone numbers: +353 (0)1 598 1234.") |
| + |
| + // "number" common across the corpus, as on the real one. |
| + add("Business & VAT Numbers", "Business & VAT Numbers", |
| + "Our company number is 566334. Our VAT number is IE3384527RH. Our DUNS/D-U-N-S number is 985632364.") |
| + add("Bank", "Banking", "The account number is 12345678 and the sort code is 93-00-11.") |
| + add("Payroll", "Payroll", "Each employee number is assigned on their first day.") |
| + add("Insurance", "Insurance", "The policy number is renewed every January.") |
| + add("Revenue", "Q3 board pack", "June revenue was 1,350,000 across all regions.") |
| + add("Costs", "Q3 board pack", "Hosting costs fell after the migration.") |
| + add("Employee Offboarding", "Processes", "Collect the laptop and the serial number of any device issued.") |
| + add("Company Registration", "Registration", "The registration number appears on every invoice.") |
| + add("Invoices", "Finance", "Every invoice carries a sequential number for the auditor.") |
| + add("Passport numbers", "Travel", "Each passport number is stored in the safe, never in a document.") |
| + add("Suppliers", "Finance", "Each supplier has a reference number in the ledger.") |
| + add("Meeting, 12 August", "Minutes", "We will not build a suite index app.") |
| + add("Sealing notes", "Sealing", "A sheet's contents are sealed at rest.") |
| + |
| + if err := ix.Add(entries...); err != nil { |
| + t.Fatal(err) |
| + } |
| + return ix |
| +} |
| + |
| +// Broken by removing the phrase operand from MatchQuery — red, then |
| +// reverted. Broken again by adding the phrase but ANDing it, which |
| +// returns the one passage and drops the other fourteen — red on the |
| +// recall assertion below, then reverted. 2026-09-05. |
| +func TestAdjacentTermsOutrankAScatteredCommonWord(t *testing.T) { |
| + ix := rankCorpus(t) |
| + hits, err := ix.Search("phone number", 25) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(hits) == 0 { |
| + t.Fatal("no hits at all") |
| + } |
| + if hits[0].ChunkID != 2 { |
| + t.Errorf("top hit is chunk %d (%q), want chunk 2 — the passage that states the number", |
| + hits[0].ChunkID, hits[0].Heading) |
| + } |
| + // Recall is the other half, and the half a phrase-only query loses. |
| + // Every passage carrying either word must still come back: narrowing |
| + // to the phrase would turn a working search into a one-result page, |
| + // which is the plausible-empty-list failure wearing a better score. |
| + if len(hits) < 10 { |
| + t.Errorf("only %d hits; the phrase boost has narrowed recall instead of reordering it", len(hits)) |
| + } |
| +} |
| + |
| +// The boost must be earned by ADJACENCY, not by both words merely being |
| +// present — which is what AND already gave, and what did not help. |
| +// |
| +// The first version of this test could not fail. It used two documents |
| +// and no filler, so both terms appeared in every document, FTS5 clamped |
| +// both IDFs to ~1e-6, every score collapsed to 0 and the order fell back |
| +// to rowid — which happened to be the order it asserted. Removing the fix |
| +// entirely left it green. It needed the filler corpus for the same reason |
| +// the test above does, and it needed the scattered passage to be WINNING |
| +// beforehand: a test where the right answer is already first proves |
| +// nothing about what put it there. |
| +// |
| +// So the scattered passage says both words three times each and the |
| +// adjacent one says them once. On term frequency alone the scattered one |
| +// wins; only adjacency turns it over. |
| +// |
| +// Broken by removing the phrase operand — red, then reverted. |
| +// 2026-09-05. |
| +func TestPhraseBoostNeedsAdjacency(t *testing.T) { |
| + ix := rankCorpus(t) |
| + if err := ix.Add( |
| + Entry{ChunkID: 90, DocumentID: 90, Ordinal: 0, Heading: "Scattered", Title: "Desk", |
| + Text: "The phone, the phone, the phone. Later, ask for a number, a number, another number."}, |
| + Entry{ChunkID: 91, DocumentID: 91, Ordinal: 0, Heading: "Adjacent", Title: "Desk", |
| + Text: "The phone number is on the desk."}, |
| + ); err != nil { |
| + t.Fatal(err) |
| + } |
| + hits, err := ix.Search("phone number", 25) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + var scattered, adjacent int |
| + for i, h := range hits { |
| + switch h.ChunkID { |
| + case 90: |
| + scattered = i + 1 |
| + case 91: |
| + adjacent = i + 1 |
| + } |
| + } |
| + if scattered == 0 || adjacent == 0 { |
| + t.Fatalf("both passages must be returned; scattered=%d adjacent=%d of %d hits", |
| + scattered, adjacent, len(hits)) |
| + } |
| + if adjacent > scattered { |
| + t.Errorf("the scattered passage ranked %d and the adjacent one %d — "+ |
| + "adjacency is not being rewarded, only presence", scattered, adjacent) |
| + } |
| +} |
| + |
| +// A single-word query has no phrase, and must not grow a spurious one. |
| +func TestSingleTermQueryIsUnchanged(t *testing.T) { |
| + if got, want := MatchQuery("revenue"), `"revenue"`; got != want { |
| + t.Errorf("MatchQuery(%q) = %q, want %q", "revenue", got, want) |
| + } |
| +} |