amadan

oficina / memoria Public

🤖 feat(index): carry source ancestry in its own low-weighted column

Stage 1 of Notion ingest, and it lands alone because it re-ranks every
document already in the corpus.

A page's meaning is largely positional: "Week 1" under "Handbook ›
Onboarding" is onboarding week 1, and the same page floating free is
nothing. memoria's chunk trail comes only from ATX headings inside one
document, so an imported page arrives with no ancestry at all.

The obvious implementation — prepend the ancestry to the heading trail —
is a trap. The heading column is weighted 3x, and ancestry repeats into
every chunk of every page beneath it, so the top of a tree would outrank
the content of it and a search for "handbook" would return the whole
handbook above the page that answers the question. Ancestry therefore
gets its own FTS column at 0.5, below the body. The 0.5 is a guess and
the constant says so; the ordering is not.

Measured, because the first version of the gate for this could not fail.
bm25() sums per-column contributions, so a row matching in path AND body
outranks a path-only match under any positive weights — the first fixture
was green with the mutation applied. Worse, the corpus had the query term
in every row, so IDF collapsed, every score came back -0.000001, and the
ordering that looked like a ranking was floating-point noise. With 200
filler chunks the numbers separate: an ancestry-only row scores -3.196 at
the path weight and -7.218 with the path folded into the heading column,
against -4.961 for the row whose body actually answers. The gate now
observes the flip, and a second test guards the fixture itself.

Also here, because Ingest had to change anyway:

  - Renames and moves now reach memoria. The unchanged-body short-circuit
    returned without touching the title, so a page renamed upstream kept
    its old name forever — and an ancestor rename does not change a
    descendant's edit time at all, so nothing else would ever notice.
    Neither bumps the revision: the passages a citation points at have
    not moved.
  - AllowTitleOnly, off for people and on for connectors. An empty paste
    is a mistake worth refusing; an empty Notion page is a fact about the
    source, and both alternatives are worse — failing the run lets one
    placeholder block an import, and skipping makes the page an absence
    with nothing to explain it.
  - Ingest takes a spec struct. Six positional strings was one transposed
    pair away from a silent bug.

Every gate here was broken, watched go red, and reverted — six mutations,
listed in the test comments with the date.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VauJkQsPeGRErbWFiQNZzC
Paul Campbell pushed by paul@keymail.dev 74cc335a42e3812ad74d5738bdefaca59c828e55 parent 4c1ef47
11 files changed, +654 −40
  • internal/memoria/chunk/chunk.go +44 −9
  • internal/memoria/handlers.go +7 −1
  • internal/memoria/index/index.go +45 −8
  • internal/memoria/index/path_test.go +178 −0
  • internal/memoria/migrations/0003_document_path.sql +10 −0
  • internal/memoria/models.go +8 −0
  • internal/memoria/seal/seal.go +5 −0
  • internal/memoria/store.go +139 −21
  • internal/memoriatest/ancestry_test.go +195 −0
  • internal/memoriatest/harness_test.go +20 −0
  • tools/seed/main.go +3 −1
diff --git a/internal/memoria/chunk/chunk.go b/internal/memoria/chunk/chunk.go
index 5311f25..c5846c5 100644
--- a/internal/memoria/chunk/chunk.go
+++ b/internal/memoria/chunk/chunk.go
@@ -45,6 +45,10 @@ type Chunk struct {
// Text is the passage, with the trail prepended when Indexable is
// called. Stored sealed.
Text string
+ // Path is the document's ancestry in its SOURCE, outermost first —
+ // distinct from Trail, which is the heading path inside this one
+ // document. Empty for anything uploaded or pasted.
+ Path []string
}
// Heading returns the trail joined for display and for indexing.
@@ -54,15 +58,7 @@ type Chunk struct {
// join then renders "› Revenue" with a leading separator and nothing
// before it. Found by looking at the page; no test saw it, because every
// fixture started at "#".
-func (c Chunk) Heading() string {
- parts := make([]string, 0, len(c.Trail))
- for _, t := range c.Trail {
- if strings.TrimSpace(t) != "" {
- parts = append(parts, t)
- }
- }
- return strings.Join(parts, " › ")
-}
+func (c Chunk) Heading() string { return joinTrail(c.Trail) }
// Indexable is what the index tokenises: the heading trail followed by
// the body. The trail is repeated into every chunk of a section
@@ -244,3 +240,42 @@ func normalise(s string) string {
}
return s
}
+
+// Path is a document's ancestry in its source, outermost first, carried
+// alongside the trail rather than merged into it.
+//
+// Kept separate because the two are scored differently and must be: a
+// heading is written inside the document, while ancestry is inherited by
+// every chunk of every document beneath it. index.PathWeight explains
+// what merging them does to ranking.
+func (c Chunk) PathString() string { return joinTrail(c.Path) }
+
+// SplitUnder is Split for a document that has ancestry. Every chunk
+// carries the same path, because every passage of a nested page is
+// equally nested.
+func SplitUnder(path []string, text string) []Chunk {
+ out := Split(text)
+ if len(path) == 0 {
+ return out
+ }
+ p := make([]string, len(path))
+ copy(p, path)
+ for i := range out {
+ out[i].Path = p
+ }
+ return out
+}
+
+// joinTrail renders a trail for display and indexing, dropping empty
+// segments. Shared by Heading and PathString so the two cannot drift —
+// an earlier bug rendered "› Revenue" because one side filtered and the
+// other did not.
+func joinTrail(parts []string) string {
+ kept := make([]string, 0, len(parts))
+ for _, t := range parts {
+ if strings.TrimSpace(t) != "" {
+ kept = append(kept, t)
+ }
+ }
+ return strings.Join(kept, " › ")
+}
diff --git a/internal/memoria/handlers.go b/internal/memoria/handlers.go
index 54758ea..e59ce35 100644
--- a/internal/memoria/handlers.go
+++ b/internal/memoria/handlers.go
@@ -180,7 +180,13 @@ func (a *app) createDocument(w http.ResponseWriter, r *http.Request) {
return
}
- if _, err := a.store.Ingest(r.Context(), source, key, title, body); err != nil {
+ // AllowTitleOnly stays OFF here. An empty upload or paste is a
+ // person's mistake and refusing it is the useful answer — the empty
+ // check above already does. Connectors turn it on, because an empty
+ // page is a fact about the source rather than a mistake.
+ if _, err := a.store.Ingest(r.Context(), IngestSpec{
+ Source: source, Key: key, Title: title, Body: body,
+ }); err != nil {
// No document content in the log line — AGENTS.md's rule, and
// the reason this carries no title either.
a.logger.Error("ingest failed", "source", source, "bytes", len(body), "err", err)
diff --git a/internal/memoria/index/index.go b/internal/memoria/index/index.go
index 30d3795..dfb2662 100644
--- a/internal/memoria/index/index.go
+++ b/internal/memoria/index/index.go
@@ -76,6 +76,7 @@ func Open() (*Index, error) {
CREATE VIRTUAL TABLE chunks USING fts5(
heading,
body,
+ path,
chunk_id UNINDEXED,
document_id UNINDEXED,
ordinal UNINDEXED,
@@ -102,6 +103,13 @@ type Entry struct {
Ordinal int
Heading string
Text string
+ // Path is the trail of the document's ANCESTORS in its source —
+ // "Handbook › Onboarding" for a Notion page nested two deep. Empty
+ // for anything uploaded or pasted, which has no ancestry.
+ //
+ // It is a separate column from Heading rather than a prefix on it,
+ // and PathWeight says why.
+ Path string
}
// Add indexes entries in one transaction.
@@ -113,13 +121,13 @@ func (i *Index) Add(entries ...Entry) error {
return err
}
defer tx.Rollback()
- st, err := tx.Prepare(`INSERT INTO chunks (heading, body, chunk_id, document_id, ordinal) VALUES (?, ?, ?, ?, ?)`)
+ st, err := tx.Prepare(`INSERT INTO chunks (heading, body, path, chunk_id, document_id, ordinal) VALUES (?, ?, ?, ?, ?, ?)`)
if err != nil {
return err
}
defer st.Close()
for _, e := range entries {
- if _, err := st.Exec(e.Heading, e.Text, e.ChunkID, e.DocumentID, e.Ordinal); err != nil {
+ if _, err := st.Exec(e.Heading, e.Text, e.Path, e.ChunkID, e.DocumentID, e.Ordinal); err != nil {
return err
}
}
@@ -146,9 +154,30 @@ func (i *Index) Count() (int, error) {
return n, err
}
-// Search runs a BM25 query. The heading column is weighted above the
-// body: a passage under "Revenue" is a better answer to "revenue" than
-// one that mentions the word in passing.
+// Column weights for bm25(), in column order: heading, body, path.
+//
+// HeadingWeight above BodyWeight because a passage under "Revenue" is a
+// better answer to "revenue" than one that mentions the word in passing.
+//
+// PathWeight is BELOW the body, and that is the load-bearing one. A
+// document's ancestry repeats into every one of its chunks — every
+// passage of every page under "Handbook" carries "Handbook" — so at the
+// heading's weight the top of a source's tree outranks the content of
+// it, and a search for "handbook" returns the whole handbook above the
+// page that answers the question. The failure would look like bad
+// search rather than a bug, which is the kind that survives.
+//
+// 0.5 IS A GUESS. It is half the body's weight because nothing has
+// measured it; the corpus it wants tuning against does not exist yet.
+// What is not a guess is the ORDERING, which TestAncestryCannotOutrank-
+// Content pins by measurement — see that test for the observed scores.
+const (
+ HeadingWeight = 3.0
+ BodyWeight = 1.0
+ PathWeight = 0.5
+)
+
+// Search runs a BM25 query, weighted by the constants above.
func (i *Index) Search(query string, limit int) ([]Hit, error) {
q := MatchQuery(query)
if q == "" {
@@ -159,14 +188,22 @@ func (i *Index) Search(query string, limit int) ([]Hit, error) {
}
i.mu.RLock()
defer i.mu.RUnlock()
+ // snippet()'s second argument is a COLUMN INDEX, and 1 is body.
+ // The path column is appended third for exactly that reason: adding
+ // it before body would silently start snippeting the wrong column,
+ // and a snippet of the wrong column still renders.
rows, err := i.db.Query(`
SELECT chunk_id, document_id, ordinal, heading,
snippet(chunks, 1, '<mark>', '</mark>', '…', 24),
- bm25(chunks, 3.0, 1.0)
+ bm25(chunks, ?, ?, ?)
FROM chunks
WHERE chunks MATCH ?
- ORDER BY bm25(chunks, 3.0, 1.0)
- LIMIT ?`, q, limit)
+ ORDER BY bm25(chunks, ?, ?, ?)
+ LIMIT ?`,
+ HeadingWeight, BodyWeight, PathWeight,
+ q,
+ HeadingWeight, BodyWeight, PathWeight,
+ limit)
if err != nil {
return nil, err
}
diff --git a/internal/memoria/index/path_test.go b/internal/memoria/index/path_test.go
new file mode 100644
index 0000000..604f003
--- /dev/null
+++ b/internal/memoria/index/path_test.go
@@ -0,0 +1,178 @@
+package index
+
+import (
+ "fmt"
+ "testing"
+)
+
+// buildTrap fills an index with the fixture that reproduces the ancestry
+// scoring trap, and returns it.
+//
+// Two things about this fixture are load-bearing, and the first version
+// of it had neither.
+//
+// FIRST, the shape. bm25() SUMS per-column contributions, so a row
+// matching in path AND body outranks a row matching only in path under
+// any positive weights — a fixture where the "good" answer also carries
+// the term in its ancestry proves nothing. The discriminating shape is
+// many pages whose ONLY match is an inherited ancestor title, against
+// one page, outside that subtree, whose BODY answers the query.
+//
+// SECOND, the filler, and this is the subtler one. Without the 200
+// unrelated chunks below, "handbook" appears in EVERY indexed row, BM25's
+// IDF term collapses to about zero, every score comes back as -1e-06, and
+// the resulting order is floating-point noise that happens to look like a
+// ranking. Measured: with no filler every score was -0.000001; with it,
+// the path-only rows score -3.196 at PathWeight and -7.218 with the path
+// folded into the heading column, against -4.961 for the body match. Only
+// the second set of numbers can distinguish a working weight from an
+// inert one.
+//
+// pathAsHeading models the mutation: ancestry folded into the heading
+// column at heading weight, which is what a naive implementation does.
+func buildTrap(t *testing.T, pathAsHeading bool) *Index {
+ t.Helper()
+ ix, err := Open()
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { ix.Close() })
+
+ var entries []Entry
+ var id int64
+ add := func(docID int64, heading, path, body string) {
+ id++
+ e := Entry{ChunkID: id, DocumentID: docID, Ordinal: 0, Heading: heading, Text: body}
+ if pathAsHeading {
+ if path != "" {
+ e.Heading = path + " › " + heading
+ }
+ } else {
+ e.Path = path
+ }
+ entries = append(entries, e)
+ }
+
+ // Filler, so the query term is rare and IDF means something.
+ for i := 0; i < 200; i++ {
+ add(int64(1000+i), fmt.Sprintf("Note %d", i), "Archive › Notes",
+ fmt.Sprintf("Meeting %d discussed procurement, latency budgets and the vendor renewal.", i))
+ }
+ // 20 pages under "Handbook". No body says "handbook"; they match only
+ // through the ancestry they inherited.
+ for i := 0; i < 20; i++ {
+ add(int64(100+i), fmt.Sprintf("Section %d", i), "Handbook › Onboarding",
+ fmt.Sprintf("Week %d covers expenses, travel booking and the equipment request form.", i))
+ }
+ // One page elsewhere whose BODY is the actual answer.
+ add(999, "Tooling", "Engineering › Platform",
+ "The handbook is generated from this repository and published nightly.")
+
+ if err := ix.Add(entries...); err != nil {
+ t.Fatal(err)
+ }
+ return ix
+}
+
+// TestAncestryCannotOutrankContent is the gate for the trap described on
+// PathWeight: a document's ancestry repeats into every chunk beneath it,
+// so ancestry scored at heading weight buries real answers under the
+// whole subtree that merely sits below a matching ancestor title.
+//
+// MUTATION-TESTED 2026-09-05: folding the path into the heading column
+// (buildTrap's pathAsHeading branch, which is what a naive implementation
+// does) makes this fail — the top three hits become Sections 0, 1 and 2,
+// and the page that answers the query falls below twenty that do not.
+// TestAncestryTrapFlipIsObserved asserts that flip so this gate cannot
+// quietly stop discriminating. Reverted.
+func TestAncestryCannotOutrankContent(t *testing.T) {
+ ix := buildTrap(t, false)
+ hits, err := ix.Search("handbook", 25)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(hits) == 0 {
+ t.Fatal("no hits: the fixture matches nothing, so this gate proves nothing")
+ }
+ if hits[0].DocumentID != 999 {
+ t.Errorf("ancestry outranked content: top hit is document %d, want 999 "+
+ "(the page whose body answers the query); %d ancestry-only pages matched",
+ hits[0].DocumentID, len(hits)-1)
+ }
+}
+
+// TestAncestryTrapFlipIsObserved fails if the mutation does NOT change the
+// ranking. A gate for a ranking property is worthless unless the ranking
+// has been seen to move, and this codebase has shipped three gates that
+// could not fail for the thing they named.
+func TestAncestryTrapFlipIsObserved(t *testing.T) {
+ top := func(pathAsHeading bool) (int64, float64) {
+ ix := buildTrap(t, pathAsHeading)
+ hits, err := ix.Search("handbook", 25)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(hits) == 0 {
+ t.Fatal("fixture matched nothing")
+ }
+ return hits[0].DocumentID, hits[0].Score
+ }
+ goodDoc, goodScore := top(false)
+ badDoc, badScore := top(true)
+ t.Logf("path column at %.1f : top doc=%d score=%.4f", PathWeight, goodDoc, goodScore)
+ t.Logf("path in heading : top doc=%d score=%.4f", badDoc, badScore)
+
+ if goodDoc != 999 {
+ t.Errorf("with the path column, top doc=%d want 999", goodDoc)
+ }
+ if badDoc == 999 {
+ t.Errorf("the mutation did NOT change the ranking: doc 999 is still top with " +
+ "ancestry at heading weight. This fixture cannot detect the trap, so " +
+ "TestAncestryCannotOutrankContent is a gate written to pass. Strengthen " +
+ "the fixture until the order flips.")
+ }
+}
+
+// TestTrapFixtureCanDiscriminate guards the FIXTURE, not production code,
+// and it exists because this fixture silently stopped being able to prove
+// anything once already.
+//
+// The first measurement of the trap was taken on a corpus where the query
+// term appeared in every indexed row. BM25's IDF term collapsed to about
+// zero, every score came back as -0.000001, and moving the path weight
+// from 0.5 to 99 changed nothing — yet the rows still came back in an
+// order, and that order looked exactly like a working ranking. A weight
+// that is silently ignored ranks the same as a weight that is wrong, so
+// TestAncestryCannotOutrankContent was passing for a reason unrelated to
+// the property it names.
+//
+// There is no production mutation for this one: it asserts that the
+// corpus and the engine can tell two weightings apart at all. Its
+// mutation is therefore to the fixture.
+//
+// MUTATION-TESTED 2026-09-05: deleting the filler loop from buildTrap
+// makes this fail on the degenerate-IDF assertion. Reverted.
+func TestTrapFixtureCanDiscriminate(t *testing.T) {
+ ix := buildTrap(t, false)
+ score := func(pw float64) float64 {
+ var v float64
+ // Document 100 is one of the ancestry-only matches, so its score
+ // depends on the path weight and nothing else.
+ err := ix.db.QueryRow(
+ `SELECT bm25(chunks, ?, ?, ?) FROM chunks WHERE chunks MATCH ? AND document_id = 100`,
+ HeadingWeight, BodyWeight, pw, MatchQuery("handbook")).Scan(&v)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return v
+ }
+ low, high := score(PathWeight), score(99)
+ t.Logf("ancestry-only row: path weight %.1f → %.4f, path weight 99 → %.4f", PathWeight, low, high)
+ if low == high {
+ t.Fatalf("path weight is inert: %.9f at both %.1f and 99", low, PathWeight)
+ }
+ if low > -1e-4 {
+ t.Fatalf("score %.9f is ~0, so IDF is degenerate and this corpus cannot "+
+ "discriminate a working weight from an inert one", low)
+ }
+}
diff --git a/internal/memoria/migrations/0003_document_path.sql b/internal/memoria/migrations/0003_document_path.sql
new file mode 100644
index 0000000..6aff5ea
--- /dev/null
+++ b/internal/memoria/migrations/0003_document_path.sql
@@ -0,0 +1,10 @@
+-- sealed_path is a document's ancestry in its source — "Handbook ›
+-- Onboarding" for a page nested two deep. Sealed under its own HKDF
+-- purpose, because a path is content: "Acquisitions › Project Bluebird"
+-- names a thing whether or not anyone opens the page.
+--
+-- Nullable rather than NOT NULL DEFAULT: existing rows came from upload
+-- and paste, which have no ancestry, and an empty blob and "no ancestry"
+-- are the same fact here. A DEFAULT would also have to be a valid
+-- sealed blob, which "" is not.
+ALTER TABLE `documents` ADD `sealed_path` blob;
diff --git a/internal/memoria/models.go b/internal/memoria/models.go
index fc5f1ee..5c442f6 100644
--- a/internal/memoria/models.go
+++ b/internal/memoria/models.go
@@ -46,6 +46,14 @@ type Document struct {
// SealedTitle and SealedBody are sealed under distinct HKDF purposes.
SealedTitle []byte `gorm:"not null"`
SealedBody []byte `gorm:"not null"`
+ // SealedPath is the document's ancestry in its source, outermost
+ // first — "Handbook › Onboarding" for a page nested two deep. Empty
+ // for uploads and paste, which have no ancestry.
+ //
+ // Sealed, and under its own purpose: a path is content. It is
+ // indexed in its own low-weighted column rather than folded into
+ // the heading trail, for the reason index.PathWeight gives.
+ SealedPath []byte
// Bytes is the plaintext length, kept so the UI can say how big a
// document is without opening it. A size is already inferable from
// the sealed length, so this leaks nothing new.
diff --git a/internal/memoria/seal/seal.go b/internal/memoria/seal/seal.go
index 1bae0ac..29957fa 100644
--- a/internal/memoria/seal/seal.go
+++ b/internal/memoria/seal/seal.go
@@ -36,6 +36,11 @@ const (
PurposeBody = "document.body"
PurposeSnapshot = "index.snapshot"
PurposeExternal = "document.external_id"
+ // PurposePath seals a document's ancestry in its source. Its own
+ // purpose rather than reusing PurposeTitle: a path and a title are
+ // different kinds of thing, and the rule here is one purpose per
+ // kind so that a mix-up cannot decrypt.
+ PurposePath = "document.path"
)
func sha256New() hash.Hash { return sha256.New() }
diff --git a/internal/memoria/store.go b/internal/memoria/store.go
index fefde48..d29a88c 100644
--- a/internal/memoria/store.go
+++ b/internal/memoria/store.go
@@ -58,27 +58,68 @@ func (s *Store) externalID(key string) (string, error) {
return s.sealer.Fingerprint(seal.PurposeExternal, []byte("key\x00"+key))
}
+// IngestSpec is one document to store.
+//
+// A struct rather than a parameter list because the connectors added two
+// more fields and a five-string call site is a bug waiting for someone
+// to transpose two of them.
+type IngestSpec struct {
+ // Source names where this came from — "upload", "paste", "notion".
+ Source string
+ // Key is the STABLE identifier in the source: a file path, a Notion
+ // page id, a pasted document's title. Not the body — see
+ // Document.ExternalID.
+ Key string
+ // Title and Body are content.
+ Title string
+ Body string
+ // Path is the document's ancestry in its source, outermost first.
+ // Empty for upload and paste, which have none.
+ Path []string
+ // AllowTitleOnly permits a document whose body is empty, storing the
+ // title as the body.
+ //
+ // Off for upload and paste, where an empty submission is a person's
+ // mistake and refusing it is the useful answer. On for connectors,
+ // where an empty page is a FACT ABOUT THE SOURCE — real workspaces
+ // are full of placeholders and title-only rows — and both
+ // alternatives are wrong: failing the run lets one empty page block
+ // an entire import, and skipping makes the page an absence with
+ // nothing to explain it.
+ AllowTitleOnly bool
+}
+
// Ingest stores a document and indexes it.
//
-// Re-ingesting the same ExternalID replaces the document's chunks and
-// bumps its revision. The old chunks are deleted from the index *and* the
-// database before the new ones land, so a passage that the new revision
-// removed cannot survive as a citation to text that is no longer there —
-// which is the stale-passage failure the spec's §4.1 refuses to allow.
-// key is the STABLE identifier in the source: a file path, a page id, a
-// pasted document's title. Not the body — see Document.ExternalID.
-func (s *Store) Ingest(ctx context.Context, source, key, title, body string) (int64, error) {
- title = strings.TrimSpace(title)
- body = strings.TrimSpace(body)
+// Re-ingesting the same Key replaces the document's chunks and bumps its
+// revision. The old chunks are deleted from the index *and* the database
+// before the new ones land, so a passage that the new revision removed
+// cannot survive as a citation to text that is no longer there — the
+// stale-passage failure the spec's §4.1 refuses to allow.
+func (s *Store) Ingest(ctx context.Context, spec IngestSpec) (int64, error) {
+ title := strings.TrimSpace(spec.Title)
+ body := strings.TrimSpace(spec.Body)
if body == "" {
- return 0, errors.New("nothing to store: the document is empty")
+ if !spec.AllowTitleOnly {
+ return 0, errors.New("nothing to store: the document is empty")
+ }
+ if title == "" {
+ return 0, errors.New("nothing to store: the document has neither a body nor a title")
+ }
+ // A title-only document IS its title. Storing an empty body
+ // instead would index nothing and make the page unfindable by
+ // the only words it has.
+ body = title
}
if title == "" {
title = firstLine(body)
}
+ key := spec.Key
if strings.TrimSpace(key) == "" {
key = title
}
+ pathStr := chunk.Chunk{Path: spec.Path}.PathString()
+
external, err := s.externalID(key)
if err != nil {
return 0, err
@@ -92,32 +133,63 @@ func (s *Store) Ingest(ctx context.Context, source, key, title, body string) (in
if err != nil {
return 0, err
}
+ var sealedPath []byte
+ if pathStr != "" {
+ if sealedPath, err = s.sealer.Seal(seal.PurposePath, []byte(pathStr)); err != nil {
+ return 0, err
+ }
+ }
sealedBody, err := s.sealer.Seal(seal.PurposeBody, []byte(body))
if err != nil {
return 0, err
}
- chunks := chunk.Split(body)
+ chunks := chunk.SplitUnder(spec.Path, body)
if len(chunks) == 0 {
return 0, errors.New("nothing to store: the document produced no passages")
}
var docID int64
- var unchanged bool
+ var unchanged, rechunk bool
err = s.db.G.WithContext(ctx).Transaction(func(tx *gormTx) error {
var existing Document
- res := tx.Where("source = ? AND external_id = ?", source, external).First(&existing)
+ res := tx.Where("source = ? AND external_id = ?", spec.Source, external).First(&existing)
switch {
case res.Error == nil:
docID = existing.ID
if existing.BodyPrint == bodyPrint {
- // Same key, same body: nothing changed. Do not bump the
- // revision — a revision that moves without the content
- // moving would make every cached authorization stale for
- // no reason.
- unchanged = true
- return nil
+ // Same key, same body — but the page may have been
+ // RENAMED or MOVED, and neither touches the body.
+ //
+ // An earlier version returned here unconditionally, so a
+ // renamed page kept its old title forever and a moved one
+ // kept its old ancestry. Worse for a connector: an
+ // ancestor rename does not change a descendant's own edit
+ // time at all, so nothing upstream ever signalled it.
+ // Found by a design review.
+ moved, err := s.titleOrPathMoved(existing, title, pathStr)
+ if err != nil {
+ return err
+ }
+ if !moved {
+ // Genuinely unchanged. Do NOT bump the revision: a
+ // revision that moves without the content moving
+ // would make every cached authorization stale for no
+ // reason.
+ unchanged = true
+ return nil
+ }
+ // Title or path only. Still no revision bump — the
+ // passages a citation points at have not moved — and the
+ // chunks are left alone, because the path lives on the
+ // document. The index is refreshed below, since it is
+ // the index that carries the path.
+ return tx.Model(&Document{}).Where("id = ?", docID).Updates(map[string]any{
+ "sealed_title": sealedTitle,
+ "sealed_path": sealedPath,
+ }).Error
}
+ rechunk = true
if err := tx.Where("document_id = ?", docID).Delete(&Chunk{}).Error; err != nil {
return err
}
@@ -125,19 +197,22 @@ func (s *Store) Ingest(ctx context.Context, source, key, title, body string) (in
"revision": existing.Revision + 1,
"sealed_title": sealedTitle,
"sealed_body": sealedBody,
+ "sealed_path": sealedPath,
"body_print": bodyPrint,
"bytes": len(body),
}).Error; err != nil {
return err
}
default:
+ rechunk = true
d := Document{
- Source: source,
+ Source: spec.Source,
ExternalID: external,
BodyPrint: bodyPrint,
Revision: 1,
SealedTitle: sealedTitle,
SealedBody: sealedBody,
+ SealedPath: sealedPath,
Bytes: len(body),
}
if err := tx.Create(&d).Error; err != nil {
@@ -146,6 +221,9 @@ func (s *Store) Ingest(ctx context.Context, source, key, title, body string) (in
docID = d.ID
}
+ if !rechunk {
+ return nil
+ }
rows := make([]Chunk, 0, len(chunks))
for _, c := range chunks {
// Seal the body ALONE, not Indexable(): the heading has its
@@ -193,8 +271,47 @@ func (s *Store) Ingest(ctx context.Context, source, key, title, body string) (in
return docID, s.indexDocument(post, docID)
}
+// titleOrPathMoved reports whether a document's title or ancestry has
+// changed, by opening what is stored and comparing plaintext.
+//
+// Comparing the sealed blobs would be cheaper and wrong: AES-GCM uses a
+// fresh nonce per seal, so identical plaintext seals to different bytes
+// every time and every document would look moved on every ingest.
+func (s *Store) titleOrPathMoved(d Document, title, pathStr string) (bool, error) {
+ cur, err := s.sealer.Open(seal.PurposeTitle, d.SealedTitle)
+ if err != nil {
+ return false, fmt.Errorf("open title %d: %w", d.ID, err)
+ }
+ if string(cur) != title {
+ return true, nil
+ }
+ var curPath string
+ if len(d.SealedPath) > 0 {
+ p, err := s.sealer.Open(seal.PurposePath, d.SealedPath)
+ if err != nil {
+ return false, fmt.Errorf("open path %d: %w", d.ID, err)
+ }
+ curPath = string(p)
+ }
+ return curPath != pathStr, nil
+}
+
// indexDocument opens a document's chunks and adds them to the index.
func (s *Store) indexDocument(ctx context.Context, docID int64) error {
+ var d Document
+ if err := s.db.G.WithContext(ctx).Where("id = ?", docID).First(&d).Error; err != nil {
+ return err
+ }
+ // The path lives on the DOCUMENT and is stamped onto every one of its
+ // entries, because every passage of a nested page is equally nested.
+ var path string
+ if len(d.SealedPath) > 0 {
+ p, err := s.sealer.Open(seal.PurposePath, d.SealedPath)
+ if err != nil {
+ return fmt.Errorf("open path %d: %w", docID, err)
+ }
+ path = string(p)
+ }
var rows []Chunk
if err := s.db.G.WithContext(ctx).Where("document_id = ?", docID).Order("ordinal").Find(&rows).Error; err != nil {
return err
@@ -215,6 +332,7 @@ func (s *Store) indexDocument(ctx context.Context, docID int64) error {
Ordinal: r.Ordinal,
Heading: string(heading),
Text: string(text),
+ Path: path,
})
}
if len(entries) == 0 {
diff --git a/internal/memoriatest/ancestry_test.go b/internal/memoriatest/ancestry_test.go
new file mode 100644
index 0000000..97ad601
--- /dev/null
+++ b/internal/memoriatest/ancestry_test.go
@@ -0,0 +1,195 @@
+package memoriatest
+
+import (
+ "context"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ memoria "amadan.net/oficina/memoria/internal/memoria"
+)
+
+// A page with ancestry, as a connector would supply it.
+func notionish(title, body string, path ...string) memoria.IngestSpec {
+ return memoria.IngestSpec{
+ Source: "notion", Key: "page-" + title, Title: title, Body: body,
+ Path: path, AllowTitleOnly: true,
+ }
+}
+
+// TestTitleOnlyIsRefusedForPeopleAndAcceptedForConnectors pins the
+// asymmetry AllowTitleOnly exists for.
+//
+// An empty paste is a person's mistake and the useful answer is to refuse
+// it. An empty Notion page is a fact about the source — real workspaces
+// are full of placeholders — and both alternatives to storing it are
+// wrong: failing the run lets one placeholder block an entire import, and
+// skipping makes the page an absence with nothing to explain it.
+//
+// MUTATION-TESTED 2026-09-05: deleting the AllowTitleOnly branch in
+// Store.Ingest makes the connector half fail with "the document is
+// empty"; defaulting AllowTitleOnly to true makes the person half fail.
+// Reverted.
+func TestTitleOnlyIsRefusedForPeopleAndAcceptedForConnectors(t *testing.T) {
+ _, store, _ := newAppWithStore(t)
+ ctx := context.Background()
+
+ if _, err := store.Ingest(ctx, memoria.IngestSpec{
+ Source: "paste", Key: "k", Title: "A title", Body: " ",
+ }); err == nil {
+ t.Error("an empty paste was accepted; a person's empty submission should be refused")
+ }
+
+ id, err := store.Ingest(ctx, notionish("Quarterly planning", "", "Handbook"))
+ if err != nil {
+ t.Fatalf("a title-only Notion page was refused: %v", err)
+ }
+ if id == 0 {
+ t.Fatal("no document id")
+ }
+ // It must be FINDABLE by the only words it has. Storing it with an
+ // empty body would index nothing and hide it.
+ hits, err := store.Search(ctx, "quarterly planning", 10)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(hits) == 0 {
+ t.Error("a title-only page was stored but is not searchable by its title")
+ }
+}
+
+// TestRenameReachesMemoriaWithoutBumpingRevision.
+//
+// Store.Ingest short-circuits when the body fingerprint is unchanged. It
+// used to return there unconditionally, so a page renamed upstream kept
+// its old title in memoria forever — the body never changes on a rename,
+// so nothing else would ever notice. Found by a design review.
+//
+// The revision must NOT move: a citation points at passages, and those
+// have not changed. A revision that moves without the content moving
+// makes every cached authorization stale for no reason.
+//
+// MUTATION-TESTED 2026-09-05: restoring the unconditional `unchanged =
+// true; return nil` makes the title assertion fail. Bumping the revision
+// in the title-only branch makes the revision assertion fail. Reverted.
+func TestRenameReachesMemoriaWithoutBumpingRevision(t *testing.T) {
+ _, store, _ := newAppWithStore(t)
+ ctx := context.Background()
+ const body = "Expenses are reimbursed within thirty days of submission."
+
+ spec := memoria.IngestSpec{Source: "notion", Key: "page-1", Title: "Old name", Body: body, AllowTitleOnly: true}
+ id, err := store.Ingest(ctx, spec)
+ if err != nil {
+ t.Fatal(err)
+ }
+ _, rev0, _, err := store.Meta(ctx, id)
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ spec.Title = "New name"
+ if _, err := store.Ingest(ctx, spec); err != nil {
+ t.Fatal(err)
+ }
+ title, _, err := store.Body(ctx, id)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if title != "New name" {
+ t.Errorf("title is %q after a rename, want %q — the rename never reached memoria", title, "New name")
+ }
+ _, rev1, _, err := store.Meta(ctx, id)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if rev1 != rev0 {
+ t.Errorf("revision moved %d→%d on a rename; the passages did not change", rev0, rev1)
+ }
+}
+
+// TestMoveReachesMemoriaAndReindexes.
+//
+// A page moved to a different parent — or one whose ANCESTOR was renamed
+// — has an unchanged body and an unchanged title. Only the path moves,
+// and the path is what the index scores in its own column, so the index
+// must be rebuilt for that document or the old ancestry stays searchable
+// and the new one never becomes searchable.
+//
+// MUTATION-TESTED 2026-09-05: dropping `sealed_path` from the title/path
+// Updates map leaves the old ancestry findable and fails this. Reverted.
+func TestMoveReachesMemoriaAndReindexes(t *testing.T) {
+ _, store, _ := newAppWithStore(t)
+ ctx := context.Background()
+ const body = "Laptops are ordered through the equipment request form."
+
+ spec := memoria.IngestSpec{
+ Source: "notion", Key: "page-2", Title: "Equipment", Body: body,
+ Path: []string{"Handbook", "Onboarding"}, AllowTitleOnly: true,
+ }
+ if _, err := store.Ingest(ctx, spec); err != nil {
+ t.Fatal(err)
+ }
+ if hits, _ := store.Search(ctx, "onboarding", 10); len(hits) == 0 {
+ t.Fatal("the original ancestry is not searchable, so this test proves nothing")
+ }
+
+ // The page moves under Engineering.
+ spec.Path = []string{"Engineering", "Platform"}
+ if _, err := store.Ingest(ctx, spec); err != nil {
+ t.Fatal(err)
+ }
+ if hits, _ := store.Search(ctx, "platform", 10); len(hits) == 0 {
+ t.Error("the new ancestry is not searchable: the move did not reach the index")
+ }
+ if hits, _ := store.Search(ctx, "onboarding", 10); len(hits) != 0 {
+ t.Error("the OLD ancestry is still searchable after a move: a stale coordinate survives")
+ }
+}
+
+// TestAncestryIsSealedInTheDatabase. A path is content — "Acquisitions ›
+// Project Bluebird" names a thing whether or not anyone opens the page —
+// so it is sealed like a title, under its own purpose.
+//
+// The existing plaintext gate cannot cover this: it drives the HTTP form,
+// and no form carries ancestry.
+//
+// MUTATION-TESTED 2026-09-05: storing pathStr with a plain []byte cast
+// instead of Seal makes this fail. Reverted.
+func TestAncestryIsSealedInTheDatabase(t *testing.T) {
+ _, store, path := newAppWithStore(t)
+ ctx := context.Background()
+ const secret = "PROJECTBLUEBIRD7742"
+
+ if _, err := store.Ingest(ctx, notionish("Deal terms",
+ "The terms were agreed in March and revisited in June.", "Acquisitions", secret)); err != nil {
+ t.Fatal(err)
+ }
+ // Vacuity guard: the ancestry must actually be indexed, or the scan
+ // below is a scan of nothing relevant.
+ if hits, _ := store.Search(ctx, secret, 10); len(hits) == 0 {
+ t.Fatal("the ancestry is not searchable, so it may not have been stored at all")
+ }
+
+ entries, err := os.ReadDir(filepath.Dir(path))
+ if err != nil {
+ t.Fatal(err)
+ }
+ checked := 0
+ for _, e := range entries {
+ if e.IsDir() {
+ continue
+ }
+ raw, err := os.ReadFile(filepath.Join(filepath.Dir(path), e.Name()))
+ if err != nil {
+ t.Fatal(err)
+ }
+ checked++
+ if strings.Contains(strings.ToUpper(string(raw)), secret) {
+ t.Errorf("%s holds the ancestry in plaintext: a path is content", e.Name())
+ }
+ }
+ if checked == 0 {
+ t.Fatal("no files to check; the gate proved nothing")
+ }
+}
diff --git a/internal/memoriatest/harness_test.go b/internal/memoriatest/harness_test.go
index 373458e..8f42ec0 100644
--- a/internal/memoriatest/harness_test.go
+++ b/internal/memoriatest/harness_test.go
@@ -71,6 +71,26 @@ func newAppAt(t *testing.T, path string) (http.Handler, string) {
return mux, path
}
+// newAppWithStore is newApp with the store exposed, for tests about
+// ingestion itself rather than about the pages that call it. Connectors
+// reach the store directly — there is no form that carries a Notion
+// page's ancestry — so the paths they use need a way to be tested.
+func newAppWithStore(t *testing.T) (http.Handler, *memoria.Store, string) {
+ t.Helper()
+ path := filepath.Join(t.TempDir(), "app.db")
+ logger := slog.New(slog.NewTextHandler(io.Discard, nil))
+ d, err := db.Open(path, logger)
+ if err != nil {
+ t.Fatalf("db.Open: %v", err)
+ }
+ t.Cleanup(func() { d.Close() })
+ mux, a, err := memoria.AppForTest(d, testOrigin, testDEK(t), logger)
+ if err != nil {
+ t.Fatalf("AppForTest: %v", err)
+ }
+ return mux, a.Store(), path
+}
+
// newAppWithLog is newApp with the log captured, for the gate that
// asserts no content reaches a log line.
func newAppWithLog(t *testing.T) (http.Handler, *bytes.Buffer) {
diff --git a/tools/seed/main.go b/tools/seed/main.go
index c668eba..2f2a1af 100644
--- a/tools/seed/main.go
+++ b/tools/seed/main.go
@@ -112,7 +112,9 @@ func main() {
title := rel
// The relative path is the stable key, so re-seeding after an
// edit updates the document rather than adding a second copy.
- if _, err := store.Ingest(ctx, *source, rel, title, string(raw)); err != nil {
+ if _, err := store.Ingest(ctx, memoria.IngestSpec{
+ Source: *source, Key: rel, Title: title, Body: string(raw),
+ }); err != nil {
// Fatal, not a skip. An earlier version counted these and
// carried on, so a missing table produced 59 skips and a
// cheerful "0 documents seeded" summary.