oficina / memoria Public

🤖 merge: Notion ingest into main

main is created here at da2e878, the commit the live instance is
actually serving, so this merge is additive to what is running rather
than a rollback of it. There was no main branch before; the deployed
lineage was worktree-oficina-works-switch.

The substantive conflict was the index. Both branches independently
added a THIRD column to the same FTS5 table, both immediately after
body, and both left a comment explaining that snippet() and bm25()
address columns by declared position so body must stay at index 1. They
do not compete — title is a document's own name, path is where it sits
in its source — so the table now carries four columns, heading, body,
title, path, weighted 3.0, 1.0, 1.0, 0.5.

That merge fixes a real gap in the Notion work for free: a page's title
is a property, not a block, so it landed in none of heading, body or
path and a page was not findable by its own name. main had already found
and fixed the same bug for pasted documents.

It also cost a measurement. The ancestry-ranking gate had to be re-taken
with titles indexed, because a title column at body weight is exactly
the ingredient that could reintroduce the trap. Two things changed:

  - The fixture now carries titles, and includes the Handbook page
    itself. That page matching "handbook" on its own name and ranking
    first is CORRECT, so the gate no longer asserts "the body match is
    top". It asserts the property that actually matters — no page which
    matched only through inherited ancestry outranks the page whose body
    answers the question.
  - TestTrapFixtureCanDiscriminate caught its own probe passing three
    bm25 weights against four columns, which SQLite accepts and which
    silently made the path weight inert. That is the test doing exactly
    what it was written for.

Re-measured: with the path column at 0.5 the body match sits at position
1 and the best ancestry-only hit at 2; with ancestry folded into the
heading column the body match falls to position 21, buried under twenty
pages that match nothing but an inherited title.

Also taken from main: ErrIndexLag, the committed-but-unsearchable
outcome, which the sync loop needs — a page whose index refresh failed
is stored, and recording it as a failure would be the refusal this app
never verified. And main's tools/drive -dir flag, which solved the
hardcoded-temp-path problem better than this branch did.

Fixed in passing: [rst-result-path] used --rst-text-mut, an undefined
token with no fallback, copied from the trail rule while it still
carried that typo. main fixed the original and added a test for the
whole class; the merge brought both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VauJkQsPeGRErbWFiQNZzC

Paul Campbell pushed by paul@keymail.dev 1 month ago f8d10a138139babca1a2eec5048790dcdd616ab3 parent da2e878parent 265039f

39 files changed, +5467 −68

.gitignore +3 −0
.impeccable/config.json +17 −0
cmd/memoria/main.go +6 −0
docs/superpowers/specs/2026-09-03-memoria-design.md +41 −4
docs/superpowers/specs/2026-09-05-notion-ingest-design.md +794 −0
internal/memoria/app.go +105 −1
internal/memoria/chunk/chunk.go +44 −9
internal/memoria/connector.go +358 −0
internal/memoria/connector_models.go +103 −0
internal/memoria/delete_test.go +8 −4
internal/memoria/handlers.go +29 −2
internal/memoria/handlers_connector.go +289 −0
internal/memoria/index/index.go +61 −15
internal/memoria/index/path_test.go +227 −0
internal/memoria/migrations/0003_document_path.sql +10 −0
internal/memoria/migrations/0004_create_connectors.sql +18 −0
internal/memoria/migrations/schema.sql +19 −1
internal/memoria/models.go +11 −0
internal/memoria/notion/client.go +332 −0
internal/memoria/notion/client_test.go +353 −0
internal/memoria/notion/fake_test.go +315 −0
internal/memoria/notion/markdown.go +325 −0
internal/memoria/notion/traverse.go +358 −0
internal/memoria/notion/verify.go +38 −0
internal/memoria/render.go +1 −1
internal/memoria/seal/seal.go +15 −0
internal/memoria/static/memoria.css +17 −0
internal/memoria/store.go +169 −28
internal/memoria/sync.go +281 −0
internal/memoria/templates/connectors.html +22 −0
internal/memoria/templates/index.html +6 −0
internal/memoria/templates/layout.html +5 −1
internal/memoria/templates/notion.html +139 −0
internal/memoriatest/ancestry_test.go +195 −0
internal/memoriatest/connector_test.go +620 −0
internal/memoriatest/gates_test.go +48 −0
internal/memoriatest/harness_test.go +20 −0
tools/drive/main.go +62 −1
tools/seed/main.go +3 −1

diff --git a/.gitignore b/.gitignore

index 1d5109f..67d776d 100644

--- a/.gitignore

+++ b/.gitignore

@@ -10,3 +10,6 @@

# The copy-review working directory: a string index, a server log and the

# submitted result. Working artefacts of a review, not source.

/copy-review/

+# impeccable: the shared config is source; its hook cache is not.

+/.impeccable/hook.cache.json

diff --git a/.impeccable/config.json b/.impeccable/config.json

new file mode 100644

index 0000000..e35c975

--- /dev/null

+++ b/.impeccable/config.json

@@ -0,0 +1,17 @@

+ "detector": {

+ "ignoreRules": [],

+ "ignoreFiles": [],

+ "ignoreValues": [

+ {

+ "rule": "side-tab",

+ "value": "*",

+ "files": [

+ "internal/memoria/static/memoria.css"

+ ],

+ "createdAt": "2026-09-05T06:00:00.000Z",

+ "reason": "Paul confirmed intentional. [rst-passage]:target draws a leading accent border on the ONE passage a citation just linked to — it is a transient anchor-target state, not ornament on a static card. Without it the page jumps and nothing says where you landed, which matters because every search result links to a specific passage ordinal. Already resolves through --rst-accent with a token fallback, so it is not a colour literal. Predates this branch; surfaced only because rst-result-path shifted its line number."

+ }

+ ]

+ }

diff --git a/cmd/memoria/main.go b/cmd/memoria/main.go

index 5001c06..5521e3e 100644

--- a/cmd/memoria/main.go

+++ b/cmd/memoria/main.go

@@ -69,6 +69,12 @@ func main() {

logger.Error("wire sign-in", "err", err)

os.Exit(1)

}

+ // The connector routes check this list. In slice 0 it is the same

+ // list the admission gate uses, so every signed-in person is an

+ // owner — requireOwner says that out loud rather than implying a

+ // restriction the app does not yet have.

+ app.SetOwners(cfg.Owners)

app.UseAuth(a)

app.MountAuth(mux, cfg.Origin)

diff --git a/docs/superpowers/specs/2026-09-03-memoria-design.md b/docs/superpowers/specs/2026-09-03-memoria-design.md

index a50f1d9..71f54a0 100644

--- a/docs/superpowers/specs/2026-09-03-memoria-design.md

+++ b/docs/superpowers/specs/2026-09-03-memoria-design.md

@@ -793,9 +793,28 @@ This is the honest resolution of gap 3, and it costs something that must

be said plainly rather than implied: **revoking a Notion page from the

integration does not revoke it in memoria.** The content was published

into memoria and memoria now owns it; removing it is an act performed *in

-memoria*. §5.1's enumeration diff is a convenience that eventually

-notices upstream deletions, not an authorization mechanism, and §3.6's

-claim does not pretend otherwise.

+memoria*. §3.6's claim does not pretend otherwise.

+**Amended 2026-09-05 by the Notion ingest design** (§3 of

+`2026-09-05-notion-ingest-design.md`). This paragraph used to end "§5.1's

+enumeration diff is a convenience that eventually notices upstream

+deletions, not an authorization mechanism". It no longer does, because

+the first connector built under it does not delete on absence at all.

+The reason is that memoria is being adopted as a *replacement* for

+Notion, not a mirror of it: under a replacement, archiving a page

+upstream after migrating it is the intended workflow, and a

+diff-and-delete connector reads that as "forget it" — deleting from the

+app of record the document the migration existed to preserve. A

+convenience that eventually notices deletions is the wrong convenience

+when deletion upstream is the goal.

+The cost is real and is carried rather than buried: a page deleted

+upstream *because it should not have been shared* stays indexed and

+readable by every admitted person, with no notice. Connectors therefore

+record `last_seen_at` per item on every enumeration, so "no longer in

+the source" is answerable later without a migration. Keeping the

+documents is a decision; being unable to list them is not licensed by

+it.

### 4.3 Admission is derived from the siblings

@@ -1026,7 +1045,7 @@ signal.

|---|---|---|---|---|---|---|

| Oficina siblings | ✅ | ✅ | ✅ team-visible mark | ✅ | ✅ tombstone | Full; §4.1 governs reads |

| Google Drive | ✅ | ✅ changes API | ❌ per-file | ✅ | ✅ | Publish-gated; §4.2 governs |

-| Notion | ✅ | ⚠️ `last_edited_time` polling | ❌ | ✅ | ❌ diff only | Publish-gated; §4.2 |

+| Notion | ✅ | ✅ traversal from shared roots | ❌ | ✅ | ❌ **none, and none used** | Publish-gated; §4.2 |

| Files | content hash | ❌ | ❌ | ✅ | ❌ | Upload only; §4.2 |

@@ -1042,6 +1061,24 @@ partially-paginated or credential-expired enumeration returns *no

information*, not *no documents*; treating the second as the first purges

the corpus silently. Gate 7.

+**Amended 2026-09-05, two changes from building the Notion connector.**

+*Notion's feed.* This table said `⚠️ last_edited_time polling`, which

+assumed `/v1/search` could enumerate a workspace. Notion documents that

+it cannot: search "is not guaranteed to return everything", and **no

+field in the response says the result was partial**. An enumerator built

+on it reports a complete import while missing pages — this design's worst

+failure shape, reachable with no error anywhere. Connectors enumerate by

+*traversal* from the roots the integration was explicitly shared with

+(search is documented as complete for those alone), which is

+cursor-verifiable and yields ancestry as a side effect.

+*Notion's delete.* Now "none, and none used" — see the amendment in §4.2.

+Gate 7's rule above stands unchanged for connectors that *do* delete;

+Notion's connector satisfies it trivially by having no path that deletes,

+which is a stronger position than a completeness predicate that has to be

+right every time.

### 5.2 Deletion, and the hard case

Chunks and vectors are rows under memoria's DEK; they go on the tombstone

diff --git a/docs/superpowers/specs/2026-09-05-notion-ingest-design.md b/docs/superpowers/specs/2026-09-05-notion-ingest-design.md

new file mode 100644

index 0000000..e62b6b1

--- /dev/null

+++ b/docs/superpowers/specs/2026-09-05-notion-ingest-design.md

@@ -0,0 +1,794 @@

+# 🤖 memoria — Notion ingest

+**Status:** draft 2. Draft 1 reviewed by `claude-fable-5-1`, REQUEST

+CHANGES, 15 findings — §1.1 records what it changed, including three

+blockers and one finding that removed the enumeration design entirely.

+**Parent:** `2026-09-03-memoria-design.md`. Where this disagrees with the

+parent, the parent wins and this document is wrong — fix it in the same

+commit. §3 does exactly that: it changes the parent, and says so.

+This is the first half of the parent's **slice 3** (§11, external

+connectors), under the parent's §4.2: content memoria owns outright,

+because there is no sibling to ask.

+---

+## 0. The one rule, as it applies here

+The parent's rule is *memoria never speaks a passage without asking

+whoever owns it whether this person may see it, right now.* For Notion

+there is nobody to ask, and the parent already says so (§4.2): **memoria

+is the app of record for what it ingests from Notion.** The sharing act

+at the far end was the publish. §12 makes sure the product says this

+rather than implying otherwise.

+The second rule is unchanged, and this document adds to what it covers: a

+Notion page's **title, body, headings and ancestry path are all content**

+and all sealed. Its page id is a coordinate — sealed reversibly, and

+separately fingerprinted for lookup (§8, and §1.1 finding 1 for why draft

+1's version could not work at all).

+---

+## 1. Settled decisions

+| # | Decision | Because |

+|---|---|---|

+| 1 | Notion ships **without** slices 1 and 2 | §4.2 needs no sibling contract, no person ref, no model. §2 |

+| 2 | Enumerate by **traversal from shared roots**, never by `/v1/search` | Notion documents that search "is not guaranteed to return everything". §6 — this is the finding that rebuilt the design |

+| 3 | A sync **never deletes on absence** | memoria is replacing Notion; a page vanishing upstream is the expected end state. **This changes the parent** — §3 |

+| 4 | One Notion page → one memoria document | The page is the unit people share, name and link to. §5 |

+| 5 | Ancestry is carried, in **its own index column at low weight** | Nesting is how a workspace is navigated; the existing heading column is 3×-weighted and would rank the whole workspace above the answer. §7 |

+| 6 | `Notion-Version: 2026-03-11`, pinned as a constant with a gate | The object model changed incompatibly at 2025-09-03 (data sources). An unpinned client is one release from silent extraction loss. §1.1 finding 5 |

+| 7 | The credential is sealed under its own HKDF purpose | Parent §3.5, verbatim: "so a chunk-store disclosure does not yield them" |

+| 8 | The run is admin-triggered and resumable, not scheduled | The app sleeps. §9 says what that costs and refuses to hide it |

+| 9 | **One package may reach the network**, host-pinned at runtime | The existing gate forbids all egress; the fix is to narrow it to something stronger, never to delete it. §11 |

+| 10 | Ship in **three stages**, ranking first | Stage 1 re-ranks every existing document and deserves its own review. §14 |

+### 1.1 What the Fable review changed

+Recorded rather than quietly folded in, in the parent's §1.2–§1.4 style.

+Findings 1, 2, 3 were blockers; finding 4 was the expensive one.

+- **The ledger could not fetch anything (blocker).** Draft 1 stored only

+ a *keyed* page id, and `seal.Fingerprint` is HMAC-SHA256

+ (`seal/seal.go:106`) — one-way. Phase 2 was specified as

+ `GET /v1/blocks/{page_id}/children` over ledger rows, so a resumed run

+ held 1,930 fingerprints and no URL to call. "Resumption is free and

+ needs no separate mechanism" was true only because the mechanism was

+ absent. Now: a **sealed** page id (`PurposeCoordinate`, reversible)

+ plus the print for lookup.

+- **Search cannot enumerate, and there is no signal when it fails to

+ (finding 4, and it removed the design).** Draft 1 promised a total

+ decoder: "every enumeration either yields every item or fails." Notion's

+ own documentation says the search endpoint is not optimized for

+ "exhaustively enumerating through all the documents that a bot has

+ access to" and that **"search is not guaranteed to return everything"**,

+ with no field in the response saying so. A clean run to

+ `has_more: false` can be missing pages, and draft 1 would have reported

+ a complete import. That is the codebase's worst failure shape produced

+ by the choice of endpoint, and **gate N1 would have stayed green while

+ the property it named was violated** — a guard written to pass, at the

+ architecture level rather than the test level. §6 replaces it with

+ traversal. The reviewer proposed a `request_status` field as the fix;

+ that field is not in the documentation and the real situation is worse

+ than the finding, which is why §6 is a redesign and not a decoder patch.

+- **Single-flight wedged the connector on the normal path (blocker).**

+ The run is a detached goroutine; when CARLOS sleeps the app mid-fetch

+ the goroutine dies without writing anything and the row stays

+ `fetching`. Nothing in draft 1 ever wrote `interrupted`, so every later

+ sync was refused as "already running", and §12's "interrupted at 412 of

+ 1,930" copy had no writer. Draft 1 described the deadlock as a feature

+ ("survives a restart that left a run marked active"). Now: boot-time

+ reclaim plus a heartbeat, §9.

+- **Gate N3 could not go red (finding 7, the best one).** FTS5's `bm25()`

+ sums per-column contributions, so a row matching in *both* path and

+ body outranks a path-only match under any positive weights. Draft 1's

+ fixture stayed green with the mutation applied — the gate written to

+ catch §7's scoring trap could not catch it. §13 specifies the fixture

+ that reproduces the trap and requires the flip to be **measured**, not

+ assumed.

+- **Renames and moves never reached memoria.** `store.go:113` short-

+ circuits on an unchanged body print without touching `sealed_title`,

+ so a renamed page keeps its old title forever; an *ancestor* rename

+ does not change a descendant's `last_edited_time` at all, so the fetch

+ phase skipped it before `Ingest` was even called. And the ledger had

+ one edit-time column, so the phase-2 comparison compared it against

+ itself. §10.

+- **Empty pages could not be ingested.** `store.go:73` refuses an empty

+ body; real workspaces are full of title-only pages and placeholders.

+ Draft 1's gate N2 asserted an empty body "ingests" and would have

+ failed before its mutation was applied. §5.

+- **A page that failed to fetch had nowhere to be recorded**, so three

+ unreadable pages inside 1,930 rendered as a complete import. §8 gives

+ every ledger row a state and an error code.

+- **"Owners only" named a distinction slice 0 does not have.**

+ `MEMORIA_OWNERS` is the entire admission list (`config.go:65`,

+ `auth.go:28`); today "owners" ≡ "signed in". Worse, `AppForTest`

+ bypasses `RequireSession`, so no identity reaches the request context

+ and the gate could not be written at all. §12.

+- **Decision 3 contradicts the parent, and draft 1 claimed it agreed.**

+ Draft 1 cited parent §3.6 as proof. §3 now argues it as a change and

+ amends the parent in this commit, which is what this document's own

+ header requires.

+- Smaller: `Retry-After` applies to **529** as well as 429; a Notion

+ 404 body quotes the page id *and* the connection name, which is a

+ concrete reason for §8's code-not-message rule; the sync route must be

+ mounted inside the CSRF and session group; `total` needs the codebase's

+ `-1` sentinel; there is no disconnect affordance though parent §3.6

+ promises one; `context.WithoutCancel` in `store.go:189` is a bounded

+ post-commit refresh, not a precedent for a multi-minute job.

+### 1.2 What building it changed

+Draft 2 was written before the code. Five things did not survive contact,

+and they are recorded here rather than quietly diverged from.

+- **One pass, not two phases.** §9 specified enumerate-then-fetch. With

+ traversal they are the same API call — fetching a page returns its body

+ *and* the child pages under it — so running them separately would

+ double every request against a limit of three per second. The loop is

+ one pass; the ledger is still written per page.

+- **Therefore the interruption cost is larger than §9 claimed, and the

+ progress display is different.** "The cost of an interruption is the

+ page in flight" is false: the *ingest* work already done survives, but

+ a resumed run repeats the traversal, because a page's children cannot

+ be discovered without asking and a child's own edit does not change its

+ parent's edit time. And since the total is only known when the walk

+ ends, the status page cannot say "412 of 1,930" mid-run. It says how

+ many pages have been seen so far and that the total is not yet known —

+ `total` stays at the `-1` sentinel until the run completes.

+- **There was no way to rotate a token.** §12 said "rotation is replacing

+ the row" and the page offered no form to do it with; the only route was

+ disconnect and reconnect. Found by mutation-testing

+ `TestTokenIsNeverRendered`, which was passing because the page it

+ looked at had no token field at all.

+- **Ancestry was indexed and never shown.** Search results rendered the

+ in-page heading trail and not the source path, so two pages called

+ "Week one" under different handbooks were indistinguishable to the

+ reader. Carrying ancestry for findability while hiding it from the

+ person reading the answer is half a feature. Found by looking at the

+ rendered page, which is the only thing that would have found it.

+- **The scheme pin became its own check.** §11 described a host pin. The

+ implementation also rejected non-loopback plaintext hosts, and that

+ second condition caught the host-pin gate's test target — so deleting

+ the host pin left the gate green. Host and scheme are now separate

+ fields with separate checks and separate gates.

+Two smaller ones: `tools/drive` wrote to a hardcoded absolute path under

+one person's home directory and panicked anywhere else, now the OS temp

+dir; and the connector page ordered token rotation above import status,

+which put a rare housekeeping act between the reader and the thing the

+page is for.

+---

+## 2. Why this is unblocked, when the parent sequenced it fourth

+The parent's §11 puts external connectors in slice 3, behind the sibling

+contract (slice 1) — which parent §13 records as "blocked on agreement,

+not on code", in *other repositories*. That ordering was never a

+dependency. Slice 3's stated requirements are least-privilege credentials

+(parent §3.5) and gate 7, both local. Notion content is memoria-owned

+under §4.2, so:

+- **No S2 call.** There is no sibling that could answer one.

+- **No person ref**, so no dependency on home §13 step 4.

+- **No content egress.** Bodies come *in*. The footer's claim — "no

+ document ever leaves this instance" — stays true and

+ `TestFooterDoesNotOverclaim` stays green.

+Draft 1 said "Notion sits on slice 0 and nothing else", which the review

+corrected: it also sits on slice 0's **test infrastructure**, because

+`TestNoOutboundNetworkInAppCode` forbids the client this design needs

+(§11), and on `AppForTest` growing an identity seam (§12). Those are

+dependencies. They are in this repository, which is the point.

+---

+## 3. Deletion: this changes the parent

+**Custody** is the integration's share list. An internal integration sees

+only what is shared with it, and that act of sharing is the publish

+(parent §5). memoria adds no second selector — parent §3.5's

+"owner-selected subsets" is satisfied by the far end already having a

+per-page control, and a second list to keep in step with a list is the

+shape decision 11 rejected.

+**Authorization** is memoria's own admission gate and nothing else

+(§4.2). Every admitted person sees every Notion document memoria holds.

+**On absence, memoria does nothing.**

+| Upstream event | memoria does |

+|---|---|

+| Page edited, renamed, or moved | re-ingest (§10 — all three, which draft 1 got wrong) |

+| Page added under a shared root | ingest |

+| Page unshared, trashed, or the token revoked | **nothing.** The document stands until someone deletes it *in memoria* |

+**This is a change to the parent, not an application of it.** Parent §4.2

+says "§5.1's enumeration diff is a convenience that eventually notices

+upstream deletions", and parent §5.1's Notion row records `Delete: diff

+only`. Draft 1 removed the diff and cited parent §3.6's README paragraph

+as evidence the parent already agreed. It is not evidence: §3.6 is about

+*disconnecting a space or revoking access*, a different event from a

+single page archived while the connection stays live. The review was

+right to call that smuggled.

+**The argument for the change, made properly.** memoria is being adopted

+as the replacement for Notion, not as a mirror of it. Under a

+replacement, archiving a page in Notion *after* migrating it is the

+intended workflow — and a diff-and-delete connector reads that as

+"forget it", deleting from the app of record the document the migration

+existed to preserve. A convenience that eventually notices deletions is

+the wrong convenience when deletion upstream is the goal.

+**What it costs, and the mitigation.** A page deleted upstream *because

+it should not have been shared* stays indexed and readable by every

+admitted person, with no notice. That is a real cost and §12 puts it in

+the interface. Every ledger row therefore carries **`last_seen_at`**,

+written on every enumeration — it costs nothing, since traversal already

+touches every row — so the "no longer in Notion" view is buildable later

+without a migration. Decision 3 keeps the documents; it does not license

+being unable to list them.

+**Parent amendments required in this commit:** §4.2's "eventually notices

+upstream deletions" and §5.1's Notion `Delete` cell. Not doing so leaves

+two specs that contradict, which is the drift CLAUDE.md exists to

+prevent.

+---

+## 4. The Notion object model we are coding against

+Pinned at **`Notion-Version: 2026-03-11`**, a constant in the client with

+a gate asserting it. This matters more than a pin usually does: version

+`2025-09-03` changed the object model incompatibly. A **database** now

+contains one or more **data sources**; a row's parent is

+`{type: "data_source_id", data_source_id, database_id}`;

+`/v1/databases/{id}/query` is replaced by `/v1/data_sources/{id}/query`.

+`2026-03-11` further renames `archived` → `in_trash` and the

+`transcription` block type → `meeting_notes`. Draft 1 described the

+pre-2025 model and listed pinning as an *open item*; it is a decision

+with consequences for every parent link the ancestry walk reads.

+Trashed content is skipped: `in_trash: true` is not ingested, and an

+already-ingested page that goes to trash is an absence, which by decision

+3 means nothing happens.

+---

+## 5. What a Notion page becomes

+One page → one `Document`, `Source = "notion"`.

+**Body** is Markdown extracted from the block tree: paragraph,

+`heading_1..3` → `#`/`##`/`###`, bulleted and numbered list items,

+`to_do` → `- [ ]`/`- [x]`, `quote`, `callout`, `toggle` (heading line

+then children), `code` (fenced, language preserved), `divider`, `table`

+(pipe table), `image`/`file`/`video`/`pdf`/`meeting_notes` (caption and

+filename only — memoria indexes text), and `child_page`/`child_database`

+as a one-line reference, **never inlined**: they are their own documents

+and inlining would duplicate every passage into all of its ancestors.

+Rich text uses Notion's `plain_text`, so an inline equation or mention

+degrades to the text Notion itself renders.

+**Unknown block types produce nothing and increment a counter.** Not an

+error — Notion adds block types, and a connector that fails a run on an

+unrecognised one breaks on Notion's release schedule. But not silence

+either: the run reports "N blocks of a type memoria does not read", so a

+page that came in thin has a reason attached rather than looking like a

+thin page.

+**Empty pages are ingested as title-only documents.** Draft 1 said an

+empty page "ingests as an empty body"; `store.go:73` and `:101` refuse

+exactly that, so the gate asserted something the store forbids. Real

+workspaces are full of placeholders and title-only rows, and both

+available alternatives were wrong: failing the run lets one placeholder

+block the whole import, and skipping makes the page an absence with no

+row to explain it. So `Store.Ingest` grows an explicit

+`AllowTitleOnly` option — off for uploads and paste, where an empty

+submission is a user error worth refusing, on for connectors, where it is

+a fact about the source. The document's body is its title; it chunks to

+one passage; the ledger row records `empty` and the run counts it.

+**Databases.** A database is not a document; each **row is a page** and

+becomes one, with properties rendered as a leading key/value block:

+```

+Status: Shipped

+Owner: Ana

+Due: 2026-03-01

+<the row page's blocks>

+```

+`relation` renders the related pages' titles, `rollup` and `formula`

+their computed value. **This is the part most likely to be wrong** — a

+40-row database becomes 40 one-paragraph documents, which may be noise

+rather than memory. §15 records it as unanswerable without a real

+workspace.

+**Title** is the page's title property. A database row with an empty

+title gets its data source's name plus its first non-empty property,

+because `firstLine(body)` over a property block titles every document

+`Status: Shipped`.

+**External id** is the Notion page id — stable across edits, renames and

+moves, which is what `Document.ExternalID`'s contract requires. It is

+keyed through `seal.Fingerprint` like every other external id (parent

+§3.2: a coordinate must not be an oracle).

+---

+## 6. Enumeration: traversal, not search

+**Draft 1 enumerated with `/v1/search` and promised a total decoder. It

+cannot be done.** Notion's documentation is explicit that search "is not

+optimized for … exhaustively enumerating through all the documents that a

+bot has access to", that **"search is not guaranteed to return

+everything"**, and that its index is eventually consistent. Crucially

+there is *no field in the response that says the result was partial*. So

+a run that pages cleanly to `has_more: false`, with no 429 and no error,

+may simply be missing pages — and would report a complete import. No

+decoder, however total, can detect it.

+The same documentation supplies the way out:

+> any pages or databases that are **directly shared** with a connection

+> are **guaranteed to be returned**

+So search is reliable for exactly one thing, and that is what we use it

+for:

+1. **Roots.** `POST /v1/search` finds the directly-shared pages and data

+ sources — guaranteed complete for *that* set, which is the set an

+ admin chose.

+2. **Traversal.** From each root, walk down: `GET /v1/blocks/{id}/children`

+ for `child_page` and `child_database`, and

+ `POST /v1/data_sources/{id}/query` for rows. Both are cursor-paginated

+ with a proper `has_more`/`next_cursor` contract, and both are

+ *complete* — they enumerate a parent's actual children, not an index's

+ opinion of them.

+**Ancestry falls out for free.** You arrive at every page *from* its

+parent, so the path is known at the moment of fetch. Draft 1 walked

+`parent` links upward, which needed raw ids of pages possibly outside the

+run, had to hop `block_id` parents for inline child pages, and truncated

+silently when an ancestor was unshared. None of that exists now.

+**The decoder is total** — the parent's §4.1 rule, applied where it can

+actually hold, because these endpoints support it:

+- `has_more: true` with no `next_cursor` → error.

+- A repeated cursor → error. (A looping API would otherwise produce a run

+ that never finishes and never says why.)

+- A short page with `has_more: true` → accepted; the *cursor* is the

+ completeness signal, not the count.

+- `429`/`529` past the retry budget → error. Both carry `Retry-After` in

+ integer seconds and both are honoured exactly; `500/502/503/504` are

+ retried only because every request here is a read.

+- An id seen twice → deduplicated **and counted**; silent dedup hides a

+ traversal bug. A cycle (a page reachable from itself) is bounded by the

+ visited set and counted the same way.

+There is no partial return anywhere in the package's surface: a caller

+cannot mistake "we got some" for "that is all there is", because "some"

+is not a value the package produces.

+**What this still does not guarantee**, said plainly: memoria enumerates

+everything reachable from the roots the integration was given. A page

+shared with nobody, or in a space the integration cannot see, is not

+missing — it was never in scope. The status page says how many roots were

+found and names them, so "I expected more" is answerable.

+---

+## 7. Ancestry, and the scoring trap in it

+A Notion page's meaning is largely positional: *"Week 1"* under

+*"Handbook › Onboarding"* is onboarding week 1; the same page floating

+free is nothing. memoria's chunk trail today comes only from ATX headings

+inside one document, so a naive import loses all of it.

+Each document therefore carries a **path**: the ancestor titles from the

+root down, known from traversal (§6).

+**The trap.** The index has two FTS5 columns scored

+`bm25(chunks, 3.0, 1.0)` — heading 3×, body 1× (`index.go:165`).

+Prepending the path onto the heading trail puts the workspace's top-level

+names into **every chunk of every page beneath them at triple weight**. A

+search for *handbook* returns the whole handbook above the page that

+answers the question, and it gets worse with depth. The symptom is bad

+ranking, not an error, which is the kind that survives.

+**Therefore:** a third FTS column, `path`, scored

+`bm25(chunks, 3.0, 1.0, 0.5)`. `chunk.Split` is unchanged; a sibling

+`chunk.SplitUnder(path, text)` carries the path without merging it into

+the document's own trail. `Store.Ingest` moves to an options struct —

+one production call site (`handlers.go:183`) and one dev tool

+(`tools/seed/main.go:115`).

+**Ancestry is also SHOWN**, above the title on every search result, not

+merely indexed. A result carrying only its in-page heading trail leaves

+two "Week one" pages under different handbooks indistinguishable to the

+reader (§1.2).

+**The 0.5 is a guess**, chosen as half the body's weight because nothing

+has measured it, and the code says so where the constant lives. §15 keeps

+it open. What is *not* a guess is that the gate under it must actually

+fail when the weighting is wrong — see §13 N3, which is where draft 1's

+version fell down.

+---

+## 8. Data

+Additive-only migrations, per AGENTS.md.

+**`Connector`** — `kind` (`"notion"`), `sealed_token`

+(`PurposeCredential`), `workspace_print`, `last_verified_at`,

+timestamps.

+**`SyncRun`** — `connector_id`, `state`

+(`enumerating|fetching|done|failed|interrupted`), `roots`, `total`,

+`fetched`, `skipped`, `empty`, `failed_pages`, `unread_blocks`,

+`error_code`, `heartbeat_at`, timestamps. `total` is `-1` until

+enumeration completes, following `handlers.go:62`'s sentinel — a total

+that could not be determined is not zero pages.

+**`error_code` is a code, never a message**, and Notion supplies the

+concrete reason: a 404 body reads *"Could not find database with ID:

+be907abe-… Make sure the relevant pages and databases are shared with

+your connection \"My Connection\""* — a page id and a connection name,

+one `%w` away from a log line. The closed set is `unauthorized`,

+`rate_limited`, `traversal_failed`, `extract_failed`, `network`,

+`interrupted`; the UI maps it to copy as `flashFor` already does.

+**`NotionPage`** — the ledger, keyed per **connector**, not per run,

+which is what makes a second run resume rather than restart:

+| Column | Why |

+|---|---|

+| `page_print` | keyed page id — lookup, and joins to `Document.ExternalID` |

+| `sealed_page_id` | the id itself, sealed under `PurposeCoordinate`. **Reversible**, because the fetch phase needs the id back (§1.1 finding 1) |

+| `sealed_path` | the ancestor trail at enumeration time |

+| `path_print` | keyed path — lets a move or an ancestor rename be detected without opening the row |

+| `seen_edited_time` | what the last enumeration saw |

+| `ingested_edited_time` | what memoria actually ingested. **Two columns, because one compares against itself** (§1.1) |

+| `error_code` | per row, so one bad page is not a whole bad run |

+| `last_seen_at` | §3's mitigation |

+| `document_id` | the row it produced |

+**`Document.SealedPath`** under a new `seal.PurposePath`. A path is

+content: *"Acquisitions › Project Bluebird"* names a thing whether or not

+anyone opens the page. Sealing it under the title's purpose would be

+cheaper and is refused — one HKDF purpose per kind of thing sealed.

+New purposes: `PurposePath`, `PurposeCredential`, `PurposeCoordinate`.

+No request body is bound onto a GORM model; the connector form goes

+through an explicit `map[string]any` and a `.Select` allowlist. Neither

+`scope.Owned` nor `scope.OwnedBy` appears — these rows are team-shared,

+which is AGENTS.md's first mechanical rule.

+---

+## 9. The run loop, and the sleep it cannot hide

+An owner posts to `/connectors/notion/sync`, **mounted inside the

+`csrf.Protect` + `requireSession` group** (`app.go:51,60`) — a route on

+the outer mux bypasses both, as `/static/` does. The handler starts the

+run and redirects to a status page that refreshes with

+`<meta http-equiv="refresh">`. Scriptless, per AGENTS.md.

+**One pass, checkpointed per page.** Draft 2 specified two phases,

+enumerate then fetch. With traversal they are the same API call —

+fetching a page returns its body *and* the child pages beneath it — so

+separating them would double every request against a limit of three per

+second (§1.2).

+The loop walks from the roots (§6); for each page it writes the ledger

+row first (coordinate, ancestry, `seen_edited`, `last_seen`), then

+fetches, then ingests if §10 says the page needs work, then records the

+outcome (`state`, `ingested_edited`, `ingested_path`).

+Writing the row BEFORE the fetch is deliberate: a page that then fails to

+read has somewhere to be named, instead of being absent.

+**What an interruption actually costs.** The ingest work already done

+survives — a resumed run skips every page that is unchanged. The

+traversal is repeated, because children cannot be discovered without

+asking for them, and a child's own edit does not move its parent's edit

+time. So a resume is cheap in the expensive currency (chunking, sealing,

+indexing) and not free in API calls.

+**Single-flight, and the deadlock draft 1 shipped.** A run holds the

+connector via a conditional update. Draft 1 called the lock's survival

+across a restart a feature; it is the bug. The goroutine dies when CARLOS

+sleeps the process, having written nothing, and the row stays `fetching`

+— so every later sync is refused as "already running", permanently, on

+the *normal* path. Two mechanisms fix it:

+- **At boot, any run in `enumerating` or `fetching` becomes

+ `interrupted`.** There can be no live run at boot, so this is sound by

+ construction rather than by timeout.

+- **A `heartbeat_at` written every N pages**, so a run wedged inside a

+ *live* process is reclaimable after a stated interval rather than

+ never.

+**What the sleep costs, said out loud.** CARLOS sleeps an idle app; a

+sleeping app syncs nothing. At ~3 requests/second a large workspace is

+tens of minutes, so runs will be interrupted. The status page says the

+sync progresses while the page is open and resumes where it stopped.

+And the honest naming, which draft 1 fudged: this is **a request-spawned

+goroutine that runs until the platform sleeps the process, kept alive by

+an open status tab** — not a background worker, and `context.WithoutCancel`

+here is not the same thing as `store.go:189`'s bounded post-commit index

+refresh. It also sits against parent §3.5's "memoria hibernates": an open

+status tab deliberately prevents that, for as long as it is open. §15

+records the scheduler that would replace it.

+---

+## 10. Re-ingest: edits, renames, and moves

+Draft 1 handled only edits, and silently. Three conditions mark a ledger

+row as needing work:

+1. `seen_edited_time != ingested_edited_time` — the page changed.

+2. `path_print` changed — the page moved, **or an ancestor was renamed**.

+ An ancestor rename does not touch a descendant's `last_edited_time` at

+ all, so without this the descendants keep a stale path forever.

+3. `state` is `pending` or `failed`.

+And in the store: `Ingest`'s unchanged-body short-circuit

+(`store.go:113`) currently returns without touching `sealed_title`, so a

+renamed page keeps its old title indefinitely. It must **update title and

+path when only those changed, without bumping the revision** — a revision

+that moves without the content moving would make every cached

+authorization stale for no reason, which is the comment already at that

+line, and it stays true.

+---

+## 11. Egress, and the gate that forbids it

+`internal/memoriatest/gates_test.go:92`'s

+`TestNoOutboundNetworkInAppCode` walks `internal/memoria/**` and `cmd/**`

+and fails on `*http.Client`, `http.NewRequest`, `net.Dial` and more. It is

+slice 0's enforcement of the parent's AI-off floor. **A Notion client

+trips it on its first line, and draft 1 did not mention it** — leaving

+the implementer's cheapest move to be widening the gate, with no rule for

+how far.

+The rule: **the gate is narrowed to something stronger, never deleted.**

+- The source grep keeps its blanket ban over `internal/memoria/**` and

+ `cmd/**`, with exactly one exemption: `internal/memoria/notion`.

+- That exemption is paid for by a **runtime** guard, which is stronger

+ than the grep it replaces there. The package's client is built on a

+ transport that **refuses any host but `api.notion.com`** and **refuses

+ to follow redirects at all**. Both matter: the token travels in an

+ `Authorization` header, and while Go drops that header across hosts, a

+ *same-host* redirect keeps it, and a base URL built from stored input

+ would send it anywhere. The base URL is a constant; nothing

+ configurable feeds it.

+- Gate N8 asserts both, and gate N9 asserts no *other* package acquired a

+ client.

+**Leak list additions** to parent §3.2:

+- memoria now holds a Notion token, sealed under its own purpose. The

+ parent's live-host entry already names "§5.1's connector credentials";

+ this makes it real rather than prospective.

+- memoria now dials `api.notion.com`, so Notion learns the instance's

+ egress address and sync cadence. No content of the team's goes *out* —

+ the traffic is Notion's own content coming in — but the parent lists

+ the sign-in dial for carrying no content either, and the same standard

+ applies.

+- The ledger leaks a page count and an edit-time distribution from the

+ bucket.

+The AI-off floor is unchanged: with the assistant off, no document of the

+team's leaves the instance, at ingest or at query.

+---

+## 12. Credentials, privilege, and surfaces

+**The token** is a Notion internal integration token, pasted by an owner,

+sealed under `PurposeCredential`, never rendered back — not even masked,

+because a masked secret is still a secret in the DOM. Verified on save

+with one `/v1/users/me` call, so a bad token fails at the form rather

+than mid-run. Rotation is replacing the row through a form on

+the connector page; there is no key history, because a replaced token has

+no use. **Disconnect removes the row** —

+parent §3.6 promises "disconnecting" and draft 1 gave no affordance for

+it. Disconnecting deletes the credential and stops future syncs; by

+decision 3 it deletes no documents, and the confirmation says so in those

+words.

+**Least privilege** (parent §3.5, "a requirement, not a nicety"): a

+read-only internal integration with no user information, shared with the

+specific pages to import rather than the workspace root. The setup copy

+says this because the connector cannot enforce it — Notion decides what a

+token can do, and memoria can only ask for the right one and refuse to

+pretend otherwise.

+**"Owner" is `Config.Owners`, and in slice 0 that is the whole admitted

+set.** Draft 1 wrote "owners only" as though it were a restriction; it

+is not one today, and stating it as a gate while it gates nothing is the

+pattern AGENTS.md opens with. So: the routes are guarded by an explicit

+`requireOwner` that reads the session identity and checks

+`Config.Owners`; it is *currently equivalent* to `requireSession` and the

+code says so at the definition. To make it testable at all,

+**`AppForTest` grows an injectable identity** — today it bypasses

+`RequireSession` entirely, so no identity reaches the context and no such

+gate could be written. §15 records that slice 1, which replaces the

+owners list with S7, must supply an owner predicate or these routes lose

+their guard.

+**Surfaces.** `/connectors` lists connectors; `/connectors/notion` is

+setup, status and history. Scriptless, attribute grammar, colour tokens,

+`theme.css` linked. Three things the interface must say, each a place

+this design could mislead:

+1. **"memoria keeps what it imports."** On the setup page, before the

+ token is entered — parent §3.6 has the wording.

+2. **Interrupted is not progress, and a running total is not a total.**

+ A run stopped by a sleep says so and offers resume. A run still going

+ reports how many pages it has SEEN so far and states that the total is

+ not known until it finishes — because with one pass (§1.2) the

+ denominator does not exist until the walk ends. `total` holds `-1`

+ until then and is never rendered as 0.

+3. **A count that could not be determined says so** — `total` is `-1`

+ until enumeration finishes, and renders as "counting", never 0.

+Plus the run's honest totals: roots found, pages fetched, skipped, empty,

+**failed**, and blocks not understood. Copy goes through the copy-review

+gate before it is written to a file.

+---

+## 13. Gates

+Every gate is written, then **broken so it goes red, then reverted** —

+per path, with the mutation and date recorded in the test comment.

+**N1 — `TestTraversalIncompleteFailsTheRun`.** A fake whose

+`blocks/{id}/children` returns `has_more: true` with no `next_cursor`,

+and one that 429s past budget. Run ends `failed` with `traversal_failed`;

+the status page does not say complete. *Mutation: return accumulated

+results with a nil error on retry exhaustion.*

+**N2 — `TestSearchIsNotUsedAsAnEnumerator`.** The traversal must not

+depend on search returning descendants: a fake whose `/v1/search` returns

+**only the roots** while children exist below must still import every

+child. *Mutation: enumerate from search results instead of traversing.*

+This is the gate that pins §6's whole reason for existing, and draft 1

+had nothing like it.

+**N3 — `TestAncestryCannotOutrankContent`.** Draft 1's version could not

+fail: BM25 sums per-column contributions, so a row matching path *and*

+body beats a path-only match under any positive weights. The fixture that

+reproduces the real trap: **20 pages under a root titled "Handbook", none

+of whose bodies contain "handbook"**, and **one page under "Engineering"

+whose body contains it once**. Query "handbook"; the Engineering page must

+rank first. *Mutation: move the path into the heading column at 3.0.*

+**The flip must be measured and both orderings recorded in the test

+comment** — if the mutation does not actually flip the order, the fixture

+is wrong and must be strengthened until it does. A gate for a ranking

+property is worthless unless the ranking has been observed to change.

+**N4 — `TestTitleOnlyPageIsIngested`.** A page with no blocks becomes a

+title-only document, the ledger row is `empty`, and the run counts it.

+*Mutation: drop `AllowTitleOnly`, so `Ingest` refuses and the run fails.*

+**N5 — `TestUnreadBlocksAreCountedNotSwallowed`.** *Mutation: drop the

+counter increment.*

+**N6 — `TestAFailedPageIsNamedNotAbsent`.** One page of 5 404s. The run

+completes, `failed_pages` is 1, the row is `failed` with a code, and the

+status page does not read as a complete import. *Mutation: `continue` on

+fetch error.*

+**N7 — `TestSyncNeverDeletesOnAbsence`.** Two pages ingested, then a

+traversal returning one. Both documents survive and the run is `done` —

+absence is not an error either. *Mutation: add the diff-and-delete draft

+1's predecessor proposed.*

+**N8 — `TestNotionClientDialsOnlyNotionAndRefusesRedirects`.** The

+transport rejects any other host, and refuses a 302 even to the same

+host. *Mutations: change the base URL constant; enable redirects.*

+**N9 — `TestOnlyTheNotionPackageReachesTheNetwork`.** The narrowed source

+grep, with the exemption asserted to be exactly one directory.

+*Mutation: add an `http.Client` to `internal/memoria/handlers.go`.*

+**N10 — `TestRenameAndMoveReachMemoria`.** Rename a page; sync; the new

+title renders. Rename an *ancestor*; sync; the descendant's path updates

+though its own `last_edited_time` never changed. *Mutation: compare only

+`seen_edited_time`.*

+**N11 — `TestResumeContinuesRatherThanRestarts`** and

+**N12 — `TestInterruptedRunDoesNotWedgeTheConnector`.** Mark a run

+`fetching`, rebuild the app (a boot), POST sync: a new run starts and the

+old is `interrupted`. *Mutations: scope the ledger by run id; remove the

+boot-time reclaim.*

+**N13 — `TestNoNotionContentInLogs`** extends gate 4 to the new

+packages: no title, body, path, page id or token in any log line, error

+string or metric label, observed over a run driven end to end against the

+fake — whose error bodies carry a canary title and a canary page id, so

+the error path is actually exercised. Parent §13 notes the existing gate

+4's credential half cannot fail; this one is built not to repeat that.

+*Mutation: log the page title at Info in the fetch loop.*

+**N14 — `TestTokenIsNeverRendered`.** With a token stored, the connector

+page's HTML contains neither the whole token nor any 8-character run of

+it. (Draft 1 said "no substring", which every single character satisfies.)

+*Mutation: render it into the input's `value`.*

+**N15 — `TestNotionVersionIsPinned`.** The client sends

+`Notion-Version: 2026-03-11`. *Mutation: drop the header.*

+**And the run is looked at in a browser** via `tools/drive`, per

+AGENTS.md — which found three bugs in its first use that the whole Go

+suite passed over. A template test proves markup exists, not that a

+sync's progress reads sensibly at 40%.

+---

+## 14. Sequencing: three stages, not one

+The review's last finding, and it is right — draft 1 was three pieces of

+work wearing one spec.

+**Stage 1 — ranking.** The third FTS column, `SplitUnder`, the `Ingest`

+options struct, `SealedPath`, `AllowTitleOnly`. No Notion code at all.

+This re-ranks **every existing document**, so it lands and is reviewed on

+its own, with N3's flip measured before anything depends on it.

+**Stage 2 — the client.** `internal/memoria/notion`: transport, host

+pin, total decoder, traversal, Markdown extraction, the fake server. Pure,

+no DB, no app types. Gates N1, N2, N8, N9, N15.

+**Stage 3 — the connector.** Credential, ledger, run loop, boot reclaim,

+surfaces, copy. Gates N4–N7, N10–N14.

+---

+## 15. Open

+- **memoria cannot replace Notion yet, and this connector does not change

+ that.** It has no editing surface. This moves the corpus, which has to

+ happen first under any plan, but until someone can write in memoria,

+ Notion remains where new pages are made. Recorded so the migration is

+ not mistaken for the replacement.

+- **No real workspace has been touched.** Every test runs against a fake

+ built from the documentation. AGENTS.md's first broken rule is about

+ guards that cannot fail for the thing they name, and a connector

+ verified only against fixtures its author also wrote is that risk in a

+ different costume — the review agreed and asked for it to be promoted.

+ **A run against the real workspace is a ship requirement**, owned by

+ Paul, blocking stage 3's merge and nothing earlier.

+- **The 0.5 path weight is unmeasured** (§7), and N3 measures only that

+ the ordering flips, not that 0.5 is the right number. Tune against a

+ real workspace and a real query set.

+- **Database rows as documents may be noise** (§5). One look at a real

+ import settles it.

+- **The scheduler.** §9's open status tab is a workaround for having no

+ background worker, and it defeats hibernation while open. Whether

+ CARLOS offers a scheduler is parent §13's open question about scratch

+ storage in a second costume; if it does, decision 8 changes.

+- **Slice 1 must supply an owner predicate** (§12), or `/connectors`

+ loses its guard when the owners list is replaced by S7.

+- **Listing what is no longer in Notion.** Decision 3 keeps such

+ documents and §3 stores `last_seen_at` so the view is buildable; the

+ view itself is not built.

+- **Review pass on draft 2.** Draft 1: `claude-fable-5-1`, REQUEST

+ CHANGES, 15 findings, §1.1. Draft 2 is unreviewed.

diff --git a/internal/memoria/app.go b/internal/memoria/app.go

index e4e8ec6..e444810 100644

--- a/internal/memoria/app.go

+++ b/internal/memoria/app.go

@@ -4,9 +4,11 @@ import (

"context"

"log/slog"

"net/http"

+ "strings"

"github.com/go-chi/chi/v5"

+ "github.com/carlosframework/rastrillo/auth"

"github.com/carlosframework/rastrillo/csrf"

"github.com/carlosframework/rastrillo/db"

"github.com/carlosframework/rastrillo/migrate"

@@ -45,7 +47,23 @@ func App(d *db.DB, origin string, sealer *seal.Sealer, logger *slog.Logger) (*ht

}

logger.Info("index rebuilt", "passages", indexed)

- a := &app{db: d, logger: logger, store: store}

+ // Reclaim runs that were in flight when the process last stopped.

+ //

+ // Sound by construction rather than by timeout: a process that has

+ // just started cannot be running a sync, so anything still marked

+ // in flight was killed by the platform sleeping the app. Without

+ // this the connector wedges permanently on its NORMAL path — the

+ // goroutine dies without writing, the row stays "fetching", and

+ // every later sync is refused as already running.

+ reclaimed, err := store.ReclaimRuns(context.Background())

+ if err != nil {

+ return nil, nil, err

+ }

+ if reclaimed > 0 {

+ logger.Info("reclaimed interrupted syncs", "runs", reclaimed)

+ }

+ a := &app{db: d, logger: logger, store: store, syncer: NewSyncer(store, logger)}

r := chi.NewRouter()

r.Use(csrf.Protect(origin))

@@ -63,6 +81,18 @@ func App(d *db.DB, origin string, sealer *seal.Sealer, logger *slog.Logger) (*ht

r.Post("/documents", a.createDocument)

r.Get("/documents/{id}", a.showDocument)

r.Post("/documents/{id}/delete", a.deleteDocument)

+ // Connector routes sit INSIDE this group, so they inherit

+ // csrf.Protect and requireSession. A route mounted on the outer

+ // mux (as /static/ is) would bypass both.

+ r.Group(func(r chi.Router) {

+ r.Use(a.requireOwner)

+ r.Get("/connectors", a.connectors)

+ r.Get("/connectors/notion", a.notionConnector)

+ r.Post("/connectors/notion", a.saveNotionConnector)

+ r.Post("/connectors/notion/sync", a.startNotionSync)

+ r.Post("/connectors/notion/disconnect", a.disconnectNotion)

+ })

})

mux := http.NewServeMux()

@@ -112,9 +142,72 @@ func (a *app) requireSession(next http.Handler) http.Handler {

})

}

+// requireOwner guards the connector routes.

+//

+// "Owner" is Config.Owners, and IN SLICE 0 THAT IS THE ENTIRE ADMITTED

+// SET — MEMORIA_OWNERS is the admission list, so this is currently

+// equivalent to requireSession. It is written as its own middleware

+// anyway, and this comment says the equivalence out loud, because an

+// earlier draft described the connector pages as "owners only" as though

+// that were a restriction. Naming a gate for a distinction the app does

+// not have is how a guard that guards nothing gets shipped.

+//

+// It fails CLOSED: no identity, or an empty owner list, refuses.

+//

+// Slice 1 replaces the owners list with S7 and there will be no list to

+// check. These routes then need an owner predicate from wherever

+// membership lives, or they lose their guard — recorded in the spec's

+// open items, not left to be discovered.

+func (a *app) requireOwner(next http.Handler) http.Handler {

+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

+ addr, ok := a.identity(r)

+ if !ok || !a.isOwner(addr) {

+ // 404, not 403: AGENTS.md's rule, and a 403 would confirm

+ // the route exists to someone who may not use it.

+ http.NotFound(w, r)

+ return

+ }

+ next.ServeHTTP(w, r)

+ })

+// identity is the signed-in address, or false.

+func (a *app) identity(r *http.Request) (string, bool) {

+ if a.testIdentity != "" {

+ return a.testIdentity, true

+ }

+ id, ok := auth.From(r)

+ if !ok {

+ return "", false

+ }

+ return strings.ToLower(strings.TrimSpace(id.Address)), true

+func (a *app) isOwner(addr string) bool {

+ if len(a.owners) == 0 {

+ return false

+ }

+ for _, o := range a.owners {

+ if o == addr {

+ return true

+ }

+ return false

+// SetOwners supplies the owner list. Called by main from Config.

+func (a *app) SetOwners(owners []string) { a.owners = owners }

// Store exposes the store, for gates that must inspect it.

func (a *app) Store() *Store { return a.store }

+// Syncer exposes the syncer, so a test can drive a run against a fake.

+func (a *app) Syncer() *Syncer { return a.syncer }

+// AsPerson sets the identity AppForTest presents, so a gate can check

+// that a non-owner is refused. Test seam only; production never sets it.

+func (a *app) AsPerson(addr string) { a.testIdentity = addr }

// AppForTest builds the app with a stub session so tests can exercise the

// routes. It is exported, obvious, and takes a testing marker so it

// cannot be reached from production by accident: cmd/memoria does not

@@ -125,5 +218,16 @@ func AppForTest(d *db.DB, origin string, sealer *seal.Sealer, logger *slog.Logge

return nil, nil, err

}

a.allowUnauthenticated = true

+ // An identity, and an owner list containing it.

+ //

+ // AppForTest used to bypass RequireSession entirely, which left no

+ // identity in the request context — so a gate on "is this person an

+ // owner" could not have been written at all, only asserted. Found by

+ // a design review.

+ a.testIdentity = TestOwner

+ a.owners = []string{TestOwner}

return mux, a, nil

}

+// TestOwner is the address AppForTest signs in as.

+const TestOwner = "owner@test"

diff --git a/internal/memoria/chunk/chunk.go b/internal/memoria/chunk/chunk.go

index 5311f25..c5846c5 100644

--- a/internal/memoria/chunk/chunk.go

+++ b/internal/memoria/chunk/chunk.go

@@ -45,6 +45,10 @@ type Chunk struct {

// Text is the passage, with the trail prepended when Indexable is

// called. Stored sealed.

Text string

+ // Path is the document's ancestry in its SOURCE, outermost first —

+ // distinct from Trail, which is the heading path inside this one

+ // document. Empty for anything uploaded or pasted.

+ Path []string

}

// Heading returns the trail joined for display and for indexing.

@@ -54,15 +58,7 @@ type Chunk struct {

// join then renders "› Revenue" with a leading separator and nothing

// before it. Found by looking at the page; no test saw it, because every

// fixture started at "#".

-func (c Chunk) Heading() string {

- parts := make([]string, 0, len(c.Trail))

- for _, t := range c.Trail {

- if strings.TrimSpace(t) != "" {

- parts = append(parts, t)

- }

- return strings.Join(parts, " › ")

+func (c Chunk) Heading() string { return joinTrail(c.Trail) }

// Indexable is what the index tokenises: the heading trail followed by

// the body. The trail is repeated into every chunk of a section

@@ -244,3 +240,42 @@ func normalise(s string) string {

}

return s

}

+// Path is a document's ancestry in its source, outermost first, carried

+// alongside the trail rather than merged into it.

+//

+// Kept separate because the two are scored differently and must be: a

+// heading is written inside the document, while ancestry is inherited by

+// every chunk of every document beneath it. index.PathWeight explains

+// what merging them does to ranking.

+func (c Chunk) PathString() string { return joinTrail(c.Path) }

+// SplitUnder is Split for a document that has ancestry. Every chunk

+// carries the same path, because every passage of a nested page is

+// equally nested.

+func SplitUnder(path []string, text string) []Chunk {

+ out := Split(text)

+ if len(path) == 0 {

+ return out

+ }

+ p := make([]string, len(path))

+ copy(p, path)

+ for i := range out {

+ out[i].Path = p

+ }

+ return out

+// joinTrail renders a trail for display and indexing, dropping empty

+// segments. Shared by Heading and PathString so the two cannot drift —

+// an earlier bug rendered "› Revenue" because one side filtered and the

+// other did not.

+func joinTrail(parts []string) string {

+ kept := make([]string, 0, len(parts))

+ for _, t := range parts {

+ if strings.TrimSpace(t) != "" {

+ kept = append(kept, t)

+ }

+ return strings.Join(kept, " › ")

diff --git a/internal/memoria/connector.go b/internal/memoria/connector.go

new file mode 100644

index 0000000..e3913e3

--- /dev/null

+++ b/internal/memoria/connector.go

@@ -0,0 +1,358 @@

+package memoria

+import (

+ "context"

+ "errors"

+ "fmt"

+ "strings"

+ "time"

+ "gorm.io/gorm"

+ "amadan.net/oficina/memoria/internal/memoria/chunk"

+ "amadan.net/oficina/memoria/internal/memoria/seal"

+// Run states.

+const (

+ RunEnumerating = "enumerating"

+ RunFetching = "fetching"

+ RunDone = "done"

+ RunFailed = "failed"

+ RunInterrupted = "interrupted"

+// Ledger row states.

+const (

+ PagePending = "pending"

+ PageFetched = "fetched"

+ PageEmpty = "empty"

+ PageFailed = "failed"

+// heartbeatStale is how long a run may go without a heartbeat before a

+// LIVE process may reclaim it. A run killed by the platform sleeping is

+// reclaimed at boot instead and does not wait for this.

+const heartbeatStale = 10 * time.Minute

+// ErrSyncInFlight is returned when a run is already active.

+var ErrSyncInFlight = errors.New("a sync is already running")

+// ErrNoConnector is returned when nothing is configured.

+var ErrNoConnector = errors.New("no connector is configured")

+// SaveConnector stores or replaces a connector's credential.

+//

+// Rotation is replacing the row. There is no key history: a replaced

+// token has no remaining use, and keeping one would be a credential

+// retained for no reason.

+func (s *Store) SaveConnector(ctx context.Context, kind, token, workspace string) error {

+ sealed, err := s.sealer.Seal(seal.PurposeCredential, []byte(token))

+ if err != nil {

+ return err

+ }

+ print, err := s.sealer.Fingerprint(seal.PurposeCredential, []byte("workspace\x00"+workspace))

+ if err != nil {

+ return err

+ }

+ now := time.Now().UTC()

+ return s.db.G.WithContext(ctx).Transaction(func(tx *gorm.DB) error {

+ var existing Connector

+ res := tx.Where("kind = ?", kind).First(&existing)

+ if res.Error == nil {

+ // An explicit allowlist, never a bound request body.

+ return tx.Model(&Connector{}).Where("id = ?", existing.ID).Updates(map[string]any{

+ "sealed_token": sealed,

+ "workspace_print": print,

+ "last_verified": now,

+ }).Error

+ }

+ return tx.Create(&Connector{

+ Kind: kind, SealedToken: sealed, WorkspacePrint: print, LastVerified: &now,

+ }).Error

+ })

+// Connector returns a configured connector, without its token.

+func (s *Store) Connector(ctx context.Context, kind string) (*Connector, error) {

+ var c Connector

+ if err := s.db.G.WithContext(ctx).Where("kind = ?", kind).First(&c).Error; err != nil {

+ if errors.Is(err, gorm.ErrRecordNotFound) {

+ return nil, ErrNoConnector

+ }

+ return nil, err

+ }

+ return &c, nil

+// ConnectorToken opens a connector's credential. The only place the

+// plaintext token exists outside the notion client.

+func (s *Store) ConnectorToken(ctx context.Context, kind string) (string, error) {

+ c, err := s.Connector(ctx, kind)

+ if err != nil {

+ return "", err

+ }

+ tok, err := s.sealer.Open(seal.PurposeCredential, c.SealedToken)

+ if err != nil {

+ return "", err

+ }

+ return string(tok), nil

+// DeleteConnector removes the credential and the ledger.

+//

+// It deletes NO DOCUMENTS. What memoria imported, memoria keeps — the

+// content was published into memoria and memoria owns it, so removing it

+// is a separate act performed in memoria. The confirmation copy says so

+// in those words rather than leaving it to be discovered.

+func (s *Store) DeleteConnector(ctx context.Context, kind string) error {

+ return s.db.G.WithContext(ctx).Transaction(func(tx *gorm.DB) error {

+ var c Connector

+ if err := tx.Where("kind = ?", kind).First(&c).Error; err != nil {

+ if errors.Is(err, gorm.ErrRecordNotFound) {

+ return nil

+ }

+ return err

+ }

+ if err := tx.Where("connector_id = ?", c.ID).Delete(&SourcePage{}).Error; err != nil {

+ return err

+ }

+ if err := tx.Where("connector_id = ?", c.ID).Delete(&SyncRun{}).Error; err != nil {

+ return err

+ }

+ return tx.Where("id = ?", c.ID).Delete(&Connector{}).Error

+ })

+// StartRun opens a run, or refuses because one is already active.

+//

+// Single-flight is a conditional insert rather than a mutex, so it

+// survives a restart — but "survives a restart" was the bug in an earlier

+// design, not the feature. The run is a goroutine; when the platform

+// sleeps the process it dies without writing anything, leaving the row

+// "fetching" and every later sync refused as already-running, forever, on

+// the NORMAL path. ReclaimRuns fixes that at boot; the heartbeat below

+// covers a run wedged inside a live process.

+func (s *Store) StartRun(ctx context.Context, connectorID int64) (*SyncRun, error) {

+ var run SyncRun

+ err := s.db.G.WithContext(ctx).Transaction(func(tx *gorm.DB) error {

+ var active []SyncRun

+ if err := tx.Where("connector_id = ? AND state IN ?", connectorID,

+ []string{RunEnumerating, RunFetching}).Find(&active).Error; err != nil {

+ return err

+ }

+ for _, a := range active {

+ if a.Heartbeat != nil && time.Since(*a.Heartbeat) > heartbeatStale {

+ if err := tx.Model(&SyncRun{}).Where("id = ?", a.ID).Updates(map[string]any{

+ "state": RunInterrupted,

+ "error_code": "interrupted",

+ }).Error; err != nil {

+ return err

+ }

+ continue

+ }

+ return ErrSyncInFlight

+ }

+ now := time.Now().UTC()

+ run = SyncRun{

+ ConnectorID: connectorID, State: RunEnumerating,

+ Total: -1, StartedAt: now, Heartbeat: &now,

+ }

+ return tx.Create(&run).Error

+ })

+ if err != nil {

+ return nil, err

+ }

+ return &run, nil

+// ReclaimRuns marks every run that was in flight as interrupted.

+//

+// Called at boot, and sound by construction rather than by timeout: a

+// process that has just started cannot be running a sync, so anything

+// still marked in flight was killed by the platform.

+func (s *Store) ReclaimRuns(ctx context.Context) (int, error) {

+ res := s.db.G.WithContext(ctx).Model(&SyncRun{}).

+ Where("state IN ?", []string{RunEnumerating, RunFetching}).

+ Updates(map[string]any{"state": RunInterrupted, "error_code": "interrupted"})

+ return int(res.RowsAffected), res.Error

+// LatestRun returns the most recent run for a connector.

+func (s *Store) LatestRun(ctx context.Context, connectorID int64) (*SyncRun, error) {

+ var run SyncRun

+ err := s.db.G.WithContext(ctx).Where("connector_id = ?", connectorID).

+ Order("id desc").First(&run).Error

+ if errors.Is(err, gorm.ErrRecordNotFound) {

+ return nil, nil

+ }

+ if err != nil {

+ return nil, err

+ }

+ return &run, nil

+// UpdateRun applies an allowlisted patch to a run.

+func (s *Store) UpdateRun(ctx context.Context, runID int64, fields map[string]any) error {

+ allowed := map[string]bool{

+ "state": true, "roots": true, "total": true, "fetched": true,

+ "skipped": true, "empty": true, "failed_pages": true,

+ "unread_blocks": true, "error_code": true, "heartbeat": true,

+ "finished_at": true,

+ }

+ patch := make(map[string]any, len(fields))

+ for k, v := range fields {

+ if !allowed[k] {

+ return fmt.Errorf("update run: field %q is not allowlisted", k)

+ }

+ patch[k] = v

+ }

+ if len(patch) == 0 {

+ return nil

+ }

+ return s.db.G.WithContext(ctx).Model(&SyncRun{}).Where("id = ?", runID).Updates(patch).Error

+// Beat records that a run is still alive.

+func (s *Store) Beat(ctx context.Context, runID int64) error {

+ now := time.Now().UTC()

+ return s.UpdateRun(ctx, runID, map[string]any{"heartbeat": now})

+// LedgerEntry is one page as the sync loop needs it: the coordinate

+// opened, and enough state to decide whether it needs work.

+type LedgerEntry struct {

+ ID int64

+ PageID string

+ Path []string

+ Title string

+ SeenEdited string

+ IngestedEdited string

+ PathPrint string

+ State string

+ DocumentID int64

+// SeeSourcePage records a page the enumeration reached.

+//

+// Upserts on (connector, page print) and writes what enumeration knows:

+// the coordinate, the ancestry, the edit time it SAW. It never touches

+// IngestedEdited — that column belongs to the fetch phase, and collapsing

+// the two into one makes the comparison compare a value against itself

+// and skip every edit.

+func (s *Store) SeeSourcePage(ctx context.Context, connectorID int64, pageID, title string, path []string, edited string) error {

+ print, err := s.sealer.Fingerprint(seal.PurposeCoordinate, []byte("page\x00"+pageID))

+ if err != nil {

+ return err

+ }

+ sealedID, err := s.sealer.Seal(seal.PurposeCoordinate, []byte(pageID))

+ if err != nil {

+ return err

+ }

+ pathStr := chunk.Chunk{Path: path}.PathString()

+ // Through the shared helper, never inline. Two places computing this

+ // fingerprint differently would make every comparison false and

+ // re-ingest the whole workspace on every sync — or make it always

+ // true and never re-ingest anything. Both fail quietly.

+ pathPrint, err := s.pathPrint(path, title)

+ if err != nil {

+ return err

+ }

+ var sealedPath []byte

+ if pathStr != "" {

+ if sealedPath, err = s.sealer.Seal(seal.PurposePath, []byte(pathStr)); err != nil {

+ return err

+ }

+ now := time.Now().UTC()

+ return s.db.G.WithContext(ctx).Transaction(func(tx *gorm.DB) error {

+ var row SourcePage

+ res := tx.Where("connector_id = ? AND page_print = ?", connectorID, print).First(&row)

+ if res.Error == nil {

+ return tx.Model(&SourcePage{}).Where("id = ?", row.ID).Updates(map[string]any{

+ "sealed_page_id": sealedID,

+ "sealed_path": sealedPath,

+ "path_print": pathPrint,

+ "seen_edited": edited,

+ "last_seen": now,

+ }).Error

+ }

+ return tx.Create(&SourcePage{

+ ConnectorID: connectorID, PagePrint: print, SealedPageID: sealedID,

+ SealedPath: sealedPath, PathPrint: pathPrint, SeenEdited: edited,

+ State: PagePending, LastSeen: now,

+ }).Error

+ })

+// PagesNeedingWork returns the ledger rows the fetch phase must visit.

+//

+// Three conditions, and the second is the one an earlier design missed

+// entirely: an ANCESTOR rename does not change a descendant's own edit

+// time, so without comparing the path print a moved or re-parented page

+// keeps stale ancestry forever and nothing upstream ever signals it.

+func (s *Store) PagesNeedingWork(ctx context.Context, connectorID int64) ([]LedgerEntry, error) {

+ var rows []SourcePage

+ if err := s.db.G.WithContext(ctx).

+ Where("connector_id = ?", connectorID).

+ Order("id").Find(&rows).Error; err != nil {

+ return nil, err

+ }

+ out := make([]LedgerEntry, 0, len(rows))

+ for _, r := range rows {

+ if r.State == PageFetched && r.SeenEdited == r.IngestedEdited && r.PathPrint == r.ingestedPathPrint() {

+ continue

+ }

+ id, err := s.sealer.Open(seal.PurposeCoordinate, r.SealedPageID)

+ if err != nil {

+ return nil, fmt.Errorf("open coordinate %d: %w", r.ID, err)

+ }

+ var path []string

+ if len(r.SealedPath) > 0 {

+ p, err := s.sealer.Open(seal.PurposePath, r.SealedPath)

+ if err != nil {

+ return nil, fmt.Errorf("open path %d: %w", r.ID, err)

+ }

+ if s := string(p); s != "" {

+ path = strings.Split(s, " › ")

+ }

+ out = append(out, LedgerEntry{

+ ID: r.ID, PageID: string(id), Path: path,

+ SeenEdited: r.SeenEdited, IngestedEdited: r.IngestedEdited,

+ PathPrint: r.PathPrint, State: r.State, DocumentID: r.DocumentID,

+ })

+ }

+ return out, nil

+// ingestedPathPrint is the path print as of the last successful ingest.

+func (r SourcePage) ingestedPathPrint() string { return r.IngestedPath }

+// MarkPage records the outcome of fetching one page.

+func (s *Store) MarkPage(ctx context.Context, id int64, state, errorCode string, documentID int64, edited, pathPrint string) error {

+ return s.db.G.WithContext(ctx).Model(&SourcePage{}).Where("id = ?", id).Updates(map[string]any{

+ "state": state,

+ "error_code": errorCode,

+ "document_id": documentID,

+ "ingested_edited": edited,

+ "ingested_path": pathPrint,

+ }).Error

+// CountPages returns how many ledger rows a connector holds, and how many

+// are in each terminal state.

+func (s *Store) CountPages(ctx context.Context, connectorID int64) (total, fetched int, err error) {

+ var t, f int64

+ q := s.db.G.WithContext(ctx).Model(&SourcePage{}).Where("connector_id = ?", connectorID)

+ if err := q.Count(&t).Error; err != nil {

+ return 0, 0, err

+ }

+ if err := s.db.G.WithContext(ctx).Model(&SourcePage{}).

+ Where("connector_id = ? AND state IN ?", connectorID,

+ []string{PageFetched, PageEmpty}).Count(&f).Error; err != nil {

+ return 0, 0, err

+ }

+ return int(t), int(f), nil

diff --git a/internal/memoria/connector_models.go b/internal/memoria/connector_models.go

new file mode 100644

index 0000000..c15b0d2

--- /dev/null

+++ b/internal/memoria/connector_models.go

@@ -0,0 +1,103 @@

+package memoria

+import "time"

+// Connector is one configured source outside the suite.

+//

+// The credential lives here, sealed under its own HKDF purpose, so a

+// disclosure of the chunk store does not also yield the key to the source

+// those chunks came from — the spec's §3.5, which calls least-privilege

+// connector credentials a requirement rather than a nicety.

+type Connector struct {

+ ID int64

+ // Kind is "notion". One row per kind.

+ Kind string `gorm:"uniqueIndex;not null"`

+ // SealedToken is the integration token. Never rendered back, not even

+ // masked: a masked secret is still a secret in the DOM.

+ SealedToken []byte `gorm:"not null"`

+ // WorkspacePrint is a keyed fingerprint of the workspace the token

+ // belongs to, so two connectors pointing at one workspace are

+ // detectable without storing which workspace that is.

+ WorkspacePrint string

+ LastVerified *time.Time

+ CreatedAt time.Time

+ UpdatedAt time.Time

+// SyncRun is one import, and the row that makes it resumable.

+type SyncRun struct {

+ ID int64

+ ConnectorID int64 `gorm:"index;not null"`

+ // State is enumerating | fetching | done | failed | interrupted.

+ State string `gorm:"index;not null"`

+ // Roots is how many top-level pages the integration could see, so

+ // "I expected more than that" is an answerable question.

+ Roots int

+ // Total is -1 until enumeration finishes.

+ //

+ // Not 0. The precedent is handlers.go's chunkCount, which returned 0

+ // on error and made the page assert an absence was real when the app

+ // had merely failed to look. A total that could not be determined is

+ // not a total of zero pages.

+ Total int `gorm:"not null;default:-1"`

+ Fetched int

+ Skipped int

+ Empty int

+ FailedPages int

+ UnreadBlocks int

+ // ErrorCode is a CODE, never a message. Notion's own error bodies

+ // carry content — a 404 quotes the page id and the connection name —

+ // and AGENTS.md forbids content in any error string.

+ ErrorCode string

+ // Heartbeat lets a run wedged inside a LIVE process be reclaimed. A

+ // run killed by the platform sleeping is reclaimed at boot instead,

+ // which is sound by construction rather than by timeout: there can be

+ // no live run at boot.

+ Heartbeat *time.Time

+ StartedAt time.Time

+ FinishedAt *time.Time

+// SourcePage is the ledger: one row per page a connector has seen.

+//

+// Keyed per CONNECTOR and not per run, which is the whole reason a second

+// run resumes instead of restarting.

+type SourcePage struct {

+ ID int64

+ ConnectorID int64 `gorm:"index;not null"`

+ // PagePrint is the keyed page id — lookup, and what joins to a

+ // Document's ExternalID.

+ PagePrint string `gorm:"index;not null"`

+ // SealedPageID is the id itself, sealed and REVERSIBLE.

+ //

+ // Both columns exist because an earlier design stored only the

+ // fingerprint, and seal.Fingerprint is HMAC-SHA256: a resumed run

+ // would have held 1,930 fingerprints and no id to fetch with. Found

+ // by a design review before any of it was written.

+ SealedPageID []byte `gorm:"not null"`

+ // SealedPath is the ancestry as enumeration saw it. PathPrint lets a

+ // move — or an ancestor rename — be detected without opening the row.

+ SealedPath []byte

+ PathPrint string

+ // SeenEdited is what the last enumeration saw; IngestedEdited is what

+ // memoria actually ingested. Two columns, because one column compares

+ // against itself and therefore skips every edit.

+ SeenEdited string

+ IngestedEdited string

+ // IngestedPath is the path print as of the last successful ingest.

+ // A row is up to date only when BOTH the edit time and the ancestry

+ // match what was ingested — an ancestor rename moves the second

+ // without moving the first.

+ IngestedPath string

+ // State is pending | fetched | empty | failed.

+ State string `gorm:"index;not null;default:'pending'"`

+ ErrorCode string

+ // LastSeen is written on every enumeration, so "no longer in the

+ // source" stays answerable without a later migration. memoria does

+ // not delete on absence; it does not follow that it should be unable

+ // to say what has gone.

+ LastSeen time.Time

+ DocumentID int64

+ CreatedAt time.Time

+ UpdatedAt time.Time

diff --git a/internal/memoria/delete_test.go b/internal/memoria/delete_test.go

index 4ba5349..fa91176 100644

--- a/internal/memoria/delete_test.go

+++ b/internal/memoria/delete_test.go

@@ -22,8 +22,10 @@ import (

func storeOneDocument(t *testing.T, a *app, term string) int64 {

t.Helper()

ctx := context.Background()

- id, err := a.store.Ingest(ctx, "upload", "regional-notes.txt", "regional notes",

- "The "+term+" terms for the departing regional lead were agreed in March.")

+ id, err := a.store.Ingest(ctx, IngestSpec{

+ Source: "upload", Key: "regional-notes.txt", Title: "regional notes",

+ Body: "The " + term + " terms for the departing regional lead were agreed in March.",

+ })

if err != nil {

t.Fatalf("Ingest: %v", err)

}

@@ -212,8 +214,10 @@ func TestASearchNeverCitesAPassageTheDocumentNoLongerHas(t *testing.T) {

t.Cleanup(func() { scratch.Close() })

live := a.store.ix

a.store.ix = scratch

- again, err := a.store.Ingest(ctx, "upload", "regional-notes.txt", "regional notes",

- "The terms for the departing regional lead were agreed in March.")

+ again, err := a.store.Ingest(ctx, IngestSpec{

+ Source: "upload", Key: "regional-notes.txt", Title: "regional notes",

+ Body: "The terms for the departing regional lead were agreed in March.",

+ })

if err != nil {

t.Fatalf("re-Ingest: %v", err)

}

diff --git a/internal/memoria/handlers.go b/internal/memoria/handlers.go

index fc9fefd..dcbd9bb 100644

--- a/internal/memoria/handlers.go

+++ b/internal/memoria/handlers.go

@@ -1,6 +1,7 @@

package memoria

import (

+ "context"

"encoding/json"

"errors"

"io"

@@ -78,10 +79,29 @@ type app struct {

store *Store

auth *auth.Auth

+ syncer *Syncer

+ // owners is the list requireOwner checks. In slice 0 it is the whole

+ // admission list — see requireOwner.

+ owners []string

// allowUnauthenticated is set only by AppForTest. Production cannot

// set it: cmd/memoria never calls that constructor, and

// TestProductionDoesNotUseTheTestConstructor fails if it starts to.

allowUnauthenticated bool

+ // testIdentity is the address AppForTest presents. Same seam, same

+ // guard: production never sets it.

+ testIdentity string

+ // verifyToken and syncClient let a test drive the connector against

+ // a fake Notion. Both nil in production, where the real client is

+ // built from the stored credential.

+ verifyToken func(ctx context.Context, token string) (string, error)

+ syncClient SourceClient

+// UseFakeSource points the connector at a stand-in Notion for tests.

+// Named for what it is; production never calls it.

+func (a *app) UseFakeSource(c SourceClient, verify func(ctx context.Context, token string) (string, error)) {

+ a.syncClient, a.verifyToken = c, verify

}

// UseAuth attaches auth after construction. App builds the router, and

@@ -735,7 +755,12 @@ func (a *app) createDocument(w http.ResponseWriter, r *http.Request) {

// serve them all, and a title box that silently titled one of

// five would be a worse answer than the filename.

title := strings.TrimSuffix(hdr.Filename, filepathExt(hdr.Filename))

- _, err := a.store.Ingest(r.Context(), "upload", hdr.Filename, title, string(raw))

+ // AllowTitleOnly stays OFF for people. An empty upload is a

+ // mistake and refusing it is the useful answer; connectors turn

+ // it on, because an empty page is a fact about the source.

+ _, err := a.store.Ingest(r.Context(), IngestSpec{

+ Source: "upload", Key: hdr.Filename, Title: title, Body: string(raw),

+ })

switch {

case errors.Is(err, ErrIndexLag):

// The row committed and the index did not take it. Counted as

@@ -765,7 +790,9 @@ func (a *app) createDocument(w http.ResponseWriter, r *http.Request) {

if len(body) > maxUpload {

refused = append(refused, fill(c.ReasonTooBig, "{name}", c.PasteName))

} else {

- _, err := a.store.Ingest(r.Context(), "paste", title, title, body)

+ _, err := a.store.Ingest(r.Context(), IngestSpec{

+ Source: "paste", Key: title, Title: title, Body: body,

+ })

switch {

case errors.Is(err, ErrIndexLag):

a.logger.Error("index refresh failed after commit", "source", "paste", "err", err)

diff --git a/internal/memoria/handlers_connector.go b/internal/memoria/handlers_connector.go

new file mode 100644

index 0000000..c560232

--- /dev/null

+++ b/internal/memoria/handlers_connector.go

@@ -0,0 +1,289 @@

+package memoria

+import (

+ "context"

+ "errors"

+ "net/http"

+ "strings"

+ "time"

+ "amadan.net/oficina/memoria/internal/memoria/notion"

+// connectorPage is what the connector templates render.

+type connectorPage struct {

+ Signed bool

+ Chunks int

+ Flash string

+ FlashKind string

+ Configured bool

+ Verified string

+ // Run is the most recent run, if any.

+ Run *runView

+ // Pages is how many pages the ledger holds, and Imported how many

+ // have actually landed.

+ Pages int

+ Imported int

+ // CountsKnown is false when the counts could not be determined. The

+ // template must then say so rather than render a confident zero.

+ CountsKnown bool

+// runView is one run, rendered.

+type runView struct {

+ State string

+ Running bool

+ Roots int

+ Total int

+ TotalKnown bool

+ Seen int

+ Fetched int

+ Skipped int

+ Empty int

+ Failed int

+ Unread int

+ ErrorCode string

+ Message string

+ Started string

+ Finished string

+// runMessage maps a run to the sentence the page shows.

+//

+// Every branch here exists because the alternative reads as something it

+// is not: an interrupted run is not a finished one, a failed run is not

+// an empty workspace, and a run still going has no total yet.

+func runMessage(r *SyncRun) string {

+ switch r.State {

+ case RunEnumerating, RunFetching:

+ return "Importing now. This runs while this page is open, and picks up where it stopped if it is interrupted."

+ case RunDone:

+ if r.FailedPages > 0 {

+ return "Import finished, but some pages could not be read. They are listed as failed below rather than left out of the count."

+ }

+ return "Import finished."

+ case RunInterrupted:

+ return "The import stopped before it finished — usually because this app went to sleep. Nothing was lost. Start it again to carry on from where it stopped."

+ case RunFailed:

+ return failMessage(r.ErrorCode)

+ }

+ return ""

+// failMessage maps an error CODE to copy. Codes travel, never messages:

+// Notion's own error bodies quote page ids and connection names.

+func failMessage(code string) string {

+ switch code {

+ case "unauthorized":

+ return "Notion refused the token. Check it is still valid and still shared with the pages you want."

+ case "rate_limited":

+ return "Notion asked us to slow down for longer than we waited. Nothing was lost — start the import again."

+ case "traversal_failed":

+ return "The import could not read all of your Notion, so it stopped rather than finish with pages missing. Nothing was lost."

+ case "network":

+ return "We could not reach Notion. Nothing was lost — start the import again."

+ case "interrupted":

+ return "The import stopped before it finished. Start it again to carry on."

+ }

+ return "The import did not finish."

+func (a *app) connectorBase(r *http.Request) connectorPage {

+ return connectorPage{Signed: true, Chunks: a.chunkCount()}

+func (a *app) connectors(w http.ResponseWriter, r *http.Request) {

+ p := a.connectorBase(r)

+ if _, err := a.store.Connector(r.Context(), "notion"); err == nil {

+ p.Configured = true

+ }

+ render(w, "connectors", p)

+func (a *app) notionConnector(w http.ResponseWriter, r *http.Request) {

+ p := a.connectorBase(r)

+ if code := r.URL.Query().Get("err"); code != "" {

+ p.Flash, p.FlashKind = connectorFlash(code), "danger"

+ }

+ if r.URL.Query().Get("saved") != "" {

+ p.Flash, p.FlashKind = "Connected. Start an import when you are ready.", "good"

+ }

+ c, err := a.store.Connector(r.Context(), "notion")

+ if errors.Is(err, ErrNoConnector) {

+ render(w, "notion", p)

+ return

+ }

+ if err != nil {

+ a.logger.Error("read connector", "err", err)

+ p.Flash, p.FlashKind = "We could not read the connector settings.", "danger"

+ render(w, "notion", p)

+ return

+ }

+ p.Configured = true

+ if c.LastVerified != nil {

+ p.Verified = c.LastVerified.Format("2 Jan 2006, 15:04")

+ }

+ // Counts. If either fails, CountsKnown stays false and the template

+ // says the number could not be determined — it never renders 0. The

+ // precedent is chunkCount, which returned 0 on error and made the

+ // page assert an absence was real when the app had failed to look.

+ if total, imported, err := a.store.CountPages(r.Context(), c.ID); err != nil {

+ a.logger.Error("count source pages", "err", err)

+ } else {

+ p.Pages, p.Imported, p.CountsKnown = total, imported, true

+ }

+ run, err := a.store.LatestRun(r.Context(), c.ID)

+ if err != nil {

+ a.logger.Error("read latest run", "err", err)

+ } else if run != nil {

+ p.Run = viewRun(run)

+ }

+ render(w, "notion", p)

+func viewRun(r *SyncRun) *runView {

+ v := &runView{

+ State: r.State, Roots: r.Roots, Fetched: r.Fetched, Skipped: r.Skipped,

+ Empty: r.Empty, Failed: r.FailedPages, Unread: r.UnreadBlocks,

+ ErrorCode: r.ErrorCode, Message: runMessage(r),

+ Running: r.State == RunEnumerating || r.State == RunFetching,

+ Started: r.StartedAt.Format("2 Jan 2006, 15:04"),

+ }

+ // Total is -1 until a run completes. Rendering that as 0 would tell

+ // the reader the workspace is empty while it is being read.

+ if r.Total >= 0 {

+ v.Total, v.TotalKnown = r.Total, true

+ }

+ v.Seen = r.Fetched + r.Skipped + r.Empty + r.FailedPages

+ if r.FinishedAt != nil {

+ v.Finished = r.FinishedAt.Format("2 Jan 2006, 15:04")

+ }

+ return v

+func connectorFlash(code string) string {

+ switch code {

+ case "token":

+ return "That did not look like a Notion integration token."

+ case "verify":

+ return "Notion would not accept that token. Check it is an internal integration token and that it has been shared with at least one page."

+ case "inflight":

+ return "An import is already running."

+ case "save":

+ return "We could not save that."

+ }

+ return "That did not work."

+// saveNotionConnector stores a token after checking Notion accepts it.

+//

+// Verified on save so a bad token fails at the form rather than part-way

+// through an import. The token is never echoed back to the page.

+func (a *app) saveNotionConnector(w http.ResponseWriter, r *http.Request) {

+ if err := r.ParseForm(); err != nil {

+ http.Redirect(w, r, "/connectors/notion?err=save", http.StatusSeeOther)

+ return

+ }

+ token := strings.TrimSpace(r.PostFormValue("token"))

+ if token == "" {

+ http.Redirect(w, r, "/connectors/notion?err=token", http.StatusSeeOther)

+ return

+ }

+ workspace, err := a.verifyNotion(r.Context(), token)

+ if err != nil {

+ // The code, never the error: a Notion error body can quote a

+ // page id and the connection's name.

+ a.logger.Error("verify notion token", "code", notion.CodeOf(err))

+ http.Redirect(w, r, "/connectors/notion?err=verify", http.StatusSeeOther)

+ return

+ }

+ if err := a.store.SaveConnector(r.Context(), "notion", token, workspace); err != nil {

+ a.logger.Error("save connector", "err", err)

+ http.Redirect(w, r, "/connectors/notion?err=save", http.StatusSeeOther)

+ return

+ }

+ http.Redirect(w, r, "/connectors/notion?saved=1", http.StatusSeeOther)

+// verifyNotion is the seam a test replaces; production calls the API.

+func (a *app) verifyNotion(ctx context.Context, token string) (string, error) {

+ if a.verifyToken != nil {

+ return a.verifyToken(ctx, token)

+ }

+ return notion.Verify(ctx, token)

+// startNotionSync begins a run and redirects to the status page.

+func (a *app) startNotionSync(w http.ResponseWriter, r *http.Request) {

+ c, err := a.store.Connector(r.Context(), "notion")

+ if err != nil {

+ http.Redirect(w, r, "/connectors/notion?err=save", http.StatusSeeOther)

+ return

+ }

+ run, err := a.store.StartRun(r.Context(), c.ID)

+ if errors.Is(err, ErrSyncInFlight) {

+ http.Redirect(w, r, "/connectors/notion?err=inflight", http.StatusSeeOther)

+ return

+ }

+ if err != nil {

+ a.logger.Error("start run", "err", err)

+ http.Redirect(w, r, "/connectors/notion?err=save", http.StatusSeeOther)

+ return

+ }

+ // context.WithoutCancel because the run outlives this request — and

+ // unlike Store.Ingest's post-commit index refresh, which is bounded

+ // and quick, this is a job of minutes.

+ //

+ // Say plainly what this is: a request-spawned goroutine that runs

+ // until the platform sleeps the process, kept alive by the status

+ // page being open. It is not a background worker; memoria has none.

+ // An interrupted run is reclaimed at boot and resumes from the

+ // ledger.

+ ctx := context.WithoutCancel(r.Context())

+ go a.runSync(ctx, c.ID, run.ID)

+ http.Redirect(w, r, "/connectors/notion", http.StatusSeeOther)

+func (a *app) runSync(ctx context.Context, connectorID, runID int64) {

+ if a.syncClient != nil {

+ a.syncer.RunWith(ctx, a.syncClient, connectorID, runID)

+ return

+ }

+ a.syncer.Run(ctx, connectorID, runID)

+// disconnectNotion removes the credential and the ledger, and NO

+// DOCUMENTS.

+//

+// What memoria imported, memoria keeps: the content was published into

+// memoria and memoria owns it, so removing it is a separate act

+// performed in memoria. The confirmation copy says that before the

+// button is pressed rather than after.

+func (a *app) disconnectNotion(w http.ResponseWriter, r *http.Request) {

+ if err := a.store.DeleteConnector(r.Context(), "notion"); err != nil {

+ a.logger.Error("disconnect connector", "err", err)

+ http.Redirect(w, r, "/connectors/notion?err=save", http.StatusSeeOther)

+ return

+ }

+ http.Redirect(w, r, "/connectors", http.StatusSeeOther)

+// waitForSync blocks until no run is active. Test seam, exported so it

+// cannot be reached by accident and obvious when it is.

+func (a *app) WaitForSync(connectorID int64, d time.Duration) bool {

+ deadline := time.Now().Add(d)

+ for time.Now().Before(deadline) {

+ run, err := a.store.LatestRun(context.Background(), connectorID)

+ if err == nil && run != nil && run.State != RunEnumerating && run.State != RunFetching {

+ return true

+ }

+ time.Sleep(5 * time.Millisecond)

+ }

+ return false

diff --git a/internal/memoria/index/index.go b/internal/memoria/index/index.go

index d5aa976..07a2dc9 100644

--- a/internal/memoria/index/index.go

+++ b/internal/memoria/index/index.go

@@ -85,8 +85,14 @@ func Open() (*Index, error) {

-- It goes AFTER body deliberately. snippet() and bm25()

-- address columns by declared position, so putting it first

-- would move body from 1 to 2 and silently change which

- -- column gets highlighted.

+ -- column gets highlighted. path goes after title for the

+ -- same reason.

title,

+ -- path is the document's ancestry in its SOURCE — "Handbook

+ -- › Onboarding" for a Notion page nested two deep. Distinct

+ -- from title (this page's own name) and from heading (where

+ -- in this page), and weighted below both: see PathWeight.

+ path,

chunk_id UNINDEXED,

document_id UNINDEXED,

ordinal UNINDEXED,

@@ -116,6 +122,13 @@ type Entry struct {

// Title is the document's title. The same value for every chunk of

// a document, as the heading trail is for every chunk of a section.

Title string

+ // Path is the trail of the document's ANCESTORS in its source —

+ // "Handbook › Onboarding" for a Notion page nested two deep. Empty

+ // for anything uploaded or pasted, which has no ancestry.

+ //

+ // A separate column from Heading rather than a prefix on it, and

+ // from Title rather than folded into it: PathWeight says why.

+ Path string

}

// Add indexes entries in one transaction.

@@ -127,13 +140,13 @@ func (i *Index) Add(entries ...Entry) error {

return err

}

defer tx.Rollback()

- st, err := tx.Prepare(`INSERT INTO chunks (heading, body, title, chunk_id, document_id, ordinal) VALUES (?, ?, ?, ?, ?, ?)`)

+ st, err := tx.Prepare(`INSERT INTO chunks (heading, body, title, path, chunk_id, document_id, ordinal) VALUES (?, ?, ?, ?, ?, ?, ?)`)

if err != nil {

return err

}

defer st.Close()

for _, e := range entries {

- if _, err := st.Exec(e.Heading, e.Text, e.Title, e.ChunkID, e.DocumentID, e.Ordinal); err != nil {

+ if _, err := st.Exec(e.Heading, e.Text, e.Title, e.Path, e.ChunkID, e.DocumentID, e.Ordinal); err != nil {

return err

}

@@ -160,17 +173,42 @@ func (i *Index) Count() (int, error) {

return n, err

}

-// Search runs a BM25 query. The heading column is weighted above the

-// body: a passage under "Revenue" is a better answer to "revenue" than

-// one that mentions the word in passing.

+// Column weights for bm25(), in column order: heading, body, title, path.

+//

+// HeadingWeight above BodyWeight because a passage under "Revenue" is a

+// better answer to "revenue" than one that mentions the word in passing.

-// Title carries the same weight as body, not more. The bug it fixes is

-// that a title term found nothing at all; ranking a title match above a

+// TitleWeight equals BodyWeight, not more. The bug it fixes is that a

+// title term found nothing at all; ranking a title match above a

// document that genuinely discusses the term would be a different and

-// worse answer. Note the consequence, which is the same one the repeated

-// heading trail already has: a title match hits every chunk of that

-// document, so a one-word title query can fill the page with one

-// document. Acceptable while documents are small; revisit with §12.

+// worse answer. Note the consequence, which the repeated heading trail

+// already has: a title match hits every chunk of that document, so a

+// one-word title query can fill the page with one document. Acceptable

+// while documents are small; revisit with §12.

+//

+// PathWeight is BELOW all three, and that is the load-bearing one. A

+// document's ancestry repeats into every one of its chunks — every

+// passage of every page under "Handbook" carries "Handbook" — so at the

+// heading's weight the top of a source's tree outranks the content of

+// it, and a search for "handbook" returns the whole handbook above the

+// page that answers the question. The failure would look like bad

+// search rather than a bug, which is the kind that survives.

+//

+// 0.5 IS A GUESS. It is half the body's weight because nothing has

+// measured it; the corpus it wants tuning against does not exist yet.

+// What is not a guess is the ORDERING, which

+// TestAncestryCannotOutrankContent pins by measurement — and that

+// measurement was re-taken once title joined the table, because a title

+// column at BodyWeight is exactly the ingredient that could reintroduce

+// the trap. See that test for the observed scores.

+const (

+ HeadingWeight = 3.0

+ BodyWeight = 1.0

+ TitleWeight = 1.0

+ PathWeight = 0.5

+// Search runs a BM25 query, weighted by the constants above.

func (i *Index) Search(query string, limit int) ([]Hit, error) {

q := MatchQuery(query)

if q == "" {

@@ -181,14 +219,22 @@ func (i *Index) Search(query string, limit int) ([]Hit, error) {

}

i.mu.RLock()

defer i.mu.RUnlock()

+ // snippet()'s second argument is a COLUMN INDEX, and 1 is body.

+ // The path column is appended third for exactly that reason: adding

+ // it before body would silently start snippeting the wrong column,

+ // and a snippet of the wrong column still renders.

rows, err := i.db.Query(`

SELECT chunk_id, document_id, ordinal, heading,

snippet(chunks, 1, '<mark>', '</mark>', '…', 24),

- bm25(chunks, 3.0, 1.0, 1.0)

+ bm25(chunks, ?, ?, ?, ?)

FROM chunks

WHERE chunks MATCH ?

- ORDER BY bm25(chunks, 3.0, 1.0, 1.0)

- LIMIT ?`, q, limit)

+ ORDER BY bm25(chunks, ?, ?, ?, ?)

+ LIMIT ?`,

+ HeadingWeight, BodyWeight, TitleWeight, PathWeight,

+ q,

+ HeadingWeight, BodyWeight, TitleWeight, PathWeight,

+ limit)

if err != nil {

return nil, err

}

diff --git a/internal/memoria/index/path_test.go b/internal/memoria/index/path_test.go

new file mode 100644

index 0000000..0ff5e65

--- /dev/null

+++ b/internal/memoria/index/path_test.go

@@ -0,0 +1,227 @@

+package index

+import (

+ "fmt"

+ "testing"

+// buildTrap fills an index with the fixture that reproduces the ancestry

+// scoring trap, and returns it.

+//

+// Two things about this fixture are load-bearing, and the first version

+// of it had neither.

+//

+// FIRST, the shape. bm25() SUMS per-column contributions, so a row

+// matching in path AND body outranks a row matching only in path under

+// any positive weights — a fixture where the "good" answer also carries

+// the term in its ancestry proves nothing. The discriminating shape is

+// many pages whose ONLY match is an inherited ancestor title, against

+// one page, outside that subtree, whose BODY answers the query.

+//

+// SECOND, the filler, and this is the subtler one. Without the 200

+// unrelated chunks below, "handbook" appears in EVERY indexed row, BM25's

+// IDF term collapses to about zero, every score comes back as -1e-06, and

+// the resulting order is floating-point noise that happens to look like a

+// ranking. Measured: with no filler every score was -0.000001; with it,

+// the path-only rows score -3.196 at PathWeight and -7.218 with the path

+// folded into the heading column, against -4.961 for the body match. Only

+// the second set of numbers can distinguish a working weight from an

+// inert one.

+//

+// pathAsHeading models the mutation: ancestry folded into the heading

+// column at heading weight, which is what a naive implementation does.

+func buildTrap(t *testing.T, pathAsHeading bool) *Index {

+ t.Helper()

+ ix, err := Open()

+ if err != nil {

+ t.Fatal(err)

+ }

+ t.Cleanup(func() { ix.Close() })

+ var entries []Entry

+ var id int64

+ add := func(docID int64, heading, path, body string) {

+ id++

+ // Title is the document's own name. Every document in the real

+ // index has one indexed at BodyWeight, so a fixture without one

+ // would be measuring a table this app no longer has — and a

+ // title column is exactly the ingredient that could reintroduce

+ // the trap.

+ e := Entry{ChunkID: id, DocumentID: docID, Ordinal: 0, Heading: heading, Text: body, Title: heading}

+ if pathAsHeading {

+ if path != "" {

+ e.Heading = path + " › " + heading

+ }

+ } else {

+ e.Path = path

+ }

+ entries = append(entries, e)

+ }

+ // Filler, so the query term is rare and IDF means something.

+ for i := 0; i < 200; i++ {

+ add(int64(1000+i), fmt.Sprintf("Note %d", i), "Archive › Notes",

+ fmt.Sprintf("Meeting %d discussed procurement, latency budgets and the vendor renewal.", i))

+ }

+ // 20 pages under "Handbook". No body says "handbook"; they match only

+ // through the ancestry they inherited.

+ for i := 0; i < 20; i++ {

+ add(int64(100+i), fmt.Sprintf("Section %d", i), "Handbook › Onboarding",

+ fmt.Sprintf("Week %d covers expenses, travel booking and the equipment request form.", i))

+ }

+ // The Handbook page itself. It matches "handbook" on its own title,

+ // and SHOULD rank — a search for "handbook" returning the handbook is

+ // the right answer. What must not happen is its twenty descendants,

+ // which match only through inherited ancestry, burying real content.

+ add(50, "Handbook", "", "The starting point for everyone new.")

+ // One page elsewhere whose BODY is the actual answer.

+ add(999, "Tooling", "Engineering › Platform",

+ "The handbook is generated from this repository and published nightly.")

+ if err := ix.Add(entries...); err != nil {

+ t.Fatal(err)

+ }

+ return ix

+// TestAncestryCannotOutrankContent is the gate for the trap described on

+// PathWeight: a document's ancestry repeats into every chunk beneath it,

+// so ancestry scored at heading weight buries real answers under the

+// whole subtree that merely sits below a matching ancestor title.

+//

+// MUTATION-TESTED 2026-09-05: folding the path into the heading column

+// (buildTrap's pathAsHeading branch, which is what a naive implementation

+// does) makes this fail — the top three hits become Sections 0, 1 and 2,

+// and the page that answers the query falls below twenty that do not.

+// TestAncestryTrapFlipIsObserved asserts that flip so this gate cannot

+// quietly stop discriminating. Reverted.

+func TestAncestryCannotOutrankContent(t *testing.T) {

+ ix := buildTrap(t, false)

+ hits, err := ix.Search("handbook", 25)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if len(hits) == 0 {

+ t.Fatal("no hits: the fixture matches nothing, so this gate proves nothing")

+ }

+ // The property is NOT "999 is first" — the Handbook page itself

+ // matching its own title and ranking highly is correct. It is that no

+ // page which matched ONLY through inherited ancestry outranks the

+ // page whose body answers the question.

+ var contentRank = -1

+ for i, h := range hits {

+ if h.DocumentID == 999 {

+ contentRank = i

+ break

+ }

+ if contentRank < 0 {

+ t.Fatal("the page whose body answers the query did not match at all")

+ }

+ for i, h := range hits {

+ if i >= contentRank {

+ break

+ }

+ if h.DocumentID >= 100 && h.DocumentID < 200 {

+ t.Errorf("ancestry outranked content: document %d matched only through its "+

+ "inherited ancestry and ranks above the page whose body answers the "+

+ "query (position %d vs %d)", h.DocumentID, i, contentRank)

+ }

+// TestAncestryTrapFlipIsObserved fails if the mutation does NOT change the

+// ranking. A gate for a ranking property is worthless unless the ranking

+// has been seen to move, and this codebase has shipped three gates that

+// could not fail for the thing they named.

+func TestAncestryTrapFlipIsObserved(t *testing.T) {

+ // The measurement is the RELATIVE position of two kinds of hit, not

+ // which document is first. Once titles are indexed the Handbook page

+ // itself is top under either weighting — legitimately, it is called

+ // Handbook — so a top-hit comparison stopped discriminating and this

+ // test said so rather than passing.

+ probe := func(pathAsHeading bool) (content int, ancestry int) {

+ ix := buildTrap(t, pathAsHeading)

+ hits, err := ix.Search("handbook", 30)

+ if err != nil {

+ t.Fatal(err)

+ }

+ content, ancestry = -1, -1

+ for i, h := range hits {

+ if h.DocumentID == 999 && content < 0 {

+ content = i

+ }

+ if h.DocumentID >= 100 && h.DocumentID < 200 && ancestry < 0 {

+ ancestry = i

+ }

+ if content < 0 || ancestry < 0 {

+ t.Fatalf("fixture did not produce both kinds of hit (content=%d ancestry=%d)", content, ancestry)

+ }

+ return content, ancestry

+ }

+ goodContent, goodAncestry := probe(false)

+ badContent, badAncestry := probe(true)

+ t.Logf("path column at %.1f : body-match at %d, best ancestry-only at %d", PathWeight, goodContent, goodAncestry)

+ t.Logf("path in heading : body-match at %d, best ancestry-only at %d", badContent, badAncestry)

+ if goodContent > goodAncestry {

+ t.Errorf("with the path column, an ancestry-only page already outranks the body match "+

+ "(%d vs %d)", goodAncestry, goodContent)

+ }

+ if badAncestry > badContent {

+ t.Errorf("the mutation did NOT change the ranking: with ancestry at heading weight the "+

+ "body match is still above every ancestry-only page (%d vs %d). This fixture cannot "+

+ "detect the trap, so TestAncestryCannotOutrankContent is a gate written to pass. "+

+ "Strengthen the fixture until the order flips.", badAncestry, badContent)

+ }

+// TestTrapFixtureCanDiscriminate guards the FIXTURE, not production code,

+// and it exists because this fixture silently stopped being able to prove

+// anything once already.

+//

+// The first measurement of the trap was taken on a corpus where the query

+// term appeared in every indexed row. BM25's IDF term collapsed to about

+// zero, every score came back as -0.000001, and moving the path weight

+// from 0.5 to 99 changed nothing — yet the rows still came back in an

+// order, and that order looked exactly like a working ranking. A weight

+// that is silently ignored ranks the same as a weight that is wrong, so

+// TestAncestryCannotOutrankContent was passing for a reason unrelated to

+// the property it names.

+//

+// There is no production mutation for this one: it asserts that the

+// corpus and the engine can tell two weightings apart at all. Its

+// mutation is therefore to the fixture.

+//

+// MUTATION-TESTED 2026-09-05: deleting the filler loop from buildTrap

+// makes this fail on the degenerate-IDF assertion. Reverted.

+func TestTrapFixtureCanDiscriminate(t *testing.T) {

+ ix := buildTrap(t, false)

+ score := func(pw float64) float64 {

+ var v float64

+ // Document 100 is one of the ancestry-only matches, so its score

+ // depends on the path weight and nothing else.

+ // FOUR weights, one per column. This probe passed three while the

+ // table had four, which SQLite accepts and which silently made

+ // the path weight inert — caught by this test's own assertion

+ // when the title column landed in a merge.

+ err := ix.db.QueryRow(

+ `SELECT bm25(chunks, ?, ?, ?, ?) FROM chunks WHERE chunks MATCH ? AND document_id = 100`,

+ HeadingWeight, BodyWeight, TitleWeight, pw, MatchQuery("handbook")).Scan(&v)

+ if err != nil {

+ t.Fatal(err)

+ }

+ return v

+ }

+ low, high := score(PathWeight), score(99)

+ t.Logf("ancestry-only row: path weight %.1f → %.4f, path weight 99 → %.4f", PathWeight, low, high)

+ if low == high {

+ t.Fatalf("path weight is inert: %.9f at both %.1f and 99", low, PathWeight)

+ }

+ if low > -1e-4 {

+ t.Fatalf("score %.9f is ~0, so IDF is degenerate and this corpus cannot "+

+ "discriminate a working weight from an inert one", low)

+ }

diff --git a/internal/memoria/migrations/0003_document_path.sql b/internal/memoria/migrations/0003_document_path.sql

new file mode 100644

index 0000000..6aff5ea

--- /dev/null

+++ b/internal/memoria/migrations/0003_document_path.sql

@@ -0,0 +1,10 @@

+-- sealed_path is a document's ancestry in its source — "Handbook ›

+-- Onboarding" for a page nested two deep. Sealed under its own HKDF

+-- purpose, because a path is content: "Acquisitions › Project Bluebird"

+-- names a thing whether or not anyone opens the page.

+--

+-- Nullable rather than NOT NULL DEFAULT: existing rows came from upload

+-- and paste, which have no ancestry, and an empty blob and "no ancestry"

+-- are the same fact here. A DEFAULT would also have to be a valid

+-- sealed blob, which "" is not.

+ALTER TABLE `documents` ADD `sealed_path` blob;

diff --git a/internal/memoria/migrations/0004_create_connectors.sql b/internal/memoria/migrations/0004_create_connectors.sql

new file mode 100644

index 0000000..f20f489

--- /dev/null

+++ b/internal/memoria/migrations/0004_create_connectors.sql

@@ -0,0 +1,18 @@

+CREATE TABLE `connectors` (`id` integer PRIMARY KEY AUTOINCREMENT,`kind` text NOT NULL,`sealed_token` blob NOT NULL,`workspace_print` text,`last_verified` datetime,`created_at` datetime,`updated_at` datetime);

+CREATE UNIQUE INDEX `idx_connectors_kind` ON `connectors`(`kind`);

+CREATE TABLE `sync_runs` (`id` integer PRIMARY KEY AUTOINCREMENT,`connector_id` integer NOT NULL,`state` text NOT NULL,`roots` integer,`total` integer NOT NULL DEFAULT -1,`fetched` integer,`skipped` integer,`empty` integer,`failed_pages` integer,`unread_blocks` integer,`error_code` text,`heartbeat` datetime,`started_at` datetime,`finished_at` datetime);

+CREATE INDEX `idx_sync_runs_state` ON `sync_runs`(`state`);

+CREATE INDEX `idx_sync_runs_connector_id` ON `sync_runs`(`connector_id`);

+CREATE TABLE `source_pages` (`id` integer PRIMARY KEY AUTOINCREMENT,`connector_id` integer NOT NULL,`page_print` text NOT NULL,`sealed_page_id` blob NOT NULL,`sealed_path` blob,`path_print` text,`seen_edited` text,`ingested_edited` text,`ingested_path` text,`state` text NOT NULL DEFAULT "pending",`error_code` text,`last_seen` datetime,`document_id` integer,`created_at` datetime,`updated_at` datetime);

+CREATE INDEX `idx_source_pages_state` ON `source_pages`(`state`);

+CREATE INDEX `idx_source_pages_page_print` ON `source_pages`(`page_print`);

+CREATE INDEX `idx_source_pages_connector_id` ON `source_pages`(`connector_id`);

diff --git a/internal/memoria/migrations/schema.sql b/internal/memoria/migrations/schema.sql

index 8338ecc..e16c256 100644

--- a/internal/memoria/migrations/schema.sql

+++ b/internal/memoria/migrations/schema.sql

@@ -3,11 +3,29 @@

CREATE TABLE `chunks` (`id` integer PRIMARY KEY AUTOINCREMENT,`document_id` integer NOT NULL,`ordinal` integer NOT NULL,`sealed_heading` blob,`sealed_text` blob NOT NULL,`created_at` datetime,CONSTRAINT `fk_documents_chunks` FOREIGN KEY (`document_id`) REFERENCES `documents`(`id`) ON DELETE CASCADE);

-CREATE TABLE `documents` (`id` integer PRIMARY KEY AUTOINCREMENT,`source` text NOT NULL,`external_id` text NOT NULL,`revision` integer NOT NULL DEFAULT 1,`sealed_title` blob NOT NULL,`sealed_body` blob NOT NULL,`bytes` integer,`created_at` datetime,`updated_at` datetime, `body_print` text NOT NULL DEFAULT "");

+CREATE TABLE `documents` (`id` integer PRIMARY KEY AUTOINCREMENT,`source` text NOT NULL,`external_id` text NOT NULL,`revision` integer NOT NULL DEFAULT 1,`sealed_title` blob NOT NULL,`sealed_body` blob NOT NULL,`bytes` integer,`created_at` datetime,`updated_at` datetime, `body_print` text NOT NULL DEFAULT "", `sealed_path` blob);

CREATE INDEX `idx_chunks_document_id` ON `chunks`(`document_id`);

+CREATE UNIQUE INDEX `idx_connectors_kind` ON `connectors`(`kind`);

CREATE INDEX `idx_documents_external_id` ON `documents`(`external_id`);

CREATE INDEX `idx_documents_source` ON `documents`(`source`);

+CREATE INDEX `idx_source_pages_connector_id` ON `source_pages`(`connector_id`);

+CREATE INDEX `idx_source_pages_page_print` ON `source_pages`(`page_print`);

+CREATE INDEX `idx_source_pages_state` ON `source_pages`(`state`);

+CREATE INDEX `idx_sync_runs_connector_id` ON `sync_runs`(`connector_id`);

+CREATE INDEX `idx_sync_runs_state` ON `sync_runs`(`state`);

diff --git a/internal/memoria/models.go b/internal/memoria/models.go

index fc5f1ee..a83be78 100644

--- a/internal/memoria/models.go

+++ b/internal/memoria/models.go

@@ -9,6 +9,9 @@ import "time"

var Models = []any{

&Document{},

&Chunk{},

+ &Connector{},

+ &SyncRun{},

+ &SourcePage{},

}

// Document is one thing memoria holds.

@@ -46,6 +49,14 @@ type Document struct {

// SealedTitle and SealedBody are sealed under distinct HKDF purposes.

SealedTitle []byte `gorm:"not null"`

SealedBody []byte `gorm:"not null"`

+ // SealedPath is the document's ancestry in its source, outermost

+ // first — "Handbook › Onboarding" for a page nested two deep. Empty

+ // for uploads and paste, which have no ancestry.

+ //

+ // Sealed, and under its own purpose: a path is content. It is

+ // indexed in its own low-weighted column rather than folded into

+ // the heading trail, for the reason index.PathWeight gives.

+ SealedPath []byte

// Bytes is the plaintext length, kept so the UI can say how big a

// document is without opening it. A size is already inferable from

// the sealed length, so this leaks nothing new.

diff --git a/internal/memoria/notion/client.go b/internal/memoria/notion/client.go

new file mode 100644

index 0000000..ccbfced

--- /dev/null

+++ b/internal/memoria/notion/client.go

@@ -0,0 +1,332 @@

+// Package notion speaks the Notion API and nothing else.

+//

+// It holds no database handle, no sealer and no app type. That boundary

+// is the testable one: everything here is exercised against a fake HTTP

+// server, and the app layer above never parses JSON.

+//

+// This is the ONLY package in memoria permitted to reach the network.

+// TestNoOutboundNetworkInAppCode forbids it everywhere else and exempts

+// exactly this directory; the exemption is paid for by the host pin

+// below, which is a runtime guard rather than a grep over source.

+package notion

+import (

+ "bytes"

+ "context"

+ "encoding/json"

+ "errors"

+ "fmt"

+ "io"

+ "net"

+ "net/http"

+ "strconv"

+ "time"

+// BaseURL is where the API lives, and it is a CONSTANT.

+//

+// Nothing configurable feeds it. The token travels in an Authorization

+// header, so a base URL built from stored or submitted input would make

+// memoria a token courier for whoever could write that input.

+const BaseURL = "https://api.notion.com"

+// APIVersion is pinned, and the pin is load-bearing rather than tidy.

+//

+// Version 2025-09-03 changed the object model incompatibly: a database

+// now contains data sources, a row's parent is a data_source_id, and

+// /v1/databases/{id}/query was replaced. 2026-03-11 further renamed

+// `archived` to `in_trash` and the `transcription` block type to

+// `meeting_notes`. An unpinned client is one Notion release away from

+// extracting nothing and reporting success.

+const APIVersion = "2026-03-11"

+// Rate limits, from Notion's published request limits: an average of

+// three requests per second per connection, bursts allowed. We pace

+// below the average rather than at it, because the second limit is

+// per-workspace and shared with every other connection the team runs —

+// so the budget we are actually spending is not ours alone.

+const defaultInterval = 350 * time.Millisecond

+// retryBudget bounds retries. On exhaustion the client returns an error;

+// it NEVER returns what it managed to collect. A partial result that

+// looks like a whole one is how an import reports success while missing

+// pages.

+const retryBudget = 5

+// Code is a closed set of error codes. It is a code and never a message

+// because Notion's own error bodies carry content: a 404 reads "Could

+// not find database with ID: be907abe-… shared with your connection "My

+// Connection"", which is a page id and a workspace name one %w away from

+// a log line AGENTS.md forbids.

+type Code string

+const (

+ CodeUnauthorized Code = "unauthorized"

+ CodeRateLimited Code = "rate_limited"

+ CodeTraversalFailed Code = "traversal_failed"

+ CodeExtractFailed Code = "extract_failed"

+ CodeNetwork Code = "network"

+ CodeForbiddenHost Code = "forbidden_host"

+// Error carries a code and an HTTP status. Deliberately no message and

+// no body: a status is a fact about the transport, a body is content.

+type Error struct {

+ Code Code

+ Status int

+func (e *Error) Error() string {

+ if e.Status != 0 {

+ return fmt.Sprintf("notion: %s (http %d)", e.Code, e.Status)

+ }

+ return "notion: " + string(e.Code)

+// CodeOf returns the code of an error, or CodeNetwork.

+func CodeOf(err error) Code {

+ var e *Error

+ if errors.As(err, &e) {

+ return e.Code

+ }

+ return CodeNetwork

+// pinnedTransport refuses every host but the one it was built for.

+//

+// This is the runtime half of the network gate. Go already drops the

+// Authorization header across a cross-host redirect, which is not

+// enough: a SAME-host redirect keeps it, and a base URL that had been

+// tampered with would never be cross-host in the first place. So the

+// check is on every request, not only on redirects — and redirects are

+// refused outright below.

+//

+// The scheme is pinned alongside the host rather than special-cased to

+// "https unless loopback". The special case looked harmless and was not:

+// it rejected a wrong HOST that also happened to be non-loopback, which

+// meant the gate for the host pin passed with the host pin deleted — the

+// neighbouring-error-path failure AGENTS.md opens with. Two fields, two

+// checks, one mutation each.

+type pinnedTransport struct {

+ host string

+ scheme string

+ next http.RoundTripper

+func (p *pinnedTransport) RoundTrip(r *http.Request) (*http.Response, error) {

+ if r.URL.Hostname() != p.host {

+ return nil, &Error{Code: CodeForbiddenHost}

+ }

+ if r.URL.Scheme != p.scheme {

+ return nil, &Error{Code: CodeForbiddenHost}

+ }

+ return p.next.RoundTrip(r)

+// Client is a Notion API client.

+type Client struct {

+ token string

+ base string

+ http *http.Client

+ interval time.Duration

+ last time.Time

+ now func() time.Time

+ sleep func(time.Duration)

+// New builds a client against the real API.

+func New(token string) (*Client, error) {

+ return newClient(token, BaseURL, defaultInterval)

+// NewForTest builds a client against an arbitrary base URL, so the

+// package can be driven by an httptest server.

+//

+// Named so it cannot be reached from production by accident, exactly as

+// memoria.AppForTest is, and held to the same standard: the host pin is

+// still applied — to the test server's host — so the transport's refusal

+// path is the one under test rather than a bypassed one.

+func NewForTest(token, base string) (*Client, error) {

+ return newClient(token, base, 0)

+func newClient(token, base string, interval time.Duration) (*Client, error) {

+ if token == "" {

+ return nil, &Error{Code: CodeUnauthorized}

+ }

+ host, scheme, err := hostOf(base)

+ if err != nil {

+ return nil, err

+ }

+ return &Client{

+ token: token,

+ base: base,

+ interval: interval,

+ now: time.Now,

+ sleep: time.Sleep,

+ http: &http.Client{

+ Timeout: 60 * time.Second,

+ Transport: &pinnedTransport{host: host, scheme: scheme, next: http.DefaultTransport},

+ CheckRedirect: func(*http.Request, []*http.Request) error {

+ // Refuse every redirect. There is no legitimate one on

+ // this API, and following one is the cheapest way to

+ // hand a bearer token to somewhere it was not meant for.

+ return &Error{Code: CodeForbiddenHost}

+ },

+ }, nil

+func hostOf(base string) (host, scheme string, err error) {

+ req, err := http.NewRequest(http.MethodGet, base, nil)

+ if err != nil {

+ return "", "", &Error{Code: CodeForbiddenHost}

+ }

+ h, sc := req.URL.Hostname(), req.URL.Scheme

+ if h == "" || sc == "" {

+ return "", "", &Error{Code: CodeForbiddenHost}

+ }

+ return h, sc, nil

+// pace holds the client below the published rate limit.

+func (c *Client) pace() {

+ if c.interval <= 0 {

+ return

+ }

+ if wait := c.interval - c.now().Sub(c.last); wait > 0 {

+ c.sleep(wait)

+ }

+ c.last = c.now()

+// post issues one API call and decodes into out, retrying only what

+// Notion says is retryable.

+func (c *Client) post(ctx context.Context, path string, body any, out any) error {

+ return c.call(ctx, http.MethodPost, path, body, out)

+func (c *Client) get(ctx context.Context, path string, out any) error {

+ return c.call(ctx, http.MethodGet, path, nil, out)

+func (c *Client) call(ctx context.Context, method, path string, body, out any) error {

+ var payload []byte

+ if body != nil {

+ var err error

+ if payload, err = json.Marshal(body); err != nil {

+ return &Error{Code: CodeExtractFailed}

+ }

+ for attempt := 0; ; attempt++ {

+ c.pace()

+ var rdr io.Reader

+ if payload != nil {

+ rdr = bytes.NewReader(payload)

+ }

+ req, err := http.NewRequestWithContext(ctx, method, c.base+path, rdr)

+ if err != nil {

+ return &Error{Code: CodeNetwork}

+ }

+ req.Header.Set("Authorization", "Bearer "+c.token)

+ req.Header.Set("Notion-Version", APIVersion)

+ if payload != nil {

+ req.Header.Set("Content-Type", "application/json")

+ }

+ res, err := c.http.Do(req)

+ if err != nil {

+ // A pinned-host or redirect refusal is not retryable: it is

+ // a statement about where this request was going.

+ var pe *Error

+ if errors.As(err, &pe) && pe.Code == CodeForbiddenHost {

+ return pe

+ }

+ if ctx.Err() != nil {

+ return &Error{Code: CodeNetwork}

+ }

+ var ne net.Error

+ if attempt < retryBudget && errors.As(err, &ne) {

+ c.backoff(attempt, 0)

+ continue

+ }

+ return &Error{Code: CodeNetwork}

+ }

+ retryable, retryAfter := classify(res.StatusCode, res.Header.Get("Retry-After"))

+ if retryable {

+ res.Body.Close()

+ if attempt < retryBudget {

+ c.backoff(attempt, retryAfter)

+ continue

+ }

+ // Budget exhausted. Return the error — never the partial

+ // work. See retryBudget.

+ code := CodeTraversalFailed

+ if res.StatusCode == http.StatusTooManyRequests || res.StatusCode == 529 {

+ code = CodeRateLimited

+ }

+ return &Error{Code: code, Status: res.StatusCode}

+ }

+ if res.StatusCode == http.StatusUnauthorized || res.StatusCode == http.StatusForbidden {

+ res.Body.Close()

+ return &Error{Code: CodeUnauthorized, Status: res.StatusCode}

+ }

+ if res.StatusCode < 200 || res.StatusCode > 299 {

+ res.Body.Close()

+ return &Error{Code: CodeTraversalFailed, Status: res.StatusCode}

+ }

+ // Cap the body. An unbounded ReadAll on a response we do not

+ // control is a memory bug waiting for a bad day.

+ raw, err := io.ReadAll(io.LimitReader(res.Body, 32<<20))

+ res.Body.Close()

+ if err != nil {

+ return &Error{Code: CodeNetwork}

+ }

+ if out == nil {

+ return nil

+ }

+ if err := json.Unmarshal(raw, out); err != nil {

+ // A body we cannot parse is NOT an empty result. Returning

+ // nil here would turn a malformed page into "no children",

+ // which is the plausible-empty-list failure this codebase

+ // ranks worst.

+ return &Error{Code: CodeTraversalFailed, Status: res.StatusCode}

+ }

+ return nil

+ }

+// classify says whether a status is worth retrying, and for how long.

+//

+// 429 and 529 both carry Retry-After in integer seconds — 529 is

+// Notion's "service_overload" and is documented to be retried the same

+// way as a 429, which is easy to miss because it is not a standard code.

+// 5xx are retried because every request this package makes is a read.

+func classify(status int, retryAfter string) (bool, time.Duration) {

+ switch status {

+ case http.StatusTooManyRequests, 529:

+ if s, err := strconv.Atoi(retryAfter); err == nil && s >= 0 {

+ return true, time.Duration(s) * time.Second

+ }

+ return true, 0

+ case http.StatusInternalServerError, http.StatusBadGateway,

+ http.StatusServiceUnavailable, http.StatusGatewayTimeout:

+ return true, 0

+ }

+ return false, 0

+func (c *Client) backoff(attempt int, after time.Duration) {

+ if c.interval <= 0 && after == 0 {

+ return // tests: no real waiting

+ }

+ if after > 0 {

+ c.sleep(after)

+ return

+ }

+ c.sleep(time.Duration(1<<attempt) * 200 * time.Millisecond)

diff --git a/internal/memoria/notion/client_test.go b/internal/memoria/notion/client_test.go

new file mode 100644

index 0000000..3e4c163

--- /dev/null

+++ b/internal/memoria/notion/client_test.go

@@ -0,0 +1,353 @@

+package notion

+import (

+ "net/http"

+ "net/http/httptest"

+ "strings"

+ "testing"

+func client(t *testing.T, f *fakeNotion) *Client {

+ t.Helper()

+ srv := f.start()

+ c, err := NewForTest("ntn_test", srv.URL)

+ if err != nil {

+ t.Fatal(err)

+ }

+ return c

+// A small workspace: a root with a child and a grandchild, none of which

+// search will admit to except the root.

+func nestedFake(t *testing.T) *fakeNotion {

+ f := newFake(t)

+ f.page("root", "Handbook", "", heading(1, "Welcome"), para("Everything a new starter needs."))

+ f.page("kid", "Onboarding", "root", para("Week one covers expenses and travel booking."))

+ f.page("grandkid", "Equipment", "kid", para("Laptops are ordered through the request form."))

+ return f

+// TestSearchIsNotUsedAsAnEnumerator is the gate for the whole traversal

+// design, and the reason draft 1 of the spec was rewritten.

+//

+// Notion documents that /v1/search "is not guaranteed to return

+// everything" and is "not optimized for exhaustively enumerating through

+// all the documents that a bot has access to" — and it returns NO field

+// saying a result was partial. So an enumerator built on search can page

+// cleanly to has_more:false, miss pages, and report a complete import,

+// with no error anywhere. The fake models exactly that: its search

+// returns only roots.

+//

+// MUTATION-TESTED 2026-09-05: replacing the traversal in walkAll with

+// "use the search results as the page list" finds 1 page instead of 3

+// and fails here. Reverted.

+func TestSearchIsNotUsedAsAnEnumerator(t *testing.T) {

+ f := nestedFake(t)

+ c := client(t, f)

+ refs, _, err := walkAll(t, c)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if len(refs) != 3 {

+ var names []string

+ for _, r := range refs {

+ names = append(names, r.Title)

+ }

+ t.Fatalf("found %d pages %v, want 3 — search returned only the root, so anything "+

+ "missing here was never traversed to", len(refs), names)

+ }

+ byID := map[string]Ref{}

+ for _, r := range refs {

+ byID[r.ID] = r

+ }

+ if got := strings.Join(byID["grandkid"].Path, " › "); got != "Handbook › Onboarding" {

+ t.Errorf("grandchild ancestry is %q, want %q", got, "Handbook › Onboarding")

+ }

+ if got := strings.Join(byID["root"].Path, " › "); got != "" {

+ t.Errorf("root ancestry is %q, want empty", got)

+ }

+// TestTraversalIncompleteFailsTheRun. Every enumeration yields everything

+// or it fails. A partial return that looks whole is how an import reports

+// success while missing pages — this codebase's worst failure shape.

+//

+// MUTATION-TESTED 2026-09-05: returning the accumulated results with a

+// nil error on retry exhaustion, or treating has_more-without-cursor as

+// the end of the list, makes each subtest pass silently. Reverted.

+func TestTraversalIncompleteFailsTheRun(t *testing.T) {

+ t.Run("has_more with no cursor", func(t *testing.T) {

+ f := nestedFake(t)

+ f.hasMoreNoCursor["blocks:root"] = true

+ _, _, err := walkAll(t, client(t, f))

+ if err == nil {

+ t.Fatal("an unfinishable enumeration was reported as complete")

+ }

+ if CodeOf(err) != CodeTraversalFailed {

+ t.Errorf("code %q, want %q", CodeOf(err), CodeTraversalFailed)

+ }

+ })

+ t.Run("repeating cursor", func(t *testing.T) {

+ f := nestedFake(t)

+ f.loopCursor["blocks:root"] = true

+ _, _, err := walkAll(t, client(t, f))

+ if err == nil {

+ t.Fatal("a looping cursor did not terminate as an error")

+ }

+ })

+ t.Run("rate limited past the budget", func(t *testing.T) {

+ f := nestedFake(t)

+ f.alwaysRateLimit["blocks:kid"] = true

+ _, _, err := walkAll(t, client(t, f))

+ if err == nil {

+ t.Fatal("exhausting the retry budget returned no error")

+ }

+ if CodeOf(err) != CodeRateLimited {

+ t.Errorf("code %q, want %q", CodeOf(err), CodeRateLimited)

+ }

+ })

+ // 529 is Notion's "service_overload". It is not a standard status, is

+ // easy to miss, and is documented to be retried exactly like a 429.

+ t.Run("529 service_overload past the budget", func(t *testing.T) {

+ f := nestedFake(t)

+ f.overload["blocks:kid"] = true

+ _, _, err := walkAll(t, client(t, f))

+ if err == nil {

+ t.Fatal("a 529 past the budget returned no error")

+ }

+ if CodeOf(err) != CodeRateLimited {

+ t.Errorf("code %q, want %q — 529 is retried and reported like a 429",

+ CodeOf(err), CodeRateLimited)

+ }

+ })

+// TestClientDialsOnlyNotion is the runtime half of the network gate. It

+// is what pays for internal/memoria/notion being exempt from the source

+// grep that forbids outbound calls everywhere else.

+//

+// The token rides in an Authorization header. Go drops that header across

+// a cross-host redirect, which is not enough on its own: a SAME-host

+// redirect keeps it, and a tampered base URL is never cross-host to begin

+// with. So the pin is checked on every request and redirects are refused

+// outright.

+//

+// MUTATION-TESTED 2026-09-05: returning p.next.RoundTrip unconditionally

+// from pinnedTransport, or returning nil from CheckRedirect, makes the

+// corresponding subtest fail. Reverted.

+func TestClientDialsOnlyNotion(t *testing.T) {

+ // The target here is "localhost" against a server pinned to

+ // "127.0.0.1": a DIFFERENT HOST that is otherwise entirely reachable,

+ // same scheme, same port, live listener.

+ //

+ // The first version of this subtest pointed at http://127.0.0.2:1 and

+ // was worthless. That address failed the transport's *scheme* check

+ // as well, so deleting the host pin left the test green — a

+ // neighbouring error path supplying the failure, which is the exact

+ // pattern AGENTS.md opens with. Found by mutation-testing this gate.

+ t.Run("another host is refused", func(t *testing.T) {

+ f := newFake(t)

+ srv := f.start()

+ c, err := NewForTest("ntn_test", srv.URL)

+ if err != nil {

+ t.Fatal(err)

+ }

+ // Point the client at the same listener by another name, as a

+ // tampered credential row or a mis-set constant would.

+ c.base = strings.Replace(srv.URL, "127.0.0.1", "localhost", 1)

+ if c.base == srv.URL {

+ t.Fatal("could not build a different-host URL; this subtest proves nothing")

+ }

+ if _, err := c.Roots(t.Context()); err == nil {

+ t.Fatal("the client called a host it was not pinned to")

+ } else if CodeOf(err) != CodeForbiddenHost {

+ t.Errorf("code %q, want %q", CodeOf(err), CodeForbiddenHost)

+ }

+ })

+ t.Run("another scheme is refused", func(t *testing.T) {

+ f := newFake(t)

+ srv := f.start()

+ c, err := NewForTest("ntn_test", srv.URL)

+ if err != nil {

+ t.Fatal(err)

+ }

+ c.base = strings.Replace(srv.URL, "http://", "https://", 1)

+ if _, err := c.Roots(t.Context()); err == nil {

+ t.Fatal("the client accepted a scheme it was not pinned to")

+ } else if CodeOf(err) != CodeForbiddenHost {

+ t.Errorf("code %q, want %q", CodeOf(err), CodeForbiddenHost)

+ }

+ })

+ t.Run("a same-host redirect is refused", func(t *testing.T) {

+ var landed bool

+ mux := http.NewServeMux()

+ mux.HandleFunc("/v1/search", func(w http.ResponseWriter, r *http.Request) {

+ http.Redirect(w, r, "/collect", http.StatusFound)

+ })

+ mux.HandleFunc("/collect", func(w http.ResponseWriter, r *http.Request) {

+ // If we get here holding the bearer token, the redirect was

+ // followed and the token went with it.

+ if r.Header.Get("Authorization") != "" {

+ landed = true

+ }

+ writeJSON(w, map[string]any{"object": "list", "results": []any{}, "has_more": false})

+ })

+ srv := httptest.NewServer(mux)

+ defer srv.Close()

+ c, err := NewForTest("ntn_test", srv.URL)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if _, err := c.Roots(t.Context()); err == nil {

+ t.Error("a redirect was followed; the bearer token travels with it")

+ }

+ if landed {

+ t.Error("the token reached the redirect target")

+ }

+ })

+// TestNotionVersionIsPinned. The object model changed incompatibly at

+// 2025-09-03 and again at 2026-03-11; an unpinned client is one release

+// from extracting nothing and calling it success. The fake refuses any

+// request that does not carry the pin.

+//

+// MUTATION-TESTED 2026-09-05: dropping the Notion-Version header makes

+// this fail. Reverted.

+func TestNotionVersionIsPinned(t *testing.T) {

+ f := newFake(t)

+ c := client(t, f)

+ var out map[string]any

+ if err := c.get(t.Context(), "/v1/users/me", &out); err != nil {

+ t.Fatalf("the pinned version was rejected by the API: %v", err)

+ }

+ if APIVersion == "" {

+ t.Fatal("no API version is pinned")

+ }

+// TestUnknownBlocksAreCountedNotSwallowed. Notion adds block types.

+// Failing a run on an unrecognised one breaks memoria on Notion's release

+// schedule; rendering nothing and saying nothing makes a page look thin

+// for no stated reason.

+//

+// MUTATION-TESTED 2026-09-05: deleting the `*unread++` in render's

+// default branch makes this fail. Reverted.

+func TestUnknownBlocksAreCountedNotSwallowed(t *testing.T) {

+ f := newFake(t)

+ f.page("root", "Notes", "",

+ para("A readable paragraph."),

+ unknownBlock("some_future_block"),

+ unknownBlock("another_new_thing"))

+ c := client(t, f)

+ _, got, err := walkAll(t, c)

+ if err != nil {

+ t.Fatal(err)

+ }

+ content := got["root"]

+ if content.UnreadBlocks != 2 {

+ t.Errorf("UnreadBlocks = %d, want 2", content.UnreadBlocks)

+ }

+ if !strings.Contains(content.Markdown, "A readable paragraph.") {

+ t.Error("the readable block was lost alongside the unreadable ones")

+ }

+// TestChildPagesAreNotInlined. A child page is its own document. Inlining

+// it would duplicate every one of its passages into every ancestor, so a

+// search would return the same text several times under different titles.

+func TestChildPagesAreNotInlined(t *testing.T) {

+ f := nestedFake(t)

+ _, got, err := walkAll(t, client(t, f))

+ if err != nil {

+ t.Fatal(err)

+ }

+ if strings.Contains(got["root"].Markdown, "Week one covers expenses") {

+ t.Error("the child page's body was inlined into its parent")

+ }

+ if !strings.Contains(got["kid"].Markdown, "Week one covers expenses") {

+ t.Error("the child page's own body is missing")

+ }

+// TestMarkdownShape covers the block renderings a Notion page actually

+// leans on, because the chunker downstream keys on Markdown headings.

+func TestMarkdownShape(t *testing.T) {

+ f := newFake(t)

+ f.page("root", "Doc", "",

+ heading(1, "Revenue"),

+ para("June was strong."),

+ map[string]any{"object": "block", "id": "t1", "type": "to_do", "has_children": false,

+ "to_do": map[string]any{"checked": true, "rich_text": []map[string]any{{"plain_text": "Ship it"}}}},

+ map[string]any{"object": "block", "id": "c1", "type": "code", "has_children": false,

+ "code": map[string]any{"language": "go", "rich_text": []map[string]any{{"plain_text": "x := 1"}}}},

+ map[string]any{"object": "block", "id": "q1", "type": "quote", "has_children": false,

+ "quote": map[string]any{"rich_text": []map[string]any{{"plain_text": "As agreed."}}}},

+ )

+ _, got, err := walkAll(t, client(t, f))

+ if err != nil {

+ t.Fatal(err)

+ }

+ md := got["root"].Markdown

+ for _, want := range []string{"# Revenue", "June was strong.", "- [x] Ship it", "```go", "x := 1", "> As agreed."} {

+ if !strings.Contains(md, want) {

+ t.Errorf("markdown missing %q\n---\n%s", want, md)

+ }

+// TestDatabaseRowsBecomePagesUnderTheDatabase.

+func TestDatabaseRowsBecomePagesUnderTheDatabase(t *testing.T) {

+ f := newFake(t)

+ f.page("root", "Workspace", "")

+ f.pages["root"].childDBs = []string{"db1"}

+ f.dbs["db1"] = []string{"ds1"}

+ f.dbParent["db1"] = "root"

+ f.page("row1", "Project Bluebird", "")

+ f.page("row2", "Project Kestrel", "")

+ f.roots = []string{"root"} // rows are reached through the database only

+ f.rows["ds1"] = []string{"row1", "row2"}

+ refs, _, err := walkAll(t, client(t, f))

+ if err != nil {

+ t.Fatal(err)

+ }

+ found := map[string][]string{}

+ for _, r := range refs {

+ found[r.Title] = r.Path

+ }

+ for _, title := range []string{"Project Bluebird", "Project Kestrel"} {

+ path, ok := found[title]

+ if !ok {

+ t.Fatalf("database row %q was not reached", title)

+ }

+ if got := strings.Join(path, " › "); got != "Workspace › Projects" {

+ t.Errorf("row %q ancestry is %q, want %q", title, got, "Workspace › Projects")

+ }

+// TestTrashedPagesAreSkipped. `in_trash` replaced `archived` at API

+// version 2026-03-11.

+func TestTrashedPagesAreSkipped(t *testing.T) {

+ f := nestedFake(t)

+ f.pages["kid"].inTrash = true

+ f.roots = []string{"root", "kid"}

+ refs, _, err := walkAll(t, client(t, f))

+ if err != nil {

+ t.Fatal(err)

+ }

+ for _, r := range refs {

+ if r.ID == "kid" && len(r.Path) == 0 {

+ t.Error("a trashed page was admitted as a root")

+ }

diff --git a/internal/memoria/notion/fake_test.go b/internal/memoria/notion/fake_test.go

new file mode 100644

index 0000000..cd5fc93

--- /dev/null

+++ b/internal/memoria/notion/fake_test.go

@@ -0,0 +1,315 @@

+package notion

+import (

+ "encoding/json"

+ "net/http"

+ "net/http/httptest"

+ "strings"

+ "testing"

+// fakeNotion is a Notion workspace built from the published API

+// reference, with switches for each failure the client must survive.

+//

+// Its /v1/search returns ONLY the roots, never the pages nested beneath

+// them. That is not a simplification — it is the behaviour Notion

+// documents ("search is not guaranteed to return everything", "not

+// optimized for exhaustively enumerating"), and modelling it is what

+// makes TestSearchIsNotUsedAsAnEnumerator able to fail.

+type fakeNotion struct {

+ t *testing.T

+ pages map[string]*fakePage

+ roots []string

+ dbs map[string][]string // database id -> data source ids

+ rows map[string][]string // data source id -> row page ids

+ // dbParent is the page a database lives under. Notion sets a child

+ // database's parent to its containing page, and getting this wrong

+ // in the fake made a nested database look like a root — a real

+ // distinction, since Roots drops any result whose parent is also in

+ // the result set, and the root version carries no ancestry.

+ dbParent map[string]string

+ // Failure switches.

+ hasMoreNoCursor map[string]bool // path -> emit has_more with no cursor

+ loopCursor map[string]bool // path -> emit the same cursor forever

+ alwaysRateLimit map[string]bool // path -> 429 with Retry-After: 0

+ notFound map[string]bool // page id -> 404

+ overload map[string]bool // path -> 529

+ calls map[string]int

+ searchLeaks []string // extra ids search returns beyond the roots

+type fakePage struct {

+ id string

+ title string

+ edited string

+ parent string

+ inTrash bool

+ blocks []map[string]any

+ children []string // child_page ids

+ childDBs []string // child_database ids

+func newFake(t *testing.T) *fakeNotion {

+ return &fakeNotion{

+ t: t, pages: map[string]*fakePage{},

+ dbs: map[string][]string{}, rows: map[string][]string{}, dbParent: map[string]string{},

+ hasMoreNoCursor: map[string]bool{}, loopCursor: map[string]bool{},

+ alwaysRateLimit: map[string]bool{}, notFound: map[string]bool{},

+ overload: map[string]bool{}, calls: map[string]int{},

+ }

+func (f *fakeNotion) page(id, title, parent string, blocks ...map[string]any) *fakePage {

+ p := &fakePage{id: id, title: title, parent: parent, edited: "2026-09-01T10:00:00.000Z", blocks: blocks}

+ f.pages[id] = p

+ if parent == "" {

+ f.roots = append(f.roots, id)

+ } else if pp, ok := f.pages[parent]; ok {

+ pp.children = append(pp.children, id)

+ }

+ return p

+func para(text string) map[string]any {

+ return map[string]any{

+ "object": "block", "id": "b-" + text[:min(6, len(text))], "type": "paragraph",

+ "has_children": false,

+ "paragraph": map[string]any{"rich_text": []map[string]any{{"plain_text": text}}},

+ }

+func heading(level int, text string) map[string]any {

+ k := "heading_" + string(rune('0'+level))

+ return map[string]any{

+ "object": "block", "id": "h-" + text[:min(6, len(text))], "type": k,

+ "has_children": false,

+ k: map[string]any{"rich_text": []map[string]any{{"plain_text": text}}},

+ }

+func unknownBlock(kind string) map[string]any {

+ return map[string]any{"object": "block", "id": "u-" + kind, "type": kind, "has_children": false}

+func min(a, b int) int {

+ if a < b {

+ return a

+ }

+ return b

+func (f *fakeNotion) object(id string) map[string]any {

+ p := f.pages[id]

+ return map[string]any{

+ "object": "page", "id": p.id, "in_trash": p.inTrash,

+ "last_edited_time": p.edited,

+ "parent": parentOf(p.parent),

+ "properties": map[string]any{

+ "Name": map[string]any{"type": "title", "title": []map[string]any{{"plain_text": p.title}}},

+ },

+ }

+func parentOf(id string) map[string]any {

+ if id == "" {

+ return map[string]any{"type": "workspace", "workspace": true}

+ }

+ return map[string]any{"type": "page_id", "page_id": id}

+// paged writes a cursor-paginated list response, honouring the switches.

+func (f *fakeNotion) paged(w http.ResponseWriter, path string, items []map[string]any, cursor string) {

+ if f.hasMoreNoCursor[path] {

+ writeJSON(w, map[string]any{"object": "list", "results": items, "has_more": true, "next_cursor": nil})

+ return

+ }

+ if f.loopCursor[path] {

+ writeJSON(w, map[string]any{"object": "list", "results": items, "has_more": true, "next_cursor": "stuck"})

+ return

+ }

+ // Real pagination: one item per page, so the cursor path is actually

+ // exercised rather than assumed.

+ if cursor == "" && len(items) > 1 {

+ writeJSON(w, map[string]any{"object": "list", "results": items[:1], "has_more": true, "next_cursor": "c1"})

+ return

+ }

+ if strings.HasPrefix(cursor, "c") {

+ n := int(cursor[1] - '0')

+ if n < len(items) {

+ rest := items[n:]

+ if len(rest) > 1 {

+ writeJSON(w, map[string]any{"object": "list", "results": rest[:1], "has_more": true,

+ "next_cursor": "c" + string(rune('0'+n+1))})

+ return

+ }

+ writeJSON(w, map[string]any{"object": "list", "results": rest, "has_more": false, "next_cursor": nil})

+ return

+ }

+ writeJSON(w, map[string]any{"object": "list", "results": items, "has_more": false, "next_cursor": nil})

+func writeJSON(w http.ResponseWriter, v any) {

+ w.Header().Set("Content-Type", "application/json")

+ _ = json.NewEncoder(w).Encode(v)

+func (f *fakeNotion) start() *httptest.Server {

+ mux := http.NewServeMux()

+ mux.HandleFunc("/v1/search", func(w http.ResponseWriter, r *http.Request) {

+ f.calls["search"]++

+ if f.reject(w, "search") {

+ return

+ }

+ var body map[string]any

+ _ = json.NewDecoder(r.Body).Decode(&body)

+ cursor, _ := body["start_cursor"].(string)

+ var items []map[string]any

+ // ONLY the roots. Children exist and search does not return them.

+ for _, id := range f.roots {

+ items = append(items, f.object(id))

+ }

+ for _, id := range f.searchLeaks {

+ items = append(items, f.object(id))

+ }

+ for db := range f.dbs {

+ items = append(items, map[string]any{

+ "object": "data_source", "id": db, "last_edited_time": "2026-09-01T10:00:00.000Z",

+ "name": "Projects", "parent": parentOf(f.dbParent[db]),

+ })

+ }

+ f.paged(w, "search", items, cursor)

+ })

+ mux.HandleFunc("/v1/blocks/", func(w http.ResponseWriter, r *http.Request) {

+ id := strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/v1/blocks/"), "/children")

+ f.calls["blocks:"+id]++

+ if f.reject(w, "blocks:"+id) {

+ return

+ }

+ if f.notFound[id] {

+ w.WriteHeader(http.StatusNotFound)

+ // Notion's real 404 body quotes the id and the connection

+ // name. Kept verbatim in shape so the log gate sees a body

+ // that actually carries a coordinate.

+ writeJSON(w, map[string]any{"object": "error", "status": 404, "code": "object_not_found",

+ "message": "Could not find block with ID: " + id + ". Make sure the relevant pages and " +

+ "databases are shared with your connection \"CANARYCONNECTION\"."})

+ return

+ }

+ p, ok := f.pages[id]

+ if !ok {

+ w.WriteHeader(http.StatusNotFound)

+ writeJSON(w, map[string]any{"object": "error", "status": 404})

+ return

+ }

+ items := append([]map[string]any{}, p.blocks...)

+ for _, c := range p.children {

+ items = append(items, map[string]any{

+ "object": "block", "id": c, "type": "child_page", "has_children": true,

+ "child_page": map[string]any{"title": f.pages[c].title},

+ })

+ }

+ for _, d := range p.childDBs {

+ items = append(items, map[string]any{

+ "object": "block", "id": d, "type": "child_database", "has_children": true,

+ "child_database": map[string]any{"title": "Projects"},

+ })

+ }

+ f.paged(w, "blocks:"+id, items, r.URL.Query().Get("start_cursor"))

+ })

+ mux.HandleFunc("/v1/databases/", func(w http.ResponseWriter, r *http.Request) {

+ id := strings.TrimPrefix(r.URL.Path, "/v1/databases/")

+ f.calls["db:"+id]++

+ sources, ok := f.dbs[id]

+ if !ok {

+ w.WriteHeader(http.StatusNotFound)

+ writeJSON(w, map[string]any{"object": "error", "status": 404})

+ return

+ }

+ var ds []map[string]any

+ for _, s := range sources {

+ ds = append(ds, map[string]any{"id": s, "name": "Projects"})

+ }

+ writeJSON(w, map[string]any{"object": "database", "id": id, "data_sources": ds})

+ })

+ mux.HandleFunc("/v1/data_sources/", func(w http.ResponseWriter, r *http.Request) {

+ id := strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/v1/data_sources/"), "/query")

+ f.calls["ds:"+id]++

+ if f.reject(w, "ds:"+id) {

+ return

+ }

+ var items []map[string]any

+ for _, rid := range f.rows[id] {

+ items = append(items, f.object(rid))

+ }

+ var body map[string]any

+ _ = json.NewDecoder(r.Body).Decode(&body)

+ cursor, _ := body["start_cursor"].(string)

+ f.paged(w, "ds:"+id, items, cursor)

+ })

+ mux.HandleFunc("/v1/users/me", func(w http.ResponseWriter, r *http.Request) {

+ if r.Header.Get("Notion-Version") != APIVersion {

+ w.WriteHeader(http.StatusBadRequest)

+ writeJSON(w, map[string]any{"object": "error", "code": "missing_version"})

+ return

+ }

+ writeJSON(w, map[string]any{"object": "user", "id": "bot", "name": "memoria"})

+ })

+ srv := httptest.NewServer(mux)

+ f.t.Cleanup(srv.Close)

+ return srv

+func (f *fakeNotion) reject(w http.ResponseWriter, key string) bool {

+ if f.alwaysRateLimit[key] {

+ w.Header().Set("Retry-After", "0")

+ w.WriteHeader(http.StatusTooManyRequests)

+ writeJSON(w, map[string]any{"object": "error", "code": "rate_limited"})

+ return true

+ }

+ if f.overload[key] {

+ w.Header().Set("Retry-After", "0")

+ w.WriteHeader(529)

+ writeJSON(w, map[string]any{"object": "error", "code": "service_overload"})

+ return true

+ }

+ return false

+// walkAll drives a full traversal the way the connector will, returning

+// every page reached, in order.

+func walkAll(t *testing.T, c *Client) ([]Ref, map[string]*Content, error) {

+ t.Helper()

+ roots, err := c.Roots(t.Context())

+ if err != nil {

+ return nil, nil, err

+ }

+ var out []Ref

+ got := map[string]*Content{}

+ seen := map[string]bool{}

+ queue := append([]Ref{}, roots...)

+ for len(queue) > 0 {

+ ref := queue[0]

+ queue = queue[1:]

+ if seen[ref.ID] {

+ continue

+ }

+ seen[ref.ID] = true

+ content, err := c.Fetch(t.Context(), ref)

+ if err != nil {

+ return out, got, err

+ }

+ out = append(out, ref)

+ got[ref.ID] = content

+ queue = append(queue, content.Children...)

+ }

+ return out, got, nil

diff --git a/internal/memoria/notion/markdown.go b/internal/memoria/notion/markdown.go

new file mode 100644

index 0000000..4ae4721

--- /dev/null

+++ b/internal/memoria/notion/markdown.go

@@ -0,0 +1,325 @@

+package notion

+import (

+ "encoding/json"

+ "errors"

+ "strconv"

+ "strings"

+func asErr(err error, target **Error) bool { return errors.As(err, target) }

+// block is one Notion block. The payload for a block's own type lives

+// under a key named after the type, so it is decoded on demand from raw

+// rather than modelled as sixty optional fields.

+type block struct {

+ ID string `json:"id"`

+ Type string `json:"type"`

+ HasChildren bool `json:"has_children"`

+ InTrash bool `json:"in_trash"`

+ raw json.RawMessage

+ children []block

+// payload decodes the type-specific body of a block.

+func (b block) payload() blockBody {

+ var wrap map[string]json.RawMessage

+ if err := json.Unmarshal(b.raw, &wrap); err != nil {

+ return blockBody{}

+ }

+ var body blockBody

+ if v, ok := wrap[b.Type]; ok {

+ _ = json.Unmarshal(v, &body)

+ }

+ return body

+type blockBody struct {

+ RichText []richText `json:"rich_text"`

+ Caption []richText `json:"caption"`

+ Title string `json:"title"`

+ Language string `json:"language"`

+ Checked bool `json:"checked"`

+ URL string `json:"url"`

+ Name string `json:"name"`

+ External struct {

+ URL string `json:"url"`

+ } `json:"external"`

+ File struct {

+ URL string `json:"url"`

+ } `json:"file"`

+ Cells [][]richText `json:"cells"`

+// render turns a page's blocks into Markdown, collecting the child pages

+// it must not inline.

+//

+// childPath is the ancestry a nested page inherits: this page's own path

+// plus this page's title. Traversal knows it because it arrived here from

+// the parent, which is why nothing walks parent links upward.

+func render(blocks []block, childPath []string) (string, []Ref, int) {

+ var b strings.Builder

+ var kids []Ref

+ unread := 0

+ renderInto(&b, blocks, childPath, &kids, &unread, 0)

+ return strings.TrimSpace(b.String()), kids, unread

+func renderInto(b *strings.Builder, blocks []block, childPath []string, kids *[]Ref, unread *int, depth int) {

+ numbered := 0

+ for _, blk := range blocks {

+ if blk.InTrash {

+ continue

+ }

+ p := blk.payload()

+ text := plain(p.RichText)

+ indent := strings.Repeat(" ", depth)

+ if blk.Type != "numbered_list_item" {

+ numbered = 0

+ }

+ switch blk.Type {

+ case "paragraph":

+ writeLine(b, indent+text)

+ case "heading_1":

+ writeLine(b, "# "+text)

+ case "heading_2":

+ writeLine(b, "## "+text)

+ case "heading_3":

+ writeLine(b, "### "+text)

+ case "bulleted_list_item", "toggle":

+ writeLine(b, indent+"- "+text)

+ case "numbered_list_item":

+ numbered++

+ writeLine(b, indent+strconv.Itoa(numbered)+". "+text)

+ case "to_do":

+ mark := " "

+ if p.Checked {

+ mark = "x"

+ }

+ writeLine(b, indent+"- ["+mark+"] "+text)

+ case "quote":

+ writeLine(b, indent+"> "+text)

+ case "callout":

+ writeLine(b, indent+"> "+text)

+ case "code":

+ writeLine(b, "```"+p.Language)

+ writeLine(b, text)

+ writeLine(b, "```")

+ case "divider":

+ writeLine(b, "---")

+ case "table_row":

+ var cells []string

+ for _, c := range p.Cells {

+ cells = append(cells, plain(c))

+ }

+ writeLine(b, "| "+strings.Join(cells, " | ")+" |")

+ case "table", "column_list", "column", "synced_block":

+ // Containers with no text of their own. Their children carry

+ // the content and are rendered below.

+ case "image", "file", "video", "pdf", "audio", "meeting_notes", "embed", "bookmark":

+ // memoria indexes text. A caption or filename is text; the

+ // asset is not, and its signed URL expires anyway.

+ if label := firstNonEmpty(plain(p.Caption), p.Name, p.Title); label != "" {

+ writeLine(b, indent+label)

+ }

+ case "child_page", "child_database":

+ // NOT inlined: its own document. Inlining would duplicate

+ // every passage into all of its ancestors.

+ title := firstNonEmpty(p.Title, text, "Untitled")

+ ref := Ref{ID: blk.ID, Title: title, Path: childPath, Kind: KindPage}

+ if blk.Type == "child_database" {

+ ref.Kind, ref.DataSourceID = KindDataSource, blk.ID

+ }

+ *kids = append(*kids, ref)

+ writeLine(b, indent+"- "+title)

+ default:

+ // Notion adds block types. Failing the run on an unknown one

+ // would break memoria on Notion's release schedule; silence

+ // would make a page look thin for no stated reason. Count it.

+ *unread++

+ if text != "" {

+ writeLine(b, indent+text)

+ }

+ if len(blk.children) > 0 {

+ nested := depth + 1

+ if blk.Type == "table" || blk.Type == "column_list" || blk.Type == "column" || blk.Type == "synced_block" {

+ nested = depth

+ }

+ renderInto(b, blk.children, childPath, kids, unread, nested)

+ }

+func writeLine(b *strings.Builder, s string) {

+ if strings.TrimSpace(s) == "" {

+ return

+ }

+ b.WriteString(s)

+ b.WriteString("\n\n")

+// Properties renders a database row's properties as a leading key/value

+// block, so a row carries its own columns into the passage that cites it.

+//

+// A row page's body alone is usually empty — the information is in the

+// columns — so without this a 40-row database becomes 40 blank documents.

+func Properties(raw json.RawMessage) string {

+ if len(raw) == 0 {

+ return ""

+ }

+ var props map[string]json.RawMessage

+ if err := json.Unmarshal(raw, &props); err != nil {

+ return ""

+ }

+ names := make([]string, 0, len(props))

+ for k := range props {

+ names = append(names, k)

+ }

+ sortStrings(names)

+ var b strings.Builder

+ for _, name := range names {

+ v := renderProperty(props[name])

+ if v == "" {

+ continue

+ }

+ b.WriteString(name)

+ b.WriteString(": ")

+ b.WriteString(v)

+ b.WriteString("\n")

+ }

+ return strings.TrimSpace(b.String())

+func renderProperty(raw json.RawMessage) string {

+ var p struct {

+ Type string `json:"type"`

+ Title []richText `json:"title"`

+ RichText []richText `json:"rich_text"`

+ Number *float64 `json:"number"`

+ Checkbox *bool `json:"checkbox"`

+ URL string `json:"url"`

+ Email string `json:"email"`

+ Phone string `json:"phone_number"`

+ Select *struct {

+ Name string `json:"name"`

+ } `json:"select"`

+ MultiSelect []struct {

+ Name string `json:"name"`

+ } `json:"multi_select"`

+ Status *struct {

+ Name string `json:"name"`

+ } `json:"status"`

+ Date *struct {

+ Start string `json:"start"`

+ End string `json:"end"`

+ } `json:"date"`

+ People []struct {

+ Name string `json:"name"`

+ } `json:"people"`

+ Formula json.RawMessage `json:"formula"`

+ Rollup json.RawMessage `json:"rollup"`

+ }

+ if err := json.Unmarshal(raw, &p); err != nil {

+ return ""

+ }

+ switch p.Type {

+ case "title":

+ return plain(p.Title)

+ case "rich_text":

+ return plain(p.RichText)

+ case "number":

+ if p.Number == nil {

+ return ""

+ }

+ return strconv.FormatFloat(*p.Number, 'f', -1, 64)

+ case "checkbox":

+ if p.Checkbox != nil && *p.Checkbox {

+ return "yes"

+ }

+ return "no"

+ case "url":

+ return p.URL

+ case "email":

+ return p.Email

+ case "phone_number":

+ return p.Phone

+ case "select":

+ if p.Select == nil {

+ return ""

+ }

+ return p.Select.Name

+ case "status":

+ if p.Status == nil {

+ return ""

+ }

+ return p.Status.Name

+ case "multi_select":

+ var names []string

+ for _, s := range p.MultiSelect {

+ names = append(names, s.Name)

+ }

+ return strings.Join(names, ", ")

+ case "date":

+ if p.Date == nil {

+ return ""

+ }

+ if p.Date.End != "" {

+ return p.Date.Start + " – " + p.Date.End

+ }

+ return p.Date.Start

+ case "people":

+ var names []string

+ for _, s := range p.People {

+ names = append(names, s.Name)

+ }

+ return strings.Join(names, ", ")

+ case "formula", "rollup":

+ return scalarOf(p.Formula) + scalarOf(p.Rollup)

+ }

+ return ""

+// scalarOf pulls a displayable value out of a formula or rollup result,

+// which nests its answer under its own result type.

+func scalarOf(raw json.RawMessage) string {

+ if len(raw) == 0 {

+ return ""

+ }

+ var v struct {

+ Type string `json:"type"`

+ String string `json:"string"`

+ Number *float64 `json:"number"`

+ Date *struct {

+ Start string `json:"start"`

+ } `json:"date"`

+ Boolean *bool `json:"boolean"`

+ }

+ if err := json.Unmarshal(raw, &v); err != nil {

+ return ""

+ }

+ switch {

+ case v.String != "":

+ return v.String

+ case v.Number != nil:

+ return strconv.FormatFloat(*v.Number, 'f', -1, 64)

+ case v.Date != nil:

+ return v.Date.Start

+ case v.Boolean != nil && *v.Boolean:

+ return "yes"

+ }

+ return ""

+func sortStrings(s []string) {

+ for i := 1; i < len(s); i++ {

+ for j := i; j > 0 && s[j] < s[j-1]; j-- {

+ s[j], s[j-1] = s[j-1], s[j]

+ }

diff --git a/internal/memoria/notion/traverse.go b/internal/memoria/notion/traverse.go

new file mode 100644

index 0000000..1ed5cd5

--- /dev/null

+++ b/internal/memoria/notion/traverse.go

@@ -0,0 +1,358 @@

+package notion

+import (

+ "context"

+ "encoding/json"

+ "strings"

+// Ref is one page memoria may hold: a coordinate plus the ancestry it

+// was reached through.

+type Ref struct {

+ // ID is the Notion page id. A coordinate, and the caller seals it.

+ ID string

+ // Title and Path are CONTENT. Never log them.

+ Title string

+ Path []string

+ // EditedTime is Notion's last_edited_time, the change detector.

+ EditedTime string

+ // Kind distinguishes an ordinary page from a data-source row, which

+ // the extractor renders differently.

+ Kind Kind

+ // DataSourceID is set when Kind is KindDataSource: the thing to

+ // query for rows.

+ DataSourceID string

+type Kind string

+const (

+ KindPage Kind = "page"

+ KindDataSource Kind = "data_source"

+// Content is one page's extracted body plus what lies beneath it.

+type Content struct {

+ Markdown string

+ // Children are the pages and data sources nested directly under this

+ // one, with their Path already resolved — traversal knows the

+ // ancestry because it arrived from the parent.

+ Children []Ref

+ // UnreadBlocks counts blocks of a type this package does not render.

+ // Reported rather than swallowed: a page that came in thin then has

+ // a reason attached instead of looking like a thin page.

+ UnreadBlocks int

+// list is one page of a paginated Notion response.

+type list struct {

+ Results []json.RawMessage `json:"results"`

+ NextCursor *string `json:"next_cursor"`

+ HasMore bool `json:"has_more"`

+// object is the shared head of every Notion object we read.

+type object struct {

+ Object string `json:"object"`

+ ID string `json:"id"`

+ InTrash bool `json:"in_trash"`

+ Parent parent `json:"parent"`

+ Properties json.RawMessage `json:"properties"`

+ Title []richText `json:"title"`

+ Name string `json:"name"`

+ LastEdited string `json:"last_edited_time"`

+type parent struct {

+ Type string `json:"type"`

+ PageID string `json:"page_id"`

+ DatabaseID string `json:"database_id"`

+ DataSourceID string `json:"data_source_id"`

+ BlockID string `json:"block_id"`

+ Workspace bool `json:"workspace"`

+// id returns the parent's id whatever kind it is, so "is my parent in

+// this set?" is one lookup rather than four.

+func (p parent) id() string {

+ switch {

+ case p.PageID != "":

+ return p.PageID

+ case p.DataSourceID != "":

+ return p.DataSourceID

+ case p.DatabaseID != "":

+ return p.DatabaseID

+ case p.BlockID != "":

+ return p.BlockID

+ }

+ return ""

+type richText struct {

+ PlainText string `json:"plain_text"`

+func plain(rt []richText) string {

+ var b strings.Builder

+ for _, r := range rt {

+ b.WriteString(r.PlainText)

+ }

+ return b.String()

+// paginate walks every page of a cursor-paginated endpoint and applies

+// the total-decoder rules. It returns everything or it returns an error;

+// there is no partial result, because a partial result that looks whole

+// is how an import reports success while missing pages.

+func (c *Client) paginate(ctx context.Context, method, path string, body map[string]any, fn func(json.RawMessage) error) error {

+ seen := map[string]bool{}

+ cursor := ""

+ for {

+ var page list

+ var err error

+ if method == "POST" {

+ b := map[string]any{"page_size": 100}

+ for k, v := range body {

+ b[k] = v

+ }

+ if cursor != "" {

+ b["start_cursor"] = cursor

+ }

+ err = c.post(ctx, path, b, &page)

+ } else {

+ p := path + "?page_size=100"

+ if cursor != "" {

+ p += "&start_cursor=" + cursor

+ }

+ err = c.get(ctx, p, &page)

+ }

+ if err != nil {

+ return err

+ }

+ for _, r := range page.Results {

+ if err := fn(r); err != nil {

+ return err

+ }

+ if !page.HasMore {

+ return nil

+ }

+ // has_more with no cursor is unfinishable. Treating it as the

+ // end would silently truncate the enumeration.

+ if page.NextCursor == nil || *page.NextCursor == "" {

+ return &Error{Code: CodeTraversalFailed}

+ }

+ // A repeating cursor is a loop. Without this the run never

+ // finishes and never says why.

+ if seen[*page.NextCursor] {

+ return &Error{Code: CodeTraversalFailed}

+ }

+ seen[*page.NextCursor] = true

+ cursor = *page.NextCursor

+ }

+// Roots returns the pages and data sources to traverse from.

+//

+// Search is used HERE AND ONLY HERE, and only for what Notion guarantees

+// it does: "any pages or databases that are directly shared with a

+// connection are guaranteed to be returned". Notion documents that

+// search is NOT an exhaustive enumerator — "search is not guaranteed to

+// return everything" — and returns no field saying a result was partial,

+// so a run that paged cleanly to the end could be missing pages and

+// would report a complete import. Everything below a root is found by

+// traversal instead, which enumerates a parent's actual children.

+//

+// A result whose parent is also in the result set is not a root: it will

+// be reached by traversal, with its ancestry known. A result whose

+// parent is absent IS a root, even if it is nested in Notion — we cannot

+// see above it, so its ancestry starts here and the status page says how

+// many roots were found.

+func (c *Client) Roots(ctx context.Context) ([]Ref, error) {

+ var objs []object

+ err := c.paginate(ctx, "POST", "/v1/search", nil, func(raw json.RawMessage) error {

+ var o object

+ if err := json.Unmarshal(raw, &o); err != nil {

+ return &Error{Code: CodeTraversalFailed}

+ }

+ objs = append(objs, o)

+ return nil

+ })

+ if err != nil {

+ return nil, err

+ }

+ present := make(map[string]bool, len(objs))

+ for _, o := range objs {

+ present[o.ID] = true

+ }

+ var roots []Ref

+ for _, o := range objs {

+ if o.InTrash {

+ continue

+ }

+ if pid := o.Parent.id(); pid != "" && present[pid] {

+ continue // reachable by traversal, with ancestry

+ }

+ r := Ref{ID: o.ID, EditedTime: o.LastEdited, Kind: KindPage}

+ switch o.Object {

+ case "data_source":

+ r.Kind, r.DataSourceID = KindDataSource, o.ID

+ r.Title = firstNonEmpty(plain(o.Title), o.Name)

+ case "database":

+ // A database is a container of data sources, not a document.

+ // Its data sources are resolved when it is traversed.

+ r.Kind, r.DataSourceID = KindDataSource, o.ID

+ r.Title = firstNonEmpty(plain(o.Title), o.Name)

+ default:

+ r.Title = titleOf(o)

+ }

+ if r.Title == "" {

+ r.Title = "Untitled"

+ }

+ roots = append(roots, r)

+ }

+ return roots, nil

+// titleOf pulls a page's title out of its properties. The title property

+// is identified by its TYPE, not its name: Notion lets it be renamed and

+// commonly is ("Name", "Task", "Project").

+func titleOf(o object) string {

+ if len(o.Title) > 0 {

+ return plain(o.Title)

+ }

+ if len(o.Properties) == 0 {

+ return ""

+ }

+ var props map[string]struct {

+ Type string `json:"type"`

+ Title []richText `json:"title"`

+ }

+ if err := json.Unmarshal(o.Properties, &props); err != nil {

+ return ""

+ }

+ for _, p := range props {

+ if p.Type == "title" {

+ return plain(p.Title)

+ }

+ return ""

+func firstNonEmpty(vals ...string) string {

+ for _, v := range vals {

+ if strings.TrimSpace(v) != "" {

+ return v

+ }

+ return ""

+// Fetch reads one page: its body as Markdown, and the pages nested

+// directly under it with their ancestry already resolved.

+func (c *Client) Fetch(ctx context.Context, ref Ref) (*Content, error) {

+ if ref.Kind == KindDataSource {

+ return c.fetchDataSource(ctx, ref)

+ }

+ blocks, err := c.blocks(ctx, ref.ID)

+ if err != nil {

+ return nil, err

+ }

+ childPath := append(append([]string{}, ref.Path...), ref.Title)

+ md, kids, unread := render(blocks, childPath)

+ return &Content{Markdown: md, Children: kids, UnreadBlocks: unread}, nil

+// fetchDataSource turns a database into its rows.

+//

+// The database itself is not a document — it is a container — so its

+// Markdown is empty and its rows are its children. Under API version

+// 2025-09-03 and later a database holds one or more DATA SOURCES and

+// rows are queried from those, not from the database.

+func (c *Client) fetchDataSource(ctx context.Context, ref Ref) (*Content, error) {

+ sources := []string{ref.DataSourceID}

+ if ids, err := c.dataSourcesOf(ctx, ref.DataSourceID); err == nil && len(ids) > 0 {

+ sources = ids

+ } else if err != nil {

+ return nil, err

+ }

+ childPath := append(append([]string{}, ref.Path...), ref.Title)

+ var kids []Ref

+ for _, ds := range sources {

+ err := c.paginate(ctx, "POST", "/v1/data_sources/"+ds+"/query", nil, func(raw json.RawMessage) error {

+ var o object

+ if err := json.Unmarshal(raw, &o); err != nil {

+ return &Error{Code: CodeTraversalFailed}

+ }

+ if o.InTrash {

+ return nil

+ }

+ title := titleOf(o)

+ if title == "" {

+ title = "Untitled"

+ }

+ kids = append(kids, Ref{

+ ID: o.ID, Title: title, Path: childPath,

+ EditedTime: o.LastEdited, Kind: KindPage,

+ })

+ return nil

+ })

+ if err != nil {

+ return nil, err

+ }

+ return &Content{Children: kids}, nil

+// dataSourcesOf resolves a database id to its data source ids. An id

+// that is already a data source answers 404, which is not an error here

+// — it means the caller had the right id all along.

+func (c *Client) dataSourcesOf(ctx context.Context, databaseID string) ([]string, error) {

+ var db struct {

+ DataSources []struct {

+ ID string `json:"id"`

+ } `json:"data_sources"`

+ }

+ if err := c.get(ctx, "/v1/databases/"+databaseID, &db); err != nil {

+ var e *Error

+ if ok := asErr(err, &e); ok && e.Status == 404 {

+ return nil, nil

+ }

+ return nil, err

+ }

+ ids := make([]string, 0, len(db.DataSources))

+ for _, d := range db.DataSources {

+ ids = append(ids, d.ID)

+ }

+ return ids, nil

+// blocks reads every block of a page, recursing into nested blocks but

+// NOT into child pages — those are their own documents, and inlining

+// them would duplicate every passage into all of its ancestors.

+func (c *Client) blocks(ctx context.Context, id string) ([]block, error) {

+ var out []block

+ err := c.paginate(ctx, "GET", "/v1/blocks/"+id+"/children", nil, func(raw json.RawMessage) error {

+ var b block

+ if err := json.Unmarshal(raw, &b); err != nil {

+ return &Error{Code: CodeTraversalFailed}

+ }

+ b.raw = raw

+ if b.HasChildren && b.Type != "child_page" && b.Type != "child_database" {

+ kids, err := c.blocks(ctx, b.ID)

+ if err != nil {

+ return err

+ }

+ b.children = kids

+ }

+ out = append(out, b)

+ return nil

+ })

+ if err != nil {

+ return nil, err

+ }

+ return out, nil

diff --git a/internal/memoria/notion/verify.go b/internal/memoria/notion/verify.go

new file mode 100644

index 0000000..490bd07

--- /dev/null

+++ b/internal/memoria/notion/verify.go

@@ -0,0 +1,38 @@

+package notion

+import "context"

+// Verify checks a token and returns an identifier for what it is bound

+// to, so a connector row can be told apart from one pointing elsewhere.

+//

+// Called when a token is saved, so a bad one fails at the form rather

+// than part-way through an import.

+//

+// The identifier is the BOT USER ID first, and the workspace name only

+// as a fallback. Notion's /v1/users/me does not expose a workspace id;

+// the name is what it gives, and a name collides between two workspaces

+// called the same thing and changes when either is renamed. The bot id

+// is unique to this integration and stable, which is the property

+// actually wanted — the caller keys it anyway, so neither reaches disk

+// in plaintext.

+func Verify(ctx context.Context, token string) (string, error) {

+ c, err := New(token)

+ if err != nil {

+ return "", err

+ }

+ return c.verify(ctx)

+func (c *Client) verify(ctx context.Context) (string, error) {

+ var out struct {

+ ID string `json:"id"`

+ Name string `json:"name"`

+ Bot struct {

+ WorkspaceName string `json:"workspace_name"`

+ } `json:"bot"`

+ }

+ if err := c.get(ctx, "/v1/users/me", &out); err != nil {

+ return "", err

+ }

+ return firstNonEmpty(out.ID, out.Bot.WorkspaceName, out.Name), nil

diff --git a/internal/memoria/render.go b/internal/memoria/render.go

index 9c60571..b02e675 100644

--- a/internal/memoria/render.go

+++ b/internal/memoria/render.go

@@ -26,7 +26,7 @@ var assets = rastrillo.NewAssets(appFS)

var pages = map[string]*template.Template{}

func init() {

- for _, name := range []string{"index", "document", "signin"} {

+ for _, name := range []string{"index", "document", "signin", "connectors", "notion"} {

pages[name] = template.Must(template.New("layout").

Funcs(template.FuncMap{

"asset": assets.Path,

diff --git a/internal/memoria/seal/seal.go b/internal/memoria/seal/seal.go

index 1bae0ac..f708e79 100644

--- a/internal/memoria/seal/seal.go

+++ b/internal/memoria/seal/seal.go

@@ -36,6 +36,21 @@ const (

PurposeBody = "document.body"

PurposeSnapshot = "index.snapshot"

PurposeExternal = "document.external_id"

+ // PurposePath seals a document's ancestry in its source. Its own

+ // purpose rather than reusing PurposeTitle: a path and a title are

+ // different kinds of thing, and the rule here is one purpose per

+ // kind so that a mix-up cannot decrypt.

+ PurposePath = "document.path"

+ // PurposeCredential seals a connector's token, so a disclosure of the

+ // chunk store does not also yield the key to the source those chunks

+ // came from.

+ PurposeCredential = "connector.token"

+ // PurposeCoordinate seals a source coordinate — a Notion page id.

+ //

+ // REVERSIBLE, unlike Fingerprint. That distinction is the whole

+ // reason this purpose exists: the fetch phase has to hand the id back

+ // to the API it came from, and an HMAC cannot be handed back.

+ PurposeCoordinate = "source.coordinate"

)

func sha256New() hash.Hash { return sha256.New() }

diff --git a/internal/memoria/static/memoria.css b/internal/memoria/static/memoria.css

index a698061..5a7a240 100644

--- a/internal/memoria/static/memoria.css

+++ b/internal/memoria/static/memoria.css

@@ -195,6 +195,23 @@

[rst-results] { list-style: none; margin: 0; padding: 0; }

[rst-result] { border-top: 1px solid var(--rst-line); padding: 0.9rem 0; }

[rst-result]:first-child { border-top: 0; }

+/* The document's ancestry in its source, above the title. Sits above

+ rather than below because it is read as a location — "Handbook ›

+ Onboarding ›" then the page — and colours come from tokens so light

+ and dark both resolve through theme.css. */

+[rst-result-path] {

+ /* --rst-text-muted, not --rst-text-mut. This rule was written by

+ * copying the trail rule above while it still carried that typo: an

+ * undefined token with no fallback makes the whole declaration invalid

+ * at computed-value time, so the path would have rendered as ordinary

+ * body text — muted in the markup, not muted on the screen. The merge

+ * brought the fix and the test that catches the class of it.

+ */

+ color: var(--rst-text-muted);

+ display: block;

+ font-size: 0.8125rem;

+ margin-bottom: 0.1rem;

[rst-result-title] { font-weight: 600; }

[rst-result-trail] {

/* The token is --rst-text-muted. This said --rst-text-mut, which

diff --git a/internal/memoria/store.go b/internal/memoria/store.go

index e0a05f0..bdb951a 100644

--- a/internal/memoria/store.go

+++ b/internal/memoria/store.go

@@ -34,9 +34,16 @@ type Result struct {

Ordinal int

Source string

Title string

- Heading string

- Snippet string

- Score float64

+ // Path is the document's ancestry in its source. SHOWN, not merely

+ // indexed: two pages called "Week one" under different handbooks are

+ // the same result to a reader who cannot see where each came from,

+ // and carrying ancestry for findability while hiding it from the

+ // person reading the answer is half a feature. Found by looking at

+ // the rendered page, which is the only thing that would have.

+ Path string

+ Heading string

+ Snippet string

+ Score float64

}

// Summary is a document as the list shows it.

@@ -44,6 +51,7 @@ type Summary struct {

ID int64

Source string

Title string

+ Path string

Bytes int

Chunks int

Updated string

@@ -58,32 +66,73 @@ func (s *Store) externalID(key string) (string, error) {

return s.sealer.Fingerprint(seal.PurposeExternal, []byte("key\x00"+key))

}

+// IngestSpec is one document to store.

+//

+// A struct rather than a parameter list because the connectors added two

+// more fields and a five-string call site is a bug waiting for someone

+// to transpose two of them.

+type IngestSpec struct {

+ // Source names where this came from — "upload", "paste", "notion".

+ Source string

+ // Key is the STABLE identifier in the source: a file path, a Notion

+ // page id, a pasted document's title. Not the body — see

+ // Document.ExternalID.

+ Key string

+ // Title and Body are content.

+ Title string

+ Body string

+ // Path is the document's ancestry in its source, outermost first.

+ // Empty for upload and paste, which have none.

+ Path []string

+ // AllowTitleOnly permits a document whose body is empty, storing the

+ // title as the body.

+ //

+ // Off for upload and paste, where an empty submission is a person's

+ // mistake and refusing it is the useful answer. On for connectors,

+ // where an empty page is a FACT ABOUT THE SOURCE — real workspaces

+ // are full of placeholders and title-only rows — and both

+ // alternatives are wrong: failing the run lets one empty page block

+ // an entire import, and skipping makes the page an absence with

+ // nothing to explain it.

+ AllowTitleOnly bool

// Ingest stores a document and indexes it.

-// Re-ingesting the same ExternalID replaces the document's chunks and

-// bumps its revision. The old chunks are deleted from the index *and* the

-// database before the new ones land, so a passage that the new revision

-// removed cannot survive as a citation to text that is no longer there —

-// which is the stale-passage failure the spec's §4.1 refuses to allow.

-// key is the STABLE identifier in the source: a file path, a page id, a

-// pasted document's title. Not the body — see Document.ExternalID.

+// Re-ingesting the same Key replaces the document's chunks and bumps its

+// revision. The old chunks are deleted from the index *and* the database

+// before the new ones land, so a passage that the new revision removed

+// cannot survive as a citation to text that is no longer there — the

+// stale-passage failure the spec's §4.1 refuses to allow.

// Two failures, and they are not the same failure. Anything before the

// commit means nothing was stored and the id comes back zero. Anything

// after it means the document IS stored and only the index is behind: the

// id comes back real and the error wraps ErrIndexLag.

-func (s *Store) Ingest(ctx context.Context, source, key, title, body string) (int64, error) {

- title = strings.TrimSpace(title)

- body = strings.TrimSpace(body)

+func (s *Store) Ingest(ctx context.Context, spec IngestSpec) (int64, error) {

+ title := strings.TrimSpace(spec.Title)

+ body := strings.TrimSpace(spec.Body)

if body == "" {

- return 0, errors.New("nothing to store: the document is empty")

+ if !spec.AllowTitleOnly {

+ return 0, errors.New("nothing to store: the document is empty")

+ }

+ if title == "" {

+ return 0, errors.New("nothing to store: the document has neither a body nor a title")

+ }

+ // A title-only document IS its title. Storing an empty body

+ // instead would index nothing and make the page unfindable by

+ // the only words it has.

+ body = title

}

if title == "" {

title = firstLine(body)

}

+ key := spec.Key

if strings.TrimSpace(key) == "" {

key = title

}

+ pathStr := chunk.Chunk{Path: spec.Path}.PathString()

external, err := s.externalID(key)

if err != nil {

return 0, err

@@ -97,32 +146,63 @@ func (s *Store) Ingest(ctx context.Context, source, key, title, body string) (in

if err != nil {

return 0, err

}

+ var sealedPath []byte

+ if pathStr != "" {

+ if sealedPath, err = s.sealer.Seal(seal.PurposePath, []byte(pathStr)); err != nil {

+ return 0, err

+ }

sealedBody, err := s.sealer.Seal(seal.PurposeBody, []byte(body))

if err != nil {

return 0, err

}

- chunks := chunk.Split(body)

+ chunks := chunk.SplitUnder(spec.Path, body)

if len(chunks) == 0 {

return 0, errors.New("nothing to store: the document produced no passages")

}

var docID int64

- var unchanged bool

+ var unchanged, rechunk bool

err = s.db.G.WithContext(ctx).Transaction(func(tx *gormTx) error {

var existing Document

- res := tx.Where("source = ? AND external_id = ?", source, external).First(&existing)

+ res := tx.Where("source = ? AND external_id = ?", spec.Source, external).First(&existing)

switch {

case res.Error == nil:

docID = existing.ID

if existing.BodyPrint == bodyPrint {

- // Same key, same body: nothing changed. Do not bump the

- // revision — a revision that moves without the content

- // moving would make every cached authorization stale for

- // no reason.

- unchanged = true

- return nil

+ // Same key, same body — but the page may have been

+ // RENAMED or MOVED, and neither touches the body.

+ //

+ // An earlier version returned here unconditionally, so a

+ // renamed page kept its old title forever and a moved one

+ // kept its old ancestry. Worse for a connector: an

+ // ancestor rename does not change a descendant's own edit

+ // time at all, so nothing upstream ever signalled it.

+ // Found by a design review.

+ moved, err := s.titleOrPathMoved(existing, title, pathStr)

+ if err != nil {

+ return err

+ }

+ if !moved {

+ // Genuinely unchanged. Do NOT bump the revision: a

+ // revision that moves without the content moving

+ // would make every cached authorization stale for no

+ // reason.

+ unchanged = true

+ return nil

+ }

+ // Title or path only. Still no revision bump — the

+ // passages a citation points at have not moved — and the

+ // chunks are left alone, because the path lives on the

+ // document. The index is refreshed below, since it is

+ // the index that carries the path.

+ return tx.Model(&Document{}).Where("id = ?", docID).Updates(map[string]any{

+ "sealed_title": sealedTitle,

+ "sealed_path": sealedPath,

+ }).Error

}

+ rechunk = true

if err := tx.Where("document_id = ?", docID).Delete(&Chunk{}).Error; err != nil {

return err

}

@@ -130,19 +210,22 @@ func (s *Store) Ingest(ctx context.Context, source, key, title, body string) (in

"revision": existing.Revision + 1,

"sealed_title": sealedTitle,

"sealed_body": sealedBody,

+ "sealed_path": sealedPath,

"body_print": bodyPrint,

"bytes": len(body),

}).Error; err != nil {

return err

}

default:

+ rechunk = true

d := Document{

- Source: source,

+ Source: spec.Source,

ExternalID: external,

BodyPrint: bodyPrint,

Revision: 1,

SealedTitle: sealedTitle,

SealedBody: sealedBody,

+ SealedPath: sealedPath,

Bytes: len(body),

}

if err := tx.Create(&d).Error; err != nil {

@@ -151,6 +234,9 @@ func (s *Store) Ingest(ctx context.Context, source, key, title, body string) (in

docID = d.ID

}

+ if !rechunk {

+ return nil

+ }

rows := make([]Chunk, 0, len(chunks))

for _, c := range chunks {

// Seal the body ALONE, not Indexable(): the heading has its

@@ -226,12 +312,38 @@ func (s *Store) Ingest(ctx context.Context, source, key, title, body string) (in

// under.

var ErrIndexLag = errors.New("the document is stored, but refreshing the index failed")

+// titleOrPathMoved reports whether a document's title or ancestry has

+// changed, by opening what is stored and comparing plaintext.

+//

+// Comparing the sealed blobs would be cheaper and wrong: AES-GCM uses a

+// fresh nonce per seal, so identical plaintext seals to different bytes

+// every time and every document would look moved on every ingest.

+func (s *Store) titleOrPathMoved(d Document, title, pathStr string) (bool, error) {

+ cur, err := s.sealer.Open(seal.PurposeTitle, d.SealedTitle)

+ if err != nil {

+ return false, fmt.Errorf("open title %d: %w", d.ID, err)

+ }

+ if string(cur) != title {

+ return true, nil

+ }

+ var curPath string

+ if len(d.SealedPath) > 0 {

+ p, err := s.sealer.Open(seal.PurposePath, d.SealedPath)

+ if err != nil {

+ return false, fmt.Errorf("open path %d: %w", d.ID, err)

+ }

+ curPath = string(p)

+ }

+ return curPath != pathStr, nil

// indexDocument opens a document's chunks and adds them to the index.

func (s *Store) indexDocument(ctx context.Context, docID int64) error {

- // The title is indexed alongside every chunk, so it is read here

- // rather than left to the caller: Ingest and Rebuild both land in

- // this function, and a title indexed on only one of those paths

- // would make a document findable until the next restart.

+ // The title and the ancestry are both indexed alongside every chunk,

+ // so both are read here rather than left to the caller: Ingest and

+ // Rebuild both land in this function, and a value indexed on only one

+ // of those paths would make a document findable until the next

+ // restart.

var doc Document

if err := s.db.G.WithContext(ctx).Where("id = ?", docID).First(&doc).Error; err != nil {

return err

@@ -240,6 +352,16 @@ func (s *Store) indexDocument(ctx context.Context, docID int64) error {

if err != nil {

return fmt.Errorf("open document title %d: %w", docID, err)

}

+ // The path lives on the DOCUMENT and is stamped onto every one of its

+ // entries, because every passage of a nested page is equally nested.

+ var path string

+ if len(doc.SealedPath) > 0 {

+ p, err := s.sealer.Open(seal.PurposePath, doc.SealedPath)

+ if err != nil {

+ return fmt.Errorf("open document path %d: %w", docID, err)

+ }

+ path = string(p)

+ }

var rows []Chunk

if err := s.db.G.WithContext(ctx).Where("document_id = ?", docID).Order("ordinal").Find(&rows).Error; err != nil {

return err

@@ -261,6 +383,7 @@ func (s *Store) indexDocument(ctx context.Context, docID int64) error {

Heading: string(heading),

Text: string(text),

Title: string(title),

+ Path: path,

})

}

if len(entries) == 0 {

@@ -308,6 +431,7 @@ func (s *Store) Search(ctx context.Context, query string, limit int) ([]Result,

}

titles := map[int64]string{}

sources := map[int64]string{}

+ paths := map[int64]string{}

for _, d := range docs {

t, err := s.sealer.Open(seal.PurposeTitle, d.SealedTitle)

if err != nil {

@@ -315,6 +439,13 @@ func (s *Store) Search(ctx context.Context, query string, limit int) ([]Result,

}

titles[d.ID] = string(t)

sources[d.ID] = d.Source

+ if len(d.SealedPath) > 0 {

+ p, err := s.sealer.Open(seal.PurposePath, d.SealedPath)

+ if err != nil {

+ return nil, fmt.Errorf("open path %d: %w", d.ID, err)

+ }

+ paths[d.ID] = string(p)

+ }

}

// livePassage is the database's answer to "is there still such a

// passage", and the database is the only thing allowed to answer it.

@@ -382,6 +513,7 @@ func (s *Store) Search(ctx context.Context, query string, limit int) ([]Result,

Ordinal: h.Ordinal,

Source: sources[h.DocumentID],

Title: titles[h.DocumentID],

+ Path: paths[h.DocumentID],

Heading: h.Heading,

Snippet: h.Snippet,

Score: h.Score,

@@ -406,10 +538,19 @@ func (s *Store) Summaries(ctx context.Context) ([]Summary, error) {

if err := s.db.G.WithContext(ctx).Model(&Chunk{}).Where("document_id = ?", d.ID).Count(&n).Error; err != nil {

return nil, err

}

+ var path string

+ if len(d.SealedPath) > 0 {

+ p, err := s.sealer.Open(seal.PurposePath, d.SealedPath)

+ if err != nil {

+ return nil, fmt.Errorf("open path %d: %w", d.ID, err)

+ }

+ path = string(p)

+ }

out = append(out, Summary{

ID: d.ID,

Source: d.Source,

Title: string(t),

+ Path: path,

Bytes: d.Bytes,

Chunks: int(n),

Updated: d.UpdatedAt.Format("2 Jan 2006, 15:04"),

diff --git a/internal/memoria/sync.go b/internal/memoria/sync.go

new file mode 100644

index 0000000..620039c

--- /dev/null

+++ b/internal/memoria/sync.go

@@ -0,0 +1,281 @@

+package memoria

+import (

+ "context"

+ "errors"

+ "log/slog"

+ "time"

+ "amadan.net/oficina/memoria/internal/memoria/chunk"

+ "amadan.net/oficina/memoria/internal/memoria/notion"

+ "amadan.net/oficina/memoria/internal/memoria/seal"

+// SourceClient is what a sync needs from a source. An interface so the

+// loop can be driven against a fake without the app package importing an

+// HTTP client — which the network gate forbids it anyway.

+type SourceClient interface {

+ Roots(ctx context.Context) ([]notion.Ref, error)

+ Fetch(ctx context.Context, ref notion.Ref) (*notion.Content, error)

+// Syncer runs imports.

+type Syncer struct {

+ store *Store

+ logger *slog.Logger

+ // dial builds a client from a token. Swapped in tests.

+ dial func(token string) (SourceClient, error)

+// NewSyncer wires a syncer against the real Notion API.

+func NewSyncer(store *Store, logger *slog.Logger) *Syncer {

+ return &Syncer{store: store, logger: logger, dial: func(token string) (SourceClient, error) {

+ return notion.New(token)

+ }}

+// beatEvery is how often the run says it is still alive.

+const beatEvery = 25

+// Run imports everything reachable from the connector's roots.

+//

+// ONE PASS, not the two phases an earlier design specified. With

+// traversal, enumeration and extraction are the same API call — fetching

+// a page returns its body AND the child pages beneath it — so separating

+// them would double every request against a limit of three per second.

+// The ledger is still written per page, so the ingest work already done

+// survives an interruption; what a resumed run repeats is the traversal,

+// because a page's children cannot be discovered without asking for them

+// and a child's own edits do not change its parent's edit time.

+//

+// Errors divide in two, and the division is load-bearing:

+//

+// - A traversal failure below a page (rate limit exhausted, a broken

+// cursor, an unreadable body) is FATAL. It means the pages beneath

+// that point were never enumerated, and a run that continued would

+// report "done" while missing a whole subtree — an absence produced

+// by a failure, which is the shape this codebase ranks worst.

+// - A single page that has gone (404 between enumeration and fetch) is

+// recorded against that page and the run carries on. One placeholder

+// must not block an import, but it must not vanish either.

+func (s *Syncer) Run(ctx context.Context, connectorID, runID int64) {

+ token, err := s.store.ConnectorToken(ctx, "notion")

+ if err != nil {

+ s.fail(ctx, runID, "unauthorized")

+ return

+ }

+ client, err := s.dial(token)

+ if err != nil {

+ s.fail(ctx, runID, "unauthorized")

+ return

+ }

+ s.RunWith(ctx, client, connectorID, runID)

+// RunWith is Run against a supplied client.

+func (s *Syncer) RunWith(ctx context.Context, client SourceClient, connectorID, runID int64) {

+ roots, err := client.Roots(ctx)

+ if err != nil {

+ s.fail(ctx, runID, string(notion.CodeOf(err)))

+ return

+ }

+ if err := s.store.UpdateRun(ctx, runID, map[string]any{

+ "roots": len(roots), "state": RunFetching,

+ }); err != nil {

+ s.fail(ctx, runID, "network")

+ return

+ }

+ var stats struct{ fetched, skipped, empty, failed, unread, seen int }

+ visited := map[string]bool{}

+ queue := append([]notion.Ref{}, roots...)

+ for len(queue) > 0 {

+ ref := queue[0]

+ queue = queue[1:]

+ if visited[ref.ID] {

+ continue

+ }

+ visited[ref.ID] = true

+ stats.seen++

+ if stats.seen%beatEvery == 0 {

+ if err := s.store.Beat(ctx, runID); err != nil {

+ s.logger.Error("sync heartbeat", "run", runID, "err", err)

+ }

+ // Record the coordinate BEFORE fetching, so a page that fails to

+ // fetch still has a row to be named in rather than being absent.

+ if err := s.store.SeeSourcePage(ctx, connectorID, ref.ID, ref.Title, ref.Path, ref.EditedTime); err != nil {

+ s.logger.Error("sync ledger write", "run", runID, "err", err)

+ s.fail(ctx, runID, "network")

+ return

+ }

+ content, err := client.Fetch(ctx, ref)

+ if err != nil {

+ if gone(err) {

+ stats.failed++

+ s.markByPage(ctx, connectorID, ref, PageFailed, string(notion.CodeOf(err)), 0, "")

+ continue

+ }

+ // Not survivable: the subtree below here was never seen.

+ s.fail(ctx, runID, string(notion.CodeOf(err)))

+ return

+ }

+ queue = append(queue, content.Children...)

+ stats.unread += content.UnreadBlocks

+ state, docID, err := s.ingest(ctx, connectorID, ref, content)

+ if err != nil {

+ stats.failed++

+ s.markByPage(ctx, connectorID, ref, PageFailed, "extract_failed", 0, "")

+ continue

+ }

+ switch state {

+ case PageEmpty:

+ stats.empty++

+ case "":

+ stats.skipped++

+ continue

+ default:

+ stats.fetched++

+ }

+ s.markByPage(ctx, connectorID, ref, state, "", docID, ref.EditedTime)

+ }

+ now := time.Now().UTC()

+ if err := s.store.UpdateRun(ctx, runID, map[string]any{

+ "state": RunDone, "total": stats.seen, "fetched": stats.fetched,

+ "skipped": stats.skipped, "empty": stats.empty, "failed_pages": stats.failed,

+ "unread_blocks": stats.unread, "finished_at": now, "heartbeat": now,

+ }); err != nil {

+ s.logger.Error("sync finish", "run", runID, "err", err)

+ }

+ // Counts only. No title, no path, no page id — AGENTS.md's rule, and

+ // the reason this line carries nothing that came from the workspace.

+ s.logger.Info("sync finished", "run", runID, "pages", stats.seen,

+ "fetched", stats.fetched, "skipped", stats.skipped,

+ "empty", stats.empty, "failed", stats.failed, "unread_blocks", stats.unread)

+// ingest stores one page, or reports that it did not need storing.

+//

+// Returns "" when the page was already up to date. A data source is a

+// container rather than a document and is never stored.

+func (s *Syncer) ingest(ctx context.Context, connectorID int64, ref notion.Ref, content *notion.Content) (string, int64, error) {

+ if ref.Kind == notion.KindDataSource {

+ return "", 0, nil

+ }

+ upToDate, err := s.upToDate(ctx, connectorID, ref)

+ if err != nil {

+ return "", 0, err

+ }

+ if upToDate {

+ return "", 0, nil

+ }

+ id, err := s.store.Ingest(ctx, IngestSpec{

+ Source: "notion",

+ Key: ref.ID,

+ Title: ref.Title,

+ Body: content.Markdown,

+ Path: ref.Path,

+ // On, because an empty page is a fact about the source. See

+ // IngestSpec.AllowTitleOnly.

+ AllowTitleOnly: true,

+ })

+ if err != nil {

+ return "", 0, err

+ }

+ if content.Markdown == "" {

+ return PageEmpty, id, nil

+ }

+ return PageFetched, id, nil

+// upToDate reports whether the ledger already holds this page at this

+// edit time AND this ancestry.

+//

+// Both, because an ancestor rename moves the ancestry without moving the

+// page's own edit time — so a check on the edit time alone leaves every

+// descendant of a renamed page stale forever, with nothing upstream ever

+// signalling it.

+func (s *Syncer) upToDate(ctx context.Context, connectorID int64, ref notion.Ref) (bool, error) {

+ rows, err := s.store.pageRow(ctx, connectorID, ref.ID)

+ if err != nil || rows == nil {

+ return false, err

+ }

+ if rows.State != PageFetched && rows.State != PageEmpty {

+ return false, nil

+ }

+ if rows.IngestedEdited != ref.EditedTime {

+ return false, nil

+ }

+ want, err := s.store.pathPrint(ref.Path, ref.Title)

+ if err != nil {

+ return false, err

+ }

+ return rows.IngestedPath == want, nil

+func (s *Syncer) markByPage(ctx context.Context, connectorID int64, ref notion.Ref, state, code string, docID int64, edited string) {

+ row, err := s.store.pageRow(ctx, connectorID, ref.ID)

+ if err != nil || row == nil {

+ return

+ }

+ print, err := s.store.pathPrint(ref.Path, ref.Title)

+ if err != nil {

+ return

+ }

+ if edited == "" {

+ print = ""

+ }

+ if err := s.store.MarkPage(ctx, row.ID, state, code, docID, edited, print); err != nil {

+ s.logger.Error("sync mark page", "err", err)

+ }

+func (s *Syncer) fail(ctx context.Context, runID int64, code string) {

+ now := time.Now().UTC()

+ if err := s.store.UpdateRun(ctx, runID, map[string]any{

+ "state": RunFailed, "error_code": code, "finished_at": now,

+ }); err != nil {

+ s.logger.Error("sync fail", "run", runID, "err", err)

+ }

+ s.logger.Error("sync failed", "run", runID, "code", code)

+func asNotionErr(err error, target **notion.Error) bool { return errors.As(err, target) }

+// gone reports whether an error is about one page rather than about the

+// traversal. Only a 404 qualifies: everything else means we do not know

+// what lies below, and continuing would report a complete import.

+func gone(err error) bool {

+ var e *notion.Error

+ if !asNotionErr(err, &e) {

+ return false

+ }

+ return e.Status == 404

+// pageRow reads one ledger row by coordinate.

+func (s *Store) pageRow(ctx context.Context, connectorID int64, pageID string) (*SourcePage, error) {

+ print, err := s.sealer.Fingerprint(seal.PurposeCoordinate, []byte("page\x00"+pageID))

+ if err != nil {

+ return nil, err

+ }

+ var row SourcePage

+ res := s.db.G.WithContext(ctx).Where("connector_id = ? AND page_print = ?", connectorID, print).First(&row)

+ if res.Error != nil {

+ return nil, nil

+ }

+ return &row, nil

+// pathPrint is the keyed fingerprint of an ancestry plus a title, so a

+// move or a rename is detectable without opening the row.

+func (s *Store) pathPrint(path []string, title string) (string, error) {

+ pathStr := chunk.Chunk{Path: path}.PathString()

+ return s.sealer.Fingerprint(seal.PurposePath, []byte("path\x00"+pathStr+"\x00"+title))

diff --git a/internal/memoria/templates/connectors.html b/internal/memoria/templates/connectors.html

new file mode 100644

index 0000000..fc13679

--- /dev/null

+++ b/internal/memoria/templates/connectors.html

@@ -0,0 +1,22 @@

+{{define "title"}}Sources · memoria{{end}}

+{{define "content"}}

+<h1>Sources</h1>

+<p rst-lede>

+ memoria can pull in work that lives outside the suite. What you connect,

+ memoria keeps: disconnecting a source stops memoria seeing new changes,

+ it does not remove what memoria already holds.

+</p>

+<ul rst-list>

+ <li rst-list-item>

+ <a href="/connectors/notion" rst-list-title>Notion</a>

+ <p rst-cell-mut>

+ {{if .Configured}}Connected.{{else}}Not connected.{{end}}

+ Imports the pages you share with a Notion integration, and everything nested under them.

+ </p>

+ </li>

+</ul>

+{{end}}

diff --git a/internal/memoria/templates/index.html b/internal/memoria/templates/index.html

index 632acce..e709867 100644

--- a/internal/memoria/templates/index.html

+++ b/internal/memoria/templates/index.html

@@ -45,6 +45,12 @@

+ {{/* Ancestry above the title, heading trail below it. Both, and in

+ that order, because they answer different questions: the path

+ says which part of the workspace this came from, the heading

+ says where in the page. A result showing only the second is

+ two indistinguishable "Week one"s under different handbooks. */}}

+ {{if .Path}}<span rst-result-path rst-cell-mut>{{.Path}}</span>{{end}}

<a href="/documents/{{.DocumentID}}#p{{.Ordinal}}" rst-result-title>{{.Title}}</a>

{{if .Heading}}<span rst-result-trail>{{.Heading}}</span>{{end}}

<p rst-result-snippet>{{.Snippet | safeSnippet}}</p>

diff --git a/internal/memoria/templates/layout.html b/internal/memoria/templates/layout.html

index 69e3bcf..275d3cb 100644

--- a/internal/memoria/templates/layout.html

+++ b/internal/memoria/templates/layout.html

@@ -20,11 +20,15 @@

scripts are linked, and it costs a querySelector to load it twice. */}}

+{{block "head" .}}{{end}}

</head>

<body>

<a href="/" rst-brand>memoria</a>

- {{if .Signed}}<form method="post" action="/signout" rst-inline><button rst-btn="quiet compact">Sign out</button></form>{{end}}

+ {{if .Signed}}<nav rst-inline>

+ <a href="/connectors" rst-btn="quiet compact">Sources</a>

+ <form method="post" action="/signout" rst-inline><button rst-btn="quiet compact">Sign out</button></form>

+ </nav>{{end}}

</header>

diff --git a/internal/memoria/templates/notion.html b/internal/memoria/templates/notion.html

new file mode 100644

index 0000000..03a6e26

--- /dev/null

+++ b/internal/memoria/templates/notion.html

@@ -0,0 +1,139 @@

+{{define "title"}}Notion · memoria{{end}}

+{{/* The status page refreshes itself ONLY while a run is active, and it

+ is a meta refresh because every affordance here works scriptless.

+ This is also the honest mechanism: the import is a goroutine that

+ lives as long as the process, and the platform sleeps an idle app,

+ so an open page is what keeps it awake. The copy below says so

+ rather than implying a background worker memoria does not have. */}}

+{{define "head"}}{{if .Run}}{{if .Run.Running}}<meta http-equiv="refresh" content="5">{{end}}{{end}}{{end}}

+{{define "content"}}

+<h1>Notion</h1>

+{{if .Flash}}<div rst-callout="{{.FlashKind}}"><p rst-callout-body>{{.Flash}}</p></div>{{end}}

+{{if not .Configured}}

+ <p rst-lede>

+ memoria imports the Notion pages you share with an integration, and

+ everything nested underneath them.

+ </p>

+ <div rst-callout="quiet">

+ <p rst-callout-body>

+ <strong>What you connect, memoria keeps.</strong> Once a page is

+ imported, it stays in memoria and stays readable by everyone here —

+ even if you later archive it in Notion, unshare it, or revoke this

+ token. Removing it is something you do in memoria.

+ </p>

+ </div>

+ <section rst-section>

+ <h2>Connect</h2>

+ <ol rst-steps>

+ <li>In Notion, create an <strong>internal integration</strong> with

+ read-only capabilities and no user information.</li>

+ <li>Share the pages you want memoria to hold with that integration.

+ Sharing a page is what publishes it. memoria imports what the

+ integration can see and nothing else.</li>

+ <li>Paste the integration token below.</li>

+ </ol>

+ <form method="post" action="/connectors/notion" rst-stack>

+ <label for="token">Integration token</label>

+ <input type="password" id="token" name="token" autocomplete="off" required>

+ <p rst-cell-mut>Checked with Notion before it is saved. It is stored sealed and is never shown again.</p>

+ <button rst-btn="primary">Connect</button>

+ </form>

+ </section>

+{{else}}

+ <section rst-section>

+ <h2>Connected</h2>

+ <p rst-cell-mut>

+ {{if .Verified}}Token last checked {{.Verified}}.{{else}}Token stored.{{end}}

+ </p>

+ {{if .CountsKnown}}

+ <p>{{.Imported}} of {{.Pages}} known page{{if ne .Pages 1}}s{{end}} imported.</p>

+ {{else}}

+ <p>The page counts could not be read, so this says nothing about how much has been imported.</p>

+ {{end}}

+ <form method="post" action="/connectors/notion/sync" rst-inline>

+ <button rst-btn="primary"{{if .Run}}{{if .Run.Running}} disabled{{end}}{{end}}>

+ {{if .Run}}{{if .Run.Running}}Importing…{{else}}Import now{{end}}{{else}}Import now{{end}}

+ </button>

+ </form>

+ </section>

+ {{if .Run}}

+ <section rst-section>

+ <h2>Last import</h2>

+ <p>{{.Run.Message}}</p>

+ <dl rst-facts>

+ <dt>Started</dt><dd>{{.Run.Started}}</dd>

+ {{if .Run.Finished}}<dt>Finished</dt><dd>{{.Run.Finished}}</dd>{{end}}

+ <dt>Top-level pages shared with the integration</dt><dd>{{.Run.Roots}}</dd>

+ <dt>Pages seen</dt>

+ <dd>

+ {{if .Run.TotalKnown}}

+ {{.Run.Total}}

+ {{else}}

+ {{.Run.Seen}} so far — the total is not known until the import finishes

+ {{end}}

+ </dd>

+ <dt>Imported</dt><dd>{{.Run.Fetched}}</dd>

+ <dt>Already up to date</dt><dd>{{.Run.Skipped}}</dd>

+ <dt>Empty pages</dt><dd>{{.Run.Empty}}</dd>

+ <dt>Could not be read</dt>

+ <dd>

+ {{.Run.Failed}}

+ {{if .Run.Failed}}<span rst-cell-mut>— counted here rather than left out of the totals</span>{{end}}

+ </dd>

+ {{if .Run.Unread}}

+ <dt>Blocks memoria does not read</dt>

+ <dd>{{.Run.Unread}} <span rst-cell-mut>— a page that came in thin may be one of these</span></dd>

+ {{end}}

+ </dl>

+ </section>

+ {{end}}

+ <section rst-section>

+ <h2>Replace the token</h2>

+ <p rst-cell-mut>

+ Rotating a token replaces the stored one. The old token is not kept.

+ Nothing already imported is affected.

+ </p>

+ <form method="post" action="/connectors/notion" rst-stack>

+ <label for="token">New integration token</label>

+ {{/* Never carries a value. The stored token is not rendered back

+ here or anywhere — not even masked, because a masked secret is

+ still a secret in the DOM. TestTokenIsNeverRendered checks

+ this branch specifically: an earlier version of that gate only

+ looked at the not-yet-connected page, which has no token field

+ at all, so it passed while proving nothing. */}}

+ <input type="password" id="token" name="token" autocomplete="off" required>

+ <button rst-btn="quiet">Replace</button>

+ </form>

+ </section>

+ <section rst-section>

+ <h2>Disconnect</h2>

+ <p>

+ This removes the token and stops future imports. <strong>It deletes

+ nothing memoria has already imported</strong> — those pages stay

+ here until someone deletes them in memoria.

+ </p>

+ <form method="post" action="/connectors/notion/disconnect" rst-inline>

+ <button rst-btn="danger">Disconnect Notion</button>

+ </form>

+ </section>

+{{end}}

diff --git a/internal/memoriatest/ancestry_test.go b/internal/memoriatest/ancestry_test.go

new file mode 100644

index 0000000..97ad601

--- /dev/null

+++ b/internal/memoriatest/ancestry_test.go

@@ -0,0 +1,195 @@

+package memoriatest

+import (

+ "context"

+ "os"

+ "path/filepath"

+ "strings"

+ "testing"

+ memoria "amadan.net/oficina/memoria/internal/memoria"

+// A page with ancestry, as a connector would supply it.

+func notionish(title, body string, path ...string) memoria.IngestSpec {

+ return memoria.IngestSpec{

+ Source: "notion", Key: "page-" + title, Title: title, Body: body,

+ Path: path, AllowTitleOnly: true,

+ }

+// TestTitleOnlyIsRefusedForPeopleAndAcceptedForConnectors pins the

+// asymmetry AllowTitleOnly exists for.

+//

+// An empty paste is a person's mistake and the useful answer is to refuse

+// it. An empty Notion page is a fact about the source — real workspaces

+// are full of placeholders — and both alternatives to storing it are

+// wrong: failing the run lets one placeholder block an entire import, and

+// skipping makes the page an absence with nothing to explain it.

+//

+// MUTATION-TESTED 2026-09-05: deleting the AllowTitleOnly branch in

+// Store.Ingest makes the connector half fail with "the document is

+// empty"; defaulting AllowTitleOnly to true makes the person half fail.

+// Reverted.

+func TestTitleOnlyIsRefusedForPeopleAndAcceptedForConnectors(t *testing.T) {

+ _, store, _ := newAppWithStore(t)

+ ctx := context.Background()

+ if _, err := store.Ingest(ctx, memoria.IngestSpec{

+ Source: "paste", Key: "k", Title: "A title", Body: " ",

+ }); err == nil {

+ t.Error("an empty paste was accepted; a person's empty submission should be refused")

+ }

+ id, err := store.Ingest(ctx, notionish("Quarterly planning", "", "Handbook"))

+ if err != nil {

+ t.Fatalf("a title-only Notion page was refused: %v", err)

+ }

+ if id == 0 {

+ t.Fatal("no document id")

+ }

+ // It must be FINDABLE by the only words it has. Storing it with an

+ // empty body would index nothing and hide it.

+ hits, err := store.Search(ctx, "quarterly planning", 10)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if len(hits) == 0 {

+ t.Error("a title-only page was stored but is not searchable by its title")

+ }

+// TestRenameReachesMemoriaWithoutBumpingRevision.

+//

+// Store.Ingest short-circuits when the body fingerprint is unchanged. It

+// used to return there unconditionally, so a page renamed upstream kept

+// its old title in memoria forever — the body never changes on a rename,

+// so nothing else would ever notice. Found by a design review.

+//

+// The revision must NOT move: a citation points at passages, and those

+// have not changed. A revision that moves without the content moving

+// makes every cached authorization stale for no reason.

+//

+// MUTATION-TESTED 2026-09-05: restoring the unconditional `unchanged =

+// true; return nil` makes the title assertion fail. Bumping the revision

+// in the title-only branch makes the revision assertion fail. Reverted.

+func TestRenameReachesMemoriaWithoutBumpingRevision(t *testing.T) {

+ _, store, _ := newAppWithStore(t)

+ ctx := context.Background()

+ const body = "Expenses are reimbursed within thirty days of submission."

+ spec := memoria.IngestSpec{Source: "notion", Key: "page-1", Title: "Old name", Body: body, AllowTitleOnly: true}

+ id, err := store.Ingest(ctx, spec)

+ if err != nil {

+ t.Fatal(err)

+ }

+ _, rev0, _, err := store.Meta(ctx, id)

+ if err != nil {

+ t.Fatal(err)

+ }

+ spec.Title = "New name"

+ if _, err := store.Ingest(ctx, spec); err != nil {

+ t.Fatal(err)

+ }

+ title, _, err := store.Body(ctx, id)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if title != "New name" {

+ t.Errorf("title is %q after a rename, want %q — the rename never reached memoria", title, "New name")

+ }

+ _, rev1, _, err := store.Meta(ctx, id)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if rev1 != rev0 {

+ t.Errorf("revision moved %d→%d on a rename; the passages did not change", rev0, rev1)

+ }

+// TestMoveReachesMemoriaAndReindexes.

+//

+// A page moved to a different parent — or one whose ANCESTOR was renamed

+// — has an unchanged body and an unchanged title. Only the path moves,

+// and the path is what the index scores in its own column, so the index

+// must be rebuilt for that document or the old ancestry stays searchable

+// and the new one never becomes searchable.

+//

+// MUTATION-TESTED 2026-09-05: dropping `sealed_path` from the title/path

+// Updates map leaves the old ancestry findable and fails this. Reverted.

+func TestMoveReachesMemoriaAndReindexes(t *testing.T) {

+ _, store, _ := newAppWithStore(t)

+ ctx := context.Background()

+ const body = "Laptops are ordered through the equipment request form."

+ spec := memoria.IngestSpec{

+ Source: "notion", Key: "page-2", Title: "Equipment", Body: body,

+ Path: []string{"Handbook", "Onboarding"}, AllowTitleOnly: true,

+ }

+ if _, err := store.Ingest(ctx, spec); err != nil {

+ t.Fatal(err)

+ }

+ if hits, _ := store.Search(ctx, "onboarding", 10); len(hits) == 0 {

+ t.Fatal("the original ancestry is not searchable, so this test proves nothing")

+ }

+ // The page moves under Engineering.

+ spec.Path = []string{"Engineering", "Platform"}

+ if _, err := store.Ingest(ctx, spec); err != nil {

+ t.Fatal(err)

+ }

+ if hits, _ := store.Search(ctx, "platform", 10); len(hits) == 0 {

+ t.Error("the new ancestry is not searchable: the move did not reach the index")

+ }

+ if hits, _ := store.Search(ctx, "onboarding", 10); len(hits) != 0 {

+ t.Error("the OLD ancestry is still searchable after a move: a stale coordinate survives")

+ }

+// TestAncestryIsSealedInTheDatabase. A path is content — "Acquisitions ›

+// Project Bluebird" names a thing whether or not anyone opens the page —

+// so it is sealed like a title, under its own purpose.

+//

+// The existing plaintext gate cannot cover this: it drives the HTTP form,

+// and no form carries ancestry.

+//

+// MUTATION-TESTED 2026-09-05: storing pathStr with a plain []byte cast

+// instead of Seal makes this fail. Reverted.

+func TestAncestryIsSealedInTheDatabase(t *testing.T) {

+ _, store, path := newAppWithStore(t)

+ ctx := context.Background()

+ const secret = "PROJECTBLUEBIRD7742"

+ if _, err := store.Ingest(ctx, notionish("Deal terms",

+ "The terms were agreed in March and revisited in June.", "Acquisitions", secret)); err != nil {

+ t.Fatal(err)

+ }

+ // Vacuity guard: the ancestry must actually be indexed, or the scan

+ // below is a scan of nothing relevant.

+ if hits, _ := store.Search(ctx, secret, 10); len(hits) == 0 {

+ t.Fatal("the ancestry is not searchable, so it may not have been stored at all")

+ }

+ entries, err := os.ReadDir(filepath.Dir(path))

+ if err != nil {

+ t.Fatal(err)

+ }

+ checked := 0

+ for _, e := range entries {

+ if e.IsDir() {

+ continue

+ }

+ raw, err := os.ReadFile(filepath.Join(filepath.Dir(path), e.Name()))

+ if err != nil {

+ t.Fatal(err)

+ }

+ checked++

+ if strings.Contains(strings.ToUpper(string(raw)), secret) {

+ t.Errorf("%s holds the ancestry in plaintext: a path is content", e.Name())

+ }

+ if checked == 0 {

+ t.Fatal("no files to check; the gate proved nothing")

+ }

diff --git a/internal/memoriatest/connector_test.go b/internal/memoriatest/connector_test.go

new file mode 100644

index 0000000..ca95afe

--- /dev/null

+++ b/internal/memoriatest/connector_test.go

@@ -0,0 +1,620 @@

+package memoriatest

+import (

+ "context"

+ "io"

+ "log/slog"

+ "net/http"

+ "net/url"

+ "path/filepath"

+ "strings"

+ "testing"

+ "time"

+ "github.com/carlosframework/rastrillo/db"

+ memoria "amadan.net/oficina/memoria/internal/memoria"

+ "amadan.net/oficina/memoria/internal/memoria/notion"

+// testApp is the app's test surface. Declared as an interface because

+// AppForTest returns an unexported type, which a test in another package

+// cannot name.

+type testApp interface {

+ Store() *memoria.Store

+ AsPerson(string)

+ SetOwners([]string)

+ UseFakeSource(memoria.SourceClient, func(context.Context, string) (string, error))

+ WaitForSync(int64, time.Duration) bool

+// fakeSource is a Notion stand-in for the run loop: a tree, some bodies,

+// and per-page errors. No HTTP — the client's own package tests that.

+type fakeSource struct {

+ roots []notion.Ref

+ children map[string][]notion.Ref

+ bodies map[string]string

+ unread map[string]int

+ failWith map[string]error

+ fetches map[string]int

+func newSource() *fakeSource {

+ return &fakeSource{

+ children: map[string][]notion.Ref{}, bodies: map[string]string{},

+ unread: map[string]int{}, failWith: map[string]error{}, fetches: map[string]int{},

+ }

+func (f *fakeSource) Roots(ctx context.Context) ([]notion.Ref, error) {

+ if err := f.failWith["__roots__"]; err != nil {

+ return nil, err

+ }

+ return f.roots, nil

+func (f *fakeSource) Fetch(ctx context.Context, ref notion.Ref) (*notion.Content, error) {

+ f.fetches[ref.ID]++

+ if err := f.failWith[ref.ID]; err != nil {

+ return nil, err

+ }

+ return &notion.Content{

+ Markdown: f.bodies[ref.ID],

+ Children: f.children[ref.ID],

+ UnreadBlocks: f.unread[ref.ID],

+ }, nil

+// add puts a page in the tree. parent "" makes it a root.

+func (f *fakeSource) add(id, title, parent, body string, path ...string) {

+ ref := notion.Ref{ID: id, Title: title, Path: path, EditedTime: "t1", Kind: notion.KindPage}

+ f.bodies[id] = body

+ if parent == "" {

+ f.roots = append(f.roots, ref)

+ return

+ }

+ f.children[parent] = append(f.children[parent], ref)

+func (f *fakeSource) edit(id, body, editedTime string) {

+ f.bodies[id] = body

+ f.retime(id, editedTime)

+func (f *fakeSource) retime(id, editedTime string) {

+ for i := range f.roots {

+ if f.roots[i].ID == id {

+ f.roots[i].EditedTime = editedTime

+ }

+ for p := range f.children {

+ for i := range f.children[p] {

+ if f.children[p][i].ID == id {

+ f.children[p][i].EditedTime = editedTime

+ }

+// rename changes a page's title wherever it appears, and rewrites the

+// inherited ancestry of everything below it — which is what Notion does

+// NOT tell us about, since an ancestor rename leaves a descendant's own

+// last_edited_time untouched.

+func (f *fakeSource) rename(id, was, now string) {

+ retitle := func(r *notion.Ref) {

+ if r.ID == id {

+ r.Title = now

+ }

+ for i, seg := range r.Path {

+ if seg == was {

+ r.Path[i] = now

+ }

+ for i := range f.roots {

+ retitle(&f.roots[i])

+ }

+ for p := range f.children {

+ for i := range f.children[p] {

+ retitle(&f.children[p][i])

+ }

+// connectorApp builds an app with a configured connector and a fake

+// Notion behind it.

+func connectorApp(t *testing.T, src *fakeSource) (http.Handler, testApp, int64, *strings.Builder) {

+ t.Helper()

+ var logs strings.Builder

+ logger := slog.New(slog.NewTextHandler(&logs, &slog.HandlerOptions{Level: slog.LevelDebug}))

+ d, err := db.Open(filepath.Join(t.TempDir(), "app.db"), logger)

+ if err != nil {

+ t.Fatalf("db.Open: %v", err)

+ }

+ t.Cleanup(func() { d.Close() })

+ mux, a, err := memoria.AppForTest(d, testOrigin, testDEK(t), logger)

+ if err != nil {

+ t.Fatalf("AppForTest: %v", err)

+ }

+ app := any(a).(testApp)

+ app.UseFakeSource(src, func(ctx context.Context, token string) (string, error) {

+ return "workspace-1", nil

+ })

+ if err := app.Store().SaveConnector(context.Background(), "notion", "ntn_secret_token_value", "workspace-1"); err != nil {

+ t.Fatal(err)

+ }

+ c, err := app.Store().Connector(context.Background(), "notion")

+ if err != nil {

+ t.Fatal(err)

+ }

+ return mux, app, c.ID, &logs

+func sync(t *testing.T, h http.Handler, app testApp, connectorID int64) {

+ t.Helper()

+ rec := post(t, h, "/connectors/notion/sync", url.Values{})

+ if rec.Code != http.StatusSeeOther {

+ t.Fatalf("sync: status %d, want 303", rec.Code)

+ }

+ if !app.WaitForSync(connectorID, 5*time.Second) {

+ t.Fatal("sync did not finish")

+ }

+func latest(t *testing.T, app testApp, connectorID int64) *memoria.SyncRun {

+ t.Helper()

+ run, err := app.Store().LatestRun(context.Background(), connectorID)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if run == nil {

+ t.Fatal("no run recorded")

+ }

+ return run

+// TestSyncImportsTheTreeWithAncestry is the happy path, and the vacuity

+// guard for everything below it: a gate about failures proves nothing if

+// the success case never worked.

+func TestSyncImportsTheTreeWithAncestry(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "# Welcome\n\nEverything a new starter needs.")

+ src.add("kid", "Onboarding", "root", "Week one covers expenses and travel booking.", "Handbook")

+ src.add("grandkid", "Equipment", "kid", "Laptops are ordered through the request form.", "Handbook", "Onboarding")

+ h, app, cid, logs := connectorApp(t, src)

+ sync(t, h, app, cid)

+ run := latest(t, app, cid)

+ if run.State != memoria.RunDone {

+ t.Fatalf("run state %q, want done (error %q)\nLOGS:\n%s", run.State, run.ErrorCode, logs.String())

+ }

+ if run.Total != 3 {

+ t.Errorf("total %d, want 3", run.Total)

+ }

+ if run.Fetched != 3 {

+ t.Errorf("fetched %d, want 3", run.Fetched)

+ }

+ // The ancestry must be searchable, or carrying it was pointless.

+ hits, err := app.Store().Search(context.Background(), "onboarding", 10)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if len(hits) == 0 {

+ t.Error("the imported ancestry is not searchable")

+ }

+// TestAFailedPageIsNamedNotAbsent. A page that has gone between

+// enumeration and fetch must be counted and recorded, never dropped: 3

+// unreadable pages inside 1,930 that simply vanish read as a complete

+// import.

+//

+// MUTATION-TESTED 2026-09-05: replacing the markByPage/stats.failed++ in

+// the gone(err) branch with a bare `continue` makes this fail. Reverted.

+func TestAFailedPageIsNamedNotAbsent(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs.")

+ src.add("kid", "Onboarding", "root", "Week one.", "Handbook")

+ src.failWith["kid"] = &notion.Error{Code: notion.CodeTraversalFailed, Status: 404}

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ run := latest(t, app, cid)

+ if run.State != memoria.RunDone {

+ t.Errorf("state %q, want done — one missing page must not fail the whole import", run.State)

+ }

+ if run.FailedPages != 1 {

+ t.Errorf("failed_pages %d, want 1 — the page vanished from the totals", run.FailedPages)

+ }

+ body := get(t, h, "/connectors/notion").Body.String()

+ if !strings.Contains(body, "Could not be read") {

+ t.Error("the status page does not name the pages that could not be read")

+ }

+// TestTraversalFailureNeverReportsDone. A failure BELOW a page means the

+// subtree was never enumerated. Continuing would report a complete

+// import while missing everything under that point — a failure rendered

+// as an absence, which this codebase ranks as the worst available.

+//

+// MUTATION-TESTED 2026-09-05: treating every fetch error as a per-page

+// failure (making gone() return true unconditionally) makes this fail.

+// Reverted.

+func TestTraversalFailureNeverReportsDone(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs.")

+ src.add("kid", "Onboarding", "root", "Week one.", "Handbook")

+ src.failWith["kid"] = &notion.Error{Code: notion.CodeRateLimited, Status: 429}

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ run := latest(t, app, cid)

+ if run.State != memoria.RunFailed {

+ t.Fatalf("state %q, want failed — the pages below the failure were never seen", run.State)

+ }

+ if run.ErrorCode != string(notion.CodeRateLimited) {

+ t.Errorf("error_code %q, want %q", run.ErrorCode, notion.CodeRateLimited)

+ }

+ body := get(t, h, "/connectors/notion").Body.String()

+ if strings.Contains(body, "Import finished.") {

+ t.Error("a failed import is described as finished")

+ }

+// TestSyncNeverDeletesOnAbsence. memoria is replacing Notion, so a page

+// disappearing upstream is the expected end state of a migration, not a

+// signal to forget it. Archiving a page in Notion after migrating it

+// must not delete it here.

+//

+// MUTATION-TESTED 2026-09-05: adding a diff-and-delete pass over ledger

+// rows not seen by the latest run makes this fail. Reverted.

+func TestSyncNeverDeletesOnAbsence(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs.")

+ src.add("kid", "Onboarding", "root", "Week one covers expenses.", "Handbook")

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ if hits, _ := app.Store().Search(context.Background(), "expenses", 10); len(hits) == 0 {

+ t.Fatal("the page was never imported, so its survival proves nothing")

+ }

+ // The page is archived in Notion: it is simply no longer returned.

+ src.children["root"] = nil

+ sync(t, h, app, cid)

+ run := latest(t, app, cid)

+ if run.State != memoria.RunDone {

+ t.Errorf("state %q, want done — an absence is not an error either", run.State)

+ }

+ if hits, _ := app.Store().Search(context.Background(), "expenses", 10); len(hits) == 0 {

+ t.Error("a page that vanished from Notion was deleted from memoria")

+ }

+// TestResumeSkipsWhatIsAlreadyImported. The ledger is per connector, not

+// per run, which is what makes a second run continue rather than start

+// over.

+//

+// MUTATION-TESTED 2026-09-05: making upToDate always return false

+// re-ingests everything and makes the skipped assertion fail. Reverted.

+func TestResumeSkipsWhatIsAlreadyImported(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs.")

+ src.add("kid", "Onboarding", "root", "Week one covers expenses.", "Handbook")

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ run := latest(t, app, cid)

+ if run.Skipped != 2 {

+ t.Errorf("skipped %d on an unchanged re-sync, want 2 — nothing changed upstream", run.Skipped)

+ }

+ if run.Fetched != 0 {

+ t.Errorf("fetched %d on an unchanged re-sync, want 0", run.Fetched)

+ }

+// TestEditAndAncestorRenameBothReachMemoria.

+//

+// The rename half is the one an earlier design missed entirely. Editing

+// a page changes its own last_edited_time; renaming its ANCESTOR does

+// not, so a sync keyed only on edit times leaves every descendant of a

+// renamed page carrying stale ancestry forever, with nothing upstream

+// ever signalling it.

+//

+// MUTATION-TESTED 2026-09-05: dropping the IngestedPath comparison from

+// upToDate (leaving only the edit-time check) makes the rename half

+// fail. Reverted.

+func TestEditAndAncestorRenameBothReachMemoria(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs.")

+ src.add("kid", "Onboarding", "root", "Week one covers expenses.", "Handbook")

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ t.Run("an edit is re-ingested", func(t *testing.T) {

+ src.edit("kid", "Week one now covers parking permits.", "t2")

+ sync(t, h, app, cid)

+ hits, _ := app.Store().Search(context.Background(), "parking permits", 10)

+ if len(hits) == 0 {

+ t.Error("an edited page was not re-ingested")

+ }

+ })

+ t.Run("an ancestor rename is re-ingested", func(t *testing.T) {

+ // Handbook becomes Playbook. The CHILD's own edit time does not

+ // move — Notion would not move it either.

+ src.rename("root", "Handbook", "Playbook")

+ sync(t, h, app, cid)

+ if hits, _ := app.Store().Search(context.Background(), "playbook", 10); len(hits) == 0 {

+ t.Error("the new ancestry is not searchable: an ancestor rename never reached memoria")

+ }

+ if hits, _ := app.Store().Search(context.Background(), "handbook", 10); len(hits) != 0 {

+ t.Error("the old ancestry is still searchable after an ancestor rename")

+ }

+ })

+// TestInterruptedRunDoesNotWedgeTheConnector.

+//

+// The run is a goroutine. When the platform sleeps the process it dies

+// having written nothing, leaving the row "fetching" — and a

+// single-flight check that trusted that row would refuse every later

+// sync as already-running, permanently, on the NORMAL path. Boot

+// reclaims them, which is sound by construction: a process that has just

+// started cannot be running a sync.

+//

+// MUTATION-TESTED 2026-09-05: removing the ReclaimRuns call from App()

+// makes this fail with "a sync is already running". Reverted.

+func TestInterruptedRunDoesNotWedgeTheConnector(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs.")

+ dir := t.TempDir()

+ path := filepath.Join(dir, "app.db")

+ logger := slog.New(slog.NewTextHandler(io.Discard, nil))

+ d, err := db.Open(path, logger)

+ if err != nil {

+ t.Fatal(err)

+ }

+ _, a, err := memoria.AppForTest(d, testOrigin, testDEK(t), logger)

+ if err != nil {

+ t.Fatal(err)

+ }

+ app := any(a).(testApp)

+ if err := app.Store().SaveConnector(context.Background(), "notion", "tok", "w"); err != nil {

+ t.Fatal(err)

+ }

+ c, err := app.Store().Connector(context.Background(), "notion")

+ if err != nil {

+ t.Fatal(err)

+ }

+ // A run starts and the process dies: the row stays in flight.

+ if _, err := app.Store().StartRun(context.Background(), c.ID); err != nil {

+ t.Fatal(err)

+ }

+ d.Close()

+ // Boot again over the same database.

+ d2, err := db.Open(path, logger)

+ if err != nil {

+ t.Fatal(err)

+ }

+ t.Cleanup(func() { d2.Close() })

+ h2, a2, err := memoria.AppForTest(d2, testOrigin, testDEK(t), logger)

+ if err != nil {

+ t.Fatal(err)

+ }

+ app2 := any(a2).(testApp)

+ app2.UseFakeSource(src, nil)

+ rec := post(t, h2, "/connectors/notion/sync", url.Values{})

+ if loc := rec.Header().Get("Location"); strings.Contains(loc, "err=inflight") {

+ t.Fatal("the connector is wedged: a run killed by a restart still blocks every later sync")

+ }

+ if !app2.WaitForSync(c.ID, 5*time.Second) {

+ t.Fatal("the new run did not finish")

+ }

+// TestNoNotionContentInLogs extends gate 4 to the connector. Titles,

+// bodies, ancestry, page ids and the token are all content or

+// coordinates, and `carlos logs` is storage we cannot grep.

+//

+// The fake's error carries a canary in a 404, which is the shape Notion

+// really uses — its 404 body quotes the page id and the connection name

+// — so the error path is exercised rather than assumed.

+//

+// MUTATION-TESTED 2026-09-05: logging ref.Title at Info in the fetch

+// loop, or %w-ing the notion error into the run's failure line, makes

+// this fail. Reverted.

+func TestNoNotionContentInLogs(t *testing.T) {

+ const secret = "WOMBATCONFIDENTIAL9931"

+ src := newSource()

+ src.add("root", secret+" handbook", "", "A passage about "+secret+" and revenue.")

+ src.add("kid", "Onboarding", "root", "Week one.", secret+" handbook")

+ src.failWith["kid"] = &notion.Error{Code: notion.CodeTraversalFailed, Status: 404}

+ h, app, cid, logs := connectorApp(t, src)

+ sync(t, h, app, cid)

+ get(t, h, "/connectors/notion")

+ got := logs.String()

+ if strings.Contains(got, secret) {

+ t.Errorf("a log line carries Notion content:\n%s", got)

+ }

+ if strings.Contains(got, "ntn_secret_token_value") {

+ t.Errorf("a log line carries the integration token:\n%s", got)

+ }

+ // Vacuity guard: the run must actually have logged something, or

+ // this scan proves nothing.

+ if !strings.Contains(got, "sync") {

+ t.Fatal("no sync log lines at all; this gate scanned nothing relevant")

+ }

+// TestTokenIsNeverRendered. A stored token must not come back to the

+// page — not in a value attribute, not masked. A masked secret is still

+// a secret in the DOM.

+//

+// "No substring" would be vacuous, since every single character is a

+// substring; the test looks for the whole token and for any run of eight

+// characters of it.

+//

+// MUTATION-TESTED 2026-09-05: rendering the token into the input's value

+// makes this fail. Reverted.

+func TestTokenIsNeverRendered(t *testing.T) {

+ const token = "ntn_secret_token_value"

+ src := newSource()

+ h, app, _, _ := connectorApp(t, src)

+ // The CONNECTED page, which is the one with a stored token and a

+ // field that could carry it. An earlier version of this gate looked

+ // only at the not-yet-connected page — which has no token field at

+ // all — so rendering the token into the rotation form left it green.

+ // Found by mutation-testing this gate.

+ body := get(t, h, "/connectors/notion").Body.String()

+ if !strings.Contains(body, `name="token"`) {

+ t.Fatal("the connected page has no token field, so this gate is scanning a page that could not leak one")

+ }

+ assertNoToken(t, "connected page", body, token)

+ // And the not-yet-connected page, for completeness.

+ if err := app.Store().DeleteConnector(context.Background(), "notion"); err != nil {

+ t.Fatal(err)

+ }

+ assertNoToken(t, "setup page", get(t, h, "/connectors/notion").Body.String(), token)

+// assertNoToken looks for the whole token and for any eight-character run

+// of it. "No substring" would be vacuous — every single character is a

+// substring of the token.

+func assertNoToken(t *testing.T, where, body, token string) {

+ t.Helper()

+ if strings.Contains(body, token) {

+ t.Errorf("%s renders the whole token", where)

+ return

+ }

+ for i := 0; i+8 <= len(token); i++ {

+ if strings.Contains(body, token[i:i+8]) {

+ t.Errorf("%s renders %q, an 8-character run of the token", where, token[i:i+8])

+ return

+ }

+// TestConnectorRoutesRefuseANonOwner.

+//

+// In slice 0 "owner" is the whole admitted set, so this gate is about

+// the mechanism rather than a live distinction — but the mechanism has

+// to exist and fail closed before slice 1 gives it something to

+// separate. It could not be written at all until AppForTest grew an

+// injectable identity.

+//

+// MUTATION-TESTED 2026-09-05: making requireOwner call next

+// unconditionally makes this fail. Reverted.

+func TestConnectorRoutesRefuseANonOwner(t *testing.T) {

+ src := newSource()

+ h, app, _, _ := connectorApp(t, src)

+ if code := get(t, h, "/connectors").Code; code != http.StatusOK {

+ t.Fatalf("an owner got %d from /connectors; this gate proves nothing if nobody can reach it", code)

+ }

+ app.AsPerson("stranger@example.com")

+ for _, target := range []string{"/connectors", "/connectors/notion"} {

+ // 404, not 403: a 403 confirms the route exists.

+ if code := get(t, h, target).Code; code != http.StatusNotFound {

+ t.Errorf("a non-owner got %d from %s, want 404", code, target)

+ }

+ if code := post(t, h, "/connectors/notion/sync", url.Values{}).Code; code != http.StatusNotFound {

+ t.Errorf("a non-owner could reach the sync route (%d)", code)

+ }

+// TestDisconnectKeepsTheDocuments. Parent §3.6 promises that

+// disconnecting stops memoria seeing new changes and does not remove

+// what memoria already holds. This is that promise as a test.

+//

+// MUTATION-TESTED 2026-09-05: deleting documents in DeleteConnector

+// makes this fail. Reverted.

+func TestDisconnectKeepsTheDocuments(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs about expenses.")

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ if hits, _ := app.Store().Search(context.Background(), "expenses", 10); len(hits) == 0 {

+ t.Fatal("nothing was imported, so keeping it proves nothing")

+ }

+ if code := post(t, h, "/connectors/notion/disconnect", url.Values{}).Code; code != http.StatusSeeOther {

+ t.Fatalf("disconnect: status %d", code)

+ }

+ if _, err := app.Store().Connector(context.Background(), "notion"); err == nil {

+ t.Error("the connector survived disconnect")

+ }

+ // The DATABASE, not only the index. An earlier version of this gate

+ // asserted searchability alone, and deleting the document rows left

+ // the in-memory index still answering — so the gate passed while the

+ // documents were gone. Found by mutation-testing it.

+ docs, err := app.Store().Summaries(context.Background())

+ if err != nil {

+ t.Fatal(err)

+ }

+ if len(docs) == 0 {

+ t.Error("disconnecting deleted the stored documents; what memoria imported, memoria keeps")

+ }

+ if hits, _ := app.Store().Search(context.Background(), "expenses", 10); len(hits) == 0 {

+ t.Error("disconnecting removed the documents from the index")

+ }

+// TestEmptyPageIsImportedAndCounted. Real workspaces are full of

+// placeholders. Failing the run on one lets a single empty page block an

+// import; skipping makes it an absence with nothing to explain it.

+//

+// MUTATION-TESTED 2026-09-05: turning AllowTitleOnly off in Syncer.ingest

+// makes the run record a failure instead of an empty page. Reverted.

+func TestEmptyPageIsImportedAndCounted(t *testing.T) {

+ src := newSource()

+ src.add("root", "Quarterly planning", "", "")

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ run := latest(t, app, cid)

+ if run.Empty != 1 {

+ t.Errorf("empty %d, want 1 (failed %d)", run.Empty, run.FailedPages)

+ }

+ if hits, _ := app.Store().Search(context.Background(), "quarterly planning", 10); len(hits) == 0 {

+ t.Error("a title-only page is not findable by the only words it has")

+ }

+// TestUnreadBlocksSurfaceOnTheStatusPage. A page that came in thin should

+// carry a reason rather than look like a thin page.

+func TestUnreadBlocksSurfaceOnTheStatusPage(t *testing.T) {

+ src := newSource()

+ src.add("root", "Notes", "", "A readable paragraph.")

+ src.unread["root"] = 3

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ if run := latest(t, app, cid); run.UnreadBlocks != 3 {

+ t.Fatalf("unread_blocks %d, want 3", run.UnreadBlocks)

+ }

+ if body := get(t, h, "/connectors/notion").Body.String(); !strings.Contains(body, "does not read") {

+ t.Error("the status page does not mention blocks memoria could not read")

+ }

diff --git a/internal/memoriatest/gates_test.go b/internal/memoriatest/gates_test.go

index 8061b9c..4a52066 100644

--- a/internal/memoriatest/gates_test.go

+++ b/internal/memoriatest/gates_test.go

@@ -164,6 +164,14 @@ func TestAMultiFileDropNeverSpillsPlaintextToATempFile(t *testing.T) {

// MUTATION-TESTED 2026-09-03: adding `http.Get("https://example.com")`

// to Store.Ingest makes this fail naming store.go. Reverted.

+//

+// AMENDED 2026-09-05 for the Notion connector, which must dial

+// api.notion.com. The blanket ban survives; one directory is exempt and

+// pays for the exemption with a runtime host pin. See allowedNetworkDirs.

+//

+// MUTATION-TESTED 2026-09-05: adding an `*http.Client` field to

+// internal/memoria/handlers.go still makes this fail, so narrowing the

+// gate did not blunt it. Reverted.

func TestNoOutboundNetworkInAppCode(t *testing.T) {

// A blacklist, and it is worth saying so: this is a source

// heuristic, not a proof. A review defeated the first version with

@@ -182,7 +190,37 @@ func TestNoOutboundNetworkInAppCode(t *testing.T) {

for _, f := range forbidden {

pats = append(pats, regexp.MustCompile(f))

}

+ // EXACTLY ONE package may reach the network, and this is the list.

+ //

+ // It is a list rather than a deleted gate because the connector had

+ // to dial api.notion.com and the cheapest way to make this test pass

+ // would have been to widen it until it named nothing. The exemption

+ // is paid for by a RUNTIME guard that is stronger than the grep it

+ // replaces here: notion.pinnedTransport refuses every host but the

+ // one it was built for, and the client refuses redirects outright,

+ // both asserted by notion.TestClientDialsOnlyNotion. A source grep

+ // cannot tell an http.Client pointed at Notion from one pointed

+ // anywhere else; that test can.

+ //

+ // Adding a second entry here should be very hard to justify.

+ allowedNetworkDirs := []string{

+ filepath.Join("..", "memoria", "notion"),

+ }

+ if len(allowedNetworkDirs) != 1 {

+ t.Fatalf("%d packages may reach the network, want exactly 1: %v",

+ len(allowedNetworkDirs), allowedNetworkDirs)

+ }

+ exempt := func(p string) bool {

+ for _, d := range allowedNetworkDirs {

+ if strings.HasPrefix(filepath.Clean(p), filepath.Clean(d)+string(filepath.Separator)) {

+ return true

+ }

+ return false

+ }

var hits []string

+ scanned := 0

walk := func(root string) error {

return filepath.Walk(root, func(p string, info os.FileInfo, err error) error {

if err != nil {

@@ -191,10 +229,14 @@ func TestNoOutboundNetworkInAppCode(t *testing.T) {

if info.IsDir() || !strings.HasSuffix(p, ".go") || strings.HasSuffix(p, "_test.go") {

return nil

}

+ if exempt(p) {

+ return nil

+ }

src, err := os.ReadFile(p)

if err != nil {

return err

}

+ scanned++

for _, pat := range pats {

if pat.Match(src) {

hits = append(hits, p+" matches "+pat.String())

@@ -211,6 +253,12 @@ func TestNoOutboundNetworkInAppCode(t *testing.T) {

t.Fatal(err)

}

+ // Vacuity guard. An exempt() that matched everything, or a walk root

+ // that moved, would leave this gate scanning nothing and passing —

+ // which is how a gate stops guarding without anyone noticing.

+ if scanned < 5 {

+ t.Fatalf("only %d files scanned; the gate is not looking at the app", scanned)

+ }

for _, h := range hits {

t.Errorf("app code can reach the network: %s", h)

}

diff --git a/internal/memoriatest/harness_test.go b/internal/memoriatest/harness_test.go

index 61d9c54..ecabc58 100644

--- a/internal/memoriatest/harness_test.go

+++ b/internal/memoriatest/harness_test.go

@@ -72,6 +72,26 @@ func newAppAt(t *testing.T, path string) (http.Handler, string) {

return mux, path

}

+// newAppWithStore is newApp with the store exposed, for tests about

+// ingestion itself rather than about the pages that call it. Connectors

+// reach the store directly — there is no form that carries a Notion

+// page's ancestry — so the paths they use need a way to be tested.

+func newAppWithStore(t *testing.T) (http.Handler, *memoria.Store, string) {

+ t.Helper()

+ path := filepath.Join(t.TempDir(), "app.db")

+ logger := slog.New(slog.NewTextHandler(io.Discard, nil))

+ d, err := db.Open(path, logger)

+ if err != nil {

+ t.Fatalf("db.Open: %v", err)

+ }

+ t.Cleanup(func() { d.Close() })

+ mux, a, err := memoria.AppForTest(d, testOrigin, testDEK(t), logger)

+ if err != nil {

+ t.Fatalf("AppForTest: %v", err)

+ }

+ return mux, a.Store(), path

// newAppWithLog is newApp with the log captured, for the gate that

// asserts no content reaches a log line.

diff --git a/tools/drive/main.go b/tools/drive/main.go

index 50b8e0d..938f360 100644

--- a/tools/drive/main.go

+++ b/tools/drive/main.go

@@ -24,9 +24,12 @@ import (

"os"

"path/filepath"

+ "time"

"github.com/carlosframework/rastrillo/db"

memoria "amadan.net/oficina/memoria/internal/memoria"

+ "amadan.net/oficina/memoria/internal/memoria/notion"

"amadan.net/oficina/memoria/internal/memoria/seal"

)

@@ -67,7 +70,7 @@ func main() {

panic(err)

}

- mux, _, err := memoria.AppForTest(d, "http://"+*addr, sealer, logger)

+ mux, app, err := memoria.AppForTest(d, "http://"+*addr, sealer, logger)

if err != nil {

panic(err)

}

@@ -76,6 +79,12 @@ func main() {

// trail, a highlight, a document list with more than one row.

seedViaHTTP(mux, *addr)

+ // And a connector with a finished import behind it, so the source

+ // pages are worth looking at. A template test proves the markup

+ // exists; it cannot show that a status page reads sensibly with real

+ // numbers in it.

+ seedConnector(app)

fmt.Printf("drive serving on http://%s (no sign-in)\n", *addr)

if err := http.ListenAndServe(*addr, mux); err != nil {

panic(err)

@@ -102,3 +111,55 @@ func seedViaHTTP(h http.Handler, addr string) {

}

+// driveSource is a small stand-in for Notion, so the connector pages have

+// a real tree and real counts behind them.

+type driveSource struct{ tree map[string][]notion.Ref }

+func (d *driveSource) Roots(ctx context.Context) ([]notion.Ref, error) {

+ return d.tree[""], nil

+func (d *driveSource) Fetch(ctx context.Context, ref notion.Ref) (*notion.Content, error) {

+ return &notion.Content{

+ Markdown: "## " + ref.Title + "\n\nThis page was imported from Notion during a drive run. " +

+ "It exists so the connector screens have something real to render.",

+ Children: d.tree[ref.ID],

+ UnreadBlocks: 2,

+ }, nil

+func seedConnector(app interface {

+ Store() *memoria.Store

+ UseFakeSource(memoria.SourceClient, func(context.Context, string) (string, error))

+ WaitForSync(int64, time.Duration) bool

+}) {

+ src := &driveSource{tree: map[string][]notion.Ref{

+ "": {{ID: "handbook", Title: "Handbook", EditedTime: "t1", Kind: notion.KindPage}},

+ "handbook": {

+ {ID: "onboarding", Title: "Onboarding", Path: []string{"Handbook"}, EditedTime: "t1", Kind: notion.KindPage},

+ {ID: "expenses", Title: "Expenses", Path: []string{"Handbook"}, EditedTime: "t1", Kind: notion.KindPage},

+ },

+ "onboarding": {

+ {ID: "week-one", Title: "Week one", Path: []string{"Handbook", "Onboarding"}, EditedTime: "t1", Kind: notion.KindPage},

+ },

+ }}

+ app.UseFakeSource(src, func(ctx context.Context, token string) (string, error) { return "drive-workspace", nil })

+ ctx := context.Background()

+ if err := app.Store().SaveConnector(ctx, "notion", "ntn_drive_token", "drive-workspace"); err != nil {

+ fmt.Fprintln(os.Stderr, "seed connector:", err)

+ return

+ }

+ c, err := app.Store().Connector(ctx, "notion")

+ if err != nil {

+ fmt.Fprintln(os.Stderr, "seed connector:", err)

+ return

+ }

+ run, err := app.Store().StartRun(ctx, c.ID)

+ if err != nil {

+ fmt.Fprintln(os.Stderr, "seed connector:", err)

+ return

+ }

+ memoria.NewSyncer(app.Store(), slog.New(slog.NewTextHandler(os.Stderr,

+ &slog.HandlerOptions{Level: slog.LevelWarn}))).RunWith(ctx, src, c.ID, run.ID)

diff --git a/tools/seed/main.go b/tools/seed/main.go

index c668eba..2f2a1af 100644

--- a/tools/seed/main.go

+++ b/tools/seed/main.go

@@ -112,7 +112,9 @@ func main() {

title := rel

// The relative path is the stable key, so re-seeding after an

// edit updates the document rather than adding a second copy.

- if _, err := store.Ingest(ctx, *source, rel, title, string(raw)); err != nil {

+ if _, err := store.Ingest(ctx, memoria.IngestSpec{

+ Source: *source, Key: rel, Title: title, Body: string(raw),

+ }); err != nil {

// Fatal, not a skip. An earlier version counted these and

// carried on, so a missing table produced 59 skips and a

// cheerful "0 documents seeded" summary.