amadan

oficina / memoria Public

🤖 feat(notion): a Notion client that enumerates by traversal, not search

Stage 2 of Notion ingest: the API client, pure and DB-free, driven
entirely by a fake server built from the published reference.

The central decision is that /v1/search is NOT the enumerator. Notion
documents that search "is not guaranteed to return everything" and is
"not optimized for exhaustively enumerating through all the documents
that a bot has access to" — and the response carries no field saying a
result was partial. An enumerator built on it can page cleanly to
has_more:false, miss pages, and report a complete import with no error
anywhere, which is this codebase's worst failure shape arriving by
design rather than by bug.

So search is used for the one thing Notion does guarantee — "pages or
databases directly shared with a connection are guaranteed to be
returned" — and everything below a root is found by walking block
children and data-source queries, which enumerate a parent's actual
children and have a real cursor contract. Ancestry then falls out for
free: you arrive at each page from its parent, so nothing walks parent
links upward, nothing hops block_id parents, and no path truncates
silently when an ancestor is unshared. The fake's search returns only
roots, so the gate for this can actually fail.

Pinned to Notion-Version 2026-03-11, because the object model changed
incompatibly at 2025-09-03 (databases now hold data sources) and again
at 2026-03-11 (archived -> in_trash). An unpinned client is one release
from extracting nothing and calling it success.

The decoder is total: has_more without a cursor, a repeating cursor, a
retry budget exhausted, or a body that will not parse are all errors.
There is no partial return anywhere in the package's surface, so a
caller cannot mistake "we got some" for "that is all there is". 429 and
529 both honour Retry-After — 529 is Notion's service_overload, is not a
standard status, and is documented to be retried exactly like a 429.

TestNoOutboundNetworkInAppCode is narrowed rather than deleted. One
directory is exempt, and pays for it with a runtime guard stronger than
the grep: the transport refuses any host or scheme but the pinned pair,
and refuses redirects outright. Both matter — the token rides in an
Authorization header, Go only drops that across hosts, and a same-host
redirect keeps it.

Mutation testing found the host-pin gate could not fail. The transport
had a second check that rejected non-loopback plaintext hosts, and the
test's target tripped that instead — so deleting the host pin left the
test green. That is the neighbouring-error-path failure AGENTS.md opens
with, reproduced exactly. The scheme is now pinned as its own field with
its own check and its own gate, and the host test targets the same live
listener under a different name so only the host pin can reject it.

All thirteen gates here were broken, watched go red, and reverted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VauJkQsPeGRErbWFiQNZzC
Paul Campbell pushed by paul@keymail.dev 8feda980d04178449f85f77f4a21ffc41140a2a5 parent 74cc335
6 files changed, +1731 −0
  • internal/memoria/notion/client.go +332 −0
  • internal/memoria/notion/client_test.go +353 −0
  • internal/memoria/notion/fake_test.go +315 −0
  • internal/memoria/notion/markdown.go +325 −0
  • internal/memoria/notion/traverse.go +358 −0
  • internal/memoriatest/gates_test.go +48 −0
diff --git a/internal/memoria/notion/client.go b/internal/memoria/notion/client.go
new file mode 100644
index 0000000..ccbfced
--- /dev/null
+++ b/internal/memoria/notion/client.go
@@ -0,0 +1,332 @@
+// Package notion speaks the Notion API and nothing else.
+//
+// It holds no database handle, no sealer and no app type. That boundary
+// is the testable one: everything here is exercised against a fake HTTP
+// server, and the app layer above never parses JSON.
+//
+// This is the ONLY package in memoria permitted to reach the network.
+// TestNoOutboundNetworkInAppCode forbids it everywhere else and exempts
+// exactly this directory; the exemption is paid for by the host pin
+// below, which is a runtime guard rather than a grep over source.
+package notion
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "net"
+ "net/http"
+ "strconv"
+ "time"
+)
+
+// BaseURL is where the API lives, and it is a CONSTANT.
+//
+// Nothing configurable feeds it. The token travels in an Authorization
+// header, so a base URL built from stored or submitted input would make
+// memoria a token courier for whoever could write that input.
+const BaseURL = "https://api.notion.com"
+
+// APIVersion is pinned, and the pin is load-bearing rather than tidy.
+//
+// Version 2025-09-03 changed the object model incompatibly: a database
+// now contains data sources, a row's parent is a data_source_id, and
+// /v1/databases/{id}/query was replaced. 2026-03-11 further renamed
+// `archived` to `in_trash` and the `transcription` block type to
+// `meeting_notes`. An unpinned client is one Notion release away from
+// extracting nothing and reporting success.
+const APIVersion = "2026-03-11"
+
+// Rate limits, from Notion's published request limits: an average of
+// three requests per second per connection, bursts allowed. We pace
+// below the average rather than at it, because the second limit is
+// per-workspace and shared with every other connection the team runs —
+// so the budget we are actually spending is not ours alone.
+const defaultInterval = 350 * time.Millisecond
+
+// retryBudget bounds retries. On exhaustion the client returns an error;
+// it NEVER returns what it managed to collect. A partial result that
+// looks like a whole one is how an import reports success while missing
+// pages.
+const retryBudget = 5
+
+// Code is a closed set of error codes. It is a code and never a message
+// because Notion's own error bodies carry content: a 404 reads "Could
+// not find database with ID: be907abe-… shared with your connection "My
+// Connection"", which is a page id and a workspace name one %w away from
+// a log line AGENTS.md forbids.
+type Code string
+
+const (
+ CodeUnauthorized Code = "unauthorized"
+ CodeRateLimited Code = "rate_limited"
+ CodeTraversalFailed Code = "traversal_failed"
+ CodeExtractFailed Code = "extract_failed"
+ CodeNetwork Code = "network"
+ CodeForbiddenHost Code = "forbidden_host"
+)
+
+// Error carries a code and an HTTP status. Deliberately no message and
+// no body: a status is a fact about the transport, a body is content.
+type Error struct {
+ Code Code
+ Status int
+}
+
+func (e *Error) Error() string {
+ if e.Status != 0 {
+ return fmt.Sprintf("notion: %s (http %d)", e.Code, e.Status)
+ }
+ return "notion: " + string(e.Code)
+}
+
+// CodeOf returns the code of an error, or CodeNetwork.
+func CodeOf(err error) Code {
+ var e *Error
+ if errors.As(err, &e) {
+ return e.Code
+ }
+ return CodeNetwork
+}
+
+// pinnedTransport refuses every host but the one it was built for.
+//
+// This is the runtime half of the network gate. Go already drops the
+// Authorization header across a cross-host redirect, which is not
+// enough: a SAME-host redirect keeps it, and a base URL that had been
+// tampered with would never be cross-host in the first place. So the
+// check is on every request, not only on redirects — and redirects are
+// refused outright below.
+//
+// The scheme is pinned alongside the host rather than special-cased to
+// "https unless loopback". The special case looked harmless and was not:
+// it rejected a wrong HOST that also happened to be non-loopback, which
+// meant the gate for the host pin passed with the host pin deleted — the
+// neighbouring-error-path failure AGENTS.md opens with. Two fields, two
+// checks, one mutation each.
+type pinnedTransport struct {
+ host string
+ scheme string
+ next http.RoundTripper
+}
+
+func (p *pinnedTransport) RoundTrip(r *http.Request) (*http.Response, error) {
+ if r.URL.Hostname() != p.host {
+ return nil, &Error{Code: CodeForbiddenHost}
+ }
+ if r.URL.Scheme != p.scheme {
+ return nil, &Error{Code: CodeForbiddenHost}
+ }
+ return p.next.RoundTrip(r)
+}
+
+// Client is a Notion API client.
+type Client struct {
+ token string
+ base string
+ http *http.Client
+ interval time.Duration
+ last time.Time
+ now func() time.Time
+ sleep func(time.Duration)
+}
+
+// New builds a client against the real API.
+func New(token string) (*Client, error) {
+ return newClient(token, BaseURL, defaultInterval)
+}
+
+// NewForTest builds a client against an arbitrary base URL, so the
+// package can be driven by an httptest server.
+//
+// Named so it cannot be reached from production by accident, exactly as
+// memoria.AppForTest is, and held to the same standard: the host pin is
+// still applied — to the test server's host — so the transport's refusal
+// path is the one under test rather than a bypassed one.
+func NewForTest(token, base string) (*Client, error) {
+ return newClient(token, base, 0)
+}
+
+func newClient(token, base string, interval time.Duration) (*Client, error) {
+ if token == "" {
+ return nil, &Error{Code: CodeUnauthorized}
+ }
+ host, scheme, err := hostOf(base)
+ if err != nil {
+ return nil, err
+ }
+ return &Client{
+ token: token,
+ base: base,
+ interval: interval,
+ now: time.Now,
+ sleep: time.Sleep,
+ http: &http.Client{
+ Timeout: 60 * time.Second,
+ Transport: &pinnedTransport{host: host, scheme: scheme, next: http.DefaultTransport},
+ CheckRedirect: func(*http.Request, []*http.Request) error {
+ // Refuse every redirect. There is no legitimate one on
+ // this API, and following one is the cheapest way to
+ // hand a bearer token to somewhere it was not meant for.
+ return &Error{Code: CodeForbiddenHost}
+ },
+ },
+ }, nil
+}
+
+func hostOf(base string) (host, scheme string, err error) {
+ req, err := http.NewRequest(http.MethodGet, base, nil)
+ if err != nil {
+ return "", "", &Error{Code: CodeForbiddenHost}
+ }
+ h, sc := req.URL.Hostname(), req.URL.Scheme
+ if h == "" || sc == "" {
+ return "", "", &Error{Code: CodeForbiddenHost}
+ }
+ return h, sc, nil
+}
+
+// pace holds the client below the published rate limit.
+func (c *Client) pace() {
+ if c.interval <= 0 {
+ return
+ }
+ if wait := c.interval - c.now().Sub(c.last); wait > 0 {
+ c.sleep(wait)
+ }
+ c.last = c.now()
+}
+
+// post issues one API call and decodes into out, retrying only what
+// Notion says is retryable.
+func (c *Client) post(ctx context.Context, path string, body any, out any) error {
+ return c.call(ctx, http.MethodPost, path, body, out)
+}
+
+func (c *Client) get(ctx context.Context, path string, out any) error {
+ return c.call(ctx, http.MethodGet, path, nil, out)
+}
+
+func (c *Client) call(ctx context.Context, method, path string, body, out any) error {
+ var payload []byte
+ if body != nil {
+ var err error
+ if payload, err = json.Marshal(body); err != nil {
+ return &Error{Code: CodeExtractFailed}
+ }
+ }
+ for attempt := 0; ; attempt++ {
+ c.pace()
+
+ var rdr io.Reader
+ if payload != nil {
+ rdr = bytes.NewReader(payload)
+ }
+ req, err := http.NewRequestWithContext(ctx, method, c.base+path, rdr)
+ if err != nil {
+ return &Error{Code: CodeNetwork}
+ }
+ req.Header.Set("Authorization", "Bearer "+c.token)
+ req.Header.Set("Notion-Version", APIVersion)
+ if payload != nil {
+ req.Header.Set("Content-Type", "application/json")
+ }
+
+ res, err := c.http.Do(req)
+ if err != nil {
+ // A pinned-host or redirect refusal is not retryable: it is
+ // a statement about where this request was going.
+ var pe *Error
+ if errors.As(err, &pe) && pe.Code == CodeForbiddenHost {
+ return pe
+ }
+ if ctx.Err() != nil {
+ return &Error{Code: CodeNetwork}
+ }
+ var ne net.Error
+ if attempt < retryBudget && errors.As(err, &ne) {
+ c.backoff(attempt, 0)
+ continue
+ }
+ return &Error{Code: CodeNetwork}
+ }
+
+ retryable, retryAfter := classify(res.StatusCode, res.Header.Get("Retry-After"))
+ if retryable {
+ res.Body.Close()
+ if attempt < retryBudget {
+ c.backoff(attempt, retryAfter)
+ continue
+ }
+ // Budget exhausted. Return the error — never the partial
+ // work. See retryBudget.
+ code := CodeTraversalFailed
+ if res.StatusCode == http.StatusTooManyRequests || res.StatusCode == 529 {
+ code = CodeRateLimited
+ }
+ return &Error{Code: code, Status: res.StatusCode}
+ }
+
+ if res.StatusCode == http.StatusUnauthorized || res.StatusCode == http.StatusForbidden {
+ res.Body.Close()
+ return &Error{Code: CodeUnauthorized, Status: res.StatusCode}
+ }
+ if res.StatusCode < 200 || res.StatusCode > 299 {
+ res.Body.Close()
+ return &Error{Code: CodeTraversalFailed, Status: res.StatusCode}
+ }
+
+ // Cap the body. An unbounded ReadAll on a response we do not
+ // control is a memory bug waiting for a bad day.
+ raw, err := io.ReadAll(io.LimitReader(res.Body, 32<<20))
+ res.Body.Close()
+ if err != nil {
+ return &Error{Code: CodeNetwork}
+ }
+ if out == nil {
+ return nil
+ }
+ if err := json.Unmarshal(raw, out); err != nil {
+ // A body we cannot parse is NOT an empty result. Returning
+ // nil here would turn a malformed page into "no children",
+ // which is the plausible-empty-list failure this codebase
+ // ranks worst.
+ return &Error{Code: CodeTraversalFailed, Status: res.StatusCode}
+ }
+ return nil
+ }
+}
+
+// classify says whether a status is worth retrying, and for how long.
+//
+// 429 and 529 both carry Retry-After in integer seconds — 529 is
+// Notion's "service_overload" and is documented to be retried the same
+// way as a 429, which is easy to miss because it is not a standard code.
+// 5xx are retried because every request this package makes is a read.
+func classify(status int, retryAfter string) (bool, time.Duration) {
+ switch status {
+ case http.StatusTooManyRequests, 529:
+ if s, err := strconv.Atoi(retryAfter); err == nil && s >= 0 {
+ return true, time.Duration(s) * time.Second
+ }
+ return true, 0
+ case http.StatusInternalServerError, http.StatusBadGateway,
+ http.StatusServiceUnavailable, http.StatusGatewayTimeout:
+ return true, 0
+ }
+ return false, 0
+}
+
+func (c *Client) backoff(attempt int, after time.Duration) {
+ if c.interval <= 0 && after == 0 {
+ return // tests: no real waiting
+ }
+ if after > 0 {
+ c.sleep(after)
+ return
+ }
+ c.sleep(time.Duration(1<<attempt) * 200 * time.Millisecond)
+}
diff --git a/internal/memoria/notion/client_test.go b/internal/memoria/notion/client_test.go
new file mode 100644
index 0000000..3e4c163
--- /dev/null
+++ b/internal/memoria/notion/client_test.go
@@ -0,0 +1,353 @@
+package notion
+
+import (
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+)
+
+func client(t *testing.T, f *fakeNotion) *Client {
+ t.Helper()
+ srv := f.start()
+ c, err := NewForTest("ntn_test", srv.URL)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return c
+}
+
+// A small workspace: a root with a child and a grandchild, none of which
+// search will admit to except the root.
+func nestedFake(t *testing.T) *fakeNotion {
+ f := newFake(t)
+ f.page("root", "Handbook", "", heading(1, "Welcome"), para("Everything a new starter needs."))
+ f.page("kid", "Onboarding", "root", para("Week one covers expenses and travel booking."))
+ f.page("grandkid", "Equipment", "kid", para("Laptops are ordered through the request form."))
+ return f
+}
+
+// TestSearchIsNotUsedAsAnEnumerator is the gate for the whole traversal
+// design, and the reason draft 1 of the spec was rewritten.
+//
+// Notion documents that /v1/search "is not guaranteed to return
+// everything" and is "not optimized for exhaustively enumerating through
+// all the documents that a bot has access to" — and it returns NO field
+// saying a result was partial. So an enumerator built on search can page
+// cleanly to has_more:false, miss pages, and report a complete import,
+// with no error anywhere. The fake models exactly that: its search
+// returns only roots.
+//
+// MUTATION-TESTED 2026-09-05: replacing the traversal in walkAll with
+// "use the search results as the page list" finds 1 page instead of 3
+// and fails here. Reverted.
+func TestSearchIsNotUsedAsAnEnumerator(t *testing.T) {
+ f := nestedFake(t)
+ c := client(t, f)
+
+ refs, _, err := walkAll(t, c)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(refs) != 3 {
+ var names []string
+ for _, r := range refs {
+ names = append(names, r.Title)
+ }
+ t.Fatalf("found %d pages %v, want 3 — search returned only the root, so anything "+
+ "missing here was never traversed to", len(refs), names)
+ }
+ byID := map[string]Ref{}
+ for _, r := range refs {
+ byID[r.ID] = r
+ }
+ if got := strings.Join(byID["grandkid"].Path, " › "); got != "Handbook › Onboarding" {
+ t.Errorf("grandchild ancestry is %q, want %q", got, "Handbook › Onboarding")
+ }
+ if got := strings.Join(byID["root"].Path, " › "); got != "" {
+ t.Errorf("root ancestry is %q, want empty", got)
+ }
+}
+
+// TestTraversalIncompleteFailsTheRun. Every enumeration yields everything
+// or it fails. A partial return that looks whole is how an import reports
+// success while missing pages — this codebase's worst failure shape.
+//
+// MUTATION-TESTED 2026-09-05: returning the accumulated results with a
+// nil error on retry exhaustion, or treating has_more-without-cursor as
+// the end of the list, makes each subtest pass silently. Reverted.
+func TestTraversalIncompleteFailsTheRun(t *testing.T) {
+ t.Run("has_more with no cursor", func(t *testing.T) {
+ f := nestedFake(t)
+ f.hasMoreNoCursor["blocks:root"] = true
+ _, _, err := walkAll(t, client(t, f))
+ if err == nil {
+ t.Fatal("an unfinishable enumeration was reported as complete")
+ }
+ if CodeOf(err) != CodeTraversalFailed {
+ t.Errorf("code %q, want %q", CodeOf(err), CodeTraversalFailed)
+ }
+ })
+
+ t.Run("repeating cursor", func(t *testing.T) {
+ f := nestedFake(t)
+ f.loopCursor["blocks:root"] = true
+ _, _, err := walkAll(t, client(t, f))
+ if err == nil {
+ t.Fatal("a looping cursor did not terminate as an error")
+ }
+ })
+
+ t.Run("rate limited past the budget", func(t *testing.T) {
+ f := nestedFake(t)
+ f.alwaysRateLimit["blocks:kid"] = true
+ _, _, err := walkAll(t, client(t, f))
+ if err == nil {
+ t.Fatal("exhausting the retry budget returned no error")
+ }
+ if CodeOf(err) != CodeRateLimited {
+ t.Errorf("code %q, want %q", CodeOf(err), CodeRateLimited)
+ }
+ })
+
+ // 529 is Notion's "service_overload". It is not a standard status, is
+ // easy to miss, and is documented to be retried exactly like a 429.
+ t.Run("529 service_overload past the budget", func(t *testing.T) {
+ f := nestedFake(t)
+ f.overload["blocks:kid"] = true
+ _, _, err := walkAll(t, client(t, f))
+ if err == nil {
+ t.Fatal("a 529 past the budget returned no error")
+ }
+ if CodeOf(err) != CodeRateLimited {
+ t.Errorf("code %q, want %q — 529 is retried and reported like a 429",
+ CodeOf(err), CodeRateLimited)
+ }
+ })
+}
+
+// TestClientDialsOnlyNotion is the runtime half of the network gate. It
+// is what pays for internal/memoria/notion being exempt from the source
+// grep that forbids outbound calls everywhere else.
+//
+// The token rides in an Authorization header. Go drops that header across
+// a cross-host redirect, which is not enough on its own: a SAME-host
+// redirect keeps it, and a tampered base URL is never cross-host to begin
+// with. So the pin is checked on every request and redirects are refused
+// outright.
+//
+// MUTATION-TESTED 2026-09-05: returning p.next.RoundTrip unconditionally
+// from pinnedTransport, or returning nil from CheckRedirect, makes the
+// corresponding subtest fail. Reverted.
+func TestClientDialsOnlyNotion(t *testing.T) {
+ // The target here is "localhost" against a server pinned to
+ // "127.0.0.1": a DIFFERENT HOST that is otherwise entirely reachable,
+ // same scheme, same port, live listener.
+ //
+ // The first version of this subtest pointed at http://127.0.0.2:1 and
+ // was worthless. That address failed the transport's *scheme* check
+ // as well, so deleting the host pin left the test green — a
+ // neighbouring error path supplying the failure, which is the exact
+ // pattern AGENTS.md opens with. Found by mutation-testing this gate.
+ t.Run("another host is refused", func(t *testing.T) {
+ f := newFake(t)
+ srv := f.start()
+ c, err := NewForTest("ntn_test", srv.URL)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // Point the client at the same listener by another name, as a
+ // tampered credential row or a mis-set constant would.
+ c.base = strings.Replace(srv.URL, "127.0.0.1", "localhost", 1)
+ if c.base == srv.URL {
+ t.Fatal("could not build a different-host URL; this subtest proves nothing")
+ }
+ if _, err := c.Roots(t.Context()); err == nil {
+ t.Fatal("the client called a host it was not pinned to")
+ } else if CodeOf(err) != CodeForbiddenHost {
+ t.Errorf("code %q, want %q", CodeOf(err), CodeForbiddenHost)
+ }
+ })
+
+ t.Run("another scheme is refused", func(t *testing.T) {
+ f := newFake(t)
+ srv := f.start()
+ c, err := NewForTest("ntn_test", srv.URL)
+ if err != nil {
+ t.Fatal(err)
+ }
+ c.base = strings.Replace(srv.URL, "http://", "https://", 1)
+ if _, err := c.Roots(t.Context()); err == nil {
+ t.Fatal("the client accepted a scheme it was not pinned to")
+ } else if CodeOf(err) != CodeForbiddenHost {
+ t.Errorf("code %q, want %q", CodeOf(err), CodeForbiddenHost)
+ }
+ })
+
+ t.Run("a same-host redirect is refused", func(t *testing.T) {
+ var landed bool
+ mux := http.NewServeMux()
+ mux.HandleFunc("/v1/search", func(w http.ResponseWriter, r *http.Request) {
+ http.Redirect(w, r, "/collect", http.StatusFound)
+ })
+ mux.HandleFunc("/collect", func(w http.ResponseWriter, r *http.Request) {
+ // If we get here holding the bearer token, the redirect was
+ // followed and the token went with it.
+ if r.Header.Get("Authorization") != "" {
+ landed = true
+ }
+ writeJSON(w, map[string]any{"object": "list", "results": []any{}, "has_more": false})
+ })
+ srv := httptest.NewServer(mux)
+ defer srv.Close()
+
+ c, err := NewForTest("ntn_test", srv.URL)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := c.Roots(t.Context()); err == nil {
+ t.Error("a redirect was followed; the bearer token travels with it")
+ }
+ if landed {
+ t.Error("the token reached the redirect target")
+ }
+ })
+}
+
+// TestNotionVersionIsPinned. The object model changed incompatibly at
+// 2025-09-03 and again at 2026-03-11; an unpinned client is one release
+// from extracting nothing and calling it success. The fake refuses any
+// request that does not carry the pin.
+//
+// MUTATION-TESTED 2026-09-05: dropping the Notion-Version header makes
+// this fail. Reverted.
+func TestNotionVersionIsPinned(t *testing.T) {
+ f := newFake(t)
+ c := client(t, f)
+ var out map[string]any
+ if err := c.get(t.Context(), "/v1/users/me", &out); err != nil {
+ t.Fatalf("the pinned version was rejected by the API: %v", err)
+ }
+ if APIVersion == "" {
+ t.Fatal("no API version is pinned")
+ }
+}
+
+// TestUnknownBlocksAreCountedNotSwallowed. Notion adds block types.
+// Failing a run on an unrecognised one breaks memoria on Notion's release
+// schedule; rendering nothing and saying nothing makes a page look thin
+// for no stated reason.
+//
+// MUTATION-TESTED 2026-09-05: deleting the `*unread++` in render's
+// default branch makes this fail. Reverted.
+func TestUnknownBlocksAreCountedNotSwallowed(t *testing.T) {
+ f := newFake(t)
+ f.page("root", "Notes", "",
+ para("A readable paragraph."),
+ unknownBlock("some_future_block"),
+ unknownBlock("another_new_thing"))
+ c := client(t, f)
+
+ _, got, err := walkAll(t, c)
+ if err != nil {
+ t.Fatal(err)
+ }
+ content := got["root"]
+ if content.UnreadBlocks != 2 {
+ t.Errorf("UnreadBlocks = %d, want 2", content.UnreadBlocks)
+ }
+ if !strings.Contains(content.Markdown, "A readable paragraph.") {
+ t.Error("the readable block was lost alongside the unreadable ones")
+ }
+}
+
+// TestChildPagesAreNotInlined. A child page is its own document. Inlining
+// it would duplicate every one of its passages into every ancestor, so a
+// search would return the same text several times under different titles.
+func TestChildPagesAreNotInlined(t *testing.T) {
+ f := nestedFake(t)
+ _, got, err := walkAll(t, client(t, f))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if strings.Contains(got["root"].Markdown, "Week one covers expenses") {
+ t.Error("the child page's body was inlined into its parent")
+ }
+ if !strings.Contains(got["kid"].Markdown, "Week one covers expenses") {
+ t.Error("the child page's own body is missing")
+ }
+}
+
+// TestMarkdownShape covers the block renderings a Notion page actually
+// leans on, because the chunker downstream keys on Markdown headings.
+func TestMarkdownShape(t *testing.T) {
+ f := newFake(t)
+ f.page("root", "Doc", "",
+ heading(1, "Revenue"),
+ para("June was strong."),
+ map[string]any{"object": "block", "id": "t1", "type": "to_do", "has_children": false,
+ "to_do": map[string]any{"checked": true, "rich_text": []map[string]any{{"plain_text": "Ship it"}}}},
+ map[string]any{"object": "block", "id": "c1", "type": "code", "has_children": false,
+ "code": map[string]any{"language": "go", "rich_text": []map[string]any{{"plain_text": "x := 1"}}}},
+ map[string]any{"object": "block", "id": "q1", "type": "quote", "has_children": false,
+ "quote": map[string]any{"rich_text": []map[string]any{{"plain_text": "As agreed."}}}},
+ )
+ _, got, err := walkAll(t, client(t, f))
+ if err != nil {
+ t.Fatal(err)
+ }
+ md := got["root"].Markdown
+ for _, want := range []string{"# Revenue", "June was strong.", "- [x] Ship it", "```go", "x := 1", "> As agreed."} {
+ if !strings.Contains(md, want) {
+ t.Errorf("markdown missing %q\n---\n%s", want, md)
+ }
+ }
+}
+
+// TestDatabaseRowsBecomePagesUnderTheDatabase.
+func TestDatabaseRowsBecomePagesUnderTheDatabase(t *testing.T) {
+ f := newFake(t)
+ f.page("root", "Workspace", "")
+ f.pages["root"].childDBs = []string{"db1"}
+ f.dbs["db1"] = []string{"ds1"}
+ f.dbParent["db1"] = "root"
+ f.page("row1", "Project Bluebird", "")
+ f.page("row2", "Project Kestrel", "")
+ f.roots = []string{"root"} // rows are reached through the database only
+ f.rows["ds1"] = []string{"row1", "row2"}
+
+ refs, _, err := walkAll(t, client(t, f))
+ if err != nil {
+ t.Fatal(err)
+ }
+ found := map[string][]string{}
+ for _, r := range refs {
+ found[r.Title] = r.Path
+ }
+ for _, title := range []string{"Project Bluebird", "Project Kestrel"} {
+ path, ok := found[title]
+ if !ok {
+ t.Fatalf("database row %q was not reached", title)
+ }
+ if got := strings.Join(path, " › "); got != "Workspace › Projects" {
+ t.Errorf("row %q ancestry is %q, want %q", title, got, "Workspace › Projects")
+ }
+ }
+}
+
+// TestTrashedPagesAreSkipped. `in_trash` replaced `archived` at API
+// version 2026-03-11.
+func TestTrashedPagesAreSkipped(t *testing.T) {
+ f := nestedFake(t)
+ f.pages["kid"].inTrash = true
+ f.roots = []string{"root", "kid"}
+ refs, _, err := walkAll(t, client(t, f))
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, r := range refs {
+ if r.ID == "kid" && len(r.Path) == 0 {
+ t.Error("a trashed page was admitted as a root")
+ }
+ }
+}
diff --git a/internal/memoria/notion/fake_test.go b/internal/memoria/notion/fake_test.go
new file mode 100644
index 0000000..cd5fc93
--- /dev/null
+++ b/internal/memoria/notion/fake_test.go
@@ -0,0 +1,315 @@
+package notion
+
+import (
+ "encoding/json"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+)
+
+// fakeNotion is a Notion workspace built from the published API
+// reference, with switches for each failure the client must survive.
+//
+// Its /v1/search returns ONLY the roots, never the pages nested beneath
+// them. That is not a simplification — it is the behaviour Notion
+// documents ("search is not guaranteed to return everything", "not
+// optimized for exhaustively enumerating"), and modelling it is what
+// makes TestSearchIsNotUsedAsAnEnumerator able to fail.
+type fakeNotion struct {
+ t *testing.T
+ pages map[string]*fakePage
+ roots []string
+ dbs map[string][]string // database id -> data source ids
+ rows map[string][]string // data source id -> row page ids
+ // dbParent is the page a database lives under. Notion sets a child
+ // database's parent to its containing page, and getting this wrong
+ // in the fake made a nested database look like a root — a real
+ // distinction, since Roots drops any result whose parent is also in
+ // the result set, and the root version carries no ancestry.
+ dbParent map[string]string
+
+ // Failure switches.
+ hasMoreNoCursor map[string]bool // path -> emit has_more with no cursor
+ loopCursor map[string]bool // path -> emit the same cursor forever
+ alwaysRateLimit map[string]bool // path -> 429 with Retry-After: 0
+ notFound map[string]bool // page id -> 404
+ overload map[string]bool // path -> 529
+ calls map[string]int
+ searchLeaks []string // extra ids search returns beyond the roots
+}
+
+type fakePage struct {
+ id string
+ title string
+ edited string
+ parent string
+ inTrash bool
+ blocks []map[string]any
+ children []string // child_page ids
+ childDBs []string // child_database ids
+}
+
+func newFake(t *testing.T) *fakeNotion {
+ return &fakeNotion{
+ t: t, pages: map[string]*fakePage{},
+ dbs: map[string][]string{}, rows: map[string][]string{}, dbParent: map[string]string{},
+ hasMoreNoCursor: map[string]bool{}, loopCursor: map[string]bool{},
+ alwaysRateLimit: map[string]bool{}, notFound: map[string]bool{},
+ overload: map[string]bool{}, calls: map[string]int{},
+ }
+}
+
+func (f *fakeNotion) page(id, title, parent string, blocks ...map[string]any) *fakePage {
+ p := &fakePage{id: id, title: title, parent: parent, edited: "2026-09-01T10:00:00.000Z", blocks: blocks}
+ f.pages[id] = p
+ if parent == "" {
+ f.roots = append(f.roots, id)
+ } else if pp, ok := f.pages[parent]; ok {
+ pp.children = append(pp.children, id)
+ }
+ return p
+}
+
+func para(text string) map[string]any {
+ return map[string]any{
+ "object": "block", "id": "b-" + text[:min(6, len(text))], "type": "paragraph",
+ "has_children": false,
+ "paragraph": map[string]any{"rich_text": []map[string]any{{"plain_text": text}}},
+ }
+}
+
+func heading(level int, text string) map[string]any {
+ k := "heading_" + string(rune('0'+level))
+ return map[string]any{
+ "object": "block", "id": "h-" + text[:min(6, len(text))], "type": k,
+ "has_children": false,
+ k: map[string]any{"rich_text": []map[string]any{{"plain_text": text}}},
+ }
+}
+
+func unknownBlock(kind string) map[string]any {
+ return map[string]any{"object": "block", "id": "u-" + kind, "type": kind, "has_children": false}
+}
+
+func min(a, b int) int {
+ if a < b {
+ return a
+ }
+ return b
+}
+
+func (f *fakeNotion) object(id string) map[string]any {
+ p := f.pages[id]
+ return map[string]any{
+ "object": "page", "id": p.id, "in_trash": p.inTrash,
+ "last_edited_time": p.edited,
+ "parent": parentOf(p.parent),
+ "properties": map[string]any{
+ "Name": map[string]any{"type": "title", "title": []map[string]any{{"plain_text": p.title}}},
+ },
+ }
+}
+
+func parentOf(id string) map[string]any {
+ if id == "" {
+ return map[string]any{"type": "workspace", "workspace": true}
+ }
+ return map[string]any{"type": "page_id", "page_id": id}
+}
+
+// paged writes a cursor-paginated list response, honouring the switches.
+func (f *fakeNotion) paged(w http.ResponseWriter, path string, items []map[string]any, cursor string) {
+ if f.hasMoreNoCursor[path] {
+ writeJSON(w, map[string]any{"object": "list", "results": items, "has_more": true, "next_cursor": nil})
+ return
+ }
+ if f.loopCursor[path] {
+ writeJSON(w, map[string]any{"object": "list", "results": items, "has_more": true, "next_cursor": "stuck"})
+ return
+ }
+ // Real pagination: one item per page, so the cursor path is actually
+ // exercised rather than assumed.
+ if cursor == "" && len(items) > 1 {
+ writeJSON(w, map[string]any{"object": "list", "results": items[:1], "has_more": true, "next_cursor": "c1"})
+ return
+ }
+ if strings.HasPrefix(cursor, "c") {
+ n := int(cursor[1] - '0')
+ if n < len(items) {
+ rest := items[n:]
+ if len(rest) > 1 {
+ writeJSON(w, map[string]any{"object": "list", "results": rest[:1], "has_more": true,
+ "next_cursor": "c" + string(rune('0'+n+1))})
+ return
+ }
+ writeJSON(w, map[string]any{"object": "list", "results": rest, "has_more": false, "next_cursor": nil})
+ return
+ }
+ }
+ writeJSON(w, map[string]any{"object": "list", "results": items, "has_more": false, "next_cursor": nil})
+}
+
+func writeJSON(w http.ResponseWriter, v any) {
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(v)
+}
+
+func (f *fakeNotion) start() *httptest.Server {
+ mux := http.NewServeMux()
+
+ mux.HandleFunc("/v1/search", func(w http.ResponseWriter, r *http.Request) {
+ f.calls["search"]++
+ if f.reject(w, "search") {
+ return
+ }
+ var body map[string]any
+ _ = json.NewDecoder(r.Body).Decode(&body)
+ cursor, _ := body["start_cursor"].(string)
+
+ var items []map[string]any
+ // ONLY the roots. Children exist and search does not return them.
+ for _, id := range f.roots {
+ items = append(items, f.object(id))
+ }
+ for _, id := range f.searchLeaks {
+ items = append(items, f.object(id))
+ }
+ for db := range f.dbs {
+ items = append(items, map[string]any{
+ "object": "data_source", "id": db, "last_edited_time": "2026-09-01T10:00:00.000Z",
+ "name": "Projects", "parent": parentOf(f.dbParent[db]),
+ })
+ }
+ f.paged(w, "search", items, cursor)
+ })
+
+ mux.HandleFunc("/v1/blocks/", func(w http.ResponseWriter, r *http.Request) {
+ id := strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/v1/blocks/"), "/children")
+ f.calls["blocks:"+id]++
+ if f.reject(w, "blocks:"+id) {
+ return
+ }
+ if f.notFound[id] {
+ w.WriteHeader(http.StatusNotFound)
+ // Notion's real 404 body quotes the id and the connection
+ // name. Kept verbatim in shape so the log gate sees a body
+ // that actually carries a coordinate.
+ writeJSON(w, map[string]any{"object": "error", "status": 404, "code": "object_not_found",
+ "message": "Could not find block with ID: " + id + ". Make sure the relevant pages and " +
+ "databases are shared with your connection \"CANARYCONNECTION\"."})
+ return
+ }
+ p, ok := f.pages[id]
+ if !ok {
+ w.WriteHeader(http.StatusNotFound)
+ writeJSON(w, map[string]any{"object": "error", "status": 404})
+ return
+ }
+ items := append([]map[string]any{}, p.blocks...)
+ for _, c := range p.children {
+ items = append(items, map[string]any{
+ "object": "block", "id": c, "type": "child_page", "has_children": true,
+ "child_page": map[string]any{"title": f.pages[c].title},
+ })
+ }
+ for _, d := range p.childDBs {
+ items = append(items, map[string]any{
+ "object": "block", "id": d, "type": "child_database", "has_children": true,
+ "child_database": map[string]any{"title": "Projects"},
+ })
+ }
+ f.paged(w, "blocks:"+id, items, r.URL.Query().Get("start_cursor"))
+ })
+
+ mux.HandleFunc("/v1/databases/", func(w http.ResponseWriter, r *http.Request) {
+ id := strings.TrimPrefix(r.URL.Path, "/v1/databases/")
+ f.calls["db:"+id]++
+ sources, ok := f.dbs[id]
+ if !ok {
+ w.WriteHeader(http.StatusNotFound)
+ writeJSON(w, map[string]any{"object": "error", "status": 404})
+ return
+ }
+ var ds []map[string]any
+ for _, s := range sources {
+ ds = append(ds, map[string]any{"id": s, "name": "Projects"})
+ }
+ writeJSON(w, map[string]any{"object": "database", "id": id, "data_sources": ds})
+ })
+
+ mux.HandleFunc("/v1/data_sources/", func(w http.ResponseWriter, r *http.Request) {
+ id := strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/v1/data_sources/"), "/query")
+ f.calls["ds:"+id]++
+ if f.reject(w, "ds:"+id) {
+ return
+ }
+ var items []map[string]any
+ for _, rid := range f.rows[id] {
+ items = append(items, f.object(rid))
+ }
+ var body map[string]any
+ _ = json.NewDecoder(r.Body).Decode(&body)
+ cursor, _ := body["start_cursor"].(string)
+ f.paged(w, "ds:"+id, items, cursor)
+ })
+
+ mux.HandleFunc("/v1/users/me", func(w http.ResponseWriter, r *http.Request) {
+ if r.Header.Get("Notion-Version") != APIVersion {
+ w.WriteHeader(http.StatusBadRequest)
+ writeJSON(w, map[string]any{"object": "error", "code": "missing_version"})
+ return
+ }
+ writeJSON(w, map[string]any{"object": "user", "id": "bot", "name": "memoria"})
+ })
+
+ srv := httptest.NewServer(mux)
+ f.t.Cleanup(srv.Close)
+ return srv
+}
+
+func (f *fakeNotion) reject(w http.ResponseWriter, key string) bool {
+ if f.alwaysRateLimit[key] {
+ w.Header().Set("Retry-After", "0")
+ w.WriteHeader(http.StatusTooManyRequests)
+ writeJSON(w, map[string]any{"object": "error", "code": "rate_limited"})
+ return true
+ }
+ if f.overload[key] {
+ w.Header().Set("Retry-After", "0")
+ w.WriteHeader(529)
+ writeJSON(w, map[string]any{"object": "error", "code": "service_overload"})
+ return true
+ }
+ return false
+}
+
+// walkAll drives a full traversal the way the connector will, returning
+// every page reached, in order.
+func walkAll(t *testing.T, c *Client) ([]Ref, map[string]*Content, error) {
+ t.Helper()
+ roots, err := c.Roots(t.Context())
+ if err != nil {
+ return nil, nil, err
+ }
+ var out []Ref
+ got := map[string]*Content{}
+ seen := map[string]bool{}
+ queue := append([]Ref{}, roots...)
+ for len(queue) > 0 {
+ ref := queue[0]
+ queue = queue[1:]
+ if seen[ref.ID] {
+ continue
+ }
+ seen[ref.ID] = true
+ content, err := c.Fetch(t.Context(), ref)
+ if err != nil {
+ return out, got, err
+ }
+ out = append(out, ref)
+ got[ref.ID] = content
+ queue = append(queue, content.Children...)
+ }
+ return out, got, nil
+}
diff --git a/internal/memoria/notion/markdown.go b/internal/memoria/notion/markdown.go
new file mode 100644
index 0000000..4ae4721
--- /dev/null
+++ b/internal/memoria/notion/markdown.go
@@ -0,0 +1,325 @@
+package notion
+
+import (
+ "encoding/json"
+ "errors"
+ "strconv"
+ "strings"
+)
+
+func asErr(err error, target **Error) bool { return errors.As(err, target) }
+
+// block is one Notion block. The payload for a block's own type lives
+// under a key named after the type, so it is decoded on demand from raw
+// rather than modelled as sixty optional fields.
+type block struct {
+ ID string `json:"id"`
+ Type string `json:"type"`
+ HasChildren bool `json:"has_children"`
+ InTrash bool `json:"in_trash"`
+
+ raw json.RawMessage
+ children []block
+}
+
+// payload decodes the type-specific body of a block.
+func (b block) payload() blockBody {
+ var wrap map[string]json.RawMessage
+ if err := json.Unmarshal(b.raw, &wrap); err != nil {
+ return blockBody{}
+ }
+ var body blockBody
+ if v, ok := wrap[b.Type]; ok {
+ _ = json.Unmarshal(v, &body)
+ }
+ return body
+}
+
+type blockBody struct {
+ RichText []richText `json:"rich_text"`
+ Caption []richText `json:"caption"`
+ Title string `json:"title"`
+ Language string `json:"language"`
+ Checked bool `json:"checked"`
+ URL string `json:"url"`
+ Name string `json:"name"`
+ External struct {
+ URL string `json:"url"`
+ } `json:"external"`
+ File struct {
+ URL string `json:"url"`
+ } `json:"file"`
+ Cells [][]richText `json:"cells"`
+}
+
+// render turns a page's blocks into Markdown, collecting the child pages
+// it must not inline.
+//
+// childPath is the ancestry a nested page inherits: this page's own path
+// plus this page's title. Traversal knows it because it arrived here from
+// the parent, which is why nothing walks parent links upward.
+func render(blocks []block, childPath []string) (string, []Ref, int) {
+ var b strings.Builder
+ var kids []Ref
+ unread := 0
+ renderInto(&b, blocks, childPath, &kids, &unread, 0)
+ return strings.TrimSpace(b.String()), kids, unread
+}
+
+func renderInto(b *strings.Builder, blocks []block, childPath []string, kids *[]Ref, unread *int, depth int) {
+ numbered := 0
+ for _, blk := range blocks {
+ if blk.InTrash {
+ continue
+ }
+ p := blk.payload()
+ text := plain(p.RichText)
+ indent := strings.Repeat(" ", depth)
+
+ if blk.Type != "numbered_list_item" {
+ numbered = 0
+ }
+
+ switch blk.Type {
+ case "paragraph":
+ writeLine(b, indent+text)
+ case "heading_1":
+ writeLine(b, "# "+text)
+ case "heading_2":
+ writeLine(b, "## "+text)
+ case "heading_3":
+ writeLine(b, "### "+text)
+ case "bulleted_list_item", "toggle":
+ writeLine(b, indent+"- "+text)
+ case "numbered_list_item":
+ numbered++
+ writeLine(b, indent+strconv.Itoa(numbered)+". "+text)
+ case "to_do":
+ mark := " "
+ if p.Checked {
+ mark = "x"
+ }
+ writeLine(b, indent+"- ["+mark+"] "+text)
+ case "quote":
+ writeLine(b, indent+"> "+text)
+ case "callout":
+ writeLine(b, indent+"> "+text)
+ case "code":
+ writeLine(b, "```"+p.Language)
+ writeLine(b, text)
+ writeLine(b, "```")
+ case "divider":
+ writeLine(b, "---")
+ case "table_row":
+ var cells []string
+ for _, c := range p.Cells {
+ cells = append(cells, plain(c))
+ }
+ writeLine(b, "| "+strings.Join(cells, " | ")+" |")
+ case "table", "column_list", "column", "synced_block":
+ // Containers with no text of their own. Their children carry
+ // the content and are rendered below.
+ case "image", "file", "video", "pdf", "audio", "meeting_notes", "embed", "bookmark":
+ // memoria indexes text. A caption or filename is text; the
+ // asset is not, and its signed URL expires anyway.
+ if label := firstNonEmpty(plain(p.Caption), p.Name, p.Title); label != "" {
+ writeLine(b, indent+label)
+ }
+ case "child_page", "child_database":
+ // NOT inlined: its own document. Inlining would duplicate
+ // every passage into all of its ancestors.
+ title := firstNonEmpty(p.Title, text, "Untitled")
+ ref := Ref{ID: blk.ID, Title: title, Path: childPath, Kind: KindPage}
+ if blk.Type == "child_database" {
+ ref.Kind, ref.DataSourceID = KindDataSource, blk.ID
+ }
+ *kids = append(*kids, ref)
+ writeLine(b, indent+"- "+title)
+ default:
+ // Notion adds block types. Failing the run on an unknown one
+ // would break memoria on Notion's release schedule; silence
+ // would make a page look thin for no stated reason. Count it.
+ *unread++
+ if text != "" {
+ writeLine(b, indent+text)
+ }
+ }
+
+ if len(blk.children) > 0 {
+ nested := depth + 1
+ if blk.Type == "table" || blk.Type == "column_list" || blk.Type == "column" || blk.Type == "synced_block" {
+ nested = depth
+ }
+ renderInto(b, blk.children, childPath, kids, unread, nested)
+ }
+ }
+}
+
+func writeLine(b *strings.Builder, s string) {
+ if strings.TrimSpace(s) == "" {
+ return
+ }
+ b.WriteString(s)
+ b.WriteString("\n\n")
+}
+
+// Properties renders a database row's properties as a leading key/value
+// block, so a row carries its own columns into the passage that cites it.
+//
+// A row page's body alone is usually empty — the information is in the
+// columns — so without this a 40-row database becomes 40 blank documents.
+func Properties(raw json.RawMessage) string {
+ if len(raw) == 0 {
+ return ""
+ }
+ var props map[string]json.RawMessage
+ if err := json.Unmarshal(raw, &props); err != nil {
+ return ""
+ }
+ names := make([]string, 0, len(props))
+ for k := range props {
+ names = append(names, k)
+ }
+ sortStrings(names)
+
+ var b strings.Builder
+ for _, name := range names {
+ v := renderProperty(props[name])
+ if v == "" {
+ continue
+ }
+ b.WriteString(name)
+ b.WriteString(": ")
+ b.WriteString(v)
+ b.WriteString("\n")
+ }
+ return strings.TrimSpace(b.String())
+}
+
+func renderProperty(raw json.RawMessage) string {
+ var p struct {
+ Type string `json:"type"`
+ Title []richText `json:"title"`
+ RichText []richText `json:"rich_text"`
+ Number *float64 `json:"number"`
+ Checkbox *bool `json:"checkbox"`
+ URL string `json:"url"`
+ Email string `json:"email"`
+ Phone string `json:"phone_number"`
+ Select *struct {
+ Name string `json:"name"`
+ } `json:"select"`
+ MultiSelect []struct {
+ Name string `json:"name"`
+ } `json:"multi_select"`
+ Status *struct {
+ Name string `json:"name"`
+ } `json:"status"`
+ Date *struct {
+ Start string `json:"start"`
+ End string `json:"end"`
+ } `json:"date"`
+ People []struct {
+ Name string `json:"name"`
+ } `json:"people"`
+ Formula json.RawMessage `json:"formula"`
+ Rollup json.RawMessage `json:"rollup"`
+ }
+ if err := json.Unmarshal(raw, &p); err != nil {
+ return ""
+ }
+ switch p.Type {
+ case "title":
+ return plain(p.Title)
+ case "rich_text":
+ return plain(p.RichText)
+ case "number":
+ if p.Number == nil {
+ return ""
+ }
+ return strconv.FormatFloat(*p.Number, 'f', -1, 64)
+ case "checkbox":
+ if p.Checkbox != nil && *p.Checkbox {
+ return "yes"
+ }
+ return "no"
+ case "url":
+ return p.URL
+ case "email":
+ return p.Email
+ case "phone_number":
+ return p.Phone
+ case "select":
+ if p.Select == nil {
+ return ""
+ }
+ return p.Select.Name
+ case "status":
+ if p.Status == nil {
+ return ""
+ }
+ return p.Status.Name
+ case "multi_select":
+ var names []string
+ for _, s := range p.MultiSelect {
+ names = append(names, s.Name)
+ }
+ return strings.Join(names, ", ")
+ case "date":
+ if p.Date == nil {
+ return ""
+ }
+ if p.Date.End != "" {
+ return p.Date.Start + " – " + p.Date.End
+ }
+ return p.Date.Start
+ case "people":
+ var names []string
+ for _, s := range p.People {
+ names = append(names, s.Name)
+ }
+ return strings.Join(names, ", ")
+ case "formula", "rollup":
+ return scalarOf(p.Formula) + scalarOf(p.Rollup)
+ }
+ return ""
+}
+
+// scalarOf pulls a displayable value out of a formula or rollup result,
+// which nests its answer under its own result type.
+func scalarOf(raw json.RawMessage) string {
+ if len(raw) == 0 {
+ return ""
+ }
+ var v struct {
+ Type string `json:"type"`
+ String string `json:"string"`
+ Number *float64 `json:"number"`
+ Date *struct {
+ Start string `json:"start"`
+ } `json:"date"`
+ Boolean *bool `json:"boolean"`
+ }
+ if err := json.Unmarshal(raw, &v); err != nil {
+ return ""
+ }
+ switch {
+ case v.String != "":
+ return v.String
+ case v.Number != nil:
+ return strconv.FormatFloat(*v.Number, 'f', -1, 64)
+ case v.Date != nil:
+ return v.Date.Start
+ case v.Boolean != nil && *v.Boolean:
+ return "yes"
+ }
+ return ""
+}
+
+func sortStrings(s []string) {
+ for i := 1; i < len(s); i++ {
+ for j := i; j > 0 && s[j] < s[j-1]; j-- {
+ s[j], s[j-1] = s[j-1], s[j]
+ }
+ }
+}
diff --git a/internal/memoria/notion/traverse.go b/internal/memoria/notion/traverse.go
new file mode 100644
index 0000000..1ed5cd5
--- /dev/null
+++ b/internal/memoria/notion/traverse.go
@@ -0,0 +1,358 @@
+package notion
+
+import (
+ "context"
+ "encoding/json"
+ "strings"
+)
+
+// Ref is one page memoria may hold: a coordinate plus the ancestry it
+// was reached through.
+type Ref struct {
+ // ID is the Notion page id. A coordinate, and the caller seals it.
+ ID string
+ // Title and Path are CONTENT. Never log them.
+ Title string
+ Path []string
+ // EditedTime is Notion's last_edited_time, the change detector.
+ EditedTime string
+ // Kind distinguishes an ordinary page from a data-source row, which
+ // the extractor renders differently.
+ Kind Kind
+ // DataSourceID is set when Kind is KindDataSource: the thing to
+ // query for rows.
+ DataSourceID string
+}
+
+type Kind string
+
+const (
+ KindPage Kind = "page"
+ KindDataSource Kind = "data_source"
+)
+
+// Content is one page's extracted body plus what lies beneath it.
+type Content struct {
+ Markdown string
+ // Children are the pages and data sources nested directly under this
+ // one, with their Path already resolved — traversal knows the
+ // ancestry because it arrived from the parent.
+ Children []Ref
+ // UnreadBlocks counts blocks of a type this package does not render.
+ // Reported rather than swallowed: a page that came in thin then has
+ // a reason attached instead of looking like a thin page.
+ UnreadBlocks int
+}
+
+// list is one page of a paginated Notion response.
+type list struct {
+ Results []json.RawMessage `json:"results"`
+ NextCursor *string `json:"next_cursor"`
+ HasMore bool `json:"has_more"`
+}
+
+// object is the shared head of every Notion object we read.
+type object struct {
+ Object string `json:"object"`
+ ID string `json:"id"`
+ InTrash bool `json:"in_trash"`
+ Parent parent `json:"parent"`
+ Properties json.RawMessage `json:"properties"`
+ Title []richText `json:"title"`
+ Name string `json:"name"`
+ LastEdited string `json:"last_edited_time"`
+}
+
+type parent struct {
+ Type string `json:"type"`
+ PageID string `json:"page_id"`
+ DatabaseID string `json:"database_id"`
+ DataSourceID string `json:"data_source_id"`
+ BlockID string `json:"block_id"`
+ Workspace bool `json:"workspace"`
+}
+
+// id returns the parent's id whatever kind it is, so "is my parent in
+// this set?" is one lookup rather than four.
+func (p parent) id() string {
+ switch {
+ case p.PageID != "":
+ return p.PageID
+ case p.DataSourceID != "":
+ return p.DataSourceID
+ case p.DatabaseID != "":
+ return p.DatabaseID
+ case p.BlockID != "":
+ return p.BlockID
+ }
+ return ""
+}
+
+type richText struct {
+ PlainText string `json:"plain_text"`
+}
+
+func plain(rt []richText) string {
+ var b strings.Builder
+ for _, r := range rt {
+ b.WriteString(r.PlainText)
+ }
+ return b.String()
+}
+
+// paginate walks every page of a cursor-paginated endpoint and applies
+// the total-decoder rules. It returns everything or it returns an error;
+// there is no partial result, because a partial result that looks whole
+// is how an import reports success while missing pages.
+func (c *Client) paginate(ctx context.Context, method, path string, body map[string]any, fn func(json.RawMessage) error) error {
+ seen := map[string]bool{}
+ cursor := ""
+ for {
+ var page list
+ var err error
+ if method == "POST" {
+ b := map[string]any{"page_size": 100}
+ for k, v := range body {
+ b[k] = v
+ }
+ if cursor != "" {
+ b["start_cursor"] = cursor
+ }
+ err = c.post(ctx, path, b, &page)
+ } else {
+ p := path + "?page_size=100"
+ if cursor != "" {
+ p += "&start_cursor=" + cursor
+ }
+ err = c.get(ctx, p, &page)
+ }
+ if err != nil {
+ return err
+ }
+ for _, r := range page.Results {
+ if err := fn(r); err != nil {
+ return err
+ }
+ }
+ if !page.HasMore {
+ return nil
+ }
+ // has_more with no cursor is unfinishable. Treating it as the
+ // end would silently truncate the enumeration.
+ if page.NextCursor == nil || *page.NextCursor == "" {
+ return &Error{Code: CodeTraversalFailed}
+ }
+ // A repeating cursor is a loop. Without this the run never
+ // finishes and never says why.
+ if seen[*page.NextCursor] {
+ return &Error{Code: CodeTraversalFailed}
+ }
+ seen[*page.NextCursor] = true
+ cursor = *page.NextCursor
+ }
+}
+
+// Roots returns the pages and data sources to traverse from.
+//
+// Search is used HERE AND ONLY HERE, and only for what Notion guarantees
+// it does: "any pages or databases that are directly shared with a
+// connection are guaranteed to be returned". Notion documents that
+// search is NOT an exhaustive enumerator — "search is not guaranteed to
+// return everything" — and returns no field saying a result was partial,
+// so a run that paged cleanly to the end could be missing pages and
+// would report a complete import. Everything below a root is found by
+// traversal instead, which enumerates a parent's actual children.
+//
+// A result whose parent is also in the result set is not a root: it will
+// be reached by traversal, with its ancestry known. A result whose
+// parent is absent IS a root, even if it is nested in Notion — we cannot
+// see above it, so its ancestry starts here and the status page says how
+// many roots were found.
+func (c *Client) Roots(ctx context.Context) ([]Ref, error) {
+ var objs []object
+ err := c.paginate(ctx, "POST", "/v1/search", nil, func(raw json.RawMessage) error {
+ var o object
+ if err := json.Unmarshal(raw, &o); err != nil {
+ return &Error{Code: CodeTraversalFailed}
+ }
+ objs = append(objs, o)
+ return nil
+ })
+ if err != nil {
+ return nil, err
+ }
+
+ present := make(map[string]bool, len(objs))
+ for _, o := range objs {
+ present[o.ID] = true
+ }
+
+ var roots []Ref
+ for _, o := range objs {
+ if o.InTrash {
+ continue
+ }
+ if pid := o.Parent.id(); pid != "" && present[pid] {
+ continue // reachable by traversal, with ancestry
+ }
+ r := Ref{ID: o.ID, EditedTime: o.LastEdited, Kind: KindPage}
+ switch o.Object {
+ case "data_source":
+ r.Kind, r.DataSourceID = KindDataSource, o.ID
+ r.Title = firstNonEmpty(plain(o.Title), o.Name)
+ case "database":
+ // A database is a container of data sources, not a document.
+ // Its data sources are resolved when it is traversed.
+ r.Kind, r.DataSourceID = KindDataSource, o.ID
+ r.Title = firstNonEmpty(plain(o.Title), o.Name)
+ default:
+ r.Title = titleOf(o)
+ }
+ if r.Title == "" {
+ r.Title = "Untitled"
+ }
+ roots = append(roots, r)
+ }
+ return roots, nil
+}
+
+// titleOf pulls a page's title out of its properties. The title property
+// is identified by its TYPE, not its name: Notion lets it be renamed and
+// commonly is ("Name", "Task", "Project").
+func titleOf(o object) string {
+ if len(o.Title) > 0 {
+ return plain(o.Title)
+ }
+ if len(o.Properties) == 0 {
+ return ""
+ }
+ var props map[string]struct {
+ Type string `json:"type"`
+ Title []richText `json:"title"`
+ }
+ if err := json.Unmarshal(o.Properties, &props); err != nil {
+ return ""
+ }
+ for _, p := range props {
+ if p.Type == "title" {
+ return plain(p.Title)
+ }
+ }
+ return ""
+}
+
+func firstNonEmpty(vals ...string) string {
+ for _, v := range vals {
+ if strings.TrimSpace(v) != "" {
+ return v
+ }
+ }
+ return ""
+}
+
+// Fetch reads one page: its body as Markdown, and the pages nested
+// directly under it with their ancestry already resolved.
+func (c *Client) Fetch(ctx context.Context, ref Ref) (*Content, error) {
+ if ref.Kind == KindDataSource {
+ return c.fetchDataSource(ctx, ref)
+ }
+ blocks, err := c.blocks(ctx, ref.ID)
+ if err != nil {
+ return nil, err
+ }
+ childPath := append(append([]string{}, ref.Path...), ref.Title)
+ md, kids, unread := render(blocks, childPath)
+ return &Content{Markdown: md, Children: kids, UnreadBlocks: unread}, nil
+}
+
+// fetchDataSource turns a database into its rows.
+//
+// The database itself is not a document — it is a container — so its
+// Markdown is empty and its rows are its children. Under API version
+// 2025-09-03 and later a database holds one or more DATA SOURCES and
+// rows are queried from those, not from the database.
+func (c *Client) fetchDataSource(ctx context.Context, ref Ref) (*Content, error) {
+ sources := []string{ref.DataSourceID}
+ if ids, err := c.dataSourcesOf(ctx, ref.DataSourceID); err == nil && len(ids) > 0 {
+ sources = ids
+ } else if err != nil {
+ return nil, err
+ }
+
+ childPath := append(append([]string{}, ref.Path...), ref.Title)
+ var kids []Ref
+ for _, ds := range sources {
+ err := c.paginate(ctx, "POST", "/v1/data_sources/"+ds+"/query", nil, func(raw json.RawMessage) error {
+ var o object
+ if err := json.Unmarshal(raw, &o); err != nil {
+ return &Error{Code: CodeTraversalFailed}
+ }
+ if o.InTrash {
+ return nil
+ }
+ title := titleOf(o)
+ if title == "" {
+ title = "Untitled"
+ }
+ kids = append(kids, Ref{
+ ID: o.ID, Title: title, Path: childPath,
+ EditedTime: o.LastEdited, Kind: KindPage,
+ })
+ return nil
+ })
+ if err != nil {
+ return nil, err
+ }
+ }
+ return &Content{Children: kids}, nil
+}
+
+// dataSourcesOf resolves a database id to its data source ids. An id
+// that is already a data source answers 404, which is not an error here
+// — it means the caller had the right id all along.
+func (c *Client) dataSourcesOf(ctx context.Context, databaseID string) ([]string, error) {
+ var db struct {
+ DataSources []struct {
+ ID string `json:"id"`
+ } `json:"data_sources"`
+ }
+ if err := c.get(ctx, "/v1/databases/"+databaseID, &db); err != nil {
+ var e *Error
+ if ok := asErr(err, &e); ok && e.Status == 404 {
+ return nil, nil
+ }
+ return nil, err
+ }
+ ids := make([]string, 0, len(db.DataSources))
+ for _, d := range db.DataSources {
+ ids = append(ids, d.ID)
+ }
+ return ids, nil
+}
+
+// blocks reads every block of a page, recursing into nested blocks but
+// NOT into child pages — those are their own documents, and inlining
+// them would duplicate every passage into all of its ancestors.
+func (c *Client) blocks(ctx context.Context, id string) ([]block, error) {
+ var out []block
+ err := c.paginate(ctx, "GET", "/v1/blocks/"+id+"/children", nil, func(raw json.RawMessage) error {
+ var b block
+ if err := json.Unmarshal(raw, &b); err != nil {
+ return &Error{Code: CodeTraversalFailed}
+ }
+ b.raw = raw
+ if b.HasChildren && b.Type != "child_page" && b.Type != "child_database" {
+ kids, err := c.blocks(ctx, b.ID)
+ if err != nil {
+ return err
+ }
+ b.children = kids
+ }
+ out = append(out, b)
+ return nil
+ })
+ if err != nil {
+ return nil, err
+ }
+ return out, nil
+}
diff --git a/internal/memoriatest/gates_test.go b/internal/memoriatest/gates_test.go
index 1d195e0..6eafa2c 100644
--- a/internal/memoriatest/gates_test.go
+++ b/internal/memoriatest/gates_test.go
@@ -89,6 +89,14 @@ func TestPlaintextNeverReachesTheDatabase(t *testing.T) {
//
// MUTATION-TESTED 2026-09-03: adding `http.Get("https://example.com")`
// to Store.Ingest makes this fail naming store.go. Reverted.
+//
+// AMENDED 2026-09-05 for the Notion connector, which must dial
+// api.notion.com. The blanket ban survives; one directory is exempt and
+// pays for the exemption with a runtime host pin. See allowedNetworkDirs.
+//
+// MUTATION-TESTED 2026-09-05: adding an `*http.Client` field to
+// internal/memoria/handlers.go still makes this fail, so narrowing the
+// gate did not blunt it. Reverted.
func TestNoOutboundNetworkInAppCode(t *testing.T) {
// A blacklist, and it is worth saying so: this is a source
// heuristic, not a proof. A review defeated the first version with
@@ -107,7 +115,37 @@ func TestNoOutboundNetworkInAppCode(t *testing.T) {
for _, f := range forbidden {
pats = append(pats, regexp.MustCompile(f))
}
+ // EXACTLY ONE package may reach the network, and this is the list.
+ //
+ // It is a list rather than a deleted gate because the connector had
+ // to dial api.notion.com and the cheapest way to make this test pass
+ // would have been to widen it until it named nothing. The exemption
+ // is paid for by a RUNTIME guard that is stronger than the grep it
+ // replaces here: notion.pinnedTransport refuses every host but the
+ // one it was built for, and the client refuses redirects outright,
+ // both asserted by notion.TestClientDialsOnlyNotion. A source grep
+ // cannot tell an http.Client pointed at Notion from one pointed
+ // anywhere else; that test can.
+ //
+ // Adding a second entry here should be very hard to justify.
+ allowedNetworkDirs := []string{
+ filepath.Join("..", "memoria", "notion"),
+ }
+ if len(allowedNetworkDirs) != 1 {
+ t.Fatalf("%d packages may reach the network, want exactly 1: %v",
+ len(allowedNetworkDirs), allowedNetworkDirs)
+ }
+ exempt := func(p string) bool {
+ for _, d := range allowedNetworkDirs {
+ if strings.HasPrefix(filepath.Clean(p), filepath.Clean(d)+string(filepath.Separator)) {
+ return true
+ }
+ }
+ return false
+ }
+
var hits []string
+ scanned := 0
walk := func(root string) error {
return filepath.Walk(root, func(p string, info os.FileInfo, err error) error {
if err != nil {
@@ -116,10 +154,14 @@ func TestNoOutboundNetworkInAppCode(t *testing.T) {
if info.IsDir() || !strings.HasSuffix(p, ".go") || strings.HasSuffix(p, "_test.go") {
return nil
}
+ if exempt(p) {
+ return nil
+ }
src, err := os.ReadFile(p)
if err != nil {
return err
}
+ scanned++
for _, pat := range pats {
if pat.Match(src) {
hits = append(hits, p+" matches "+pat.String())
@@ -136,6 +178,12 @@ func TestNoOutboundNetworkInAppCode(t *testing.T) {
t.Fatal(err)
}
}
+ // Vacuity guard. An exempt() that matched everything, or a walk root
+ // that moved, would leave this gate scanning nothing and passing —
+ // which is how a gate stops guarding without anyone noticing.
+ if scanned < 5 {
+ t.Fatalf("only %d files scanned; the gate is not looking at the app", scanned)
+ }
for _, h := range hits {
t.Errorf("app code can reach the network: %s", h)
}