oficina / memoria Public

🤖 feat(notion): the connector — credential, ledger, run loop, surfaces

Stage 3, and the last of the three. An owner pastes a read-only Notion
integration token, memoria verifies it against the API before storing it
sealed, and an import walks everything reachable from the pages that
integration was shared with.

Deletion, decided by what memoria is for: a sync never deletes on
absence. memoria is replacing Notion, so archiving a page upstream after
migrating it is the intended workflow — a diff-and-delete connector would
read that as "forget it" and remove from the app of record the document
the migration existed to preserve. Disconnecting removes the token and
the ledger and no documents, and says so before the button is pressed.
Every ledger row records last_seen anyway, so "no longer in Notion" stays
answerable later without a migration.

Errors divide in two and the division is the load-bearing part. A
traversal failure below a page is FATAL, because the pages beneath it
were never enumerated and a run that carried on would report "done" while
missing a subtree — a failure rendered as an absence. A single page that
404s between enumeration and fetch is recorded against that page and the
run continues: one placeholder must not block an import, but it must not
vanish from the totals either.

The run is one pass, not the two phases the spec described. With
traversal, enumeration and extraction are the same API call, so splitting
them would double every request against a limit of three per second. The
spec is amended rather than diverged from, along with what that costs: a
resume repeats the traversal and skips the ingest, and the status page
cannot show "412 of 1,930" because the denominator does not exist until
the walk ends. It says what it knows and says the total is not known yet.

Interrupted runs are reclaimed at boot — sound by construction, since a
process that has just started cannot be running a sync — with a heartbeat
for a run wedged inside a live process. Without that the connector
wedges permanently on its normal path.

requireOwner exists and says out loud that in slice 0 it is equivalent to
requireSession, because MEMORIA_OWNERS is the whole admission list.
Naming a gate for a distinction the app does not have is how a guard that
guards nothing gets shipped. AppForTest grew an injectable identity so
the gate could be written at all rather than merely asserted.

Two bugs came from looking at the rendered page, which no template test
would have found:

  - Ancestry was indexed and never shown. Results carried the in-page
    heading trail only, so two "Week one" pages under different handbooks
    were indistinguishable to a reader. The source path now renders above
    the title.
  - There was no way to rotate a token. The spec said rotation is
    replacing the row and the page offered no form to do it with. Found
    by mutation-testing TestTokenIsNeverRendered, which was green because
    the page it looked at had no token field at all.

Also fixed: tools/drive wrote to a hardcoded absolute path under one
person's home and panicked anywhere else.

Eleven gates here, each broken, watched go red, and reverted — thirty
across the three stages. Two of them were vacuous when first written and
are recorded as such in their comments.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VauJkQsPeGRErbWFiQNZzC

Paul Campbell pushed by paul@keymail.dev 1 month ago c70d6ffd52146b5745c9ca20ae599fb74f24c458 parent 8feda98

22 files changed, +2225 −20

cmd/memoria/main.go +6 −0
docs/superpowers/specs/2026-09-05-notion-ingest-design.md +77 −11
internal/memoria/app.go +105 −1
internal/memoria/connector.go +358 −0
internal/memoria/connector_models.go +103 −0
internal/memoria/handlers.go +20 −0
internal/memoria/handlers_connector.go +289 −0
internal/memoria/migrations/0004_create_connectors.sql +18 −0
internal/memoria/migrations/schema.sql +19 −1
internal/memoria/models.go +3 −0
internal/memoria/notion/verify.go +38 −0
internal/memoria/render.go +1 −1
internal/memoria/seal/seal.go +10 −0
internal/memoria/static/memoria.css +10 −0
internal/memoria/store.go +29 −3
internal/memoria/sync.go +281 −0
internal/memoria/templates/connectors.html +22 −0
internal/memoria/templates/index.html +6 −0
internal/memoria/templates/layout.html +5 −1
internal/memoria/templates/notion.html +139 −0
internal/memoriatest/connector_test.go +620 −0
tools/drive/main.go +66 −2

diff --git a/cmd/memoria/main.go b/cmd/memoria/main.go

index 5001c06..5521e3e 100644

--- a/cmd/memoria/main.go

+++ b/cmd/memoria/main.go

@@ -69,6 +69,12 @@ func main() {

logger.Error("wire sign-in", "err", err)

os.Exit(1)

}

+ // The connector routes check this list. In slice 0 it is the same

+ // list the admission gate uses, so every signed-in person is an

+ // owner — requireOwner says that out loud rather than implying a

+ // restriction the app does not yet have.

+ app.SetOwners(cfg.Owners)

app.UseAuth(a)

app.MountAuth(mux, cfg.Origin)

diff --git a/docs/superpowers/specs/2026-09-05-notion-ingest-design.md b/docs/superpowers/specs/2026-09-05-notion-ingest-design.md

index 069ee62..e62b6b1 100644

--- a/docs/superpowers/specs/2026-09-05-notion-ingest-design.md

+++ b/docs/superpowers/specs/2026-09-05-notion-ingest-design.md

@@ -119,6 +119,49 @@ Findings 1, 2, 3 were blockers; finding 4 was the expensive one.

promises one; `context.WithoutCancel` in `store.go:189` is a bounded

post-commit refresh, not a precedent for a multi-minute job.

+### 1.2 What building it changed

+Draft 2 was written before the code. Five things did not survive contact,

+and they are recorded here rather than quietly diverged from.

+- **One pass, not two phases.** §9 specified enumerate-then-fetch. With

+ traversal they are the same API call — fetching a page returns its body

+ *and* the child pages under it — so running them separately would

+ double every request against a limit of three per second. The loop is

+ one pass; the ledger is still written per page.

+- **Therefore the interruption cost is larger than §9 claimed, and the

+ progress display is different.** "The cost of an interruption is the

+ page in flight" is false: the *ingest* work already done survives, but

+ a resumed run repeats the traversal, because a page's children cannot

+ be discovered without asking and a child's own edit does not change its

+ parent's edit time. And since the total is only known when the walk

+ ends, the status page cannot say "412 of 1,930" mid-run. It says how

+ many pages have been seen so far and that the total is not yet known —

+ `total` stays at the `-1` sentinel until the run completes.

+- **There was no way to rotate a token.** §12 said "rotation is replacing

+ the row" and the page offered no form to do it with; the only route was

+ disconnect and reconnect. Found by mutation-testing

+ `TestTokenIsNeverRendered`, which was passing because the page it

+ looked at had no token field at all.

+- **Ancestry was indexed and never shown.** Search results rendered the

+ in-page heading trail and not the source path, so two pages called

+ "Week one" under different handbooks were indistinguishable to the

+ reader. Carrying ancestry for findability while hiding it from the

+ person reading the answer is half a feature. Found by looking at the

+ rendered page, which is the only thing that would have found it.

+- **The scheme pin became its own check.** §11 described a host pin. The

+ implementation also rejected non-loopback plaintext hosts, and that

+ second condition caught the host-pin gate's test target — so deleting

+ the host pin left the gate green. Host and scheme are now separate

+ fields with separate checks and separate gates.

+Two smaller ones: `tools/drive` wrote to a hardcoded absolute path under

+one person's home directory and panicked anywhere else, now the OS temp

+dir; and the connector page ordered token rotation above import status,

+which put a rare housekeeping act between the reader and the thing the

+page is for.

---

## 2. Why this is unblocked, when the parent sequenced it fourth

@@ -367,6 +410,11 @@ the document's own trail. `Store.Ingest` moves to an options struct —

one production call site (`handlers.go:183`) and one dev tool

(`tools/seed/main.go:115`).

+**Ancestry is also SHOWN**, above the title on every search result, not

+merely indexed. A result carrying only its in-page heading trail leaves

+two "Week one" pages under different handbooks indistinguishable to the

+reader (§1.2).

**The 0.5 is a guess**, chosen as half the body's weight because nothing

has measured it, and the code says so where the constant lives. §15 keeps

it open. What is *not* a guess is that the gate under it must actually

@@ -436,12 +484,26 @@ the outer mux bypasses both, as `/static/` does. The handler starts the

run and redirects to a status page that refreshes with

`<meta http-equiv="refresh">`. Scriptless, per AGENTS.md.

-**Two phases, both checkpointed.** *Enumerate*: traverse from the roots

-(§6), upserting ledger rows and writing `seen_edited_time`,

-`sealed_path`, `path_print` and `last_seen_at`. *Fetch*: walk rows needing

-work (§10), extract, `Ingest`, set `ingested_edited_time` and `state`.

-The checkpoint is the row itself, so an interruption costs the page in

-flight.

+**One pass, checkpointed per page.** Draft 2 specified two phases,

+enumerate then fetch. With traversal they are the same API call —

+fetching a page returns its body *and* the child pages beneath it — so

+separating them would double every request against a limit of three per

+second (§1.2).

+The loop walks from the roots (§6); for each page it writes the ledger

+row first (coordinate, ancestry, `seen_edited`, `last_seen`), then

+fetches, then ingests if §10 says the page needs work, then records the

+outcome (`state`, `ingested_edited`, `ingested_path`).

+Writing the row BEFORE the fetch is deliberate: a page that then fails to

+read has somewhere to be named, instead of being absent.

+**What an interruption actually costs.** The ingest work already done

+survives — a resumed run skips every page that is unchanged. The

+traversal is repeated, because children cannot be discovered without

+asking for them, and a child's own edit does not move its parent's edit

+time. So a resume is cheap in the expensive currency (chunking, sealing,

+indexing) and not free in API calls.

**Single-flight, and the deadlock draft 1 shipped.** A run holds the

connector via a conditional update. Draft 1 called the lock's survival

@@ -542,8 +604,9 @@ team's leaves the instance, at ingest or at query.

sealed under `PurposeCredential`, never rendered back — not even masked,

because a masked secret is still a secret in the DOM. Verified on save

with one `/v1/users/me` call, so a bad token fails at the form rather

-than mid-run. Rotation is replacing the row; there is no key history,

-because a replaced token has no use. **Disconnect removes the row** —

+than mid-run. Rotation is replacing the row through a form on

+the connector page; there is no key history, because a replaced token has

+no use. **Disconnect removes the row** —

parent §3.6 promises "disconnecting" and draft 1 gave no affordance for

it. Disconnecting deletes the credential and stops future syncs; by

decision 3 it deletes no documents, and the confirmation says so in those

@@ -576,9 +639,12 @@ this design could mislead:

1. **"memoria keeps what it imports."** On the setup page, before the

token is entered — parent §3.6 has the wording.

-2. **Interrupted is not progress.** A run stopped by a sleep says

- `interrupted at 412 of 1,930` and offers resume; it never renders as a

- smaller number that looks like it is still going.

+2. **Interrupted is not progress, and a running total is not a total.**

+ A run stopped by a sleep says so and offers resume. A run still going

+ reports how many pages it has SEEN so far and states that the total is

+ not known until it finishes — because with one pass (§1.2) the

+ denominator does not exist until the walk ends. `total` holds `-1`

+ until then and is never rendered as 0.

3. **A count that could not be determined says so** — `total` is `-1`

until enumeration finishes, and renders as "counting", never 0.

diff --git a/internal/memoria/app.go b/internal/memoria/app.go

index e4e8ec6..e444810 100644

--- a/internal/memoria/app.go

+++ b/internal/memoria/app.go

@@ -4,9 +4,11 @@ import (

"context"

"log/slog"

"net/http"

+ "strings"

"github.com/go-chi/chi/v5"

+ "github.com/carlosframework/rastrillo/auth"

"github.com/carlosframework/rastrillo/csrf"

"github.com/carlosframework/rastrillo/db"

"github.com/carlosframework/rastrillo/migrate"

@@ -45,7 +47,23 @@ func App(d *db.DB, origin string, sealer *seal.Sealer, logger *slog.Logger) (*ht

}

logger.Info("index rebuilt", "passages", indexed)

- a := &app{db: d, logger: logger, store: store}

+ // Reclaim runs that were in flight when the process last stopped.

+ //

+ // Sound by construction rather than by timeout: a process that has

+ // just started cannot be running a sync, so anything still marked

+ // in flight was killed by the platform sleeping the app. Without

+ // this the connector wedges permanently on its NORMAL path — the

+ // goroutine dies without writing, the row stays "fetching", and

+ // every later sync is refused as already running.

+ reclaimed, err := store.ReclaimRuns(context.Background())

+ if err != nil {

+ return nil, nil, err

+ }

+ if reclaimed > 0 {

+ logger.Info("reclaimed interrupted syncs", "runs", reclaimed)

+ }

+ a := &app{db: d, logger: logger, store: store, syncer: NewSyncer(store, logger)}

r := chi.NewRouter()

r.Use(csrf.Protect(origin))

@@ -63,6 +81,18 @@ func App(d *db.DB, origin string, sealer *seal.Sealer, logger *slog.Logger) (*ht

r.Post("/documents", a.createDocument)

r.Get("/documents/{id}", a.showDocument)

r.Post("/documents/{id}/delete", a.deleteDocument)

+ // Connector routes sit INSIDE this group, so they inherit

+ // csrf.Protect and requireSession. A route mounted on the outer

+ // mux (as /static/ is) would bypass both.

+ r.Group(func(r chi.Router) {

+ r.Use(a.requireOwner)

+ r.Get("/connectors", a.connectors)

+ r.Get("/connectors/notion", a.notionConnector)

+ r.Post("/connectors/notion", a.saveNotionConnector)

+ r.Post("/connectors/notion/sync", a.startNotionSync)

+ r.Post("/connectors/notion/disconnect", a.disconnectNotion)

+ })

})

mux := http.NewServeMux()

@@ -112,9 +142,72 @@ func (a *app) requireSession(next http.Handler) http.Handler {

})

}

+// requireOwner guards the connector routes.

+//

+// "Owner" is Config.Owners, and IN SLICE 0 THAT IS THE ENTIRE ADMITTED

+// SET — MEMORIA_OWNERS is the admission list, so this is currently

+// equivalent to requireSession. It is written as its own middleware

+// anyway, and this comment says the equivalence out loud, because an

+// earlier draft described the connector pages as "owners only" as though

+// that were a restriction. Naming a gate for a distinction the app does

+// not have is how a guard that guards nothing gets shipped.

+//

+// It fails CLOSED: no identity, or an empty owner list, refuses.

+//

+// Slice 1 replaces the owners list with S7 and there will be no list to

+// check. These routes then need an owner predicate from wherever

+// membership lives, or they lose their guard — recorded in the spec's

+// open items, not left to be discovered.

+func (a *app) requireOwner(next http.Handler) http.Handler {

+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

+ addr, ok := a.identity(r)

+ if !ok || !a.isOwner(addr) {

+ // 404, not 403: AGENTS.md's rule, and a 403 would confirm

+ // the route exists to someone who may not use it.

+ http.NotFound(w, r)

+ return

+ }

+ next.ServeHTTP(w, r)

+ })

+// identity is the signed-in address, or false.

+func (a *app) identity(r *http.Request) (string, bool) {

+ if a.testIdentity != "" {

+ return a.testIdentity, true

+ }

+ id, ok := auth.From(r)

+ if !ok {

+ return "", false

+ }

+ return strings.ToLower(strings.TrimSpace(id.Address)), true

+func (a *app) isOwner(addr string) bool {

+ if len(a.owners) == 0 {

+ return false

+ }

+ for _, o := range a.owners {

+ if o == addr {

+ return true

+ }

+ return false

+// SetOwners supplies the owner list. Called by main from Config.

+func (a *app) SetOwners(owners []string) { a.owners = owners }

// Store exposes the store, for gates that must inspect it.

func (a *app) Store() *Store { return a.store }

+// Syncer exposes the syncer, so a test can drive a run against a fake.

+func (a *app) Syncer() *Syncer { return a.syncer }

+// AsPerson sets the identity AppForTest presents, so a gate can check

+// that a non-owner is refused. Test seam only; production never sets it.

+func (a *app) AsPerson(addr string) { a.testIdentity = addr }

// AppForTest builds the app with a stub session so tests can exercise the

// routes. It is exported, obvious, and takes a testing marker so it

// cannot be reached from production by accident: cmd/memoria does not

@@ -125,5 +218,16 @@ func AppForTest(d *db.DB, origin string, sealer *seal.Sealer, logger *slog.Logge

return nil, nil, err

}

a.allowUnauthenticated = true

+ // An identity, and an owner list containing it.

+ //

+ // AppForTest used to bypass RequireSession entirely, which left no

+ // identity in the request context — so a gate on "is this person an

+ // owner" could not have been written at all, only asserted. Found by

+ // a design review.

+ a.testIdentity = TestOwner

+ a.owners = []string{TestOwner}

return mux, a, nil

}

+// TestOwner is the address AppForTest signs in as.

+const TestOwner = "owner@test"

diff --git a/internal/memoria/connector.go b/internal/memoria/connector.go

new file mode 100644

index 0000000..e3913e3

--- /dev/null

+++ b/internal/memoria/connector.go

@@ -0,0 +1,358 @@

+package memoria

+import (

+ "context"

+ "errors"

+ "fmt"

+ "strings"

+ "time"

+ "gorm.io/gorm"

+ "amadan.net/oficina/memoria/internal/memoria/chunk"

+ "amadan.net/oficina/memoria/internal/memoria/seal"

+// Run states.

+const (

+ RunEnumerating = "enumerating"

+ RunFetching = "fetching"

+ RunDone = "done"

+ RunFailed = "failed"

+ RunInterrupted = "interrupted"

+// Ledger row states.

+const (

+ PagePending = "pending"

+ PageFetched = "fetched"

+ PageEmpty = "empty"

+ PageFailed = "failed"

+// heartbeatStale is how long a run may go without a heartbeat before a

+// LIVE process may reclaim it. A run killed by the platform sleeping is

+// reclaimed at boot instead and does not wait for this.

+const heartbeatStale = 10 * time.Minute

+// ErrSyncInFlight is returned when a run is already active.

+var ErrSyncInFlight = errors.New("a sync is already running")

+// ErrNoConnector is returned when nothing is configured.

+var ErrNoConnector = errors.New("no connector is configured")

+// SaveConnector stores or replaces a connector's credential.

+//

+// Rotation is replacing the row. There is no key history: a replaced

+// token has no remaining use, and keeping one would be a credential

+// retained for no reason.

+func (s *Store) SaveConnector(ctx context.Context, kind, token, workspace string) error {

+ sealed, err := s.sealer.Seal(seal.PurposeCredential, []byte(token))

+ if err != nil {

+ return err

+ }

+ print, err := s.sealer.Fingerprint(seal.PurposeCredential, []byte("workspace\x00"+workspace))

+ if err != nil {

+ return err

+ }

+ now := time.Now().UTC()

+ return s.db.G.WithContext(ctx).Transaction(func(tx *gorm.DB) error {

+ var existing Connector

+ res := tx.Where("kind = ?", kind).First(&existing)

+ if res.Error == nil {

+ // An explicit allowlist, never a bound request body.

+ return tx.Model(&Connector{}).Where("id = ?", existing.ID).Updates(map[string]any{

+ "sealed_token": sealed,

+ "workspace_print": print,

+ "last_verified": now,

+ }).Error

+ }

+ return tx.Create(&Connector{

+ Kind: kind, SealedToken: sealed, WorkspacePrint: print, LastVerified: &now,

+ }).Error

+ })

+// Connector returns a configured connector, without its token.

+func (s *Store) Connector(ctx context.Context, kind string) (*Connector, error) {

+ var c Connector

+ if err := s.db.G.WithContext(ctx).Where("kind = ?", kind).First(&c).Error; err != nil {

+ if errors.Is(err, gorm.ErrRecordNotFound) {

+ return nil, ErrNoConnector

+ }

+ return nil, err

+ }

+ return &c, nil

+// ConnectorToken opens a connector's credential. The only place the

+// plaintext token exists outside the notion client.

+func (s *Store) ConnectorToken(ctx context.Context, kind string) (string, error) {

+ c, err := s.Connector(ctx, kind)

+ if err != nil {

+ return "", err

+ }

+ tok, err := s.sealer.Open(seal.PurposeCredential, c.SealedToken)

+ if err != nil {

+ return "", err

+ }

+ return string(tok), nil

+// DeleteConnector removes the credential and the ledger.

+//

+// It deletes NO DOCUMENTS. What memoria imported, memoria keeps — the

+// content was published into memoria and memoria owns it, so removing it

+// is a separate act performed in memoria. The confirmation copy says so

+// in those words rather than leaving it to be discovered.

+func (s *Store) DeleteConnector(ctx context.Context, kind string) error {

+ return s.db.G.WithContext(ctx).Transaction(func(tx *gorm.DB) error {

+ var c Connector

+ if err := tx.Where("kind = ?", kind).First(&c).Error; err != nil {

+ if errors.Is(err, gorm.ErrRecordNotFound) {

+ return nil

+ }

+ return err

+ }

+ if err := tx.Where("connector_id = ?", c.ID).Delete(&SourcePage{}).Error; err != nil {

+ return err

+ }

+ if err := tx.Where("connector_id = ?", c.ID).Delete(&SyncRun{}).Error; err != nil {

+ return err

+ }

+ return tx.Where("id = ?", c.ID).Delete(&Connector{}).Error

+ })

+// StartRun opens a run, or refuses because one is already active.

+//

+// Single-flight is a conditional insert rather than a mutex, so it

+// survives a restart — but "survives a restart" was the bug in an earlier

+// design, not the feature. The run is a goroutine; when the platform

+// sleeps the process it dies without writing anything, leaving the row

+// "fetching" and every later sync refused as already-running, forever, on

+// the NORMAL path. ReclaimRuns fixes that at boot; the heartbeat below

+// covers a run wedged inside a live process.

+func (s *Store) StartRun(ctx context.Context, connectorID int64) (*SyncRun, error) {

+ var run SyncRun

+ err := s.db.G.WithContext(ctx).Transaction(func(tx *gorm.DB) error {

+ var active []SyncRun

+ if err := tx.Where("connector_id = ? AND state IN ?", connectorID,

+ []string{RunEnumerating, RunFetching}).Find(&active).Error; err != nil {

+ return err

+ }

+ for _, a := range active {

+ if a.Heartbeat != nil && time.Since(*a.Heartbeat) > heartbeatStale {

+ if err := tx.Model(&SyncRun{}).Where("id = ?", a.ID).Updates(map[string]any{

+ "state": RunInterrupted,

+ "error_code": "interrupted",

+ }).Error; err != nil {

+ return err

+ }

+ continue

+ }

+ return ErrSyncInFlight

+ }

+ now := time.Now().UTC()

+ run = SyncRun{

+ ConnectorID: connectorID, State: RunEnumerating,

+ Total: -1, StartedAt: now, Heartbeat: &now,

+ }

+ return tx.Create(&run).Error

+ })

+ if err != nil {

+ return nil, err

+ }

+ return &run, nil

+// ReclaimRuns marks every run that was in flight as interrupted.

+//

+// Called at boot, and sound by construction rather than by timeout: a

+// process that has just started cannot be running a sync, so anything

+// still marked in flight was killed by the platform.

+func (s *Store) ReclaimRuns(ctx context.Context) (int, error) {

+ res := s.db.G.WithContext(ctx).Model(&SyncRun{}).

+ Where("state IN ?", []string{RunEnumerating, RunFetching}).

+ Updates(map[string]any{"state": RunInterrupted, "error_code": "interrupted"})

+ return int(res.RowsAffected), res.Error

+// LatestRun returns the most recent run for a connector.

+func (s *Store) LatestRun(ctx context.Context, connectorID int64) (*SyncRun, error) {

+ var run SyncRun

+ err := s.db.G.WithContext(ctx).Where("connector_id = ?", connectorID).

+ Order("id desc").First(&run).Error

+ if errors.Is(err, gorm.ErrRecordNotFound) {

+ return nil, nil

+ }

+ if err != nil {

+ return nil, err

+ }

+ return &run, nil

+// UpdateRun applies an allowlisted patch to a run.

+func (s *Store) UpdateRun(ctx context.Context, runID int64, fields map[string]any) error {

+ allowed := map[string]bool{

+ "state": true, "roots": true, "total": true, "fetched": true,

+ "skipped": true, "empty": true, "failed_pages": true,

+ "unread_blocks": true, "error_code": true, "heartbeat": true,

+ "finished_at": true,

+ }

+ patch := make(map[string]any, len(fields))

+ for k, v := range fields {

+ if !allowed[k] {

+ return fmt.Errorf("update run: field %q is not allowlisted", k)

+ }

+ patch[k] = v

+ }

+ if len(patch) == 0 {

+ return nil

+ }

+ return s.db.G.WithContext(ctx).Model(&SyncRun{}).Where("id = ?", runID).Updates(patch).Error

+// Beat records that a run is still alive.

+func (s *Store) Beat(ctx context.Context, runID int64) error {

+ now := time.Now().UTC()

+ return s.UpdateRun(ctx, runID, map[string]any{"heartbeat": now})

+// LedgerEntry is one page as the sync loop needs it: the coordinate

+// opened, and enough state to decide whether it needs work.

+type LedgerEntry struct {

+ ID int64

+ PageID string

+ Path []string

+ Title string

+ SeenEdited string

+ IngestedEdited string

+ PathPrint string

+ State string

+ DocumentID int64

+// SeeSourcePage records a page the enumeration reached.

+//

+// Upserts on (connector, page print) and writes what enumeration knows:

+// the coordinate, the ancestry, the edit time it SAW. It never touches

+// IngestedEdited — that column belongs to the fetch phase, and collapsing

+// the two into one makes the comparison compare a value against itself

+// and skip every edit.

+func (s *Store) SeeSourcePage(ctx context.Context, connectorID int64, pageID, title string, path []string, edited string) error {

+ print, err := s.sealer.Fingerprint(seal.PurposeCoordinate, []byte("page\x00"+pageID))

+ if err != nil {

+ return err

+ }

+ sealedID, err := s.sealer.Seal(seal.PurposeCoordinate, []byte(pageID))

+ if err != nil {

+ return err

+ }

+ pathStr := chunk.Chunk{Path: path}.PathString()

+ // Through the shared helper, never inline. Two places computing this

+ // fingerprint differently would make every comparison false and

+ // re-ingest the whole workspace on every sync — or make it always

+ // true and never re-ingest anything. Both fail quietly.

+ pathPrint, err := s.pathPrint(path, title)

+ if err != nil {

+ return err

+ }

+ var sealedPath []byte

+ if pathStr != "" {

+ if sealedPath, err = s.sealer.Seal(seal.PurposePath, []byte(pathStr)); err != nil {

+ return err

+ }

+ now := time.Now().UTC()

+ return s.db.G.WithContext(ctx).Transaction(func(tx *gorm.DB) error {

+ var row SourcePage

+ res := tx.Where("connector_id = ? AND page_print = ?", connectorID, print).First(&row)

+ if res.Error == nil {

+ return tx.Model(&SourcePage{}).Where("id = ?", row.ID).Updates(map[string]any{

+ "sealed_page_id": sealedID,

+ "sealed_path": sealedPath,

+ "path_print": pathPrint,

+ "seen_edited": edited,

+ "last_seen": now,

+ }).Error

+ }

+ return tx.Create(&SourcePage{

+ ConnectorID: connectorID, PagePrint: print, SealedPageID: sealedID,

+ SealedPath: sealedPath, PathPrint: pathPrint, SeenEdited: edited,

+ State: PagePending, LastSeen: now,

+ }).Error

+ })

+// PagesNeedingWork returns the ledger rows the fetch phase must visit.

+//

+// Three conditions, and the second is the one an earlier design missed

+// entirely: an ANCESTOR rename does not change a descendant's own edit

+// time, so without comparing the path print a moved or re-parented page

+// keeps stale ancestry forever and nothing upstream ever signals it.

+func (s *Store) PagesNeedingWork(ctx context.Context, connectorID int64) ([]LedgerEntry, error) {

+ var rows []SourcePage

+ if err := s.db.G.WithContext(ctx).

+ Where("connector_id = ?", connectorID).

+ Order("id").Find(&rows).Error; err != nil {

+ return nil, err

+ }

+ out := make([]LedgerEntry, 0, len(rows))

+ for _, r := range rows {

+ if r.State == PageFetched && r.SeenEdited == r.IngestedEdited && r.PathPrint == r.ingestedPathPrint() {

+ continue

+ }

+ id, err := s.sealer.Open(seal.PurposeCoordinate, r.SealedPageID)

+ if err != nil {

+ return nil, fmt.Errorf("open coordinate %d: %w", r.ID, err)

+ }

+ var path []string

+ if len(r.SealedPath) > 0 {

+ p, err := s.sealer.Open(seal.PurposePath, r.SealedPath)

+ if err != nil {

+ return nil, fmt.Errorf("open path %d: %w", r.ID, err)

+ }

+ if s := string(p); s != "" {

+ path = strings.Split(s, " › ")

+ }

+ out = append(out, LedgerEntry{

+ ID: r.ID, PageID: string(id), Path: path,

+ SeenEdited: r.SeenEdited, IngestedEdited: r.IngestedEdited,

+ PathPrint: r.PathPrint, State: r.State, DocumentID: r.DocumentID,

+ })

+ }

+ return out, nil

+// ingestedPathPrint is the path print as of the last successful ingest.

+func (r SourcePage) ingestedPathPrint() string { return r.IngestedPath }

+// MarkPage records the outcome of fetching one page.

+func (s *Store) MarkPage(ctx context.Context, id int64, state, errorCode string, documentID int64, edited, pathPrint string) error {

+ return s.db.G.WithContext(ctx).Model(&SourcePage{}).Where("id = ?", id).Updates(map[string]any{

+ "state": state,

+ "error_code": errorCode,

+ "document_id": documentID,

+ "ingested_edited": edited,

+ "ingested_path": pathPrint,

+ }).Error

+// CountPages returns how many ledger rows a connector holds, and how many

+// are in each terminal state.

+func (s *Store) CountPages(ctx context.Context, connectorID int64) (total, fetched int, err error) {

+ var t, f int64

+ q := s.db.G.WithContext(ctx).Model(&SourcePage{}).Where("connector_id = ?", connectorID)

+ if err := q.Count(&t).Error; err != nil {

+ return 0, 0, err

+ }

+ if err := s.db.G.WithContext(ctx).Model(&SourcePage{}).

+ Where("connector_id = ? AND state IN ?", connectorID,

+ []string{PageFetched, PageEmpty}).Count(&f).Error; err != nil {

+ return 0, 0, err

+ }

+ return int(t), int(f), nil

diff --git a/internal/memoria/connector_models.go b/internal/memoria/connector_models.go

new file mode 100644

index 0000000..c15b0d2

--- /dev/null

+++ b/internal/memoria/connector_models.go

@@ -0,0 +1,103 @@

+package memoria

+import "time"

+// Connector is one configured source outside the suite.

+//

+// The credential lives here, sealed under its own HKDF purpose, so a

+// disclosure of the chunk store does not also yield the key to the source

+// those chunks came from — the spec's §3.5, which calls least-privilege

+// connector credentials a requirement rather than a nicety.

+type Connector struct {

+ ID int64

+ // Kind is "notion". One row per kind.

+ Kind string `gorm:"uniqueIndex;not null"`

+ // SealedToken is the integration token. Never rendered back, not even

+ // masked: a masked secret is still a secret in the DOM.

+ SealedToken []byte `gorm:"not null"`

+ // WorkspacePrint is a keyed fingerprint of the workspace the token

+ // belongs to, so two connectors pointing at one workspace are

+ // detectable without storing which workspace that is.

+ WorkspacePrint string

+ LastVerified *time.Time

+ CreatedAt time.Time

+ UpdatedAt time.Time

+// SyncRun is one import, and the row that makes it resumable.

+type SyncRun struct {

+ ID int64

+ ConnectorID int64 `gorm:"index;not null"`

+ // State is enumerating | fetching | done | failed | interrupted.

+ State string `gorm:"index;not null"`

+ // Roots is how many top-level pages the integration could see, so

+ // "I expected more than that" is an answerable question.

+ Roots int

+ // Total is -1 until enumeration finishes.

+ //

+ // Not 0. The precedent is handlers.go's chunkCount, which returned 0

+ // on error and made the page assert an absence was real when the app

+ // had merely failed to look. A total that could not be determined is

+ // not a total of zero pages.

+ Total int `gorm:"not null;default:-1"`

+ Fetched int

+ Skipped int

+ Empty int

+ FailedPages int

+ UnreadBlocks int

+ // ErrorCode is a CODE, never a message. Notion's own error bodies

+ // carry content — a 404 quotes the page id and the connection name —

+ // and AGENTS.md forbids content in any error string.

+ ErrorCode string

+ // Heartbeat lets a run wedged inside a LIVE process be reclaimed. A

+ // run killed by the platform sleeping is reclaimed at boot instead,

+ // which is sound by construction rather than by timeout: there can be

+ // no live run at boot.

+ Heartbeat *time.Time

+ StartedAt time.Time

+ FinishedAt *time.Time

+// SourcePage is the ledger: one row per page a connector has seen.

+//

+// Keyed per CONNECTOR and not per run, which is the whole reason a second

+// run resumes instead of restarting.

+type SourcePage struct {

+ ID int64

+ ConnectorID int64 `gorm:"index;not null"`

+ // PagePrint is the keyed page id — lookup, and what joins to a

+ // Document's ExternalID.

+ PagePrint string `gorm:"index;not null"`

+ // SealedPageID is the id itself, sealed and REVERSIBLE.

+ //

+ // Both columns exist because an earlier design stored only the

+ // fingerprint, and seal.Fingerprint is HMAC-SHA256: a resumed run

+ // would have held 1,930 fingerprints and no id to fetch with. Found

+ // by a design review before any of it was written.

+ SealedPageID []byte `gorm:"not null"`

+ // SealedPath is the ancestry as enumeration saw it. PathPrint lets a

+ // move — or an ancestor rename — be detected without opening the row.

+ SealedPath []byte

+ PathPrint string

+ // SeenEdited is what the last enumeration saw; IngestedEdited is what

+ // memoria actually ingested. Two columns, because one column compares

+ // against itself and therefore skips every edit.

+ SeenEdited string

+ IngestedEdited string

+ // IngestedPath is the path print as of the last successful ingest.

+ // A row is up to date only when BOTH the edit time and the ancestry

+ // match what was ingested — an ancestor rename moves the second

+ // without moving the first.

+ IngestedPath string

+ // State is pending | fetched | empty | failed.

+ State string `gorm:"index;not null;default:'pending'"`

+ ErrorCode string

+ // LastSeen is written on every enumeration, so "no longer in the

+ // source" stays answerable without a later migration. memoria does

+ // not delete on absence; it does not follow that it should be unable

+ // to say what has gone.

+ LastSeen time.Time

+ DocumentID int64

+ CreatedAt time.Time

+ UpdatedAt time.Time

diff --git a/internal/memoria/handlers.go b/internal/memoria/handlers.go

index e59ce35..1522719 100644

--- a/internal/memoria/handlers.go

+++ b/internal/memoria/handlers.go

@@ -1,6 +1,7 @@

package memoria

import (

+ "context"

"io"

"log/slog"

"net/http"

@@ -27,10 +28,29 @@ type app struct {

store *Store

auth *auth.Auth

+ syncer *Syncer

+ // owners is the list requireOwner checks. In slice 0 it is the whole

+ // admission list — see requireOwner.

+ owners []string

// allowUnauthenticated is set only by AppForTest. Production cannot

// set it: cmd/memoria never calls that constructor, and

// TestProductionDoesNotUseTheTestConstructor fails if it starts to.

allowUnauthenticated bool

+ // testIdentity is the address AppForTest presents. Same seam, same

+ // guard: production never sets it.

+ testIdentity string

+ // verifyToken and syncClient let a test drive the connector against

+ // a fake Notion. Both nil in production, where the real client is

+ // built from the stored credential.

+ verifyToken func(ctx context.Context, token string) (string, error)

+ syncClient SourceClient

+// UseFakeSource points the connector at a stand-in Notion for tests.

+// Named for what it is; production never calls it.

+func (a *app) UseFakeSource(c SourceClient, verify func(ctx context.Context, token string) (string, error)) {

+ a.syncClient, a.verifyToken = c, verify

}

// UseAuth attaches auth after construction. App builds the router, and

diff --git a/internal/memoria/handlers_connector.go b/internal/memoria/handlers_connector.go

new file mode 100644

index 0000000..c560232

--- /dev/null

+++ b/internal/memoria/handlers_connector.go

@@ -0,0 +1,289 @@

+package memoria

+import (

+ "context"

+ "errors"

+ "net/http"

+ "strings"

+ "time"

+ "amadan.net/oficina/memoria/internal/memoria/notion"

+// connectorPage is what the connector templates render.

+type connectorPage struct {

+ Signed bool

+ Chunks int

+ Flash string

+ FlashKind string

+ Configured bool

+ Verified string

+ // Run is the most recent run, if any.

+ Run *runView

+ // Pages is how many pages the ledger holds, and Imported how many

+ // have actually landed.

+ Pages int

+ Imported int

+ // CountsKnown is false when the counts could not be determined. The

+ // template must then say so rather than render a confident zero.

+ CountsKnown bool

+// runView is one run, rendered.

+type runView struct {

+ State string

+ Running bool

+ Roots int

+ Total int

+ TotalKnown bool

+ Seen int

+ Fetched int

+ Skipped int

+ Empty int

+ Failed int

+ Unread int

+ ErrorCode string

+ Message string

+ Started string

+ Finished string

+// runMessage maps a run to the sentence the page shows.

+//

+// Every branch here exists because the alternative reads as something it

+// is not: an interrupted run is not a finished one, a failed run is not

+// an empty workspace, and a run still going has no total yet.

+func runMessage(r *SyncRun) string {

+ switch r.State {

+ case RunEnumerating, RunFetching:

+ return "Importing now. This runs while this page is open, and picks up where it stopped if it is interrupted."

+ case RunDone:

+ if r.FailedPages > 0 {

+ return "Import finished, but some pages could not be read. They are listed as failed below rather than left out of the count."

+ }

+ return "Import finished."

+ case RunInterrupted:

+ return "The import stopped before it finished — usually because this app went to sleep. Nothing was lost. Start it again to carry on from where it stopped."

+ case RunFailed:

+ return failMessage(r.ErrorCode)

+ }

+ return ""

+// failMessage maps an error CODE to copy. Codes travel, never messages:

+// Notion's own error bodies quote page ids and connection names.

+func failMessage(code string) string {

+ switch code {

+ case "unauthorized":

+ return "Notion refused the token. Check it is still valid and still shared with the pages you want."

+ case "rate_limited":

+ return "Notion asked us to slow down for longer than we waited. Nothing was lost — start the import again."

+ case "traversal_failed":

+ return "The import could not read all of your Notion, so it stopped rather than finish with pages missing. Nothing was lost."

+ case "network":

+ return "We could not reach Notion. Nothing was lost — start the import again."

+ case "interrupted":

+ return "The import stopped before it finished. Start it again to carry on."

+ }

+ return "The import did not finish."

+func (a *app) connectorBase(r *http.Request) connectorPage {

+ return connectorPage{Signed: true, Chunks: a.chunkCount()}

+func (a *app) connectors(w http.ResponseWriter, r *http.Request) {

+ p := a.connectorBase(r)

+ if _, err := a.store.Connector(r.Context(), "notion"); err == nil {

+ p.Configured = true

+ }

+ render(w, "connectors", p)

+func (a *app) notionConnector(w http.ResponseWriter, r *http.Request) {

+ p := a.connectorBase(r)

+ if code := r.URL.Query().Get("err"); code != "" {

+ p.Flash, p.FlashKind = connectorFlash(code), "danger"

+ }

+ if r.URL.Query().Get("saved") != "" {

+ p.Flash, p.FlashKind = "Connected. Start an import when you are ready.", "good"

+ }

+ c, err := a.store.Connector(r.Context(), "notion")

+ if errors.Is(err, ErrNoConnector) {

+ render(w, "notion", p)

+ return

+ }

+ if err != nil {

+ a.logger.Error("read connector", "err", err)

+ p.Flash, p.FlashKind = "We could not read the connector settings.", "danger"

+ render(w, "notion", p)

+ return

+ }

+ p.Configured = true

+ if c.LastVerified != nil {

+ p.Verified = c.LastVerified.Format("2 Jan 2006, 15:04")

+ }

+ // Counts. If either fails, CountsKnown stays false and the template

+ // says the number could not be determined — it never renders 0. The

+ // precedent is chunkCount, which returned 0 on error and made the

+ // page assert an absence was real when the app had failed to look.

+ if total, imported, err := a.store.CountPages(r.Context(), c.ID); err != nil {

+ a.logger.Error("count source pages", "err", err)

+ } else {

+ p.Pages, p.Imported, p.CountsKnown = total, imported, true

+ }

+ run, err := a.store.LatestRun(r.Context(), c.ID)

+ if err != nil {

+ a.logger.Error("read latest run", "err", err)

+ } else if run != nil {

+ p.Run = viewRun(run)

+ }

+ render(w, "notion", p)

+func viewRun(r *SyncRun) *runView {

+ v := &runView{

+ State: r.State, Roots: r.Roots, Fetched: r.Fetched, Skipped: r.Skipped,

+ Empty: r.Empty, Failed: r.FailedPages, Unread: r.UnreadBlocks,

+ ErrorCode: r.ErrorCode, Message: runMessage(r),

+ Running: r.State == RunEnumerating || r.State == RunFetching,

+ Started: r.StartedAt.Format("2 Jan 2006, 15:04"),

+ }

+ // Total is -1 until a run completes. Rendering that as 0 would tell

+ // the reader the workspace is empty while it is being read.

+ if r.Total >= 0 {

+ v.Total, v.TotalKnown = r.Total, true

+ }

+ v.Seen = r.Fetched + r.Skipped + r.Empty + r.FailedPages

+ if r.FinishedAt != nil {

+ v.Finished = r.FinishedAt.Format("2 Jan 2006, 15:04")

+ }

+ return v

+func connectorFlash(code string) string {

+ switch code {

+ case "token":

+ return "That did not look like a Notion integration token."

+ case "verify":

+ return "Notion would not accept that token. Check it is an internal integration token and that it has been shared with at least one page."

+ case "inflight":

+ return "An import is already running."

+ case "save":

+ return "We could not save that."

+ }

+ return "That did not work."

+// saveNotionConnector stores a token after checking Notion accepts it.

+//

+// Verified on save so a bad token fails at the form rather than part-way

+// through an import. The token is never echoed back to the page.

+func (a *app) saveNotionConnector(w http.ResponseWriter, r *http.Request) {

+ if err := r.ParseForm(); err != nil {

+ http.Redirect(w, r, "/connectors/notion?err=save", http.StatusSeeOther)

+ return

+ }

+ token := strings.TrimSpace(r.PostFormValue("token"))

+ if token == "" {

+ http.Redirect(w, r, "/connectors/notion?err=token", http.StatusSeeOther)

+ return

+ }

+ workspace, err := a.verifyNotion(r.Context(), token)

+ if err != nil {

+ // The code, never the error: a Notion error body can quote a

+ // page id and the connection's name.

+ a.logger.Error("verify notion token", "code", notion.CodeOf(err))

+ http.Redirect(w, r, "/connectors/notion?err=verify", http.StatusSeeOther)

+ return

+ }

+ if err := a.store.SaveConnector(r.Context(), "notion", token, workspace); err != nil {

+ a.logger.Error("save connector", "err", err)

+ http.Redirect(w, r, "/connectors/notion?err=save", http.StatusSeeOther)

+ return

+ }

+ http.Redirect(w, r, "/connectors/notion?saved=1", http.StatusSeeOther)

+// verifyNotion is the seam a test replaces; production calls the API.

+func (a *app) verifyNotion(ctx context.Context, token string) (string, error) {

+ if a.verifyToken != nil {

+ return a.verifyToken(ctx, token)

+ }

+ return notion.Verify(ctx, token)

+// startNotionSync begins a run and redirects to the status page.

+func (a *app) startNotionSync(w http.ResponseWriter, r *http.Request) {

+ c, err := a.store.Connector(r.Context(), "notion")

+ if err != nil {

+ http.Redirect(w, r, "/connectors/notion?err=save", http.StatusSeeOther)

+ return

+ }

+ run, err := a.store.StartRun(r.Context(), c.ID)

+ if errors.Is(err, ErrSyncInFlight) {

+ http.Redirect(w, r, "/connectors/notion?err=inflight", http.StatusSeeOther)

+ return

+ }

+ if err != nil {

+ a.logger.Error("start run", "err", err)

+ http.Redirect(w, r, "/connectors/notion?err=save", http.StatusSeeOther)

+ return

+ }

+ // context.WithoutCancel because the run outlives this request — and

+ // unlike Store.Ingest's post-commit index refresh, which is bounded

+ // and quick, this is a job of minutes.

+ //

+ // Say plainly what this is: a request-spawned goroutine that runs

+ // until the platform sleeps the process, kept alive by the status

+ // page being open. It is not a background worker; memoria has none.

+ // An interrupted run is reclaimed at boot and resumes from the

+ // ledger.

+ ctx := context.WithoutCancel(r.Context())

+ go a.runSync(ctx, c.ID, run.ID)

+ http.Redirect(w, r, "/connectors/notion", http.StatusSeeOther)

+func (a *app) runSync(ctx context.Context, connectorID, runID int64) {

+ if a.syncClient != nil {

+ a.syncer.RunWith(ctx, a.syncClient, connectorID, runID)

+ return

+ }

+ a.syncer.Run(ctx, connectorID, runID)

+// disconnectNotion removes the credential and the ledger, and NO

+// DOCUMENTS.

+//

+// What memoria imported, memoria keeps: the content was published into

+// memoria and memoria owns it, so removing it is a separate act

+// performed in memoria. The confirmation copy says that before the

+// button is pressed rather than after.

+func (a *app) disconnectNotion(w http.ResponseWriter, r *http.Request) {

+ if err := a.store.DeleteConnector(r.Context(), "notion"); err != nil {

+ a.logger.Error("disconnect connector", "err", err)

+ http.Redirect(w, r, "/connectors/notion?err=save", http.StatusSeeOther)

+ return

+ }

+ http.Redirect(w, r, "/connectors", http.StatusSeeOther)

+// waitForSync blocks until no run is active. Test seam, exported so it

+// cannot be reached by accident and obvious when it is.

+func (a *app) WaitForSync(connectorID int64, d time.Duration) bool {

+ deadline := time.Now().Add(d)

+ for time.Now().Before(deadline) {

+ run, err := a.store.LatestRun(context.Background(), connectorID)

+ if err == nil && run != nil && run.State != RunEnumerating && run.State != RunFetching {

+ return true

+ }

+ time.Sleep(5 * time.Millisecond)

+ }

+ return false

diff --git a/internal/memoria/migrations/0004_create_connectors.sql b/internal/memoria/migrations/0004_create_connectors.sql

new file mode 100644

index 0000000..f20f489

--- /dev/null

+++ b/internal/memoria/migrations/0004_create_connectors.sql

@@ -0,0 +1,18 @@

+CREATE TABLE `connectors` (`id` integer PRIMARY KEY AUTOINCREMENT,`kind` text NOT NULL,`sealed_token` blob NOT NULL,`workspace_print` text,`last_verified` datetime,`created_at` datetime,`updated_at` datetime);

+CREATE UNIQUE INDEX `idx_connectors_kind` ON `connectors`(`kind`);

+CREATE TABLE `sync_runs` (`id` integer PRIMARY KEY AUTOINCREMENT,`connector_id` integer NOT NULL,`state` text NOT NULL,`roots` integer,`total` integer NOT NULL DEFAULT -1,`fetched` integer,`skipped` integer,`empty` integer,`failed_pages` integer,`unread_blocks` integer,`error_code` text,`heartbeat` datetime,`started_at` datetime,`finished_at` datetime);

+CREATE INDEX `idx_sync_runs_state` ON `sync_runs`(`state`);

+CREATE INDEX `idx_sync_runs_connector_id` ON `sync_runs`(`connector_id`);

+CREATE TABLE `source_pages` (`id` integer PRIMARY KEY AUTOINCREMENT,`connector_id` integer NOT NULL,`page_print` text NOT NULL,`sealed_page_id` blob NOT NULL,`sealed_path` blob,`path_print` text,`seen_edited` text,`ingested_edited` text,`ingested_path` text,`state` text NOT NULL DEFAULT "pending",`error_code` text,`last_seen` datetime,`document_id` integer,`created_at` datetime,`updated_at` datetime);

+CREATE INDEX `idx_source_pages_state` ON `source_pages`(`state`);

+CREATE INDEX `idx_source_pages_page_print` ON `source_pages`(`page_print`);

+CREATE INDEX `idx_source_pages_connector_id` ON `source_pages`(`connector_id`);

diff --git a/internal/memoria/migrations/schema.sql b/internal/memoria/migrations/schema.sql

index 8338ecc..e16c256 100644

--- a/internal/memoria/migrations/schema.sql

+++ b/internal/memoria/migrations/schema.sql

@@ -3,11 +3,29 @@

CREATE TABLE `chunks` (`id` integer PRIMARY KEY AUTOINCREMENT,`document_id` integer NOT NULL,`ordinal` integer NOT NULL,`sealed_heading` blob,`sealed_text` blob NOT NULL,`created_at` datetime,CONSTRAINT `fk_documents_chunks` FOREIGN KEY (`document_id`) REFERENCES `documents`(`id`) ON DELETE CASCADE);

-CREATE TABLE `documents` (`id` integer PRIMARY KEY AUTOINCREMENT,`source` text NOT NULL,`external_id` text NOT NULL,`revision` integer NOT NULL DEFAULT 1,`sealed_title` blob NOT NULL,`sealed_body` blob NOT NULL,`bytes` integer,`created_at` datetime,`updated_at` datetime, `body_print` text NOT NULL DEFAULT "");

+CREATE TABLE `documents` (`id` integer PRIMARY KEY AUTOINCREMENT,`source` text NOT NULL,`external_id` text NOT NULL,`revision` integer NOT NULL DEFAULT 1,`sealed_title` blob NOT NULL,`sealed_body` blob NOT NULL,`bytes` integer,`created_at` datetime,`updated_at` datetime, `body_print` text NOT NULL DEFAULT "", `sealed_path` blob);

CREATE INDEX `idx_chunks_document_id` ON `chunks`(`document_id`);

+CREATE UNIQUE INDEX `idx_connectors_kind` ON `connectors`(`kind`);

CREATE INDEX `idx_documents_external_id` ON `documents`(`external_id`);

CREATE INDEX `idx_documents_source` ON `documents`(`source`);

+CREATE INDEX `idx_source_pages_connector_id` ON `source_pages`(`connector_id`);

+CREATE INDEX `idx_source_pages_page_print` ON `source_pages`(`page_print`);

+CREATE INDEX `idx_source_pages_state` ON `source_pages`(`state`);

+CREATE INDEX `idx_sync_runs_connector_id` ON `sync_runs`(`connector_id`);

+CREATE INDEX `idx_sync_runs_state` ON `sync_runs`(`state`);

diff --git a/internal/memoria/models.go b/internal/memoria/models.go

index 5c442f6..a83be78 100644

--- a/internal/memoria/models.go

+++ b/internal/memoria/models.go

@@ -9,6 +9,9 @@ import "time"

var Models = []any{

&Document{},

&Chunk{},

+ &Connector{},

+ &SyncRun{},

+ &SourcePage{},

}

// Document is one thing memoria holds.

diff --git a/internal/memoria/notion/verify.go b/internal/memoria/notion/verify.go

new file mode 100644

index 0000000..490bd07

--- /dev/null

+++ b/internal/memoria/notion/verify.go

@@ -0,0 +1,38 @@

+package notion

+import "context"

+// Verify checks a token and returns an identifier for what it is bound

+// to, so a connector row can be told apart from one pointing elsewhere.

+//

+// Called when a token is saved, so a bad one fails at the form rather

+// than part-way through an import.

+//

+// The identifier is the BOT USER ID first, and the workspace name only

+// as a fallback. Notion's /v1/users/me does not expose a workspace id;

+// the name is what it gives, and a name collides between two workspaces

+// called the same thing and changes when either is renamed. The bot id

+// is unique to this integration and stable, which is the property

+// actually wanted — the caller keys it anyway, so neither reaches disk

+// in plaintext.

+func Verify(ctx context.Context, token string) (string, error) {

+ c, err := New(token)

+ if err != nil {

+ return "", err

+ }

+ return c.verify(ctx)

+func (c *Client) verify(ctx context.Context) (string, error) {

+ var out struct {

+ ID string `json:"id"`

+ Name string `json:"name"`

+ Bot struct {

+ WorkspaceName string `json:"workspace_name"`

+ } `json:"bot"`

+ }

+ if err := c.get(ctx, "/v1/users/me", &out); err != nil {

+ return "", err

+ }

+ return firstNonEmpty(out.ID, out.Bot.WorkspaceName, out.Name), nil

diff --git a/internal/memoria/render.go b/internal/memoria/render.go

index 9c60571..b02e675 100644

--- a/internal/memoria/render.go

+++ b/internal/memoria/render.go

@@ -26,7 +26,7 @@ var assets = rastrillo.NewAssets(appFS)

var pages = map[string]*template.Template{}

func init() {

- for _, name := range []string{"index", "document", "signin"} {

+ for _, name := range []string{"index", "document", "signin", "connectors", "notion"} {

pages[name] = template.Must(template.New("layout").

Funcs(template.FuncMap{

"asset": assets.Path,

diff --git a/internal/memoria/seal/seal.go b/internal/memoria/seal/seal.go

index 29957fa..f708e79 100644

--- a/internal/memoria/seal/seal.go

+++ b/internal/memoria/seal/seal.go

@@ -41,6 +41,16 @@ const (

// different kinds of thing, and the rule here is one purpose per

// kind so that a mix-up cannot decrypt.

PurposePath = "document.path"

+ // PurposeCredential seals a connector's token, so a disclosure of the

+ // chunk store does not also yield the key to the source those chunks

+ // came from.

+ PurposeCredential = "connector.token"

+ // PurposeCoordinate seals a source coordinate — a Notion page id.

+ //

+ // REVERSIBLE, unlike Fingerprint. That distinction is the whole

+ // reason this purpose exists: the fetch phase has to hand the id back

+ // to the API it came from, and an HMAC cannot be handed back.

+ PurposeCoordinate = "source.coordinate"

)

func sha256New() hash.Hash { return sha256.New() }

diff --git a/internal/memoria/static/memoria.css b/internal/memoria/static/memoria.css

index 6bcbdac..628f7aa 100644

--- a/internal/memoria/static/memoria.css

+++ b/internal/memoria/static/memoria.css

@@ -67,6 +67,16 @@

[rst-results] { list-style: none; margin: 0; padding: 0; }

[rst-result] { border-top: 1px solid var(--rst-line); padding: 0.9rem 0; }

[rst-result]:first-child { border-top: 0; }

+/* The document's ancestry in its source, above the title. Sits above

+ rather than below because it is read as a location — "Handbook ›

+ Onboarding ›" then the page — and colours come from tokens so light

+ and dark both resolve through theme.css. */

+[rst-result-path] {

+ color: var(--rst-text-mut);

+ display: block;

+ font-size: 0.8125rem;

+ margin-bottom: 0.1rem;

[rst-result-title] { font-weight: 600; }

[rst-result-trail] {

color: var(--rst-text-mut);

diff --git a/internal/memoria/store.go b/internal/memoria/store.go

index d29a88c..2326596 100644

--- a/internal/memoria/store.go

+++ b/internal/memoria/store.go

@@ -34,9 +34,16 @@ type Result struct {

Ordinal int

Source string

Title string

- Heading string

- Snippet string

- Score float64

+ // Path is the document's ancestry in its source. SHOWN, not merely

+ // indexed: two pages called "Week one" under different handbooks are

+ // the same result to a reader who cannot see where each came from,

+ // and carrying ancestry for findability while hiding it from the

+ // person reading the answer is half a feature. Found by looking at

+ // the rendered page, which is the only thing that would have.

+ Path string

+ Heading string

+ Snippet string

+ Score float64

}

// Summary is a document as the list shows it.

@@ -44,6 +51,7 @@ type Summary struct {

ID int64

Source string

Title string

+ Path string

Bytes int

Chunks int

Updated string

@@ -380,6 +388,7 @@ func (s *Store) Search(ctx context.Context, query string, limit int) ([]Result,

}

titles := map[int64]string{}

sources := map[int64]string{}

+ paths := map[int64]string{}

for _, d := range docs {

t, err := s.sealer.Open(seal.PurposeTitle, d.SealedTitle)

if err != nil {

@@ -387,6 +396,13 @@ func (s *Store) Search(ctx context.Context, query string, limit int) ([]Result,

}

titles[d.ID] = string(t)

sources[d.ID] = d.Source

+ if len(d.SealedPath) > 0 {

+ p, err := s.sealer.Open(seal.PurposePath, d.SealedPath)

+ if err != nil {

+ return nil, fmt.Errorf("open path %d: %w", d.ID, err)

+ }

+ paths[d.ID] = string(p)

+ }

}

out := make([]Result, 0, len(hits))

for _, h := range hits {

@@ -396,6 +412,7 @@ func (s *Store) Search(ctx context.Context, query string, limit int) ([]Result,

Ordinal: h.Ordinal,

Source: sources[h.DocumentID],

Title: titles[h.DocumentID],

+ Path: paths[h.DocumentID],

Heading: h.Heading,

Snippet: h.Snippet,

Score: h.Score,

@@ -420,10 +437,19 @@ func (s *Store) Summaries(ctx context.Context) ([]Summary, error) {

if err := s.db.G.WithContext(ctx).Model(&Chunk{}).Where("document_id = ?", d.ID).Count(&n).Error; err != nil {

return nil, err

}

+ var path string

+ if len(d.SealedPath) > 0 {

+ p, err := s.sealer.Open(seal.PurposePath, d.SealedPath)

+ if err != nil {

+ return nil, fmt.Errorf("open path %d: %w", d.ID, err)

+ }

+ path = string(p)

+ }

out = append(out, Summary{

ID: d.ID,

Source: d.Source,

Title: string(t),

+ Path: path,

Bytes: d.Bytes,

Chunks: int(n),

Updated: d.UpdatedAt.Format("2 Jan 2006, 15:04"),

diff --git a/internal/memoria/sync.go b/internal/memoria/sync.go

new file mode 100644

index 0000000..620039c

--- /dev/null

+++ b/internal/memoria/sync.go

@@ -0,0 +1,281 @@

+package memoria

+import (

+ "context"

+ "errors"

+ "log/slog"

+ "time"

+ "amadan.net/oficina/memoria/internal/memoria/chunk"

+ "amadan.net/oficina/memoria/internal/memoria/notion"

+ "amadan.net/oficina/memoria/internal/memoria/seal"

+// SourceClient is what a sync needs from a source. An interface so the

+// loop can be driven against a fake without the app package importing an

+// HTTP client — which the network gate forbids it anyway.

+type SourceClient interface {

+ Roots(ctx context.Context) ([]notion.Ref, error)

+ Fetch(ctx context.Context, ref notion.Ref) (*notion.Content, error)

+// Syncer runs imports.

+type Syncer struct {

+ store *Store

+ logger *slog.Logger

+ // dial builds a client from a token. Swapped in tests.

+ dial func(token string) (SourceClient, error)

+// NewSyncer wires a syncer against the real Notion API.

+func NewSyncer(store *Store, logger *slog.Logger) *Syncer {

+ return &Syncer{store: store, logger: logger, dial: func(token string) (SourceClient, error) {

+ return notion.New(token)

+ }}

+// beatEvery is how often the run says it is still alive.

+const beatEvery = 25

+// Run imports everything reachable from the connector's roots.

+//

+// ONE PASS, not the two phases an earlier design specified. With

+// traversal, enumeration and extraction are the same API call — fetching

+// a page returns its body AND the child pages beneath it — so separating

+// them would double every request against a limit of three per second.

+// The ledger is still written per page, so the ingest work already done

+// survives an interruption; what a resumed run repeats is the traversal,

+// because a page's children cannot be discovered without asking for them

+// and a child's own edits do not change its parent's edit time.

+//

+// Errors divide in two, and the division is load-bearing:

+//

+// - A traversal failure below a page (rate limit exhausted, a broken

+// cursor, an unreadable body) is FATAL. It means the pages beneath

+// that point were never enumerated, and a run that continued would

+// report "done" while missing a whole subtree — an absence produced

+// by a failure, which is the shape this codebase ranks worst.

+// - A single page that has gone (404 between enumeration and fetch) is

+// recorded against that page and the run carries on. One placeholder

+// must not block an import, but it must not vanish either.

+func (s *Syncer) Run(ctx context.Context, connectorID, runID int64) {

+ token, err := s.store.ConnectorToken(ctx, "notion")

+ if err != nil {

+ s.fail(ctx, runID, "unauthorized")

+ return

+ }

+ client, err := s.dial(token)

+ if err != nil {

+ s.fail(ctx, runID, "unauthorized")

+ return

+ }

+ s.RunWith(ctx, client, connectorID, runID)

+// RunWith is Run against a supplied client.

+func (s *Syncer) RunWith(ctx context.Context, client SourceClient, connectorID, runID int64) {

+ roots, err := client.Roots(ctx)

+ if err != nil {

+ s.fail(ctx, runID, string(notion.CodeOf(err)))

+ return

+ }

+ if err := s.store.UpdateRun(ctx, runID, map[string]any{

+ "roots": len(roots), "state": RunFetching,

+ }); err != nil {

+ s.fail(ctx, runID, "network")

+ return

+ }

+ var stats struct{ fetched, skipped, empty, failed, unread, seen int }

+ visited := map[string]bool{}

+ queue := append([]notion.Ref{}, roots...)

+ for len(queue) > 0 {

+ ref := queue[0]

+ queue = queue[1:]

+ if visited[ref.ID] {

+ continue

+ }

+ visited[ref.ID] = true

+ stats.seen++

+ if stats.seen%beatEvery == 0 {

+ if err := s.store.Beat(ctx, runID); err != nil {

+ s.logger.Error("sync heartbeat", "run", runID, "err", err)

+ }

+ // Record the coordinate BEFORE fetching, so a page that fails to

+ // fetch still has a row to be named in rather than being absent.

+ if err := s.store.SeeSourcePage(ctx, connectorID, ref.ID, ref.Title, ref.Path, ref.EditedTime); err != nil {

+ s.logger.Error("sync ledger write", "run", runID, "err", err)

+ s.fail(ctx, runID, "network")

+ return

+ }

+ content, err := client.Fetch(ctx, ref)

+ if err != nil {

+ if gone(err) {

+ stats.failed++

+ s.markByPage(ctx, connectorID, ref, PageFailed, string(notion.CodeOf(err)), 0, "")

+ continue

+ }

+ // Not survivable: the subtree below here was never seen.

+ s.fail(ctx, runID, string(notion.CodeOf(err)))

+ return

+ }

+ queue = append(queue, content.Children...)

+ stats.unread += content.UnreadBlocks

+ state, docID, err := s.ingest(ctx, connectorID, ref, content)

+ if err != nil {

+ stats.failed++

+ s.markByPage(ctx, connectorID, ref, PageFailed, "extract_failed", 0, "")

+ continue

+ }

+ switch state {

+ case PageEmpty:

+ stats.empty++

+ case "":

+ stats.skipped++

+ continue

+ default:

+ stats.fetched++

+ }

+ s.markByPage(ctx, connectorID, ref, state, "", docID, ref.EditedTime)

+ }

+ now := time.Now().UTC()

+ if err := s.store.UpdateRun(ctx, runID, map[string]any{

+ "state": RunDone, "total": stats.seen, "fetched": stats.fetched,

+ "skipped": stats.skipped, "empty": stats.empty, "failed_pages": stats.failed,

+ "unread_blocks": stats.unread, "finished_at": now, "heartbeat": now,

+ }); err != nil {

+ s.logger.Error("sync finish", "run", runID, "err", err)

+ }

+ // Counts only. No title, no path, no page id — AGENTS.md's rule, and

+ // the reason this line carries nothing that came from the workspace.

+ s.logger.Info("sync finished", "run", runID, "pages", stats.seen,

+ "fetched", stats.fetched, "skipped", stats.skipped,

+ "empty", stats.empty, "failed", stats.failed, "unread_blocks", stats.unread)

+// ingest stores one page, or reports that it did not need storing.

+//

+// Returns "" when the page was already up to date. A data source is a

+// container rather than a document and is never stored.

+func (s *Syncer) ingest(ctx context.Context, connectorID int64, ref notion.Ref, content *notion.Content) (string, int64, error) {

+ if ref.Kind == notion.KindDataSource {

+ return "", 0, nil

+ }

+ upToDate, err := s.upToDate(ctx, connectorID, ref)

+ if err != nil {

+ return "", 0, err

+ }

+ if upToDate {

+ return "", 0, nil

+ }

+ id, err := s.store.Ingest(ctx, IngestSpec{

+ Source: "notion",

+ Key: ref.ID,

+ Title: ref.Title,

+ Body: content.Markdown,

+ Path: ref.Path,

+ // On, because an empty page is a fact about the source. See

+ // IngestSpec.AllowTitleOnly.

+ AllowTitleOnly: true,

+ })

+ if err != nil {

+ return "", 0, err

+ }

+ if content.Markdown == "" {

+ return PageEmpty, id, nil

+ }

+ return PageFetched, id, nil

+// upToDate reports whether the ledger already holds this page at this

+// edit time AND this ancestry.

+//

+// Both, because an ancestor rename moves the ancestry without moving the

+// page's own edit time — so a check on the edit time alone leaves every

+// descendant of a renamed page stale forever, with nothing upstream ever

+// signalling it.

+func (s *Syncer) upToDate(ctx context.Context, connectorID int64, ref notion.Ref) (bool, error) {

+ rows, err := s.store.pageRow(ctx, connectorID, ref.ID)

+ if err != nil || rows == nil {

+ return false, err

+ }

+ if rows.State != PageFetched && rows.State != PageEmpty {

+ return false, nil

+ }

+ if rows.IngestedEdited != ref.EditedTime {

+ return false, nil

+ }

+ want, err := s.store.pathPrint(ref.Path, ref.Title)

+ if err != nil {

+ return false, err

+ }

+ return rows.IngestedPath == want, nil

+func (s *Syncer) markByPage(ctx context.Context, connectorID int64, ref notion.Ref, state, code string, docID int64, edited string) {

+ row, err := s.store.pageRow(ctx, connectorID, ref.ID)

+ if err != nil || row == nil {

+ return

+ }

+ print, err := s.store.pathPrint(ref.Path, ref.Title)

+ if err != nil {

+ return

+ }

+ if edited == "" {

+ print = ""

+ }

+ if err := s.store.MarkPage(ctx, row.ID, state, code, docID, edited, print); err != nil {

+ s.logger.Error("sync mark page", "err", err)

+ }

+func (s *Syncer) fail(ctx context.Context, runID int64, code string) {

+ now := time.Now().UTC()

+ if err := s.store.UpdateRun(ctx, runID, map[string]any{

+ "state": RunFailed, "error_code": code, "finished_at": now,

+ }); err != nil {

+ s.logger.Error("sync fail", "run", runID, "err", err)

+ }

+ s.logger.Error("sync failed", "run", runID, "code", code)

+func asNotionErr(err error, target **notion.Error) bool { return errors.As(err, target) }

+// gone reports whether an error is about one page rather than about the

+// traversal. Only a 404 qualifies: everything else means we do not know

+// what lies below, and continuing would report a complete import.

+func gone(err error) bool {

+ var e *notion.Error

+ if !asNotionErr(err, &e) {

+ return false

+ }

+ return e.Status == 404

+// pageRow reads one ledger row by coordinate.

+func (s *Store) pageRow(ctx context.Context, connectorID int64, pageID string) (*SourcePage, error) {

+ print, err := s.sealer.Fingerprint(seal.PurposeCoordinate, []byte("page\x00"+pageID))

+ if err != nil {

+ return nil, err

+ }

+ var row SourcePage

+ res := s.db.G.WithContext(ctx).Where("connector_id = ? AND page_print = ?", connectorID, print).First(&row)

+ if res.Error != nil {

+ return nil, nil

+ }

+ return &row, nil

+// pathPrint is the keyed fingerprint of an ancestry plus a title, so a

+// move or a rename is detectable without opening the row.

+func (s *Store) pathPrint(path []string, title string) (string, error) {

+ pathStr := chunk.Chunk{Path: path}.PathString()

+ return s.sealer.Fingerprint(seal.PurposePath, []byte("path\x00"+pathStr+"\x00"+title))

diff --git a/internal/memoria/templates/connectors.html b/internal/memoria/templates/connectors.html

new file mode 100644

index 0000000..fc13679

--- /dev/null

+++ b/internal/memoria/templates/connectors.html

@@ -0,0 +1,22 @@

+{{define "title"}}Sources · memoria{{end}}

+{{define "content"}}

+<h1>Sources</h1>

+<p rst-lede>

+ memoria can pull in work that lives outside the suite. What you connect,

+ memoria keeps: disconnecting a source stops memoria seeing new changes,

+ it does not remove what memoria already holds.

+</p>

+<ul rst-list>

+ <li rst-list-item>

+ <a href="/connectors/notion" rst-list-title>Notion</a>

+ <p rst-cell-mut>

+ {{if .Configured}}Connected.{{else}}Not connected.{{end}}

+ Imports the pages you share with a Notion integration, and everything nested under them.

+ </p>

+ </li>

+</ul>

+{{end}}

diff --git a/internal/memoria/templates/index.html b/internal/memoria/templates/index.html

index 699173b..cbd10f4 100644

--- a/internal/memoria/templates/index.html

+++ b/internal/memoria/templates/index.html

@@ -25,6 +25,12 @@

+ {{/* Ancestry above the title, heading trail below it. Both, and in

+ that order, because they answer different questions: the path

+ says which part of the workspace this came from, the heading

+ says where in the page. A result showing only the second is

+ two indistinguishable "Week one"s under different handbooks. */}}

+ {{if .Path}}<span rst-result-path rst-cell-mut>{{.Path}}</span>{{end}}

<a href="/documents/{{.DocumentID}}#p{{.Ordinal}}" rst-result-title>{{.Title}}</a>

{{if .Heading}}<span rst-result-trail>{{.Heading}}</span>{{end}}

<p rst-result-snippet>{{.Snippet | safeSnippet}}</p>

diff --git a/internal/memoria/templates/layout.html b/internal/memoria/templates/layout.html

index 459d8fa..56fa66c 100644

--- a/internal/memoria/templates/layout.html

+++ b/internal/memoria/templates/layout.html

@@ -15,11 +15,15 @@

+{{block "head" .}}{{end}}

</head>

<body>

<a href="/" rst-brand>memoria</a>

- {{if .Signed}}<form method="post" action="/signout" rst-inline><button rst-btn="quiet compact">Sign out</button></form>{{end}}

+ {{if .Signed}}<nav rst-inline>

+ <a href="/connectors" rst-btn="quiet compact">Sources</a>

+ <form method="post" action="/signout" rst-inline><button rst-btn="quiet compact">Sign out</button></form>

+ </nav>{{end}}

</header>

diff --git a/internal/memoria/templates/notion.html b/internal/memoria/templates/notion.html

new file mode 100644

index 0000000..231b279

--- /dev/null

+++ b/internal/memoria/templates/notion.html

@@ -0,0 +1,139 @@

+{{define "title"}}Notion · memoria{{end}}

+{{/* The status page refreshes itself ONLY while a run is active, and it

+ is a meta refresh because every affordance here works scriptless.

+ This is also the honest mechanism: the import is a goroutine that

+ lives as long as the process, and the platform sleeps an idle app,

+ so an open page is what keeps it awake. The copy below says so

+ rather than implying a background worker memoria does not have. */}}

+{{define "head"}}{{if .Run}}{{if .Run.Running}}<meta http-equiv="refresh" content="5">{{end}}{{end}}{{end}}

+{{define "content"}}

+<h1>Notion</h1>

+{{if .Flash}}<div rst-callout="{{.FlashKind}}"><p rst-callout-body>{{.Flash}}</p></div>{{end}}

+{{if not .Configured}}

+ <p rst-lede>

+ memoria imports the Notion pages you share with an integration, and

+ everything nested underneath them.

+ </p>

+ <div rst-callout="quiet">

+ <p rst-callout-body>

+ <strong>What you connect, memoria keeps.</strong> Once a page is

+ imported, it stays in memoria and stays readable by everyone here —

+ even if you later archive it in Notion, unshare it, or revoke this

+ token. Removing it is something you do in memoria.

+ </p>

+ </div>

+ <section rst-section>

+ <h2>Connect</h2>

+ <ol rst-steps>

+ <li>In Notion, create an <strong>internal integration</strong> with

+ read-only capabilities and no user information.</li>

+ <li>Share the pages you want memoria to hold with that integration.

+ Sharing a page is what publishes it — memoria imports what the

+ integration can see and nothing else.</li>

+ <li>Paste the integration token below.</li>

+ </ol>

+ <form method="post" action="/connectors/notion" rst-stack>

+ <label for="token">Integration token</label>

+ <input type="password" id="token" name="token" autocomplete="off" required>

+ <p rst-cell-mut>Checked with Notion before it is saved. It is stored sealed and is never shown again.</p>

+ <button rst-btn="primary">Connect</button>

+ </form>

+ </section>

+{{else}}

+ <section rst-section>

+ <h2>Connected</h2>

+ <p rst-cell-mut>

+ {{if .Verified}}Token last checked {{.Verified}}.{{else}}Token stored.{{end}}

+ </p>

+ {{if .CountsKnown}}

+ <p>{{.Imported}} of {{.Pages}} known page{{if ne .Pages 1}}s{{end}} imported.</p>

+ {{else}}

+ <p>The page counts could not be read, so this says nothing about how much has been imported.</p>

+ {{end}}

+ <form method="post" action="/connectors/notion/sync" rst-inline>

+ <button rst-btn="primary"{{if .Run}}{{if .Run.Running}} disabled{{end}}{{end}}>

+ {{if .Run}}{{if .Run.Running}}Importing…{{else}}Import now{{end}}{{else}}Import now{{end}}

+ </button>

+ </form>

+ </section>

+ {{if .Run}}

+ <section rst-section>

+ <h2>Last import</h2>

+ <p>{{.Run.Message}}</p>

+ <dl rst-facts>

+ <dt>Started</dt><dd>{{.Run.Started}}</dd>

+ {{if .Run.Finished}}<dt>Finished</dt><dd>{{.Run.Finished}}</dd>{{end}}

+ <dt>Top-level pages shared with the integration</dt><dd>{{.Run.Roots}}</dd>

+ <dt>Pages seen</dt>

+ <dd>

+ {{if .Run.TotalKnown}}

+ {{.Run.Total}}

+ {{else}}

+ {{.Run.Seen}} so far — the total is not known until the import finishes

+ {{end}}

+ </dd>

+ <dt>Imported</dt><dd>{{.Run.Fetched}}</dd>

+ <dt>Already up to date</dt><dd>{{.Run.Skipped}}</dd>

+ <dt>Empty pages</dt><dd>{{.Run.Empty}}</dd>

+ <dt>Could not be read</dt>

+ <dd>

+ {{.Run.Failed}}

+ {{if .Run.Failed}}<span rst-cell-mut>— counted here rather than left out of the totals</span>{{end}}

+ </dd>

+ {{if .Run.Unread}}

+ <dt>Blocks memoria does not read</dt>

+ <dd>{{.Run.Unread}} <span rst-cell-mut>— a page that came in thin may be one of these</span></dd>

+ {{end}}

+ </dl>

+ </section>

+ {{end}}

+ <section rst-section>

+ <h2>Replace the token</h2>

+ <p rst-cell-mut>

+ Rotating a token replaces the stored one. The old token is not kept.

+ Nothing already imported is affected.

+ </p>

+ <form method="post" action="/connectors/notion" rst-stack>

+ <label for="token">New integration token</label>

+ {{/* Never carries a value. The stored token is not rendered back

+ here or anywhere — not even masked, because a masked secret is

+ still a secret in the DOM. TestTokenIsNeverRendered checks

+ this branch specifically: an earlier version of that gate only

+ looked at the not-yet-connected page, which has no token field

+ at all, so it passed while proving nothing. */}}

+ <input type="password" id="token" name="token" autocomplete="off" required>

+ <button rst-btn="quiet">Replace</button>

+ </form>

+ </section>

+ <section rst-section>

+ <h2>Disconnect</h2>

+ <p>

+ This removes the token and stops future imports. <strong>It deletes

+ nothing memoria has already imported</strong> — those pages stay

+ here until someone deletes them in memoria.

+ </p>

+ <form method="post" action="/connectors/notion/disconnect" rst-inline>

+ <button rst-btn="danger">Disconnect Notion</button>

+ </form>

+ </section>

+{{end}}

diff --git a/internal/memoriatest/connector_test.go b/internal/memoriatest/connector_test.go

new file mode 100644

index 0000000..ca95afe

--- /dev/null

+++ b/internal/memoriatest/connector_test.go

@@ -0,0 +1,620 @@

+package memoriatest

+import (

+ "context"

+ "io"

+ "log/slog"

+ "net/http"

+ "net/url"

+ "path/filepath"

+ "strings"

+ "testing"

+ "time"

+ "github.com/carlosframework/rastrillo/db"

+ memoria "amadan.net/oficina/memoria/internal/memoria"

+ "amadan.net/oficina/memoria/internal/memoria/notion"

+// testApp is the app's test surface. Declared as an interface because

+// AppForTest returns an unexported type, which a test in another package

+// cannot name.

+type testApp interface {

+ Store() *memoria.Store

+ AsPerson(string)

+ SetOwners([]string)

+ UseFakeSource(memoria.SourceClient, func(context.Context, string) (string, error))

+ WaitForSync(int64, time.Duration) bool

+// fakeSource is a Notion stand-in for the run loop: a tree, some bodies,

+// and per-page errors. No HTTP — the client's own package tests that.

+type fakeSource struct {

+ roots []notion.Ref

+ children map[string][]notion.Ref

+ bodies map[string]string

+ unread map[string]int

+ failWith map[string]error

+ fetches map[string]int

+func newSource() *fakeSource {

+ return &fakeSource{

+ children: map[string][]notion.Ref{}, bodies: map[string]string{},

+ unread: map[string]int{}, failWith: map[string]error{}, fetches: map[string]int{},

+ }

+func (f *fakeSource) Roots(ctx context.Context) ([]notion.Ref, error) {

+ if err := f.failWith["__roots__"]; err != nil {

+ return nil, err

+ }

+ return f.roots, nil

+func (f *fakeSource) Fetch(ctx context.Context, ref notion.Ref) (*notion.Content, error) {

+ f.fetches[ref.ID]++

+ if err := f.failWith[ref.ID]; err != nil {

+ return nil, err

+ }

+ return &notion.Content{

+ Markdown: f.bodies[ref.ID],

+ Children: f.children[ref.ID],

+ UnreadBlocks: f.unread[ref.ID],

+ }, nil

+// add puts a page in the tree. parent "" makes it a root.

+func (f *fakeSource) add(id, title, parent, body string, path ...string) {

+ ref := notion.Ref{ID: id, Title: title, Path: path, EditedTime: "t1", Kind: notion.KindPage}

+ f.bodies[id] = body

+ if parent == "" {

+ f.roots = append(f.roots, ref)

+ return

+ }

+ f.children[parent] = append(f.children[parent], ref)

+func (f *fakeSource) edit(id, body, editedTime string) {

+ f.bodies[id] = body

+ f.retime(id, editedTime)

+func (f *fakeSource) retime(id, editedTime string) {

+ for i := range f.roots {

+ if f.roots[i].ID == id {

+ f.roots[i].EditedTime = editedTime

+ }

+ for p := range f.children {

+ for i := range f.children[p] {

+ if f.children[p][i].ID == id {

+ f.children[p][i].EditedTime = editedTime

+ }

+// rename changes a page's title wherever it appears, and rewrites the

+// inherited ancestry of everything below it — which is what Notion does

+// NOT tell us about, since an ancestor rename leaves a descendant's own

+// last_edited_time untouched.

+func (f *fakeSource) rename(id, was, now string) {

+ retitle := func(r *notion.Ref) {

+ if r.ID == id {

+ r.Title = now

+ }

+ for i, seg := range r.Path {

+ if seg == was {

+ r.Path[i] = now

+ }

+ for i := range f.roots {

+ retitle(&f.roots[i])

+ }

+ for p := range f.children {

+ for i := range f.children[p] {

+ retitle(&f.children[p][i])

+ }

+// connectorApp builds an app with a configured connector and a fake

+// Notion behind it.

+func connectorApp(t *testing.T, src *fakeSource) (http.Handler, testApp, int64, *strings.Builder) {

+ t.Helper()

+ var logs strings.Builder

+ logger := slog.New(slog.NewTextHandler(&logs, &slog.HandlerOptions{Level: slog.LevelDebug}))

+ d, err := db.Open(filepath.Join(t.TempDir(), "app.db"), logger)

+ if err != nil {

+ t.Fatalf("db.Open: %v", err)

+ }

+ t.Cleanup(func() { d.Close() })

+ mux, a, err := memoria.AppForTest(d, testOrigin, testDEK(t), logger)

+ if err != nil {

+ t.Fatalf("AppForTest: %v", err)

+ }

+ app := any(a).(testApp)

+ app.UseFakeSource(src, func(ctx context.Context, token string) (string, error) {

+ return "workspace-1", nil

+ })

+ if err := app.Store().SaveConnector(context.Background(), "notion", "ntn_secret_token_value", "workspace-1"); err != nil {

+ t.Fatal(err)

+ }

+ c, err := app.Store().Connector(context.Background(), "notion")

+ if err != nil {

+ t.Fatal(err)

+ }

+ return mux, app, c.ID, &logs

+func sync(t *testing.T, h http.Handler, app testApp, connectorID int64) {

+ t.Helper()

+ rec := post(t, h, "/connectors/notion/sync", url.Values{})

+ if rec.Code != http.StatusSeeOther {

+ t.Fatalf("sync: status %d, want 303", rec.Code)

+ }

+ if !app.WaitForSync(connectorID, 5*time.Second) {

+ t.Fatal("sync did not finish")

+ }

+func latest(t *testing.T, app testApp, connectorID int64) *memoria.SyncRun {

+ t.Helper()

+ run, err := app.Store().LatestRun(context.Background(), connectorID)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if run == nil {

+ t.Fatal("no run recorded")

+ }

+ return run

+// TestSyncImportsTheTreeWithAncestry is the happy path, and the vacuity

+// guard for everything below it: a gate about failures proves nothing if

+// the success case never worked.

+func TestSyncImportsTheTreeWithAncestry(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "# Welcome\n\nEverything a new starter needs.")

+ src.add("kid", "Onboarding", "root", "Week one covers expenses and travel booking.", "Handbook")

+ src.add("grandkid", "Equipment", "kid", "Laptops are ordered through the request form.", "Handbook", "Onboarding")

+ h, app, cid, logs := connectorApp(t, src)

+ sync(t, h, app, cid)

+ run := latest(t, app, cid)

+ if run.State != memoria.RunDone {

+ t.Fatalf("run state %q, want done (error %q)\nLOGS:\n%s", run.State, run.ErrorCode, logs.String())

+ }

+ if run.Total != 3 {

+ t.Errorf("total %d, want 3", run.Total)

+ }

+ if run.Fetched != 3 {

+ t.Errorf("fetched %d, want 3", run.Fetched)

+ }

+ // The ancestry must be searchable, or carrying it was pointless.

+ hits, err := app.Store().Search(context.Background(), "onboarding", 10)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if len(hits) == 0 {

+ t.Error("the imported ancestry is not searchable")

+ }

+// TestAFailedPageIsNamedNotAbsent. A page that has gone between

+// enumeration and fetch must be counted and recorded, never dropped: 3

+// unreadable pages inside 1,930 that simply vanish read as a complete

+// import.

+//

+// MUTATION-TESTED 2026-09-05: replacing the markByPage/stats.failed++ in

+// the gone(err) branch with a bare `continue` makes this fail. Reverted.

+func TestAFailedPageIsNamedNotAbsent(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs.")

+ src.add("kid", "Onboarding", "root", "Week one.", "Handbook")

+ src.failWith["kid"] = &notion.Error{Code: notion.CodeTraversalFailed, Status: 404}

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ run := latest(t, app, cid)

+ if run.State != memoria.RunDone {

+ t.Errorf("state %q, want done — one missing page must not fail the whole import", run.State)

+ }

+ if run.FailedPages != 1 {

+ t.Errorf("failed_pages %d, want 1 — the page vanished from the totals", run.FailedPages)

+ }

+ body := get(t, h, "/connectors/notion").Body.String()

+ if !strings.Contains(body, "Could not be read") {

+ t.Error("the status page does not name the pages that could not be read")

+ }

+// TestTraversalFailureNeverReportsDone. A failure BELOW a page means the

+// subtree was never enumerated. Continuing would report a complete

+// import while missing everything under that point — a failure rendered

+// as an absence, which this codebase ranks as the worst available.

+//

+// MUTATION-TESTED 2026-09-05: treating every fetch error as a per-page

+// failure (making gone() return true unconditionally) makes this fail.

+// Reverted.

+func TestTraversalFailureNeverReportsDone(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs.")

+ src.add("kid", "Onboarding", "root", "Week one.", "Handbook")

+ src.failWith["kid"] = &notion.Error{Code: notion.CodeRateLimited, Status: 429}

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ run := latest(t, app, cid)

+ if run.State != memoria.RunFailed {

+ t.Fatalf("state %q, want failed — the pages below the failure were never seen", run.State)

+ }

+ if run.ErrorCode != string(notion.CodeRateLimited) {

+ t.Errorf("error_code %q, want %q", run.ErrorCode, notion.CodeRateLimited)

+ }

+ body := get(t, h, "/connectors/notion").Body.String()

+ if strings.Contains(body, "Import finished.") {

+ t.Error("a failed import is described as finished")

+ }

+// TestSyncNeverDeletesOnAbsence. memoria is replacing Notion, so a page

+// disappearing upstream is the expected end state of a migration, not a

+// signal to forget it. Archiving a page in Notion after migrating it

+// must not delete it here.

+//

+// MUTATION-TESTED 2026-09-05: adding a diff-and-delete pass over ledger

+// rows not seen by the latest run makes this fail. Reverted.

+func TestSyncNeverDeletesOnAbsence(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs.")

+ src.add("kid", "Onboarding", "root", "Week one covers expenses.", "Handbook")

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ if hits, _ := app.Store().Search(context.Background(), "expenses", 10); len(hits) == 0 {

+ t.Fatal("the page was never imported, so its survival proves nothing")

+ }

+ // The page is archived in Notion: it is simply no longer returned.

+ src.children["root"] = nil

+ sync(t, h, app, cid)

+ run := latest(t, app, cid)

+ if run.State != memoria.RunDone {

+ t.Errorf("state %q, want done — an absence is not an error either", run.State)

+ }

+ if hits, _ := app.Store().Search(context.Background(), "expenses", 10); len(hits) == 0 {

+ t.Error("a page that vanished from Notion was deleted from memoria")

+ }

+// TestResumeSkipsWhatIsAlreadyImported. The ledger is per connector, not

+// per run, which is what makes a second run continue rather than start

+// over.

+//

+// MUTATION-TESTED 2026-09-05: making upToDate always return false

+// re-ingests everything and makes the skipped assertion fail. Reverted.

+func TestResumeSkipsWhatIsAlreadyImported(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs.")

+ src.add("kid", "Onboarding", "root", "Week one covers expenses.", "Handbook")

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ run := latest(t, app, cid)

+ if run.Skipped != 2 {

+ t.Errorf("skipped %d on an unchanged re-sync, want 2 — nothing changed upstream", run.Skipped)

+ }

+ if run.Fetched != 0 {

+ t.Errorf("fetched %d on an unchanged re-sync, want 0", run.Fetched)

+ }

+// TestEditAndAncestorRenameBothReachMemoria.

+//

+// The rename half is the one an earlier design missed entirely. Editing

+// a page changes its own last_edited_time; renaming its ANCESTOR does

+// not, so a sync keyed only on edit times leaves every descendant of a

+// renamed page carrying stale ancestry forever, with nothing upstream

+// ever signalling it.

+//

+// MUTATION-TESTED 2026-09-05: dropping the IngestedPath comparison from

+// upToDate (leaving only the edit-time check) makes the rename half

+// fail. Reverted.

+func TestEditAndAncestorRenameBothReachMemoria(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs.")

+ src.add("kid", "Onboarding", "root", "Week one covers expenses.", "Handbook")

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ t.Run("an edit is re-ingested", func(t *testing.T) {

+ src.edit("kid", "Week one now covers parking permits.", "t2")

+ sync(t, h, app, cid)

+ hits, _ := app.Store().Search(context.Background(), "parking permits", 10)

+ if len(hits) == 0 {

+ t.Error("an edited page was not re-ingested")

+ }

+ })

+ t.Run("an ancestor rename is re-ingested", func(t *testing.T) {

+ // Handbook becomes Playbook. The CHILD's own edit time does not

+ // move — Notion would not move it either.

+ src.rename("root", "Handbook", "Playbook")

+ sync(t, h, app, cid)

+ if hits, _ := app.Store().Search(context.Background(), "playbook", 10); len(hits) == 0 {

+ t.Error("the new ancestry is not searchable: an ancestor rename never reached memoria")

+ }

+ if hits, _ := app.Store().Search(context.Background(), "handbook", 10); len(hits) != 0 {

+ t.Error("the old ancestry is still searchable after an ancestor rename")

+ }

+ })

+// TestInterruptedRunDoesNotWedgeTheConnector.

+//

+// The run is a goroutine. When the platform sleeps the process it dies

+// having written nothing, leaving the row "fetching" — and a

+// single-flight check that trusted that row would refuse every later

+// sync as already-running, permanently, on the NORMAL path. Boot

+// reclaims them, which is sound by construction: a process that has just

+// started cannot be running a sync.

+//

+// MUTATION-TESTED 2026-09-05: removing the ReclaimRuns call from App()

+// makes this fail with "a sync is already running". Reverted.

+func TestInterruptedRunDoesNotWedgeTheConnector(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs.")

+ dir := t.TempDir()

+ path := filepath.Join(dir, "app.db")

+ logger := slog.New(slog.NewTextHandler(io.Discard, nil))

+ d, err := db.Open(path, logger)

+ if err != nil {

+ t.Fatal(err)

+ }

+ _, a, err := memoria.AppForTest(d, testOrigin, testDEK(t), logger)

+ if err != nil {

+ t.Fatal(err)

+ }

+ app := any(a).(testApp)

+ if err := app.Store().SaveConnector(context.Background(), "notion", "tok", "w"); err != nil {

+ t.Fatal(err)

+ }

+ c, err := app.Store().Connector(context.Background(), "notion")

+ if err != nil {

+ t.Fatal(err)

+ }

+ // A run starts and the process dies: the row stays in flight.

+ if _, err := app.Store().StartRun(context.Background(), c.ID); err != nil {

+ t.Fatal(err)

+ }

+ d.Close()

+ // Boot again over the same database.

+ d2, err := db.Open(path, logger)

+ if err != nil {

+ t.Fatal(err)

+ }

+ t.Cleanup(func() { d2.Close() })

+ h2, a2, err := memoria.AppForTest(d2, testOrigin, testDEK(t), logger)

+ if err != nil {

+ t.Fatal(err)

+ }

+ app2 := any(a2).(testApp)

+ app2.UseFakeSource(src, nil)

+ rec := post(t, h2, "/connectors/notion/sync", url.Values{})

+ if loc := rec.Header().Get("Location"); strings.Contains(loc, "err=inflight") {

+ t.Fatal("the connector is wedged: a run killed by a restart still blocks every later sync")

+ }

+ if !app2.WaitForSync(c.ID, 5*time.Second) {

+ t.Fatal("the new run did not finish")

+ }

+// TestNoNotionContentInLogs extends gate 4 to the connector. Titles,

+// bodies, ancestry, page ids and the token are all content or

+// coordinates, and `carlos logs` is storage we cannot grep.

+//

+// The fake's error carries a canary in a 404, which is the shape Notion

+// really uses — its 404 body quotes the page id and the connection name

+// — so the error path is exercised rather than assumed.

+//

+// MUTATION-TESTED 2026-09-05: logging ref.Title at Info in the fetch

+// loop, or %w-ing the notion error into the run's failure line, makes

+// this fail. Reverted.

+func TestNoNotionContentInLogs(t *testing.T) {

+ const secret = "WOMBATCONFIDENTIAL9931"

+ src := newSource()

+ src.add("root", secret+" handbook", "", "A passage about "+secret+" and revenue.")

+ src.add("kid", "Onboarding", "root", "Week one.", secret+" handbook")

+ src.failWith["kid"] = &notion.Error{Code: notion.CodeTraversalFailed, Status: 404}

+ h, app, cid, logs := connectorApp(t, src)

+ sync(t, h, app, cid)

+ get(t, h, "/connectors/notion")

+ got := logs.String()

+ if strings.Contains(got, secret) {

+ t.Errorf("a log line carries Notion content:\n%s", got)

+ }

+ if strings.Contains(got, "ntn_secret_token_value") {

+ t.Errorf("a log line carries the integration token:\n%s", got)

+ }

+ // Vacuity guard: the run must actually have logged something, or

+ // this scan proves nothing.

+ if !strings.Contains(got, "sync") {

+ t.Fatal("no sync log lines at all; this gate scanned nothing relevant")

+ }

+// TestTokenIsNeverRendered. A stored token must not come back to the

+// page — not in a value attribute, not masked. A masked secret is still

+// a secret in the DOM.

+//

+// "No substring" would be vacuous, since every single character is a

+// substring; the test looks for the whole token and for any run of eight

+// characters of it.

+//

+// MUTATION-TESTED 2026-09-05: rendering the token into the input's value

+// makes this fail. Reverted.

+func TestTokenIsNeverRendered(t *testing.T) {

+ const token = "ntn_secret_token_value"

+ src := newSource()

+ h, app, _, _ := connectorApp(t, src)

+ // The CONNECTED page, which is the one with a stored token and a

+ // field that could carry it. An earlier version of this gate looked

+ // only at the not-yet-connected page — which has no token field at

+ // all — so rendering the token into the rotation form left it green.

+ // Found by mutation-testing this gate.

+ body := get(t, h, "/connectors/notion").Body.String()

+ if !strings.Contains(body, `name="token"`) {

+ t.Fatal("the connected page has no token field, so this gate is scanning a page that could not leak one")

+ }

+ assertNoToken(t, "connected page", body, token)

+ // And the not-yet-connected page, for completeness.

+ if err := app.Store().DeleteConnector(context.Background(), "notion"); err != nil {

+ t.Fatal(err)

+ }

+ assertNoToken(t, "setup page", get(t, h, "/connectors/notion").Body.String(), token)

+// assertNoToken looks for the whole token and for any eight-character run

+// of it. "No substring" would be vacuous — every single character is a

+// substring of the token.

+func assertNoToken(t *testing.T, where, body, token string) {

+ t.Helper()

+ if strings.Contains(body, token) {

+ t.Errorf("%s renders the whole token", where)

+ return

+ }

+ for i := 0; i+8 <= len(token); i++ {

+ if strings.Contains(body, token[i:i+8]) {

+ t.Errorf("%s renders %q, an 8-character run of the token", where, token[i:i+8])

+ return

+ }

+// TestConnectorRoutesRefuseANonOwner.

+//

+// In slice 0 "owner" is the whole admitted set, so this gate is about

+// the mechanism rather than a live distinction — but the mechanism has

+// to exist and fail closed before slice 1 gives it something to

+// separate. It could not be written at all until AppForTest grew an

+// injectable identity.

+//

+// MUTATION-TESTED 2026-09-05: making requireOwner call next

+// unconditionally makes this fail. Reverted.

+func TestConnectorRoutesRefuseANonOwner(t *testing.T) {

+ src := newSource()

+ h, app, _, _ := connectorApp(t, src)

+ if code := get(t, h, "/connectors").Code; code != http.StatusOK {

+ t.Fatalf("an owner got %d from /connectors; this gate proves nothing if nobody can reach it", code)

+ }

+ app.AsPerson("stranger@example.com")

+ for _, target := range []string{"/connectors", "/connectors/notion"} {

+ // 404, not 403: a 403 confirms the route exists.

+ if code := get(t, h, target).Code; code != http.StatusNotFound {

+ t.Errorf("a non-owner got %d from %s, want 404", code, target)

+ }

+ if code := post(t, h, "/connectors/notion/sync", url.Values{}).Code; code != http.StatusNotFound {

+ t.Errorf("a non-owner could reach the sync route (%d)", code)

+ }

+// TestDisconnectKeepsTheDocuments. Parent §3.6 promises that

+// disconnecting stops memoria seeing new changes and does not remove

+// what memoria already holds. This is that promise as a test.

+//

+// MUTATION-TESTED 2026-09-05: deleting documents in DeleteConnector

+// makes this fail. Reverted.

+func TestDisconnectKeepsTheDocuments(t *testing.T) {

+ src := newSource()

+ src.add("root", "Handbook", "", "Everything a new starter needs about expenses.")

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ if hits, _ := app.Store().Search(context.Background(), "expenses", 10); len(hits) == 0 {

+ t.Fatal("nothing was imported, so keeping it proves nothing")

+ }

+ if code := post(t, h, "/connectors/notion/disconnect", url.Values{}).Code; code != http.StatusSeeOther {

+ t.Fatalf("disconnect: status %d", code)

+ }

+ if _, err := app.Store().Connector(context.Background(), "notion"); err == nil {

+ t.Error("the connector survived disconnect")

+ }

+ // The DATABASE, not only the index. An earlier version of this gate

+ // asserted searchability alone, and deleting the document rows left

+ // the in-memory index still answering — so the gate passed while the

+ // documents were gone. Found by mutation-testing it.

+ docs, err := app.Store().Summaries(context.Background())

+ if err != nil {

+ t.Fatal(err)

+ }

+ if len(docs) == 0 {

+ t.Error("disconnecting deleted the stored documents; what memoria imported, memoria keeps")

+ }

+ if hits, _ := app.Store().Search(context.Background(), "expenses", 10); len(hits) == 0 {

+ t.Error("disconnecting removed the documents from the index")

+ }

+// TestEmptyPageIsImportedAndCounted. Real workspaces are full of

+// placeholders. Failing the run on one lets a single empty page block an

+// import; skipping makes it an absence with nothing to explain it.

+//

+// MUTATION-TESTED 2026-09-05: turning AllowTitleOnly off in Syncer.ingest

+// makes the run record a failure instead of an empty page. Reverted.

+func TestEmptyPageIsImportedAndCounted(t *testing.T) {

+ src := newSource()

+ src.add("root", "Quarterly planning", "", "")

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ run := latest(t, app, cid)

+ if run.Empty != 1 {

+ t.Errorf("empty %d, want 1 (failed %d)", run.Empty, run.FailedPages)

+ }

+ if hits, _ := app.Store().Search(context.Background(), "quarterly planning", 10); len(hits) == 0 {

+ t.Error("a title-only page is not findable by the only words it has")

+ }

+// TestUnreadBlocksSurfaceOnTheStatusPage. A page that came in thin should

+// carry a reason rather than look like a thin page.

+func TestUnreadBlocksSurfaceOnTheStatusPage(t *testing.T) {

+ src := newSource()

+ src.add("root", "Notes", "", "A readable paragraph.")

+ src.unread["root"] = 3

+ h, app, cid, _ := connectorApp(t, src)

+ sync(t, h, app, cid)

+ if run := latest(t, app, cid); run.UnreadBlocks != 3 {

+ t.Fatalf("unread_blocks %d, want 3", run.UnreadBlocks)

+ }

+ if body := get(t, h, "/connectors/notion").Body.String(); !strings.Contains(body, "does not read") {

+ t.Error("the status page does not mention blocks memoria could not read")

+ }

diff --git a/tools/drive/main.go b/tools/drive/main.go

index 8c1461c..f3cabaf 100644

--- a/tools/drive/main.go

+++ b/tools/drive/main.go

@@ -24,9 +24,12 @@ import (

"os"

"path/filepath"

+ "time"

"github.com/carlosframework/rastrillo/db"

memoria "amadan.net/oficina/memoria/internal/memoria"

+ "amadan.net/oficina/memoria/internal/memoria/notion"

"amadan.net/oficina/memoria/internal/memoria/seal"

)

@@ -35,7 +38,10 @@ func main() {

flag.Parse()

logger := slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelWarn}))

- dir, err := os.MkdirTemp("/home/paulca/memoriadrive", "drive")

+ // The OS temp dir, not a hardcoded path under one person's home:

+ // the previous absolute path did not exist on any other machine, and

+ // MkdirTemp panics rather than falling back.

+ dir, err := os.MkdirTemp("", "memoria-drive")

if err != nil {

panic(err)

}

@@ -54,7 +60,7 @@ func main() {

panic(err)

}

- mux, _, err := memoria.AppForTest(d, "http://"+*addr, sealer, logger)

+ mux, app, err := memoria.AppForTest(d, "http://"+*addr, sealer, logger)

if err != nil {

panic(err)

}

@@ -63,6 +69,12 @@ func main() {

// trail, a highlight, a document list with more than one row.

seedViaHTTP(mux, *addr)

+ // And a connector with a finished import behind it, so the source

+ // pages are worth looking at. A template test proves the markup

+ // exists; it cannot show that a status page reads sensibly with real

+ // numbers in it.

+ seedConnector(app)

fmt.Printf("drive serving on http://%s (no sign-in)\n", *addr)

if err := http.ListenAndServe(*addr, mux); err != nil {

panic(err)

@@ -89,3 +101,55 @@ func seedViaHTTP(h http.Handler, addr string) {

}

+// driveSource is a small stand-in for Notion, so the connector pages have

+// a real tree and real counts behind them.

+type driveSource struct{ tree map[string][]notion.Ref }

+func (d *driveSource) Roots(ctx context.Context) ([]notion.Ref, error) {

+ return d.tree[""], nil

+func (d *driveSource) Fetch(ctx context.Context, ref notion.Ref) (*notion.Content, error) {

+ return &notion.Content{

+ Markdown: "## " + ref.Title + "\n\nThis page was imported from Notion during a drive run. " +

+ "It exists so the connector screens have something real to render.",

+ Children: d.tree[ref.ID],

+ UnreadBlocks: 2,

+ }, nil

+func seedConnector(app interface {

+ Store() *memoria.Store

+ UseFakeSource(memoria.SourceClient, func(context.Context, string) (string, error))

+ WaitForSync(int64, time.Duration) bool

+}) {

+ src := &driveSource{tree: map[string][]notion.Ref{

+ "": {{ID: "handbook", Title: "Handbook", EditedTime: "t1", Kind: notion.KindPage}},

+ "handbook": {

+ {ID: "onboarding", Title: "Onboarding", Path: []string{"Handbook"}, EditedTime: "t1", Kind: notion.KindPage},

+ {ID: "expenses", Title: "Expenses", Path: []string{"Handbook"}, EditedTime: "t1", Kind: notion.KindPage},

+ },

+ "onboarding": {

+ {ID: "week-one", Title: "Week one", Path: []string{"Handbook", "Onboarding"}, EditedTime: "t1", Kind: notion.KindPage},

+ },

+ }}

+ app.UseFakeSource(src, func(ctx context.Context, token string) (string, error) { return "drive-workspace", nil })

+ ctx := context.Background()

+ if err := app.Store().SaveConnector(ctx, "notion", "ntn_drive_token", "drive-workspace"); err != nil {

+ fmt.Fprintln(os.Stderr, "seed connector:", err)

+ return

+ }

+ c, err := app.Store().Connector(ctx, "notion")

+ if err != nil {

+ fmt.Fprintln(os.Stderr, "seed connector:", err)

+ return

+ }

+ run, err := app.Store().StartRun(ctx, c.ID)

+ if err != nil {

+ fmt.Fprintln(os.Stderr, "seed connector:", err)

+ return

+ }

+ memoria.NewSyncer(app.Store(), slog.New(slog.NewTextHandler(os.Stderr,

+ &slog.HandlerOptions{Level: slog.LevelWarn}))).RunWith(ctx, src, c.ID, run.ID)