amadan

oficina / memoria Public

feat: a seed tool, and a README you can follow at 8am

seed ingests a directory straight into the store — same chunking, same
sealing, no network, and it refuses to run without the DEK the app uses.
It exists so a fresh instance opens on a corpus worth searching, and it
is the honest ancestor of slice 3's connectors.

The local instance is seeded with every .md in the oficina tree: 55
documents, 649 passages. The first thing memoria can answer questions
about is the suite's own specs, which is a fair test of whether the
retrieval is any good.

Sign-in turned out to go through keymail OAuth rather than self-hosted
magic links, so the owners list carries both of Paul's addresses.
Verified keymail accepts a loopback client_id, so the local flow works in
a browser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SiWVaR69Vb2Lzc9wgfit1v
Paul Campbell pushed by paul@keymail.dev 2bf09d9015ade1f3e2c8e406ee08422e4a3d3d5c parent d14d4ef
8 files changed, +413 −8
  • README.md +102 −0
  • go.mod +1 −0
  • go.sum +2 −0
  • internal/memoria/index/index.go +33 −6
  • internal/memoria/index/index_test.go +40 −0
  • internal/memoria/render.go +2 −2
  • internal/memoriatest/app_test.go +116 −0
  • tools/seed/main.go +117 −0
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..84f67a8
--- /dev/null
+++ b/README.md
@@ -0,0 +1,102 @@
+# 🤖 memoria
+
+The Oficina suite's memory. One instance per team.
+
+Ask a question, get an answer grounded in your team's own documents —
+with a citation you can open, always.
+
+**Slice 0 is what exists today: a search engine, not an assistant.** It
+holds text you give it, seals it, and searches it. There is no model call
+anywhere in this binary, which means the honest version of the trust
+claim is unusually strong right now:
+
+> memoria's passages and its index are sealed at rest under this team's
+> own key. **Nothing leaves this instance** — not at query time, and not
+> at indexing time, because nothing is sent anywhere to be indexed.
+
+That changes the day the assistant arrives, and the design spec is
+explicit about how: embeddings come from a hosted API, so switching the
+assistant on sends **every document in the corpus** to a named provider
+once, at indexing time — not just the passages behind an answer. See
+§3.2 and §3.6 of the design.
+
+## Trying it locally
+
+```
+cd /home/paulca/memoriadrive
+./run.sh
+```
+
+Then open **http://127.0.0.1:8823** and sign in. Sign-in goes through
+keymail, so use an address on the owners list in `run.sh`
+(`paul@tito.io` or `paul@keymail.dev` as it stands).
+
+It comes pre-seeded with 55 documents and 649 passages — every `.md` in
+the `oficina` tree, so the first thing you can search is the suite's own
+specs. Try `sealed at rest`, `grants join`, `revoke`, `escritorio`,
+`plausible empty list`.
+
+Things worth poking at, because each is a decision rather than an
+accident:
+
+- Search **ORs** its terms. Ask it a question in words — "what is the
+ worst available failure shape" — and it works. ANDing would return
+ nothing, and nothing is indistinguishable from having nothing.
+- A no-match page tells you how much *is* indexed, so an absent answer
+ never looks like an empty index.
+- Every result cites a document and a passage number, and the passage
+ anchors when you follow it.
+
+### Adding your own
+
+Paste or upload text on the front page, or seed a directory:
+
+```
+cd /home/paulca/memoriadrive
+MEMORIA_DEK="$(cat DEK.txt)" ./seed --db m.db --dir /path/to/notes --source notes
+```
+
+The seed tool uses the same store, chunker and sealing as the app. It is
+the honest ancestor of slice 3's connectors, and it bypasses nothing —
+without the right DEK it refuses to run.
+
+### The DEK
+
+`DEK.txt` in `memoriadrive` is the key everything is sealed with. **Lose
+it and the corpus is gone**, exactly as with Sheets' `SHEETS_DEK`. The
+app never generates one: a generated fallback would seal this boot's data
+under a key the next boot cannot derive, losing the data quietly instead
+of refusing loudly.
+
+## Deploying
+
+Not done — this has never run anywhere but localhost. When you want it on
+the estate:
+
+```
+make release
+carlos deploy --app memoria --account bds --channel edge --timeout 5m ./releases/memoria-arm64
+```
+
+Flags go **before** the binary path; Go's flag package stops parsing at
+the first non-flag argument, so a path first makes `--account` invisible.
+`carlos deploy`, never `carlos ship` on its own. Then poll
+`GET /api/version` rather than trusting `X-Carlos-Version`, which is the
+router's adopted release and flips before the process cycles.
+
+## The design
+
+`docs/superpowers/specs/2026-09-03-memoria-design.md`. Draft 2, which
+answers independent reviews by GPT Sol and Fable that both returned
+REQUEST CHANGES against draft 1. §1.2 records what changed and why — the
+central correction being that authorization is asked of whichever app
+owns a document, per person, per query, rather than derived from a
+membership table memoria keeps.
+
+`QUESTIONS-FOR-PAUL.md` is what I could not decide alone.
+
+## What is not here
+
+No assistant, no embeddings, no sibling apps, no external connectors, no
+snapshot of the index (it rebuilds at boot — milliseconds at this size,
+and §12 carries the trigger for changing that). The sequencing is §11.
diff --git a/go.mod b/go.mod
index 2854485..088d0aa 100644
--- a/go.mod
+++ b/go.mod
@@ -11,6 +11,7 @@ require (
)
require (
+ github.com/BurntSushi/toml v1.6.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
diff --git a/go.sum b/go.sum
index 8024e0f..6ee94a5 100644
--- a/go.sum
+++ b/go.sum
@@ -1,3 +1,5 @@
+github.com/BurntSushi/toml v1.6.0 h1:dRaEfpa2VI55EwlIW72hMRHdWouJeRF7TPYhI+AUQjk=
+github.com/BurntSushi/toml v1.6.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
github.com/carlosframework/rastrillo v0.23.0 h1:tKAAlwPt4Pg/rkqk7o6jvV12bbZ0r34oEVpuOlKfX7w=
github.com/carlosframework/rastrillo v0.23.0/go.mod h1:pZlrE5F5OhspvZFzef0DrfjHAjA8FXycb9J5/8vc8Tw=
github.com/chromedp/cdproto v0.0.0-20250724212937-08a3db8b4327 h1:UQ4AU+BGti3Sy/aLU8KVseYKNALcX9UXY6DfpwQ6J8E=
diff --git a/internal/memoria/index/index.go b/internal/memoria/index/index.go
index 0d9f9b3..d06af09 100644
--- a/internal/memoria/index/index.go
+++ b/internal/memoria/index/index.go
@@ -13,7 +13,9 @@
package index
import (
+ "crypto/rand"
"database/sql"
+ "encoding/hex"
"fmt"
"strings"
"sync"
@@ -44,12 +46,25 @@ type Index struct {
// Open builds an empty index. The DSN is in-memory with a shared cache so
// the single handle is stable; nothing is written to any file.
func Open() (*Index, error) {
- // `file::memory:` with cache=shared keeps one database alive across
- // pooled connections. Without it each connection gets its own empty
- // database and searches return nothing — a plausible empty list,
- // which is the failure shape this codebase treats as the worst
- // available, so it is worth the explicit DSN.
- db, err := sql.Open("sqlite", "file:memoria-index?mode=memory&cache=shared")
+ // `cache=shared` keeps one database alive across pooled connections.
+ // Without it each connection gets its own empty database and searches
+ // return nothing — a plausible empty list, which is the failure shape
+ // this codebase treats as the worst available, so it is worth the
+ // explicit DSN.
+ //
+ // The name must be UNIQUE PER INDEX, and that is the load-bearing
+ // half. A shared cache is keyed by name, so a fixed name makes every
+ // Index in the process the same database. Two apps in one process
+ // then share one index: the second Open fails with "table chunks
+ // already exists" if you are lucky, and if the schema ever matched it
+ // would instead merge two teams' passages into one searchable
+ // corpus — the cross-team leak §6.3 exists to make impossible. Found
+ // by the test suite building two apps in one process.
+ name, err := randomName()
+ if err != nil {
+ return nil, err
+ }
+ db, err := sql.Open("sqlite", "file:"+name+"?mode=memory&cache=shared")
if err != nil {
return nil, err
}
@@ -218,3 +233,15 @@ func MatchQuery(raw string) string {
}
return strings.Join(terms, " OR ")
}
+
+// randomName gives each in-memory index its own shared-cache key. Random
+// rather than a counter: a counter resets when the process restarts, and
+// two indexes alive across a restart boundary in a test binary would
+// collide again.
+func randomName() (string, error) {
+ b := make([]byte, 8)
+ if _, err := rand.Read(b); err != nil {
+ return "", fmt.Errorf("index: name: %w", err)
+ }
+ return "memoria-index-" + hex.EncodeToString(b), nil
+}
diff --git a/internal/memoria/index/index_test.go b/internal/memoria/index/index_test.go
index 24edb99..d241c87 100644
--- a/internal/memoria/index/index_test.go
+++ b/internal/memoria/index/index_test.go
@@ -155,3 +155,43 @@ func TestOrRecallStillRanksBestMatchFirst(t *testing.T) {
t.Errorf("best hit is chunk %d, want 2 — ranking is not compensating for OR", hits[0].ChunkID)
}
}
+
+// Two indexes in one process must be independent. They were not: the
+// shared-cache DSN used a fixed name, so every Index was the same
+// in-memory database. The visible symptom was "table chunks already
+// exists"; the invisible one, had the schema matched, would have been two
+// teams' passages merged into one searchable corpus.
+func TestTwoIndexesAreIndependent(t *testing.T) {
+ a, err := Open()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer a.Close()
+ b, err := Open()
+ if err != nil {
+ t.Fatalf("opening a second index failed: %v", err)
+ }
+ defer b.Close()
+
+ if err := a.Add(Entry{ChunkID: 1, DocumentID: 1, Heading: "A", Text: "alpha secret"}); err != nil {
+ t.Fatal(err)
+ }
+ hits, err := b.Search("alpha", 10)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(hits) != 0 {
+ t.Fatalf("index B can see index A's passages: %+v — indexes are sharing state", hits)
+ }
+ na, err := a.Count()
+ if err != nil {
+ t.Fatal(err)
+ }
+ nb, err := b.Count()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if na != 1 || nb != 0 {
+ t.Errorf("counts are A=%d B=%d, want 1 and 0", na, nb)
+ }
+}
diff --git a/internal/memoria/render.go b/internal/memoria/render.go
index f77a30d..9c60571 100644
--- a/internal/memoria/render.go
+++ b/internal/memoria/render.go
@@ -35,8 +35,8 @@ func init() {
// chosen delivery needs — empty for the vendored-inline
// default, so the layout calls it unconditionally and
// switching delivery later needs no template edit.
- "icon": icons.Icon,
- "iconAssets": icons.Assets,
+ "icon": icons.Icon,
+ "iconAssets": icons.Assets,
"safeSnippet": safeSnippet,
}).
ParseFS(appFS, "templates/layout.html", "templates/"+name+".html"))
diff --git a/internal/memoriatest/app_test.go b/internal/memoriatest/app_test.go
new file mode 100644
index 0000000..8c66f52
--- /dev/null
+++ b/internal/memoriatest/app_test.go
@@ -0,0 +1,116 @@
+package memoriatest
+
+import (
+ "net/http"
+ "strings"
+ "testing"
+)
+
+func TestStoreThenFind(t *testing.T) {
+ h := newApp(t)
+ if rec := store(t, h, "Q3 board pack", "## Revenue\n\nJune revenue was 1,350,000 across all regions.\n\n## Costs\n\nHosting fell after the migration."); rec.Code != http.StatusSeeOther {
+ t.Fatalf("store: status %d, want 303", rec.Code)
+ }
+ body := get(t, h, "/?q=june+revenue").Body.String()
+ if !strings.Contains(body, "Q3 board pack") {
+ t.Errorf("search does not show the document title:\n%s", body)
+ }
+ if !strings.Contains(body, "Revenue") {
+ t.Error("search result carries no heading trail")
+ }
+ if !strings.Contains(body, "<mark>") {
+ t.Error("search result carries no highlight")
+ }
+}
+
+// The list must distinguish "nothing matched" from "nothing indexed".
+// Conflating them is the plausible-empty-list failure.
+func TestNoMatchSaysTheCorpusIsNotEmpty(t *testing.T) {
+ h := newApp(t)
+ store(t, h, "Notes", strings.Repeat("hosting migration notes. ", 20))
+ body := get(t, h, "/?q=xylophone").Body.String()
+ if !strings.Contains(body, "0 results") {
+ t.Errorf("no-match page does not state the result count:\n%s", body)
+ }
+ if !strings.Contains(body, "indexed") {
+ t.Error("no-match page does not say how much is indexed, so a reader " +
+ "cannot tell an absent answer from an empty index")
+ }
+}
+
+// Stored text is untrusted. A document containing markup must not
+// execute in a search result or on the document page.
+func TestStoredMarkupIsEscaped(t *testing.T) {
+ h := newApp(t)
+ const nasty = `<script>alert('xss')</script>`
+ store(t, h, "Payload "+nasty, "## Heading "+nasty+"\n\nA passage containing "+nasty+" inline and revenue too.")
+
+ for _, target := range []string{"/?q=revenue", "/documents/1"} {
+ body := get(t, h, target).Body.String()
+ if strings.Contains(body, "<script>") {
+ t.Errorf("%s renders an unescaped <script>:\n%s", target, body)
+ }
+ if !strings.Contains(body, "&lt;script&gt;") {
+ t.Errorf("%s does not show the escaped form, so the payload may have been dropped "+
+ "rather than escaped — check the snippet path", target)
+ }
+ }
+}
+
+// Re-storing the same text updates one document rather than making two
+// that disagree. The ExternalID is the content hash, so this is what
+// makes a re-upload idempotent.
+func TestReStoringSameTextDoesNotDuplicate(t *testing.T) {
+ h := newApp(t)
+ const text = "## Revenue\n\nJune revenue was 1,350,000 across every region we serve."
+ store(t, h, "First", text)
+ store(t, h, "Second", text)
+ body := get(t, h, "/").Body.String()
+ if !strings.Contains(body, "1 document") {
+ t.Errorf("re-storing identical text made more than one document:\n%s", body)
+ }
+}
+
+// A changed document must not leave its old passages searchable: a
+// citation to text that is no longer there is worse than a missing one.
+func TestRevisingRemovesTheOldPassages(t *testing.T) {
+ h := newApp(t)
+ store(t, h, "Doc", "## Revenue\n\nThe zebrafinch figure was 100.")
+ if !strings.Contains(get(t, h, "/?q=zebrafinch").Body.String(), "Doc") {
+ t.Fatal("the first version was not indexed")
+ }
+ // Same title, different body: a new document under this design,
+ // since the hash differs. Delete the first and confirm it is gone.
+ if rec := post(t, h, "/documents/1/delete", nil); rec.Code != http.StatusSeeOther {
+ t.Fatalf("delete: status %d, want 303", rec.Code)
+ }
+ body := get(t, h, "/?q=zebrafinch").Body.String()
+ if strings.Contains(body, "<mark>") {
+ t.Errorf("a deleted document is still searchable:\n%s", body)
+ }
+}
+
+func TestMissingDocument404s(t *testing.T) {
+ if rec := get(t, newApp(t), "/documents/999"); rec.Code != http.StatusNotFound {
+ t.Errorf("GET /documents/999: status %d, want 404", rec.Code)
+ }
+}
+
+func TestEmptySubmissionIsRefusedWithCopy(t *testing.T) {
+ h := newApp(t)
+ if rec := store(t, h, "", " "); rec.Code != http.StatusSeeOther {
+ t.Fatalf("empty store: status %d, want 303", rec.Code)
+ }
+ if !strings.Contains(get(t, h, "/?err=empty").Body.String(), "nothing to store") {
+ t.Error("the empty-submission page does not say what went wrong")
+ }
+}
+
+// The footer states the no-egress property on every screen. It is the
+// product's main honest claim in slice 0, so it is worth a test.
+func TestFooterStatesNoEgress(t *testing.T) {
+ body := get(t, newApp(t), "/").Body.String()
+ if !strings.Contains(body, "nothing leaves this instance") {
+ t.Errorf("the page does not state the no-egress property:\n%s", body)
+ }
+}
diff --git a/tools/seed/main.go b/tools/seed/main.go
new file mode 100644
index 0000000..9ac7dbf
--- /dev/null
+++ b/tools/seed/main.go
@@ -0,0 +1,117 @@
+// Command seed ingests text files straight into a memoria database,
+// without going through HTTP.
+//
+// It exists so a fresh instance can be opened on a corpus that is already
+// worth searching, and it is the honest ancestor of slice 3's connectors:
+// same Store, same chunking, same sealing, no network. Nothing here
+// bypasses the trust model — it needs the same DEK the app runs with, and
+// what it writes is sealed exactly as the app would seal it.
+package main
+
+import (
+ "context"
+ "encoding/base64"
+ "flag"
+ "fmt"
+ "io/fs"
+ "log/slog"
+ "os"
+ "path/filepath"
+ "strings"
+
+ "github.com/carlosframework/rastrillo/db"
+
+ memoria "amadan.net/oficina/memoria/internal/memoria"
+ "amadan.net/oficina/memoria/internal/memoria/index"
+ "amadan.net/oficina/memoria/internal/memoria/seal"
+)
+
+func main() {
+ dbPath := flag.String("db", "", "path to the memoria database")
+ root := flag.String("dir", "", "directory to walk for .md and .txt files")
+ source := flag.String("source", "seed", "the Source recorded on each document")
+ flag.Parse()
+ if *dbPath == "" || *root == "" {
+ fmt.Fprintln(os.Stderr, "usage: seed --db <path> --dir <path> [--source name]")
+ os.Exit(2)
+ }
+
+ dekRaw := os.Getenv("MEMORIA_DEK")
+ if dekRaw == "" {
+ fmt.Fprintln(os.Stderr, "MEMORIA_DEK is not set; refusing to seed with a key the app cannot derive")
+ os.Exit(1)
+ }
+ dek, err := base64.StdEncoding.DecodeString(dekRaw)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "MEMORIA_DEK is not valid base64: %v\n", err)
+ os.Exit(1)
+ }
+ sealer, err := seal.New(dek)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "MEMORIA_DEK rejected: %v\n", err)
+ os.Exit(1)
+ }
+
+ logger := slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelWarn}))
+ d, err := db.Open(*dbPath, logger)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "open database: %v\n", err)
+ os.Exit(1)
+ }
+ defer d.Close()
+
+ ix, err := index.Open()
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "open index: %v\n", err)
+ os.Exit(1)
+ }
+ defer ix.Close()
+
+ store := memoria.NewStore(d, sealer, ix)
+ ctx := context.Background()
+
+ var n, skipped int
+ err = filepath.WalkDir(*root, func(p string, e fs.DirEntry, err error) error {
+ if err != nil {
+ return err
+ }
+ if e.IsDir() {
+ // Anything under a dot-directory or node_modules is not the
+ // team's writing.
+ base := e.Name()
+ if strings.HasPrefix(base, ".") || base == "node_modules" || base == "releases" {
+ return fs.SkipDir
+ }
+ return nil
+ }
+ switch strings.ToLower(filepath.Ext(p)) {
+ case ".md", ".txt", ".markdown":
+ default:
+ return nil
+ }
+ raw, err := os.ReadFile(p)
+ if err != nil {
+ return err
+ }
+ if len(strings.TrimSpace(string(raw))) == 0 {
+ skipped++
+ return nil
+ }
+ rel, _ := filepath.Rel(*root, p)
+ title := rel
+ if _, err := store.Ingest(ctx, *source, title, string(raw)); err != nil {
+ fmt.Fprintf(os.Stderr, "skip %s: %v\n", rel, err)
+ skipped++
+ return nil
+ }
+ n++
+ fmt.Printf("seeded %s\n", rel)
+ return nil
+ })
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "walk: %v\n", err)
+ os.Exit(1)
+ }
+ count, _ := store.Count()
+ fmt.Printf("\n%d documents seeded, %d skipped, %d passages indexed\n", n, skipped, count)
+}