amadan

oficina / memoria Public

perf: measure the index instead of asserting it, and find §7 was wrong

A review of draft 2 said every figure in §7 was an unrecorded assertion
and that the FTS size could not describe the stated schema. Both fair, so
tools/bench now measures it on a real corpus chunked by memoria's own
chunker, and docs/BENCHMARKS.md records the machine and command.

Three results.

The cosine table was right: 3.2 GFLOP/s measured, and every latency in
draft 2 correct to within a millisecond.

A memoria chunk averages 202 words / 1325 bytes, not the 400 words the
spec assumed — the chunker folds runts and never straddles a heading, and
real documents have headings every few paragraphs.

The FTS claim of 104 MB was wrong by half: a default FTS5 table keeps its
own copy of the text, and the real figure is 200 MB at 90k chunks. A
contentless table is 49 MB and queries just as fast, but cannot serve
snippet(). Recomputing the wake footprint puts the ceiling at ~90-110k
chunks, not draft 2's 150k — so draft 2 moved that number the wrong way
while claiming to have fixed the arithmetic. Draft 1's ~100k was closer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SiWVaR69Vb2Lzc9wgfit1v
Paul Campbell pushed by paul@keymail.dev 6f6ecea49b7b9d6d5678d57e3b12f3a878c15f6f parent 2bf09d9
2 files changed, +289 −0
  • docs/BENCHMARKS.md +86 −0
  • tools/bench/main.go +203 −0
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
new file mode 100644
index 0000000..2ed23af
--- /dev/null
+++ b/docs/BENCHMARKS.md
@@ -0,0 +1,86 @@
+# Recorded measurements
+
+A review of draft 2 observed that every performance figure in §7 was an
+assertion with no benchmark, command, machine or sample behind it, and
+that the FTS size could not be reconciled with the stated schema. Both
+were fair. This file is the answer: `tools/bench` produces it, and §7
+cites it rather than restating numbers.
+
+## Machine and command
+
+```
+linux/amd64, 16 cpu, go1.26.5
+go build -o bench ./tools/bench
+./bench --corpus /home/paulca/amadan.net/oficina --chunks 90000
+```
+
+Run 2026-09-03. The corpus is every `.md` in the oficina tree, chunked by
+**memoria's own chunker** (`internal/memoria/chunk`), then cycled to reach
+the target count. Cycling repeats vocabulary, so the FTS figures are
+**optimistic**: genuinely varied English produces a larger postings list.
+
+## What a memoria chunk actually is
+
+650 chunks from the real corpus, **mean 202 words / 1325 bytes**.
+
+This corrects draft 2, which assumed 400 words and ~1.6 KB. The chunker
+targets 400 words but folds runts and never straddles a heading, and real
+documents have headings every few paragraphs — so the mean lands at about
+half the target. Every figure below follows from the measured size, not
+the intended one.
+
+At 90,000 chunks that is **113.8 MB of text**.
+
+## FTS5
+
+| schema | index size | bytes/chunk | build | one BM25 query |
+|---|---|---|---|---|
+| `fts5(body, tokenize='porter unicode61')` | **200.0 MB** | 2329 | 6.8 s | 1.09 ms |
+| `fts5(body, content='', tokenize=…)` | **49.3 MB** | 574 | 6.2 s | 1.09 ms |
+
+**Draft 2 claimed 104 MB. That was wrong by about half.** A default FTS5
+table keeps its own copy of the inserted text, which the earlier
+measurement did not account for; the review caught this before the
+arithmetic downstream of it was trusted.
+
+The contentless table is **4× smaller** and just as fast to query, but it
+cannot return column values, so `snippet()` does not work on it and
+memoria would have to build its own highlight from the sealed chunk. That
+is a real trade and §7 records it as a decision rather than an accident.
+
+## Brute-force cosine, one goroutine, pure Go, no SIMD
+
+Best of three runs. Draft 2 asserted 3.2 GFLOP/s; measured, it is 3.2
+GFLOP/s, and every latency in draft 2's table was right to within a
+millisecond.
+
+| chunks × dims | measured | GFLOP/s | vectors f32 | vectors int8 |
+|---|---|---|---|---|
+| 10,000 × 512 | 3.18 ms | 3.2 | 20 MB | 5 MB |
+| 90,000 × 512 | 28.9 ms | 3.2 | 176 MB | 44 MB |
+| 100,000 × 768 | 49.2 ms | 3.1 | 293 MB | 73 MB |
+| 250,000 × 512 | 80.0 ms | 3.2 | 488 MB | 122 MB |
+
+## The wake footprint, recomputed
+
+At 90,000 chunks:
+
+| design | FTS | vectors | separate text | total |
+|---|---|---|---|---|
+| default FTS (text lives in the index) | 200 MB | 44 MB | — | **244 MB** |
+| contentless FTS + text for snippets | 49 MB | 44 MB | 114 MB | **207 MB** |
+
+So the ceiling under a 250 MB / 5-second budget is **about 90,000–110,000
+chunks**, not the 150,000 draft 2 derived. Draft 1's ~100,000 was closer,
+and draft 2's revision moved the number the wrong way while claiming to
+have fixed the arithmetic.
+
+## Still not measured
+
+- Bucket read bandwidth and AES-GCM throughput on the platform. The
+ 50–100 MB/s and ~1 GB/s in §7.3 remain assumptions, and they are the
+ inputs the ceiling is most sensitive to.
+- The rest of a cold start: opening SQLite, allocating the vector matrix,
+ parsing the snapshot, building id maps. The 5-second budget is spent on
+ transfer and decryption alone, which flatters it.
+- A corpus of genuinely varied English rather than a cycled one.
diff --git a/tools/bench/main.go b/tools/bench/main.go
new file mode 100644
index 0000000..9a280b4
--- /dev/null
+++ b/tools/bench/main.go
@@ -0,0 +1,203 @@
+// Command bench measures the index, so §7's numbers are recorded
+// measurements rather than assertions.
+//
+// A review of draft 2 pointed out that every figure in §7 — 3.2 GFLOP/s,
+// 104 MB, 19 s — had no benchmark, command or machine behind it, and that
+// the FTS size could not be reconciled with the stated schema, because a
+// default FTS5 table keeps its own copy of the inserted text. Both are
+// fair, and both are answerable by measuring instead of arguing.
+package main
+
+import (
+ "crypto/rand"
+ "database/sql"
+ "flag"
+ "fmt"
+ "math"
+ "os"
+ "path/filepath"
+ "runtime"
+ "strings"
+ "time"
+
+ _ "modernc.org/sqlite"
+
+ "amadan.net/oficina/memoria/internal/memoria/chunk"
+)
+
+func main() {
+ corpus := flag.String("corpus", "", "directory of real text to build the corpus from")
+ target := flag.Int("chunks", 90000, "how many chunks to grow the corpus to")
+ flag.Parse()
+
+ fmt.Printf("machine: %s/%s, %d cpu, go %s\n\n", runtime.GOOS, runtime.GOARCH, runtime.NumCPU(), runtime.Version())
+
+ passages := realPassages(*corpus)
+ if len(passages) == 0 {
+ fmt.Fprintln(os.Stderr, "no corpus found; pass --corpus")
+ os.Exit(1)
+ }
+ avgWords := 0
+ avgBytes := 0
+ for _, p := range passages {
+ avgWords += len(strings.Fields(p))
+ avgBytes += len(p)
+ }
+ fmt.Printf("real corpus: %d passages, mean %d words / %d bytes per passage\n",
+ len(passages), avgWords/len(passages), avgBytes/len(passages))
+
+ // Grow to the target by cycling the real passages. Cycling repeats
+ // vocabulary, which makes the postings list SMALLER than genuinely
+ // varied English would — so the FTS figure below is optimistic, and
+ // says so.
+ grown := make([]string, 0, *target)
+ for i := 0; i < *target; i++ {
+ grown = append(grown, passages[i%len(passages)])
+ }
+ totalBytes := 0
+ for _, p := range grown {
+ totalBytes += len(p)
+ }
+ fmt.Printf("grown corpus: %d chunks, %.1f MB of text\n\n", len(grown), float64(totalBytes)/(1<<20))
+
+ for _, contentless := range []bool{false, true} {
+ measureFTS(grown, contentless)
+ }
+ measureCosine()
+}
+
+func measureFTS(chunks []string, contentless bool) {
+ label := "default (keeps its own copy of the text)"
+ schema := `CREATE VIRTUAL TABLE c USING fts5(body, tokenize='porter unicode61');`
+ if contentless {
+ label = "contentless (content='')"
+ schema = `CREATE VIRTUAL TABLE c USING fts5(body, content='', tokenize='porter unicode61');`
+ }
+ dir, err := os.MkdirTemp("/home/paulca/memoriadrive", "bench")
+ if err != nil {
+ panic(err)
+ }
+ defer os.RemoveAll(dir)
+ path := filepath.Join(dir, "fts.db")
+ db, err := sql.Open("sqlite", "file:"+path)
+ if err != nil {
+ panic(err)
+ }
+ defer db.Close()
+ if _, err := db.Exec(schema); err != nil {
+ panic(err)
+ }
+
+ start := time.Now()
+ tx, _ := db.Begin()
+ st, err := tx.Prepare(`INSERT INTO c (body) VALUES (?)`)
+ if err != nil {
+ panic(err)
+ }
+ for _, ch := range chunks {
+ if _, err := st.Exec(ch); err != nil {
+ panic(err)
+ }
+ }
+ st.Close()
+ if err := tx.Commit(); err != nil {
+ panic(err)
+ }
+ if _, err := db.Exec(`INSERT INTO c(c) VALUES('optimize');`); err != nil {
+ panic(err)
+ }
+ build := time.Since(start)
+ db.Exec(`PRAGMA wal_checkpoint(TRUNCATE);`)
+
+ var size int64
+ for _, suffix := range []string{"", "-wal"} {
+ if fi, err := os.Stat(path + suffix); err == nil {
+ size += fi.Size()
+ }
+ }
+
+ // A query, warm.
+ qstart := time.Now()
+ var n int
+ row := db.QueryRow(`SELECT count(*) FROM c WHERE c MATCH '"revenue" OR "sealed"'`)
+ if err := row.Scan(&n); err != nil {
+ fmt.Printf(" query failed: %v\n", err)
+ }
+ q := time.Since(qstart)
+
+ fmt.Printf("FTS5 %s\n", label)
+ fmt.Printf(" index on disk : %.1f MB (%d bytes/chunk)\n", float64(size)/(1<<20), size/int64(len(chunks)))
+ fmt.Printf(" build time : %s\n", build.Round(time.Millisecond))
+ fmt.Printf(" one BM25 query: %s (%d rows matched)\n\n", q.Round(time.Microsecond), n)
+}
+
+// measureCosine times a brute-force pass, the number §7's latency table
+// is derived from.
+func measureCosine() {
+ fmt.Println("brute-force cosine, one goroutine, pure Go, no SIMD")
+ for _, c := range []struct{ n, d int }{{10000, 512}, {90000, 512}, {100000, 768}, {250000, 512}} {
+ vecs := make([]float32, c.n*c.d)
+ buf := make([]byte, len(vecs)*4)
+ rand.Read(buf)
+ for i := range vecs {
+ vecs[i] = float32(int8(buf[i*4])) / 128
+ }
+ q := make([]float32, c.d)
+ for i := range q {
+ q[i] = float32(i%17) / 17
+ }
+ // Three runs, report the best — a slow run measures the
+ // scheduler, not the arithmetic.
+ best := time.Duration(math.MaxInt64)
+ for run := 0; run < 3; run++ {
+ start := time.Now()
+ var bestScore float32
+ for i := 0; i+c.d <= len(vecs); i += c.d {
+ var dot float32
+ row := vecs[i : i+c.d]
+ for j, v := range row {
+ dot += v * q[j]
+ }
+ if dot > bestScore {
+ bestScore = dot
+ }
+ }
+ if e := time.Since(start); e < best {
+ best = e
+ }
+ }
+ flops := float64(2 * c.n * c.d)
+ fmt.Printf(" %7d chunks x %d dims: %8s (%.1f GFLOP/s) vectors %.0f MB f32 / %.0f MB int8\n",
+ c.n, c.d, best.Round(time.Microsecond), flops/best.Seconds()/1e9,
+ float64(c.n*c.d*4)/(1<<20), float64(c.n*c.d)/(1<<20))
+ }
+}
+
+// realPassages reads .md files and chunks them with MEMORIA'S OWN
+// chunker, so the measurement describes the real product rather than an
+// approximation of it. An earlier version split on blank lines and
+// reported a mean of 81 words per passage, which is a paragraph, not a
+// memoria chunk.
+func realPassages(dir string) []string {
+ var out []string
+ if dir == "" {
+ return nil
+ }
+ filepath.Walk(dir, func(p string, info os.FileInfo, err error) error {
+ if err != nil || info.IsDir() {
+ return nil
+ }
+ if !strings.HasSuffix(p, ".md") || strings.Contains(p, "/.") || strings.Contains(p, "node_modules") {
+ return nil
+ }
+ raw, err := os.ReadFile(p)
+ if err != nil {
+ return nil
+ }
+ for _, c := range chunk.Split(string(raw)) {
+ out = append(out, c.Indexable())
+ }
+ return nil
+ })
+ return out
+}