| new file mode 100644 |
| index 0000000..3580c0f |
| --- /dev/null |
| +++ b/docs/sealed-search-index.md |
| @@ -0,0 +1,161 @@ |
| +# 🤖 Sealed search index — plan for message search at business scale |
| + |
| +**Status: designed 2026-07-14, not built.** This documents the agreed upgrade |
| +path for client-side message search before large (Business-tier, ~1 TB |
| +allowance) spaces make today's design hurt. Nothing here is urgent yet — the |
| +triggers below say when to build it. Companion decisions this plan leans on: |
| +the storage/durability ladder (`tier-economics.md § Durability per tier`) and |
| +the R2 attachment offload (#550), which keeps media out of the search path |
| +entirely. |
| + |
| +## Where search stands today |
| + |
| +Search is **local on every platform** — the query never leaves the device, |
| +and the server never gains a search surface. That's a consequence of the E2EE |
| +rule, not a feature toggle, and this plan does not bend it. |
| + |
| +- **Web** (#586 + the #228 "time machine"): the local store (`store.js`) |
| + holds raw **ciphertext** rows in IndexedDB; `search.js`'s `SearchIndex` |
| + decrypts each thread **once per session into an in-memory index** (never |
| + persisted), and queries are substring scans over it. "Search earlier |
| + messages" walks incomplete threads' history into the store (`runBackfill`) |
| + and streams new hits in. The design's own comments accept the limits and |
| + say "revisit if a real space ever makes this hurt". |
| +- **iOS / Mac**: the App-Group `MessageStore` keeps **decrypted payloads** in |
| + SQLite with an **FTS5** twin table (`messages_fts`); `MessageStore.search` |
| + is a `MATCH` join, newest-first, limit 50. At-rest protection is the OS's |
| + (file-level Data Protection), which is the platform-normal posture — |
| + Signal and WhatsApp do the same. |
| + |
| +Two honest asymmetries fall out of that, worth naming because this plan |
| +resolves one and keeps the other: |
| + |
| +1. **Semantics differ**: web matches *substrings* (`matchText` uses |
| + `includes`), native matches FTS5 *word/prefix terms*. Searching "ana" |
| + finds "banana" on web and not on iOS. This plan aligns both on |
| + word-prefix (below). |
| +2. **At-rest posture differs**: native stores plaintext under OS file |
| + encryption; web deliberately stores only ciphertext. This plan **keeps** |
| + web's stricter posture — that's what "sealed" means. |
| + |
| +## Why web search will hurt, and when |
| + |
| +The in-memory index costs one full-store decrypt per tab session plus RAM |
| +proportional to the whole corpus's text. Media never enters the path (the |
| +store caches message rows only; attachment bytes fetch on demand), so the |
| +problem scales with **history length, not storage bytes**: |
| + |
| +| Space | Rows on device | First search (est.) | Index RAM (est.) | |
| +|---|---|---|---| |
| +| Family, years | ~50k | ~1–2 s | ~10 MB | |
| +| Community, years | ~300k | ~10 s | ~50 MB | |
| +| Business, a decade | ~2M | minutes | 300+ MB | |
| + |
| +A phone browser will evict or kill the tab well before the last row. The |
| +scan is fine to roughly **100–200k rows per device**; beyond that every |
| +session pays the full decrypt again. |
| + |
| +**Triggers to build this** (client-side heuristics, no analytics — measure |
| +locally and log to console): a device's store crosses ~150k rows, or a |
| +measured first-search ensure pass exceeds ~3 s. Either observed in a real |
| +space = schedule the work. |
| + |
| +## Design: a sealed, disposable index beside the ciphertext store |
| + |
| +One sentence: **persist the derived index the session index already builds, |
| +encrypted at rest under a device-local key, and query it via postings so a |
| +search decrypts a few shards and a handful of candidate rows instead of the |
| +whole store.** |
| + |
| +### Invariants (the non-negotiables) |
| + |
| +- **Never plaintext at rest in web storage.** Index shards are AES-GCM |
| + ciphertext in IndexedDB, sealed under a **device-local, non-extractable |
| + WebCrypto key** stored alongside (structured-clone of a `CryptoKey`). |
| + `store.js`'s ciphertext-only rule survives intact. |
| +- **The index is a cache, never truth.** It is derived from the ciphertext |
| + store, deletable at any time, rebuilt lazily from the store. Schema-version |
| + mismatch → drop and rebuild. Sign-out → delete with everything else. |
| +- **Postings are hints.** Every hit re-decrypts its row from the store and |
| + re-verifies the match before display — so stale postings from edits, |
| + deletes, and tombstones fall out silently rather than needing synchronous |
| + index maintenance. |
| +- **Per-device coverage, honestly stated.** The index covers what this |
| + device's store holds; the existing backfill narration ("searching back to |
| + 2024…") already sets that expectation and stays. |
| +- **No dependencies, no build step.** Tokenization via the built-in |
| + `Intl.Segmenter`; everything else is plain ES modules. (Running the |
| + indexer in a module Worker to keep the UI responsive is an implementation |
| + freedom, not a requirement.) |
| + |
| +### Shape |
| + |
| +- **Stores** (new object stores in the existing `lchat-store` DB): |
| + `idx-shards` (sealed term-shard blobs), `idx-meta` (per-thread indexed |
| + interval + schema version), `idx-key` (the CryptoKey). |
| +- **Tokenization**: `Intl.Segmenter` word segmentation, lowercased; message |
| + text plus sender name. **Query semantics become word-prefix AND** — each |
| + query term must prefix-match an indexed term. This matches native FTS5, |
| + fixing asymmetry #1, and is what every mainstream chat app does. (The |
| + substring-parity alternative — a trigram index — was rejected: ~3× the |
| + size for a semantics nobody expects.) |
| +- **Shards**: terms bucketed by normalized prefix, target ~128–512 KB per |
| + shard sealed; postings are `(threadId, messageId)` pairs, delta-encoded. |
| + Sizing: ~2M messages × ~12 tokens ≈ 24M postings ≈ well under 100 MB |
| + sealed — comfortable IndexedDB territory. |
| +- **Write path — piggyback on decryption that already happens**: live |
| + receive, thread render, `SearchIndex.ensureThread`, and `runBackfill`'s |
| + `onPage` hook (whose docstring already reserves it "for incremental |
| + indexing"). A buffered writer batches term→posting appends and |
| + read-modify-writes sealed shards. Per-thread coverage is tracked as an |
| + **interval** (`[minIndexed, maxIndexed]`) rather than the session index's |
| + seen-`Set` — backfill extends it downward, live traffic upward. (Same |
| + known gap-above caveat as the store's backfill watermark; same acceptance.) |
| +- **Query path**: segment the query → decrypt the few shards its term |
| + prefixes land in → intersect postings (AND) → decrypt candidate rows only, |
| + newest-first, limit 50 → verify with the real matcher → render through the |
| + existing `#msg-results` scaffold and jump. First search on a warm index: |
| + milliseconds of shard decrypt plus tens of row decrypts. |
| +- **Compaction**: lazy and occasional — rewrite shards dropping postings the |
| + verify step has seen dead (deleted/edited/left threads). Nothing correct |
| + depends on it; it only reclaims space, because hints are re-verified. |
| + |
| +### What sealing buys, and what it doesn't |
| + |
| +Sealing keeps derived plaintext (a term list is readable content!) out of |
| +casual reach: disk images, sync'd browser-profile backups, and forensic |
| +strolls through IndexedDB see only ciphertext — the same at-rest level as |
| +the message store beside it. It does **not** defend against an attacker who |
| +controls the same origin at runtime (XSS) or who lifts the whole profile and |
| +runs the browser — but neither does anything else in the client: the user's |
| +private key and wrapped thread keys live in the same origin storage. The |
| +sealed index adds **no new exposure class**; that's the bar it has to meet, |
| +and the reason a plaintext web index (native-style) was rejected — browser |
| +storage lacks the OS file-protection story App-Group SQLite gets. |
| + |
| +## Phases |
| + |
| +1. **P0 — semantics + measurement** (small, anytime): align web `matchText` |
| + on word-prefix AND (shared test vectors web ↔ native so the tokenizer/ |
| + match contract is pinned the way event vectors pin folds), and add the |
| + local first-search timing log that arms the trigger. |
| +2. **P1 — the sealed index** (the feature): stores, writer, query path, |
| + rebuild-on-version-mismatch. Scan remains the fallback for anything the |
| + index doesn't cover yet. |
| +3. **P2 — compaction + polish**: lazy shard compaction, leave-thread |
| + cleanup, coverage surfacing in the results footer. |
| +4. **P3 — richer queries** (optional, later): attachment-filename terms |
| + (filenames ride inside encrypted payloads, so they index client-side like |
| + any other text), `from:` / `in:` filters (the entry fields already |
| + exist), quoted phrases (positional postings — only if wanted). |
| + |
| +## Rejected |
| + |
| +- **Any server-side search**, including searchable-encryption schemes — |
| + they leak query and access patterns to exactly the party the product |
| + promises can see nothing. Off the table permanently. |
| +- **Trigram/substring index** — 3× size for unfamiliar semantics. |
| +- **Cross-device index sync** — each device derives its own from the store |
| + it already syncs; shipping index blobs around adds a surface for zero |
| + user-visible gain. |
| +- **Plaintext index on web** — see the sealing analysis above. |
amadan