amadan
docs: tamper-evident hosting — design for #573 (#751)
* docs: tamper-evident hosting design — settle #573's open questions Settles log granularity (one commons log, sealed space tags), the invite #fragment interaction (bind MAC extends the anchor; verify-before-wrap), and checkpoint distribution (escrow → member gossip → commons log), and fixes the build order. Companion to the checkpoint-chain slice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: key custody + baseline updates from the review pass Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> ---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+307
−0
docs/tamper-evidence.md+307 −0
diff --git a/docs/tamper-evidence.md b/docs/tamper-evidence.md| new file mode 100644 |
| index 0000000..578f894 |
| --- /dev/null |
| +++ b/docs/tamper-evidence.md |
| @@ -0,0 +1,307 @@ |
| +# 🤖 Tamper-evident hosting |
| + |
| +**Status: design.** This document settles the design questions left open in |
| +issue #573 (spun out of #550's provider-marketplace discussion) and defines the |
| +build order. The first slice — chained, signed checkpoints over an append-only |
| +event journal — is implemented; everything else here is committed design, not |
| +code. |
| + |
| +The one rule says: *if the server is compromised, the attacker gets nothing.* |
| +Today that holds for **confidentiality** — message content, names, and files |
| +are ciphertext under keys the server never sees. This work extends the rule to |
| +**integrity**: a malicious host (a compromised us, or any third-party provider |
| +we later delegate spaces to) must not be able to silently rewrite membership |
| +or history. "Silently" is the load-bearing word — a host can always destroy or |
| +withhold data; the goal is that *every* such act leaves evidence a client or |
| +monitor can catch, and where possible **prove** to a third party. |
| + |
| +## What a malicious host can do today |
| + |
| +1. **Attack membership** (the key-distribution MITM). A joiner's new public |
| + key travels through the server unauthenticated: the invite secret `T` |
| + authenticates the *server → joiner* direction (the wrapped key blob only |
| + opens under a `T`-derived key), but nothing in the *joiner → members* |
| + direction binds the published public key to `T`. The host can substitute |
| + its own key as "the new member," receive wrapped thread keys from honest |
| + members, and re-wrap to the real joiner — a full read of every thread the |
| + fake member is wrapped into. |
| +2. **Roll back.** Serve an old DB snapshot. Clients that were online notice |
| + their own messages missing; clients that were offline can't tell. |
| +3. **Withhold or reorder** (bounded nuisance): drop or delay individual |
| + events. Clients notice gaps against their own state but can't prove |
| + misbehaviour. |
| + |
| +Three mechanisms close these, in dependency order: |
| + |
| +| Mechanism | Closes | Depends on | |
| +|---|---|---| |
| +| Signed checkpoint chain (this slice) | 2 becomes provable | nothing | |
| +| Key transparency log | 1 | checkpoint chain, commons log | |
| +| Client sampling verification | 3 detectable; escrow conformance | checkpoint chain, restore-read | |
| + |
| +--- |
| + |
| +## Part 1 — the event journal and the checkpoint chain (built) |
| + |
| +### Why a journal: the message table is not append-only, on purpose |
| + |
| +Edits are in-place `UPDATE`s and unsend is a hard `DELETE` — the ciphertext is |
| +really gone, which is the product promise. A Merkle tree over the *current* |
| +`messages` rows would therefore change under legitimate edits and unsends, and |
| +"legitimate mutation" would be indistinguishable from "host rewrote history." |
| + |
| +So the unit of commitment is not table state but an **append-only event |
| +journal**: every insert, edit, and delete appends one journal row in the same |
| +transaction. The journal stores **hashes of ciphertext, never ciphertext** — |
| +unsend still destroys the blob; what the journal retains is that *an* event |
| +happened, not what it said. |
| + |
| +``` |
| +event_journal( |
| + seq INTEGER PRIMARY KEY AUTOINCREMENT, -- total order, never reused |
| + ts INTEGER, -- unix seconds, server clock |
| + kind TEXT, -- 'msg' | 'edit' | 'delete' (roster kinds join later) |
| + msg_id INTEGER, -- the messages.id this event concerns |
| + thread_id TEXT, |
| + blob_hash TEXT -- sha256 hex of the ciphertext ('' for delete) |
| +) |
| +``` |
| + |
| +**Privacy accounting.** Every journal column is metadata the server already |
| +holds (it stores the blobs and serves them by thread and id). The one *new* |
| +retention is temporal: after an unsend, the server previously forgot the |
| +message ever existed; now a hash-shaped fossil (`msg_id`, `thread_id`, `ts`, |
| +`blob_hash`) remains. That is inherent to tamper evidence — you cannot make |
| +deletion verifiable while also forgetting it happened — and it reveals nothing |
| +about content: a SHA-256 of AES-GCM ciphertext is doubly opaque. This is the |
| +same bargain Certificate Transparency makes, and it is why the journal must |
| +never grow a plaintext column. |
| + |
| +### Checkpoints |
| + |
| +A background sweep (same shape as the mini-app scheduler: a `time.Tick` loop |
| +delegating to a pure, test-callable unit of work) periodically commits to all |
| +new journal rows: |
| + |
| +- **Leaf hash**: `SHA-256("lchat-ckpt-leaf\n" ‖ canonical encoding of the |
| + journal row)`. Domain-separated so leaves can't be confused with interior |
| + nodes or other protocol hashes. |
| +- **Batch root**: an RFC 6962-style Merkle tree over the leaves |
| + `[prev.seq_to+1 … seq_to]`. Batching means each checkpoint is O(new events), |
| + not O(history), and inclusion proofs (a later slice) are proof-in-batch plus |
| + the chain to the head. |
| +- **Chain hash**: `SHA-256("lchat-ckpt-chain\n" ‖ prev_chain_hex ‖ root_hex ‖ |
| + seq_to)`. Each checkpoint commits to its predecessor, so two checkpoints |
| + presenting different histories for the same range are *cryptographic proof* |
| + of equivocation, and serving a state older than a head a client has seen is |
| + *provable* rollback — not just an anomaly. |
| +- **Signature**: Ed25519 over the base64 JSON payload string, exactly the |
| + `release.go` signed-manifest pattern (verify the signature over the encoded |
| + string first, decode after — no canonicalization to get wrong). |
| + |
| +Checkpoints live in a `checkpoints` table and are served, member-token-gated, |
| +at `GET /api/checkpoints` (cursorable, newest known to the caller onward). |
| +Member-gated rather than public because the chain leaks activity volume and |
| +timing; monitors hold member credentials anyway (see distribution, below). |
| + |
| +### The signing key is not the security |
| + |
| +The per-instance Ed25519 seed lives **in the DB config by default**, minted on |
| +first use. That placement is deliberate and operational: every path that moves |
| +instance state — hibernation wake, `VACUUM INTO` migration, escrow restore — |
| +carries *only* the DB, and a key that silently changes on routine ops would |
| +make every legitimate restore read as forgery, poisoning the system with false |
| +alarms. The trade is that anyone holding a DB copy (including the escrow) can |
| +sign as the instance; we accept it, because no key location survives "the host |
| +is malicious" anyway — the host runs the process and holds the seed. **The |
| +security property is chain continuity plus external witnessing of heads** — a |
| +host (or a DB-copy holder) can forge a fresh chain, but it cannot make one |
| +that both (a) extends the head a client or the escrow already observed and |
| +(b) omits what that head committed to. The signature's job is modest: it makes |
| +a checkpoint attributable *given* a witnessed head, and stops anyone without |
| +the seed from fabricating checkpoints in the instance's name. Two supporting |
| +choices: every checkpoint payload **embeds its own public key**, so a key |
| +change is an explicit, investigable event rather than old checkpoints suddenly |
| +failing verification; and `LCHAT_CHECKPOINT_KEY_FILE` lets an operator keep |
| +the seed in a file outside the DB instead, owning the carry-it-across-restores |
| +burden themselves. |
| + |
| +### What this slice deliberately does not do |
| + |
| +No inclusion proofs endpoint yet, no client-side verification yet, no roster |
| +events in the journal yet (that's key transparency's job, below), and no |
| +UI. It is the evidence *substrate*: from the moment it deploys, every instance |
| +is accumulating a committed history that later verifiers can hold it to. |
| +Deploying this early matters — evidence only covers events after the journal |
| +starts. Messages that predate the journal are **baselined** at migration (one |
| +`msg` row each, committing to the ciphertext as it stood at upgrade): the |
| +baseline can't retroactively prove the past, but without it, editing or |
| +unsending an old message would journal an event for a message the journal |
| +never saw — a built-in false alarm. |
| + |
| +--- |
| + |
| +## Part 2 — the three open design questions, settled |
| + |
| +### Q1: Log granularity — one commons log with sealed space identifiers |
| + |
| +**Decision: one global, commons-operated transparency log; spaces appear |
| +under sealed (HKDF-derived) identifiers. Per-space logs are rejected.** |
| + |
| +A per-space log kept by the space's own host adds nothing against that host — |
| +the adversary operates the log. The trust win comes precisely from the log |
| +living in a *different* trust domain than the space's host, which is the whole |
| +point once spaces can be hosted by third-party providers (#575): a provider |
| +can't equivocate about a roster without the commons log catching it, and a |
| +compromised commons log can't read anything — it holds only commitments. |
| + |
| +One global log also amortizes what actually costs effort: monitoring. CONIKS |
| +died and WhatsApp KT shipped largely on this axis — auditing one log is a |
| +commons-scale job; auditing ten thousand per-space logs is nobody's job. |
| + |
| +**Sealed identifiers.** A space appears in the log as |
| +`space_tag = HKDF(instance_secret, "lchat-kt-space-tag")` — stable within a |
| +space (members and monitors derive it; they hold the instance secret, which |
| +already exists as key material wrapped to every member) but meaningless to the |
| +log operator and unlinkable across spaces. Roster entries under a tag are the |
| +sealed shape of "key K joined / rotated / left," carrying no names — screen |
| +names are E2EE and stay that way. The log learns: some space, some volume of |
| +membership churn. It must never learn which space, or who. |
| + |
| +**What the instance's own chain (Part 1) is, then:** the per-space *history* |
| +commitment stays local-with-witnessing (it commits every message event — |
| +far too voluminous and too revealing for a global log). The global log carries |
| +only (a) roster/key events under sealed tags and (b) periodic instance chain |
| +heads, which is what makes local chains fork-evident. Two layers, one trust |
| +story: the instance chain says *what this host claims happened*; the commons |
| +log pins *which claim the host is stuck with*. |
| + |
| +### Q2: The invite `#fragment` bootstraps the log entry — extends, never replaced |
| + |
| +**Decision: `T` stays the root of trust for admission; the log makes what `T` |
| +authorized auditable. Concretely, a join produces a log entry that carries a |
| +`T`-derived binding of the joiner's public key, verified by the inviter's |
| +client.** |
| + |
| +The mechanism mirrors the existing `deriveInvite` derivations (id, wrap key, |
| +claim secret — all HKDF domain-separated from `T`). Add one more: |
| + |
| +- `bind_key = HKDF(T, "lchat-invite-bind")`, held by whoever minted the link |
| + (their client already had `T`) and derivable by the joiner. |
| +- At claim time the joiner publishes, alongside its new public key, |
| + `bind_mac = HMAC(bind_key, joiner_pubkey)`. |
| +- The **inviter's client** — the one party besides the joiner who ever held |
| + `T` — verifies `bind_mac` against the *published roster key* before it (or |
| + anyone it vouches to) wraps a single thread key to the new member, and then |
| + signs the join's log entry as attester. |
| + |
| +This closes the substitution attack at its exact seam: a host that swaps the |
| +published key can't produce a valid `bind_mac` for its own key (it never held |
| +`T` — it stores only `claim_hash`), so the inviter's client refuses to wrap |
| +and the join fails loudly instead of succeeding silently. |
| + |
| +Subsequent lifecycle events are self-certifying without `T`: |
| + |
| +- **Key rotation / add-device**: signed by the key being replaced (continuity |
| + chain rooted in the `T`-bound original). |
| +- **Removal**: signed by an admin identity, itself a `T`-rooted chain (the |
| + setup invite is the genesis entry). |
| +- **Wrap discipline**: clients wrap thread keys only to keys whose lineage |
| + appears in the log — "verify before wrap" is the client-side rule that makes |
| + the log load-bearing rather than advisory. |
| + |
| +Bots and the eleven CLI follow the same path — they are ordinary members whose |
| +`T` arrives via the agent-invite flow; nothing special-cases them. |
| + |
| +**Why not replace the fragment with the log?** Because the fragment is the |
| +only *out-of-band* channel in the system — the one path the host never sees. |
| +A log can prove consistency (everyone sees the same roster) but cannot prove |
| +*legitimacy* (that a roster entry was authorized by a human who held the |
| +link). Consistency without an anchor just means the host lies to everyone |
| +identically. `T` supplies legitimacy; the log makes it globally consistent |
| +and equivocation provable. They compose; neither substitutes for the other. |
| + |
| +### Q3: Checkpoint distribution — escrow first, member gossip second, commons log third |
| + |
| +**Decision: three witnessing channels, shipped in this order, each valuable |
| +alone.** |
| + |
| +1. **Escrow replica (near-free, ships with Part 1).** The `checkpoints` table |
| + rides the existing Litestream replication to S3, and S3 versioning |
| + timestamps every generation of it — a target the *provider* doesn't control |
| + once escrow is commons-held (#550 Part 10). Honest history: monotone chain |
| + growth across replica generations. Rollback: a later generation whose head |
| + is an ancestor of an earlier generation's. The host could stop replicating, |
| + but going dark is itself the detectable event the escrow lease already |
| + surfaces. Limit: a fully self-hosted instance with no escrow gets nothing |
| + from this channel — hence channels 2 and 3. |
| +2. **Member gossip inside E2EE traffic (the novel one, and ours to use: |
| + members already share an authenticated, host-opaque channel — the chat |
| + itself).** Clients periodically tuck the head they last verified |
| + (`seq_to`, `chain_hash`) inside their normal encrypted payloads; peers |
| + compare against their own view. A host that forks — serving member A one |
| + history and member B another — is betrayed by its own transport: it can't |
| + read the gossip to filter it, and dropping messages wholesale is exactly |
| + the withholding that members notice. Split-view detection with zero new |
| + infrastructure, working even for self-hosted spaces with no escrow and no |
| + commons. This is the distribution answer the issue was looking for. |
| +3. **Commons log witnessing (with Part 3/KT).** Instances push heads under |
| + their sealed tag on a slow cadence; the commons log's own consistency |
| + proofs then pin every instance's chain globally. Strongest guarantee, |
| + biggest dependency — which is why it's last, not first. |
| + |
| +Ordering rationale: 1 is a table already replicating; 2 is a small client |
| +change with outsized reach; 3 requires operating a new commons service and |
| +should be built once, for KT, not twice. |
| + |
| +--- |
| + |
| +## Part 3 — key transparency (design sketch, next after this slice) |
| + |
| +With Q1–Q3 settled, KT's shape is: roster/key events become journal kinds |
| +(`join`, `rotate`, `remove`) carrying the Q2 evidence (bind MACs, continuity |
| +signatures); the instance chain commits to them like any event; instances |
| +publish roster events + chain heads to the commons log under their sealed tag; |
| +clients enforce verify-before-wrap; monitors (any client, or the commons) |
| +audit for equivocation. The commons log itself is CT-shaped: append-only, |
| +Merkle, publicly auditable, dumb — it stores sealed commitments and serves |
| +proofs, nothing else. Detailed protocol (proof formats, monitor cadence, |
| +anti-flap for offline devices) is its own design pass **after** the journal |
| +substrate has soaked in production. It is deliberately not specified further |
| +here: the failure mode of KT designs is specifying the whole cathedral before |
| +the foundation has carried load. |
| + |
| +## Part 4 — client sampling verification (design sketch) |
| + |
| +Because clients hold plaintext truth, whole-DB verification needs no |
| +proofs-of-retrievability: each client periodically samples k message ids *from |
| +its own local store*, fetches those rows through a **restore-read endpoint |
| +over the escrow replica** (not the live host — the point is to check what |
| +escrow would restore), checks each blob decrypts and matches, and — once |
| +inclusion proofs land — proves each against a committed root. Independent |
| +sampling across members covers the DB probabilistically at trivial per-client |
| +cost, and the pass/fail signal doubles as the client-visible end of provider |
| +conformance, which #575's payment gating needs. Depends on: checkpoint chain |
| +(shipped), inclusion-proof endpoint, restore-read endpoint. Build after KT's |
| +journal extension, before the marketplace. |
| + |
| +## Out of scope (unchanged from #573) |
| + |
| +Provider-served **web assets** remain the weakest trust point: a malicious |
| +host can serve JavaScript that exfiltrates keys, and no log catches that. |
| +Signed releases + native apps carry the answer for now; a Code Verify-style |
| +extension is the eventual one. Separate work, tracked in #550 Part 10. |
| + |
| +## Build order |
| + |
| +1. **Event journal + signed checkpoint chain** — this slice. Server-only, |
| + additive migration, zero client changes. Starts the evidence clock. |
| +2. **Member gossip of chain heads** (Q3 channel 2) — small client change, |
| + makes forks member-detectable everywhere, no new services. |
| +3. **Inclusion proofs endpoint** + verify-on-load in clients. |
| +4. **KT journal kinds + bind MAC at claim + verify-before-wrap** (Q2) — the |
| + membership fix, still commons-free. |
| +5. **Commons log + sealed tags + monitors** (Q1, Q3 channel 3) — the global |
| + equivocation pin. Gates #575. |
| +6. **Restore-read + client sampling** (Part 4). Gates #575's payment |
| + conformance. |