| 1 | # 🤖 Tamper-evident hosting |
| 2 | |
| 3 | **Status: design.** This document settles the design questions left open in |
| 4 | issue #573 (spun out of #550's provider-marketplace discussion) and defines the |
| 5 | build order. The first slice — chained, signed checkpoints over an append-only |
| 6 | event journal — is implemented; everything else here is committed design, not |
| 7 | code. |
| 8 | |
| 9 | The one rule says: *if the server is compromised, the attacker gets nothing.* |
| 10 | Today that holds for **confidentiality** — message content, names, and files |
| 11 | are ciphertext under keys the server never sees. This work extends the rule to |
| 12 | **integrity**: a malicious host (a compromised us, or any third-party provider |
| 13 | we later delegate spaces to) must not be able to silently rewrite membership |
| 14 | or history. "Silently" is the load-bearing word — a host can always destroy or |
| 15 | withhold data; the goal is that *every* such act leaves evidence a client or |
| 16 | monitor can catch, and where possible **prove** to a third party. |
| 17 | |
| 18 | ## What a malicious host can do today |
| 19 | |
| 20 | 1. **Attack membership** (the key-distribution MITM). A joiner's new public |
| 21 | key travels through the server unauthenticated: the invite secret `T` |
| 22 | authenticates the *server → joiner* direction (the wrapped key blob only |
| 23 | opens under a `T`-derived key), but nothing in the *joiner → members* |
| 24 | direction binds the published public key to `T`. The host can substitute |
| 25 | its own key as "the new member," receive wrapped thread keys from honest |
| 26 | members, and re-wrap to the real joiner — a full read of every thread the |
| 27 | fake member is wrapped into. |
| 28 | 2. **Roll back.** Serve an old DB snapshot. Clients that were online notice |
| 29 | their own messages missing; clients that were offline can't tell. |
| 30 | 3. **Withhold or reorder** (bounded nuisance): drop or delay individual |
| 31 | events. Clients notice gaps against their own state but can't prove |
| 32 | misbehaviour. |
| 33 | |
| 34 | Three mechanisms close these, in dependency order: |
| 35 | |
| 36 | | Mechanism | Closes | Depends on | |
| 37 | |---|---|---| |
| 38 | | Signed checkpoint chain (this slice) | 2 becomes provable | nothing | |
| 39 | | Key transparency log | 1 | checkpoint chain, commons log | |
| 40 | | Client sampling verification | 3 detectable; escrow conformance | checkpoint chain, restore-read | |
| 41 | |
| 42 | --- |
| 43 | |
| 44 | ## Part 1 — the event journal and the checkpoint chain (built) |
| 45 | |
| 46 | ### Why a journal: the message table is not append-only, on purpose |
| 47 | |
| 48 | Edits are in-place `UPDATE`s and unsend is a hard `DELETE` — the ciphertext is |
| 49 | really gone, which is the product promise. A Merkle tree over the *current* |
| 50 | `messages` rows would therefore change under legitimate edits and unsends, and |
| 51 | "legitimate mutation" would be indistinguishable from "host rewrote history." |
| 52 | |
| 53 | So the unit of commitment is not table state but an **append-only event |
| 54 | journal**: every insert, edit, and delete appends one journal row in the same |
| 55 | transaction. The journal stores **hashes of ciphertext, never ciphertext** — |
| 56 | unsend still destroys the blob; what the journal retains is that *an* event |
| 57 | happened, not what it said. |
| 58 | |
| 59 | ``` |
| 60 | event_journal( |
| 61 | seq INTEGER PRIMARY KEY AUTOINCREMENT, -- total order, never reused |
| 62 | ts INTEGER, -- unix seconds, server clock |
| 63 | kind TEXT, -- 'msg' | 'edit' | 'delete' (roster kinds join later) |
| 64 | msg_id INTEGER, -- the messages.id this event concerns |
| 65 | thread_id TEXT, |
| 66 | blob_hash TEXT -- sha256 hex of the ciphertext ('' for delete) |
| 67 | ) |
| 68 | ``` |
| 69 | |
| 70 | **Privacy accounting.** Every journal column is metadata the server already |
| 71 | holds (it stores the blobs and serves them by thread and id). The one *new* |
| 72 | retention is temporal: after an unsend, the server previously forgot the |
| 73 | message ever existed; now a hash-shaped fossil (`msg_id`, `thread_id`, `ts`, |
| 74 | `blob_hash`) remains. That is inherent to tamper evidence — you cannot make |
| 75 | deletion verifiable while also forgetting it happened — and it reveals nothing |
| 76 | about content: a SHA-256 of AES-GCM ciphertext is doubly opaque. This is the |
| 77 | same bargain Certificate Transparency makes, and it is why the journal must |
| 78 | never grow a plaintext column. |
| 79 | |
| 80 | ### Checkpoints |
| 81 | |
| 82 | A background sweep (same shape as the mini-app scheduler: a `time.Tick` loop |
| 83 | delegating to a pure, test-callable unit of work) periodically commits to all |
| 84 | new journal rows: |
| 85 | |
| 86 | - **Leaf hash**: `SHA-256("lchat-ckpt-leaf\n" ‖ canonical encoding of the |
| 87 | journal row)`. Domain-separated so leaves can't be confused with interior |
| 88 | nodes or other protocol hashes. |
| 89 | - **Batch root**: an RFC 6962-style Merkle tree over the leaves |
| 90 | `[prev.seq_to+1 … seq_to]`. Batching means each checkpoint is O(new events), |
| 91 | not O(history), and inclusion proofs (a later slice) are proof-in-batch plus |
| 92 | the chain to the head. |
| 93 | - **Chain hash**: `SHA-256("lchat-ckpt-chain\n" ‖ prev_chain_hex ‖ root_hex ‖ |
| 94 | seq_to)`. Each checkpoint commits to its predecessor, so two checkpoints |
| 95 | presenting different histories for the same range are *cryptographic proof* |
| 96 | of equivocation, and serving a state older than a head a client has seen is |
| 97 | *provable* rollback — not just an anomaly. |
| 98 | - **Signature**: Ed25519 over the base64 JSON payload string, exactly the |
| 99 | `release.go` signed-manifest pattern (verify the signature over the encoded |
| 100 | string first, decode after — no canonicalization to get wrong). |
| 101 | |
| 102 | Checkpoints live in a `checkpoints` table and are served, member-token-gated, |
| 103 | at `GET /api/checkpoints` (cursorable, newest known to the caller onward). |
| 104 | Member-gated rather than public because the chain leaks activity volume and |
| 105 | timing; monitors hold member credentials anyway (see distribution, below). |
| 106 | |
| 107 | ### The signing key is not the security |
| 108 | |
| 109 | The per-instance Ed25519 seed lives **in the DB config by default**, minted on |
| 110 | first use. That placement is deliberate and operational: every path that moves |
| 111 | instance state — hibernation wake, `VACUUM INTO` migration, escrow restore — |
| 112 | carries *only* the DB, and a key that silently changes on routine ops would |
| 113 | make every legitimate restore read as forgery, poisoning the system with false |
| 114 | alarms. The trade is that anyone holding a DB copy (including the escrow) can |
| 115 | sign as the instance; we accept it, because no key location survives "the host |
| 116 | is malicious" anyway — the host runs the process and holds the seed. **The |
| 117 | security property is chain continuity plus external witnessing of heads** — a |
| 118 | host (or a DB-copy holder) can forge a fresh chain, but it cannot make one |
| 119 | that both (a) extends the head a client or the escrow already observed and |
| 120 | (b) omits what that head committed to. The signature's job is modest: it makes |
| 121 | a checkpoint attributable *given* a witnessed head, and stops anyone without |
| 122 | the seed from fabricating checkpoints in the instance's name. Two supporting |
| 123 | choices: every checkpoint payload **embeds its own public key**, so a key |
| 124 | change is an explicit, investigable event rather than old checkpoints suddenly |
| 125 | failing verification; and `LCHAT_CHECKPOINT_KEY_FILE` lets an operator keep |
| 126 | the seed in a file outside the DB instead, owning the carry-it-across-restores |
| 127 | burden themselves. |
| 128 | |
| 129 | ### What this slice deliberately does not do |
| 130 | |
| 131 | No inclusion proofs endpoint yet, no client-side verification yet, no roster |
| 132 | events in the journal yet (that's key transparency's job, below), and no |
| 133 | UI. It is the evidence *substrate*: from the moment it deploys, every instance |
| 134 | is accumulating a committed history that later verifiers can hold it to. |
| 135 | Deploying this early matters — evidence only covers events after the journal |
| 136 | starts. Messages that predate the journal are **baselined** at migration (one |
| 137 | `msg` row each, committing to the ciphertext as it stood at upgrade): the |
| 138 | baseline can't retroactively prove the past, but without it, editing or |
| 139 | unsending an old message would journal an event for a message the journal |
| 140 | never saw — a built-in false alarm. |
| 141 | |
| 142 | --- |
| 143 | |
| 144 | ## Part 2 — the three open design questions, settled |
| 145 | |
| 146 | ### Q1: Log granularity — one commons log with sealed space identifiers |
| 147 | |
| 148 | **Decision: one global, commons-operated transparency log; spaces appear |
| 149 | under sealed (HKDF-derived) identifiers. Per-space logs are rejected.** |
| 150 | |
| 151 | A per-space log kept by the space's own host adds nothing against that host — |
| 152 | the adversary operates the log. The trust win comes precisely from the log |
| 153 | living in a *different* trust domain than the space's host, which is the whole |
| 154 | point once spaces can be hosted by third-party providers (#575): a provider |
| 155 | can't equivocate about a roster without the commons log catching it, and a |
| 156 | compromised commons log can't read anything — it holds only commitments. |
| 157 | |
| 158 | One global log also amortizes what actually costs effort: monitoring. CONIKS |
| 159 | died and WhatsApp KT shipped largely on this axis — auditing one log is a |
| 160 | commons-scale job; auditing ten thousand per-space logs is nobody's job. |
| 161 | |
| 162 | **Sealed identifiers.** A space appears in the log as |
| 163 | `space_tag = HKDF(instance_secret, "lchat-kt-space-tag")` — stable within a |
| 164 | space (members and monitors derive it; they hold the instance secret, which |
| 165 | already exists as key material wrapped to every member) but meaningless to the |
| 166 | log operator and unlinkable across spaces. Roster entries under a tag are the |
| 167 | sealed shape of "key K joined / rotated / left," carrying no names — screen |
| 168 | names are E2EE and stay that way. The log learns: some space, some volume of |
| 169 | membership churn. It must never learn which space, or who. |
| 170 | |
| 171 | **What the instance's own chain (Part 1) is, then:** the per-space *history* |
| 172 | commitment stays local-with-witnessing (it commits every message event — |
| 173 | far too voluminous and too revealing for a global log). The global log carries |
| 174 | only (a) roster/key events under sealed tags and (b) periodic instance chain |
| 175 | heads, which is what makes local chains fork-evident. Two layers, one trust |
| 176 | story: the instance chain says *what this host claims happened*; the commons |
| 177 | log pins *which claim the host is stuck with*. |
| 178 | |
| 179 | ### Q2: The invite `#fragment` bootstraps the log entry — extends, never replaced |
| 180 | |
| 181 | **Decision: `T` stays the root of trust for admission; the log makes what `T` |
| 182 | authorized auditable. Concretely, a join produces a log entry that carries a |
| 183 | `T`-derived binding of the joiner's public key, verified by the inviter's |
| 184 | client.** |
| 185 | |
| 186 | The mechanism mirrors the existing `deriveInvite` derivations (id, wrap key, |
| 187 | claim secret — all HKDF domain-separated from `T`). Add one more: |
| 188 | |
| 189 | - `bind_key = HKDF(T, "lchat-invite-bind")`, held by whoever minted the link |
| 190 | (their client already had `T`) and derivable by the joiner. |
| 191 | - At claim time the joiner publishes, alongside its new public key, |
| 192 | `bind_mac = HMAC(bind_key, joiner_pubkey)`. |
| 193 | - The **inviter's client** — the one party besides the joiner who ever held |
| 194 | `T` — verifies `bind_mac` against the *published roster key* before it (or |
| 195 | anyone it vouches to) wraps a single thread key to the new member, and then |
| 196 | signs the join's log entry as attester. |
| 197 | |
| 198 | This closes the substitution attack at its exact seam: a host that swaps the |
| 199 | published key can't produce a valid `bind_mac` for its own key (it never held |
| 200 | `T` — it stores only `claim_hash`), so the inviter's client refuses to wrap |
| 201 | and the join fails loudly instead of succeeding silently. |
| 202 | |
| 203 | Subsequent lifecycle events are self-certifying without `T`: |
| 204 | |
| 205 | - **Key rotation / add-device**: signed by the key being replaced (continuity |
| 206 | chain rooted in the `T`-bound original). |
| 207 | - **Removal**: signed by an admin identity, itself a `T`-rooted chain (the |
| 208 | setup invite is the genesis entry). |
| 209 | - **Wrap discipline**: clients wrap thread keys only to keys whose lineage |
| 210 | appears in the log — "verify before wrap" is the client-side rule that makes |
| 211 | the log load-bearing rather than advisory. |
| 212 | |
| 213 | Bots and the eleven CLI follow the same path — they are ordinary members whose |
| 214 | `T` arrives via the agent-invite flow; nothing special-cases them. |
| 215 | |
| 216 | **Why not replace the fragment with the log?** Because the fragment is the |
| 217 | only *out-of-band* channel in the system — the one path the host never sees. |
| 218 | A log can prove consistency (everyone sees the same roster) but cannot prove |
| 219 | *legitimacy* (that a roster entry was authorized by a human who held the |
| 220 | link). Consistency without an anchor just means the host lies to everyone |
| 221 | identically. `T` supplies legitimacy; the log makes it globally consistent |
| 222 | and equivocation provable. They compose; neither substitutes for the other. |
| 223 | |
| 224 | ### Q3: Checkpoint distribution — escrow first, member gossip second, commons log third |
| 225 | |
| 226 | **Decision: three witnessing channels, shipped in this order, each valuable |
| 227 | alone.** |
| 228 | |
| 229 | 1. **Escrow replica (near-free, ships with Part 1).** The `checkpoints` table |
| 230 | rides the existing Litestream replication to S3, and S3 versioning |
| 231 | timestamps every generation of it — a target the *provider* doesn't control |
| 232 | once escrow is commons-held (#550 Part 10). Honest history: monotone chain |
| 233 | growth across replica generations. Rollback: a later generation whose head |
| 234 | is an ancestor of an earlier generation's. The host could stop replicating, |
| 235 | but going dark is itself the detectable event the escrow lease already |
| 236 | surfaces. Limit: a fully self-hosted instance with no escrow gets nothing |
| 237 | from this channel — hence channels 2 and 3. |
| 238 | 2. **Member gossip inside E2EE traffic (the novel one, and ours to use: |
| 239 | members already share an authenticated, host-opaque channel — the chat |
| 240 | itself).** Clients periodically tuck the head they last verified |
| 241 | (`seq_to`, `chain_hash`) inside their normal encrypted payloads; peers |
| 242 | compare against their own view. A host that forks — serving member A one |
| 243 | history and member B another — is betrayed by its own transport: it can't |
| 244 | read the gossip to filter it, and dropping messages wholesale is exactly |
| 245 | the withholding that members notice. Split-view detection with zero new |
| 246 | infrastructure, working even for self-hosted spaces with no escrow and no |
| 247 | commons. This is the distribution answer the issue was looking for. |
| 248 | 3. **Commons log witnessing (with Part 3/KT).** Instances push heads under |
| 249 | their sealed tag on a slow cadence; the commons log's own consistency |
| 250 | proofs then pin every instance's chain globally. Strongest guarantee, |
| 251 | biggest dependency — which is why it's last, not first. |
| 252 | |
| 253 | Ordering rationale: 1 is a table already replicating; 2 is a small client |
| 254 | change with outsized reach; 3 requires operating a new commons service and |
| 255 | should be built once, for KT, not twice. |
| 256 | |
| 257 | --- |
| 258 | |
| 259 | ## Part 3 — key transparency (design sketch, next after this slice) |
| 260 | |
| 261 | With Q1–Q3 settled, KT's shape is: roster/key events become journal kinds |
| 262 | (`join`, `rotate`, `remove`) carrying the Q2 evidence (bind MACs, continuity |
| 263 | signatures); the instance chain commits to them like any event; instances |
| 264 | publish roster events + chain heads to the commons log under their sealed tag; |
| 265 | clients enforce verify-before-wrap; monitors (any client, or the commons) |
| 266 | audit for equivocation. The commons log itself is CT-shaped: append-only, |
| 267 | Merkle, publicly auditable, dumb — it stores sealed commitments and serves |
| 268 | proofs, nothing else. Detailed protocol (proof formats, monitor cadence, |
| 269 | anti-flap for offline devices) is its own design pass **after** the journal |
| 270 | substrate has soaked in production. It is deliberately not specified further |
| 271 | here: the failure mode of KT designs is specifying the whole cathedral before |
| 272 | the foundation has carried load. |
| 273 | |
| 274 | **Part 3a — roster/key journal events (built).** The first, safe increment: |
| 275 | membership changes are journaled like message events, so they're committed to |
| 276 | the signed checkpoint chain and a monitor catches a server rewriting the |
| 277 | roster. A member's first key publish appends a `join` event; removal appends a |
| 278 | `remove`. They reuse the journal's generic columns as the kind's detail record |
| 279 | (`thread_id` = the member's user id, `blob_hash` = sha256 of the public key for |
| 280 | `join` / empty for `remove`, `msg_id` unused) — the leaf encoding already |
| 281 | covers these fields, so no wire-format change. Re-registering the same key |
| 282 | (add-device) is not a new join. This touches no crypto and no join-path |
| 283 | decision; it just makes the roster tamper-evident on the instance chain. |
| 284 | |
| 285 | **Part 3b — the bind MAC + verify-before-wrap (next, review-gated).** The |
| 286 | security-critical increment that closes the key-substitution attack (Q2): |
| 287 | `bind_key = HKDF(T, "lchat-invite-bind")`; the joiner publishes |
| 288 | `bind_mac = HMAC(bind_key, joiner_pubkey)` at claim; the inviter's client |
| 289 | verifies it against the published roster key before wrapping any thread key, |
| 290 | and clients enforce verify-before-wrap generally. Because it changes the |
| 291 | join/key-distribution path — the crown jewel of the one rule — it ships as its |
| 292 | own PR with an adversarial review pass and a canary before it reaches any real |
| 293 | space. |
| 294 | |
| 295 | ## Part 4 — client sampling verification (design sketch) |
| 296 | |
| 297 | Because clients hold plaintext truth, whole-DB verification needs no |
| 298 | proofs-of-retrievability: each client periodically samples k message ids *from |
| 299 | its own local store*, fetches those rows through a **restore-read endpoint |
| 300 | over the escrow replica** (not the live host — the point is to check what |
| 301 | escrow would restore), checks each blob decrypts and matches, and — once |
| 302 | inclusion proofs land — proves each against a committed root. Independent |
| 303 | sampling across members covers the DB probabilistically at trivial per-client |
| 304 | cost, and the pass/fail signal doubles as the client-visible end of provider |
| 305 | conformance, which #575's payment gating needs. Depends on: checkpoint chain |
| 306 | (shipped), inclusion-proof endpoint, restore-read endpoint. Build after KT's |
| 307 | journal extension, before the marketplace. |
| 308 | |
| 309 | ## Out of scope (unchanged from #573) |
| 310 | |
| 311 | Provider-served **web assets** remain the weakest trust point: a malicious |
| 312 | host can serve JavaScript that exfiltrates keys, and no log catches that. |
| 313 | Signed releases + native apps carry the answer for now; a Code Verify-style |
| 314 | extension is the eventual one. Separate work, tracked in #550 Part 10. |
| 315 | |
| 316 | ## Build order |
| 317 | |
| 318 | 1. **Event journal + signed checkpoint chain** — this slice. Server-only, |
| 319 | additive migration, zero client changes. Starts the evidence clock. |
| 320 | 2. **Member gossip of chain heads** (Q3 channel 2) — small client change, |
| 321 | makes forks member-detectable everywhere, no new services. |
| 322 | 3. **Inclusion proofs endpoint** + verify-on-load in clients. |
| 323 | 4. **KT journal kinds + bind MAC at claim + verify-before-wrap** (Q2) — the |
| 324 | membership fix, still commons-free. |
| 325 | 5. **Commons log + sealed tags + monitors** (Q1, Q3 channel 3) — the global |
| 326 | equivocation pin. Gates #575. |
| 327 | 6. **Restore-read + client sampling** (Part 4). Gates #575's payment |
| 328 | conformance. |
| 329 | |
amadan