🤖 Tamper-evident hosting

Status: design. This document settles the design questions left open in issue #573 (spun out of #550's provider-marketplace discussion) and defines the build order. The first slice — chained, signed checkpoints over an append-only event journal — is implemented; everything else here is committed design, not code.

The one rule says: if the server is compromised, the attacker gets nothing. Today that holds for confidentiality — message content, names, and files are ciphertext under keys the server never sees. This work extends the rule to integrity: a malicious host (a compromised us, or any third-party provider we later delegate spaces to) must not be able to silently rewrite membership or history. "Silently" is the load-bearing word — a host can always destroy or withhold data; the goal is that every such act leaves evidence a client or monitor can catch, and where possible prove to a third party.

What a malicious host can do today

1. Attack membership (the key-distribution MITM). A joiner's new public key travels through the server unauthenticated: the invite secret T authenticates the server → joiner direction (the wrapped key blob only opens under a T-derived key), but nothing in the joiner → members direction binds the published public key to T. The host can substitute its own key as "the new member," receive wrapped thread keys from honest members, and re-wrap to the real joiner — a full read of every thread the fake member is wrapped into.
2. Roll back. Serve an old DB snapshot. Clients that were online notice their own messages missing; clients that were offline can't tell.
3. Withhold or reorder (bounded nuisance): drop or delay individual events. Clients notice gaps against their own state but can't prove misbehaviour.

Three mechanisms close these, in dependency order:

| Mechanism | Closes | Depends on | |---|---|---| | Signed checkpoint chain (this slice) | 2 becomes provable | nothing | | Key transparency log | 1 | checkpoint chain, commons log | | Client sampling verification | 3 detectable; escrow conformance | checkpoint chain, restore-read |

---

Part 1 — the event journal and the checkpoint chain (built)

Why a journal: the message table is not append-only, on purpose

Edits are in-place UPDATEs and unsend is a hard DELETE — the ciphertext is really gone, which is the product promise. A Merkle tree over the current messages rows would therefore change under legitimate edits and unsends, and "legitimate mutation" would be indistinguishable from "host rewrote history."

So the unit of commitment is not table state but an append-only event journal: every insert, edit, and delete appends one journal row in the same transaction. The journal stores hashes of ciphertext, never ciphertext — unsend still destroys the blob; what the journal retains is that an event happened, not what it said.

event_journal(
  seq        INTEGER PRIMARY KEY AUTOINCREMENT,  -- total order, never reused
  ts         INTEGER,                            -- unix seconds, server clock
  kind       TEXT,     -- 'msg' | 'edit' | 'delete' (roster kinds join later)
  msg_id     INTEGER,  -- the messages.id this event concerns
  thread_id  TEXT,
  blob_hash  TEXT      -- sha256 hex of the ciphertext ('' for delete)
)

Privacy accounting. Every journal column is metadata the server already holds (it stores the blobs and serves them by thread and id). The one new retention is temporal: after an unsend, the server previously forgot the message ever existed; now a hash-shaped fossil (msg_id, thread_id, ts, blob_hash) remains. That is inherent to tamper evidence — you cannot make deletion verifiable while also forgetting it happened — and it reveals nothing about content: a SHA-256 of AES-GCM ciphertext is doubly opaque. This is the same bargain Certificate Transparency makes, and it is why the journal must never grow a plaintext column.

Checkpoints

A background sweep (same shape as the mini-app scheduler: a time.Tick loop delegating to a pure, test-callable unit of work) periodically commits to all new journal rows:

• Leaf hash: SHA-256("lchat-ckpt-leaf\n" ‖ canonical encoding of the journal row). Domain-separated so leaves can't be confused with interior nodes or other protocol hashes.
• Batch root: an RFC 6962-style Merkle tree over the leaves [prev.seq_to+1 … seq_to]. Batching means each checkpoint is O(new events), not O(history), and inclusion proofs (a later slice) are proof-in-batch plus the chain to the head.
• Chain hash: SHA-256("lchat-ckpt-chain\n" ‖ prev_chain_hex ‖ root_hex ‖ seq_to). Each checkpoint commits to its predecessor, so two checkpoints presenting different histories for the same range are cryptographic proof of equivocation, and serving a state older than a head a client has seen is provable rollback — not just an anomaly.
• Signature: Ed25519 over the base64 JSON payload string, exactly the release.go signed-manifest pattern (verify the signature over the encoded string first, decode after — no canonicalization to get wrong).

Checkpoints live in a checkpoints table and are served, member-token-gated, at GET /api/checkpoints (cursorable, newest known to the caller onward). Member-gated rather than public because the chain leaks activity volume and timing; monitors hold member credentials anyway (see distribution, below).

The signing key is not the security

The per-instance Ed25519 seed lives in the DB config by default, minted on first use. That placement is deliberate and operational: every path that moves instance state — hibernation wake, VACUUM INTO migration, escrow restore — carries only the DB, and a key that silently changes on routine ops would make every legitimate restore read as forgery, poisoning the system with false alarms. The trade is that anyone holding a DB copy (including the escrow) can sign as the instance; we accept it, because no key location survives "the host is malicious" anyway — the host runs the process and holds the seed. The security property is chain continuity plus external witnessing of heads — a host (or a DB-copy holder) can forge a fresh chain, but it cannot make one that both (a) extends the head a client or the escrow already observed and (b) omits what that head committed to. The signature's job is modest: it makes a checkpoint attributable given a witnessed head, and stops anyone without the seed from fabricating checkpoints in the instance's name. Two supporting choices: every checkpoint payload embeds its own public key, so a key change is an explicit, investigable event rather than old checkpoints suddenly failing verification; and LCHAT_CHECKPOINT_KEY_FILE lets an operator keep the seed in a file outside the DB instead, owning the carry-it-across-restores burden themselves.

What this slice deliberately does not do

No inclusion proofs endpoint yet, no client-side verification yet, no roster events in the journal yet (that's key transparency's job, below), and no UI. It is the evidence substrate: from the moment it deploys, every instance is accumulating a committed history that later verifiers can hold it to. Deploying this early matters — evidence only covers events after the journal starts. Messages that predate the journal are baselined at migration (one msg row each, committing to the ciphertext as it stood at upgrade): the baseline can't retroactively prove the past, but without it, editing or unsending an old message would journal an event for a message the journal never saw — a built-in false alarm.

---

Part 2 — the three open design questions, settled

Q1: Log granularity — one commons log with sealed space identifiers

Decision: one global, commons-operated transparency log; spaces appear under sealed (HKDF-derived) identifiers. Per-space logs are rejected.

A per-space log kept by the space's own host adds nothing against that host — the adversary operates the log. The trust win comes precisely from the log living in a different trust domain than the space's host, which is the whole point once spaces can be hosted by third-party providers (#575): a provider can't equivocate about a roster without the commons log catching it, and a compromised commons log can't read anything — it holds only commitments.

One global log also amortizes what actually costs effort: monitoring. CONIKS died and WhatsApp KT shipped largely on this axis — auditing one log is a commons-scale job; auditing ten thousand per-space logs is nobody's job.

Sealed identifiers. A space appears in the log as space_tag = HKDF(instance_secret, "lchat-kt-space-tag") — stable within a space (members and monitors derive it; they hold the instance secret, which already exists as key material wrapped to every member) but meaningless to the log operator and unlinkable across spaces. Roster entries under a tag are the sealed shape of "key K joined / rotated / left," carrying no names — screen names are E2EE and stay that way. The log learns: some space, some volume of membership churn. It must never learn which space, or who.

What the instance's own chain (Part 1) is, then: the per-space history commitment stays local-with-witnessing (it commits every message event — far too voluminous and too revealing for a global log). The global log carries only (a) roster/key events under sealed tags and (b) periodic instance chain heads, which is what makes local chains fork-evident. Two layers, one trust story: the instance chain says what this host claims happened; the commons log pins which claim the host is stuck with.

Q2: The invite #fragment bootstraps the log entry — extends, never replaced

Decision: T stays the root of trust for admission; the log makes what T authorized auditable. Concretely, a join produces a log entry that carries a T-derived binding of the joiner's public key, verified by the inviter's client.

The mechanism mirrors the existing deriveInvite derivations (id, wrap key, claim secret — all HKDF domain-separated from T). Add one more:

• bind_key = HKDF(T, "lchat-invite-bind"), held by whoever minted the link (their client already had T) and derivable by the joiner.
• At claim time the joiner publishes, alongside its new public key, bind_mac = HMAC(bind_key, joiner_pubkey).
• The inviter's client — the one party besides the joiner who ever held T — verifies bind_mac against the published roster key before it (or anyone it vouches to) wraps a single thread key to the new member, and then signs the join's log entry as attester.

This closes the substitution attack at its exact seam: a host that swaps the published key can't produce a valid bind_mac for its own key (it never held T — it stores only claim_hash), so the inviter's client refuses to wrap and the join fails loudly instead of succeeding silently.

Subsequent lifecycle events are self-certifying without T:

• Key rotation / add-device: signed by the key being replaced (continuity chain rooted in the T-bound original).
• Removal: signed by an admin identity, itself a T-rooted chain (the setup invite is the genesis entry).
• Wrap discipline: clients wrap thread keys only to keys whose lineage appears in the log — "verify before wrap" is the client-side rule that makes the log load-bearing rather than advisory.

Bots and the eleven CLI follow the same path — they are ordinary members whose T arrives via the agent-invite flow; nothing special-cases them.

Why not replace the fragment with the log? Because the fragment is the only out-of-band channel in the system — the one path the host never sees. A log can prove consistency (everyone sees the same roster) but cannot prove legitimacy (that a roster entry was authorized by a human who held the link). Consistency without an anchor just means the host lies to everyone identically. T supplies legitimacy; the log makes it globally consistent and equivocation provable. They compose; neither substitutes for the other.

Q3: Checkpoint distribution — escrow first, member gossip second, commons log third

Decision: three witnessing channels, shipped in this order, each valuable alone.

1. Escrow replica (near-free, ships with Part 1). The checkpoints table rides the existing Litestream replication to S3, and S3 versioning timestamps every generation of it — a target the provider doesn't control once escrow is commons-held (#550 Part 10). Honest history: monotone chain growth across replica generations. Rollback: a later generation whose head is an ancestor of an earlier generation's. The host could stop replicating, but going dark is itself the detectable event the escrow lease already surfaces. Limit: a fully self-hosted instance with no escrow gets nothing from this channel — hence channels 2 and 3.
2. Member gossip inside E2EE traffic (the novel one, and ours to use: members already share an authenticated, host-opaque channel — the chat itself). Clients periodically tuck the head they last verified (seq_to, chain_hash) inside their normal encrypted payloads; peers compare against their own view. A host that forks — serving member A one history and member B another — is betrayed by its own transport: it can't read the gossip to filter it, and dropping messages wholesale is exactly the withholding that members notice. Split-view detection with zero new infrastructure, working even for self-hosted spaces with no escrow and no commons. This is the distribution answer the issue was looking for.
3. Commons log witnessing (with Part 3/KT). Instances push heads under their sealed tag on a slow cadence; the commons log's own consistency proofs then pin every instance's chain globally. Strongest guarantee, biggest dependency — which is why it's last, not first.

Ordering rationale: 1 is a table already replicating; 2 is a small client change with outsized reach; 3 requires operating a new commons service and should be built once, for KT, not twice.

---

Part 3 — key transparency (design sketch, next after this slice)

With Q1–Q3 settled, KT's shape is: roster/key events become journal kinds (join, rotate, remove) carrying the Q2 evidence (bind MACs, continuity signatures); the instance chain commits to them like any event; instances publish roster events + chain heads to the commons log under their sealed tag; clients enforce verify-before-wrap; monitors (any client, or the commons) audit for equivocation. The commons log itself is CT-shaped: append-only, Merkle, publicly auditable, dumb — it stores sealed commitments and serves proofs, nothing else. Detailed protocol (proof formats, monitor cadence, anti-flap for offline devices) is its own design pass after the journal substrate has soaked in production. It is deliberately not specified further here: the failure mode of KT designs is specifying the whole cathedral before the foundation has carried load.

Part 3a — roster/key journal events (built). The first, safe increment: membership changes are journaled like message events, so they're committed to the signed checkpoint chain and a monitor catches a server rewriting the roster. A member's first key publish appends a join event; removal appends a remove. They reuse the journal's generic columns as the kind's detail record (thread_id = the member's user id, blob_hash = sha256 of the public key for join / empty for remove, msg_id unused) — the leaf encoding already covers these fields, so no wire-format change. Re-registering the same key (add-device) is not a new join. This touches no crypto and no join-path decision; it just makes the roster tamper-evident on the instance chain.

Part 3b — the bind MAC + verify-before-wrap (next, review-gated). The security-critical increment that closes the key-substitution attack (Q2): bind_key = HKDF(T, "lchat-invite-bind"); the joiner publishes bind_mac = HMAC(bind_key, joiner_pubkey) at claim; the inviter's client verifies it against the published roster key before wrapping any thread key, and clients enforce verify-before-wrap generally. Because it changes the join/key-distribution path — the crown jewel of the one rule — it ships as its own PR with an adversarial review pass and a canary before it reaches any real space.

Part 4 — client sampling verification (design sketch)

Because clients hold plaintext truth, whole-DB verification needs no proofs-of-retrievability: each client periodically samples k message ids from its own local store, fetches those rows through a restore-read endpoint over the escrow replica (not the live host — the point is to check what escrow would restore), checks each blob decrypts and matches, and — once inclusion proofs land — proves each against a committed root. Independent sampling across members covers the DB probabilistically at trivial per-client cost, and the pass/fail signal doubles as the client-visible end of provider conformance, which #575's payment gating needs. Depends on: checkpoint chain (shipped), inclusion-proof endpoint, restore-read endpoint. Build after KT's journal extension, before the marketplace.

Out of scope (unchanged from #573)

Provider-served web assets remain the weakest trust point: a malicious host can serve JavaScript that exfiltrates keys, and no log catches that. Signed releases + native apps carry the answer for now; a Code Verify-style extension is the eventual one. Separate work, tracked in #550 Part 10.

Build order

1. Event journal + signed checkpoint chain — this slice. Server-only, additive migration, zero client changes. Starts the evidence clock.
2. Member gossip of chain heads (Q3 channel 2) — small client change, makes forks member-detectable everywhere, no new services.
3. Inclusion proofs endpoint + verify-on-load in clients.
4. KT journal kinds + bind MAC at claim + verify-before-wrap (Q2) — the membership fix, still commons-free.
5. Commons log + sealed tags + monitors (Q1, Q3 channel 3) — the global equivocation pin. Gates #575.
6. Restore-read + client sampling (Part 4). Gates #575's payment conformance.