🤖 Incoming webhooks: the ingest sidecar (lchat hooks run)

Issue #610. External services (GitHub, Stripe, a greenhouse sensor) can't encrypt to a thread key — but their events belong in the product. This design lets an admin hand an outside service a URL such that every HTTP request it sends becomes an encrypted, typed event (webhooks.<ns>) in one chosen thread — with the server never seeing plaintext content and the receiving process never able to read anything back.

The shape: an inverted bot sidecar

The bot sidecar (lchat bot run, docs/bots.md) is a pure outbound client: it dials out, holds keys, and must never listen. The hooks sidecar (lchat hooks run, internal/hooks/) is its inverse — it exists to accept inbound connections, and is strictly write-only against the instance:

• it never opens a websocket and never calls GET /api/messages — nothing in the process can read a conversation;
• it never holds the instance key, so it can't decrypt profile blobs, member apps, charters, or anything else instance-key-sealed;
• per webhook it holds exactly that webhook's private key and device token, which unwrap exactly one thread's key (to encrypt, and only encrypt).

The two invariants stay unconditional by staying in separate processes: the bot sidecar remains "no inbound ports, relocatable anywhere"; the hooks sidecar is inbound-by-purpose and lives behind the router. Compromising the hooks process yields the threads its webhooks feed — never the rest of the space, never a bot's keys, never an Anthropic key (it has none).

Reaching it: in.<host>, a plain registry route

Each instance's ingest surface is a sibling host — in.family.11ch.at for family.11ch.at — which is just a router registry row (kind: "hooks", socket /run/lchat/<host>-hooks.sock). No router code special-cases it: the router proxies any registered host to its socket.

TLS/DNS: the *.11ch.at wildcard certificate covers only one label, so in.<sub>.11ch.at takes the router's per-host autocert path instead — the cert is minted via HTTP-01 on first contact (HostPolicy allows it because the host is in the registry). Wildcard DNS does resolve two labels down when no closer record exists, so 11ch.at spaces need no zone change. (This is why the earlier /hooks/* path-split design — PR #617 — is superseded: the subdomain shape needs no router changes at all. The one real cost: a custom-domain space (chat.paulca.com) needs its owner to add an in.chat.paulca.com DNS record pointing at the same box.)

The URL an outside service gets is https://in.<host>/h/<secret> — the secret is the capability, exactly the GitHub/Slack convention. The sidecar alone knows which secrets are live; the router forwards blindly and unknown secrets 404 at the sidecar, so the server can't even enumerate webhooks.

One webhook = one agent identity = one thread

A webhook is an ordinary agent member (roster-visible, labelled with the webhook's name, badged webhook), created by the admin's browser:

1. The admin opens the Webhooks app (admin-only, /webhooks or the Apps picker) and names a webhook + its event namespace (github → events of type webhooks.github) + its target thread (a group they're in, or the space thread).
2. Their browser mints the webhook's keypair, wraps the target thread's key to it, and calls POST /api/hooks (admin-gated) — the server creates the agent user, mints its device token (returned once), and writes the membership row carrying the wrapped key. For the space thread this is the explicit-membership opt-in threadMember was built to honour: agents are fenced out of implicit membership and open_key, and a webhook gets the space key only because an admin deliberately wrapped it to that one identity.
3. The browser generates the URL secret, then seals {id, user_id, name, ns, thread_id, secret, token, priv} to the hooks service identity's pubkey (ECIES sealForPub) → def_blob, and encrypts {name, ns, thread_id, secret} under the instance key → meta_blob (the app's list view). PUT /api/hooks/{id}/blobs stores both. The server holds only ciphertext: no key, no token plaintext (tokens are stored hashed), no secret, no target service.

The sidecar polls GET /api/hooks/defs (~10s, sidecar-marked), opens each definition with its service-identity private key, materializes its route table, and acks new webhooks via POST /api/hooks/{id}/deployed — a plaintext timestamp, the only thing the server learns, which the app shows as "Deploying… → Active". Because the client minted the secret, the URL is displayed instantly at creation; the ack just confirms the route is serving. (#610's discussion had the sidecar minting the URL and feeding it back; client-minting is the deliberate simplification — a sidecar-minted URL would have to travel back encrypted, which needs the sidecar to hold the instance key, the very thing this design keeps away from it.)

Removal (DELETE /api/hooks/{id}) deletes the row and sweeps the whole identity (removeUser: token, memberships, roster row); the sidecar's next poll drops the route and the URL dies.

The service identity is provisioned once per instance by lchat add-hooks <host> (box-side, mirrors add-bot): it mints an agent invite flagged hooks_service — recorded in config as hooks_service_user, served to members at GET /api/hooks/service as the seal target — writes /data/<host>-hooks.env, registers the in.<host> route, and enables lchat-hooks@<host>. The sidecar self-provisions from the invite on first boot (bot.EnsureSession, the #806 zero-touch path). It's exempt from the bots gate: webhooks are integrations with their own admin-only gate, not AI.

The event: dumb ingest, smart render

On each inbound request the sidecar builds (pinned by test/event-vectors/webhooks.json):

{
  "type": "webhooks.<ns>",
  "name": "<webhook name>", "uid": "<webhook user id>", "ts": 1752710400000,
  "hook": { "id": "wh-…", "ns": "<ns>" },
  "req": {
    "method": "POST", "path": "/<subpath after the secret>",
    "query": { "site": ["7"] },
    "headers": { "Content-Type": "…", "X-Github-Event": "…" },
    "body": "<raw body>", "body_b64": false
  }
}

encrypts it to the thread key, and POST /api/messages as the webhook — the Announcer's outbound-only walk. The sidecar never transforms: the raw request rides inside the event forever, so rendering can improve later without re-ingesting.

• No default rendering. Webhook events are consumed by apps — a CI dashboard app claims exactly webhooks.github in its events map and folds from there. Until an app claims a type, every client silently ignores it: web via foldEvent/isAppType (no owner → dropped), native via MessagePayload.isRenderableChat (typed non-chat → not rendered, not notified), the service worker via its typed-events-stay-silent rule. Old clients degrade to exactly this. The management app itself claims no types.
• Headers are allowlisted (Content-Type, User-Agent, provider event/signature headers). Full headers routinely carry sender credentials (Authorization, cookies) that must never enter the log — even encrypted, every thread member could read them. req.path is the sub-path after the secret segment, so the capability never rides inside an event.
• Notifications: space-thread deliveries never push at all (push.go skips kind='space'); group deliveries push ciphertext but render nothing (typed, unclaimed) — so a webhook can't buzz a phone until a consumer app decides events are worth words.

Ingest hygiene

• Unknown secret → 404; nothing distinguishes "never existed" from "removed".
• Body over ~40 KB → 413, never truncated (the encrypted event must fit the server's 64 KB message-blob cap).
• Per-webhook token bucket (60 burst, 1/s sustained) → 429; the server's content-blind message brake backstops it independently.
• Non-UTF-8 bodies ride base64url with body_b64: true.
• Delivery failure upstream → 502, so senders with retry logic retry.
• Hibernation-aware: the recurring definition poll is sidecar-marked (never holds a sleeping instance awake; the last-known route table keeps serving), while an actual delivery is unmarked — a webhook arrival is real external activity and wakes the instance to land its event.

Dev story

./tools/dev up runs the whole loop locally, no router: it mints the service invite, spawns lchat hooks run on a sticky local TCP port (the sidecar's identity persists in .lchat-dev-hooks.json, exactly like the dev bot's), and passes the server LCHAT_HOOKS_BASE=http://127.0.0.1:<port> — which the service endpoint echoes as ingest_base, so the Webhooks app displays URLs that actually work on your machine. Dev-you is seeded as an admin member, so the app is right there under Apps → Webhooks; create one, wait for the "Active" chip (~10s poll), and curl the URL — the event lands encrypted in the chosen thread. --no-hooks opts out.

Hand-run equivalent (any server): lchat hooks run --addr :8090 --base http://localhost:1747 with LCHAT_INVITE from an agent invite minted with hooks_service: true, and LCHAT_HOOKS_BASE=http://127.0.0.1:8090 on the server so displayed URLs match. In production leave LCHAT_HOOKS_BASE unset — clients derive https://in.<host>.

Later (deliberately not in v1)

• Member-mintable webhooks with quotas (creation is admin-only today).
• Sender signature enforcement (e.g. verify X-Hub-Signature-256 at the sidecar with a per-webhook shared secret; reject unsigned).
• Consumer apps — the point of the exercise: e.g. a GitHub Actions dashboard claiming webhooks.github, parsing workflow_run payloads into a live runs board.
• Multi-thread webhooks / re-targeting — v1 binds one webhook to one thread at creation; re-point by remove + recreate.