| 1 | # 🤖 Incoming webhooks: the ingest sidecar (`lchat hooks run`) |
| 2 | |
| 3 | Issue #610. External services (GitHub, Stripe, a greenhouse sensor) can't |
| 4 | encrypt to a thread key — but their events belong in the product. This design |
| 5 | lets an admin hand an outside service a URL such that every HTTP request it |
| 6 | sends becomes an **encrypted, typed event** (`webhooks.<ns>`) in **one chosen |
| 7 | thread** — with the server never seeing plaintext content and the receiving |
| 8 | process never able to read anything back. |
| 9 | |
| 10 | ## The shape: an inverted bot sidecar |
| 11 | |
| 12 | The bot sidecar (`lchat bot run`, docs/bots.md) is a **pure outbound client**: |
| 13 | it dials out, holds keys, and must never listen. The hooks sidecar |
| 14 | (`lchat hooks run`, `internal/hooks/`) is its inverse — it exists to accept |
| 15 | inbound connections, and is **strictly write-only** against the instance: |
| 16 | |
| 17 | - it never opens a websocket and never calls `GET /api/messages` — nothing in |
| 18 | the process can read a conversation; |
| 19 | - it never holds the instance key, so it can't decrypt profile blobs, member |
| 20 | apps, charters, or anything else instance-key-sealed; |
| 21 | - per webhook it holds exactly that webhook's private key and device token, |
| 22 | which unwrap exactly **one** thread's key (to encrypt, and only encrypt). |
| 23 | |
| 24 | The two invariants stay unconditional by staying in separate processes: the |
| 25 | bot sidecar remains "no inbound ports, relocatable anywhere"; the hooks |
| 26 | sidecar is inbound-by-purpose and lives behind the router. Compromising the |
| 27 | hooks process yields the threads its webhooks feed — never the rest of the |
| 28 | space, never a bot's keys, never an Anthropic key (it has none). |
| 29 | |
| 30 | ## Reaching it: `in.<host>`, a plain registry route |
| 31 | |
| 32 | Each instance's ingest surface is a sibling host — `in.family.11ch.at` for |
| 33 | `family.11ch.at` — which is just a router registry row |
| 34 | (`kind: "hooks"`, socket `/run/lchat/<host>-hooks.sock`). No router code |
| 35 | special-cases it: the router proxies any registered host to its socket. |
| 36 | |
| 37 | TLS/DNS: the `*.11ch.at` **wildcard certificate** covers only one label, so |
| 38 | `in.<sub>.11ch.at` takes the router's **per-host autocert** path instead — the |
| 39 | cert is minted via HTTP-01 on first contact (HostPolicy allows it because the |
| 40 | host is in the registry). Wildcard **DNS** does resolve two labels down when |
| 41 | no closer record exists, so 11ch.at spaces need no zone change. (This is why |
| 42 | the earlier `/hooks/*` path-split design — PR #617 — is superseded: the |
| 43 | subdomain shape needs no router changes at all. The one real cost: a |
| 44 | **custom-domain** space (`chat.paulca.com`) needs its owner to add an |
| 45 | `in.chat.paulca.com` DNS record pointing at the same box.) |
| 46 | |
| 47 | The URL an outside service gets is `https://in.<host>/h/<secret>` — the |
| 48 | secret is the capability, exactly the GitHub/Slack convention. The sidecar |
| 49 | alone knows which secrets are live; the router forwards blindly and unknown |
| 50 | secrets 404 **at the sidecar**, so the server can't even enumerate webhooks. |
| 51 | |
| 52 | ## One webhook = one agent identity = one thread |
| 53 | |
| 54 | A webhook is an ordinary **agent member** (roster-visible, labelled with the |
| 55 | webhook's name, badged `webhook`), created by the admin's browser: |
| 56 | |
| 57 | 1. The admin opens the **Webhooks** app (admin-only, `/webhooks` or the Apps |
| 58 | picker) and names a webhook + its **event namespace** (`github` → |
| 59 | events of type `webhooks.github`) + its **target thread** (a group they're |
| 60 | in, or the space thread). |
| 61 | 2. Their browser mints the webhook's **keypair**, wraps the target thread's |
| 62 | key to it, and calls `POST /api/hooks` (admin-gated) — the server creates |
| 63 | the agent user, mints its device token (returned once), and writes the |
| 64 | membership row carrying the wrapped key. For the **space thread** this is |
| 65 | the explicit-membership opt-in `threadMember` was built to honour: agents |
| 66 | are fenced out of implicit membership and `open_key`, and a webhook gets |
| 67 | the space key only because an admin deliberately wrapped it to that one |
| 68 | identity. |
| 69 | 3. The browser generates the URL **secret**, then seals |
| 70 | `{id, user_id, name, ns, thread_id, secret, token, priv}` to the **hooks |
| 71 | service identity's pubkey** (ECIES `sealForPub`) → `def_blob`, and encrypts |
| 72 | `{name, ns, thread_id, secret}` under the **instance key** → `meta_blob` |
| 73 | (the app's list view). `PUT /api/hooks/{id}/blobs` stores both. The server |
| 74 | holds only ciphertext: no key, no token plaintext (tokens are stored |
| 75 | hashed), no secret, no target service. |
| 76 | |
| 77 | The sidecar polls `GET /api/hooks/defs` (~10s, sidecar-marked), opens each |
| 78 | definition with its service-identity private key, materializes its route |
| 79 | table, and acks new webhooks via `POST /api/hooks/{id}/deployed` — a plaintext |
| 80 | timestamp, the only thing the server learns, which the app shows as |
| 81 | "Deploying… → Active". Because the client minted the secret, the URL is |
| 82 | displayed **instantly** at creation; the ack just confirms the route is |
| 83 | serving. (#610's discussion had the sidecar minting the URL and feeding it |
| 84 | back; client-minting is the deliberate simplification — a sidecar-minted URL |
| 85 | would have to travel back encrypted, which needs the sidecar to hold the |
| 86 | instance key, the very thing this design keeps away from it.) |
| 87 | |
| 88 | **Removal** (`DELETE /api/hooks/{id}`) deletes the row and sweeps the whole |
| 89 | identity (`removeUser`: token, memberships, roster row); the sidecar's next |
| 90 | poll drops the route and the URL dies. |
| 91 | |
| 92 | The **service identity** is provisioned once per instance by |
| 93 | `lchat add-hooks <host>` (box-side, mirrors `add-bot`): it mints an agent |
| 94 | invite flagged `hooks_service` — recorded in config as `hooks_service_user`, |
| 95 | served to members at `GET /api/hooks/service` as the seal target — writes |
| 96 | `/data/<host>-hooks.env`, registers the `in.<host>` route, and enables |
| 97 | `lchat-hooks@<host>`. The sidecar self-provisions from the invite on first |
| 98 | boot (`bot.EnsureSession`, the #806 zero-touch path). It's exempt from the |
| 99 | bots gate: webhooks are integrations with their own admin-only gate, not AI. |
| 100 | |
| 101 | ## The event: dumb ingest, smart render |
| 102 | |
| 103 | On each inbound request the sidecar builds (pinned by |
| 104 | `test/event-vectors/webhooks.json`): |
| 105 | |
| 106 | ```json |
| 107 | { |
| 108 | "type": "webhooks.<ns>", |
| 109 | "name": "<webhook name>", "uid": "<webhook user id>", "ts": 1752710400000, |
| 110 | "hook": { "id": "wh-…", "ns": "<ns>" }, |
| 111 | "req": { |
| 112 | "method": "POST", "path": "/<subpath after the secret>", |
| 113 | "query": { "site": ["7"] }, |
| 114 | "headers": { "Content-Type": "…", "X-Github-Event": "…" }, |
| 115 | "body": "<raw body>", "body_b64": false |
| 116 | } |
| 117 | } |
| 118 | ``` |
| 119 | |
| 120 | encrypts it to the thread key, and `POST /api/messages` as the webhook — the |
| 121 | Announcer's outbound-only walk. The sidecar **never transforms**: the raw |
| 122 | request rides inside the event forever, so rendering can improve later without |
| 123 | re-ingesting. |
| 124 | |
| 125 | - **No default rendering.** Webhook events are consumed by *apps* — a CI |
| 126 | dashboard app claims exactly `webhooks.github` in its `events` map and folds |
| 127 | from there. Until an app claims a type, every client silently ignores it: |
| 128 | web via `foldEvent`/`isAppType` (no owner → dropped), native via |
| 129 | `MessagePayload.isRenderableChat` (typed non-chat → not rendered, not |
| 130 | notified), the service worker via its typed-events-stay-silent rule. Old |
| 131 | clients degrade to exactly this. The management app itself claims **no** |
| 132 | types. |
| 133 | - **Headers are allowlisted** (`Content-Type`, `User-Agent`, provider |
| 134 | event/signature headers). Full headers routinely carry sender credentials |
| 135 | (`Authorization`, cookies) that must never enter the log — even encrypted, |
| 136 | every thread member could read them. `req.path` is the sub-path *after* the |
| 137 | secret segment, so the capability never rides inside an event. |
| 138 | - **Notifications:** space-thread deliveries never push at all (push.go skips |
| 139 | `kind='space'`); group deliveries push ciphertext but render nothing (typed, |
| 140 | unclaimed) — so a webhook can't buzz a phone until a consumer app decides |
| 141 | events are worth words. |
| 142 | |
| 143 | ## Ingest hygiene |
| 144 | |
| 145 | - Unknown secret → **404**; nothing distinguishes "never existed" from |
| 146 | "removed". |
| 147 | - Body over ~40 KB → **413**, never truncated (the encrypted event must fit |
| 148 | the server's 64 KB message-blob cap). |
| 149 | - Per-webhook token bucket (60 burst, 1/s sustained) → **429**; the server's |
| 150 | content-blind message brake backstops it independently. |
| 151 | - Non-UTF-8 bodies ride base64url with `body_b64: true`. |
| 152 | - Delivery failure upstream → **502**, so senders with retry logic retry. |
| 153 | - Hibernation-aware: the recurring definition poll is sidecar-marked (never |
| 154 | holds a sleeping instance awake; the last-known route table keeps serving), |
| 155 | while an actual delivery is unmarked — a webhook arrival is real external |
| 156 | activity and **wakes** the instance to land its event. |
| 157 | |
| 158 | ## Dev story |
| 159 | |
| 160 | `./tools/dev up` runs the whole loop locally, no router: it mints the service |
| 161 | invite, spawns `lchat hooks run` on a sticky local TCP port (the sidecar's |
| 162 | identity persists in `.lchat-dev-hooks.json`, exactly like the dev bot's), and |
| 163 | passes the server `LCHAT_HOOKS_BASE=http://127.0.0.1:<port>` — which the |
| 164 | service endpoint echoes as `ingest_base`, so the Webhooks app displays URLs |
| 165 | that actually work on your machine. Dev-you is seeded as an admin member, so |
| 166 | the app is right there under Apps → Webhooks; create one, wait for the |
| 167 | "Active" chip (~10s poll), and `curl` the URL — the event lands encrypted in |
| 168 | the chosen thread. `--no-hooks` opts out. |
| 169 | |
| 170 | Hand-run equivalent (any server): `lchat hooks run --addr :8090 --base |
| 171 | http://localhost:1747` with `LCHAT_INVITE` from an agent invite minted with |
| 172 | `hooks_service: true`, and `LCHAT_HOOKS_BASE=http://127.0.0.1:8090` on the |
| 173 | server so displayed URLs match. In production leave `LCHAT_HOOKS_BASE` unset — |
| 174 | clients derive `https://in.<host>`. |
| 175 | |
| 176 | ## Later (deliberately not in v1) |
| 177 | |
| 178 | - **Member-mintable webhooks** with quotas (creation is admin-only today). |
| 179 | - **Sender signature enforcement** (e.g. verify `X-Hub-Signature-256` at the |
| 180 | sidecar with a per-webhook shared secret; reject unsigned). |
| 181 | - **Consumer apps** — the point of the exercise: e.g. a GitHub Actions |
| 182 | dashboard claiming `webhooks.github`, parsing `workflow_run` payloads into |
| 183 | a live runs board. |
| 184 | - **Multi-thread webhooks / re-targeting** — v1 binds one webhook to one |
| 185 | thread at creation; re-point by remove + recreate. |
| 186 | |
amadan