| 1 | # 🤖 Integrations — a suite of connected services, from one worked example |
| 2 | |
| 3 | Status: **built on main** (branch `integrations-github`). Generalises the |
| 4 | DNSimple proof (`docs/connected-services.md`) into a catalog anyone can |
| 5 | extend: GitHub, Stripe, HubSpot and friends as installable integrations, with |
| 6 | read/write permissions scoped per grant — and therefore per bot — and every |
| 7 | operation available to bots as a tool behind the same consent chips the rest |
| 8 | of the platform uses. GitHub is the first registry-native service, with its |
| 9 | own companion app (`web/static/apps/github.js`). |
| 10 | |
| 11 | ## The shape: manifests are the palette |
| 12 | |
| 13 | Everything hangs off one idea, borrowed from bot recipes: **an integration is |
| 14 | data, not code**. A service ships as a JSON manifest |
| 15 | (`integrations/registry/`) declaring its auth, the ONE API host its |
| 16 | operations may touch, and a closed list of operations — each `read` or |
| 17 | `write`, writes each carrying the confirmation sentence a person taps before |
| 18 | it runs. The generic executor (`integrations/exec.go`) can only do what a |
| 19 | manifest spells; per-service Go code (the old `dnsimple.go`) stops being the |
| 20 | growth path. |
| 21 | |
| 22 | **Responses are as reviewable as requests.** Each op may declare a `pick` |
| 23 | projection — a root envelope to descend plus the exact fields that may flow |
| 24 | back. The executor drops everything else before a result ever leaves the |
| 25 | sidecar, which is both a privacy bound (the manifest review covers what data |
| 26 | can enter a thread) and a practical one: an op result rides inside ONE |
| 27 | end-to-end-encrypted message (64 KB blob cap), while a raw GitHub |
| 28 | `/user/repos` page is ~400 KB of mostly-unwanted fields. Oversized results |
| 29 | refuse honestly; nothing is ever silently truncated. |
| 30 | |
| 31 | What each party holds is unchanged from the DNSimple design: |
| 32 | |
| 33 | | Party | Holds | |
| 34 | |---|---| |
| 35 | | Messenger server | opaque sealed credential blobs, as ever — nothing new | |
| 36 | | Connect broker | OAuth client secrets (one per OAuth-app service) + the public registry (`GET /registry`) | |
| 37 | | Member's client | the credential in the clear only with the member present; seals scoped grants to bots | |
| 38 | | Bot sidecar | a grant in the clear during its window; the manifest-pinned executor | |
| 39 | |
| 40 | ## Read/write, per bot |
| 41 | |
| 42 | A grant sealed to a bot carries `access: "read" | "write"` alongside the |
| 43 | token and expiry (an access-less grant is legacy = read-only). Because a |
| 44 | grant is sealed to exactly one bot's public key, **access is per-bot by |
| 45 | construction**: seal Eleven a write grant for DNSimple and your reporting bot |
| 46 | a read-only one, and the executor enforces the difference — a read grant |
| 47 | cannot run a write op, whatever the model asks for. Least privilege stacks in |
| 48 | four layers: |
| 49 | |
| 50 | 1. **The token itself** — mint scoped PATs (GitHub fine-grained, Stripe |
| 51 | restricted keys, DNSimple domain-scoped); the manifest's `pat_help` points |
| 52 | at each service's scoping page, and the catalog verifies a pasted token |
| 53 | immediately through the manifest's `viewer` op when one exists. |
| 54 | 2. **The connection cap** — chosen by the member AT CONNECT TIME (read-only |
| 55 | is the default; read & write is the explicit choice), sealed inside the |
| 56 | credential blob so the server never learns it. The member's client refuses |
| 57 | to mint any grant beyond the cap (`capAllows`, `ui.js`) — a read-only |
| 58 | connection cannot produce a write grant whatever a chip or proposal asks — |
| 59 | and for OAuth services whose manifests declare `scopes` / `scopes_write`, |
| 60 | the cap also picks the provider-side scope set on the authorize URL, so a |
| 61 | read-only connection's token is born read-only at the provider too. The |
| 62 | cap is changeable later in Integrations (re-sealing the blob; the member's |
| 63 | presence is the consent). Connections stored before caps existed carry no |
| 64 | cap and behave as before. |
| 65 | 3. **The grant** — read/write + expiry, sealed per bot, minted per operation |
| 66 | (the apps and chips use one-shot ~2-minute grants). |
| 67 | 4. **The tap** — every write op renders its `confirm` line as a consent chip |
| 68 | ("Register blip.com? This buys the domain and costs real money."); |
| 69 | nothing irreversible runs from model output alone. |
| 70 | |
| 71 | ## Bots use connections as tools |
| 72 | |
| 73 | Interactive: *"@eleven go and register blip.com"* → the brain's `service_op` |
| 74 | tool (its description is GENERATED from the registry — the closed list the |
| 75 | manifests declare is the closed list the model can propose) → a whitelisted |
| 76 | chip whose label, for writes, is the manifest's rendered confirm line → the |
| 77 | tap seals a one-shot access-scoped grant and posts the `connect.op` → the |
| 78 | sidecar executor runs it → the outcome lands as a private notice, never a |
| 79 | data dump. Reads work the same minus the drama. `TestServiceOpEndToEnd` |
| 80 | drives the whole flow, including the negative (a read grant refused before |
| 81 | the registrar is touched). |
| 82 | |
| 83 | **Research sessions** widen the interactive flow one notch: a question that |
| 84 | needs *reading* one or more connected services — especially cross-referencing |
| 85 | two ("which of my DNSimple domains point at GitHub Pages?") — gets a single |
| 86 | `service_research` consent chip naming the services. The tap seals a |
| 87 | ~10-minute READ grant per service to the bot; the sidecar then runs a bounded |
| 88 | tool loop (`internal/bot/research.go`) whose palette is exactly the granted |
| 89 | manifests' read ops — write ops are never in the model's world here, whatever |
| 90 | the grants say — and posts one answer in the thread that asked. Results ride |
| 91 | fenced in `tool_result` blocks (data, never instructions), each is |
| 92 | pick-projected by the same pinned executor, and a provider pair rotated |
| 93 | mid-loop is handed back sealed to the member exactly like a single op's. |
| 94 | |
| 95 | **What the model sees — and never sees.** The op *results* (pick-projected |
| 96 | data) enter the model's context — that is the point of research — and so, like |
| 97 | any conversation a bot takes part in, they transit the Anthropic API from the |
| 98 | sidecar. The *credentials* never do: tokens, grants, and refresh secrets live |
| 99 | only in the sidecar handler's locals and are never interpolated into a prompt, |
| 100 | a tool description, or a tool_result (`TestResearchEndToEnd` pins this — no |
| 101 | request to the model may contain the provider token). Say it to members the |
| 102 | honest way round: "the bot can read what you let it read, and nobody — not |
| 103 | the server, not the model — is ever handed your keys." |
| 104 | |
| 105 | Standing (designed, deliberately not built): a member-bot charter gains |
| 106 | `connections: [{service, access}]` — shown on the creation consent card like |
| 107 | sources are — and at approval the client seals a standing grant to the new |
| 108 | bot, renewed by presence like the DNSimple autonomous-window design |
| 109 | (`docs/connected-services.md § token custody`). Task runs then expose the |
| 110 | granted services' READ ops as model tools; writes stay interactive-only until |
| 111 | there's a reviewed answer for unattended spending. This is the natural meet |
| 112 | point with member bots (issue #432). |
| 113 | |
| 114 | ## Connected-service apps ride the same rail |
| 115 | |
| 116 | A service's companion app (DNSimple's domain table, GitHub's repos/issues) is |
| 117 | a normal mini-app gated by `requiresService` and fed by |
| 118 | `ctx.host.connected.requestOp(service, op, args, access)` — the app renders, |
| 119 | the bot executes under the grant, the `pick`-bounded result folds into the |
| 120 | app's state as an ordinary typed event. That last step is what plugs |
| 121 | integrations into the app **sharing model**: folded events are the thread's |
| 122 | log, so the data syncs across the member's devices via snapshots exactly like |
| 123 | a to-do list, and a future `groupCapable` service app would share one fetched |
| 124 | view with a whole group through the same log — no new machinery. |
| 125 | |
| 126 | ## The external registry repo |
| 127 | |
| 128 | The registry directory is destined to be its own public repository |
| 129 | (`elevenmessenger/integrations`) — the audit surface for what integrations |
| 130 | can do, growable by PR without touching this codebase. The pipeline: |
| 131 | |
| 132 | 1. **Author** a manifest (PR to the registry repo; review = reading the ops — |
| 133 | including `pick`, which is the review of what data can flow back). |
| 134 | `integrations.Parse` is exported precisely so the registry repo's CI can |
| 135 | validate a PR'd manifest with the same fail-closed rules the binary uses. |
| 136 | 2. **OAuth services**: register the OAuth app once, secret into the connect |
| 137 | broker's config — the same one-place-holds-the-secret move as the push |
| 138 | relay's `.p8`. PAT-only services (GitHub, Stripe, HubSpot) need nothing |
| 139 | beyond the manifest. |
| 140 | 3. **Serve**: the broker embeds the registry and serves `GET /registry` |
| 141 | (CORS-open) to every instance's web client; each instance also serves its |
| 142 | own embedded copy at `GET /api/integrations` (same-origin, what the |
| 143 | catalog uses today); sidecars embed a snapshot at build time |
| 144 | (single-binary ethos) — the broker copy lets the *catalog UI* grow between |
| 145 | deploys, while *execution* trusts only the reviewed, embedded snapshot it |
| 146 | shipped with. Repo split + a broker sync-from-repo job are the ship-time |
| 147 | steps (creating the GitHub repo is deliberately a human act). |
| 148 | |
| 149 | ## What's built (this branch) |
| 150 | |
| 151 | 1. `integrations/registry/` — schema + README + five manifests (DNSimple, |
| 152 | GitHub, Stripe, HubSpot, Linear; reads with `pick` projections, confirmed |
| 153 | writes). Linear is the first GraphQL service and needs no executor |
| 154 | special-casing: each op is a POST to `/graphql` whose `body_json` carries a |
| 155 | static query string plus member input in `variables`, `pick`'s dot-path |
| 156 | root descends the `data.…` envelope, and its personal API keys ride |
| 157 | `"authorization": "none"` (Linear rejects a Bearer prefix on them — the |
| 158 | header is the bare key). Linear is also the first **mixed-scheme** service — |
| 159 | its OAuth tokens DO want Bearer — which is why the sealed credential now |
| 160 | records its `kind` ("pat" | "oauth") at connect time: grants carry it, and |
| 161 | the executor (`ExecuteAs`) sends an OAuth token as Bearer per RFC 6750 |
| 162 | whatever the manifest's pasted-key scheme says (empty kind = legacy |
| 163 | credential = manifest scheme, unchanged). Its 24-hour OAuth tokens ride the |
| 164 | expiring-pair rotation below unchanged; `scopes`/`scopes_write` map the |
| 165 | connection cap to Linear's `read` vs `read issues:create`. One authoring |
| 166 | rule keeps GraphQL text out of the |
| 167 | placeholder machinery: every brace in query text keeps a space inside |
| 168 | (`{ nodes ` never `{nodes`), so nothing can parse as a `{arg}` placeholder — |
| 169 | the fail-closed loader rejects any slip. |
| 170 | 2. `integrations` Go package — fail-closed loader + the pinned executor |
| 171 | (arg validation, path/query escaping, JSON/form bodies, host pinning, |
| 172 | response caps, `pick` projection), embedded into the binary. |
| 173 | 3. Sidecar `connect.op` v2 — any manifest op, grant-scope enforcement |
| 174 | (read/write, expiry, service match); the legacy DNSimple `list` path is |
| 175 | untouched. |
| 176 | 4. The interactive flow — the brain's `service_op` tool → whitelisted consent |
| 177 | chip (labels sized for a whole consent sentence) → the client seals the |
| 178 | scoped one-shot grant and round-trips the op → outcome as a private notice. |
| 179 | 5. The catalog — `GET /api/integrations` + broker `GET /registry`; the |
| 180 | Integrations app lists every service with paste-a-scoped-PAT (verified on |
| 181 | save via `viewer`) and a "Connect with X" button for any service the |
| 182 | broker has an OAuth app for. |
| 183 | 6. The GitHub app — repos, per-repo open issues/PRs, open-an-issue (the first |
| 184 | write a member performs from inside an app, consented by the form itself). |
| 185 | 7. **The OAuth relay is service-generic.** The golden path is: OAuth on the |
| 186 | provider's own site whenever the broker has an app (the catalog probes |
| 187 | `/api/oauth/config?service=` and only shows the button when the flow can |
| 188 | actually start); paste-a-scoped-token is the fallback. The broker holds one |
| 189 | confidential client per service — DNSimple on its legacy `LCHAT_DNSIMPLE_*` |
| 190 | envs, everything else on `LCHAT_OAUTH_<ID>_CLIENT_ID`/`_CLIENT_SECRET` |
| 191 | (provider endpoints come from the manifest's `auth.oauth`; the redirect |
| 192 | defaults to the broker's own `/oauth/<id>/callback`). Enabling GitHub OAuth |
| 193 | is therefore: create the GitHub App (callback = the broker's |
| 194 | `/oauth/github/callback`), set the two envs, restart the broker — no |
| 195 | instance or client deploy. Prefer a **GitHub App** over a classic OAuth |
| 196 | app: members pick which repositories at consent time (least privilege as a |
| 197 | consent screen), and its exchange rides the same confidential broker flow. |
| 198 | |
| 199 | 8. **Expiring tokens rotate themselves.** Providers with short-lived user |
| 200 | tokens (GitHub Apps: 8-hour access + single-use rotating refresh tokens) |
| 201 | work first-class: the stored credential is the whole pair |
| 202 | (`{secret, refresh, expires_at}`, still sealed to the member's own key), |
| 203 | the pair rides inside the op grant, and the SIDECAR is the only refresher — |
| 204 | it rotates a stale pair before the call (or on a surprise 401, retrying |
| 205 | once), serialized per credential so two near-simultaneous ops share one |
| 206 | rotation instead of burning the single-use token. The fresh pair rides |
| 207 | back **on the op result**, sealed to the member, and the requesting device |
| 208 | re-stores it. Rotation is refused outright when the fresh pair couldn't be |
| 209 | handed back (no resolvable member key) — a failed op beats a silently |
| 210 | severed connection. Broker side: `POST /api/refresh`, same auth and |
| 211 | nothing-persisted shape as the exchange. This is the temporal-custody |
| 212 | model from `connected-services.md` made native: a stolen blob ages out in |
| 213 | hours. Failure mode is honest: a dead refresh token (revoked, >6 months |
| 214 | idle) errors with "reconnect it in Integrations". |
| 215 | |
| 216 | Out of scope, called out honestly: standing charter grants (above), |
| 217 | registering the OAuth apps themselves (a human act per provider — the Stripe |
| 218 | and HubSpot manifests are OAuth-ready, scopes mapped to the connection cap |
| 219 | [HubSpot: CRM read scopes always, contacts.write only on a read & write |
| 220 | connection. Stripe: `read_write` for BOTH caps — Stripe refuses the |
| 221 | `read_only` OAuth scope unless support grants the platform read-only |
| 222 | connections, so until that's requested the connection cap alone enforces |
| 223 | read-only there, and the provider token is broader than the cap; ask Stripe |
| 224 | support for read-only connections before go-live if that residue matters]; |
| 225 | the broker offers each Connect button |
| 226 | the moment `LCHAT_OAUTH_<ID>_CLIENT_ID`/`_CLIENT_SECRET` land in its env, |
| 227 | and until then the catalog offers paste-a-token alone. NB Stripe's OAuth |
| 228 | `client_secret` is the platform's own secret key — treat its env value with |
| 229 | that weight. HubSpot's 30-minute access tokens ride the expiring-pair |
| 230 | rotation above unchanged), |
| 231 | Disconnect-revokes-provider-side, native catalog UI (connected services |
| 232 | remain web-only), and the Node reference sidecar (`agent/`) neither executes |
| 233 | registry ops nor proposes `service_op` — the Go sidecar is the production |
| 234 | path. |
| 235 | |
amadan