| 1 | # 🤖 The public-web fetch service (`lchat fetch run`) |
| 2 | |
| 3 | A shared, identity-free service that fronts **public web content** for |
| 4 | clients: OpenGraph unfurls and a camo-style signed image proxy for the cards' |
| 5 | images. One binary subcommand (`fetch.go`), one sqlite file, fronted by the |
| 6 | instance router like links/push/home. Born in the unfurl-through-Finland |
| 7 | discussion (PRs #1237/#1240 → this): if a service is going to see posted |
| 8 | URLs at all, it should be a service that knows *nothing else*. |
| 9 | |
| 10 | ## Why it exists |
| 11 | |
| 12 | Unfurling is the messenger's one documented E2E exception: to build a link |
| 13 | card, *some* server must see the URL. Today that server is the member's own |
| 14 | instance. Moving the fetch to the shared service buys, in rough order of value: |
| 15 | |
| 16 | 1. **The instance can stop seeing URLs entirely** (once clients call the service |
| 17 | directly — the phase-2 flip below). The instance server then stores and |
| 18 | relays only ciphertext, full stop; the unfurl exception moves to a service |
| 19 | that holds no members, threads, tokens, or instance list. |
| 20 | 2. **Viewers stop leaking their IPs to card-image hosts.** The web client |
| 21 | sets `img.src` to the raw `og:image` URL today, so every *viewer* of a |
| 22 | card fetches from a third-party host; `http:`-only images die silently as |
| 23 | mixed content. Its `/img` proxy fixes both. |
| 24 | 3. **Self-hosters stop exposing their egress IP.** A member can post a link |
| 25 | to a server they control and learn the instance's IP — for a home |
| 26 | self-hoster, their home IP. Pointing unfurl at the shared service hides it; running |
| 27 | their own keeps full autonomy. |
| 28 | 4. **A shared warm cache.** Popular links unfurl once per language across |
| 29 | every space that uses the service, and the poster/recipient in-flight window |
| 30 | mostly disappears because the cache is usually already warm. |
| 31 | |
| 32 | ## Trust model |
| 33 | |
| 34 | - **Sees:** posted URLs + an `Accept-Language` tag, transiently, attached to |
| 35 | nothing — no token, no instance id, no member id. Behind the router it |
| 36 | doesn't even see client IPs (nothing forwards them, deliberately). |
| 37 | - **Stores:** the sealed preview cache — same convergent-encryption scheme as |
| 38 | the instance's (`unfurl.go`): row key = salted hash of language+URL, blob |
| 39 | opens only with a key derived from language+URL, service-wide salt (minted |
| 40 | on first boot, lives only in the service's DB) defeats precomputed |
| 41 | dictionaries. The cache at rest reveals nothing a reader doesn't already |
| 42 | know the URL for. No request logs; the only log line is startup. |
| 43 | - **Never:** member content. Member photos/files are encrypted client-side |
| 44 | (`processImage` + `encryptBytes`) and are categorically out of scope — |
| 45 | its remit is content already on the public web. Any future endpoint that |
| 46 | would touch member bytes is a design error; say no. |
| 47 | - **Not an open proxy:** `/img` serves only URLs the service itself HMAC-signed |
| 48 | while building a card (GitHub-camo shape), so third parties can't drive it |
| 49 | as a generic fetcher. Signing is at response time; the cache stores raw |
| 50 | URLs, so the key could rotate without invalidating cached cards. |
| 51 | - **SSRF:** every outbound dial (unfurl page + image, every redirect hop) |
| 52 | re-validates the resolved IP via `guardDialControl` — same guard as the |
| 53 | instance, links, and push. |
| 54 | |
| 55 | ## Abuse posture — hashcash admission (shipped, off by default) |
| 56 | |
| 57 | Set `LCHAT_FETCH_POW_BITS` (0/unset = off) and `/api/unfurl` + `/api/gif` |
| 58 | demand proof of work — home's space-creation scheme (`fetchpow.go`, kept in |
| 59 | step with `home/pow.go`): a 401 hands out a sealed single-use challenge, |
| 60 | the client grinds SHA-256 to that many leading zero bits ONCE, the solved |
| 61 | request is served and answered with a sealed ~1h **ticket**, and every call |
| 62 | after rides (and is rate-bucketed by) the ticket — one solve per client per |
| 63 | hour, never per unfurl. Admission travels in query params so cross-origin |
| 64 | callers pay no CORS preflight; `/img` needs none (its HMAC signature is the |
| 65 | admission). Both web (`fetchServiceGET`, shell.js) and native |
| 66 | (`LChatAPI.fetchServiceGET` + the `FetchService` registry) solve and hold |
| 67 | tickets. Ungated, the per-IP buckets remain — which behind the router |
| 68 | collapse into global buckets (no forwarded IPs — a privacy feature): arm |
| 69 | the gate before the shared service is load-bearing for the hosted fleet. |
| 70 | |
| 71 | ## Deployment |
| 72 | |
| 73 | Standard sibling-service shape: `lchat fetch run` behind the router on a unix |
| 74 | socket (`-socket /run/lchat/fetch.sock`), DB at `/data/fetch.db` |
| 75 | (`LCHAT_FETCH_DB`), service route + `/etc/hosts` pin like cloud/home. Not |
| 76 | deployed anywhere yet; when it ships, the planet-scale guide gains its row. |
| 77 | No secrets in env — the salt and signing key mint themselves into the DB on |
| 78 | first boot. |
| 79 | |
| 80 | ## The client flip (shipped, env-gated) |
| 81 | |
| 82 | `LCHAT_FETCH_URL` on an instance routes its clients here: the chat page CSP |
| 83 | grows the origin in `connect-src` + `window.__fetchURL` injects it (web), and |
| 84 | `fetch_url` rides `/api/group`'s member-gated read (native — `FeatureGates`, |
| 85 | latched so a failed read never reroutes). Both platforms prefer the service |
| 86 | with instance-unfurl fallback on transport failure. Self-host default stays |
| 87 | *local fetch* — a self-hosted instance silently shipping its members' URLs to |
| 88 | our shared service would betray the self-host story; pointing at the shared |
| 89 | service (or their own) is opt-in config. |
| 90 | |
| 91 | ## Resize (shipped) |
| 92 | |
| 93 | `/img` re-encodes oversized JPEG/PNG card images down to card width |
| 94 | (`shrinkCardImage`: >1600px wide → 1200px, CatmullRom, same format) — an |
| 95 | og:image is routinely thousands of px for a card rendering at a few hundred, |
| 96 | and every viewer pays that transfer. GIFs pass through (resizing freezes the |
| 97 | animation), as does anything that fails to decode, claims bomb-scale pixel |
| 98 | dims, or comes out bigger "shrunk" — polish never breaks a card. Public web |
| 99 | images only; member photos keep their client-side pipeline |
| 100 | (`processImage`), always. |
| 101 | |
| 102 | ## Giphy (`GET /api/gif`) |
| 103 | |
| 104 | The authenticated-upstream sibling: search/trending fronted with the |
| 105 | **platform's** Giphy key (`LCHAT_GIPHY_KEY` in the unit's env file — the one |
| 106 | env secret this service has), so a space needs no key of its own and the |
| 107 | get-an-API-key wall disappears. The response is Giphy's own page shape with |
| 108 | every giphy-hosted media URL rewritten to the signed `/img` proxy, so the |
| 109 | browser never talks to Giphy: previews and the posted GIF's bytes all ride |
| 110 | this service's origin, then encrypt client-side and post as ordinary |
| 111 | attachments exactly as before. |
| 112 | |
| 113 | Honest trade vs the old path: direct client→Giphy meant *we* saw nothing and |
| 114 | Giphy saw the member's IP + query; via the service, the service sees search |
| 115 | terms (identity-free, unlogged) and Giphy sees only the service. No key |
| 116 | configured → `/api/gif` 404s and the client falls back to the space's |
| 117 | bring-your-own-key path (docs/app-credentials.md), which remains the |
| 118 | self-host/override story. |
| 119 | |
amadan