🤖 The public-web fetch service (lchat fetch run)

A shared, identity-free service that fronts public web content for clients: OpenGraph unfurls and a camo-style signed image proxy for the cards' images. One binary subcommand (fetch.go), one sqlite file, fronted by the instance router like links/push/home. Born in the unfurl-through-Finland discussion (PRs #1237/#1240 → this): if a service is going to see posted URLs at all, it should be a service that knows nothing else.

Why it exists

Unfurling is the messenger's one documented E2E exception: to build a link card, some server must see the URL. Today that server is the member's own instance. Moving the fetch to the shared service buys, in rough order of value:

1. The instance can stop seeing URLs entirely (once clients call the service directly — the phase-2 flip below). The instance server then stores and relays only ciphertext, full stop; the unfurl exception moves to a service that holds no members, threads, tokens, or instance list.
2. Viewers stop leaking their IPs to card-image hosts. The web client sets img.src to the raw og:image URL today, so every viewer of a card fetches from a third-party host; http:-only images die silently as mixed content. Its /img proxy fixes both.
3. Self-hosters stop exposing their egress IP. A member can post a link to a server they control and learn the instance's IP — for a home self-hoster, their home IP. Pointing unfurl at the shared service hides it; running their own keeps full autonomy.
4. A shared warm cache. Popular links unfurl once per language across every space that uses the service, and the poster/recipient in-flight window mostly disappears because the cache is usually already warm.

Trust model

• Sees: posted URLs + an Accept-Language tag, transiently, attached to nothing — no token, no instance id, no member id. Behind the router it doesn't even see client IPs (nothing forwards them, deliberately).
• Stores: the sealed preview cache — same convergent-encryption scheme as the instance's (unfurl.go): row key = salted hash of language+URL, blob opens only with a key derived from language+URL, service-wide salt (minted on first boot, lives only in the service's DB) defeats precomputed dictionaries. The cache at rest reveals nothing a reader doesn't already know the URL for. No request logs; the only log line is startup.
• Never: member content. Member photos/files are encrypted client-side (processImage + encryptBytes) and are categorically out of scope — its remit is content already on the public web. Any future endpoint that would touch member bytes is a design error; say no.
• Not an open proxy: /img serves only URLs the service itself HMAC-signed while building a card (GitHub-camo shape), so third parties can't drive it as a generic fetcher. Signing is at response time; the cache stores raw URLs, so the key could rotate without invalidating cached cards.
• SSRF: every outbound dial (unfurl page + image, every redirect hop) re-validates the resolved IP via guardDialControl — same guard as the instance, links, and push.

Abuse posture — hashcash admission (shipped, off by default)

Set LCHAT_FETCH_POW_BITS (0/unset = off) and /api/unfurl + /api/gif demand proof of work — home's space-creation scheme (fetchpow.go, kept in step with home/pow.go): a 401 hands out a sealed single-use challenge, the client grinds SHA-256 to that many leading zero bits ONCE, the solved request is served and answered with a sealed ~1h ticket, and every call after rides (and is rate-bucketed by) the ticket — one solve per client per hour, never per unfurl. Admission travels in query params so cross-origin callers pay no CORS preflight; /img needs none (its HMAC signature is the admission). Both web (fetchServiceGET, shell.js) and native (LChatAPI.fetchServiceGET + the FetchService registry) solve and hold tickets. Ungated, the per-IP buckets remain — which behind the router collapse into global buckets (no forwarded IPs — a privacy feature): arm the gate before the shared service is load-bearing for the hosted fleet.

Deployment

Standard sibling-service shape: lchat fetch run behind the router on a unix socket (-socket /run/lchat/fetch.sock), DB at /data/fetch.db (LCHAT_FETCH_DB), service route + /etc/hosts pin like cloud/home. Not deployed anywhere yet; when it ships, the planet-scale guide gains its row. No secrets in env — the salt and signing key mint themselves into the DB on first boot.

The client flip (shipped, env-gated)

LCHAT_FETCH_URL on an instance routes its clients here: the chat page CSP grows the origin in connect-src + window.__fetchURL injects it (web), and fetch_url rides /api/group's member-gated read (native — FeatureGates, latched so a failed read never reroutes). Both platforms prefer the service with instance-unfurl fallback on transport failure. Self-host default stays local fetch — a self-hosted instance silently shipping its members' URLs to our shared service would betray the self-host story; pointing at the shared service (or their own) is opt-in config.

Resize (shipped)

/img re-encodes oversized JPEG/PNG card images down to card width (shrinkCardImage: >1600px wide → 1200px, CatmullRom, same format) — an og:image is routinely thousands of px for a card rendering at a few hundred, and every viewer pays that transfer. GIFs pass through (resizing freezes the animation), as does anything that fails to decode, claims bomb-scale pixel dims, or comes out bigger "shrunk" — polish never breaks a card. Public web images only; member photos keep their client-side pipeline (processImage), always.

Giphy (GET /api/gif)

The authenticated-upstream sibling: search/trending fronted with the platform's Giphy key (LCHAT_GIPHY_KEY in the unit's env file — the one env secret this service has), so a space needs no key of its own and the get-an-API-key wall disappears. The response is Giphy's own page shape with every giphy-hosted media URL rewritten to the signed /img proxy, so the browser never talks to Giphy: previews and the posted GIF's bytes all ride this service's origin, then encrypt client-side and post as ordinary attachments exactly as before.

Honest trade vs the old path: direct client→Giphy meant we saw nothing and Giphy saw the member's IP + query; via the service, the service sees search terms (identity-free, unlogged) and Giphy sees only the service. No key configured → /api/gif 404s and the client falls back to the space's bring-your-own-key path (docs/app-credentials.md), which remains the self-host/override story.