amadan
docs: federated blob storage — providers contribute buckets, not trust (#990)
The plan of record for taking attachments from one bucket (#987) to many providers — the storage twin of the co-op compute model (#550 Part 10). Core arguments: write-once ciphertext makes storage the easy half of planet-scale (no leases, no consensus — sweeps and bookkeeping); E2EE inverts provider trust to just availability + integrity, both mechanically checkable (hash audits, mirror copies, published standing); contribution = four values (S3 endpoint/bucket/keys/region), no custom protocol. Copy count per plan = the media rung of the durability ladder. Phases: A two-stores-one-owner (ready to build) / B store registry + placement / C provider onboarding + contribution metering. Non-goals locked: DB replicas stay first-party, no erasure coding, no DHT, no provider daemon.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+160
−0
docs/federated-blob-storage.md+160 −0
diff --git a/docs/federated-blob-storage.md b/docs/federated-blob-storage.md| new file mode 100644 |
| index 0000000..12c0d9a |
| --- /dev/null |
| +++ b/docs/federated-blob-storage.md |
| @@ -0,0 +1,160 @@ |
| +# 🤖 Federated blob storage — providers contribute buckets, not trust |
| + |
| +**Status: designed 2026-07-15, Phase A ready to build.** The plan of record |
| +for taking attachment storage from one bucket (#550 Phase 1, the s3BlobStore |
| +merged as #987) to many providers — the storage twin of the co-op compute |
| +model (`docs/planet-scale.md` Part 10). Companion decisions this leans on: |
| +the durability ladder (`tier-economics.md § Durability per tier`) and the |
| +E2EE rule (`README.md` / `CLAUDE.md` — the whole design only works because |
| +of it). |
| + |
| +## Why storage is the easy half of planet-scale |
| + |
| +Hibernation needed leases, heartbeats, and a mesh because a SQLite database |
| +is **mutable with exactly one legitimate writer** — every hard problem in |
| +that design flows from that fact. Attachment blobs are the opposite: |
| +**write-once ciphertext under server-minted ids**. Nothing is ever rewritten, |
| +so there are no conflicts, no single-writer lock, no consensus, and no |
| +ordering. Mirroring immutable objects is order-free and idempotent — a copy |
| +sweep can crash anywhere and rerun from the top. The distributed-systems hard |
| +part evaporates; what remains is sweeps and bookkeeping, which is exactly the |
| +shape this codebase already trusts (the meter sweep, the quota push, the |
| +reaper). |
| + |
| +## The trust inversion E2EE buys |
| + |
| +A storage provider holds ciphertext under opaque ids. They can read |
| +*nothing* — confidentiality is not a trust axis at all. What a provider is |
| +actually trusted with: |
| + |
| +- **Availability** — they might vanish. Answer: a second copy on a second |
| + provider, plus a sweep that notices and re-mirrors. A vanished provider |
| + costs a re-copy, never data. |
| +- **Integrity** — they might corrupt or withhold. Answer: every blob's |
| + ciphertext hash lives in the instance's `blobs` table (recorded at |
| + upload); reads can verify, and an **audit sweep** samples ranged reads per |
| + provider and checks them. A cheating provider is *caught*, published, and |
| + evicted — trust-but-verify with teeth. |
| + |
| +Two leaks remain, and the design closes both: |
| + |
| +- **Access patterns + member IPs**: a provider serving members directly |
| + (presigned URLs) would learn who fetches what, when, from where. So |
| + co-op-provider blobs are served **proxied through the box, never |
| + presigned** — providers see the fleet's IP and nothing about members. |
| + (First-party stores may use presigned URLs later; that's a separate, |
| + deliberate choice per store.) |
| +- **Sizes and timing**: unavoidable for any object store, identical to what |
| + the server itself already sees. Nothing new is exposed. |
| + |
| +## The model, kept boring |
| + |
| +**One wire dialect: S3.** It is the lingua franca — R2, Hetzner Object |
| +Storage, and AWS speak it natively, and MinIO/Garage turn any box with disks |
| +into it. A storage contribution is therefore four values: endpoint, bucket, |
| +key pair, region. No custom protocol, no agent, no inbound port on the |
| +provider's side. The s3BlobStore (#987) already consumes exactly these. |
| + |
| +**A store registry**, beside the route registry: one row per store — name, |
| +endpoint, bucket, region, provider, declared capacity, standing |
| +(`primary-eligible` / `mirror-only` / `draining`). Tens of rows, not |
| +thousands; a table a human reads whole. |
| + |
| +**Placement per space**: each space's row records which store holds its |
| +primary and which its mirror — the same per-space shape as `Route.Provider` |
| +(the co-op seam that has been waiting in the schema since #550 Part 10). |
| +Placement policy stays legible: region affinity first, capacity second, |
| +never more clever than a sentence can state. |
| + |
| +**Two sweeps**: |
| + |
| +- **Mirror sweep**: the `blobs` table enumerates every id a space owns; copy |
| + whatever the mirror store lacks. Idempotent, restartable, journal-free — |
| + the table *is* the journal. |
| +- **Audit sweep**: per store, sample a few blobs, ranged-GET a window, |
| + verify against the stored hash. Probabilistic coverage over time at |
| + trivial request cost — the same philosophy as tamper-evidence inclusion |
| + sampling (#573). Results feed the status pages: provider standing is |
| + public, like everything else about how this fleet runs. |
| + |
| +**Reads**: primary first, then mirror, then the local-disk fallback the |
| +s3BlobStore already has. A store outage degrades to slower reads, never |
| +errors, and the audit sweep turns persistent failures into an eviction. |
| + |
| +**Migration = copy + flip.** Because stores are dumb ciphertext buckets |
| +keyed by id, moving a space between providers is: mirror to the new store, |
| +flip the placement row, drain the old. The same no-lock-in shape as instance |
| +migration — a provider leaving the co-op is an afternoon's sweep, not an |
| +event. |
| + |
| +## One ladder, two knobs |
| + |
| +The durability ladder (`stream`/`batched`/`daily`) governs how fast a |
| +space's *messages* reach the replica. **Copy count is the same product shape |
| +for media**: how many providers hold a space's blobs. One member-facing |
| +story — *your plan's durability covers your messages and your media* — and |
| +plausible rungs (final numbers belong to a pricing session, not this doc): |
| + |
| +| Tier | Message replication | Blob copies | |
| +|---|---|---| |
| +| Free | daily | 1 | |
| +| Family / Community | daily / batched | 2 | |
| +| Business | stream | 2, region-diverse | |
| + |
| +## Contribution accounting |
| + |
| +A provider's contribution is meterable with nothing but the registry and the |
| +audit sweep: **bytes-months held × audited availability**. Both numbers are |
| +publishable (the books, `tier-economics.md`, status pages) — the storage twin |
| +of a co-op member contributing a compute box, credited in whatever form the |
| +Part 10 economics settle on. No usage-secret accounting: the fleet publishes |
| +what it measures, providers can dispute with their own logs. |
| + |
| +**Onboarding ratchet**: a new provider's store joins as `mirror-only` and |
| +carries second copies through an audit burn-in (weeks, not hours). Only |
| +stores that survive it become primary-eligible — a hobbyist NAS strengthens |
| +the pool without ever being load-bearing alone. Standing is a ratchet the |
| +audit sweep can lower but only time can raise. |
| + |
| +## Phases |
| + |
| +- **Phase A — two stores, one owner (ready to build).** Multi-store under |
| + fleet control: a `mirror` store config beside the primary, the ciphertext |
| + hash column + recording at upload, the mirror sweep, read fallback |
| + primary→mirror→disk, and the audit sweep's first form (verify-on-read |
| + counters). Ships dark behind env like #987. This alone closes the |
| + vendor-concentration exposure (Hetzner primary + the existing AWS bucket |
| + as mirror) and exercises every rail federation needs. |
| +- **Phase B — the registry.** Store rows + per-space placement + placement |
| + at provision (cloud push, like the replication class), migration sweep |
| + (copy + flip + drain), audit results to the status pages. |
| +- **Phase C — contribution.** Provider onboarding (four values + burn-in), |
| + standing lifecycle, contribution metering into the books, proxy-only |
| + serving enforced for co-op stores. |
| + |
| +## Non-goals, deliberately |
| + |
| +- **DB replicas stay first-party.** Small, critical, mutable, |
| + lease-guarded — federating them buys pennies and costs the crown jewels' |
| + simplicity. Media is where the volume, cost, and federation win live. |
| +- **No erasure coding.** Two full copies are legible, cheap at object-store |
| + prices, and degrade gracefully. Erasure across heterogeneous providers is |
| + clever; the second rule (`CLAUDE.md`) says legibility wins. |
| +- **No DHT, no consensus, no gossip protocol for placement.** Placement is |
| + rows in a registry a human reads. The fleet is tens of boxes and tens of |
| + stores for the foreseeable future; designs for millions can wait for |
| + millions. |
| +- **No custom provider daemon.** If a provider must run something, it's |
| + MinIO/Garage — maintained by people whose whole job that is. Contribution |
| + stays four values. |
| + |
| +## Decisions needed before Phase A flips live |
| + |
| +1. **Enable Hetzner Object Storage** on the fleet account (primary store; |
| + ~€4.9/TB-mo, egress €1/TB, same DC as the box) — one console action + |
| + key pair into the vault. |
| +2. **Confirm the mirror target**: the existing AWS account (a dedicated |
| + bucket; storage-only cost, writes incur no egress) is the default |
| + assumption. |
| +3. Pack pricing and pay-flips-paid (already on Paul's desk) — independent, |
| + but the same pricing session should set the blob copy-count rungs above. |