🤖 Federated blob storage — providers contribute buckets, not trust

Status: designed 2026-07-15, Phase A ready to build. The plan of record for taking attachment storage from one bucket (#550 Phase 1, the s3BlobStore merged as #987) to many providers — the storage twin of the co-op compute model (docs/planet-scale.md Part 10). Companion decisions this leans on: the durability ladder (tier-economics.md § Durability per tier) and the E2EE rule (README.md / CLAUDE.md — the whole design only works because of it).

Why storage is the easy half of planet-scale

Hibernation needed leases, heartbeats, and a mesh because a SQLite database is mutable with exactly one legitimate writer — every hard problem in that design flows from that fact. Attachment blobs are the opposite: write-once ciphertext under server-minted ids. Nothing is ever rewritten, so there are no conflicts, no single-writer lock, no consensus, and no ordering. Mirroring immutable objects is order-free and idempotent — a copy sweep can crash anywhere and rerun from the top. The distributed-systems hard part evaporates; what remains is sweeps and bookkeeping, which is exactly the shape this codebase already trusts (the meter sweep, the quota push, the reaper).

The trust inversion E2EE buys

A storage provider holds ciphertext under opaque ids. They can read nothing — confidentiality is not a trust axis at all. What a provider is actually trusted with:

• Availability — they might vanish. Answer: a second copy on a second provider, plus a sweep that notices and re-mirrors. A vanished provider costs a re-copy, never data.
• Integrity — they might corrupt or withhold. Answer: every blob's ciphertext hash lives in the instance's blobs table (recorded at upload); reads can verify, and an audit sweep samples ranged reads per provider and checks them. A cheating provider is caught, published, and evicted — trust-but-verify with teeth.

Two leaks remain, and the design closes both:

• Access patterns + member IPs: a provider serving members directly (presigned URLs) would learn who fetches what, when, from where. So co-op-provider blobs are served proxied through the box, never presigned — providers see the fleet's IP and nothing about members. (First-party stores may use presigned URLs later; that's a separate, deliberate choice per store.)
• Sizes and timing: unavoidable for any object store, identical to what the server itself already sees. Nothing new is exposed.

The model, kept boring

One wire dialect: S3. It is the lingua franca — R2, Hetzner Object Storage, and AWS speak it natively, and MinIO/Garage turn any box with disks into it. A storage contribution is therefore four values: endpoint, bucket, key pair, region. No custom protocol, no agent, no inbound port on the provider's side. The s3BlobStore (#987) already consumes exactly these.

A store registry, beside the route registry: one row per store — name, endpoint, bucket, region, provider, declared capacity, standing (primary-eligible / mirror-only / draining). Tens of rows, not thousands; a table a human reads whole.

Placement per space: each space's row records which store holds its primary and which its mirror — the same per-space shape as Route.Provider (the co-op seam that has been waiting in the schema since #550 Part 10). Placement policy stays legible: region affinity first, capacity second, never more clever than a sentence can state.

Two sweeps:

• Mirror sweep: the blobs table enumerates every id a space owns; copy whatever the mirror store lacks. Idempotent, restartable, journal-free — the table is the journal.
• Audit sweep: per store, sample a few blobs, ranged-GET a window, verify against the stored hash. Probabilistic coverage over time at trivial request cost — the same philosophy as tamper-evidence inclusion sampling (#573). Results feed the status pages: provider standing is public, like everything else about how this fleet runs.

Reads: primary first, then mirror, then the local-disk fallback the s3BlobStore already has. A store outage degrades to slower reads, never errors, and the audit sweep turns persistent failures into an eviction.

Migration = copy + flip. Because stores are dumb ciphertext buckets keyed by id, moving a space between providers is: mirror to the new store, flip the placement row, drain the old. The same no-lock-in shape as instance migration — a provider leaving the co-op is an afternoon's sweep, not an event.

One ladder, two knobs

The durability ladder (stream/batched/daily) governs how fast a space's messages reach the replica. Copy count is the same product shape for media: how many providers hold a space's blobs. One member-facing story — your plan's durability covers your messages and your media — and plausible rungs (final numbers belong to a pricing session, not this doc):

| Tier | Message replication | Blob copies | |---|---|---| | Free | daily | 1 | | Family / Community | daily / batched | 2 | | Business | stream | 2, region-diverse |

Contribution accounting

A provider's contribution is meterable with nothing but the registry and the audit sweep: bytes-months held × audited availability. Both numbers are publishable (the books, tier-economics.md, status pages) — the storage twin of a co-op member contributing a compute box, credited in whatever form the Part 10 economics settle on. No usage-secret accounting: the fleet publishes what it measures, providers can dispute with their own logs.

Onboarding ratchet: a new provider's store joins as mirror-only and carries second copies through an audit burn-in (weeks, not hours). Only stores that survive it become primary-eligible — a hobbyist NAS strengthens the pool without ever being load-bearing alone. Standing is a ratchet the audit sweep can lower but only time can raise.

Phases

• Phase A — two stores, one owner (ready to build). Multi-store under fleet control: a mirror store config beside the primary, the ciphertext hash column + recording at upload, the mirror sweep, read fallback primary→mirror→disk, and the audit sweep's first form (verify-on-read counters). Ships dark behind env like #987. This alone closes the vendor-concentration exposure (Hetzner primary + the existing AWS bucket as mirror) and exercises every rail federation needs.
• Phase B — the registry. Store rows + per-space placement + placement at provision (cloud push, like the replication class), migration sweep (copy + flip + drain), audit results to the status pages.
• Phase C — contribution. Provider onboarding (four values + burn-in), standing lifecycle, contribution metering into the books, proxy-only serving enforced for co-op stores.

Non-goals, deliberately

• DB replicas stay first-party. Small, critical, mutable, lease-guarded — federating them buys pennies and costs the crown jewels' simplicity. Media is where the volume, cost, and federation win live.
• No erasure coding. Two full copies are legible, cheap at object-store prices, and degrade gracefully. Erasure across heterogeneous providers is clever; the second rule (CLAUDE.md) says legibility wins.
• No DHT, no consensus, no gossip protocol for placement. Placement is rows in a registry a human reads. The fleet is tens of boxes and tens of stores for the foreseeable future; designs for millions can wait for millions.
• No custom provider daemon. If a provider must run something, it's MinIO/Garage — maintained by people whose whole job that is. Contribution stays four values.

Decisions needed before Phase A flips live

1. Enable Hetzner Object Storage on the fleet account (primary store; ~€4.9/TB-mo, egress €1/TB, same DC as the box) — one console action + key pair into the vault.
2. Confirm the mirror target: the existing AWS account (a dedicated bucket; storage-only cost, writes incur no egress) is the default assumption.
3. Pack pricing and pay-flips-paid (already on Paul's desk) — independent, but the same pricing session should set the blob copy-count rungs above.