eleven / messenger Public

Download this file

1	# 🤖 Federated blob storage — providers contribute buckets, not trust
2
3	**Status: designed 2026-07-15, Phase A ready to build.** The plan of record
4	for taking attachment storage from one bucket (#550 Phase 1, the s3BlobStore
5	merged as #987) to many providers — the storage twin of the co-op compute
6	model (`docs/planet-scale.md` Part 10). Companion decisions this leans on:
7	the durability ladder (`tier-economics.md § Durability per tier`) and the
8	E2EE rule (`README.md` / `CLAUDE.md` — the whole design only works because
9	of it).
10
11	## Why storage is the easy half of planet-scale
12
13	Hibernation needed leases, heartbeats, and a mesh because a SQLite database
14	is **mutable with exactly one legitimate writer** — every hard problem in
15	that design flows from that fact. Attachment blobs are the opposite:
16	**write-once ciphertext under server-minted ids**. Nothing is ever rewritten,
17	so there are no conflicts, no single-writer lock, no consensus, and no
18	ordering. Mirroring immutable objects is order-free and idempotent — a copy
19	sweep can crash anywhere and rerun from the top. The distributed-systems hard
20	part evaporates; what remains is sweeps and bookkeeping, which is exactly the
21	shape this codebase already trusts (the meter sweep, the quota push, the
22	reaper).
23
24	## The trust inversion E2EE buys
25
26	A storage provider holds ciphertext under opaque ids. They can read
27	*nothing* — confidentiality is not a trust axis at all. What a provider is
28	actually trusted with:
29
30	- **Availability** — they might vanish. Answer: a second copy on a second
31	provider, plus a sweep that notices and re-mirrors. A vanished provider
32	costs a re-copy, never data.
33	- **Integrity** — they might corrupt or withhold. Answer: every blob's
34	ciphertext hash lives in the instance's `blobs` table (recorded at
35	upload); reads can verify, and an **audit sweep** samples ranged reads per
36	provider and checks them. A cheating provider is *caught*, published, and
37	evicted — trust-but-verify with teeth.
38
39	Two leaks remain, and the design closes both:
40
41	- **Access patterns + member IPs**: a provider serving members directly
42	(presigned URLs) would learn who fetches what, when, from where. So
43	co-op-provider blobs are served **proxied through the box, never
44	presigned** — providers see the fleet's IP and nothing about members.
45	(First-party stores may use presigned URLs later; that's a separate,
46	deliberate choice per store.)
47	- **Sizes and timing**: unavoidable for any object store, identical to what
48	the server itself already sees. Nothing new is exposed.
49
50	## The model, kept boring
51
52	**One wire dialect: S3.** It is the lingua franca — R2, Hetzner Object
53	Storage, and AWS speak it natively, and MinIO/Garage turn any box with disks
54	into it. A storage contribution is therefore four values: endpoint, bucket,
55	key pair, region. No custom protocol, no agent, no inbound port on the
56	provider's side. The s3BlobStore (#987) already consumes exactly these.
57
58	**A store registry**, beside the route registry: one row per store — name,
59	endpoint, bucket, region, provider, declared capacity, standing
60	(`primary-eligible` / `mirror-only` / `draining`). Tens of rows, not
61	thousands; a table a human reads whole.
62
63	**Placement per space**: each space's row records which store holds its
64	primary and which its mirror — the same per-space shape as `Route.Provider`
65	(the co-op seam that has been waiting in the schema since #550 Part 10).
66	Placement policy stays legible: region affinity first, capacity second,
67	never more clever than a sentence can state.
68
69	**Two sweeps**:
70
71	- **Mirror sweep**: the `blobs` table enumerates every id a space owns; copy
72	whatever the mirror store lacks. Idempotent, restartable, journal-free —
73	the table *is* the journal.
74	- **Audit sweep**: per store, sample a few blobs, ranged-GET a window,
75	verify against the stored hash. Probabilistic coverage over time at
76	trivial request cost — the same philosophy as tamper-evidence inclusion
77	sampling (#573). Results feed the status pages: provider standing is
78	public, like everything else about how this fleet runs.
79
80	**Reads**: primary first, then mirror, then the local-disk fallback the
81	s3BlobStore already has. A store outage degrades to slower reads, never
82	errors, and the audit sweep turns persistent failures into an eviction.
83
84	**Migration = copy + flip.** Because stores are dumb ciphertext buckets
85	keyed by id, moving a space between providers is: mirror to the new store,
86	flip the placement row, drain the old. The same no-lock-in shape as instance
87	migration — a provider leaving the co-op is an afternoon's sweep, not an
88	event.
89
90	## One ladder, two knobs
91
92	The durability ladder (`stream`/`batched`/`daily`) governs how fast a
93	space's *messages* reach the replica. **Copy count is the same product shape
94	for media**: how many providers hold a space's blobs. One member-facing
95	story — *your plan's durability covers your messages and your media* — and
96	plausible rungs (final numbers belong to a pricing session, not this doc):
97
98	| Tier | Message replication | Blob copies |
99	|---|---|---|
100	| Free | daily | 1 |
101	| Family / Community | daily / batched | 2 |
102	| Business | stream | 2, region-diverse |
103
104	## Contribution accounting
105
106	A provider's contribution is meterable with nothing but the registry and the
107	audit sweep: **bytes-months held × audited availability**. Both numbers are
108	publishable (the books, `tier-economics.md`, status pages) — the storage twin
109	of a co-op member contributing a compute box, credited in whatever form the
110	Part 10 economics settle on. No usage-secret accounting: the fleet publishes
111	what it measures, providers can dispute with their own logs.
112
113	**Onboarding ratchet**: a new provider's store joins as `mirror-only` and
114	carries second copies through an audit burn-in (weeks, not hours). Only
115	stores that survive it become primary-eligible — a hobbyist NAS strengthens
116	the pool without ever being load-bearing alone. Standing is a ratchet the
117	audit sweep can lower but only time can raise.
118
119	## Phases
120
121	- **Phase A — two stores, one owner (ready to build).** Multi-store under
122	fleet control: a `mirror` store config beside the primary, the ciphertext
123	hash column + recording at upload, the mirror sweep, read fallback
124	primary→mirror→disk, and the audit sweep's first form (verify-on-read
125	counters). Ships dark behind env like #987. This alone closes the
126	vendor-concentration exposure (Hetzner primary + the existing AWS bucket
127	as mirror) and exercises every rail federation needs.
128	- **Phase B — the registry.** Store rows + per-space placement + placement
129	at provision (cloud push, like the replication class), migration sweep
130	(copy + flip + drain), audit results to the status pages.
131	- **Phase C — contribution.** Provider onboarding (four values + burn-in),
132	standing lifecycle, contribution metering into the books, proxy-only
133	serving enforced for co-op stores.
134
135	## Non-goals, deliberately
136
137	- **DB replicas stay first-party.** Small, critical, mutable,
138	lease-guarded — federating them buys pennies and costs the crown jewels'
139	simplicity. Media is where the volume, cost, and federation win live.
140	- **No erasure coding.** Two full copies are legible, cheap at object-store
141	prices, and degrade gracefully. Erasure across heterogeneous providers is
142	clever; the second rule (`CLAUDE.md`) says legibility wins.
143	- **No DHT, no consensus, no gossip protocol for placement.** Placement is
144	rows in a registry a human reads. The fleet is tens of boxes and tens of
145	stores for the foreseeable future; designs for millions can wait for
146	millions.
147	- **No custom provider daemon.** If a provider must run something, it's
148	MinIO/Garage — maintained by people whose whole job that is. Contribution
149	stays four values.
150
151	## Decisions needed before Phase A flips live
152
153	1. **Enable Hetzner Object Storage** on the fleet account (primary store;
154	~€4.9/TB-mo, egress €1/TB, same DC as the box) — one console action +
155	key pair into the vault.
156	2. **Confirm the mirror target**: the existing AWS account (a dedicated
157	bucket; storage-only cost, writes incur no egress) is the default
158	assumption.
159	3. Pack pricing and pay-flips-paid (already on Paul's desk) — independent,
160	but the same pricing session should set the blob copy-count rungs above.
161