eleven / messenger Public

Download this file

1	# 🤖 Planet scale — the running system
2
3	This is the distillation of issue **#550** ("use an object store as the only
4	database — planet-scale plan", twelve parts across two working sessions).
5	Most of what that thread designed has since shipped; this doc describes the
6	system as it **runs**, with the still-unbuilt parts clearly marked as roadmap.
7	Where the issue and the code differ, the code wins, and the difference is
8	noted. #550 stays open only as the historical record of how we got here and
9	closes when this merges.
10
11	## The one idea
12
13	The E2EE rule did the hard part before any of this started. Home's server-side
14	state is opaque client-encrypted ciphertext plus a few kilobytes of public-key
15	metadata, with zero cross-user queries — so home scales by **removing the
16	database**: an S3 bucket, used raw, is a strictly better fit than SQLite ever
17	was. An instance's SQLite file is opaque blobs plus routing metadata the server
18	already sees — so replicating it to a bucket is threat-model-free, and
19	instances scale by **removing the idle process**: state parks in S3 as
20	ciphertext, a process exists only while someone's talking, and creating a
21	space becomes (nearly) a metadata write. Launch-day load lands on S3's
22	capacity, not ours.
23
24	## What runs today
25
26	### Home: the store is a seam, and S3 is implementation #2
27
28	`home/store.go` (PR #611) extracts **`vaultStore`**, home's whole persistence
29	surface, shaped like an object store on purpose: point reads by key,
30	conditional writes (CAS on an *opaque version string*, with the one universal
31	sentinel `"0"` = "no blob yet"), keep-last-N blob versions, prefix deletes,
32	and write-once pointers. SQLite is implementation #1 and stays the zero-config
33	self-hoster default forever — dual backend is a feature, not migration debt.
34
35	`home/s3store.go` (PR #627) is implementation #2: **one bucket, no database**,
36	laid out exactly as the issue drew it —
37
38	```
39	blobs/<uid> the vault; S3-native versioning, version = the ETag
40	users/<uid> one metadata JSON per user (credentials + devices),
41	every mutation an If-Match CAS loop
42	creds/<credId> write-once pointer, body = uid
43	tokens/<sha256> write-once pointer, body = uid
44	```
45
46	CAS rides S3 conditional writes (`If-Match` to replace; the `"0"` sentinel
47	maps to `If-None-Match: *` for create-only). Pointers are never authoritative:
48	resolving one re-loads `users/<uid>` and verifies the credential/token is
49	still listed, so the stale pointer a revocation leaves behind denies by
50	construction. Two open questions from the issue were settled at PR time,
51	both worth recording:
52
53	- **The client is a hand-rolled SigV4 signer** (`home/sigv4.go`), not the AWS
54	SDK — the surface home needs is six calls, and home keeps its tiny dependency
55	footprint. (The *activator* side, below, does carry `aws-sdk-go-v2`, brought
56	in by litestream — the two made opposite calls for good local reasons.)
57	- **Keep-last-5 is enforced at the interface, not by the bucket**:
58	`listBlobVersions` caps at five; the bucket's lifecycle rule prunes
59	noncurrent versions asynchronously, days later, as a backstop. And S3 cannot
60	distinguish "a version string that was never ours" from a plain mismatch, so
61	the SQLite backend's `errBadVersion` (→ 400) is deliberately backend-specific;
62	on S3 both read as conflicts (→ 409).
63
64	Selection is by env (`home/main.go`): `LCHAT_HOME_STORE` unset or `sqlite` is
65	the default; `s3` requires `LCHAT_HOME_S3_BUCKET` + `LCHAT_HOME_S3_REGION`
66	plus AWS credentials in the standard env vars, with optional
67	`LCHAT_HOME_S3_PREFIX` (test isolation) and `LCHAT_HOME_S3_ENDPOINT`
68	(path-style, for S3-compatible stores). Misconfiguration fails at boot, never
69	limps.
70
71	The wire contract was already ETag-opaque: PR #612 made the blob `version` a
72	JSON string on the wire (clients tolerated both) and coarsened link-token
73	`last_seen` to daily, so an object-store backend never pays a write per
74	authenticated read.
75
76	### Home is stateless
77
78	`home/stateless.go` (PR #613) removed the last two things a home process
79	remembered between requests. Ceremony proofs became **HMAC-signed tokens**
80	(uid + expiry, 5-minute window) the client carries; in-flight WebAuthn
81	sessions are **AES-GCM-sealed into the `?session=` query param**. Both keys
82	derive from one server secret, `LCHAT_HOME_SESSION_KEY` — auth plumbing like
83	the admin token, it cannot open a byte of any vault. Unset, an ephemeral key
84	is minted at boot (with a logged warning): single-process self-hosters keep
85	working with zero config, and ceremonies just don't survive a restart, which
86	is exactly what the old in-memory map gave them. With the bucket backend and
87	the stateless ceremonies together, home is a pure function from
88	(request, bucket) → response — any replica behind a load balancer can finish
89	what another began.
90
91	### Fleet backup: replication as a property of normal operation
92
93	`deploy/fleet-backup.sh` (PRs #637, #643) closed the actual gap the plan
94	started from — instance DBs living on one box with no off-box copy. It runs on
95	the router box, idempotent, regenerating `/etc/litestream.yml` from what's in
96	`/data`: a CLI **Litestream** (0.5.14) streams every `/data/*.db` — instances
97	plus home/push/cloud/registry — to the fleet-backup bucket
98	(`eleven-prod-fleet-backup-…`, one prefix per database), under the
99	`lchat-litestream` unit and a scoped IAM user that can touch objects but never
100	versioning, lifecycle, or the bucket itself. Versioning + a 14-day noncurrent
101	window is the ransomware posture: an overwrite or delete with the box's own
102	credentials leaves recoverable versions behind. Continuous replication, not
103	snapshots — seconds of RPO, point-in-time recovery, and it supersedes the
104	parked R2-snapshot (`r2d2.sh`) plan entirely. The restore drill is one
105	command, documented in the script's header; run it against a scratch path,
106	never over a live file.
107
108	One exclusion, added when hibernation went live (PR #658): **hibernating hosts
109	are skipped** — the activator's embedded Litestream owns their replication (to
110	the instances bucket, under the lease), and two replicators on one WAL would
111	fight over checkpointing.
112
113	### Instance hibernation, end to end
114
115	This is Part 4 v2 of the issue, running. Four layers:
116
117	**The lease** (`internal/activator/replica.go`, PR #630). Litestream is
118	embedded **as a Go library** — pure `modernc.org/sqlite`, no cgo, so the
119	single-binary property survives. Its S3 lease (`lock.json` acquired via
120	conditional writes) is the distributed single-writer lock: a second claimant
121	fails loudly with a typed `*litestream.LeaseExistsError`. This is a
122	correction the plan earned by testing (recorded on #550): the issue assumed
123	the lease was CLI config, but **the lease exists only in the library API — the
124	CLI silently absorbs a second replicator**, orphaning the loser's writes with
125	no error signal. Embedding it turned that from a footgun into the mechanism.
126	TTL is 15 m, renewed every 5 m (each renewal is one S3 Tier-1 PUT — at the
127	original 30 s/10 s cadence the lease heartbeat alone was ~70% of the fleet's
128	S3 request bill, ~$16/yr per always-awake instance against €5/y plans); a
129	failed renewal means leadership is lost and the holder stands down. The
130	failover window follows the TTL: a wedged or hard-killed process is claimable
131	by a healthy peer within ~15 m of its last renewal — every graceful path
132	(sleep, deploy, shutdown) releases explicitly and hands over in seconds, so
133	the window prices only real crashes. Restores run
134	*under* the held lease (restoring outside it could resurrect a stale copy that
135	shadows the true holder's newer state).
136
137	**The activator** (`activator.go`, PR #641). With `LCHAT_HIBERNATE_BUCKET`
138	set, the router stops relying on systemd to keep flagged instances alive and
139	runs them as its **own children, on demand** — "units are for the router,
140	instances are its children", exactly as the issue put it. The per-host state
141	machine is asleep → waking → awake → stopping:
142
143	- **Wake** (a request for a host with no live process): claim the per-host
144	lease at `<prefix>/<host>/lock.json`; restore the DB from S3 only if the
145	local copy is gone (a present local DB is the warm cache — on one box, under
146	the lease, it can only be our own last checkpoint); spawn the instance
147	(exec of the same versioned `live/<host>` binary its systemd unit would run,
148	same env file); start replication under the lease. Concurrent requests
149	coalesce on one wake (singleflight); lease contention and wake timeouts are
150	503 + `Retry-After` — "sleeping spaces are slow to wake", never a 5xx
151	cascade.
152	- **Sleep** (idle past `LCHAT_HIBERNATE_IDLE`, default 15 m — idleness is
153	in-flight-count *and* time, so one open WebSocket holds an instance awake):
154	probe the next due time, SIGTERM the child (its own graceful drain is the
155	final write), final checkpoint to S3, release the lease, forget the entry.
156	The local DB stays as the warm cache.
157	- **Failure is the wake path**: a dead child or a lost lease stands the entry
158	down, and the next request re-wakes it. S3 blips leave warm instances
159	serving (local SQLite is authoritative; replication queues and drains) —
160	only cold wakes stall.
161
162	**Due-time wakes** (PR #653). Scheduled deliveries (future messages,
163	reminders) fire on time while their instance sleeps: at sleep the stop path
164	asks the instance for its earliest pending delivery (`GET /api/next-due`,
165	admin-gated, over the socket) and records it in the registry
166	(`routes.next_due` — persisted, so it survives router restarts); the sweeper
167	wakes any sleeping host whose time has arrived, no client request needed.
168	Worst-case lateness ≈ one sweep interval (≤30 s) + spawn + one scheduler tick
169	(≤5 s). One honest gap the issue glossed: **member-bot charter schedules run
170	in the sidecar** (`docs/member-bots.md`), so they didn't wake a sleeping
171	instance — the index covered server-side schedules only. Closed by
172	**charter-wake registration**: after every schedule recompute the charter host
173	reports its earliest next run to the instance (`POST /api/charter-next-due`,
174	bot-token-gated — one bare timestamp; the charters stay ciphertext), the
175	instance folds it into `GET /api/next-due`, and the same index wakes for it —
176	zero activator changes. A stale report from a dead sidecar is ignored once
177	past (it changes only when the sidecar re-reports, so honoring it would re-wake
178	after every sleep): at most one wasted wake, never a loop — and a sidecar that
179	can't come up couldn't run the charter anyway. The due-wake's other half lives
180	in the sidecar: a co-spawned host that loses the delivery race (the instance's
181	scheduler fires the tick before the host's cursor baseline) treats a live
182	charter with no pending alarm at connect as owing a run now, so the woken
183	charter actually runs instead of skipping to its next occurrence.
184
185	**Sidecar co-spawn** (PR #670). A host with a bot sidecar staged
186	(`/data/<host>-bot.env` — the same file `lchat-bot@.service` reads) gets its
187	`bot run` child spawned right after the instance turns healthy and SIGTERMed
188	first at sleep: the sidecar exists exactly while its instance is awake, so its
189	lifecycle solves itself — the issue's "co-located over loopback" default,
190	built. The subtlety that makes it work: the sidecar dials its instance's
191	public URL back through this very router, so it stamps every request
192	(`X-Lchat-Sidecar`, `internal/bot/sidecarmark.go`) and the activator makes
193	that traffic **activity-invisible** — no idle-clock credit while awake, a 503
194	(never a wake) while asleep. Without that, the pair could never sleep and an
195	orphaned sidecar's reconnect loop would wake its instance right back, forever.
196	Spoofing the marker only downgrades the sender's own requests. Supervision is
197	sweep-driven Restart=always with capped exponential backoff; a sidecar that
198	won't start never fails the wake — members before bots.
199
200	### The flip, per host, and creates that are born asleep
201
202	Opt-in is **per host** (`routes.hibernate` in the registry, PR #659), which is
203	what makes a production canary safe: `LCHAT_HIBERNATE_BUCKET` ships fleet-wide
204	as a no-op, then
205
206	```sh
207	lchat hibernate <host> --on # stop+disable its unit; the router wakes it per request
208	lchat hibernate <host> --off # activator releases (checkpoint + lease back), unit restarts
209	lchat instances # HIBERNATE column shows who's activator-managed
210	```
211
212	migrates exactly one instance, and back. The flip's ordering is deliberate
213	(flag → stop unit → reload router → warm request) so the host is never left
214	unserveable, and `--off` waits for the child to actually release the socket
215	before the unit starts (an instance refuses to clobber a live listener). A
216	staged bot sidecar hands over both ways in the same flip.
217
218	One lesson found live, now baked in: **env-file ownership.** Systemd reads a
219	root-written `EnvironmentFile` as root; the activator reads it as the service
220	user — the canary's first wake died on permission denied. `add-instance` and
221	both flips now chown the env (and bot session dirs) to match the data dir.
222
223	Creation splits into **register** (the row: host, minted-or-imported token,
224	placement fields — pure metadata, no capacity) and **materialize** (env file,
225	staged binary, unit, health check, router reload, setup invite) — PR #625,
226	exactly the seam the issue asked for. A `--hibernate` create (and, behind
227	`LCHAT_CONTROL_HIBERNATE_CREATES=1` on the control daemon, **every
228	provisioner-created space** — PR #663) skips only the systemd half: no unit,
229	no resident process, no DB — the first wake finds no replica in S3 and boots
230	fresh. Capacity for hibernating rows is a row ceiling
231	(`LCHAT_MAX_REGISTERED`, default 10× `LCHAT_MAX_INSTANCES`), not a process
232	cap. Honest delta from the issue's "create = a registry row + an S3 prefix,
233	microseconds, no box contacted": today's hibernating create still writes the
234	env file, stages the binary, reloads the router — and its readiness probe *is*
235	the first wake, because minting the setup invite needs a running instance to
236	pre-seed the admin user. Cheap, but not yet free; the fully-deferred
237	materialization remains roadmap.
238
239	The **reaper** got the property Part 5 promised — reclaiming never-claimed
240	spaces **without waking anything**. A hibernating space's stats
241	(`setup_complete` / members / `has_content`) are read straight off its local
242	DB file, read-only — wake-free because the warm cache is the newest truth on
243	one box. The case the code had to add: a hibernating space that has *never
244	woken* has no DB at all, and before the fix that read as a stats error — and
245	a stats error is a skip-never-reap, so register-only spaces would have been
246	immortal. The stats handler now answers those **definitionally from the
247	registry row** (0 members, never set up, no content); a missing DB on a
248	*non*-hibernating row stays a 404, meaning a failed create, skipped as before.
249
250	### The mesh: multiple boxes, one owner per instance
251
252	The multi-box mechanism from Part 4 v2 shipped as PR #678, gated on
253	`LCHAT_MESH_SELF` (a peer-reachable base URL); absent means zero mesh
254	codepaths, and the current single-box deployment runs exactly as before.
255
256	- **Membership is S3 heartbeats**, not gossip: each box PUTs one object per
257	60 seconds under the hibernation prefix's `mesh/members/`, and the object
258	*key* is the record (base64url of the peer URL) so a single LIST yields the
259	whole membership. Liveness reads S3's own `LastModified`, so a box with a
260	wrong clock can't poison the staleness judgment. Stale after 3 m (three
261	missed beats — well inside the lease expiry that actually gates takeover).
262	- **Rendezvous (HRW) hashing is the placement hint** — `owner(host)` is a
263	pure function of the live member set; adding a box moves only the hosts
264	that land on it. **The lease stays the truth**: a box that wrongly believes
265	it owns a host loses the CAS race loudly and proxies instead, so transient
266	membership disagreement structurally cannot split-brain.
267	- **Wrong-box requests reverse-proxy to the owner**, TLS verified against the
268	instance hostname with the connection pinned to the peer's address (no
269	`InsecureSkipVerify` anywhere); a one-hop marker turns disagreement loops
270	into a wrong-box wake the lease resolves.
271	- **Drain deletes the heartbeat first**, then checkpoints and releases as
272	usual — so deploys, scale-down, and box death are the same code path. The
273	kill-a-box property is a live test, both ways: a drained owner's instances
274	are serving on the survivor ~2 seconds later; a SIGKILLed owner's once its
275	leases expire (the TTL — proven live at the original 30 s TTL, since raised
276	to 15 m for S3 request cost), data intact in both.
277
278	Hardening that same week (PR #681, both lessons from the first real
279	production flip): flipping a host refuses while the box-wide backup
280	replicator still tracks it — two litestream processes on one DB share a
281	derived meta directory and corrupt each other — and the lease release runs
282	under its own budget so a wedged final checkpoint can't strand the lease
283	until TTL.
284
285	### The seams cut for what's next
286
287	Small PRs that exist so later phases are config, not surgery:
288
289	- **Attachments** sit behind a `blobStore` interface (`blobstore.go`, PR
290	#636): write-once immutable bytes, no CAS or versioning needed — sized and
291	thread-tagged in SQLite, ciphertext in the store. Disk is implementation #1;
292	the object-store implementation (R2 for zero-egress media, per the issue) is
293	roadmap.
294	- **The cert cache** is behind `autocert.Cache` (`certcache.go`, PR #661):
295	`LCHAT_CERT_CACHE` unset/`dir` is today's `/data/acme` layout; `s3` is
296	reserved for the pooled-router future and **refuses to start** until it
297	exists — failing loudly beats minting certs into a local dir the rest of a
298	pool can't see.
299	- **Placement fields**: the registry row carries `home_region` and `provider`
300	(PR #625), single-valued and inert today — the geo (Part 7) and co-op
301	(Part 10) seams, as schema.
302	- **Billing rails** (PR #628): the cloud ledger's spaces table holds exactly
303	Part 11's shape — `stripe_customer_id` (an opaque pointer into Stripe) and
304	an `entitlement` flag — no email, name, or card detail anywhere in our
305	stores (identity lives in Stripe, where the operator can see it). Dormant
306	until the Stripe env is set.
307	- **The release-feed mirror** (PR #662, `deploy/release-server.sh` +
308	`updater.go`): the fleet self-updates from GitHub, whose CDN has served
309	stale manifests — the Part 9 SPOF. Publishing now mirrors the signed feed
310	and binaries to S3 (`LCHAT_RELEASE_MIRROR_BUCKET`), and updaters fall back
311	(`LCHAT_RELEASE_MIRROR_URL`) when the primary fails *or verifies but is
312	stale* against the newest feed the box ever accepted. Signature verification
313	is identical on both paths: the mirror adds availability, never trust.
314	- **The secrets inventory** (`docs/secrets.md`, PR #639): every server-side
315	secret, blast radius, and rotation story — Part 12's operability item 11.
316	- **Infrastructure as code** (`infra/`, PRs #609, #637, #638, #658): OpenTofu,
317	managing the static skeleton only — dev and prod as separate states. Dev
318	holds the proof-build buckets and a `t4g.small` box; prod holds the
319	fleet-backup and hibernation buckets (versioning, lifecycle, Block Public
320	Access, scoped IAM users whose keys never enter tofu state) plus CloudTrail.
321	The boundary test from the issue holds and is written into `infra/README.md`:
322	**if the IaC needs to know an instance hostname, the design is wrong.**
323
324	## Decision log
325
326	Part 12's table, one line per settled call, updated with what shipped. "Held"
327	means the decision stands with nothing built against it yet.
328
329	| # | decision | chosen | status |
330	|---|---|---|---|
331	| 1 | home store | S3 as the only database (over R2-sole / DynamoDB) | **Shipped** #611/#627 — dual backend, SQLite stays the self-hoster default; minimal SigV4 client, not the SDK |
332	| 2 | home runtime | stateless Go binary, Lambda-able later (over a Workers rewrite) | **Shipped** #613 — statelessness done; Lambda step not taken |
333	| 3 | multi-region writes | single write region + CRR warm standby (over active-active, which breaks per-region CAS) | Held — single region today, no CRR yet |
334	| 4 | metadata at rest | sealed bucket: HMAC(pepper) keys + pepper-encrypted metadata | **Not built** — object keys are plain uids; bucket stays private (Block Public Access) |
335	| 5 | public bucket | design as if public; the flip is its own reviewed decision (takedown process prerequisite) | Held — posture unchanged |
336	| 6 | instance runtime | hibernating processes over S3: local SQLite + Litestream, lease = single-writer lock (over Lambda / S3-as-only-DB / cells / renting DO-Fly) | **Shipped** #630/#641 — with the correction: the lease is Go-library-only; the CLI silently absorbs a second writer, so the activator embeds litestream |
337	| 7 | placement | rendezvous hash as hint, S3 lease as truth (over a routing table) | **Shipped** (#678) — S3-heartbeat membership + HRW + proxy-to-owner; kill-a-box passes live both ways. **Live in production 2026-07-13** across two boxes: domain-scoped placement (#820), automatic route replication + `mesh-seed` (#821), and fleet-wide status aggregation partitioned by ownership (#822/#849/#851/#852) |
338	| 8 | instance data | one SQLite per space, forever (shared DB tier is the anti-goal) | Held, load-bearing |
339	| 9 | attachments | blob interface → R2 from day one | Seam **shipped** #636; disk impl only, R2 roadmap |
340	| 10 | provisioning | create = register only; materialize at first wake; controld dissolves | Split **shipped** #625, hibernate-by-default creates #663; full deferral + controld dissolution roadmap |
341	| 11 | bootstrap | thin golden AMI + `lchat bootstrap`; release channel owns versions | `bootstrap` + channels predate #550; AMI roadmap |
342	| 12 | migration posture | dual plane — Hetzner as-a-self-hoster + Eleven Cloud on AWS; origin-preserving moves | Held — production is the Hetzner plane; AWS holds buckets + a dev box |
343	| 13 | custom domains | CNAME → `<sqid>.11ch.at`, per-host autocert, cert cache → S3 | Cache seam **shipped** #661 (`s3` reserved, refuses); rest roadmap |
344	| 14 | IaC | OpenTofu, static skeleton only (over Terraform/Pulumi/CFN) | **Shipped** #609/#637/#638/#658 |
345	| 15 | capacity | ASG target-tracking on warm-set RAM; one shared drain path; Spot slice | Roadmap |
346	| 16 | geography | per-space `home_region`; relocate while hibernated | Field **shipped** #625, single-valued and inert; behavior roadmap |
347	| 17 | push relay | small pool of long-lived processes; self-serve relay tokens | Relay predates #550; self-serve registration roadmap |
348	| 18 | co-op | commons/provider seam; delegation unit = the space; ciphertext escrow | `provider` field **shipped** #625, inert; rest roadmap |
349	| 19 | billing | Stripe holds all customer data; our record = opaque pointer + entitlement flag; public ledger | Rails **shipped** #628 (dormant until env set); public ledger roadmap (`docs/tier-economics.md` is the cost record) |
350
351	Corrections the build taught us, beyond the table: the lease-is-library-only
352	finding (#6 above); env-file ownership must follow the reader (root-written
353	files, service-user activator — a wake died on it live); a never-woken
354	hibernating space has no DB, so reaper stats answer from the registry row or
355	register-only spaces become immortal; keep-last-N on S3 is an interface cap
356	with the lifecycle as backstop, not a bucket guarantee; `errBadVersion` is a
357	SQLite-only nicety S3 cannot express; and the first live mesh flip (#7,
358	2026-07-13) taught that a replicated route is a registry row only — it never
359	stages a binary or creates a `live` symlink, and the sandboxed router
360	(`ProtectSystem=strict`) cannot do that itself, so a replicated route's first
361	wake 503ed outright until the wake path was rerouted through the unsandboxed
362	control daemon (#844) instead of writing `/opt/lchat` directly.
363
364	## The invariants — the review checklist
365
366	Part 12's twelve lines, refined where building sharpened them. Every future
367	PR that touches this territory gets checked against these:
368
369	1. The server — and any future *provider* — only ever holds ciphertext plus
370	auth plumbing; never a usable key. (The root rule; everything below is it,
371	applied.)
372	2. No cross-user state in home — the zero-join property *is* the scalability.
373	3. Never consolidate spaces into a shared DB tier — per-space isolation is the
374	sharding, security, and migration story at once.
375	4. DB migrations are additive-only — this is what makes wake-onto-new-binary
376	and channel-following safe.
377	5. The IaC layer never learns an instance hostname.
378	6. Wire contracts stay backend-opaque — clients never learn (or depend on) the
379	storage backend; versions are opaque strings, `"0"` the one sentinel.
380	7. The wildcard TLS private key never crosses a trust domain.
381	8. A space's hostname survives every migration — re-keying is never the cost
382	of moving (region, plane, or provider).
383	9. The bot sidecar remains a pure outbound client and bot code never runs in
384	the server process — co-spawn makes it a router *child*, but still a
385	separate process whose traffic is marked, activity-invisible, and never a
386	wake source.
387	10. No cross-provider protocol, ever — spaces are islands; if a feature seems
388	to need one, the design is wrong.
389	11. The 18 KiB vault pad/cap and the no-per-user-analytics / no-email absences
390	move only by deliberate decision.
391	12. No PII in our stores, ever — customer identity lives in Stripe; we hold
392	opaque pointers and entitlement flags. Income publishes aggregated;
393	expenses and disbursements publish itemized.
394
395	## Roadmap — designed, not built
396
397	Everything below is settled design from #550 with no code behind it (or only
398	a seam). Build triggers are events, not dates.
399
400	- **Multi-region / `home_region`** — one write region per space, wakes
401	deterministic, relocation *while hibernated* (copy the prefix, flip the
402	field, next wake claims the lease elsewhere). The field exists; nothing
403	keys on it. Trigger: member-latency data from a real non-EU/US cohort. The
404	full design that builds on this — region as a *mutable property* (not encoded
405	in the address), stable-address relocation, and multi-edge replication, with
406	the append-only/CRDT reasoning that makes it tractable — is
407	**`docs/multi-edge.md`** (agreed direction, 2026-07-18).
408	- **The co-op / provider seam** — `provider` as a real interface with
409	ourselves as provider #1, the Hetzner box formalized as #2, escrow replicas
410	in storage the provider doesn't control. Federation without a federation
411	protocol.
412	- **Lambda-able home / autoscaling runtime** — statelessness (the
413	prerequisite) is done; the same binary on an autoscaling platform is the
414	next rung, an edge rewrite deliberately skipped.
415	- **Sealed bucket, then the public-bucket decision** — pepper-keyed object
416	names and pepper-encrypted metadata first (hardens the private design too);
417	flipping public-read stays its own reviewed decision with a written
418	takedown process as prerequisite.
419	- **Attachments to an object store** — the `blobStore` seam is cut; R2 for
420	zero-egress media is the intended implementation #2.
421	- **S3 cert cache** — `LCHAT_CERT_CACHE=s3` is reserved and refusing; needed
422	the day two boxes share issuance.
423	- **CRR + second-provider mirror** — the region checkbox and the Ente-style
424	read-only mirror; everything is ciphertext, so copies are threat-model-free.
425	- **The AWS front plane** — NLB (TCP passthrough), ASG on warm-set RAM, Spot
426	slice, golden AMI: Part 7's ~$35/mo shape. The account, state buckets, and
427	IAM posture exist; the compute plane doesn't. Trigger: warm-RAM pressure or
428	a launch date.
429	- **Observability gauges** — wake latency, lease conflicts, replication lag
430	are loud log lines today, not metrics; Part 9's E2EE-safe metrics discipline
431	is still owed.
432	- **The public ledger** — Part 11's append-only books repo. Partially
433	shipped: the public *status dashboards* exist (fleet state + aggregated
434	Stripe income, published from the box every 5 minutes — `docs/status.md`);
435	the itemized expense/disbursement ledger repo is still to come.
436
437	## The historical record
438
439	The full twelve-part design conversation — including the superseded "cells"
440	Part 4, the cost math, the rejected alternatives in their original words, and
441	the lease experiment writeup — is issue
442	[#550](https://github.com/elevenmessenger/messenger/issues/550). Related
443	docs: `docs/instance-router.md` (the router this grew inside),
444	`docs/cloud-provisioner.md` (create-a-space), `docs/server-releases.md` (the
445	signed feed + mirror), `docs/secrets.md` (the key inventory),
446	`infra/README.md` (the IaC skeleton), `docs/bots.md` / `docs/member-bots.md`
447	(the sidecar model co-spawn inherits).
448
449	One sentence, carried over intact because it survived contact with the build:
450	**the E2EE rule did the hard part everywhere — home scales by removing the
451	database, instances scale by removing the idle process, the fleet scales by
452	removing the operator, the co-op scales by removing us — and the books
453	balance in public.**
454