| new file mode 100644 |
| index 0000000..0c5d303 |
| --- /dev/null |
| +++ b/docs/comparison-infrastructure.md |
| @@ -0,0 +1,150 @@ |
| +# 🤖 How the infrastructure compares |
| + |
| +Where the Eleven *infrastructure* sits among prior art, honestly — the |
| +sibling of [comparison.md](comparison.md), which compares the product. |
| +The architecture in question is the |
| +[#550 plan](https://github.com/elevenmessenger/messenger/issues/550): |
| +one fully-isolated single-binary instance per space, hibernating when idle, |
| +state parked in S3 as ciphertext, woken on demand by an activator mesh that |
| +places instances by lease + rendezvous hashing — no orchestrator, no |
| +control plane. |
| + |
| +The short version: **every mechanism has shipped somewhere, and two systems |
| +are nearly the same shape — but nobody has shipped this combination.** The |
| +reasons are datable: the load-bearing primitives mostly didn't exist before |
| +2021–2024, and everyone big enough to build it had business incentives |
| +pointing the other way. The people who *did* build the pattern sold it as a |
| +rentable platform instead of putting it inside a product. |
| + |
| +## At a glance |
| + |
| +| | Durable Objects | Sandstorm (2014) | Orleans / Erlang | Fly Machines | **Eleven (#550)** | |
| +|---|---|---|---|---|---| |
| +| Unit | JS actor + private SQLite | per-document container ("grain") | in-process actor over a shared store | micro-VM | **OS process + own SQLite** | |
| +| Wakes on demand / sleeps idle | Yes (incl. WebSocket hibernation) | Yes | Yes (activate/deactivate) | Yes (stop/start) | **Yes** | |
| +| Durability layer | Cloudflare's storage | local disk | pluggable shared store | volume (+ Litestream) | **S3, continuous replication** | |
| +| Placement | Cloudflare's routing layer | single server | directory service | Fly's scheduler | **S3 lease is truth; HRW hash is the hint — no directory** | |
| +| Self-hostable | No | Yes (its whole point) | Yes (a framework) | No | **Yes — same binary as the flagship** | |
| +| State is ciphertext to the operator | No | No | No | No | **Yes — the E2E rule** | |
| + |
| +## The nearest neighbours |
| + |
| +**[Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/)** |
| +is almost isomorphic: a per-entity actor addressed by name, woken on |
| +request, hibernating when idle — including WebSocket hibernation, where |
| +connections stay open while the object sleeps — each with its |
| +[own private SQLite database](https://blog.cloudflare.com/sqlite-in-durable-objects/) |
| +(2024). "A chat room per object" is literally Cloudflare's canonical demo, |
| +and Cloudflare Workflows V2 proved the density story at tens of thousands |
| +of concurrent SQLite-backed instances. What it isn't: self-hostable, a |
| +normal OS process, or E2EE — it's a JS runtime inside Cloudflare's control |
| +plane. Renting it would put the core mechanic on someone else's platform |
| +and fork the self-hosted shape, which is why #550 rejects it by name. |
| + |
| +**[Sandstorm](https://sandstorm.io/) (2014)** was the self-hosted version, |
| +a decade too early. Grains — per-document isolated containers, spun up on |
| +first request, suspended when idle, capability-secured — are the closest |
| +anyone came to "the unit of the system is a tiny personal server that |
| +hibernates." The company wound down around 2017 with essentially no |
| +revenue: containers were heavy, every app had to be ported into its |
| +sandbox, and none of the enablers below existed yet. The instructive coda: |
| +its founder, [Kenton Varda](https://github.com/kentonv), went to Cloudflare |
| +and built Workers and Durable Objects. The grain idea didn't fail — it got |
| +captured into a proprietary platform. |
| + |
| +**[Orleans](https://learn.microsoft.com/en-us/dotnet/orleans/overview) and |
| +Erlang/OTP** are the 15–40-year-old theory. Orleans "virtual actors" |
| +(Microsoft Research 2011, Halo's backend by 2012) activate on demand, |
| +deactivate when idle, and place across a cluster automatically. The |
| +difference: actors are code objects over a *shared* store with a directory |
| +service; Eleven's unit is a whole isolated process with its own database, |
| +and placement needs no directory because the S3 lease is the truth. The |
| +activator itself descends from `inetd` (1986) — wake-on-connection is the |
| +oldest trick in Unix, and 90s shared PHP hosting was economically |
| +process-per-site scale-to-zero. |
| + |
| +**[Fly.io](https://fly.io/)** built the tooling and evangelized the |
| +pattern — Machines that stop when idle and wake on request, plus |
| +[Litestream](https://fly.io/blog/all-in-on-sqlite-litestream/), whose |
| +author it hired to push server-side SQLite — explicitly pitching "one app |
| +per customer." Again: rented placement. |
| + |
| +**The S3-as-the-only-disk wave is happening right now** — WarpStream, |
| +SlateDB, turbopuffer, [Turso's](https://turso.tech/) database-per-tenant, |
| +Litestream-under-workflow-instances. The storage layer of #550 isn't |
| +swimming against the current; it's riding a 2024–2026 wave that only became |
| +possible when S3 shipped conditional writes. |
| + |
| +**Product-shape neighbours:** [Snikket](https://snikket.org/) |
| +(instance-per-family XMPP, invite links) and 37signals' |
| +[ONCE Campfire](https://once.com/) (chat sold as one self-hosted unit per |
| +group) share the small-instance-per-group philosophy — but neither |
| +hibernates, meshes, or offers a planet-scale hosted plane. |
| +[Tarsnap](https://www.tarsnap.com/), [Ente](https://ente.io/), and |
| +Bitwarden pioneered client-encrypted-blobs-in-a-bucket, the storage half of |
| +the trust model. |
| + |
| +## What only the combination gives you |
| + |
| +1. **E2EE as an infrastructure enabler, not a feature.** Everyone else |
| + doing hibernation-over-object-storage has to encrypt at rest, manage |
| + keys, and trust the bucket. Because an instance's state is already |
| + ciphertext by [the one rule](../README.md), parking it in S3 is |
| + threat-model-free — the crypto design and the infra design pay for each |
| + other. Nobody has built the hibernation pattern *on top of* a |
| + ciphertext-only invariant. |
| +2. **The self-hosted unit and the planet-scale unit are the same binary in |
| + the same shape.** Durable Objects and Fly are platforms you rent; |
| + Sandstorm and Snikket were self-hosted with no flagship-scale story. |
| + "One box with no S3 is the degenerate case of the mesh" as a design |
| + invariant appears to be new. |
| +3. **No control plane.** Orleans has a directory, Durable Objects has |
| + Cloudflare's routing layer, Kubernetes *is* a control plane. |
| + Lease-as-truth + rendezvous hashing keeps placement entirely inside the |
| + app — the IaC layer never learns an instance hostname. |
| + |
| +## Why hasn't this been done before? |
| + |
| +**The primitives are datable, and they're recent.** |
| + |
| +| primitive | arrived | what it unlocks | |
| +|---|---|---| |
| +| S3 conditional writes (`If-None-Match` / `If-Match`) | Aug / Nov 2024 | the lease — a distributed single-writer lock with no DynamoDB/ZooKeeper | |
| +| Litestream (v0.5 leases) | 2021 (2025) | continuous SQLite→S3 replication; restore-anywhere | |
| +| Wildcard ACME via DNS-01 | 2018 | per-instance TLS at zero marginal cost | |
| +| Pure-Go SQLite + `go:embed` | ~2021 | the one-file, no-cgo, no-build-step binary | |
| +| Passkeys with PRF | 2023+ | a keyless, stateless home server | |
| +| Server-side SQLite respectability | ~2022–2024 | Fly, Tailscale, then Rails 8 defaulting to it | |
| + |
| +Anyone attempting this stack in 2019 would have had to build four missing |
| +layers first — which is roughly what killed Sandstorm. |
| + |
| +**The incentive gradient pointed the other way for everyone capable of |
| +it.** The standard SaaS playbook wants one big multi-tenant database: it |
| +maximizes density and enables everything the business model runs on — |
| +search, analytics, growth loops, and (for messengers) the global social |
| +graph. E2EE-plus-isolation forecloses all of that *on purpose*. Signal |
| +centralized; Meta needs the graph; enterprise vendors sell isolation as |
| +expensive "dedicated instances," not free family chats. Nobody's revenue |
| +model demanded a million tiny free isolated servers costing fractions of a |
| +cent each — which is exactly why the ones who built the mechanics |
| +(Cloudflare, Fly) monetized them as infrastructure-for-rent rather than |
| +shipping a product on them. |
| + |
| +**And a team-economics reason that's only just dissolving:** a homegrown |
| +activator mesh instead of Kubernetes used to be a staffing decision — |
| +bespoke placement layers were for Google-shaped orgs. A tiny team choosing |
| +"we'll write our own orchestration inside the app binary" is newly sane in |
| +the LLM era, which may be the quietest reason this combination is appearing |
| +now and not in 2020. |
| + |
| +## Sources |
| + |
| +- [Zero-latency SQLite storage in every Durable Object (Cloudflare)](https://blog.cloudflare.com/sqlite-in-durable-objects/) |
| +- [I'm All-In on Server-Side SQLite (Fly)](https://fly.io/blog/all-in-on-sqlite-litestream/) |
| +- [Orleans virtual actors (Microsoft)](https://learn.microsoft.com/en-us/dotnet/orleans/overview) |
| +- [Sandstorm](https://sandstorm.io/) · [Kenton Varda](https://github.com/kentonv) |
| +- [SQLite durable workflows over Litestream](https://byteiota.com/sqlite-durable-workflows-skip-temporal/) · [turbolite — SQLite VFS over S3](https://github.com/russellromney/turbolite) |
| +- The #550 plan itself: [home: use an object store (S3) as the only database — planet-scale plan](https://github.com/elevenmessenger/messenger/issues/550) |
| + |
| +*Snapshot as of July 2026 — this field is moving fast; corrections welcome.* |
amadan