| new file mode 100644 |
| index 0000000..d1de754 |
| --- /dev/null |
| +++ b/docs/planet-scale.md |
| @@ -0,0 +1,393 @@ |
| +# 🤖 Planet scale — the running system |
| + |
| +This is the distillation of issue **#550** ("use an object store as the only |
| +database — planet-scale plan", twelve parts across two working sessions). |
| +Most of what that thread designed has since shipped; this doc describes the |
| +system as it **runs**, with the still-unbuilt parts clearly marked as roadmap. |
| +Where the issue and the code differ, the code wins, and the difference is |
| +noted. #550 stays open only as the historical record of how we got here and |
| +closes when this merges. |
| + |
| +## The one idea |
| + |
| +The E2EE rule did the hard part before any of this started. Home's server-side |
| +state is opaque client-encrypted ciphertext plus a few kilobytes of public-key |
| +metadata, with zero cross-user queries — so home scales by **removing the |
| +database**: an S3 bucket, used raw, is a strictly better fit than SQLite ever |
| +was. An instance's SQLite file is opaque blobs plus routing metadata the server |
| +already sees — so replicating it to a bucket is threat-model-free, and |
| +instances scale by **removing the idle process**: state parks in S3 as |
| +ciphertext, a process exists only while someone's talking, and creating a |
| +space becomes (nearly) a metadata write. Launch-day load lands on S3's |
| +capacity, not ours. |
| + |
| +## What runs today |
| + |
| +### Home: the store is a seam, and S3 is implementation #2 |
| + |
| +`home/store.go` (PR #611) extracts **`vaultStore`**, home's whole persistence |
| +surface, shaped like an object store on purpose: point reads by key, |
| +conditional writes (CAS on an *opaque version string*, with the one universal |
| +sentinel `"0"` = "no blob yet"), keep-last-N blob versions, prefix deletes, |
| +and write-once pointers. SQLite is implementation #1 and stays the zero-config |
| +self-hoster default forever — dual backend is a feature, not migration debt. |
| + |
| +`home/s3store.go` (PR #627) is implementation #2: **one bucket, no database**, |
| +laid out exactly as the issue drew it — |
| + |
| +``` |
| +blobs/<uid> the vault; S3-native versioning, version = the ETag |
| +users/<uid> one metadata JSON per user (credentials + devices), |
| + every mutation an If-Match CAS loop |
| +creds/<credId> write-once pointer, body = uid |
| +tokens/<sha256> write-once pointer, body = uid |
| +``` |
| + |
| +CAS rides S3 conditional writes (`If-Match` to replace; the `"0"` sentinel |
| +maps to `If-None-Match: *` for create-only). Pointers are never authoritative: |
| +resolving one re-loads `users/<uid>` and verifies the credential/token is |
| +still listed, so the stale pointer a revocation leaves behind denies by |
| +construction. Two open questions from the issue were settled at PR time, |
| +both worth recording: |
| + |
| +- **The client is a hand-rolled SigV4 signer** (`home/sigv4.go`), not the AWS |
| + SDK — the surface home needs is six calls, and home keeps its tiny dependency |
| + footprint. (The *activator* side, below, does carry `aws-sdk-go-v2`, brought |
| + in by litestream — the two made opposite calls for good local reasons.) |
| +- **Keep-last-5 is enforced at the interface, not by the bucket**: |
| + `listBlobVersions` caps at five; the bucket's lifecycle rule prunes |
| + noncurrent versions asynchronously, days later, as a backstop. And S3 cannot |
| + distinguish "a version string that was never ours" from a plain mismatch, so |
| + the SQLite backend's `errBadVersion` (→ 400) is deliberately backend-specific; |
| + on S3 both read as conflicts (→ 409). |
| + |
| +Selection is by env (`home/main.go`): `LCHAT_HOME_STORE` unset or `sqlite` is |
| +the default; `s3` requires `LCHAT_HOME_S3_BUCKET` + `LCHAT_HOME_S3_REGION` |
| +plus AWS credentials in the standard env vars, with optional |
| +`LCHAT_HOME_S3_PREFIX` (test isolation) and `LCHAT_HOME_S3_ENDPOINT` |
| +(path-style, for S3-compatible stores). Misconfiguration fails at boot, never |
| +limps. |
| + |
| +The wire contract was already ETag-opaque: PR #612 made the blob `version` a |
| +JSON string on the wire (clients tolerated both) and coarsened link-token |
| +`last_seen` to daily, so an object-store backend never pays a write per |
| +authenticated read. |
| + |
| +### Home is stateless |
| + |
| +`home/stateless.go` (PR #613) removed the last two things a home process |
| +remembered between requests. Ceremony proofs became **HMAC-signed tokens** |
| +(uid + expiry, 5-minute window) the client carries; in-flight WebAuthn |
| +sessions are **AES-GCM-sealed into the `?session=` query param**. Both keys |
| +derive from one server secret, `LCHAT_HOME_SESSION_KEY` — auth plumbing like |
| +the admin token, it cannot open a byte of any vault. Unset, an ephemeral key |
| +is minted at boot (with a logged warning): single-process self-hosters keep |
| +working with zero config, and ceremonies just don't survive a restart, which |
| +is exactly what the old in-memory map gave them. With the bucket backend and |
| +the stateless ceremonies together, home is a pure function from |
| +(request, bucket) → response — any replica behind a load balancer can finish |
| +what another began. |
| + |
| +### Fleet backup: replication as a property of normal operation |
| + |
| +`deploy/fleet-backup.sh` (PRs #637, #643) closed the actual gap the plan |
| +started from — instance DBs living on one box with no off-box copy. It runs on |
| +the router box, idempotent, regenerating `/etc/litestream.yml` from what's in |
| +`/data`: a CLI **Litestream** (0.5.14) streams every `/data/*.db` — instances |
| +plus home/push/cloud/registry — to the fleet-backup bucket |
| +(`eleven-prod-fleet-backup-…`, one prefix per database), under the |
| +`lchat-litestream` unit and a scoped IAM user that can touch objects but never |
| +versioning, lifecycle, or the bucket itself. Versioning + a 14-day noncurrent |
| +window is the ransomware posture: an overwrite or delete with the box's own |
| +credentials leaves recoverable versions behind. Continuous replication, not |
| +snapshots — seconds of RPO, point-in-time recovery, and it supersedes the |
| +parked R2-snapshot (`r2d2.sh`) plan entirely. The restore drill is one |
| +command, documented in the script's header; run it against a scratch path, |
| +never over a live file. |
| + |
| +One exclusion, added when hibernation went live (PR #658): **hibernating hosts |
| +are skipped** — the activator's embedded Litestream owns their replication (to |
| +the instances bucket, under the lease), and two replicators on one WAL would |
| +fight over checkpointing. |
| + |
| +### Instance hibernation, end to end |
| + |
| +This is Part 4 v2 of the issue, running. Four layers: |
| + |
| +**The lease** (`internal/activator/replica.go`, PR #630). Litestream is |
| +embedded **as a Go library** — pure `modernc.org/sqlite`, no cgo, so the |
| +single-binary property survives. Its S3 lease (`lock.json` acquired via |
| +conditional writes) is the distributed single-writer lock: a second claimant |
| +fails loudly with a typed `*litestream.LeaseExistsError`. This is a |
| +correction the plan earned by testing (recorded on #550): the issue assumed |
| +the lease was CLI config, but **the lease exists only in the library API — the |
| +CLI silently absorbs a second replicator**, orphaning the loser's writes with |
| +no error signal. Embedding it turned that from a footgun into the mechanism. |
| +TTL is 30 s, renewed every 10 s; a failed renewal means leadership is lost and |
| +the holder stands down — a wedged process is claimable by a healthy peer |
| +within ~30 s, which is the failover window of the whole design. Restores run |
| +*under* the held lease (restoring outside it could resurrect a stale copy that |
| +shadows the true holder's newer state). |
| + |
| +**The activator** (`activator.go`, PR #641). With `LCHAT_HIBERNATE_BUCKET` |
| +set, the router stops relying on systemd to keep flagged instances alive and |
| +runs them as its **own children, on demand** — "units are for the router, |
| +instances are its children", exactly as the issue put it. The per-host state |
| +machine is asleep → waking → awake → stopping: |
| + |
| +- **Wake** (a request for a host with no live process): claim the per-host |
| + lease at `<prefix>/<host>/lock.json`; restore the DB from S3 only if the |
| + local copy is gone (a present local DB is the warm cache — on one box, under |
| + the lease, it can only be our own last checkpoint); spawn the instance |
| + (exec of the same versioned `live/<host>` binary its systemd unit would run, |
| + same env file); start replication under the lease. Concurrent requests |
| + coalesce on one wake (singleflight); lease contention and wake timeouts are |
| + 503 + `Retry-After` — "sleeping spaces are slow to wake", never a 5xx |
| + cascade. |
| +- **Sleep** (idle past `LCHAT_HIBERNATE_IDLE`, default 15 m — idleness is |
| + in-flight-count *and* time, so one open WebSocket holds an instance awake): |
| + probe the next due time, SIGTERM the child (its own graceful drain is the |
| + final write), final checkpoint to S3, release the lease, forget the entry. |
| + The local DB stays as the warm cache. |
| +- **Failure is the wake path**: a dead child or a lost lease stands the entry |
| + down, and the next request re-wakes it. S3 blips leave warm instances |
| + serving (local SQLite is authoritative; replication queues and drains) — |
| + only cold wakes stall. |
| + |
| +**Due-time wakes** (PR #653). Scheduled deliveries (future messages, |
| +reminders) fire on time while their instance sleeps: at sleep the stop path |
| +asks the instance for its earliest pending delivery (`GET /api/next-due`, |
| +admin-gated, over the socket) and records it in the registry |
| +(`routes.next_due` — persisted, so it survives router restarts); the sweeper |
| +wakes any sleeping host whose time has arrived, no client request needed. |
| +Worst-case lateness ≈ one sweep interval (≤30 s) + spawn + one scheduler tick |
| +(≤5 s). One honest gap the issue glossed: **member-bot charter schedules run |
| +in the sidecar** (`docs/member-bots.md`), so they don't wake a sleeping |
| +instance — the index covers server-side schedules only. |
| + |
| +**Sidecar co-spawn** (PR #670). A host with a bot sidecar staged |
| +(`/data/<host>-bot.env` — the same file `lchat-bot@.service` reads) gets its |
| +`bot run` child spawned right after the instance turns healthy and SIGTERMed |
| +first at sleep: the sidecar exists exactly while its instance is awake, so its |
| +lifecycle solves itself — the issue's "co-located over loopback" default, |
| +built. The subtlety that makes it work: the sidecar dials its instance's |
| +public URL back through this very router, so it stamps every request |
| +(`X-Lchat-Sidecar`, `internal/bot/sidecarmark.go`) and the activator makes |
| +that traffic **activity-invisible** — no idle-clock credit while awake, a 503 |
| +(never a wake) while asleep. Without that, the pair could never sleep and an |
| +orphaned sidecar's reconnect loop would wake its instance right back, forever. |
| +Spoofing the marker only downgrades the sender's own requests. Supervision is |
| +sweep-driven Restart=always with capped exponential backoff; a sidecar that |
| +won't start never fails the wake — members before bots. |
| + |
| +### The flip, per host, and creates that are born asleep |
| + |
| +Opt-in is **per host** (`routes.hibernate` in the registry, PR #659), which is |
| +what makes a production canary safe: `LCHAT_HIBERNATE_BUCKET` ships fleet-wide |
| +as a no-op, then |
| + |
| +```sh |
| +lchat hibernate <host> --on # stop+disable its unit; the router wakes it per request |
| +lchat hibernate <host> --off # activator releases (checkpoint + lease back), unit restarts |
| +lchat instances # HIBERNATE column shows who's activator-managed |
| +``` |
| + |
| +migrates exactly one instance, and back. The flip's ordering is deliberate |
| +(flag → stop unit → reload router → warm request) so the host is never left |
| +unserveable, and `--off` waits for the child to actually release the socket |
| +before the unit starts (an instance refuses to clobber a live listener). A |
| +staged bot sidecar hands over both ways in the same flip. |
| + |
| +One lesson found live, now baked in: **env-file ownership.** Systemd reads a |
| +root-written `EnvironmentFile` as root; the activator reads it as the service |
| +user — the canary's first wake died on permission denied. `add-instance` and |
| +both flips now chown the env (and bot session dirs) to match the data dir. |
| + |
| +Creation splits into **register** (the row: host, minted-or-imported token, |
| +placement fields — pure metadata, no capacity) and **materialize** (env file, |
| +staged binary, unit, health check, router reload, setup invite) — PR #625, |
| +exactly the seam the issue asked for. A `--hibernate` create (and, behind |
| +`LCHAT_CONTROL_HIBERNATE_CREATES=1` on the control daemon, **every |
| +provisioner-created space** — PR #663) skips only the systemd half: no unit, |
| +no resident process, no DB — the first wake finds no replica in S3 and boots |
| +fresh. Capacity for hibernating rows is a row ceiling |
| +(`LCHAT_MAX_REGISTERED`, default 10× `LCHAT_MAX_INSTANCES`), not a process |
| +cap. Honest delta from the issue's "create = a registry row + an S3 prefix, |
| +microseconds, no box contacted": today's hibernating create still writes the |
| +env file, stages the binary, reloads the router — and its readiness probe *is* |
| +the first wake, because minting the setup invite needs a running instance to |
| +pre-seed the admin user. Cheap, but not yet free; the fully-deferred |
| +materialization remains roadmap. |
| + |
| +The **reaper** got the property Part 5 promised — reclaiming never-claimed |
| +spaces **without waking anything**. A hibernating space's stats |
| +(`setup_complete` / members / `has_content`) are read straight off its local |
| +DB file, read-only — wake-free because the warm cache is the newest truth on |
| +one box. The case the code had to add: a hibernating space that has *never |
| +woken* has no DB at all, and before the fix that read as a stats error — and |
| +a stats error is a skip-never-reap, so register-only spaces would have been |
| +immortal. The stats handler now answers those **definitionally from the |
| +registry row** (0 members, never set up, no content); a missing DB on a |
| +*non*-hibernating row stays a 404, meaning a failed create, skipped as before. |
| + |
| +### The seams cut for what's next |
| + |
| +Small PRs that exist so later phases are config, not surgery: |
| + |
| +- **Attachments** sit behind a `blobStore` interface (`blobstore.go`, PR |
| + #636): write-once immutable bytes, no CAS or versioning needed — sized and |
| + thread-tagged in SQLite, ciphertext in the store. Disk is implementation #1; |
| + the object-store implementation (R2 for zero-egress media, per the issue) is |
| + roadmap. |
| +- **The cert cache** is behind `autocert.Cache` (`certcache.go`, PR #661): |
| + `LCHAT_CERT_CACHE` unset/`dir` is today's `/data/acme` layout; `s3` is |
| + reserved for the pooled-router future and **refuses to start** until it |
| + exists — failing loudly beats minting certs into a local dir the rest of a |
| + pool can't see. |
| +- **Placement fields**: the registry row carries `home_region` and `provider` |
| + (PR #625), single-valued and inert today — the geo (Part 7) and co-op |
| + (Part 10) seams, as schema. |
| +- **Billing rails** (PR #628): the cloud ledger's spaces table holds exactly |
| + Part 11's shape — `stripe_customer_id` (an opaque pointer into Stripe) and |
| + an `entitlement` flag — no email, name, or card detail anywhere in our |
| + stores. Dormant until the Stripe env is set. |
| +- **The release-feed mirror** (PR #662, `deploy/release-server.sh` + |
| + `updater.go`): the fleet self-updates from GitHub, whose CDN has served |
| + stale manifests — the Part 9 SPOF. Publishing now mirrors the signed feed |
| + and binaries to S3 (`LCHAT_RELEASE_MIRROR_BUCKET`), and updaters fall back |
| + (`LCHAT_RELEASE_MIRROR_URL`) when the primary fails *or verifies but is |
| + stale* against the newest feed the box ever accepted. Signature verification |
| + is identical on both paths: the mirror adds availability, never trust. |
| +- **The secrets inventory** (`docs/secrets.md`, PR #639): every server-side |
| + secret, blast radius, and rotation story — Part 12's operability item 11. |
| +- **Infrastructure as code** (`infra/`, PRs #609, #637, #638, #658): OpenTofu, |
| + managing the static skeleton only — dev and prod as separate states. Dev |
| + holds the proof-build buckets and a `t4g.small` box; prod holds the |
| + fleet-backup and hibernation buckets (versioning, lifecycle, Block Public |
| + Access, scoped IAM users whose keys never enter tofu state) plus CloudTrail. |
| + The boundary test from the issue holds and is written into `infra/README.md`: |
| + **if the IaC needs to know an instance hostname, the design is wrong.** |
| + |
| +## Decision log |
| + |
| +Part 12's table, one line per settled call, updated with what shipped. "Held" |
| +means the decision stands with nothing built against it yet. |
| + |
| +| # | decision | chosen | status | |
| +|---|---|---|---| |
| +| 1 | home store | S3 as the only database (over R2-sole / DynamoDB) | **Shipped** #611/#627 — dual backend, SQLite stays the self-hoster default; minimal SigV4 client, not the SDK | |
| +| 2 | home runtime | stateless Go binary, Lambda-able later (over a Workers rewrite) | **Shipped** #613 — statelessness done; Lambda step not taken | |
| +| 3 | multi-region writes | single write region + CRR warm standby (over active-active, which breaks per-region CAS) | Held — single region today, no CRR yet | |
| +| 4 | metadata at rest | sealed bucket: HMAC(pepper) keys + pepper-encrypted metadata | **Not built** — object keys are plain uids; bucket stays private (Block Public Access) | |
| +| 5 | public bucket | design as if public; the flip is its own reviewed decision (takedown process prerequisite) | Held — posture unchanged | |
| +| 6 | instance runtime | hibernating processes over S3: local SQLite + Litestream, lease = single-writer lock (over Lambda / S3-as-only-DB / cells / renting DO-Fly) | **Shipped** #630/#641 — with the correction: the lease is Go-library-only; the CLI silently absorbs a second writer, so the activator embeds litestream | |
| +| 7 | placement | rendezvous hash as hint, S3 lease as truth (over a routing table) | Lease half **shipped**; gossip/HRW is roadmap — one box, hash picks self | |
| +| 8 | instance data | one SQLite per space, forever (shared DB tier is the anti-goal) | Held, load-bearing | |
| +| 9 | attachments | blob interface → R2 from day one | Seam **shipped** #636; disk impl only, R2 roadmap | |
| +| 10 | provisioning | create = register only; materialize at first wake; controld dissolves | Split **shipped** #625, hibernate-by-default creates #663; full deferral + controld dissolution roadmap | |
| +| 11 | bootstrap | thin golden AMI + `lchat bootstrap`; release channel owns versions | `bootstrap` + channels predate #550; AMI roadmap | |
| +| 12 | migration posture | dual plane — Hetzner as-a-self-hoster + Eleven Cloud on AWS; origin-preserving moves | Held — production is the Hetzner plane; AWS holds buckets + a dev box | |
| +| 13 | custom domains | CNAME → `<sqid>.11ch.at`, per-host autocert, cert cache → S3 | Cache seam **shipped** #661 (`s3` reserved, refuses); rest roadmap | |
| +| 14 | IaC | OpenTofu, static skeleton only (over Terraform/Pulumi/CFN) | **Shipped** #609/#637/#638/#658 | |
| +| 15 | capacity | ASG target-tracking on warm-set RAM; one shared drain path; Spot slice | Roadmap | |
| +| 16 | geography | per-space `home_region`; relocate while hibernated | Field **shipped** #625, single-valued and inert; behavior roadmap | |
| +| 17 | push relay | small pool of long-lived processes; self-serve relay tokens | Relay predates #550; self-serve registration roadmap | |
| +| 18 | co-op | commons/provider seam; delegation unit = the space; ciphertext escrow | `provider` field **shipped** #625, inert; rest roadmap | |
| +| 19 | billing | Stripe holds all customer data; our record = opaque pointer + entitlement flag; public ledger | Rails **shipped** #628 (dormant until env set); public ledger roadmap (`docs/tier-economics.md` is the cost record) | |
| + |
| +Corrections the build taught us, beyond the table: the lease-is-library-only |
| +finding (#6 above); env-file ownership must follow the reader (root-written |
| +files, service-user activator — a wake died on it live); a never-woken |
| +hibernating space has no DB, so reaper stats answer from the registry row or |
| +register-only spaces become immortal; keep-last-N on S3 is an interface cap |
| +with the lifecycle as backstop, not a bucket guarantee; and `errBadVersion` |
| +is a SQLite-only nicety S3 cannot express. |
| + |
| +## The invariants — the review checklist |
| + |
| +Part 12's twelve lines, refined where building sharpened them. Every future |
| +PR that touches this territory gets checked against these: |
| + |
| +1. The server — and any future *provider* — only ever holds ciphertext plus |
| + auth plumbing; never a usable key. (The root rule; everything below is it, |
| + applied.) |
| +2. No cross-user state in home — the zero-join property *is* the scalability. |
| +3. Never consolidate spaces into a shared DB tier — per-space isolation is the |
| + sharding, security, and migration story at once. |
| +4. DB migrations are additive-only — this is what makes wake-onto-new-binary |
| + and channel-following safe. |
| +5. The IaC layer never learns an instance hostname. |
| +6. Wire contracts stay backend-opaque — clients never learn (or depend on) the |
| + storage backend; versions are opaque strings, `"0"` the one sentinel. |
| +7. The wildcard TLS private key never crosses a trust domain. |
| +8. A space's hostname survives every migration — re-keying is never the cost |
| + of moving (region, plane, or provider). |
| +9. The bot sidecar remains a pure outbound client and bot code never runs in |
| + the server process — co-spawn makes it a router *child*, but still a |
| + separate process whose traffic is marked, activity-invisible, and never a |
| + wake source. |
| +10. No cross-provider protocol, ever — spaces are islands; if a feature seems |
| + to need one, the design is wrong. |
| +11. The 18 KiB vault pad/cap and the no-per-user-analytics / no-email absences |
| + move only by deliberate decision. |
| +12. No PII in our stores, ever — customer identity lives in Stripe; we hold |
| + opaque pointers and entitlement flags. Income publishes aggregated; |
| + expenses and disbursements publish itemized. |
| + |
| +## Roadmap — designed, not built |
| + |
| +Everything below is settled design from #550 with no code behind it (or only |
| +a seam). Build triggers are events, not dates. |
| + |
| +- **The activator mesh** — the "second box" work, in progress: membership |
| + (gossip or S3 heartbeats), rendezvous hashing as the placement hint, peer |
| + proxying to the hash-owner. The lease already guarantees split-brain can't |
| + happen; the mesh only decides who *tries* first. Today's single router box |
| + is the degenerate case — membership of one, hash picks self. |
| +- **Multi-region / `home_region`** — one write region per space, wakes |
| + deterministic, relocation *while hibernated* (copy the prefix, flip the |
| + field, next wake claims the lease elsewhere). The field exists; nothing |
| + keys on it. Trigger: member-latency data from a real non-EU/US cohort. |
| +- **The co-op / provider seam** — `provider` as a real interface with |
| + ourselves as provider #1, the Hetzner box formalized as #2, escrow replicas |
| + in storage the provider doesn't control. Federation without a federation |
| + protocol. |
| +- **Lambda-able home / autoscaling runtime** — statelessness (the |
| + prerequisite) is done; the same binary on an autoscaling platform is the |
| + next rung, an edge rewrite deliberately skipped. |
| +- **Sealed bucket, then the public-bucket decision** — pepper-keyed object |
| + names and pepper-encrypted metadata first (hardens the private design too); |
| + flipping public-read stays its own reviewed decision with a written |
| + takedown process as prerequisite. |
| +- **Attachments to an object store** — the `blobStore` seam is cut; R2 for |
| + zero-egress media is the intended implementation #2. |
| +- **S3 cert cache** — `LCHAT_CERT_CACHE=s3` is reserved and refusing; needed |
| + the day two boxes share issuance. |
| +- **CRR + second-provider mirror** — the region checkbox and the Ente-style |
| + read-only mirror; everything is ciphertext, so copies are threat-model-free. |
| +- **The AWS front plane** — NLB (TCP passthrough), ASG on warm-set RAM, Spot |
| + slice, golden AMI: Part 7's ~$35/mo shape. The account, state buckets, and |
| + IAM posture exist; the compute plane doesn't. Trigger: warm-RAM pressure or |
| + a launch date. |
| +- **Observability gauges** — wake latency, lease conflicts, replication lag |
| + are loud log lines today, not metrics; Part 9's E2EE-safe metrics discipline |
| + is still owed. |
| +- **The public ledger** — Part 11's append-only books repo + static viewer. |
| + |
| +## The historical record |
| + |
| +The full twelve-part design conversation — including the superseded "cells" |
| +Part 4, the cost math, the rejected alternatives in their original words, and |
| +the lease experiment writeup — is issue |
| +[#550](https://github.com/elevenmessenger/messenger/issues/550). Related |
| +docs: `docs/instance-router.md` (the router this grew inside), |
| +`docs/cloud-provisioner.md` (create-a-space), `docs/server-releases.md` (the |
| +signed feed + mirror), `docs/secrets.md` (the key inventory), |
| +`infra/README.md` (the IaC skeleton), `docs/bots.md` / `docs/member-bots.md` |
| +(the sidecar model co-spawn inherits). |
| + |
| +One sentence, carried over intact because it survived contact with the build: |
| +**the E2EE rule did the hard part everywhere — home scales by removing the |
| +database, instances scale by removing the idle process, the fleet scales by |
| +removing the operator, the co-op scales by removing us — and the books |
| +balance in public.** |
amadan