🤖 Planet scale — the running system

This is the distillation of issue #550 ("use an object store as the only database — planet-scale plan", twelve parts across two working sessions). Most of what that thread designed has since shipped; this doc describes the system as it runs, with the still-unbuilt parts clearly marked as roadmap. Where the issue and the code differ, the code wins, and the difference is noted. #550 stays open only as the historical record of how we got here and closes when this merges.

The one idea

The E2EE rule did the hard part before any of this started. Home's server-side state is opaque client-encrypted ciphertext plus a few kilobytes of public-key metadata, with zero cross-user queries — so home scales by removing the database: an S3 bucket, used raw, is a strictly better fit than SQLite ever was. An instance's SQLite file is opaque blobs plus routing metadata the server already sees — so replicating it to a bucket is threat-model-free, and instances scale by removing the idle process: state parks in S3 as ciphertext, a process exists only while someone's talking, and creating a space becomes (nearly) a metadata write. Launch-day load lands on S3's capacity, not ours.

What runs today

Home: the store is a seam, and S3 is implementation #2

home/store.go (PR #611) extracts vaultStore, home's whole persistence surface, shaped like an object store on purpose: point reads by key, conditional writes (CAS on an opaque version string, with the one universal sentinel "0" = "no blob yet"), keep-last-N blob versions, prefix deletes, and write-once pointers. SQLite is implementation #1 and stays the zero-config self-hoster default forever — dual backend is a feature, not migration debt.

home/s3store.go (PR #627) is implementation #2: one bucket, no database, laid out exactly as the issue drew it —

blobs/<uid>        the vault; S3-native versioning, version = the ETag
users/<uid>        one metadata JSON per user (credentials + devices),
                   every mutation an If-Match CAS loop
creds/<credId>     write-once pointer, body = uid
tokens/<sha256>    write-once pointer, body = uid

CAS rides S3 conditional writes (If-Match to replace; the "0" sentinel maps to If-None-Match: * for create-only). Pointers are never authoritative: resolving one re-loads users/<uid> and verifies the credential/token is still listed, so the stale pointer a revocation leaves behind denies by construction. Two open questions from the issue were settled at PR time, both worth recording:

• The client is a hand-rolled SigV4 signer (home/sigv4.go), not the AWS SDK — the surface home needs is six calls, and home keeps its tiny dependency footprint. (The activator side, below, does carry aws-sdk-go-v2, brought in by litestream — the two made opposite calls for good local reasons.)
• Keep-last-5 is enforced at the interface, not by the bucket: listBlobVersions caps at five; the bucket's lifecycle rule prunes noncurrent versions asynchronously, days later, as a backstop. And S3 cannot distinguish "a version string that was never ours" from a plain mismatch, so the SQLite backend's errBadVersion (→ 400) is deliberately backend-specific; on S3 both read as conflicts (→ 409).

Selection is by env (home/main.go): LCHAT_HOME_STORE unset or sqlite is the default; s3 requires LCHAT_HOME_S3_BUCKET + LCHAT_HOME_S3_REGION plus AWS credentials in the standard env vars, with optional LCHAT_HOME_S3_PREFIX (test isolation) and LCHAT_HOME_S3_ENDPOINT (path-style, for S3-compatible stores). Misconfiguration fails at boot, never limps.

The wire contract was already ETag-opaque: PR #612 made the blob version a JSON string on the wire (clients tolerated both) and coarsened link-token last_seen to daily, so an object-store backend never pays a write per authenticated read.

Home is stateless

home/stateless.go (PR #613) removed the last two things a home process remembered between requests. Ceremony proofs became HMAC-signed tokens (uid + expiry, 5-minute window) the client carries; in-flight WebAuthn sessions are AES-GCM-sealed into the ?session= query param. Both keys derive from one server secret, LCHAT_HOME_SESSION_KEY — auth plumbing like the admin token, it cannot open a byte of any vault. Unset, an ephemeral key is minted at boot (with a logged warning): single-process self-hosters keep working with zero config, and ceremonies just don't survive a restart, which is exactly what the old in-memory map gave them. With the bucket backend and the stateless ceremonies together, home is a pure function from (request, bucket) → response — any replica behind a load balancer can finish what another began.

Fleet backup: replication as a property of normal operation

deploy/fleet-backup.sh (PRs #637, #643) closed the actual gap the plan started from — instance DBs living on one box with no off-box copy. It runs on the router box, idempotent, regenerating /etc/litestream.yml from what's in /data: a CLI Litestream (0.5.14) streams every /data/*.db — instances plus home/push/cloud/registry — to the fleet-backup bucket (eleven-prod-fleet-backup-…, one prefix per database), under the lchat-litestream unit and a scoped IAM user that can touch objects but never versioning, lifecycle, or the bucket itself. Versioning + a 14-day noncurrent window is the ransomware posture: an overwrite or delete with the box's own credentials leaves recoverable versions behind. Continuous replication, not snapshots — seconds of RPO, point-in-time recovery, and it supersedes the parked R2-snapshot (r2d2.sh) plan entirely. The restore drill is one command, documented in the script's header; run it against a scratch path, never over a live file.

One exclusion, added when hibernation went live (PR #658): hibernating hosts are skipped — the activator's embedded Litestream owns their replication (to the instances bucket, under the lease), and two replicators on one WAL would fight over checkpointing.

Instance hibernation, end to end

This is Part 4 v2 of the issue, running. Four layers:

The lease (internal/activator/replica.go, PR #630). Litestream is embedded as a Go library — pure modernc.org/sqlite, no cgo, so the single-binary property survives. Its S3 lease (lock.json acquired via conditional writes) is the distributed single-writer lock: a second claimant fails loudly with a typed *litestream.LeaseExistsError. This is a correction the plan earned by testing (recorded on #550): the issue assumed the lease was CLI config, but the lease exists only in the library API — the CLI silently absorbs a second replicator, orphaning the loser's writes with no error signal. Embedding it turned that from a footgun into the mechanism. TTL is 15 m, renewed every 5 m (each renewal is one S3 Tier-1 PUT — at the original 30 s/10 s cadence the lease heartbeat alone was ~70% of the fleet's S3 request bill, ~$16/yr per always-awake instance against €5/y plans); a failed renewal means leadership is lost and the holder stands down. The failover window follows the TTL: a wedged or hard-killed process is claimable by a healthy peer within ~15 m of its last renewal — every graceful path (sleep, deploy, shutdown) releases explicitly and hands over in seconds, so the window prices only real crashes. Restores run under the held lease (restoring outside it could resurrect a stale copy that shadows the true holder's newer state).

The activator (activator.go, PR #641). With LCHAT_HIBERNATE_BUCKET set, the router stops relying on systemd to keep flagged instances alive and runs them as its own children, on demand — "units are for the router, instances are its children", exactly as the issue put it. The per-host state machine is asleep → waking → awake → stopping:

• Wake (a request for a host with no live process): claim the per-host lease at <prefix>/<host>/lock.json; restore the DB from S3 only if the local copy is gone (a present local DB is the warm cache — on one box, under the lease, it can only be our own last checkpoint); spawn the instance (exec of the same versioned live/<host> binary its systemd unit would run, same env file); start replication under the lease. Concurrent requests coalesce on one wake (singleflight); lease contention and wake timeouts are 503 + Retry-After — "sleeping spaces are slow to wake", never a 5xx cascade.
• Sleep (idle past LCHAT_HIBERNATE_IDLE, default 15 m — idleness is in-flight-count and time, so one open WebSocket holds an instance awake): probe the next due time, SIGTERM the child (its own graceful drain is the final write), final checkpoint to S3, release the lease, forget the entry. The local DB stays as the warm cache.
• Failure is the wake path: a dead child or a lost lease stands the entry down, and the next request re-wakes it. S3 blips leave warm instances serving (local SQLite is authoritative; replication queues and drains) — only cold wakes stall.

Due-time wakes (PR #653). Scheduled deliveries (future messages, reminders) fire on time while their instance sleeps: at sleep the stop path asks the instance for its earliest pending delivery (GET /api/next-due, admin-gated, over the socket) and records it in the registry (routes.next_due — persisted, so it survives router restarts); the sweeper wakes any sleeping host whose time has arrived, no client request needed. Worst-case lateness ≈ one sweep interval (≤30 s) + spawn + one scheduler tick (≤5 s). One honest gap the issue glossed: member-bot charter schedules run in the sidecar (docs/member-bots.md), so they didn't wake a sleeping instance — the index covered server-side schedules only. Closed by charter-wake registration: after every schedule recompute the charter host reports its earliest next run to the instance (POST /api/charter-next-due, bot-token-gated — one bare timestamp; the charters stay ciphertext), the instance folds it into GET /api/next-due, and the same index wakes for it — zero activator changes. A stale report from a dead sidecar is ignored once past (it changes only when the sidecar re-reports, so honoring it would re-wake after every sleep): at most one wasted wake, never a loop — and a sidecar that can't come up couldn't run the charter anyway. The due-wake's other half lives in the sidecar: a co-spawned host that loses the delivery race (the instance's scheduler fires the tick before the host's cursor baseline) treats a live charter with no pending alarm at connect as owing a run now, so the woken charter actually runs instead of skipping to its next occurrence.

Sidecar co-spawn (PR #670). A host with a bot sidecar staged (/data/<host>-bot.env — the same file lchat-bot@.service reads) gets its bot run child spawned right after the instance turns healthy and SIGTERMed first at sleep: the sidecar exists exactly while its instance is awake, so its lifecycle solves itself — the issue's "co-located over loopback" default, built. The subtlety that makes it work: the sidecar dials its instance's public URL back through this very router, so it stamps every request (X-Lchat-Sidecar, internal/bot/sidecarmark.go) and the activator makes that traffic activity-invisible — no idle-clock credit while awake, a 503 (never a wake) while asleep. Without that, the pair could never sleep and an orphaned sidecar's reconnect loop would wake its instance right back, forever. Spoofing the marker only downgrades the sender's own requests. Supervision is sweep-driven Restart=always with capped exponential backoff; a sidecar that won't start never fails the wake — members before bots.

The flip, per host, and creates that are born asleep

Opt-in is per host (routes.hibernate in the registry, PR #659), which is what makes a production canary safe: LCHAT_HIBERNATE_BUCKET ships fleet-wide as a no-op, then

lchat hibernate <host> --on     # stop+disable its unit; the router wakes it per request
lchat hibernate <host> --off    # activator releases (checkpoint + lease back), unit restarts
lchat instances                 # HIBERNATE column shows who's activator-managed

migrates exactly one instance, and back. The flip's ordering is deliberate (flag → stop unit → reload router → warm request) so the host is never left unserveable, and --off waits for the child to actually release the socket before the unit starts (an instance refuses to clobber a live listener). A staged bot sidecar hands over both ways in the same flip.

One lesson found live, now baked in: env-file ownership. Systemd reads a root-written EnvironmentFile as root; the activator reads it as the service user — the canary's first wake died on permission denied. add-instance and both flips now chown the env (and bot session dirs) to match the data dir.

Creation splits into register (the row: host, minted-or-imported token, placement fields — pure metadata, no capacity) and materialize (env file, staged binary, unit, health check, router reload, setup invite) — PR #625, exactly the seam the issue asked for. A --hibernate create (and, behind LCHAT_CONTROL_HIBERNATE_CREATES=1 on the control daemon, every provisioner-created space — PR #663) skips only the systemd half: no unit, no resident process, no DB — the first wake finds no replica in S3 and boots fresh. Capacity for hibernating rows is a row ceiling (LCHAT_MAX_REGISTERED, default 10× LCHAT_MAX_INSTANCES), not a process cap. Honest delta from the issue's "create = a registry row + an S3 prefix, microseconds, no box contacted": today's hibernating create still writes the env file, stages the binary, reloads the router — and its readiness probe is the first wake, because minting the setup invite needs a running instance to pre-seed the admin user. Cheap, but not yet free; the fully-deferred materialization remains roadmap.

The reaper got the property Part 5 promised — reclaiming never-claimed spaces without waking anything. A hibernating space's stats (setup_complete / members / has_content) are read straight off its local DB file, read-only — wake-free because the warm cache is the newest truth on one box. The case the code had to add: a hibernating space that has never woken has no DB at all, and before the fix that read as a stats error — and a stats error is a skip-never-reap, so register-only spaces would have been immortal. The stats handler now answers those definitionally from the registry row (0 members, never set up, no content); a missing DB on a non-hibernating row stays a 404, meaning a failed create, skipped as before.

The mesh: multiple boxes, one owner per instance

The multi-box mechanism from Part 4 v2 shipped as PR #678, gated on LCHAT_MESH_SELF (a peer-reachable base URL); absent means zero mesh codepaths, and the current single-box deployment runs exactly as before.

• Membership is S3 heartbeats, not gossip: each box PUTs one object per 60 seconds under the hibernation prefix's mesh/members/, and the object key is the record (base64url of the peer URL) so a single LIST yields the whole membership. Liveness reads S3's own LastModified, so a box with a wrong clock can't poison the staleness judgment. Stale after 3 m (three missed beats — well inside the lease expiry that actually gates takeover).
• Rendezvous (HRW) hashing is the placement hint — owner(host) is a pure function of the live member set; adding a box moves only the hosts that land on it. The lease stays the truth: a box that wrongly believes it owns a host loses the CAS race loudly and proxies instead, so transient membership disagreement structurally cannot split-brain.
• Wrong-box requests reverse-proxy to the owner, TLS verified against the instance hostname with the connection pinned to the peer's address (no InsecureSkipVerify anywhere); a one-hop marker turns disagreement loops into a wrong-box wake the lease resolves.
• Drain deletes the heartbeat first, then checkpoints and releases as usual — so deploys, scale-down, and box death are the same code path. The kill-a-box property is a live test, both ways: a drained owner's instances are serving on the survivor ~2 seconds later; a SIGKILLed owner's once its leases expire (the TTL — proven live at the original 30 s TTL, since raised to 15 m for S3 request cost), data intact in both.

Hardening that same week (PR #681, both lessons from the first real production flip): flipping a host refuses while the box-wide backup replicator still tracks it — two litestream processes on one DB share a derived meta directory and corrupt each other — and the lease release runs under its own budget so a wedged final checkpoint can't strand the lease until TTL.

The seams cut for what's next

Small PRs that exist so later phases are config, not surgery:

• Attachments sit behind a blobStore interface (blobstore.go, PR #636): write-once immutable bytes, no CAS or versioning needed — sized and thread-tagged in SQLite, ciphertext in the store. Disk is implementation #1; the object-store implementation (R2 for zero-egress media, per the issue) is roadmap.
• The cert cache is behind autocert.Cache (certcache.go, PR #661): LCHAT_CERT_CACHE unset/dir is today's /data/acme layout; s3 is reserved for the pooled-router future and refuses to start until it exists — failing loudly beats minting certs into a local dir the rest of a pool can't see.
• Placement fields: the registry row carries home_region and provider (PR #625), single-valued and inert today — the geo (Part 7) and co-op (Part 10) seams, as schema.
• Billing rails (PR #628): the cloud ledger's spaces table holds exactly Part 11's shape — stripe_customer_id (an opaque pointer into Stripe) and an entitlement flag — no email, name, or card detail anywhere in our stores (identity lives in Stripe, where the operator can see it). Dormant until the Stripe env is set.
• The release-feed mirror (PR #662, deploy/release-server.sh + updater.go): the fleet self-updates from GitHub, whose CDN has served stale manifests — the Part 9 SPOF. Publishing now mirrors the signed feed and binaries to S3 (LCHAT_RELEASE_MIRROR_BUCKET), and updaters fall back (LCHAT_RELEASE_MIRROR_URL) when the primary fails or verifies but is stale against the newest feed the box ever accepted. Signature verification is identical on both paths: the mirror adds availability, never trust.
• The secrets inventory (docs/secrets.md, PR #639): every server-side secret, blast radius, and rotation story — Part 12's operability item 11.
• Infrastructure as code (infra/, PRs #609, #637, #638, #658): OpenTofu, managing the static skeleton only — dev and prod as separate states. Dev holds the proof-build buckets and a t4g.small box; prod holds the fleet-backup and hibernation buckets (versioning, lifecycle, Block Public Access, scoped IAM users whose keys never enter tofu state) plus CloudTrail. The boundary test from the issue holds and is written into infra/README.md: if the IaC needs to know an instance hostname, the design is wrong.

Decision log

Part 12's table, one line per settled call, updated with what shipped. "Held" means the decision stands with nothing built against it yet.

| # | decision | chosen | status | |---|---|---|---| | 1 | home store | S3 as the only database (over R2-sole / DynamoDB) | Shipped #611/#627 — dual backend, SQLite stays the self-hoster default; minimal SigV4 client, not the SDK | | 2 | home runtime | stateless Go binary, Lambda-able later (over a Workers rewrite) | Shipped #613 — statelessness done; Lambda step not taken | | 3 | multi-region writes | single write region + CRR warm standby (over active-active, which breaks per-region CAS) | Held — single region today, no CRR yet | | 4 | metadata at rest | sealed bucket: HMAC(pepper) keys + pepper-encrypted metadata | Not built — object keys are plain uids; bucket stays private (Block Public Access) | | 5 | public bucket | design as if public; the flip is its own reviewed decision (takedown process prerequisite) | Held — posture unchanged | | 6 | instance runtime | hibernating processes over S3: local SQLite + Litestream, lease = single-writer lock (over Lambda / S3-as-only-DB / cells / renting DO-Fly) | Shipped #630/#641 — with the correction: the lease is Go-library-only; the CLI silently absorbs a second writer, so the activator embeds litestream | | 7 | placement | rendezvous hash as hint, S3 lease as truth (over a routing table) | Shipped (#678) — S3-heartbeat membership + HRW + proxy-to-owner; kill-a-box passes live both ways. Live in production 2026-07-13 across two boxes: domain-scoped placement (#820), automatic route replication + mesh-seed (#821), and fleet-wide status aggregation partitioned by ownership (#822/#849/#851/#852) | | 8 | instance data | one SQLite per space, forever (shared DB tier is the anti-goal) | Held, load-bearing | | 9 | attachments | blob interface → R2 from day one | Seam shipped #636; disk impl only, R2 roadmap | | 10 | provisioning | create = register only; materialize at first wake; controld dissolves | Split shipped #625, hibernate-by-default creates #663; full deferral + controld dissolution roadmap | | 11 | bootstrap | thin golden AMI + lchat bootstrap; release channel owns versions | bootstrap + channels predate #550; AMI roadmap | | 12 | migration posture | dual plane — Hetzner as-a-self-hoster + Eleven Cloud on AWS; origin-preserving moves | Held — production is the Hetzner plane; AWS holds buckets + a dev box | | 13 | custom domains | CNAME → <sqid>.11ch.at, per-host autocert, cert cache → S3 | Cache seam shipped #661 (s3 reserved, refuses); rest roadmap | | 14 | IaC | OpenTofu, static skeleton only (over Terraform/Pulumi/CFN) | Shipped #609/#637/#638/#658 | | 15 | capacity | ASG target-tracking on warm-set RAM; one shared drain path; Spot slice | Roadmap | | 16 | geography | per-space home_region; relocate while hibernated | Field shipped #625, single-valued and inert; behavior roadmap | | 17 | push relay | small pool of long-lived processes; self-serve relay tokens | Relay predates #550; self-serve registration roadmap | | 18 | co-op | commons/provider seam; delegation unit = the space; ciphertext escrow | provider field shipped #625, inert; rest roadmap | | 19 | billing | Stripe holds all customer data; our record = opaque pointer + entitlement flag; public ledger | Rails shipped #628 (dormant until env set); public ledger roadmap (docs/tier-economics.md is the cost record) |

Corrections the build taught us, beyond the table: the lease-is-library-only finding (#6 above); env-file ownership must follow the reader (root-written files, service-user activator — a wake died on it live); a never-woken hibernating space has no DB, so reaper stats answer from the registry row or register-only spaces become immortal; keep-last-N on S3 is an interface cap with the lifecycle as backstop, not a bucket guarantee; errBadVersion is a SQLite-only nicety S3 cannot express; and the first live mesh flip (#7, 2026-07-13) taught that a replicated route is a registry row only — it never stages a binary or creates a live symlink, and the sandboxed router (ProtectSystem=strict) cannot do that itself, so a replicated route's first wake 503ed outright until the wake path was rerouted through the unsandboxed control daemon (#844) instead of writing /opt/lchat directly.

The invariants — the review checklist

Part 12's twelve lines, refined where building sharpened them. Every future PR that touches this territory gets checked against these:

1. The server — and any future provider — only ever holds ciphertext plus auth plumbing; never a usable key. (The root rule; everything below is it, applied.)
2. No cross-user state in home — the zero-join property is the scalability.
3. Never consolidate spaces into a shared DB tier — per-space isolation is the sharding, security, and migration story at once.
4. DB migrations are additive-only — this is what makes wake-onto-new-binary and channel-following safe.
5. The IaC layer never learns an instance hostname.
6. Wire contracts stay backend-opaque — clients never learn (or depend on) the storage backend; versions are opaque strings, "0" the one sentinel.
7. The wildcard TLS private key never crosses a trust domain.
8. A space's hostname survives every migration — re-keying is never the cost of moving (region, plane, or provider).
9. The bot sidecar remains a pure outbound client and bot code never runs in the server process — co-spawn makes it a router child, but still a separate process whose traffic is marked, activity-invisible, and never a wake source.
10. No cross-provider protocol, ever — spaces are islands; if a feature seems to need one, the design is wrong.
11. The 18 KiB vault pad/cap and the no-per-user-analytics / no-email absences move only by deliberate decision.
12. No PII in our stores, ever — customer identity lives in Stripe; we hold opaque pointers and entitlement flags. Income publishes aggregated; expenses and disbursements publish itemized.

Roadmap — designed, not built

Everything below is settled design from #550 with no code behind it (or only a seam). Build triggers are events, not dates.

• Multi-region / home_region — one write region per space, wakes deterministic, relocation while hibernated (copy the prefix, flip the field, next wake claims the lease elsewhere). The field exists; nothing keys on it. Trigger: member-latency data from a real non-EU/US cohort. The full design that builds on this — region as a mutable property (not encoded in the address), stable-address relocation, and multi-edge replication, with the append-only/CRDT reasoning that makes it tractable — is docs/multi-edge.md (agreed direction, 2026-07-18).
• The co-op / provider seam — provider as a real interface with ourselves as provider #1, the Hetzner box formalized as #2, escrow replicas in storage the provider doesn't control. Federation without a federation protocol.
• Lambda-able home / autoscaling runtime — statelessness (the prerequisite) is done; the same binary on an autoscaling platform is the next rung, an edge rewrite deliberately skipped.
• Sealed bucket, then the public-bucket decision — pepper-keyed object names and pepper-encrypted metadata first (hardens the private design too); flipping public-read stays its own reviewed decision with a written takedown process as prerequisite.
• Attachments to an object store — the blobStore seam is cut; R2 for zero-egress media is the intended implementation #2.
• S3 cert cache — LCHAT_CERT_CACHE=s3 is reserved and refusing; needed the day two boxes share issuance.
• CRR + second-provider mirror — the region checkbox and the Ente-style read-only mirror; everything is ciphertext, so copies are threat-model-free.
• The AWS front plane — NLB (TCP passthrough), ASG on warm-set RAM, Spot slice, golden AMI: Part 7's ~$35/mo shape. The account, state buckets, and IAM posture exist; the compute plane doesn't. Trigger: warm-RAM pressure or a launch date.
• Observability gauges — wake latency, lease conflicts, replication lag are loud log lines today, not metrics; Part 9's E2EE-safe metrics discipline is still owed.
• The public ledger — Part 11's append-only books repo. Partially shipped: the public status dashboards exist (fleet state + aggregated Stripe income, published from the box every 5 minutes — docs/status.md); the itemized expense/disbursement ledger repo is still to come.

The historical record

The full twelve-part design conversation — including the superseded "cells" Part 4, the cost math, the rejected alternatives in their original words, and the lease experiment writeup — is issue #550. Related docs: docs/instance-router.md (the router this grew inside), docs/cloud-provisioner.md (create-a-space), docs/server-releases.md (the signed feed + mirror), docs/secrets.md (the key inventory), infra/README.md (the IaC skeleton), docs/bots.md / docs/member-bots.md (the sidecar model co-spawn inherits).

One sentence, carried over intact because it survived contact with the build: the E2EE rule did the hard part everywhere — home scales by removing the database, instances scale by removing the idle process, the fleet scales by removing the operator, the co-op scales by removing us — and the books balance in public.