| 1 | # 🤖 The cloud provisioner — self-serve space creation |
| 2 | |
| 3 | Status: **shipped** (server side). The app UI ("+ → Sign in → new space") is |
| 4 | deliberately not built yet; this doc covers everything behind it. |
| 5 | |
| 6 | ## The idea |
| 7 | |
| 8 | Creating an instance used to be an operator SSHing to the box and running |
| 9 | `lchat add-instance`. The end state we want: a member signed in to their home |
| 10 | server taps **+** in the app and gets a fresh, fully-isolated Eleven space — |
| 11 | address, TLS, admin-granting setup link — in a few seconds. Two pieces make |
| 12 | that possible: |
| 13 | |
| 14 | - **The cloud provisioner** (`lchat cloud`, e.g. at `cloud.elevenmessenger.com`) |
| 15 | can create instances on its box, gated by caller tokens, per-user quotas, and |
| 16 | a box capacity limit. It's a subcommand of the same binary that runs the |
| 17 | router, so **any host running Eleven can enable it** and get multi-instance |
| 18 | auto-provisioning — it isn't infrastructure only we can run. |
| 19 | - **The home server** gains one endpoint (`POST /api/spaces`): a signed-in user |
| 20 | asks *their* home server for a space; home forwards the request to the cloud |
| 21 | provisioner it's paired with and relays the result. |
| 22 | |
| 23 | ``` |
| 24 | app ── link token ──▶ home ── caller token ──▶ lchat cloud ── control token ──▶ lchat controld ──▶ systemctl… |
| 25 | (lchat-home unit) (lchat user, sandboxed, (root; local unix socket |
| 26 | public via the router) /run/lchat/control.sock) |
| 27 | ``` |
| 28 | |
| 29 | The client then opens the returned **setup link** (`/join/setup#<T>`), claims |
| 30 | it, becomes the space's admin, and runs the normal setup conversation. Adding |
| 31 | the new space to the user's encrypted backup is the client's move, exactly like |
| 32 | any other server — the blob stays content-blind. |
| 33 | |
| 34 | ## The privilege split |
| 35 | |
| 36 | Creating an instance needs root (systemctl, writes to `/opt/lchat`), but |
| 37 | everything network-facing on the box runs as the unprivileged `lchat` user |
| 38 | under a strict systemd sandbox. So provisioning is two processes: |
| 39 | |
| 40 | - **`lchat controld`** (`lchat-control.service`, **root**) — the god surface |
| 41 | `instance-router.md` reserved: `POST /instances` on a local unix socket, |
| 42 | never routed publicly. It executes the same creation core as |
| 43 | `lchat add-instance` (`addInstance` in `control.go`), serialized, and is the |
| 44 | single authority on **capacity** (below). |
| 45 | - **`lchat cloud`** (`lchat-cloud.service`, `lchat` user, sandboxed) — the |
| 46 | public policy layer: caller tokens, per-user quotas, friendly slug |
| 47 | generation. It holds no root powers; everything it can do to the box, it does |
| 48 | by asking controld. |
| 49 | |
| 50 | ### Who holds which token |
| 51 | |
| 52 | | Token | Lives | Grants | |
| 53 | |---|---|---| |
| 54 | | control token | `/etc/lchat/control.token`, **root-only 0600**; handed to the cloud unit alone via systemd `LoadCredential` | create instances on the box | |
| 55 | | cloud admin token (`LCHAT_ADMIN_TOKEN`) | `/data/cloud.env` | mint/list/revoke caller tokens | |
| 56 | | caller token | the paired home server's env (`LCHAT_CLOUD_TOKEN`) | provision spaces, bounded by quota | |
| 57 | | device link token | the user's device | ask *their* home server for a space | |
| 58 | |
| 59 | The root-only control token matters because **every service on the box shares |
| 60 | the `lchat` uid** — socket permissions can't distinguish the cloud service from |
| 61 | a compromised instance. A compromised instance can read `/data` (a known, |
| 62 | accepted box-level exposure), but it cannot read the control token, so it |
| 63 | cannot mint instances or reach the creation path. `lchat bootstrap` mints the |
| 64 | token idempotently. |
| 65 | |
| 66 | ## Capacity: the box says no before it falls over |
| 67 | |
| 68 | controld enforces two limits, checked atomically with each create |
| 69 | (`LCHAT_MAX_INSTANCES`, default **40**; `LCHAT_MIN_FREE_GB`, default **5**): |
| 70 | |
| 71 | - **Instance ceiling.** The educated guess for a 4 GB box: the OS, router, |
| 72 | push, home, and the existing fleet use ~1 GB; an idle instance is ~25–40 MB |
| 73 | (no server-side crypto or media work, by design), so 40 spaces ≈ 1.4 GB with |
| 74 | ~1.5 GB headroom for spikes. When Go bot sidecars land (~25 MB each, vs |
| 75 | ~120 MB for today's Node sidecar), 40 spaces *with* bots still fits. |
| 76 | - **Disk floor.** Disk, not RAM, is the long-run binding resource (attachments |
| 77 | are blobs in SQLite), so creation also refuses when `/data` has less than the |
| 78 | floor free. |
| 79 | |
| 80 | Past either limit the API answers `503 {error: "at_capacity"}` and the cloud |
| 81 | layer relays the member-facing message: *"We can't create a new Eleven space |
| 82 | right now — please try again later."* Clients show it verbatim and stay calm. |
| 83 | |
| 84 | ### Hibernate-by-default creates (`LCHAT_CONTROL_HIBERNATE_CREATES=1`) |
| 85 | |
| 86 | With this env set on `lchat-control` (off by default — the flip is a per-box |
| 87 | config decision, not a code deploy), every provisioner-created space is born |
| 88 | **hibernating** (#550): the create registers the route, writes the env file, |
| 89 | stages the binary — but starts **no systemd unit and no process**. The |
| 90 | create-path readiness probe is the space's one *first wake* (it mints the setup |
| 91 | link and warms the cert through the router's activator); after the idle window |
| 92 | the activator puts it to sleep, and a never-claimed space costs **zero RAM** |
| 93 | from then on. It requires the **router** to be running with |
| 94 | `LCHAT_HIBERNATE_BUCKET`, or created spaces never answer. Direct |
| 95 | `lchat add-instance` keeps its explicit `--hibernate` flag either way. |
| 96 | |
| 97 | Under the flag the capacity policy changes shape: |
| 98 | |
| 99 | - **`LCHAT_MAX_INSTANCES` counts *materialized* instances only** — the |
| 100 | systemd-managed, always-on ones that actually hold RAM. Hibernating rows |
| 101 | don't consume it, so the cap stops being the constraint on self-serve |
| 102 | creation. |
| 103 | - **`LCHAT_MAX_REGISTERED`** (default **10× `LCHAT_MAX_INSTANCES`**) is a |
| 104 | sanity ceiling on *total* instance rows, hibernating included — registry |
| 105 | rows, env files, and staged symlinks aren't free forever. |
| 106 | - The **disk floor** applies unchanged; disk stays the real long-run bound. |
| 107 | |
| 108 | ## Addresses |
| 109 | |
| 110 | Spaces get generated friendly slugs — `amber-fox-42.<base-domain>` — from small |
| 111 | embedded wordlists (collisions re-roll; a reserved blocklist keeps sibling |
| 112 | names like `home`/`push`/`cloud` unmintable). The display name of the group is |
| 113 | set in the setup conversation; the hostname is just an address, and hostnames |
| 114 | are public in certificate-transparency logs regardless. User-chosen slugs (or |
| 115 | custom domains) are a possible later feature, deliberately not v1. |
| 116 | |
| 117 | ### Addressing (sqids) |
| 118 | |
| 119 | A space's subdomain label is **`sqids(id)`** — the sqid of its integer primary |
| 120 | key in the `spaces` table. So `a4f.<base-domain>` both *is* short and *encodes* |
| 121 | the id reversibly: `spaceID("a4f")` recovers the id, i.e. "which server this is." |
| 122 | SQLite supplies the id (`INTEGER PRIMARY KEY AUTOINCREMENT`); Go's `sqids-go` |
| 123 | does the encoding (lowercase base36, min length 3 so labels grow only as ids do, |
| 124 | default blocklist so nothing rude is minted, `AUTOINCREMENT` so a torn-down |
| 125 | space's id — and thus its label — is never reused). Creation reserves the id |
| 126 | first (quota-checked), turns it into the hostname, then creates the instance; |
| 127 | there's no wordlist and no collision re-roll, because distinct ids give distinct |
| 128 | labels by construction. |
| 129 | |
| 130 | ### Certs + readiness |
| 131 | |
| 132 | The base domain (`LCHAT_CLOUD_BASE_DOMAIN`) needs **wildcard DNS** pointing at |
| 133 | the box. Certs are per-host (autocert HTTP-01) unless the box is configured for a |
| 134 | wildcard cert (DNS-01 — see `instance-router.md`). Either way, provisioning |
| 135 | **confirms readiness through the public path before returning the setup link**: |
| 136 | after the instance passes its own socket health check, `add-instance` issues an |
| 137 | HTTPS request *through the local router* (dialed at `:443` with the new host's |
| 138 | SNI) and waits for a non-5xx reply. That guarantees the moment a human clicks the |
| 139 | setup link they reach a ready admin sign-on — never a mid-provision error — and, |
| 140 | on a per-host-cert box, warms the cert as a side effect (the handshake triggers |
| 141 | issuance) so there's no first-hit "not secure" race. Best-effort: on timeout it |
| 142 | returns anyway (the instance already health-checked; the public path is a beat |
| 143 | behind at most). The home UI keeps a spinner up for the whole call, so the |
| 144 | "Create a new space" button only reveals the link once the space is confirmed up. |
| 145 | |
| 146 | A wildcard record also means typo'd or retired subdomains resolve to the |
| 147 | router. Set the router's `LCHAT_FALLBACK_URL` (e.g. the apex marketing site) so |
| 148 | those redirect somewhere useful instead of 404ing. On a per-host-cert box an |
| 149 | unknown host fails TLS before any redirect can happen (no cert is ever minted |
| 150 | for it), so the redirect only covers `:80` and hosts whose cert outlived their |
| 151 | route — but with the **wildcard cert** (DNS-01, `instance-router.md § Wildcard |
| 152 | TLS`) any subdomain of the base domain completes the handshake, so garbage |
| 153 | subdomains land on the fallback over https too. |
| 154 | |
| 155 | ## API |
| 156 | |
| 157 | ### cloud (`lchat cloud`) |
| 158 | |
| 159 | ``` |
| 160 | POST /api/instances bearer: caller token {requester: <opaque id>} |
| 161 | 201 {host, url, setup_link} |
| 162 | 409 {error: quota_exceeded} per (caller, requester) FREE-space quota, |
| 163 | default 1 (LCHAT_CLOUD_QUOTA; paid spaces |
| 164 | never count, and upgrading frees the slot) |
| 165 | 503 {error: at_capacity, message} the box is full |
| 166 | GET /api/instances?requester=… this requester's spaces (+ entitlement/tier, |
| 167 | and a top-level billing price book) |
| 168 | POST /api/instances/{host}/checkout bearer: caller token |
| 169 | {requester, tier, success_url, cancel_url} |
| 170 | 200 {url} the hosted Stripe Checkout page |
| 171 | 400 unknown tier | 404 not yours | 409 already_upgraded | 501 billing off |
| 172 | POST /api/instances/{host}/portal bearer: caller token {requester, return_url} |
| 173 | 200 {url} the hosted Billing Portal (cancel/card/invoices) |
| 174 | 404 not yours | 409 not_subscribed | 501 billing off |
| 175 | POST /api/stripe/webhook signature-gated (Stripe calls it; see Billing) |
| 176 | POST /api/admin/tokens bearer: admin token {label} → {id, token} (shown once) |
| 177 | GET /api/admin/tokens list (never the token values) |
| 178 | DELETE /api/admin/tokens/{id} revoke |
| 179 | ``` |
| 180 | |
| 181 | ### controld (`lchat controld`, local socket only) |
| 182 | |
| 183 | ``` |
| 184 | POST /instances bearer: control token {host, admin_token?, …env fields} |
| 185 | 201 {host, url, setup_link, admin_link} |
| 186 | 400 invalid_host | 409 host_exists | 503 at_capacity | 500 create_failed |
| 187 | ``` |
| 188 | |
| 189 | ### home |
| 190 | |
| 191 | ``` |
| 192 | POST /api/spaces bearer: device link token |
| 193 | → relays the cloud response (201 / 409 / 503 shapes above) |
| 194 | 403 {error: not_eligible} caller has no saved space yet (see gate below) |
| 195 | 501 {error: unavailable} this home server has no provisioner configured |
| 196 | GET /api/spaces bearer: device link token |
| 197 | → the hosts this user provisioned (drives the "Delete space" affordance) |
| 198 | DELETE /api/spaces/{host} X-Ceremony-Proof: fresh passkey ceremony ONLY |
| 199 | → relays cloud's 200 / 404 |
| 200 | POST /api/spaces/{host}/checkout bearer: device link token {tier} |
| 201 | → relays cloud's 200 {url} / 400 / 404 / 409 / 501; return URLs point back at home |
| 202 | POST /api/spaces/{host}/portal bearer: device link token |
| 203 | → relays cloud's 200 {url} / 404 / 409 / 501 (Manage on a paid space's row) |
| 204 | ``` |
| 205 | |
| 206 | 🤖 **Per-country provisioning (optional, `LCHAT_CLOUD_REGIONS`).** By default |
| 207 | home talks to one provisioner (`LCHAT_CLOUD_URL` + `LCHAT_CLOUD_TOKEN`). Setting |
| 208 | `LCHAT_CLOUD_REGIONS` — a JSON array of `{code,name,url,token}` — plus |
| 209 | `LCHAT_CLOUD_DEFAULT_REGION` lets a signed-in user's space be placed in one of |
| 210 | several country provisioners (the dev mesh runs `ie` + `fi`; see |
| 211 | `docs/dev-mesh.md`). It supersedes the single `LCHAT_CLOUD_URL`; the default |
| 212 | region's provisioner also backs demo and every host-agnostic fallback. Routing: |
| 213 | `POST /api/spaces` takes an optional **`region`** (empty = auto-assign to the |
| 214 | default; an unknown code quietly auto-assigns rather than failing); `GET |
| 215 | /api/spaces` fans out across every region and merges, so a user sees spaces they |
| 216 | hold anywhere; host-specific calls (delete, checkout, portal) route by the region |
| 217 | label the host encodes (`<sqid>.<code>.…`). `GET /api/features` advertises the |
| 218 | regions — `code` + `name` only, never a url or token — so the create UI can offer |
| 219 | the choice; single-region home omits the key. (The create-page region control is |
| 220 | a follow-on; today the choice rides the `region` field.) |
| 221 | |
| 222 | Creation and listing are link-token-authed (additive/reads) and creation is |
| 223 | rate-limited per IP; the real abuse bound is the provisioner's quota. **With |
| 224 | `LCHAT_POW_BITS` set on home (default off), creation also demands hashcash**: |
| 225 | `POST /api/spaces/challenge` mints a sealed, expiring, single-use challenge |
| 226 | bound to the account; the client grinds SHA-256(challenge:nonce) to that many |
| 227 | leading zero bits during the fader's build phase and submits it with the |
| 228 | create. Verification is one hash; a consumed challenge never buys a second |
| 229 | space. This prices the Sybil loop (invite yourself → fresh account → 3 more |
| 230 | free spaces) in CPU. |
| 231 | |
| 232 | Above all of that sits the **fleet-wide creation budget** |
| 233 | (`LCHAT_CLOUD_MAX_CREATES_PER_DAY`, 0/unset = uncapped): a trailing-24h |
| 234 | circuit breaker counted against an append-only log, so create-and-delete |
| 235 | can't reset it. Past the budget, creation answers the same member-facing |
| 236 | at-capacity message. This is the pre-production insurance dial — the |
| 237 | per-account gates do the fine-grained work; this bounds a bad day. |
| 238 | |
| 239 | On bot detection generally, the position is deliberate: **passkeys are the |
| 240 | bot check**. Creating an account requires a platform-authenticator WebAuthn |
| 241 | ceremony; there is no fingerprinting, UA sniffing, or behavioral scoring |
| 242 | anywhere, and there won't be — the layers are cryptographic (passkeys), |
| 243 | economic (hashcash, storage caps), and quantitative (quotas, rate limits, |
| 244 | the budget, the reaper). **Creation |
| 245 | is also gated to existing members:** `POST /api/spaces` returns `403 not_eligible` |
| 246 | unless the caller already has a backup blob. A blob only exists once you've |
| 247 | enrolled ≥1 space, and enrolling needs a space you were *invited* into — so this |
| 248 | keeps creation invite-gated (not open to the world) while it's early, without the |
| 249 | server ever reading the E2E-encrypted server list. The client hides the "+" when |
| 250 | the list is empty; the 403 is the boundary behind that. Lift it later by removing |
| 251 | the check (the invite-only funnel becomes self-serve). **Deletion is |
| 252 | ceremony-gated** — same rule as revoking devices or deleting the backup: a |
| 253 | stolen link token must never be able to destroy a space. The UI adds its own |
| 254 | stern step (type the space's address to arm the button) before the ceremony, |
| 255 | and the cloud layer re-checks ownership against its ledger regardless: only the |
| 256 | exact (caller, requester) that provisioned a host can delete it, so spaces |
| 257 | added to a list by URL are never deletable through this surface. "Remove from |
| 258 | list" (vault-only, the space keeps running) is a separate, lighter client-side |
| 259 | action. On the box, deletion **archives** the instance's DB/env to |
| 260 | `/data/trash/<host>-<unix>/` rather than destroying them — the operator safety |
| 261 | net until the trash is purged. |
| 262 | |
| 263 | ## What this changes about who-sees-what |
| 264 | |
| 265 | Provisioning is the one place the home server (and the provisioner's ledger) |
| 266 | learns a relationship between a user and an instance hostname — `user X |
| 267 | provisioned host Y`. That's **operational metadata, held outside the blob**, |
| 268 | and it's deliberate: quotas need a ledger. The requester id is the home |
| 269 | server's random opaque uid (no email, no name), the blob stays content-blind, |
| 270 | and a user who never provisions never appears in the ledger. Members who join |
| 271 | by invite are invisible here, unchanged. |
| 272 | |
| 273 | ## Reaping never-claimed spaces |
| 274 | |
| 275 | Self-serve creation hands out a capacity slot the instant a space is made. A |
| 276 | space whose admin never finishes setup — the setup invite expires after 7 days — |
| 277 | would otherwise sit there forever: empty, but holding a slot and a disk. The |
| 278 | **reaper** (in `cloud`, `reaper.go`) reclaims them: |
| 279 | |
| 280 | - Every `LCHAT_CLOUD_REAP_INTERVAL_HOURS` (default 6) it sweeps the ledger for |
| 281 | finalized spaces older than `LCHAT_CLOUD_REAP_AFTER_DAYS` (**opt-in — 0/unset |
| 282 | disables it**; a value comfortably past the 7-day invite expiry, e.g. 14, is |
| 283 | the intent). |
| 284 | - For each candidate it asks the control daemon `GET /instances/{host}/stats`, |
| 285 | which reads the instance DB **read-only** and returns `setup_complete`, member |
| 286 | count, and `has_content`. A candidate is reaped only when **all three guards** |
| 287 | agree it's an abandoned shell: |
| 288 | 1. **`setup_complete` absent** — the admin never finished the setup conversation. |
| 289 | (The signal is `setup_complete`, **not** member count: `add-instance` |
| 290 | pre-seeds the admin user so the setup invite has an identity, and claiming |
| 291 | *becomes* that user — so a space has exactly one member whether or not it was |
| 292 | ever set up.) |
| 293 | 2. **`members <= 1`** — nobody beyond the seeded admin joined. |
| 294 | 3. **`has_content` false** — zero messages and zero threads. The instant anyone |
| 295 | posts anything, the space is protected regardless of the other signals; this |
| 296 | is the decisive guard against a false positive on a used-but-not-marked space. |
| 297 | |
| 298 | **Anything set up, joined, with content, or whose status can't be read is left |
| 299 | untouched** — a stats error is always a skip, never a delete. |
| 300 | - A reaped candidate is deprovisioned through the normal controld delete, which |
| 301 | **archives** the DB/env to `/data/trash` (never destroys), so a false positive |
| 302 | is recoverable. Its ledger row is cleared, freeing the quota slot. |
| 303 | - **Per-sweep cap:** `LCHAT_CLOUD_REAP_MAX_PER_SWEEP` (default 5) bounds how many a |
| 304 | single sweep reclaims; the rest defer to the next sweep. This caps the blast |
| 305 | radius of any future mis-classification to a handful of recoverable archives |
| 306 | before an operator would notice. |
| 307 | - `LCHAT_CLOUD_REAP_DRY_RUN=1` logs exactly what it *would* reclaim without |
| 308 | deleting — run it once before arming the real sweep. |
| 309 | |
| 310 | Because the reaper only ever considers hosts **in the cloud ledger**, operator-run |
| 311 | instances (family.…, home, push, …) are never candidates. |
| 312 | |
| 313 | ### Self-reclaim: a quota-blocked create frees the account's own dead weight |
| 314 | |
| 315 | With the free quota at 1, a fumbled create (made a space, lost the tab, never |
| 316 | set it up) must not lock the account out until the reaper's 14-day sweep. So |
| 317 | when a create hits `quota_exceeded`, the cloud first tries to reclaim **one of |
| 318 | the requester's own abandoned allocations** and, if that frees a slot, proceeds |
| 319 | with the create instead of failing: |
| 320 | |
| 321 | - Candidates are only the same (caller, requester)'s free-entitlement rows |
| 322 | **older than one hour** — setup takes minutes, so the age gate cleanly |
| 323 | separates "abandoned, retrying" from "setting it up in another tab right |
| 324 | now" (the home page keeps the setup link for continuing inside the window). |
| 325 | - A **pending row** (a create that died between allocate and finalize — it |
| 326 | would otherwise hold the slot forever) is cleared ledger-only. |
| 327 | - A **provisioned space** must clear the reaper's exact three guards above |
| 328 | (never set up, nobody joined, zero content); it archives to `/data/trash` |
| 329 | through the same controld delete. A stats error is a skip, never a reclaim. |
| 330 | - Anything in real use stays untouched and the 409 stands — the way out then |
| 331 | is upgrading a space (paid spaces don't count) or deleting one from home. |
| 332 | |
| 333 | **Hibernating spaces never wake for the reaper.** Stats always come from local |
| 334 | state, not from the instance: an *asleep* space is read from its on-disk DB (the |
| 335 | warm cache the activator left at its last sleep — on one box, the newest truth), |
| 336 | and a space that **never woke** has no DB at all, so its stats are definitional — |
| 337 | `members 0, setup_complete false, has_content false`, answered from the registry |
| 338 | row alone. A sweep over thousands of dormant spaces stays a pure metadata read. |
| 339 | Reaping a hibernating space follows the same delete path (unit-stop becomes |
| 340 | best-effort — there is no unit; the route delete + router reload retires any |
| 341 | awake child via the activator's reconcile) and archives to `/data/trash` as |
| 342 | usual. The space's **S3 replica prefix is deliberately not deleted**: controld |
| 343 | holds no bucket credentials, the prefix is opaque ciphertext of a DB whose |
| 344 | authoritative copy just moved to trash, and the versioned bucket's |
| 345 | noncurrent-version expiry means it ages rather than grows — orphan-prefix |
| 346 | cleanup is deferred to the S3 lifecycle work. |
| 347 | |
| 348 | A companion **trash purge** in `controld` reclaims the archive after a longer |
| 349 | window: `LCHAT_TRASH_KEEP_DAYS` (opt-in — 0/unset keeps trash forever) removes |
| 350 | `/data/trash/<host>-<unix>` entries older than that, swept daily. Keep it well |
| 351 | above the reap threshold so a mistaken reap stays recoverable for a while. |
| 352 | |
| 353 | ## Demo spaces (docs/demo-spaces.md slice 1) — 24h, discarded, no backups |
| 354 | |
| 355 | With `LCHAT_DEMO_TTL_HOURS` set (unset = the shipped default, demos off), a |
| 356 | create may carry `kind: "demo"`: an `entitlement='demo'` ledger row that |
| 357 | skips the free-space quota and is bounded instead by `LCHAT_DEMO_MAX_LIVE` |
| 358 | (default 10, pending rows included — refusals answer `demo_busy`) plus the |
| 359 | shared creates-per-day breaker. The create response carries `expires_at`. |
| 360 | Three deliberate inversions of the free-space rules: |
| 361 | |
| 362 | - **Reaped unconditionally** at `created_at + TTL` by an hourly demo sweep — |
| 363 | no stats call, no guards: content protects a free space, never a demo (the |
| 364 | 24h death is the disclosed contract). `LCHAT_CLOUD_REAP_DRY_RUN` is |
| 365 | honoured. The main reaper never sees demos (its candidate query is |
| 366 | free-only). |
| 367 | - **Discarded, not archived**: the reap deletes via controld |
| 368 | `DELETE /instances/{host}?discard=1`, which removes the files outright |
| 369 | (blobs included) instead of renaming into `/data/trash` — the one exception |
| 370 | to archive-everything. |
| 371 | - **Never activator-managed**: controld forces `Hibernate=false` for a demo |
| 372 | create even on a hibernate-by-default box, so a demo never touches S3 — no |
| 373 | lease, no litestream replica, no parked prefix. It holds a materialized |
| 374 | slot for its 24h life; the live-demo cap bounds how many. |
| 375 | |
| 376 | Demos take a `demo` key in `LCHAT_CLOUD_STORAGE_CAPS` (falling back to the |
| 377 | `free` cap when absent). |
| 378 | |
| 379 | **The home-side gates** (`home/demo.go`, on with `LCHAT_HOME_DEMO=1` — |
| 380 | default off): `POST /api/demo/challenge` + `POST /api/demo`, the one |
| 381 | account-free creation path, so the graduated-friction ladder is the whole |
| 382 | boundary — **mandatory adaptive hashcash** (base `LCHAT_DEMO_POW_BITS`, |
| 383 | default 18; +1 bit at 5 creates in the trailing hour and another per |
| 384 | doubling, with the difficulty sealed inside each challenge so escalation |
| 385 | never voids honest work in flight), and **one demo per IP bucket per |
| 386 | cycle** (IPv6 at /56; blocked from creation until expiry + one full TTL |
| 387 | again, so serial recreation converges to one rolling slot). Buckets are |
| 388 | HMACs under a per-process random key — no raw IP is ever stored, nothing |
| 389 | links across a restart, and a failed create never burns the bucket. The |
| 390 | client IP comes from the router's `X-Forwarded-For` stamp. |
| 391 | `GET /api/features` advertises `demo` so clients can show the button only |
| 392 | where it works. **The home surface**: the sign-in card offers "Try a demo |
| 393 | space" (feature-gated, with the honest one-liner about the 24 hours), the |
| 394 | client grinds the hashcash inline — the burn is the wait — and lands in |
| 395 | the fresh space's setup link same-tab. `POST /api/spaces/claim` |
| 396 | (link-token) relays a claim to the cloud so a signed-in account can adopt |
| 397 | a demo as its free space; folding the claimed space into the encrypted |
| 398 | backup is the client's move, like a create. Still to come: the in-space |
| 399 | countdown/claim banner (an instance-side surface). |
| 400 | |
| 401 | **Claiming** (`POST /api/instances/{host}/claim`, caller-token): the happy |
| 402 | exit — a signed-in account adopts a demo, transferring ownership and |
| 403 | flipping `entitlement` to `free` in one guarded UPDATE (a racing claim or |
| 404 | reaper loses cleanly). The claim consumes the account's free-quota slot |
| 405 | (same rules as a create, including the courtesy self-reclaim), refuses |
| 406 | expired demos (`410 demo_expired` — past-TTL is dead by contract even |
| 407 | before the sweep), and re-pushes the instance's entitlement env + free |
| 408 | storage cap + replication class. Claims are never pool-gated: the space's |
| 409 | spend was already counted while it was a demo. Flipping the claimed space |
| 410 | onto the activator (hibernation + S3 backups) is an operator/follow-up |
| 411 | step — the claim itself never restarts anything. |
| 412 | |
| 413 | ## The storage meter (#668 slice 1) |
| 414 | |
| 415 | Every sweep interval (`LCHAT_CLOUD_METER_INTERVAL_HOURS`, default 6, 0 |
| 416 | disables), the cloud service asks controld for each ledger space's on-disk |
| 417 | footprint — the SQLite files plus the attachment blob dir, **pure filesystem |
| 418 | metadata: nothing is read, and a hibernating space is never woken**. The |
| 419 | reading lands on the ledger row (`storage_bytes`, `metered_at`), rides the |
| 420 | space listing, and home shows it to the member ("2.1 GB stored"). Bytes of |
| 421 | ciphertext are exactly the kind of thing this ledger is allowed to know |
| 422 | (#562: price only what ciphertext exposes). |
| 423 | |
| 424 | **Storage caps (#561, insurance-grade).** With `LCHAT_CLOUD_STORAGE_CAPS` set |
| 425 | ("free=2GB,family=5GB,…" — tier keys from the price book plus `free`), the |
| 426 | cloud pushes each ledger space's cap into its instance env |
| 427 | (`LCHAT_BLOB_QUOTA_BYTES`, via controld `POST /instances/{host}/quota`), where |
| 428 | the instance's own upload guard enforces it: uploads past the cap answer 507, |
| 429 | nothing is ever deleted, and text messaging is never gated. Pushes happen on |
| 430 | tier changes (checkout/subscription webhooks), at creation, and the meter |
| 431 | sweep reconciles drift — self-healing, best-effort at every step. A |
| 432 | unit-managed instance is bounced (`try-restart`, seconds) so the cap applies |
| 433 | now; a hibernating one picks it up at next wake — never wake a space to limit |
| 434 | it. Unset = no enforcement, and operator-run instances (not in this ledger) |
| 435 | are never touched. Caps follow the 80/20 margin |
| 436 | rule in `tier-economics.md` (an allowance is what 20% of a tier's net buys |
| 437 | in storage, forever) and always match the published allowance table. |
| 438 | Business is per-seat (`business=11GB/seat` — cap = 11 GB × the space's live |
| 439 | head count, pooled); a big team's pool exceeds the box's disk — honored on |
| 440 | demand (attach a Hetzner volume) until attachments move off-box (#550). |
| 441 | |
| 442 | ## The usage pipe (#857, #668 slices 2–3) — AI usage into the ledger |
| 443 | |
| 444 | With `LCHAT_CLOUD_USAGE_URL` set (the cloud's own public `/api/usage` URL), |
| 445 | the same meter sweep mints each ledger space a **meter token** and pushes it |
| 446 | into the instance env (`LCHAT_METER_URL` + `LCHAT_METER_TOKEN`, via controld |
| 447 | `POST /instances/{host}/meter`, hash-only in the ledger, same self-healing |
| 448 | best-effort shape as the quota push). The flow is sidecar → instance → cloud: |
| 449 | |
| 450 | - The **sidecar** counts model spend — it's the only place spend is visible — |
| 451 | as (day, model, dimension) quantities: `llm_requests`, the four token |
| 452 | counters (input / output / cache-write / cache-read, priced differently), |
| 453 | and `charter_runs`. It flushes every few minutes to its instance |
| 454 | (`POST /api/bot-usage`, bot-token-gated), keeping its egress surface at the |
| 455 | two destinations `docs/bots.md` promises. |
| 456 | - The **instance** is a stateless relay: validate, forward with the meter |
| 457 | token, hand the response's budget state back to the sidecar. Unconfigured |
| 458 | (self-hosted, BYO key) it accepts and drops — answer `"unmetered"`, nothing |
| 459 | recorded anywhere. |
| 460 | - The **cloud** upserts into `usage(space_id, day, model, dimension, |
| 461 | quantity)` — quantities only, never content, never per-member attribution |
| 462 | (#668's rule) — and answers a budget state, `"ok"` unconditionally until |
| 463 | the phase-3 allowance/budget math lands (docs/ai-provisioning.md). |
| 464 | |
| 465 | Unset `LCHAT_CLOUD_USAGE_URL` = the pipe is off fleet-wide: no tokens minted, |
| 466 | no env pushes, instances answer sidecars `"unmetered"`. |
| 467 | **Replication classes (the durability ladder).** With |
| 468 | `LCHAT_CLOUD_REPLICATION_CLASSES` set |
| 469 | (`free=daily,family=daily,community=batched,business=stream` — same key |
| 470 | convention as storage caps), the cloud maps each ledger space's billing state |
| 471 | to a replication class on the activator's S3 replica and pushes it via |
| 472 | controld (`POST /instances/{host}/replication` → the registry route + a router |
| 473 | reload, only on actual change). Same event-driven-plus-sweep shape as caps: |
| 474 | spaces are **born classed** (the create request carries the free class), |
| 475 | Stripe webhooks re-push on every entitlement move (a lapse drops the space |
| 476 | back to the free rung), and the meter sweep reconciles drift. The class is |
| 477 | wake-time config — nothing is restarted, no sleeping space is woken, and an |
| 478 | awake instance keeps its current sync tick until it next sleeps. An unknown |
| 479 | **paid** tier maps to `stream` on purpose: a config typo must fail toward |
| 480 | durability, never away from what the member paid for. Unset = no pushes, |
| 481 | every space streams (the pre-ladder status quo). What the classes mean — |
| 482 | sync cadence, loss windows, and the graceful-flush guarantee — lives with |
| 483 | the mechanism in `registry.go` and the ops guide. |
| 484 | |
| 485 | ## The free-pool gauge (docs/free-pool.md slice 1) |
| 486 | |
| 487 | The free pool runs the free tier as a budget: free + demo spend must never |
| 488 | exceed a configured share of total cost, funded by paid revenue with an |
| 489 | operator floor. **Slice 1 is a gauge only** — it computes and records, gates |
| 490 | nothing, and ships off by default. At the end of each meter sweep the cloud |
| 491 | service estimates every ledger space's daily cost from a declared cost model |
| 492 | (`LCHAT_POOL_COSTS`, defaults = the docs/free-pool.md snapshot), derives |
| 493 | |
| 494 | ``` |
| 495 | budget_day = min( X% × total_cost_day, max(floor, Y% × net_revenue_day) ) |
| 496 | pressure = free_cost_day / budget_day |
| 497 | ``` |
| 498 | |
| 499 | and records one `pool_meter` row (pruned past ~400 days) plus a log line. |
| 500 | Knobs: `LCHAT_POOL_MAX_SHARE` (X — **0/unset = gauge off**), |
| 501 | `LCHAT_POOL_REVENUE_SHARE` (Y, default 30), `LCHAT_POOL_BASE_EUR_DAY` |
| 502 | (default 0.20), `LCHAT_POOL_DEMO_SHARE` (default 20, reserved for the demo |
| 503 | tier). Revenue comes from an additive `net_eur_year` field per tier in |
| 504 | `LCHAT_STRIPE_PRICES` (entries without it fund the pool as €0 and log a |
| 505 | startup warning). `GET /api/pool` (admin token) serves the latest reading — |
| 506 | labelled an estimate, because it is one: derived from declared constants, |
| 507 | not metered. |
| 508 | |
| 509 | **Slice 2 — admission control.** At `pressure ≥ 1` on the latest reading, |
| 510 | new FREE creates answer `503 free_pool_full` with the honest member-facing |
| 511 | message (home's 503 passthrough relays it verbatim). The gate **fails |
| 512 | open** on any uncertainty — pool off, no reading yet, ledger error — and |
| 513 | never touches anything else: demos are exempt (their carve-out + TTL bound |
| 514 | them), paid creates are never gated (they fund the pool), existing spaces |
| 515 | are never touched, and deleting your own space still works at red. |
| 516 | `LCHAT_POOL_DRY_RUN=1` computes and logs every would-refuse without |
| 517 | refusing — arm it the reaper's way: gauge first, dry-run second, live |
| 518 | third. Budget + pressure now use revenue **smoothed over the trailing ~90 |
| 519 | days** of stored readings (one big churn shrinks the pool gradually); each |
| 520 | `pool_meter` row still records the instantaneous figure. |
| 521 | |
| 522 | ## Billing (Stripe) — tiers, quota exemption, no metering |
| 523 | |
| 524 | Optional space upgrades (`billing.go`): a person who created a space can pay a |
| 525 | subscription to upgrade it at one of a small set of **tiers** (Family / Friends / |
| 526 | Business — config, not code). Per-tier powers (storage, member counts, bots) and |
| 527 | metering are #561/#562 and layer on later; today a paid space of ANY tier gets |
| 528 | the same two things. |
| 529 | |
| 530 | **The flow.** The home page shows a quiet *Upgrade* on spaces this account |
| 531 | created (only when the provisioner advertises billing). It opens a plan picker |
| 532 | (the price book from the listing response); the chosen tier key goes home → |
| 533 | cloud, which mints a Stripe **Checkout Session** for that tier's price; the |
| 534 | browser goes to Stripe's hosted page and comes back to home |
| 535 | (`?upgraded=<host>`). Stripe then confirms through the **webhook** — |
| 536 | `checkout.session.completed` flips the space's ledger row to `paid` and records |
| 537 | the tier, and `customer.subscription.updated/deleted` track it for the life of |
| 538 | the subscription (`past_due` keeps access through Stripe's dunning window; |
| 539 | deletion drops the space back to `free`). The row's numeric ledger id rides in |
| 540 | the session's `client_reference_id` and the subscription's metadata, the tier |
| 541 | key in both metadata blobs — that's how events route back to a space. |
| 542 | |
| 543 | Creating a space itself is **never pay-gated** (#1374): the create step is |
| 544 | just the name, and "who is this space for?" is asked by Eleven inside the |
| 545 | space, priced from the cloud's `GET /api/pricebook` (caller-token'd; home |
| 546 | republishes it unauthenticated and CORS-open at its own `/api/pricebook`, |
| 547 | cached, for space origins to read). The create screen's tier cards appear |
| 548 | only when a create bounces off the free quota or the pool gate — the two |
| 549 | errors paying genuinely fixes; picking one there runs the pay-first |
| 550 | checkout-create path below. |
| 551 | |
| 552 | **What paying does today:** |
| 553 | |
| 554 | - a paid space **doesn't count toward the creation quota** (`allocateSpace` |
| 555 | counts only free spaces — upgrading a space frees its slot, and every extra |
| 556 | space carries itself); |
| 557 | - a paid space is **never a reap candidate** (excluded in the ledger query |
| 558 | itself, the strongest of the guards above); |
| 559 | - the entitlement + tier come back on space listings so clients render the |
| 560 | badge. Per-space limits read the same columns when #561 lands. |
| 561 | |
| 562 | **Privacy (the point):** the ledger stores two opaque Stripe ids |
| 563 | (`stripe_customer_id`, `stripe_subscription_id`), the `entitlement` flag, and |
| 564 | `paid_until`. **Never an email, name, address, or card detail** — Stripe holds |
| 565 | those on its side, and the operator can see them in Stripe's dashboard. |
| 566 | Paying is not anonymous; the invariant (#562's documented residual) is that |
| 567 | the *ledger* carries no billing identity. Checkout Sessions are created with no customer |
| 568 | fields at all. The webhook is the one public unauthenticated-by-token endpoint; |
| 569 | its gate is Stripe's HMAC signature over the raw payload (±5 min replay window). |
| 570 | |
| 571 | **Config** (`/data/cloud.env`; all-or-nothing — partial config refuses to start): |
| 572 | |
| 573 | ``` |
| 574 | LCHAT_STRIPE_SECRET_KEY restricted API key (write: Checkout Sessions; read: nothing else needed) |
| 575 | LCHAT_STRIPE_WEBHOOK_SECRET the webhook endpoint's signing secret |
| 576 | LCHAT_STRIPE_PRICES the price book: a JSON array of tiers, e.g. |
| 577 | [{"key":"family","name":"Family","label":"$5/year","price":"price_…"}, …] |
| 578 | (display strings live here verbatim — the server never |
| 579 | asks Stripe what a price costs, so a Stripe outage can't |
| 580 | block a cloud restart; the checkout charges the real price) |
| 581 | Optional per-tier: "eur" (numeric sticker, feeds the |
| 582 | picker's ≈ local-currency display), "business": true |
| 583 | (ex-VAT tier: checkout collects a VAT ID), and "bands" |
| 584 | — the honor prices of docs/pricing-bands.md: |
| 585 | {"supporter"|"reduced"|"solidarity": |
| 586 | {"label","price","eur","net_eur_year"}}. |
| 587 | A tier without bands behaves exactly as before; a |
| 588 | price id may appear only once in the whole book. |
| 589 | LCHAT_STRIPE_API_BASE default https://api.stripe.com (tests point it at a stub) |
| 590 | LCHAT_STRIPE_AUTOMATIC_TAX "1" opts Checkout Sessions into Stripe Tax |
| 591 | (docs/pricing-bands.md § VAT). Flip it WITH the |
| 592 | band-price rollout — every live Price must carry a |
| 593 | tax_behavior (or an account default) first. |
| 594 | ``` |
| 595 | |
| 596 | Unset = billing off: checkout answers 501, listings say `billing: false`, the |
| 597 | UI shows nothing. The Stripe-side webhook endpoint must subscribe to |
| 598 | `checkout.session.completed` and `customer.subscription.*`, pointed at |
| 599 | `https://<cloud host>/api/stripe/webhook`. |
| 600 | |
| 601 | **Managing a subscription** is Stripe's hosted **Billing Portal**: Manage on a |
| 602 | paid space's row → home → cloud mints a portal session for that space's own |
| 603 | Stripe customer (same ownership gate as checkout). The portal also offers |
| 604 | **plan switching** (upgrade/downgrade, monthly↔annual, prorated by Stripe); |
| 605 | the webhook derives the tier from the subscription's current PRICE (metadata |
| 606 | only seeds the first value), so the badge and the pushed storage cap follow a |
| 607 | switch automatically. **Cancelling** is at-period-end: the space keeps its |
| 608 | tier until the paid time runs out, then the deletion event drops it to free — |
| 609 | the space survives, its storage cap returns to the free allowance (existing |
| 610 | content is never deleted; only new uploads pause if over), and it counts |
| 611 | toward the creation quota again. Portals are per-space by |
| 612 | construction — every space is a separate Stripe customer, deliberately (no |
| 613 | cross-space identity exists to merge). `LCHAT_STRIPE_PORTAL_CONFIG` optionally |
| 614 | pins a portal configuration id; unset uses the account's default (which must |
| 615 | exist — save the portal settings once in the Stripe dashboard, or create a |
| 616 | configuration via the API). With bands, `LCHAT_STRIPE_PORTAL_CONFIGS` |
| 617 | (JSON `{band: configuration id}`) picks a per-band configuration — each |
| 618 | band's portal offers plan switches inside that band only; band changes go |
| 619 | through the home surface, where they can be messaged. Bands without an entry |
| 620 | fall back to `LCHAT_STRIPE_PORTAL_CONFIG`. |
| 621 | |
| 622 | ## Enabling it on a box |
| 623 | |
| 624 | ```sh |
| 625 | lchat bootstrap # installs units, mints /etc/lchat/control.token |
| 626 | systemctl enable --now lchat-control |
| 627 | cat > /data/cloud.env <<EOF # then: chown lchat:lchat, chmod 600 |
| 628 | LCHAT_ADMIN_TOKEN=<mint one> |
| 629 | LCHAT_CLOUD_BASE_DOMAIN=<domain with wildcard DNS at this box> |
| 630 | LCHAT_CLOUD_REAP_AFTER_DAYS=14 # optional: reclaim never-claimed spaces (0/unset = off) |
| 631 | EOF |
| 632 | systemctl enable --now lchat-cloud |
| 633 | # optional trash purge (controld): set LCHAT_TRASH_KEEP_DAYS in /data/control.env, restart lchat-control |
| 634 | lchat add-service cloud.<domain> --socket /run/lchat/cloud.sock |
| 635 | # mint the home server's caller token: |
| 636 | curl -sX POST https://cloud.<domain>/api/admin/tokens \ |
| 637 | -H "Authorization: Bearer $ADMIN" -d '{"label":"home.<domain>"}' |
| 638 | # hand it to home: LCHAT_CLOUD_URL + LCHAT_CLOUD_TOKEN in /data/home.env, restart lchat-home |
| 639 | ``` |
| 640 | |
| 641 | ## Phase 2 (designed for, not built) |
| 642 | |
| 643 | - **App UI** — "+ → Sign in (home.elevenmessenger.com) → new space", and the |
| 644 | server list syncing/sorting through the existing encrypted blob. |
| 645 | - ~~**Bot sidecars on provisioned spaces**~~ — SHIPPED (#858): controld's |
| 646 | `POST /instances/{host}/bot` stages an **Eleven-only** sidecar (idempotent; |
| 647 | the model key comes from controld's own env, never the instance's). The |
| 648 | trigger is the admin's AI opt-in: setup's dedicated `ai-optin` step (or any |
| 649 | later flip of the AI toggle) makes the instance signal |
| 650 | `POST /api/ai-optin` on the cloud over its meter token, and the cloud calls |
| 651 | controld. Hibernating hosts stage the env only — the activator co-spawns at |
| 652 | wake, and its sweep adopts an already-awake child within seconds. Meter |
| 653 | tokens are now also minted eagerly at create, so a fresh space can opt in |
| 654 | during onboarding without waiting for the sweep. |
| 655 | - **Multi-box placement** — a cloud front choosing among several router boxes. |
| 656 | |
amadan