| 1 | <!doctype html> |
| 2 | <html lang="en"> |
| 3 | <head> |
| 4 | <meta charset="utf-8" /> |
| 5 | <meta name="viewport" content="width=device-width, initial-scale=1" /> |
| 6 | <title>Creating and removing chat spaces — Running the fleet</title> |
| 7 | <link rel="stylesheet" href="guide.css" /> |
| 8 | <script src="guide.js" defer></script> |
| 9 | </head> |
| 10 | <body data-page="spaces.html"> |
| 11 | <div class="frame"> |
| 12 | <nav class="side"></nav> |
| 13 | <main class="content"> |
| 14 | <div class="wrap"> |
| 15 | |
| 16 | <header class="top"> |
| 17 | <p class="eyebrow"><span class="robot">🤖</span> Running the fleet · Part 05 of 10</p> |
| 18 | <h1>Creating and removing chat spaces</h1> |
| 19 | <p class="thesis">A new chat space is a hostname, a minted admin token, an environment file, a registry row, and (usually) a started process. This page walks the manual commands first — <strong><code>add-instance</code> and <code>remove</code></strong> — and then the self-serve pipeline that lets signed-in users create their own spaces: <strong>the control daemon, the cloud provisioner, and the reaper</strong> that quietly reclaims spaces nobody ever claimed.</p> |
| 20 | </header> |
| 21 | |
| 22 | <div class="sysmap" data-highlight="control"></div> |
| 23 | <p class="sysmap-caption">You are here: the control daemon and cloud provisioner. The manual commands on this page do the same work the control daemon does, from your terminal instead of over its socket.</p> |
| 24 | |
| 25 | <h2 id="adding-manual">Adding a space by hand: <code>lchat add-instance</code></h2> |
| 26 | <p>One command creates a fully working space. Before running it, the only prerequisite is DNS: the hostname must resolve to the box. With a wildcard DNS record (<code>*.example.com → the box’s address</code>) every subdomain already resolves, so there is no per-space DNS step at all.</p> |
| 27 | <figure class="cmd"> |
| 28 | <figcaption>Run on the box, as root</figcaption> |
| 29 | <pre><code>lchat add-instance chat-c.example.com |
| 30 | <span class="out">setup link: https://chat-c.example.com/#setup=Xk3… (one-time; give this to the space’s owner) |
| 31 | admin link: https://chat-c.example.com/admin#token=9f2…</span></code></pre> |
| 32 | </figure> |
| 33 | <p>What just happened, in order — this sequence is worth reading once carefully, because <code>remove</code>, hibernation, and migration all make sense in terms of it:</p> |
| 34 | <ol> |
| 35 | <li><strong>Validate and register.</strong> The hostname is checked (it becomes a filename, a unit name, and a socket path, so it must be a clean domain name), an admin token is minted, and a row is written to the <a href="router.html#the-registry">registry</a> with status <code>creating</code>.</li> |
| 36 | <li><strong>Write the environment file.</strong> <code>/data/chat-c.example.com.env</code> (mode 0600) receives <code>LCHAT_ADMIN_TOKEN=…</code> plus any per-instance settings passed as flags. Remember from <a href="processes.html#data">part 03</a>: the database will store only a <em>hash</em> of the admin token, so this file is the token’s only recoverable copy.</li> |
| 37 | <li><strong>Point the version link.</strong> The currently running build is staged into the version store if it is not already there, and <code>/opt/lchat/live/chat-c.example.com</code> is created pointing at it — so the new instance starts on a known build (see <a href="deploys.html#mental-model">part 04</a>).</li> |
| 38 | <li><strong>Start and health-check the process.</strong> <code>systemctl enable --now lchat@chat-c.example.com</code>, then wait up to fifteen seconds for the socket to answer. On failure the row is marked <code>failed</code> with the error recorded (visible in <code>lchat instances</code>), and the files are left in place for diagnosis.</li> |
| 39 | <li><strong>Publish the route and warm the certificate.</strong> The registry row flips to <code>ready</code>, the router reloads, and the command makes one HTTPS request through the router with the real hostname — which triggers the Let’s Encrypt issuance (<a href="router.html#http01">part 02</a>) so the first member does not pay that wait.</li> |
| 40 | <li><strong>Mint the setup invite.</strong> A one-time setup link is created (this also pre-seeds the space’s admin member). The setup link is what you hand to the person who asked for the space; the admin link is the operator-level console.</li> |
| 41 | </ol> |
| 42 | <p>Useful flags, all optional: <code>--admin-token <tok></code> imports an existing token instead of minting one (essential when <a href="operations.html#migration">migrating</a> a space between boxes, because a new token would invalidate every saved admin link); <code>--push-relay-url</code> / <code>--push-relay-token</code>, <code>--home-origin</code>, <code>--app-store-url</code> record per-instance service configuration into the environment file; and <code>--hibernate</code> registers the space as activator-managed from birth (<a href="#hibernating-creates">below</a>).</p> |
| 43 | <div class="warn"> |
| 44 | <span class="label">Store the admin token somewhere safe, immediately</span> |
| 45 | <p>Treat the printed admin link like a password: store it in your secrets manager as soon as the command returns. If it is lost, <code>lchat admin-link chat-c.example.com</code> recovers it from the environment file — but only while that file exists on that box.</p> |
| 46 | </div> |
| 47 | |
| 48 | <h2 id="removing">Removing a space: <code>lchat remove</code>, and why deletion archives</h2> |
| 49 | <p>Removal is deliberately layered, because “stop serving this hostname” and “destroy this data” are different decisions:</p> |
| 50 | <ol class="procedure"> |
| 51 | <li><p class="step-title">Take the space off the internet (data untouched).</p> |
| 52 | <figure class="cmd"> |
| 53 | <figcaption>Run on the box, as root</figcaption> |
| 54 | <pre><code>lchat remove chat-c.example.com --stop</code></pre> |
| 55 | </figure> |
| 56 | <p><code>remove</code> deletes the registry row and reloads the router, so the hostname stops being served; <code>--stop</code> also stops and disables the instance unit and its bot sidecar unit. Without <code>--stop</code>, the process keeps running but receives no traffic. Either way, <strong>the database, blobs, and environment file remain on disk</strong> — re-running <code>add-instance</code> for the same host (with <code>--admin-token</code> from the old environment file) brings the space straight back.</p></li> |
| 57 | <li><p class="step-title">Only if the data should go: archive it into the trash.</p> |
| 58 | <p>Full deletion — the path the self-serve pipeline uses — moves the database, environment file, write-ahead log, push key, and blob directory into <code>/data/trash/<host>-<timestamp>/</code>, and removes the certificate and version link. Nothing in this system unlinks user data directly; the trash directory is the safety net, and a mistaken deletion is recovered by moving the files back and re-running <code>add-instance</code>.</p></li> |
| 59 | <li><p class="step-title">Eventually, the trash is purged (only if configured).</p> |
| 60 | <p>The control daemon sweeps <code>/data/trash</code> daily and deletes archives older than <code>LCHAT_TRASH_KEEP_DAYS</code> days. The default is <code>0</code>, which means <strong>keep forever</strong> — purging is opt-in, and the retention window should comfortably exceed the reaper’s thresholds below so that any automated reclaim stays reversible for weeks.</p></li> |
| 61 | </ol> |
| 62 | <div class="plain"> |
| 63 | <span class="label">In plain words</span> |
| 64 | <p>Removing a space is like taking a folder out of the filing cabinet and putting it in a banker’s box in the basement, with the date written on it. Only a separately configured janitor ever shreds the boxes, and by default there is no janitor.</p> |
| 65 | </div> |
| 66 | <p>One deliberate exception: for a <a href="hibernation.html">hibernating</a> space, the parked copy of its database in the S3 bucket is <em>not</em> deleted by the deletion path — the box’s control daemon holds no bucket credentials on purpose. Cleaning old parked prefixes is left to the bucket’s lifecycle rules.</p> |
| 67 | |
| 68 | <h2 id="self-serve">The self-serve pipeline: how a member creates a space with no operator</h2> |
| 69 | <p>Members of the product can create new spaces themselves (“Create a space” in the app). No human runs <code>add-instance</code> for those — instead a request flows through three services, each with a narrow job, and understanding the chain makes the failure modes obvious:</p> |
| 70 | <figure class="diagram"> |
| 71 | <svg viewBox="0 0 740 150" role="img" aria-label="Diagram: a member's device calls the home server, which calls the cloud provisioner, which calls the control daemon, which creates the instance."> |
| 72 | <rect x="16" y="50" width="130" height="52" rx="7" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| 73 | <text x="81" y="72" font-size="11.5" font-weight="600" text-anchor="middle">member’s device</text> |
| 74 | <text x="81" y="88" font-size="9.5" fill="var(--muted)" text-anchor="middle">signed in to home</text> |
| 75 | <rect x="196" y="50" width="130" height="52" rx="7" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| 76 | <text x="261" y="72" font-size="11.5" font-weight="600" text-anchor="middle">home server</text> |
| 77 | <text x="261" y="88" font-size="9.5" fill="var(--muted)" text-anchor="middle">eligibility gate</text> |
| 78 | <rect x="376" y="50" width="130" height="52" rx="7" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| 79 | <text x="441" y="72" font-size="11.5" font-weight="600" text-anchor="middle">cloud provisioner</text> |
| 80 | <text x="441" y="88" font-size="9.5" fill="var(--muted)" text-anchor="middle">policy: quotas, names</text> |
| 81 | <rect x="556" y="50" width="130" height="52" rx="7" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| 82 | <text x="621" y="72" font-size="11.5" font-weight="600" text-anchor="middle">control daemon</text> |
| 83 | <text x="621" y="88" font-size="9.5" fill="var(--muted)" text-anchor="middle">capacity + creation</text> |
| 84 | <line x1="146" y1="76" x2="196" y2="76" stroke="var(--muted)" stroke-width="1.2"></line> |
| 85 | <line x1="326" y1="76" x2="376" y2="76" stroke="var(--muted)" stroke-width="1.2"></line> |
| 86 | <line x1="506" y1="76" x2="556" y2="76" stroke="var(--muted)" stroke-width="1.2"></line> |
| 87 | <text x="171" y="66" font-size="9" fill="var(--muted)" text-anchor="middle">HTTPS</text> |
| 88 | <text x="351" y="66" font-size="9" fill="var(--muted)" text-anchor="middle">caller token</text> |
| 89 | <text x="531" y="66" font-size="9" fill="var(--muted)" text-anchor="middle">root-only token,</text> |
| 90 | <text x="531" y="76" font-size="9" fill="var(--muted)" text-anchor="middle">local socket</text> |
| 91 | <text x="370" y="130" font-size="10" fill="var(--muted)" text-anchor="middle">Each hop narrows trust: the internet-facing services hold policy; only the local, root-only daemon can actually change the box.</text> |
| 92 | </svg> |
| 93 | </figure> |
| 94 | |
| 95 | <h3 id="controld">The control daemon (<code>lchat controld</code>): the only process allowed to change the box</h3> |
| 96 | <p>The control daemon is the “god surface”: it is the one process permitted to run <code>systemctl</code> and write the software store, which is why it runs as root, listens <strong>only on a local unix socket</strong> (<code>/run/lchat/control.sock</code>), and is never given a public hostname. Its API is four endpoints — create an instance, delete an instance, report a space’s statistics, set a space’s storage quota — and every call must present a bearer token whose only copy lives in the root-only file <code>/etc/lchat/control.token</code>. (Socket file permissions alone cannot be the gate, because every sandboxed service on the box shares the same <code>lchat</code> user; the root-only token is what distinguishes the cloud provisioner, which is handed the token by systemd’s <code>LoadCredential</code> mechanism, from a hypothetically compromised instance, which cannot read it.)</p> |
| 97 | <p>Its create endpoint enforces <strong>capacity</strong> before doing anything: at most <code>LCHAT_MAX_INSTANCES</code> running instances (default 40), and at least <code>LCHAT_MIN_FREE_GB</code> of free disk (default 5). At capacity it answers <code>503 at_capacity</code> and the refusal propagates up the chain as a polite “try again later” to the member. Configure these in <code>/data/control.env</code> and restart <code>lchat-control</code>.</p> |
| 98 | |
| 99 | <h3 id="cloud">The cloud provisioner (<code>lchat cloud</code>): policy, in public</h3> |
| 100 | <p>The cloud provisioner is the internet-facing half, running unprivileged behind the router like any other service (registered with <code>lchat add-service</code>). It holds everything that is <em>policy</em> rather than <em>power</em>:</p> |
| 101 | <ul> |
| 102 | <li><strong>Caller tokens.</strong> Services that may request creations (in practice: the home server) authenticate with tokens the operator mints at runtime — <code>POST /api/admin/tokens</code> with the provisioner’s admin token; tokens are stored hashed and are revocable individually.</li> |
| 103 | <li><strong>Quotas.</strong> Each requesting user may hold at most <code>LCHAT_CLOUD_QUOTA</code> (default 3) <em>free</em> spaces at a time; a deleted space frees its slot, and a paid space does not count. There is also an optional fleet-wide circuit breaker, <code>LCHAT_CLOUD_MAX_CREATES_PER_DAY</code>, whose count deliberately survives create-and-delete games.</li> |
| 104 | <li><strong>Names.</strong> New spaces get short subdomains like <code>k3f.example.com</code> — an encoding (sqids) of the space’s ledger row number, so names are unique by construction, never reused, and carry no information. The base domain comes from <code>LCHAT_CLOUD_BASE_DOMAIN</code> and needs the wildcard DNS record.</li> |
| 105 | <li><strong>The ledger.</strong> Its own small database (<code>/data/cloud.db</code>) records who created what, plus the dormant billing rails: an opaque Stripe customer pointer and an entitlement flag per space — deliberately no email, name, or card data in the ledger itself (identity lives in Stripe, where the operator can see it). <span class="status partial">partially built</span> (dormant until the Stripe environment variables are set).</li> |
| 106 | </ul> |
| 107 | <p>Upstream of both sits the <strong>home server’s eligibility gate</strong>: it forwards a creation request only for a signed-in account that already holds at least one space backup, which keeps space creation invitation-shaped without the server reading anything encrypted. The member’s identity crosses to the provisioner only as an opaque random identifier — the quota is per that identifier, and no personal information flows down the chain.</p> |
| 108 | |
| 109 | <h3 id="enabling-self-serve">Enabling the pipeline on a box, once</h3> |
| 110 | <ol class="procedure"> |
| 111 | <li><p class="step-title">Start the control daemon.</p> |
| 112 | <figure class="cmd"><figcaption>Run on the box, as root</figcaption> |
| 113 | <pre><code>systemctl enable --now lchat-control # the token at /etc/lchat/control.token was minted by bootstrap</code></pre></figure></li> |
| 114 | <li><p class="step-title">Configure and start the cloud provisioner.</p> |
| 115 | <figure class="cmd"><figcaption>Run on the box, as root — /data/cloud.env, then start</figcaption> |
| 116 | <pre><code>cat > /data/cloud.env <<'EOF' |
| 117 | LCHAT_ADMIN_TOKEN=<a fresh secret for the provisioner’s own admin API> |
| 118 | LCHAT_CLOUD_BASE_DOMAIN=example.com |
| 119 | LCHAT_CLOUD_QUOTA=3 |
| 120 | EOF |
| 121 | chown lchat:lchat /data/cloud.env && chmod 600 /data/cloud.env |
| 122 | systemctl enable --now lchat-cloud</code></pre></figure></li> |
| 123 | <li><p class="step-title">Give the provisioner a public hostname through the router.</p> |
| 124 | <figure class="cmd"><figcaption>Run on the box, as root</figcaption> |
| 125 | <pre><code>lchat add-service cloud.example.com --socket /run/lchat/cloud.sock</code></pre></figure></li> |
| 126 | <li><p class="step-title">Mint a caller token and hand it to the home server.</p> |
| 127 | <figure class="cmd"><figcaption>Run on the box</figcaption> |
| 128 | <pre><code>curl -s -X POST https://cloud.example.com/api/admin/tokens \ |
| 129 | -H "Authorization: Bearer <the provisioner admin token>" -d '{"label":"home"}' |
| 130 | <span class="out">{"id":1,"token":"<shown exactly once — copy it now>"}</span> |
| 131 | # then add to /data/home.env: LCHAT_CLOUD_URL=https://cloud.example.com |
| 132 | # LCHAT_CLOUD_TOKEN=<the minted token> |
| 133 | systemctl restart lchat-home</code></pre></figure></li> |
| 134 | </ol> |
| 135 | |
| 136 | <h2 id="reaper">The reaper: reclaiming spaces nobody ever claimed</h2> |
| 137 | <p>Self-serve creation produces abandonment: someone creates a space, never finishes setting it up, and its slot counts against capacity forever. The reaper is the cloud provisioner’s answer, and it is engineered to be terrified of false positives:</p> |
| 138 | <ul> |
| 139 | <li><strong>Only ledger spaces are candidates.</strong> It considers only spaces the self-serve pipeline itself created (rows in <code>/data/cloud.db</code>) — operator-created instances, the home server, and the push relay are structurally not candidates. Paid spaces are excluded at the query level.</li> |
| 140 | <li><strong>“Never claimed” requires three signals to agree.</strong> A space is reclaimed only if its setup was never completed, <em>and</em> it has at most its one pre-seeded admin member, <em>and</em> it contains no messages or threads at all. Any error reading a space’s statistics is a skip, never a reap. (A hibernating space that has never even woken has no database file; it is answered definitionally from its registry row rather than being treated as an error — otherwise register-only spaces could never be reclaimed.)</li> |
| 141 | <li><strong>It is off by default and bounded when on.</strong> Enable by setting <code>LCHAT_CLOUD_REAP_AFTER_DAYS</code> (choose a value comfortably past the seven-day life of the setup invite — fourteen is sensible); it sweeps every <code>LCHAT_CLOUD_REAP_INTERVAL_HOURS</code> (default 6), reclaims at most <code>LCHAT_CLOUD_REAP_MAX_PER_SWEEP</code> (default 5) spaces per sweep, and honours <code>LCHAT_CLOUD_REAP_DRY_RUN=1</code>, which logs what it <em>would</em> reclaim. Run the dry run for a few days before trusting it.</li> |
| 142 | <li><strong>Reaping is the archiving deletion</strong> described <a href="#removing">above</a> — into <code>/data/trash</code>, recoverable until the (separately configured, default-never) trash purge.</li> |
| 143 | </ul> |
| 144 | |
| 145 | <h2 id="hibernating-creates">Spaces born asleep</h2> |
| 146 | <p>On a box with <a href="hibernation.html">hibernation</a> enabled, creation can skip starting a process at all: <code>add-instance --hibernate</code> (or, for the whole self-serve pipeline, <code>LCHAT_CONTROL_HIBERNATE_CREATES=1</code> in <code>/data/control.env</code>) registers the space as activator-managed from birth. No systemd unit is created and no process runs; the first request to the hostname performs the first wake, which finds no parked database in the bucket and simply starts fresh. Capacity accounting changes accordingly: hibernating registrations are bounded by a row ceiling (<code>LCHAT_MAX_REGISTERED</code>, default ten times <code>LCHAT_MAX_INSTANCES</code>) rather than the running-process cap, because a sleeping space costs disk, not memory.</p> |
| 147 | <div class="roadmap"> |
| 148 | <span class="label">Partially built: creation is cheap, not yet free</span> |
| 149 | <p>The design’s end state is that creating a space writes <em>only</em> a registry row and an object-storage prefix — microseconds, no files, with everything else materialising at first wake. Today’s hibernating create is close but honest about the gap: it still writes the environment file, stages the binary, reloads the router, and performs one immediate wake, because minting the setup invite needs a running instance to pre-seed the admin member. Fully deferred materialisation remains roadmap, as does the eventual dissolution of the control daemon into the registry-only model.</p> |
| 150 | </div> |
| 151 | |
| 152 | </div> |
| 153 | </main> |
| 154 | </div> |
| 155 | </body> |
| 156 | </html> |
| 157 | |