| 1 | <!doctype html> |
| 2 | <html lang="en"> |
| 3 | <head> |
| 4 | <meta charset="utf-8" /> |
| 5 | <meta name="viewport" content="width=device-width, initial-scale=1" /> |
| 6 | <title>Complete reference — Running the fleet</title> |
| 7 | <link rel="stylesheet" href="guide.css" /> |
| 8 | <script src="guide.js" defer></script> |
| 9 | </head> |
| 10 | <body data-page="reference.html"> |
| 11 | <div class="frame"> |
| 12 | <nav class="side"></nav> |
| 13 | <main class="content"> |
| 14 | <div class="wrap"> |
| 15 | |
| 16 | <header class="top"> |
| 17 | <p class="eyebrow"><span class="robot">🤖</span> Running the fleet · Part 10 of 10</p> |
| 18 | <h1>Complete reference: commands, files, variables</h1> |
| 19 | <p class="thesis">Everything in one place, for looking up rather than reading through: <strong>every operator command with its flags, every environment variable with its default, every file path, every systemd unit, and the glossary.</strong> Each entry links back to the page that explains it.</p> |
| 20 | </header> |
| 21 | |
| 22 | <div class="sysmap" data-highlight=""></div> |
| 23 | <p class="sysmap-caption">The map one last time, unhighlighted. If a term below is unfamiliar, the <a href="#glossary">glossary</a> at the bottom of this page defines it.</p> |
| 24 | |
| 25 | <h2 id="commands">Command reference: the <code>lchat</code> control commands</h2> |
| 26 | <p>All of these run on a box (over <code>ssh</code>, which rides the WireGuard tunnel — <a href="operations.html#hardening">part 08</a>), as a named operator invoking <code>/opt/lchat/lchat</code> through the committed sudo allowlist (root SSH is for first bootstrap only). Common conventions: <code><host></code> is always a full domain name; flags may come before or after the host; most commands accept <code>--registry <path></code> to point at a non-default registry (you will essentially never need it — the default is <code>/data/registry.db</code>).</p> |
| 27 | |
| 28 | <h3 id="cmd-lifecycle">Box and instance lifecycle</h3> |
| 29 | <dl> |
| 30 | <dt><code>lchat bootstrap [--links-apex <host>] [--no-update-timer] [--no-start] [--collectors-only]</code></dt> |
| 31 | <dd>One-time box setup: writes every systemd unit file, creates the <code>lchat</code> system user and the <code>/data</code>, <code>/run/lchat</code>, and certificate-cache directories, mints the root-only control token, and enables and starts the router and the hourly update timer. Idempotent — safe to re-run to refresh unit files, and re-running is the <em>only</em> thing that installs a newly-added unit on an existing box (<code>deploy</code> restarts only units that already exist). <code>--links-apex <host></code> makes this box the <a href="#cmd-links">short-link apex</a> for that host; without it, the box converges to whatever the registry already says. <code>--no-update-timer</code> <strong>disables</strong> the hourly auto-updater rather than enabling it, for boxes that must never track a release channel — the dev mesh runs unmerged code, so an updater pass would replace it with the current stable build. It disables rather than merely skips, so the end state follows the flag rather than the box’s history. Explained in <a href="deploys.html#bootstrap">part 04</a>.</dd> |
| 32 | <dt><code>lchat add-instance <host> [--admin-token <tok>] [--hibernate] [--home-region <cc>] [--app-store-url <url>] [--home-origin <url>] [--push-relay-url <url>] [--push-relay-token <tok>]</code></dt> |
| 33 | <dd>Creates a space: mints (or, with <code>--admin-token</code>, imports) the admin token, writes <code>/data/<host>.env</code>, registers the route, points the version link, starts and health-checks the unit (skipped with <code>--hibernate</code>), reloads the router, warms the certificate, and prints the one-time setup link and the admin link. Explained in <a href="spaces.html#adding-manual">part 05</a>.</dd> |
| 34 | <dt><code>lchat add-service <host> --socket <path></code></dt> |
| 35 | <dd>Registers a <code>service</code>-kind route (home server, push relay, cloud provisioner, or any third-party backend) after verifying the socket answers. No unit is created and no version is tracked — the router only forwards. Explained in <a href="processes.html#sibling-services">part 03</a>.</dd> |
| 36 | <dt><code>lchat add-bot <host> [--model <model>]</code></dt> |
| 37 | <dd>Provisions the bot sidecar for an instance: mints one agent invite per identity, writes <code>/data/<host>-bot.env</code> and the session directory, and enables <code>lchat-bot@<host></code> (on a hibernating host the unit is skipped; the activator co-spawns instead). Reads <code>ANTHROPIC_API_KEY</code> from the calling environment if AI behaviour is wanted. Explained in <a href="processes.html#bot-sidecars">part 03</a>.</dd> |
| 38 | <dt><code>lchat add-hooks <host></code> <span class="status partial">built, not yet deployed</span></dt> |
| 39 | <dd>Provisions the webhook-ingest sidecar for an instance (incoming webhooks, issue #610): mints the service identity’s agent invite, writes <code>/data/<host>-hooks.env</code> and the session directory, registers the <code>in.<host></code> registry route (kind <code>hooks</code>) pointing at <code>/run/lchat/<host>-hooks.sock</code>, and enables <code>lchat-hooks@<host></code>. Admins then create webhooks from the app’s Webhooks panel. Design and trust model: <code>docs/webhooks.md</code>.</dd> |
| 40 | <dt><code>lchat remove <host> [--stop]</code></dt> |
| 41 | <dd>Deletes the host’s registry row and reloads the router (traffic stops); with <code>--stop</code> also stops and disables the instance and sidecar units. <strong>Never touches data.</strong> Explained in <a href="spaces.html#removing">part 05</a>.</dd> |
| 42 | <dt><code>lchat hibernate <host> --on | --off</code></dt> |
| 43 | <dd>Moves one host between systemd management and activator management, in either direction, safely ordered so the host is never unserveable. Requires the router to be running with a hibernation bucket for <code>--on</code>. Explained in <a href="hibernation.html#host-switch">part 06</a>.</dd> |
| 44 | <dt><code>lchat replication <host> [stream | batched | daily]</code></dt> |
| 45 | <dd>Reads (no argument) or sets a hibernating host’s <a href="hibernation.html#replication-classes">replication class</a> — how often awake writes sync to the bucket. A set reloads the router and applies at the host’s next wake; on provisioned fleets billing drives this instead. Explained in <a href="hibernation.html#replication-classes">part 06</a>.</dd> |
| 46 | <dt><code>lchat instances</code></dt> |
| 47 | <dd>Prints the registry: HOST, KIND, STATUS, VERSION, HIBERNATE, REPLICATION, SOCKET, plus a <code>last error</code> line for any failed row. The first stop for “what does this box think it is serving?”.</dd> |
| 48 | <dt><code>lchat mesh-seed [--region <cc>] <peer-url></code></dt> |
| 49 | <dd>Run on an existing mesh box, pointed at a newly joined one: bulk-replicates every in-scope hibernating route to the peer in one pass, authenticated by <code>LCHAT_MESH_TOKEN</code>. The one-time catch-up step of <a href="mesh.html#add-box">adding a box to the mesh</a>. On a region-scoped fleet, pass <code>--region</code> with the joining box’s <code>LCHAT_MESH_REGION</code> so routes it owns by <code>home_region</code> are seeded too, even ones outside this seed’s <code>LCHAT_MESH_DOMAINS</code> (the multi-edge region-owner union); omitting it seeds by domain exactly as before. <a href="mesh.html#mesh-replication">Part 07</a>.</dd> |
| 50 | <dt><code>lchat relocate <host> --to <region> [--dst-bucket <b>] [--move-dns --dns-target <ipv4>]</code></dt> |
| 51 | <dd>Move a hibernating space to another region without changing its address: copies its store to the destination region (a no-op on a shared bucket), flips <code>home_region</code>, and re-fans-out the route so the mesh re-homes it. The <a href="hibernation.html#the-lease">lease</a> keeps a single writer throughout. Run on a mesh box with <code>/data/router.env</code> sourced. <a href="mesh.html#relocate">Part 07</a>.</dd> |
| 52 | <dt><code>lchat admin-link <host></code> / <code>lchat setup-invite <host></code></dt> |
| 53 | <dd>Recover the admin link (from the environment file — the only plaintext copy of the token) / mint a fresh one-time setup invite.</dd> |
| 54 | </dl> |
| 55 | |
| 56 | <h3 id="cmd-deploy">Versions and deploys</h3> |
| 57 | <dl> |
| 58 | <dt><code>lchat stage --file <binary> [--version <ver>]</code></dt> |
| 59 | <dd>Copies a built binary into the version store under its build identifier (asked of the binary itself unless overridden). Inert and always safe. <a href="deploys.html#manual-deploy">Part 04</a>.</dd> |
| 60 | <dt><code>lchat deploy <host>|--all [--version <ver>] [--pin] [--include-pinned]</code></dt> |
| 61 | <dd>With <code>--version</code>: repoint the host’s live link, restart, poll <code>/api/version</code> for the exact build, auto-roll-back on mismatch, restart the bot sidecar. Without: restart on the current build. <code>--all</code> rolls one host at a time, stops on first failure, and skips pinned hosts unless <code>--include-pinned</code>. <code>--pin</code> (single host + version only) freezes the host there. Hibernating hosts: pure repoint, effective at next wake. <a href="deploys.html#manual-deploy">Part 04</a>.</dd> |
| 62 | <dt><code>lchat rollback <host> | --all</code></dt> |
| 63 | <dd>Repoints to the recorded previous version, restarts, health-checks, and swaps current/previous (so rollback twice returns forward). <a href="deploys.html#rollback">Part 04</a>.</dd> |
| 64 | <dt><code>lchat init-versions [--from <binary>]</code></dt> |
| 65 | <dd>One-time adoption when moving an existing box onto versioned deploys: stages a seed build and creates missing live links. Idempotent; never repoints an existing link.</dd> |
| 66 | <dt><code>lchat update run [--dry-run]</code> / <code>status</code> / <code>pin <host> [<ver>]</code> / <code>unpin <host></code></dt> |
| 67 | <dd>One updater pass (what the hourly timer fires) / the fleet update table (HOST, CHANNEL, AUTO, CURRENT, AVAILABLE) / freeze a host on a version / resume its channel. <a href="deploys.html#auto-updates">Part 04</a>.</dd> |
| 68 | <dt><code>lchat version</code></dt> |
| 69 | <dd>Prints the binary’s own build identifier (also served by every instance at <code>GET /api/version</code> as <code>build</code>).</dd> |
| 70 | </dl> |
| 71 | |
| 72 | <h3 id="cmd-daemons">Daemons and utilities</h3> |
| 73 | <dl> |
| 74 | <dt><code>lchat router</code></dt> |
| 75 | <dd>The router process itself — run by <code>lchat-router.service</code>, not by hand. Flags <code>--registry</code>, <code>--acme-cache</code>, <code>--https</code>, <code>--http</code>, <code>--fallback-url</code>, <code>--wildcard-domains</code>. <a href="router.html">Part 02</a>.</dd> |
| 76 | <dt><code>lchat controld</code></dt> |
| 77 | <dd>The root-only control daemon — run by <code>lchat-control.service</code>. Flags/env for socket, token file, and capacity policy are under <a href="#env-control">environment variables</a>. <a href="spaces.html#controld">Part 05</a>.</dd> |
| 78 | <dt><code>lchat cloud --socket <path> --db <path></code></dt> |
| 79 | <dd>The self-serve provisioner — run by <code>lchat-cloud.service</code>. <a href="spaces.html#cloud">Part 05</a>.</dd> |
| 80 | <dt id="cmd-links"><code>lchat links run --socket <path></code></dt> |
| 81 | <dd>The short-link redirector behind <code>11ch.at</code> (message permalinks, issue #905) — run by <code>lchat-links.service</code>. Exactly <strong>one box in a fleet is the short-link apex</strong>, and only that box runs the redirector; everywhere else the unit is written and left disabled. You declare the apex with <code>lchat bootstrap --links-apex <host></code>, which starts the redirector and registers <code><host></code> as a <code>service</code> route pointing at <code>/run/lchat/links.sock</code>. |
| 82 | <p>That route — not the flag — is then the source of truth: later plain <code>bootstrap</code> runs read the registry and converge to it, so re-running bootstrap on the apex box keeps short links up, and running it on any other box keeps the redirector off. (A flag-keyed design would silently switch the apex off the first time someone re-bootstrapped it without remembering the flag.)</p> |
| 83 | <p>Note the apex needs its own DNS A record and certificate: a <code>*.example.com</code> wildcard does <em>not</em> cover bare <code>example.com</code>. The router issues the certificate automatically once the route exists — an unrouted host gets no certificate at all, which is how a missing route surfaces: a TLS handshake failure rather than a 404.</p> |
| 84 | <p><strong>Ours to serve, in practice.</strong> Two values in <code>linkscmd.go</code> are Eleven-specific and not configurable: the apex bounces to <code>https://elevenmessenger.com</code>, and the <code>apple-app-site-association</code> advertises our iOS app id. Pointing your own apex here would send your visitors to our marketing site and publish our app id under your domain. Making those configurable is what self-hostable short links would take.</p></dd> |
| 85 | <dt><code>lchat bot run [--base <url>]</code> / <code>lchat bot provision</code></dt> |
| 86 | <dd>Run a bot sidecar (with no arguments, configuration comes from the unit’s environment file) / claim an agent invite into a session file. <a href="processes.html#bot-sidecars">Part 03</a>.</dd> |
| 87 | <dt><code>lchat release keygen|sign|verify|payload</code></dt> |
| 88 | <dd>The release-feed signing plumbing — used by the publishing pipeline, not in box operation. <a href="deploys.html#signing">Part 04</a>.</dd> |
| 89 | <dt><code>lchat help</code></dt> |
| 90 | <dd>Prints every command on this page, one line each — the fastest way to check a verb’s exact name on a box. Note that an <em>unrecognised</em> command (a typo like <code>lchat routes</code>) now prints this same list to stderr and exits 2. It used to fall through and start a messenger server instead, which either died on a port clash or, if the port was free, kept running unnoticed.</dd> |
| 91 | </dl> |
| 92 | |
| 93 | <h2 id="env">Environment variable reference</h2> |
| 94 | <p>Grouped by the file each is read from. All files are plain <code>KEY=value</code> lines, mode 0600, under <code>/data/</code> unless noted.</p> |
| 95 | |
| 96 | <h3 id="env-router">The router — <code>/data/router.env</code></h3> |
| 97 | <div class="tablewrap"> |
| 98 | <table> |
| 99 | <tr><th>Variable</th><th>Default</th><th>Meaning</th></tr> |
| 100 | <tr><td><code>LCHAT_REGISTRY</code></td><td><code>/data/registry.db</code></td><td>The registry database.</td></tr> |
| 101 | <tr><td><code>LCHAT_ACME_CACHE</code></td><td><code>/data/acme</code></td><td>Certificate cache directory.</td></tr> |
| 102 | <tr><td><code>LCHAT_HTTPS_ADDR</code> / <code>LCHAT_HTTP_ADDR</code></td><td><code>:443</code> / <code>:80</code></td><td>Public listen addresses.</td></tr> |
| 103 | <tr><td><code>LCHAT_FALLBACK_URL</code></td><td>unset (404)</td><td>Redirect target for unknown hostnames.</td></tr> |
| 104 | <tr><td><code>LCHAT_WILDCARD_DOMAINS</code></td><td>unset</td><td>Comma list of domains to hold wildcard certificates for (needs the next variable). <a href="router.html#dns01">Part 02</a>.</td></tr> |
| 105 | <tr><td><code>LCHAT_DNSIMPLE_TOKEN</code></td><td>unset</td><td>DNS-provider API token used for wildcard issuance.</td></tr> |
| 106 | <tr><td><code>LCHAT_DNS_PROVIDER</code></td><td><code>dnsimple</code></td><td>DNS-01 backend for wildcard issuance: <code>dnsimple</code> (default, needs the token above) or <code>route53</code> (additive AWS off-ramp — authenticates via the box’s own AWS role/user scoped to one zone, no token on disk). <a href="mesh.html#region-neutral">Part 07</a>.</td></tr> |
| 107 | <tr><td><code>LCHAT_HIBERNATE_BUCKET</code></td><td>unset (activator off)</td><td>The hibernation bucket — the activator’s master switch. <a href="hibernation.html#activator-config">Part 06</a>.</td></tr> |
| 108 | <tr><td><code>LCHAT_HIBERNATE_REGION</code> / <code>LCHAT_HIBERNATE_PREFIX</code></td><td>empty</td><td>Bucket region / key prefix.</td></tr> |
| 109 | <tr><td><code>LCHAT_HIBERNATE_IDLE</code></td><td><code>15m</code></td><td>Idle window before sleep (a duration such as <code>15m</code> or <code>90s</code>; must be positive).</td></tr> |
| 110 | <tr><td><code>LCHAT_MESH_SELF</code></td><td>unset (mesh off)</td><td>This box’s peer-reachable base URL — the mesh’s master switch. <a href="mesh.html#mesh-config">Part 07</a>.</td></tr> |
| 111 | <tr><td><code>LCHAT_MESH_DOMAINS</code></td><td>unset</td><td>Comma list of domains whose hosts participate in mesh placement. Required whenever <code>LCHAT_MESH_SELF</code> is set.</td></tr> |
| 112 | <tr><td><code>LCHAT_MESH_TOKEN</code></td><td>unset</td><td>Shared secret authenticating route replication between mesh members. Required whenever <code>LCHAT_MESH_SELF</code> is set; identical on every box. <a href="mesh.html#mesh-config">Part 07</a>.</td></tr> |
| 113 | <tr><td><code>LCHAT_MESH_REGION</code></td><td>unset (agnostic)</td><td>This box’s placement region label (e.g. <code>ie</code>) — advertised in the heartbeat body; the mesh owns a space on a box whose region matches its <code>home_region</code>. Distinct from <code>LCHAT_BOX_REGION</code> (that is the dashboard display string). <a href="mesh.html#region-placement">Part 07</a>.</td></tr> |
| 114 | <tr><td><code>LCHAT_BOX_REGION</code></td><td>unset (omitted)</td><td>This box’s coarse continent + country string, published to peers over the mesh stats channel for the Fleet dashboard’s <code>region</code> field (e.g. “EU · Finland”) — never a city, datacenter, IP, or hostname. <a href="operations.html#observability">Part 08</a>.</td></tr> |
| 115 | </table> |
| 116 | </div> |
| 117 | |
| 118 | <h3 id="env-instance">Each instance — <code>/data/<host>.env</code></h3> |
| 119 | <div class="tablewrap"> |
| 120 | <table> |
| 121 | <tr><th>Variable</th><th>Meaning</th></tr> |
| 122 | <tr><td><code>LCHAT_ADMIN_TOKEN</code></td><td>The instance’s admin token. The database stores only its hash; <strong>this file is the only recoverable copy</strong>.</td></tr> |
| 123 | <tr><td><code>LCHAT_APP_STORE_URL</code>, <code>LCHAT_HOME_ORIGIN</code>, <code>LCHAT_PUSH_RELAY_URL</code>, <code>LCHAT_PUSH_RELAY_TOKEN</code></td><td>Per-instance pointers to the app-store listing, the home server, and the push relay (with its per-instance bearer token).</td></tr> |
| 124 | <tr><td><code>LCHAT_BLOB_QUOTA_BYTES</code></td><td>Attachment-storage cap for the space (written by the control daemon’s quota endpoint; uploads beyond it are refused).</td></tr> |
| 125 | <tr><td><code>LCHAT_BLOB_S3_BUCKET</code> (+ <code>_PREFIX</code>, <code>_REGION</code>, <code>_ENDPOINT</code>, <code>_ACCESS_KEY</code>, <code>_SECRET_KEY</code>)</td><td>Attachment bytes in an S3-compatible object store. Set = new uploads land there, the on-disk dir keeps serving pre-flip blobs; unset = today’s disk layout. Prefix defaults to the host; key pair falls back to the AWS default chain. <span class="status partial">built, not yet flipped</span></td></tr> |
| 126 | <tr><td><code>LCHAT_BLOB_S3_MIRROR_*</code> (same six)</td><td>A second copy on a second provider (federated blobs Phase A — <code>docs/federated-blob-storage.md</code>): an idempotent sweep mirrors every store-backed blob, reads chain primary→mirror→disk, deletes reap both, and an hourly audit spot-checks both stores against each blob’s recorded sha256. Requires the primary. Changing the mirror <em>target</em> later: <code>UPDATE blobs SET mirrored_at=0</code> re-mirrors everything. <span class="status partial">built, not yet flipped</span></td></tr> |
| 127 | </table> |
| 128 | </div> |
| 129 | |
| 130 | <h3 id="env-control">The control daemon — <code>/data/control.env</code></h3> |
| 131 | <div class="tablewrap"> |
| 132 | <table> |
| 133 | <tr><th>Variable</th><th>Default</th><th>Meaning</th></tr> |
| 134 | <tr><td><code>LCHAT_MAX_INSTANCES</code></td><td>40</td><td>Ceiling on running (materialised) instances.</td></tr> |
| 135 | <tr><td><code>LCHAT_MIN_FREE_GB</code></td><td>5</td><td>Disk floor; creations refuse below it.</td></tr> |
| 136 | <tr><td><code>LCHAT_CONTROL_HIBERNATE_CREATES</code></td><td>off</td><td><code>1</code> = every provisioner-created space is born hibernating. <a href="spaces.html#hibernating-creates">Part 05</a>.</td></tr> |
| 137 | <tr><td><code>LCHAT_MAX_REGISTERED</code></td><td>0 (= 10 × max instances)</td><td>Row ceiling counting hibernating registrations.</td></tr> |
| 138 | <tr><td><code>LCHAT_TRASH_KEEP_DAYS</code></td><td>0 (keep forever)</td><td>Age at which archived (deleted) spaces in <code>/data/trash</code> are purged.</td></tr> |
| 139 | <tr><td><code>LCHAT_CONTROL_HOME_ORIGIN</code></td><td>empty (compiled production default)</td><td>Default <code>LCHAT_HOME_ORIGIN</code> for provisioner creates that name none — set on non-production fleets so their spaces point at the right home server (backup nudges, the demo banner's live status read).</td></tr> |
| 140 | <tr><td><code>LCHAT_CONTROL_SOCKET</code> / <code>LCHAT_CONTROL_TOKEN_FILE</code></td><td><code>/run/lchat/control.sock</code> / <code>/etc/lchat/control.token</code></td><td>Where it listens / the root-only bearer token gating it.</td></tr> |
| 141 | </table> |
| 142 | </div> |
| 143 | |
| 144 | <h3 id="env-cloud">The cloud provisioner — <code>/data/cloud.env</code></h3> |
| 145 | <div class="tablewrap"> |
| 146 | <table> |
| 147 | <tr><th>Variable</th><th>Default</th><th>Meaning</th></tr> |
| 148 | <tr><td><code>LCHAT_ADMIN_TOKEN</code></td><td>required</td><td>Gates the caller-token minting API.</td></tr> |
| 149 | <tr><td><code>LCHAT_CLOUD_BASE_DOMAIN</code></td><td>required</td><td>Domain new spaces are created under (needs wildcard DNS).</td></tr> |
| 150 | <tr><td><code>LCHAT_CLOUD_REGION</code></td><td>unset (agnostic)</td><td>Stamps each newly-provisioned space’s <code>home_region</code>, so it is placed in this region from first wake. <a href="mesh.html#region-neutral">Part 07</a>.</td></tr> |
| 151 | <tr><td><code>LCHAT_CLOUD_ID_LEDGER</code></td><td><code>0</code> (local ids)</td><td>When truthy, mint hostnames from the shared S3-backed global id counter (so region-neutral names never collide across regions) instead of the local DB primary key. Explicit opt-in: enabling it for a namespace that already has spaces needs the counter seeded above the current max first. <a href="mesh.html#region-neutral">Part 07</a>.</td></tr> |
| 152 | <tr><td><code>LCHAT_CLOUD_QUOTA</code></td><td>3</td><td>Free spaces each requesting user may hold at once.</td></tr> |
| 153 | <tr><td><code>LCHAT_CLOUD_MAX_CREATES_PER_DAY</code></td><td>0 (uncapped)</td><td>Fleet-wide daily creation circuit breaker.</td></tr> |
| 154 | <tr><td><code>LCHAT_CLOUD_REAP_AFTER_DAYS</code></td><td>0 (reaper off)</td><td>Age before a never-claimed space is reclaimable. <a href="spaces.html#reaper">Part 05</a>.</td></tr> |
| 155 | <tr><td><code>LCHAT_CLOUD_REAP_INTERVAL_HOURS</code> / <code>LCHAT_CLOUD_REAP_MAX_PER_SWEEP</code> / <code>LCHAT_CLOUD_REAP_DRY_RUN</code></td><td>6 / 5 / off</td><td>Reaper sweep cadence / per-sweep cap / log-only mode.</td></tr> |
| 156 | <tr><td><code>LCHAT_CLOUD_STORAGE_CAPS</code>, <code>LCHAT_STRIPE_*</code></td><td>unset</td><td>Per-tier storage caps and the dormant billing rails. <span class="status partial">partially built</span></td></tr> |
| 157 | <tr><td><code>LCHAT_CLOUD_REPLICATION_CLASSES</code></td><td>unset (every space streams)</td><td>Per-tier durability rungs, e.g. <code>free=daily,family=daily,community=batched,business=stream</code>. <a href="hibernation.html#replication-classes">Part 06</a>.</td></tr> |
| 158 | </table> |
| 159 | </div> |
| 160 | |
| 161 | <h3 id="env-other">Other files</h3> |
| 162 | <div class="tablewrap"> |
| 163 | <table> |
| 164 | <tr><th>File</th><th>Notable variables</th></tr> |
| 165 | <tr><td><code>/data/<host>-bot.env</code></td><td><code>LCHAT_BASE</code> (the instance’s public URL), the per-identity session paths, optional <code>ANTHROPIC_API_KEY</code> and <code>LCHAT_MODEL</code>. <a href="processes.html#bot-sidecars">Part 03</a>.</td></tr> |
| 166 | <tr><td><code>/data/update.env</code></td><td><code>LCHAT_UPDATE_MANIFEST_URL</code> (point a box at a different release feed); mirror fallback via <code>LCHAT_RELEASE_MIRROR_URL</code>. <a href="deploys.html#signing">Part 04</a>.</td></tr> |
| 167 | <tr><td><code>/data/home.env</code> (required)</td><td><code>LCHAT_HOME_ORIGIN</code> (the passkey origin — must never change), <code>LCHAT_CLOUD_URL</code> + <code>LCHAT_CLOUD_TOKEN</code> (enables self-serve creation), <code>LCHAT_HOME_STORE</code> (<code>sqlite</code> default, or <code>s3</code> with <code>LCHAT_HOME_S3_BUCKET</code>/<code>_REGION</code>).</td></tr> |
| 168 | <tr><td><code>/data/push.env</code> (required)</td><td>The Apple push signing key and its identifiers — the one copy in the fleet, plus the relay’s admin token.</td></tr> |
| 169 | </table> |
| 170 | </div> |
| 171 | |
| 172 | <h2 id="paths">File path reference</h2> |
| 173 | <div class="tablewrap"> |
| 174 | <table> |
| 175 | <tr><th>Path</th><th>What it is</th><th>Back up?</th></tr> |
| 176 | <tr><td><code>/opt/lchat/lchat</code>, <code>/opt/lchat/home</code>, <code>/opt/lchat/push</code></td><td>The binaries.</td><td>No — rebuildable.</td></tr> |
| 177 | <tr><td><code>/opt/lchat/versions/<ver>/lchat</code></td><td>The version store (every staged build).</td><td>No.</td></tr> |
| 178 | <tr><td><code>/opt/lchat/live/<host></code></td><td>The per-instance version pointer (a symlink). <em>Which build a host runs.</em></td><td>No (recorded in the registry).</td></tr> |
| 179 | <tr><td><code>/data/<host>.db</code> (+ <code>-wal</code>, <code>-shm</code>)</td><td>A space’s entire database (ciphertext).</td><td><strong>Yes</strong> — continuously, via the backup replicator.</td></tr> |
| 180 | <tr><td><code>/data/<host>.env</code></td><td>A space’s configuration and its only plaintext admin token.</td><td><strong>Yes.</strong></td></tr> |
| 181 | <tr><td><code>/data/<host>-blobs/</code></td><td>A space’s encrypted attachments.</td><td><strong>Yes</strong> (file-level).</td></tr> |
| 182 | <tr><td><code>/data/<host>.vapid</code></td><td>A space’s web-push key. Losing it breaks push subscriptions.</td><td><strong>Yes.</strong></td></tr> |
| 183 | <tr><td><code>/data/<host>-bot.env</code>, <code>/data/<host>-bot/</code></td><td>Bot sidecar configuration and identities.</td><td><strong>Yes.</strong></td></tr> |
| 184 | <tr><td><code>/data/registry.db</code></td><td>The box’s routing table and instance ledger.</td><td><strong>Yes.</strong></td></tr> |
| 185 | <tr><td><code>/data/acme/</code></td><td>TLS certificates and their private keys.</td><td>Yes (or accept re-issuance).</td></tr> |
| 186 | <tr><td><code>/data/trash/<host>-<timestamp>/</code></td><td>Archived deleted spaces.</td><td>Optional.</td></tr> |
| 187 | <tr><td><code>/run/lchat/*.sock</code></td><td>The unix sockets (transient).</td><td>No.</td></tr> |
| 188 | <tr><td><code>/etc/lchat/control.token</code></td><td>The root-only control-daemon secret.</td><td><strong>Yes</strong> (secretly).</td></tr> |
| 189 | <tr><td><code>/etc/litestream.yml</code></td><td>Generated backup-replication configuration.</td><td>No — regenerate with <code>deploy/fleet-backup.sh</code>.</td></tr> |
| 190 | <tr><td><code>s3://<bucket>/<prefix>/<host>/</code></td><td>A hibernating space’s parked database and its <code>lock.json</code> lease.</td><td>The bucket’s own versioning covers it.</td></tr> |
| 191 | <tr><td><code>s3://<bucket>/<prefix>/mesh/members/</code></td><td>The mesh membership heartbeats.</td><td>No — transient by design.</td></tr> |
| 192 | </table> |
| 193 | </div> |
| 194 | |
| 195 | <h2 id="units">systemd unit reference</h2> |
| 196 | <p>Covered in narrative form in <a href="processes.html#census">part 03</a>; the lookup table:</p> |
| 197 | <div class="tablewrap"> |
| 198 | <table> |
| 199 | <tr><th>Unit</th><th>Runs</th><th>Enabled by</th><th>Journal</th></tr> |
| 200 | <tr><td><code>lchat-router.service</code></td><td>The router (+ activator + mesh)</td><td><code>bootstrap</code></td><td><code>journalctl -u lchat-router</code></td></tr> |
| 201 | <tr><td><code>lchat@<host>.service</code></td><td>One instance</td><td><code>add-instance</code></td><td><code>journalctl -u 'lchat@<host>'</code></td></tr> |
| 202 | <tr><td><code>lchat-bot@<host>.service</code></td><td>One bot sidecar</td><td><code>add-bot</code></td><td><code>journalctl -u 'lchat-bot@<host>'</code></td></tr> |
| 203 | <tr><td><code>lchat-hooks@<host>.service</code></td><td>One webhook-ingest sidecar</td><td><code>add-hooks</code></td><td><code>journalctl -u 'lchat-hooks@<host>'</code></td></tr> |
| 204 | <tr><td><code>lchat-home.service</code> / <code>lchat-push.service</code></td><td>Home server / push relay</td><td>manually, once</td><td><code>journalctl -u lchat-home</code> etc.</td></tr> |
| 205 | <tr><td><code>lchat-control.service</code></td><td>Control daemon (root)</td><td>manually, once</td><td><code>journalctl -u lchat-control</code></td></tr> |
| 206 | <tr><td><code>lchat-cloud.service</code></td><td>Cloud provisioner</td><td>manually, once</td><td><code>journalctl -u lchat-cloud</code></td></tr> |
| 207 | <tr><td><code>lchat-update.timer</code> → <code>.service</code></td><td>Hourly update pass (root)</td><td><code>bootstrap</code></td><td><code>journalctl -u lchat-update.service</code></td></tr> |
| 208 | <tr><td><code>lchat-litestream.service</code></td><td>Fleet backup replicator</td><td><code>deploy/fleet-backup.sh</code></td><td><code>journalctl -u lchat-litestream</code></td></tr> |
| 209 | </table> |
| 210 | </div> |
| 211 | |
| 212 | <h2 id="endpoints">Health and inspection endpoints</h2> |
| 213 | <div class="tablewrap"> |
| 214 | <table> |
| 215 | <tr><th>Endpoint</th><th>On</th><th>Returns</th></tr> |
| 216 | <tr><td><code>GET /api/version</code></td><td>Every instance (no authentication)</td><td><code>{"version": <frontend build>, "build": <binary build id>}</code> — the health check deploys poll.</td></tr> |
| 217 | <tr><td><code>GET /healthz</code></td><td>The cloud provisioner</td><td><code>ok</code>.</td></tr> |
| 218 | <tr><td><code>GET /api/next-due</code> / <code>POST /api/charter-next-due</code></td><td>Every instance (token-gated, used by the activator and sidecar)</td><td>The earliest scheduled delivery — the machinery behind <a href="hibernation.html#due-time-wakes">scheduled wakes</a>. Not an operator surface.</td></tr> |
| 219 | </table> |
| 220 | </div> |
| 221 | |
| 222 | <h2 id="glossary">Glossary</h2> |
| 223 | <dl> |
| 224 | <dt id="gloss-space">space / instance / host</dt> |
| 225 | <dd>One chat group; its server-side unit (one process + one SQLite database + one subdomain); and that subdomain itself, which names everything belonging to the instance. See <a href="index.html#vocabulary">the vocabulary table</a>.</dd> |
| 226 | <dt id="gloss-box">box</dt> |
| 227 | <dd>One Linux server in the fleet, running the router and everything in <a href="processes.html">part 03</a>.</dd> |
| 228 | <dt id="gloss-systemd">systemd</dt> |
| 229 | <dd>The standard Linux service manager: it starts processes at boot, restarts them when they crash, applies sandboxing, and keeps their logs (the “journal”). Controlled with <code>systemctl</code>, read with <code>journalctl</code>.</dd> |
| 230 | <dt id="gloss-unit">unit / template unit</dt> |
| 231 | <dd>A systemd service definition; a template unit (name containing <code>@</code>) is stamped out per instance name, which is how one file defines the service for every chat space.</dd> |
| 232 | <dt id="gloss-unix-socket">unix socket</dt> |
| 233 | <dd>A file (here under <code>/run/lchat/</code>) that local processes connect to like a network address. Invisible to the network — the reason instances need no firewall rules of their own.</dd> |
| 234 | <dt id="gloss-registry">the registry</dt> |
| 235 | <dd>The SQLite file <code>/data/registry.db</code>: one row per hostname a box serves. The router’s routing table and the fleet commands’ ledger. <a href="router.html#the-registry">Part 02</a>.</dd> |
| 236 | <dt id="gloss-dns">DNS / wildcard DNS</dt> |
| 237 | <dd>The system translating names to network addresses. A <em>wildcard</em> record (<code>*.example.com</code>) points every subdomain at once, which is why creating a space needs no DNS step.</dd> |
| 238 | <dt id="gloss-le">Let’s Encrypt / ACME / HTTP-01 / DNS-01</dt> |
| 239 | <dd>The free certificate authority and its automated issuance protocol. HTTP-01 proves control of a hostname by answering a challenge on port 80 (used per host); DNS-01 proves control of a whole domain via a temporary DNS record (used for wildcard certificates). <a href="router.html#tls">Part 02</a>.</dd> |
| 240 | <dt id="gloss-s3">object storage / S3 / bucket / prefix</dt> |
| 241 | <dd>Storage addressed as named objects in a “bucket” rather than files on a disk — durable, replicated, and shared. A “prefix” is the leading part of object names, used like a directory. Amazon S3 is the implementation used here.</dd> |
| 242 | <dt id="gloss-litestream">Litestream</dt> |
| 243 | <dd>A tool that continuously streams a SQLite database’s changes into object storage, allowing restore to any recent moment. Used twice in this system, in two forms: a stock copy for <a href="operations.html#backups">fleet backups</a>, and embedded inside the router for <a href="hibernation.html#the-lease">hibernation</a>.</dd> |
| 244 | <dt id="gloss-lease">the lease</dt> |
| 245 | <dd>The per-space lock object (<code>lock.json</code>) in the hibernation bucket, held while a process serves the space and renewed every few minutes. The system’s single source of truth about who may write a space’s data. <a href="hibernation.html#the-lease">Part 06</a>.</dd> |
| 246 | <dt id="gloss-replication-class">replication class</dt> |
| 247 | <dd>A hibernating host’s durability rung (<code>stream</code> / <code>batched</code> / <code>daily</code>): how often its awake writes sync to the bucket, set per plan by billing or per host by <code>lchat replication</code>. Graceful sleeps and deploys always flush in full regardless. <a href="hibernation.html#replication-classes">Part 06</a>.</dd> |
| 248 | <dt id="gloss-activator">the activator</dt> |
| 249 | <dd>The component inside the router that wakes sleeping spaces on demand and puts idle ones to sleep. Exists only when <code>LCHAT_HIBERNATE_BUCKET</code> is set. <a href="hibernation.html">Part 06</a>.</dd> |
| 250 | <dt id="gloss-hrw">rendezvous hashing</dt> |
| 251 | <dd>The rule mesh boxes use to agree which of them owns each space without communicating: every box scores every (box, space) pair the same way, and the highest score wins. Adding or removing a box only moves the spaces whose winner changed. <a href="mesh.html#placement">Part 07</a>.</dd> |
| 252 | <dt id="gloss-sidecar">bot sidecar</dt> |
| 253 | <dd>The separate process hosting an instance’s bot members. It holds the bots’ encryption keys (which must never live in the server process) and connects outward as an ordinary client. <a href="processes.html#bot-sidecars">Part 03</a>.</dd> |
| 254 | <dt id="gloss-canary">canary / pin</dt> |
| 255 | <dd>Deploying a new build to one instance to watch it before promoting it everywhere; and the mark that freezes an instance on a version so the hourly auto-updater leaves the experiment alone. <a href="deploys.html#pinning">Part 04</a>.</dd> |
| 256 | <dt id="gloss-vapid">web push / VAPID key</dt> |
| 257 | <dd>Browser push notifications, signed per instance with a key pair (the <code>.vapid</code> file). Distinct from Apple device push, which goes through the push relay.</dd> |
| 258 | </dl> |
| 259 | |
| 260 | </div> |
| 261 | </main> |
| 262 | </div> |
| 263 | </body> |
| 264 | </html> |
| 265 | |