amadan
docs: multi-page HTML operator guide to the planet-scale system (#825)
A nine-page, human-focussed guide (docs/planet-scale-guide/) covering the router, every process on a box and its systemd unit, versioned deploys and release channels, space creation/removal and the self-serve pipeline, hibernation, the mesh, a day-to-day runbook, and a complete reference with glossary. Plain HTML + one shared stylesheet + one shared script (no build step); every page opens with the same system map highlighting the component it covers; status badges keep shipped vs partially-built vs designed-only honest (mesh marked dormant, registry replication marked reverted). CLAUDE.md gains a convention: PRs that change the fleet machinery's operator-visible behaviour update the guide in the same PR.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
13 files changed,
+2210
−0
CLAUDE.md+11 −0docs/planet-scale-guide/README.html+67 −0docs/planet-scale-guide/deploys.html+165 −0docs/planet-scale-guide/guide.css+444 −0docs/planet-scale-guide/guide.js+210 −0docs/planet-scale-guide/hibernation.html+152 −0docs/planet-scale-guide/index.html+142 −0docs/planet-scale-guide/mesh.html+122 −0docs/planet-scale-guide/operations.html+153 −0docs/planet-scale-guide/processes.html+193 −0docs/planet-scale-guide/reference.html+236 −0docs/planet-scale-guide/router.html+159 −0docs/planet-scale-guide/spaces.html+156 −0
diff --git a/CLAUDE.md b/CLAUDE.md| index 22996f9..d4af15d 100644 |
| --- a/CLAUDE.md |
| +++ b/CLAUDE.md |
| @@ -343,6 +343,17 @@ open with the note itself) carry the 🤖 on their first line. |
| downloadable (TestFlight for iOS; GitHub Releases/Sparkle for Mac) — so a |
| native feature PR ships *without* an entry, and the PR that mints the release |
| adds one covering everything in that build. |
| +- 🤖 **Keep the fleet operations guide current.** `docs/planet-scale-guide/` is |
| + the multi-page, human-focussed HTML guide to the planet-scale machinery — the |
| + router, the process/systemd model, versioned deploys and channels, space |
| + creation/removal, hibernation, and the mesh. A PR that changes any of that |
| + machinery's operator-visible behaviour (commands, flags, env vars, unit files, |
| + filesystem layout, timings/defaults, or the shipped-vs-roadmap status of a |
| + feature) updates the guide in the same PR. Keep its status badges honest — |
| + "running today" / "partially built" / "designed, not built" — and when a |
| + roadmap item ships, promote its badge and move the text out of the dashed |
| + roadmap panel. How the guide is structured (shared nav in `guide.js`, shared |
| + styles, page conventions): `docs/planet-scale-guide/README.html`. |
| - **Refresh Eleven's stats periodically.** `internal/bot/knowledge.md` §16 has a |
| *"By the numbers"* block (LOC + binary/app sizes) that Eleven quotes to members |
| (this file is Eleven's reference — the single source both sidecars use: the Go |
diff --git a/docs/planet-scale-guide/README.html b/docs/planet-scale-guide/README.html| new file mode 100644 |
| index 0000000..f04e196 |
| --- /dev/null |
| +++ b/docs/planet-scale-guide/README.html |
| @@ -0,0 +1,67 @@ |
| +<!doctype html> |
| +<html lang="en"> |
| +<head> |
| +<meta charset="utf-8" /> |
| +<meta name="viewport" content="width=device-width, initial-scale=1" /> |
| +<title>Maintaining this guide — Running the fleet</title> |
| +<link rel="stylesheet" href="guide.css" /> |
| +<script src="guide.js" defer></script> |
| +</head> |
| +<body data-page="README.html"> |
| +<div class="frame"> |
| +<nav class="side"></nav> |
| +<main class="content"> |
| +<div class="wrap"> |
| + |
| +<header class="top"> |
| + <p class="eyebrow"><span class="robot">🤖</span> Running the fleet · About this guide</p> |
| + <h1>Maintaining this guide</h1> |
| + <p class="thesis">The guide is nine plain HTML pages, one shared stylesheet, and one shared script — <strong>no build step, no dependencies, nothing to install</strong>. Open any page in a browser straight from the repository, or serve the directory statically. This page explains how to keep it accurate and how to extend it.</p> |
| +</header> |
| + |
| +<h2 id="layout">How the directory is put together</h2> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>File</th><th>Role</th></tr> |
| + <tr><td><code>index.html</code> … <code>reference.html</code></td><td>The nine content pages. Each is self-contained semantic HTML; nothing is generated.</td></tr> |
| + <tr><td><code>guide.css</code></td><td>The single stylesheet. Colors, typography, panels, badges, tables, diagrams — edit here and every page follows. Light and dark themes are both defined at the top.</td></tr> |
| + <tr><td><code>guide.js</code></td><td>The single script, and the guide’s one source of structure: the <code>PAGES</code> list (which drives the sidebar and the previous/next links on every page) and the shared “you are here” system-map diagram. All injected markup is built from static in-code strings only, per the repository rule about <code>innerHTML</code>.</td></tr> |
| + <tr><td><code>README.html</code></td><td>This page.</td></tr> |
| +</table> |
| +</div> |
| + |
| +<h2 id="editing">Editing an existing page</h2> |
| +<ul> |
| + <li><strong>The code is the source of truth.</strong> Before changing a factual claim, check it against the code (<code>control.go</code>, <code>router.go</code>, <code>activator.go</code>, <code>mesh.go</code>, <code>controld.go</code>, <code>cloud.go</code>, <code>reaper.go</code>, <code>deploy/systemd/</code>) or the design documents (<code>docs/planet-scale.md</code>, <code>docs/instance-router.md</code>). Where this guide and the code disagree, fix the guide.</li> |
| + <li><strong>Keep the register.</strong> The guide deliberately avoids contractions, avoids unexplained jargon, spells out commands in full, and repeats key ideas where they matter. Match that when adding text.</li> |
| + <li><strong>Keep the status badges honest.</strong> Anything not operable today must carry <span class="status partial">partially built</span> or <span class="status planned">designed, not built</span>, and aspirational behaviour belongs in a dashed <code>.roadmap</code> panel, never in running prose. When a roadmap item ships, promote its badge and move the text out of the panel in the same change.</li> |
| + <li><strong>Use the shared components</strong> rather than inventing new markup: <code>figure.cmd</code> for labelled command blocks, <code>ol.procedure</code> for step-by-step tasks, <code>.plain</code> / <code>.warn</code> / <code>.why</code> panels for recaps, cautions, and rationale, and <code>.status</code> badges. Examples of each are on every page.</li> |
| +</ul> |
| + |
| +<h2 id="adding">Adding a page</h2> |
| +<ol class="procedure"> |
| + <li><p class="step-title">Copy an existing page as the template.</p> |
| + <p>Keep the <code><head></code>, the <code>.frame</code>/<code>nav.side</code>/<code>main.content</code> skeleton, the header block, and the <code>div.sysmap</code>. Set <code>data-page</code> on <code><body></code> to the new file name.</p></li> |
| + <li><p class="step-title">Add one entry to the <code>PAGES</code> list in <code>guide.js</code>, in reading order.</p> |
| + <p>That single edit updates every page’s sidebar and previous/next links. Renumber the <code>num</code> fields and each page’s eyebrow (“Part NN of NN”) if the new page changes the ordering.</p></li> |
| + <li><p class="step-title">Choose the system-map highlight.</p> |
| + <p>Set <code>data-highlight</code> on the <code>div.sysmap</code> to the node keys the page covers (<code>members router instances services control updater bucket backup peer</code>, space-separated). If the page introduces a genuinely new component, add a node for it to <code>sysmapSVG</code> in <code>guide.js</code> so the map stays a true whole-system picture.</p></li> |
| +</ol> |
| + |
| +<h2 id="checking">Checking your changes</h2> |
| +<figure class="cmd"> |
| +<figcaption>Run in the repository</figcaption> |
| +<pre><code>node --check docs/planet-scale-guide/guide.js # the script must stay parseable |
| +# then open the pages and click through: |
| +open docs/planet-scale-guide/index.html # macOS; or any static file server</code></pre> |
| +</figure> |
| +<p>Eyeball both color schemes (the pages follow the operating system’s light/dark preference), confirm the sidebar highlights the page you are on, and confirm the previous/next links at the bottom of each page still make sense.</p> |
| + |
| +<h2 id="authorship">Authorship marking</h2> |
| +<p>Per the repository’s authorship convention (<code>CLAUDE.md</code>), these pages were written by an LLM and carry the visible 🤖 marker in each page’s eyebrow. A human who substantially rewrites a page should remove the marker from that page (or add a person marker to certify it); an LLM editing further keeps it.</p> |
| + |
| +</div> |
| +</main> |
| +</div> |
| +</body> |
| +</html> |
diff --git a/docs/planet-scale-guide/deploys.html b/docs/planet-scale-guide/deploys.html| new file mode 100644 |
| index 0000000..aed8fd1 |
| --- /dev/null |
| +++ b/docs/planet-scale-guide/deploys.html |
| @@ -0,0 +1,165 @@ |
| +<!doctype html> |
| +<html lang="en"> |
| +<head> |
| +<meta charset="utf-8" /> |
| +<meta name="viewport" content="width=device-width, initial-scale=1" /> |
| +<title>Installing and updating the software — Running the fleet</title> |
| +<link rel="stylesheet" href="guide.css" /> |
| +<script src="guide.js" defer></script> |
| +</head> |
| +<body data-page="deploys.html"> |
| +<div class="frame"> |
| +<nav class="side"></nav> |
| +<main class="content"> |
| +<div class="wrap"> |
| + |
| +<header class="top"> |
| + <p class="eyebrow"><span class="robot">🤖</span> Running the fleet · Part 04 of 09</p> |
| + <h1>Installing and updating the software</h1> |
| + <p class="thesis">Every instance on a box can run a <strong>different version</strong> of the software at the same time. This page explains the machinery that makes that true — the version store and the per-instance symbolic link — and then the three ways software changes: <strong>a manual staged deploy, an automatic hourly update, and rollback.</strong></p> |
| +</header> |
| + |
| +<div class="sysmap" data-highlight="updater"></div> |
| +<p class="sysmap-caption">You are here: the auto-updater node — though this page equally concerns the version store under <code>/opt/lchat/</code> and the per-instance links that decide which build every instance runs.</p> |
| + |
| +<h2 id="mental-model">The mental model: a store of builds and a pointer per instance</h2> |
| +<p>Two filesystem facts, introduced in <a href="processes.html#opt">part 03</a>, carry this entire page:</p> |
| +<ul> |
| + <li><strong>The version store.</strong> Every build of the software that has ever been staged on the box lives at <code>/opt/lchat/versions/<ver>/lchat</code>, where <code><ver></code> is the build’s identifier — a short git commit hash such as <code>1b4bd808</code>, stamped into the binary when it was compiled. Builds sit side by side and never overwrite each other.</li> |
| + <li><strong>The per-instance pointer.</strong> Each instance’s systemd unit runs <code>/opt/lchat/live/<host></code>, which is a symbolic link into the version store. <em>Which build an instance runs is exactly where its link points, nothing more.</em></li> |
| +</ul> |
| +<p>So “deploy version X to instance Y” decomposes into: repoint <code>live/Y</code> at <code>versions/X/lchat</code>, restart Y’s process, and verify Y now reports build X. That is genuinely all a deploy is, and every command below is a wrapper around those three steps with safety checks. It also makes the two most valuable properties fall out for free:</p> |
| +<ul> |
| + <li><strong>Canarying:</strong> point <em>one</em> instance at a new build while the rest stay put, watch it, then promote the rest.</li> |
| + <li><strong>Rollback:</strong> the previous build is still sitting in the store, and the registry remembers which one it was — stepping back is repointing the link again.</li> |
| +</ul> |
| +<figure class="diagram"> |
| +<svg viewBox="0 0 720 210" role="img" aria-label="Diagram: three live links point into a store of versioned builds; one instance is canaried on a newer build."> |
| + <rect x="20" y="20" width="290" height="170" rx="10" fill="none" stroke="var(--rule)" stroke-dasharray="4 4"></rect> |
| + <text x="34" y="42" font-size="10" font-weight="700" letter-spacing="0.08em" fill="var(--muted)">/opt/lchat/live/</text> |
| + <rect x="34" y="56" width="240" height="30" rx="6" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| + <text x="44" y="75" font-size="11" font-family="var(--mono)">chat-a.example.com</text> |
| + <rect x="34" y="96" width="240" height="30" rx="6" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| + <text x="44" y="115" font-size="11" font-family="var(--mono)">chat-b.example.com</text> |
| + <rect x="34" y="136" width="240" height="30" rx="6" fill="var(--accent-soft)" stroke="var(--accent)"></rect> |
| + <text x="44" y="155" font-size="11" font-family="var(--mono)">test.example.com (canary)</text> |
| + <rect x="430" y="20" width="270" height="170" rx="10" fill="none" stroke="var(--rule)" stroke-dasharray="4 4"></rect> |
| + <text x="444" y="42" font-size="10" font-weight="700" letter-spacing="0.08em" fill="var(--muted)">/opt/lchat/versions/</text> |
| + <rect x="444" y="56" width="240" height="44" rx="6" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| + <text x="454" y="75" font-size="11" font-family="var(--mono)">9c21e77e/lchat</text> |
| + <text x="454" y="91" font-size="9.5" fill="var(--muted)">the stable build</text> |
| + <rect x="444" y="116" width="240" height="44" rx="6" fill="var(--accent-soft)" stroke="var(--accent)"></rect> |
| + <text x="454" y="135" font-size="11" font-family="var(--mono)">4f0a2d19/lchat</text> |
| + <text x="454" y="151" font-size="9.5" fill="var(--muted)">the new build being canaried</text> |
| + <path d="M 274 71 C 360 71, 360 78, 444 78" fill="none" stroke="var(--muted)" stroke-width="1.2"></path> |
| + <path d="M 274 111 C 360 111, 360 84, 444 84" fill="none" stroke="var(--muted)" stroke-width="1.2"></path> |
| + <path d="M 274 151 C 360 151, 360 138, 444 138" fill="none" stroke="var(--accent)" stroke-width="1.6"></path> |
| +</svg> |
| +<figcaption>Each instance’s link chooses its build independently. Here two instances run the stable build while one canaries a newer one.</figcaption> |
| +</figure> |
| + |
| +<h2 id="bootstrap">Setting up a brand-new box: <code>lchat bootstrap</code></h2> |
| +<p>Because the software is one static binary, provisioning a fresh Linux server needs no compiler, no language runtime, and no packages. The one-time sequence is:</p> |
| +<ol class="procedure"> |
| + <li><p class="step-title">Copy the binaries to the box (or use the installer).</p> |
| + <p>Either copy a built <code>lchat</code> binary to <code>/opt/lchat/lchat</code> yourself, or run <code>deploy/install.sh</code>, which downloads the latest stable release, verifies its checksum, and performs the next step for you.</p></li> |
| + <li><p class="step-title">Run <code>lchat bootstrap</code>, as root, once.</p> |
| + <p>This writes all the systemd unit files described in <a href="processes.html#census">part 03</a>, creates the <code>lchat</code> system user, creates <code>/data</code>, <code>/run/lchat</code>, and the certificate cache directory, mints the root-only control token at <code>/etc/lchat/control.token</code>, then enables and starts the router and the hourly update timer. It is idempotent: re-running it refreshes the unit files without touching data.</p></li> |
| + <li><p class="step-title">Open ports 80 and 443 (and 22 for yourself) in the firewall.</p> |
| + <p>Nothing else needs a public port — see <a href="operations.html#hardening">part 08</a>.</p></li> |
| +</ol> |
| + |
| +<h2 id="manual-deploy">A manual deploy, end to end: build, stage, canary, promote</h2> |
| +<p>This is the full ceremony for shipping a build you compiled yourself. It is the procedure behind every “deploy the new version” task, and each step exists for a reason given alongside it.</p> |
| +<ol class="procedure"> |
| + <li><p class="step-title">Build the Linux binary on your own machine, stamped with its version.</p> |
| + <figure class="cmd"> |
| + <figcaption>Run on your own machine, in the repository</figcaption> |
| + <pre><code>./deploy/build-linux.sh |
| +<span class="out"># produces lchat-linux-amd64, with the current git commit hash compiled in</span></code></pre> |
| + </figure> |
| + <p>The build script passes the git commit hash into the binary itself, and the running server reports it at <code>GET /api/version</code> as the <code>build</code> field. This self-identification is what makes health-checked deploys possible: the machinery can ask a process which build it <em>actually is</em>, rather than trusting that a restart took.</p></li> |
| + <li><p class="step-title">Copy the binary to the box and stage it into the version store.</p> |
| + <figure class="cmd"> |
| + <figcaption>Run on your own machine</figcaption> |
| + <pre><code>scp lchat-linux-amd64 root@box-1:/opt/lchat/incoming |
| +ssh root@box-1 /opt/lchat/lchat stage --file /opt/lchat/incoming |
| +<span class="out">staged 4f0a2d19</span></code></pre> |
| + </figure> |
| + <p><code>stage</code> asks the binary for its own build identifier and copies it to <code>/opt/lchat/versions/4f0a2d19/lchat</code>. Nothing is running it yet; staging is inert and always safe.</p></li> |
| + <li><p class="step-title">Deploy it to one canary instance first.</p> |
| + <figure class="cmd"> |
| + <figcaption>Run on the box (or via ssh)</figcaption> |
| + <pre><code>lchat deploy test.example.com --version 4f0a2d19</code></pre> |
| + </figure> |
| + <p>This repoints <code>live/test.example.com</code>, restarts the unit, and then <strong>polls <code>/api/version</code> until the instance reports build <code>4f0a2d19</code></strong> — not merely “answers 200”. If the report never matches (the new build crashes on start, say), the command automatically repoints the link back to the previous version, restarts again, and exits with an error. A failed deploy therefore leaves the instance running the old build, not down.</p></li> |
| + <li><p class="step-title">Verify the canary by using it, then promote the whole box.</p> |
| + <figure class="cmd"> |
| + <figcaption>Run on the box</figcaption> |
| + <pre><code>lchat deploy --all --version 4f0a2d19</code></pre> |
| + </figure> |
| + <p><code>--all</code> rolls instance by instance, health-checking each, and <strong>stops at the first failure</strong> (instances already promoted stay promoted). It also <strong>skips pinned instances</strong> — a pin marks somebody’s deliberate canary, and a fleet-wide promote must not stomp it; <code>--include-pinned</code> overrides that deliberately, and naming a host explicitly always deploys it. Two more useful forms: <code>lchat deploy <host></code> with no <code>--version</code> simply restarts the instance on its current build, and <code>--pin</code> combined with <code>--version</code> deploys <em>and</em> pins in one step (the standard way to start a canary that the auto-updater must leave alone).</p></li> |
| + <li><p class="step-title">If anything looks wrong later, step back.</p> |
| + <figure class="cmd"> |
| + <figcaption>Run on the box</figcaption> |
| + <pre><code>lchat rollback test.example.com |
| +<span class="out">rolled back test.example.com: 4f0a2d19 -> 9c21e77e</span></code></pre> |
| + </figure> |
| + <p id="rollback"><code>rollback</code> repoints the link at the registry’s recorded previous version, restarts, health-checks, and swaps the “current” and “previous” records — so a second rollback returns you forward again. <code>rollback --all</code> steps every instance back, skipping any that have no recorded previous version.</p></li> |
| +</ol> |
| +<div class="plain"> |
| + <span class="label">In plain words</span> |
| + <p>You never overwrite the program an instance is running. You put the new program on the shelf next to the old one, point one test instance at it, let the machinery confirm the test instance really is running the new program, and only then point the others at it too. The old program stays on the shelf, so going back is always one command.</p> |
| +</div> |
| +<p>Three background details that protect you during all of this: a <strong>deploy lock</strong> (<code>/opt/lchat/.deploy.lock</code>) makes concurrent deploys, rollbacks, and updater passes take strict turns — the loser fails immediately with “another deploy/rollback/update is running” rather than interleaving. The <strong>router’s dial retry</strong> (<a href="router.html#restart-tolerance">part 02</a>) bridges each instance’s restart gap so members see at most a moment of added latency. And each deploy <strong>restarts the instance’s bot sidecar</strong> right after the instance passes its health check, because the sidecar execs the same versioned link and must track it.</p> |
| + |
| +<h2 id="auto-updates">The automatic path: release channels and the hourly updater</h2> |
| +<p>Most of the time nobody drives deploys by hand: boxes keep themselves current. Three pieces cooperate:</p> |
| +<h3 id="signing">The signed release feed</h3> |
| +<p>Published releases live in a public feed: a manifest file listing, for each <strong>channel</strong>, the newest version, its download URL, and its checksum. There are exactly three channels — <code>stable</code> (the default), <code>beta</code>, and <code>edge</code> (tracks the main branch). The manifest is signed with an Ed25519 key whose public half is compiled into every binary, and every box remembers the newest manifest timestamp it has ever accepted (<code>/opt/lchat/manifest-stamp</code>) and refuses anything older. Together those two checks mean a compromised or stale download server cannot push the fleet backwards or substitute binaries: only the holder of the signing key can move a channel. There is also an optional mirror of the feed in object storage (<code>LCHAT_RELEASE_MIRROR_URL</code>) that boxes fall back to when the primary fails or serves something stale — the mirror adds availability, never trust, because the signature checks are identical on both paths.</p> |
| +<h3 id="update-pass">What one updater pass does</h3> |
| +<p>The timer <code>lchat-update.timer</code> fires <code>lchat update run</code> hourly (with a random spread of up to ten minutes, so a fleet of boxes does not hit the feed simultaneously). One pass, in order: fetch and verify the manifest; ask each instance for its <em>own admin’s preferences</em> (channel choice, auto-update on or off, download-only) over its socket; then, per instance that wants updating: download, checksum, stage, and apply through <em>exactly the same</em> deploy-with-health-check-and-auto-rollback dance as a manual deploy. A build that fails its health check on one instance is <strong>poisoned for the rest of the pass</strong> — the updater will not try it on instances that still pass. One unreachable instance logs and is skipped; it never strands the rest of the fleet. After the pass, builds in the version store that nothing references any more (no instance’s current or previous version, no channel head, no pin) are garbage-collected, so hourly updates on the edge channel stay disk-bounded.</p> |
| +<h3 id="pinning">Pinning: how a canary survives the hourly pass</h3> |
| +<p>A <strong>pin</strong> freezes one instance on one exact version. The updater skips a pinned instance entirely, and its pinned build survives garbage collection. Pins are how a deliberately deployed canary is protected from being “corrected” back to the channel head an hour later. The pin is stored as instance state (the instance’s admin can also see and clear it in the admin panel), and the fleet commands read and write it:</p> |
| +<figure class="cmd"> |
| +<figcaption>Run on the box — the pinning vocabulary</figcaption> |
| +<pre><code>lchat deploy test.example.com --version 4f0a2d19 --pin # deploy a canary and freeze it, one command |
| +lchat update pin test.example.com # freeze on whatever it runs right now |
| +lchat update unpin test.example.com # resume following its channel (next hourly pass) |
| +lchat update status # the fleet update table: |
| +<span class="out">HOST CHANNEL AUTO CURRENT AVAILABLE |
| +chat-a.example.com stable dl+upd 9c21e77e 9c21e77e |
| +test.example.com stable pinned 4f0a2d19 9c21e77e</span></code></pre> |
| +</figure> |
| +<div class="warn"> |
| + <span class="label">Before any fleet-wide action, look for pins</span> |
| + <p>Every <code>pinned</code> row in <code>lchat update status</code> is somebody’s live experiment. <code>deploy --all --version</code> skips them by itself, but channel changes and unpin sweeps do not — check the table first, and leave pinned rows alone unless you know whose they are.</p> |
| +</div> |
| +<p><code>lchat update run --dry-run</code> prints what a pass <em>would</em> do without doing it — useful before enabling anything new. Hibernating instances get updates too, by a quieter route: the updater reads their preferences straight from their on-disk database (read-only) and simply repoints their <code>live</code> link, so they come up on the new build at their next wake, with no health check (a bad build on a hibernator is recovered with <code>rollback</code>).</p> |
| + |
| +<h2 id="sibling-deploys">Deploying the sibling services and the control binary</h2> |
| +<p>The home server and push relay are <em>not</em> versioned instances — they show <code>-</code> in the version column and deploy the plain way: build the <code>home</code> or <code>push</code> binary, copy it over <code>/opt/lchat/home</code> or <code>/opt/lchat/push</code>, and <code>systemctl restart lchat-home</code> (or <code>lchat-push</code>). The control/router binary <code>/opt/lchat/lchat</code> itself is refreshed by copying the new build over it atomically (copy to a temporary name in the same directory, then rename) and restarting <code>lchat-router</code>; the auto-updater does not manage it.</p> |
| + |
| +<h2 id="verifying">Verifying what is deployed where</h2> |
| +<figure class="cmd"> |
| +<figcaption>Run on the box — the questions and their answers</figcaption> |
| +<pre><code># Which build does each instance run? (VERSION column) |
| +lchat instances |
| + |
| +# Which build is a specific instance really running, asked of the process itself? |
| +curl -s https://chat-a.example.com/api/version |
| +<span class="out">{"version":"a41c9f2","build":"9c21e77e"}</span> |
| + |
| +# Which build would the unit start next time? (where the link points) |
| +readlink /opt/lchat/live/chat-a.example.com |
| + |
| +# What builds are on the shelf? |
| +ls /opt/lchat/versions/</code></pre> |
| +</figure> |
| +<p>When the registry, the link, and the process disagree, believe the process (<code>/api/version</code>) about the present and the link about the future — and expect the next restart to reconcile them.</p> |
| + |
| +</div> |
| +</main> |
| +</div> |
| +</body> |
| +</html> |
diff --git a/docs/planet-scale-guide/guide.css b/docs/planet-scale-guide/guide.css| new file mode 100644 |
| index 0000000..328a3f7 |
| --- /dev/null |
| +++ b/docs/planet-scale-guide/guide.css |
| @@ -0,0 +1,444 @@ |
| +/* 🤖 Shared stylesheet for the planet-scale operations guide. |
| + One file styles every page; edit here and every page follows. |
| + Design notes: warm paper + slate-blue ink, serif body for long reading, |
| + sans headings, mono for anything typed into a terminal. Light and dark |
| + are both first-class (prefers-color-scheme plus a data-theme override). */ |
| + |
| +:root { |
| + --paper: #fbfaf7; |
| + --ink: #232f3c; |
| + --muted: #5d6b77; |
| + --accent: #29618e; |
| + --accent-ink: #1d4d75; |
| + --accent-soft: rgba(41, 97, 142, 0.09); |
| + --signal: #a05a18; |
| + --signal-soft: rgba(160, 90, 24, 0.10); |
| + --good: #2f7a55; |
| + --good-soft: rgba(47, 122, 85, 0.10); |
| + --card: #ffffff; |
| + --rule: #dfdcd2; |
| + --shade: #f2f0e9; |
| + --diagram-box: #eef1ec; |
| + --mono: ui-monospace, "SF Mono", SFMono-Regular, Menlo, Consolas, monospace; |
| + --serif: Charter, "Bitstream Charter", "Iowan Old Style", Georgia, serif; |
| + --sans: -apple-system, "Segoe UI", system-ui, sans-serif; |
| +} |
| +@media (prefers-color-scheme: dark) { |
| + :root { |
| + --paper: #141b22; |
| + --ink: #d9e2ea; |
| + --muted: #8fa0ae; |
| + --accent: #7fb3dc; |
| + --accent-ink: #9cc6e8; |
| + --accent-soft: rgba(127, 179, 220, 0.12); |
| + --signal: #dca35c; |
| + --signal-soft: rgba(220, 163, 92, 0.12); |
| + --good: #6cbf94; |
| + --good-soft: rgba(108, 191, 148, 0.12); |
| + --card: #1b242e; |
| + --rule: #2a3641; |
| + --shade: #19222b; |
| + --diagram-box: #1e2933; |
| + } |
| +} |
| +:root[data-theme="dark"] { |
| + --paper: #141b22; --ink: #d9e2ea; --muted: #8fa0ae; --accent: #7fb3dc; |
| + --accent-ink: #9cc6e8; --accent-soft: rgba(127, 179, 220, 0.12); |
| + --signal: #dca35c; --signal-soft: rgba(220, 163, 92, 0.12); |
| + --good: #6cbf94; --good-soft: rgba(108, 191, 148, 0.12); |
| + --card: #1b242e; --rule: #2a3641; --shade: #19222b; --diagram-box: #1e2933; |
| +} |
| +:root[data-theme="light"] { |
| + --paper: #fbfaf7; --ink: #232f3c; --muted: #5d6b77; --accent: #29618e; |
| + --accent-ink: #1d4d75; --accent-soft: rgba(41, 97, 142, 0.09); |
| + --signal: #a05a18; --signal-soft: rgba(160, 90, 24, 0.10); |
| + --good: #2f7a55; --good-soft: rgba(47, 122, 85, 0.10); |
| + --card: #ffffff; --rule: #dfdcd2; --shade: #f2f0e9; --diagram-box: #eef1ec; |
| +} |
| + |
| +html { background: var(--paper); scroll-behavior: smooth; } |
| +@media (prefers-reduced-motion: reduce) { |
| + html { scroll-behavior: auto; } |
| + * { animation: none !important; transition: none !important; } |
| +} |
| +body { |
| + margin: 0; |
| + color: var(--ink); |
| + background: var(--paper); |
| + font: 17px/1.7 var(--serif); |
| + -webkit-text-size-adjust: 100%; |
| +} |
| + |
| +/* ---- page frame: sidebar + content ---- */ |
| +.frame { display: flex; min-height: 100vh; } |
| +nav.side { |
| + width: 250px; |
| + flex: none; |
| + border-right: 1px solid var(--rule); |
| + padding: 28px 20px 40px; |
| + box-sizing: border-box; |
| + position: sticky; |
| + top: 0; |
| + align-self: flex-start; |
| + max-height: 100vh; |
| + overflow-y: auto; |
| + font-family: var(--sans); |
| +} |
| +nav.side .brand { |
| + font: 700 15px/1.3 var(--sans); |
| + margin: 0 0 4px; |
| + letter-spacing: -0.01em; |
| +} |
| +nav.side .brand a { color: var(--ink); text-decoration: none; } |
| +nav.side .brand-sub { |
| + font: 400 12px/1.5 var(--sans); |
| + color: var(--muted); |
| + margin: 0 0 20px; |
| +} |
| +nav.side ol { list-style: none; margin: 0; padding: 0; } |
| +nav.side li { margin: 0 0 2px; } |
| +nav.side a { |
| + display: flex; |
| + gap: 9px; |
| + align-items: baseline; |
| + padding: 6px 9px; |
| + border-radius: 7px; |
| + color: var(--ink); |
| + text-decoration: none; |
| + font: 400 13.5px/1.4 var(--sans); |
| +} |
| +nav.side a .num { |
| + font: 600 10.5px/1 var(--mono); |
| + color: var(--muted); |
| + min-width: 16px; |
| +} |
| +nav.side a:hover { background: var(--accent-soft); } |
| +nav.side a[aria-current="page"] { |
| + background: var(--accent-soft); |
| + color: var(--accent-ink); |
| + font-weight: 600; |
| +} |
| +nav.side a[aria-current="page"] .num { color: var(--accent-ink); } |
| +nav.side a:focus-visible, |
| +a:focus-visible, button:focus-visible, summary:focus-visible { |
| + outline: 2px solid var(--accent); |
| + outline-offset: 2px; |
| +} |
| + |
| +main.content { flex: 1; min-width: 0; } |
| +.wrap { max-width: 730px; margin: 0 auto; padding: 0 28px 110px; } |
| + |
| +@media (max-width: 900px) { |
| + .frame { display: block; } |
| + nav.side { |
| + position: static; |
| + width: auto; |
| + max-height: none; |
| + border-right: none; |
| + border-bottom: 1px solid var(--rule); |
| + padding: 18px 20px; |
| + } |
| + nav.side ol { display: flex; flex-wrap: wrap; gap: 2px 6px; } |
| + .wrap { padding: 0 20px 80px; } |
| +} |
| + |
| +/* ---- page header ---- */ |
| +header.top { padding: 54px 0 6px; } |
| +.eyebrow { |
| + font: 600 12px/1.4 var(--sans); |
| + letter-spacing: 0.13em; |
| + text-transform: uppercase; |
| + color: var(--accent); |
| + margin: 0 0 14px; |
| +} |
| +.eyebrow .robot { letter-spacing: 0; margin-right: 6px; } |
| +h1 { |
| + font: 700 clamp(29px, 4.6vw, 38px)/1.14 var(--sans); |
| + letter-spacing: -0.015em; |
| + margin: 0 0 14px; |
| + text-wrap: balance; |
| +} |
| +.thesis { |
| + font-size: 19px; |
| + line-height: 1.6; |
| + color: var(--muted); |
| + margin: 0 0 10px; |
| + text-wrap: balance; |
| +} |
| +.thesis strong { color: var(--ink); font-weight: 600; } |
| + |
| +h2 { |
| + font: 700 23px/1.28 var(--sans); |
| + letter-spacing: -0.01em; |
| + margin: 62px 0 12px; |
| + text-wrap: balance; |
| +} |
| +h3 { font: 650 17.5px/1.35 var(--sans); margin: 36px 0 10px; } |
| +h4 { font: 650 15px/1.4 var(--sans); margin: 26px 0 8px; } |
| +p { margin: 0 0 17px; } |
| +ul, ol { margin: 0 0 17px; padding-left: 26px; } |
| +li { margin: 0 0 7px; } |
| +a { color: var(--accent-ink); text-decoration-color: color-mix(in srgb, var(--accent) 45%, transparent); text-underline-offset: 2.5px; } |
| +a:hover { text-decoration-color: var(--accent); } |
| +strong { font-weight: 650; } |
| +hr { border: none; border-top: 1px solid var(--rule); margin: 48px 0; } |
| + |
| +/* Anchored headings get a hover ¶ link (added by guide.js). */ |
| +.hlink { |
| + margin-left: 8px; |
| + font: 400 0.75em var(--sans); |
| + color: var(--muted); |
| + text-decoration: none; |
| + opacity: 0; |
| +} |
| +h2:hover .hlink, h3:hover .hlink, .hlink:focus-visible { opacity: 1; } |
| + |
| +/* ---- code ---- */ |
| +code { |
| + font: 0.85em var(--mono); |
| + background: var(--accent-soft); |
| + padding: 1px 5px; |
| + border-radius: 4px; |
| + overflow-wrap: break-word; |
| +} |
| +pre { |
| + font: 13px/1.65 var(--mono); |
| + background: var(--card); |
| + border: 1px solid var(--rule); |
| + border-radius: 10px; |
| + padding: 14px 16px; |
| + overflow-x: auto; |
| + margin: 16px 0; |
| +} |
| +pre code { background: transparent; padding: 0; } |
| +pre .comment { color: var(--muted); } |
| +pre .out { color: var(--muted); } |
| + |
| +/* A command block with a labelled title bar: where to run it, as whom. */ |
| +figure.cmd { margin: 22px 0; } |
| +figure.cmd figcaption { |
| + font: 600 11.5px/1.4 var(--sans); |
| + letter-spacing: 0.08em; |
| + text-transform: uppercase; |
| + color: var(--muted); |
| + background: var(--shade); |
| + border: 1px solid var(--rule); |
| + border-bottom: none; |
| + border-radius: 10px 10px 0 0; |
| + padding: 7px 16px; |
| +} |
| +figure.cmd pre { margin: 0; border-radius: 0 0 10px 10px; } |
| + |
| +/* ---- asides / panels ---- */ |
| +.plain { |
| + background: var(--accent-soft); |
| + border-left: 3px solid var(--accent); |
| + border-radius: 0 10px 10px 0; |
| + padding: 14px 18px 4px; |
| + margin: 24px 0; |
| +} |
| +.plain .label, .warn .label, .why .label { |
| + display: block; |
| + font: 700 11.5px/1 var(--sans); |
| + letter-spacing: 0.11em; |
| + text-transform: uppercase; |
| + margin: 0 0 9px; |
| +} |
| +.plain .label { color: var(--accent-ink); } |
| +.plain p { margin-bottom: 12px; } |
| +.warn { |
| + background: var(--signal-soft); |
| + border-left: 3px solid var(--signal); |
| + border-radius: 0 10px 10px 0; |
| + padding: 14px 18px 4px; |
| + margin: 24px 0; |
| +} |
| +.warn .label { color: var(--signal); } |
| +.warn p { margin-bottom: 12px; } |
| +.why { |
| + background: var(--good-soft); |
| + border-left: 3px solid var(--good); |
| + border-radius: 0 10px 10px 0; |
| + padding: 14px 18px 4px; |
| + margin: 24px 0; |
| +} |
| +.why .label { color: var(--good); } |
| +.why p { margin-bottom: 12px; } |
| + |
| +/* ---- step-by-step procedures ---- */ |
| +ol.procedure { list-style: none; padding: 0; margin: 26px 0; counter-reset: step; } |
| +ol.procedure > li { |
| + counter-increment: step; |
| + position: relative; |
| + padding: 0 0 26px 58px; |
| + margin: 0; |
| +} |
| +ol.procedure > li::before { |
| + content: counter(step); |
| + position: absolute; |
| + left: 0; |
| + top: 1px; |
| + width: 34px; |
| + height: 34px; |
| + border-radius: 50%; |
| + background: var(--accent-soft); |
| + color: var(--accent-ink); |
| + font: 700 15px/34px var(--sans); |
| + text-align: center; |
| +} |
| +ol.procedure > li::after { |
| + content: ""; |
| + position: absolute; |
| + left: 16.5px; |
| + top: 42px; |
| + bottom: 6px; |
| + width: 1px; |
| + background: var(--rule); |
| +} |
| +ol.procedure > li:last-child::after { display: none; } |
| +ol.procedure > li > .step-title { |
| + font: 650 16px/1.4 var(--sans); |
| + margin: 5px 0 8px; |
| +} |
| +ol.procedure pre { margin: 12px 0; } |
| +ol.procedure figure.cmd { margin: 12px 0; } |
| + |
| +/* ---- tables ---- */ |
| +.tablewrap { overflow-x: auto; margin: 20px 0; } |
| +table { |
| + border-collapse: collapse; |
| + width: 100%; |
| + font: 14px/1.55 var(--sans); |
| +} |
| +th { |
| + text-align: left; |
| + font: 650 12px/1.4 var(--sans); |
| + letter-spacing: 0.07em; |
| + text-transform: uppercase; |
| + color: var(--muted); |
| + border-bottom: 2px solid var(--rule); |
| + padding: 8px 14px 8px 0; |
| + white-space: nowrap; |
| +} |
| +td { |
| + border-bottom: 1px solid var(--rule); |
| + padding: 9px 14px 9px 0; |
| + vertical-align: top; |
| +} |
| +td code, th code { white-space: nowrap; } |
| +td:last-child, th:last-child { padding-right: 0; } |
| + |
| +/* ---- the system map (the "you are here" diagram) ---- */ |
| +.sysmap { |
| + margin: 30px 0 8px; |
| + border: 1px solid var(--rule); |
| + border-radius: 12px; |
| + background: var(--card); |
| + padding: 14px 10px 6px; |
| +} |
| +.sysmap svg { display: block; width: 100%; height: auto; } |
| +.sysmap-caption { |
| + font: 13px/1.6 var(--sans); |
| + color: var(--muted); |
| + margin: 8px 0 26px; |
| +} |
| +.sysmap text { font-family: var(--sans); fill: var(--ink); } |
| +.sysmap .boxlabel { font-size: 11.5px; font-weight: 600; } |
| +.sysmap .sublabel { font-size: 9.5px; fill: var(--muted); } |
| +.sysmap .zonelabel { font-size: 9.5px; font-weight: 700; letter-spacing: 0.09em; fill: var(--muted); } |
| +.sysmap rect.node { fill: var(--diagram-box); stroke: var(--rule); stroke-width: 1; rx: 7; } |
| +.sysmap rect.zone { fill: none; stroke: var(--rule); stroke-width: 1; stroke-dasharray: 4 4; rx: 10; } |
| +.sysmap line.wire, .sysmap path.wire { stroke: var(--muted); stroke-width: 1.2; fill: none; } |
| +.sysmap .here rect.node { fill: var(--accent-soft); stroke: var(--accent); stroke-width: 1.6; } |
| +.sysmap .here text.boxlabel { fill: var(--accent-ink); } |
| + |
| +/* generic figure/diagram captions */ |
| +.caption { font: 13px/1.6 var(--sans); color: var(--muted); margin: 10px 0 24px; } |
| +figure { margin: 26px 0; } |
| +figure.diagram { |
| + border: 1px solid var(--rule); |
| + border-radius: 12px; |
| + background: var(--card); |
| + padding: 16px 12px 8px; |
| +} |
| +figure.diagram svg { display: block; width: 100%; height: auto; } |
| +figure.diagram text { font-family: var(--sans); fill: var(--ink); } |
| +figure.diagram figcaption { font: 13px/1.6 var(--sans); color: var(--muted); padding: 8px 8px 6px; } |
| + |
| +/* ---- definition lists (for flags, env vars) ---- */ |
| +dl { margin: 0 0 20px; } |
| +dt { font: 600 14.5px/1.5 var(--sans); margin: 18px 0 4px; } |
| +dt code { font-size: 13.5px; } |
| +dd { margin: 0 0 10px 0; padding-left: 18px; border-left: 2px solid var(--rule); } |
| + |
| +/* ---- build-status badges: what is real and what is aspirational ---- */ |
| +.status { |
| + display: inline-block; |
| + font: 700 10.5px/1 var(--sans); |
| + letter-spacing: 0.08em; |
| + text-transform: uppercase; |
| + padding: 4px 9px 3px; |
| + border-radius: 999px; |
| + vertical-align: 2px; |
| + white-space: nowrap; |
| +} |
| +.status.built { background: var(--good-soft); color: var(--good); border: 1px solid color-mix(in srgb, var(--good) 40%, transparent); } |
| +.status.partial { background: var(--signal-soft); color: var(--signal); border: 1px solid color-mix(in srgb, var(--signal) 40%, transparent); } |
| +.status.planned { background: var(--shade); color: var(--muted); border: 1px dashed color-mix(in srgb, var(--muted) 55%, transparent); } |
| + |
| +/* A framed panel for describing designed-but-unbuilt behaviour, so aspirational |
| + material never reads as something an operator could run today. */ |
| +.roadmap { |
| + border: 1px dashed color-mix(in srgb, var(--muted) 55%, transparent); |
| + border-radius: 12px; |
| + background: var(--shade); |
| + padding: 16px 20px 6px; |
| + margin: 24px 0; |
| +} |
| +.roadmap .label { |
| + display: block; |
| + font: 700 11.5px/1 var(--sans); |
| + letter-spacing: 0.11em; |
| + text-transform: uppercase; |
| + color: var(--muted); |
| + margin: 0 0 9px; |
| +} |
| +.roadmap p { margin-bottom: 12px; } |
| + |
| +/* ---- prev / next footer ---- */ |
| +footer.pager { |
| + display: flex; |
| + gap: 14px; |
| + justify-content: space-between; |
| + margin: 70px 0 0; |
| + padding-top: 26px; |
| + border-top: 1px solid var(--rule); |
| + font-family: var(--sans); |
| +} |
| +footer.pager a { |
| + flex: 1; |
| + max-width: 48%; |
| + border: 1px solid var(--rule); |
| + border-radius: 10px; |
| + padding: 12px 16px; |
| + text-decoration: none; |
| + color: var(--ink); |
| + background: var(--card); |
| +} |
| +footer.pager a:hover { border-color: var(--accent); } |
| +footer.pager .dir { |
| + display: block; |
| + font: 600 11px/1 var(--sans); |
| + letter-spacing: 0.1em; |
| + text-transform: uppercase; |
| + color: var(--muted); |
| + margin-bottom: 6px; |
| +} |
| +footer.pager .dest { font: 600 14px/1.4 var(--sans); color: var(--accent-ink); } |
| +footer.pager a.next { text-align: right; margin-left: auto; } |
| +footer.colophon { |
| + font: 12.5px/1.7 var(--sans); |
| + color: var(--muted); |
| + margin-top: 30px; |
| +} |
| +footer.colophon a { color: var(--muted); } |
diff --git a/docs/planet-scale-guide/guide.js b/docs/planet-scale-guide/guide.js| new file mode 100644 |
| index 0000000..d60644d |
| --- /dev/null |
| +++ b/docs/planet-scale-guide/guide.js |
| @@ -0,0 +1,210 @@ |
| +// 🤖 Shared script for the planet-scale operations guide. |
| +// |
| +// This file is the single source of truth for the guide's structure. It: |
| +// 1. renders the sidebar navigation on every page (from the PAGES list below), |
| +// 2. renders the previous / next links at the bottom of every page, |
| +// 3. renders the "you are here" system map at the top of every page, |
| +// 4. adds a small anchor link to every heading that has an id. |
| +// |
| +// To add a page: create the HTML file (copy an existing page as a template), |
| +// then add one entry to PAGES below, in reading order. Nothing else to update — |
| +// every page's sidebar and pager follow this list automatically. |
| +// |
| +// All markup injected here is built from static, in-code strings only (the |
| +// repo rule: never innerHTML untrusted content). |
| + |
| +"use strict"; |
| + |
| +const PAGES = [ |
| + { file: "index.html", num: "01", short: "Overview", title: "Overview and orientation" }, |
| + { file: "router.html", num: "02", short: "The router", title: "The router: the front door of every box" }, |
| + { file: "processes.html", num: "03", short: "Processes on a box", title: "Everything that runs on a box" }, |
| + { file: "deploys.html", num: "04", short: "Deploys and updates", title: "Installing and updating the software" }, |
| + { file: "spaces.html", num: "05", short: "Creating spaces", title: "Creating and removing chat spaces" }, |
| + { file: "hibernation.html", num: "06", short: "Hibernation", title: "Hibernation: how idle spaces sleep in object storage" }, |
| + { file: "mesh.html", num: "07", short: "The mesh", title: "The mesh: running more than one box" }, |
| + { file: "operations.html", num: "08", short: "Day-to-day operations", title: "Day-to-day operations and checks" }, |
| + { file: "reference.html", num: "09", short: "Reference", title: "Complete reference: commands, files, variables" }, |
| +]; |
| + |
| +function currentPage() { |
| + const explicit = document.body.dataset.page; |
| + if (explicit) return explicit; |
| + const leaf = location.pathname.split("/").pop(); |
| + return leaf === "" ? "index.html" : leaf; |
| +} |
| + |
| +// ---- sidebar ---- |
| +function renderSidebar() { |
| + const nav = document.querySelector("nav.side"); |
| + if (!nav) return; |
| + const here = currentPage(); |
| + const items = PAGES.map(p => { |
| + const current = p.file === here ? ' aria-current="page"' : ""; |
| + return `<li><a href="${p.file}"${current}><span class="num">${p.num}</span><span>${p.short}</span></a></li>`; |
| + }).join(""); |
| + nav.innerHTML = |
| + `<p class="brand"><a href="index.html">Running the fleet</a></p>` + |
| + `<p class="brand-sub">An operator’s guide to the planet-scale system</p>` + |
| + `<ol>${items}</ol>`; |
| + nav.setAttribute("aria-label", "Guide contents"); |
| +} |
| + |
| +// ---- previous / next pager + colophon ---- |
| +function renderPager() { |
| + const wrap = document.querySelector("main.content .wrap"); |
| + if (!wrap) return; |
| + const here = currentPage(); |
| + const idx = PAGES.findIndex(p => p.file === here); |
| + if (idx === -1) return; |
| + const prev = PAGES[idx - 1]; |
| + const next = PAGES[idx + 1]; |
| + const pager = document.createElement("footer"); |
| + pager.className = "pager"; |
| + let html = ""; |
| + if (prev) html += `<a class="prev" href="${prev.file}"><span class="dir">Previous</span><span class="dest">${prev.num} · ${prev.short}</span></a>`; |
| + if (next) html += `<a class="next" href="${next.file}"><span class="dir">Next</span><span class="dest">${next.num} · ${next.short}</span></a>`; |
| + pager.innerHTML = html; |
| + wrap.appendChild(pager); |
| + |
| + const colophon = document.createElement("footer"); |
| + colophon.className = "colophon"; |
| + colophon.innerHTML = |
| + `\u{1F916} This guide was written by an LLM from the code and design documents in this repository. ` + |
| + `The code is the source of truth; if the guide and the code disagree, trust the code and please fix the guide. ` + |
| + `How to update it: <a href="README.html">maintaining this guide</a>. ` + |
| + `Deeper design rationale: <code>docs/planet-scale.md</code> and <code>docs/instance-router.md</code>.`; |
| + wrap.appendChild(colophon); |
| +} |
| + |
| +// ---- heading anchors ---- |
| +function renderHeadingAnchors() { |
| + document.querySelectorAll("h2[id], h3[id]").forEach(h => { |
| + const a = document.createElement("a"); |
| + a.className = "hlink"; |
| + a.href = "#" + h.id; |
| + a.textContent = "#"; |
| + a.setAttribute("aria-label", "Link to this section"); |
| + h.appendChild(a); |
| + }); |
| +} |
| + |
| +// ---- the system map (the "you are here" diagram) ---- |
| +// |
| +// One SVG, drawn identically on every page; the page highlights its own |
| +// component(s) via <div class="sysmap" data-highlight="router bucket">. |
| +// Node keys: members, router, instances, services, control, updater, |
| +// bucket, backup, peer. |
| +function sysmapSVG(highlights) { |
| + const hl = key => highlights.includes(key) ? ' class="here"' : ""; |
| + return ` |
| +<svg viewBox="0 0 960 400" role="img" aria-label="Map of the whole system: members' devices reach a box over the internet; on the box a router forwards traffic to chat space instances and sibling services; a control daemon and provisioner create spaces; an updater installs new versions; object storage holds parked spaces, leases, heartbeats and backups; peer boxes form the mesh."> |
| + <!-- zone: the internet --> |
| + <rect class="zone" x="10" y="34" width="140" height="330" rx="10"></rect> |
| + <text class="zonelabel" x="24" y="24">THE INTERNET</text> |
| + <g${hl("members")}> |
| + <rect class="node" x="26" y="150" width="108" height="72" rx="7"></rect> |
| + <text class="boxlabel" x="80" y="180" text-anchor="middle">Members’</text> |
| + <text class="boxlabel" x="80" y="196" text-anchor="middle">devices</text> |
| + <text class="sublabel" x="80" y="211" text-anchor="middle">browsers, apps</text> |
| + </g> |
| + |
| + <!-- zone: one box --> |
| + <rect class="zone" x="180" y="34" width="440" height="330" rx="10"></rect> |
| + <text class="zonelabel" x="194" y="24">ONE BOX (A LINUX SERVER)</text> |
| + |
| + <g${hl("router")}> |
| + <rect class="node" x="200" y="60" width="150" height="86" rx="7"></rect> |
| + <text class="boxlabel" x="275" y="88" text-anchor="middle">lchat router</text> |
| + <text class="sublabel" x="275" y="104" text-anchor="middle">TLS on ports 80 / 443</text> |
| + <text class="sublabel" x="275" y="119" text-anchor="middle">host → unix socket</text> |
| + <text class="sublabel" x="275" y="134" text-anchor="middle">activator lives inside</text> |
| + </g> |
| + |
| + <g${hl("instances")}> |
| + <rect class="node" x="420" y="52" width="180" height="34" rx="7"></rect> |
| + <text class="boxlabel" x="510" y="73" text-anchor="middle">instance: chat-a.example.com</text> |
| + <rect class="node" x="420" y="94" width="180" height="34" rx="7"></rect> |
| + <text class="boxlabel" x="510" y="115" text-anchor="middle">instance: chat-b.example.com</text> |
| + <rect class="node" x="420" y="136" width="180" height="34" rx="7"></rect> |
| + <text class="boxlabel" x="510" y="157" text-anchor="middle">bot sidecars (one per instance)</text> |
| + </g> |
| + |
| + <g${hl("services")}> |
| + <rect class="node" x="420" y="186" width="180" height="46" rx="7"></rect> |
| + <text class="boxlabel" x="510" y="205" text-anchor="middle">sibling services</text> |
| + <text class="sublabel" x="510" y="221" text-anchor="middle">home server, push relay</text> |
| + </g> |
| + |
| + <g${hl("control")}> |
| + <rect class="node" x="200" y="186" width="150" height="62" rx="7"></rect> |
| + <text class="boxlabel" x="275" y="207" text-anchor="middle">control daemon +</text> |
| + <text class="boxlabel" x="275" y="222" text-anchor="middle">cloud provisioner</text> |
| + <text class="sublabel" x="275" y="238" text-anchor="middle">creates chat spaces</text> |
| + </g> |
| + |
| + <g${hl("updater")}> |
| + <rect class="node" x="200" y="272" width="150" height="62" rx="7"></rect> |
| + <text class="boxlabel" x="275" y="295" text-anchor="middle">auto-updater</text> |
| + <text class="sublabel" x="275" y="311" text-anchor="middle">hourly timer; installs</text> |
| + <text class="sublabel" x="275" y="325" text-anchor="middle">signed releases</text> |
| + </g> |
| + |
| + <!-- zone: object storage --> |
| + <rect class="zone" x="650" y="34" width="300" height="190" rx="10"></rect> |
| + <text class="zonelabel" x="664" y="24">OBJECT STORAGE (AMAZON S3)</text> |
| + |
| + <g${hl("bucket")}> |
| + <rect class="node" x="668" y="52" width="264" height="86" rx="7"></rect> |
| + <text class="boxlabel" x="800" y="76" text-anchor="middle">hibernation bucket</text> |
| + <text class="sublabel" x="800" y="93" text-anchor="middle">parked space databases · per-space locks</text> |
| + <text class="sublabel" x="800" y="108" text-anchor="middle">mesh membership heartbeats</text> |
| + <text class="sublabel" x="800" y="123" text-anchor="middle">one prefix per chat space</text> |
| + </g> |
| + |
| + <g${hl("backup")}> |
| + <rect class="node" x="668" y="152" width="264" height="56" rx="7"></rect> |
| + <text class="boxlabel" x="800" y="176" text-anchor="middle">fleet-backup bucket</text> |
| + <text class="sublabel" x="800" y="192" text-anchor="middle">continuous copies of every database</text> |
| + </g> |
| + |
| + <!-- zone: peer boxes --> |
| + <rect class="zone" x="650" y="252" width="300" height="112" rx="10"></rect> |
| + <text class="zonelabel" x="664" y="246">OTHER BOXES — THE MESH</text> |
| + |
| + <g${hl("peer")}> |
| + <rect class="node" x="668" y="268" width="264" height="38" rx="7"></rect> |
| + <text class="boxlabel" x="800" y="292" text-anchor="middle">peer box 2 (same software)</text> |
| + <rect class="node" x="668" y="314" width="264" height="38" rx="7"></rect> |
| + <text class="boxlabel" x="800" y="338" text-anchor="middle">peer box 3 (same software)</text> |
| + </g> |
| + |
| + <!-- wires --> |
| + <line class="wire" x1="134" y1="186" x2="200" y2="110" marker-end="url(#arr)"></line> |
| + <line class="wire" x1="350" y1="90" x2="420" y2="70" marker-end="url(#arr)"></line> |
| + <line class="wire" x1="350" y1="100" x2="420" y2="110" marker-end="url(#arr)"></line> |
| + <line class="wire" x1="350" y1="120" x2="420" y2="205" marker-end="url(#arr)"></line> |
| + <line class="wire" x1="350" y1="80" x2="668" y2="80" marker-end="url(#arr)"></line> |
| + <line class="wire" x1="600" y1="180" x2="668" y2="180" marker-end="url(#arr)"></line> |
| + <line class="wire" x1="800" y1="268" x2="800" y2="224" marker-end="url(#arr)"></line> |
| + <defs> |
| + <marker id="arr" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse"> |
| + <path d="M0,0 L8,4 L0,8 z" fill="currentColor" opacity="0.55"></path> |
| + </marker> |
| + </defs> |
| +</svg>`; |
| +} |
| + |
| +function renderSysmap() { |
| + const holder = document.querySelector(".sysmap"); |
| + if (!holder) return; |
| + const highlights = (holder.dataset.highlight || "").split(/\s+/).filter(Boolean); |
| + holder.innerHTML = sysmapSVG(highlights); |
| +} |
| + |
| +document.addEventListener("DOMContentLoaded", () => { |
| + renderSidebar(); |
| + renderSysmap(); |
| + renderHeadingAnchors(); |
| + renderPager(); |
| +}); |
diff --git a/docs/planet-scale-guide/hibernation.html b/docs/planet-scale-guide/hibernation.html| new file mode 100644 |
| index 0000000..fd63283 |
| --- /dev/null |
| +++ b/docs/planet-scale-guide/hibernation.html |
| @@ -0,0 +1,152 @@ |
| +<!doctype html> |
| +<html lang="en"> |
| +<head> |
| +<meta charset="utf-8" /> |
| +<meta name="viewport" content="width=device-width, initial-scale=1" /> |
| +<title>Hibernation — Running the fleet</title> |
| +<link rel="stylesheet" href="guide.css" /> |
| +<script src="guide.js" defer></script> |
| +</head> |
| +<body data-page="hibernation.html"> |
| +<div class="frame"> |
| +<nav class="side"></nav> |
| +<main class="content"> |
| +<div class="wrap"> |
| + |
| +<header class="top"> |
| + <p class="eyebrow"><span class="robot">🤖</span> Running the fleet · Part 06 of 09</p> |
| + <h1>Hibernation: how idle spaces sleep in object storage</h1> |
| + <p class="thesis">A chat space with nobody talking does not deserve a running process. With hibernation enabled, an idle space’s process <strong>exits</strong>, its database is <strong>parked in an S3 bucket</strong>, and the next request (or the next scheduled reminder) <strong>wakes it in a few seconds</strong>. This page explains the machinery — the activator, the lease, and the replication — and every command and configuration knob around it.</p> |
| +</header> |
| + |
| +<div class="sysmap" data-highlight="router bucket"></div> |
| +<p class="sysmap-caption">You are here: the activator lives <em>inside the router process</em>, and its durable state lives in the hibernation bucket. Nothing new runs on the box — hibernation is a behaviour of the router.</p> |
| + |
| +<h2 id="why-safe">Why parking chat data in a bucket is safe at all</h2> |
| +<p>This whole design is only possible because of the product’s encryption rule: an instance’s database contains end-to-end ciphertext plus routing metadata the server already sees, and never a usable decryption key. Copying that file into an object-storage bucket therefore changes nothing about who can read messages — nobody new can. The operational risk of hibernation is availability and consistency, not confidentiality, and the rest of this page is about how consistency is guaranteed.</p> |
| + |
| +<h2 id="activator-config">Enabling hibernation: one bucket variable, then one flag per host</h2> |
| +<p>Hibernation is enabled in two deliberately separate steps, so that turning the machinery on is fleet-safe and adopting it is gradual:</p> |
| +<h3 id="box-switch">Step one: give the router a bucket (per box)</h3> |
| +<p>The <strong>activator</strong> — the component that wakes and sleeps instances — is part of the router process, and it exists only when the router’s environment names a bucket. In <code>/data/router.env</code>:</p> |
| +<figure class="cmd"> |
| +<figcaption>Contents of /data/router.env (then: systemctl restart lchat-router)</figcaption> |
| +<pre><code>LCHAT_HIBERNATE_BUCKET=my-hibernation-bucket # the master switch; unset = the activator does not exist |
| +LCHAT_HIBERNATE_REGION=eu-west-1 # optional; the AWS default chain applies otherwise |
| +LCHAT_HIBERNATE_PREFIX=prod # optional key prefix inside the bucket |
| +LCHAT_HIBERNATE_IDLE=15m # optional idle window before sleep; default 15m</code></pre> |
| +</figure> |
| +<p>The router also needs AWS credentials able to read and write that bucket, supplied the standard AWS ways (environment variables in the same file, or an instance role). Setting the bucket alone changes <em>nothing</em> for any instance: it only makes the activator available.</p> |
| +<h3 id="host-switch">Step two: flip individual hosts (per host)</h3> |
| +<p>Each host opts in via the <code>hibernate</code> flag on its registry row, flipped by one command in each direction. This per-host granularity is what made the feature safe to prove in production: one canary space migrated first, everything else untouched.</p> |
| +<figure class="cmd"> |
| +<figcaption>Run on the box, as root</figcaption> |
| +<pre><code>lchat hibernate chat-b.example.com --on # hand the host to the activator |
| +lchat hibernate chat-b.example.com --off # hand it back to systemd |
| +lchat instances # the HIBERNATE column shows who is activator-managed</code></pre> |
| +</figure> |
| +<p>What <code>--on</code> actually does, in order: fixes the ownership of the host’s environment file (the activator reads it as the service user, whereas systemd read it as root — a real wake failed on exactly this before it was baked in); sets the registry flag; <strong>refuses to proceed if the box-wide backup replicator still tracks this database</strong> (two replicators on one database corrupt each other’s state — re-run <code>deploy/fleet-backup.sh</code> to regenerate the backup configuration without this host, then retry); stops and disables the host’s systemd unit (and its bot sidecar unit, if present); reloads the router; and finally makes one warm request so the host is immediately woken by the activator rather than left to the first member. The ordering means the host is never unserveable: requests arriving in the brief socket-dead gap ride the router’s <a href="router.html#restart-tolerance">thirteen-second dial retry</a>.</p> |
| +<p><code>--off</code> is the mirror image: clear the flag, reload the router (which makes the activator checkpoint the space to the bucket, release its lock, and stop the child), <em>wait for the child to actually release the socket</em>, then re-enable the systemd unit and health-check it, then re-enable the sidecar unit. The wait matters — an instance refuses to start over a socket that still has a listener.</p> |
| + |
| +<h2 id="the-lease">The lease: one writer, enforced by the bucket itself</h2> |
| +<p>The central safety problem of hibernation is: <em>what guarantees that two processes never write the same space’s database at once?</em> (Two boxes might both think they should wake a space; a wedged old process might still be running when a new one starts.) The answer is <strong>the lease</strong> — a small lock object in the bucket, at <code><prefix>/<host>/lock.json</code>, acquired using S3’s conditional writes (an operation that atomically fails if the object already exists). Whoever holds a space’s lease is the one writer; everyone else fails loudly and immediately.</p> |
| +<ul> |
| + <li>The lease has a <strong>30-second lifetime</strong> and its holder renews it roughly every <strong>10 seconds</strong>. A process that stops renewing — crashed, wedged, unplugged — loses the lease within about 30 seconds, and a healthy process elsewhere can then claim it. <strong>That ~30 seconds is the failover window of the entire design.</strong></li> |
| + <li>A failed renewal is terminal on purpose: the holder <em>stands down</em> (checkpoints nothing further, stops serving) rather than retrying, because a retry could collide with a new holder that legitimately took over after expiry.</li> |
| + <li>Restoring a database from the bucket happens only <em>while holding the lease</em>. Restoring outside it could resurrect a stale copy that shadows the true holder’s newer state.</li> |
| +</ul> |
| +<p>Replication itself is performed by <a href="reference.html#gloss-litestream">Litestream</a> — a well-known tool that continuously streams SQLite changes to object storage — embedded <em>as a library</em> inside the router. The embedding is not an implementation detail but a hard-won correction: the Litestream command-line tool, run twice against one database, <em>silently</em> absorbs the second copy and orphans one side’s writes with no error at all; only the library interface exposes the lease that turns a second writer into a loud, typed failure. The build chose the loud failure.</p> |
| +<div class="plain"> |
| + <span class="label">In plain words</span> |
| + <p>Every sleeping space has a key hanging on a hook in the shared storage room, and the rule is absolute: you may only touch a space’s data while you are holding its key, and the key dissolves in your hand if you stop checking in every few seconds. Two people can never hold it at once, and a collapsed keyholder drops it within half a minute.</p> |
| +</div> |
| + |
| +<h2 id="lifecycle">The life of a hibernating space</h2> |
| +<figure class="diagram"> |
| +<svg viewBox="0 0 740 120" role="img" aria-label="State machine: asleep, then waking on a request, then awake, then stopping after idleness, then asleep again."> |
| + <rect x="20" y="35" width="120" height="50" rx="8" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| + <text x="80" y="58" font-size="12" font-weight="600" text-anchor="middle">asleep</text> |
| + <text x="80" y="74" font-size="9.5" fill="var(--muted)" text-anchor="middle">no process at all</text> |
| + <rect x="210" y="35" width="120" height="50" rx="8" fill="var(--accent-soft)" stroke="var(--accent)"></rect> |
| + <text x="270" y="58" font-size="12" font-weight="600" text-anchor="middle">waking</text> |
| + <text x="270" y="74" font-size="9.5" fill="var(--muted)" text-anchor="middle">lease · restore · spawn</text> |
| + <rect x="400" y="35" width="120" height="50" rx="8" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| + <text x="460" y="58" font-size="12" font-weight="600" text-anchor="middle">awake</text> |
| + <text x="460" y="74" font-size="9.5" fill="var(--muted)" text-anchor="middle">serving; replicating</text> |
| + <rect x="590" y="35" width="120" height="50" rx="8" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| + <text x="650" y="58" font-size="12" font-weight="600" text-anchor="middle">stopping</text> |
| + <text x="650" y="74" font-size="9.5" fill="var(--muted)" text-anchor="middle">checkpoint · release</text> |
| + <line x1="140" y1="60" x2="210" y2="60" stroke="var(--muted)" stroke-width="1.2"></line> |
| + <text x="175" y="52" font-size="9" fill="var(--muted)" text-anchor="middle">request</text> |
| + <line x1="330" y1="60" x2="400" y2="60" stroke="var(--muted)" stroke-width="1.2"></line> |
| + <line x1="520" y1="60" x2="590" y2="60" stroke="var(--muted)" stroke-width="1.2"></line> |
| + <text x="555" y="52" font-size="9" fill="var(--muted)" text-anchor="middle">idle 15m</text> |
| + <path d="M 650 85 C 650 110, 80 110, 80 85" fill="none" stroke="var(--muted)" stroke-width="1.2"></path> |
| +</svg> |
| +<figcaption>The four states. “Asleep” is literally the absence of any process or bookkeeping entry — a sleeping space costs only disk.</figcaption> |
| +</figure> |
| +<h3 id="wake">Waking: what happens when a request arrives for a sleeping space</h3> |
| +<ol> |
| + <li>The router sees a request for a host flagged as hibernating with no live process, and hands it to the activator. Many simultaneous requests coalesce into <strong>one</strong> wake — the rest simply wait for it.</li> |
| + <li>The activator <strong>claims the lease</strong> at <code><prefix>/<host>/lock.json</code>. If another holder has it (for example, another box in a <a href="mesh.html">mesh</a>), the request is answered <code>503</code> with a <code>Retry-After</code> header — deliberately: “sleeping spaces are slow to wake” must degrade to a polite retry, never a conflict.</li> |
| + <li>It <strong>restores the database from the bucket only if the local file is missing</strong>. A database already on disk is trusted as-is — on a single box, under the lease, the local copy can only be this box’s own last checkpoint, so most wakes skip the download entirely. This is the “warm cache”, and it is why routine wakes are fast.</li> |
| + <li>It <strong>starts the instance as a direct child process</strong> of the router — the same versioned binary its systemd unit would run (<code>/opt/lchat/live/<host></code>), with the same environment file (<code>/data/<host>.env</code>) — and waits up to 30 seconds for the socket to answer.</li> |
| + <li>It <strong>starts replication under the held lease</strong>, so every subsequent write streams to the bucket within about a second, and finally spawns the bot sidecar if one is staged (<a href="#sidecar-co-spawn">below</a>). Then the waiting requests proceed.</li> |
| +</ol> |
| +<p>The entire wake has a five-minute budget; a queued request waits at most 30 seconds before giving up with a 503. A <em>failed</em> wake discards all bookkeeping so the next request retries from scratch — one bad wake never becomes a stuck host.</p> |
| +<h3 id="sleep">Sleeping: what happens when a space goes idle</h3> |
| +<p>A background sweep (every few seconds) puts an instance to sleep when it has had <strong>zero requests in flight for the whole idle window</strong> (default fifteen minutes, set by <code>LCHAT_HIBERNATE_IDLE</code>). Both conditions matter: an open WebSocket counts as an in-flight request for its entire life, so <em>a space with anyone connected never sleeps</em>, no matter how quiet the conversation. The stop sequence, in order:</p> |
| +<ol> |
| + <li><strong>Stop the bot sidecar first</strong> (an orphaned sidecar trying to reconnect would immediately re-wake the space).</li> |
| + <li><strong>Ask the instance for its next scheduled delivery</strong> (the earliest future message or reminder) and record it in the registry’s <code>next_due</code> column — the hook for <a href="#due-time-wakes">scheduled wakes</a>.</li> |
| + <li><strong>Terminate the child gracefully</strong> (its own drain is the final write), with a 20-second budget before a hard kill.</li> |
| + <li><strong>Take the final checkpoint to the bucket and release the lease</strong>, each under its own time budget so a wedged upload can never strand the lease until it expires on its own.</li> |
| + <li>Forget the entry. The local database file <strong>stays on disk as the warm cache</strong>; the bucket copy is the durable truth, guarded by the (now released) lease.</li> |
| +</ol> |
| +<h3 id="failure">Failure is just the wake path again</h3> |
| +<p>Every failure mode collapses into “stand down; the next request re-wakes”: a child that dies unexpectedly, a lease lost to a renewal failure, a checkpoint that errors. And an S3 outage while spaces are <em>awake</em> is survivable by design — the local SQLite file is authoritative while the lease is held, replication queues and drains when the bucket returns, and only <em>cold</em> wakes (no local file) actually require the bucket to answer.</p> |
| + |
| +<h2 id="due-time-wakes">Scheduled deliveries wake sleeping spaces on time</h2> |
| +<p>Spaces can hold scheduled events — a reminder due at 9:00, a message scheduled for tomorrow. A naive hibernation design would sleep through them. Instead, at sleep time the space reports its earliest pending delivery, the activator persists that timestamp in the registry (so it survives router restarts), and the same background sweep wakes any sleeping host whose time has arrived — no member request needed. Worst-case lateness is about one sweep interval (thirty seconds) plus a wake. Bot schedules created by members (“send me a news summary every morning”) live encrypted in the sidecar, which the server cannot read — so the sidecar itself reports its earliest scheduled run to the instance (one bare timestamp crosses; the schedule stays ciphertext), and the instance folds it into the same answer.</p> |
| + |
| +<h2 id="sidecar-co-spawn">Bot sidecars on hibernating spaces: co-spawned, and invisible to the idle clock</h2> |
| +<p>A persistent sidecar service would defeat hibernation twice over — its always-open WebSocket would hold the space awake forever, and if the space did sleep, the sidecar’s reconnection attempts would wake it right back. So on a hibernating host the sidecar’s systemd unit is disabled (the <code>hibernate --on</code> flip does this) and the activator <strong>co-spawns</strong> the sidecar as its own second child: started right after the instance turns healthy, terminated first at sleep. The sidecar exists exactly while its space is awake. Two details make the arrangement sound:</p> |
| +<ul> |
| + <li>The sidecar marks every request it makes with a header (<code>X-Lchat-Sidecar</code>), and the activator makes that traffic <strong>activity-invisible</strong>: it earns no idle-clock credit while the space is awake and receives a 503 (never a wake) while it sleeps. Forging the marker is harmless — it only downgrades the forger’s own requests, and the router strips the header from genuine outside traffic anyway.</li> |
| + <li>If the co-spawned sidecar dies, the activator restarts it with capped exponential backoff; if it will not start at all, the space still serves — members before bots. The old Node-based reference sidecar does not send the marker and is therefore unsuitable for hibernating hosts; the Go sidecar (the default) is required there.</li> |
| +</ul> |
| + |
| +<h2 id="children">Observing hibernating spaces (they are not in systemctl)</h2> |
| +<p>This is the operational surprise worth internalising from this page: <strong>an awake hibernating instance has no systemd unit.</strong> It is a child process of the router. Your usual reflexes translate as follows:</p> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Question</th><th>For a systemd-managed host</th><th>For a hibernating host</th></tr> |
| + <tr><td>Is it running?</td><td><code>systemctl status lchat@<host></code></td><td><code>ss -xlH | grep <host></code> — a bound socket means awake; no socket means asleep (which is normal, not a fault)</td></tr> |
| + <tr><td>Its logs?</td><td><code>journalctl -u lchat@<host></code></td><td><code>journalctl -u lchat-router</code> — the child’s output flows through the router’s journal, alongside the activator’s own wake/sleep/lease lines</td></tr> |
| + <tr><td>Restart it?</td><td><code>systemctl restart lchat@<host></code></td><td>Usually unnecessary — but any request wakes it, and <code>lchat hibernate <host> --off</code> then <code>--on</code> is the full reset</td></tr> |
| + <tr><td>Change its configuration?</td><td>Edit <code>/data/<host>.env</code>, restart the unit</td><td>Edit the same file; it is re-read at the next wake automatically</td></tr> |
| + <tr><td>Deploy a version to it?</td><td><code>lchat deploy <host> --version <ver></code> (restart + health check)</td><td>Same command — but it only repoints the link; the new build runs at the next wake, with no health check, so keep <code>rollback</code> in mind</td></tr> |
| +</table> |
| +</div> |
| +<figure class="cmd"> |
| +<figcaption>Run on the box — watching a wake happen</figcaption> |
| +<pre><code>journalctl -u lchat-router -f & |
| +curl -s https://chat-b.example.com/api/version |
| +<span class="out">activator: chat-b.example.com waking (lease acquired) |
| +activator: chat-b.example.com restored: local db present, skipping restore |
| +activator: chat-b.example.com awake (2.1s) |
| +{"version":"a41c9f2","build":"1b4bd808"}</span></code></pre> |
| +</figure> |
| + |
| +<h2 id="hibernation-gotchas">The sharp edges, learned in production</h2> |
| +<ul> |
| + <li><strong>Never let two replicators track one database.</strong> The box-wide fleet backup (<a href="operations.html#backups">part 08</a>) and the activator’s embedded replication must divide the databases between them: the <code>hibernate --on</code> flip refuses while the backup configuration still lists the host, and the backup script skips hibernating hosts when regenerating. If you ever edit <code>/etc/litestream.yml</code> by hand, preserve that invariant — the failure mode is mutual corruption of replication state.</li> |
| + <li><strong>Environment file ownership follows the reader.</strong> Systemd read <code>/data/<host>.env</code> as root; the activator reads it as the service user. The flip commands and <code>add-instance</code> fix ownership automatically now, but a hand-copied environment file with root-only permissions will fail its first wake with a permission error.</li> |
| + <li><strong>The 503s are a feature.</strong> A lease conflict or a wake timeout answers <code>503 with Retry-After</code>. Clients treat it as “try again in a moment”, which is the honest truth. A burst of 503s on one host is worth investigating; scattered occasional ones during wakes are the design working.</li> |
| +</ul> |
| + |
| +</div> |
| +</main> |
| +</div> |
| +</body> |
| +</html> |
diff --git a/docs/planet-scale-guide/index.html b/docs/planet-scale-guide/index.html| new file mode 100644 |
| index 0000000..7383826 |
| --- /dev/null |
| +++ b/docs/planet-scale-guide/index.html |
| @@ -0,0 +1,142 @@ |
| +<!doctype html> |
| +<html lang="en"> |
| +<head> |
| +<meta charset="utf-8" /> |
| +<meta name="viewport" content="width=device-width, initial-scale=1" /> |
| +<title>Overview and orientation — Running the fleet</title> |
| +<link rel="stylesheet" href="guide.css" /> |
| +<script src="guide.js" defer></script> |
| +</head> |
| +<body data-page="index.html"> |
| +<div class="frame"> |
| +<nav class="side"></nav> |
| +<main class="content"> |
| +<div class="wrap"> |
| + |
| +<header class="top"> |
| + <p class="eyebrow"><span class="robot">🤖</span> Running the fleet · Part 01 of 09</p> |
| + <h1>Overview and orientation</h1> |
| + <p class="thesis">This guide explains, from the ground up, how the Eleven Messenger fleet is deployed and operated: <strong>what runs where, how each piece is started and supervised, and exactly which commands you type</strong> to add, remove, inspect, update, and repair things.</p> |
| +</header> |
| + |
| +<div class="sysmap" data-highlight=""></div> |
| +<p class="sysmap-caption">The whole system on one map. Every detailed page in this guide reproduces this map with the part it covers highlighted, so you always know where you are.</p> |
| + |
| +<h2 id="who-this-is-for">Who this guide is for, and what it assumes</h2> |
| +<p>This guide is written for a person who needs to operate this system — not for the people who wrote it. It assumes you are comfortable with the basics of system administration: you can open a terminal, connect to a server with <code>ssh</code>, edit a file, and run a command. It assumes only a passing familiarity with <a href="reference.html#gloss-systemd">systemd</a> (the Linux service manager), <a href="reference.html#gloss-dns">DNS</a>, and <a href="reference.html#gloss-s3">object storage</a> — every one of those is explained where it first matters, and again in the <a href="reference.html#glossary">glossary</a>.</p> |
| +<p>Everything here is deliberately repetitive in places. Key ideas — for example, that <em>one chat space is one process with one database file</em>, or that <em>the object-storage lock is the real source of truth about who may write</em> — are restated wherever they matter, because internalising those few ideas makes every command in this guide predictable rather than memorised.</p> |
| + |
| +<div class="plain"> |
| + <span class="label">In plain words</span> |
| + <p>You are looking after a small number of Linux servers. Each server runs one program, called <code>lchat</code>, which plays several roles at once. Chat groups live on these servers as small, separate processes, one per group. Idle groups are put to sleep and their data is parked in Amazon’s object storage. This guide teaches you how to look after all of that.</p> |
| +</div> |
| + |
| +<h2 id="what-the-product-is">What the product is, in one paragraph</h2> |
| +<p>Eleven Messenger is an end-to-end encrypted group chat product. Every chat group — called a <strong>space</strong> — runs as its own fully isolated server-side unit: its own process, its own SQLite database file, its own admin token, and its own subdomain (for example <code>chat-a.example.com</code>). The server side only ever stores and relays <em>ciphertext</em> — encrypted bytes it cannot read — because all encryption and decryption happens on members’ devices. This matters operationally: <strong>nothing you do as an operator can expose message content</strong>, because the servers never hold the keys. Copying a database file, replicating it to object storage, or moving it between machines is safe by construction.</p> |
| + |
| +<h2 id="the-one-idea">The one idea behind “planet scale”</h2> |
| +<p>The system is designed so that it can grow to a very large number of spaces without a large number of servers, and the design is captured in the repository document <code>docs/planet-scale.md</code>. The core idea has three parts:</p> |
| +<ul> |
| + <li><strong>One space, one process, one database file — forever.</strong> Spaces are never consolidated into a shared database. This is simultaneously the isolation story (one space crashing cannot hurt another), the sharding story (a space can move to any machine by moving one file), and the security story.</li> |
| + <li><strong>An idle space does not deserve a running process.</strong> Because a space’s entire state is one SQLite file of ciphertext, that file can be parked in object storage (Amazon S3) when nobody is talking, and the process can exit. When someone next opens the space, the system restores the file and starts the process again, in a few seconds. This is called <a href="hibernation.html"><strong>hibernation</strong></a>.</li> |
| + <li><strong>Several servers can share the work without a central coordinator.</strong> Each server (this guide calls them <strong>boxes</strong>) announces itself by writing a small heartbeat object into the shared S3 bucket. A well-known hashing rule decides which box should host which space, and a per-space lock object in S3 guarantees that two boxes can never write the same space’s data at the same time. This is called <a href="mesh.html"><strong>the mesh</strong></a>.</li> |
| +</ul> |
| + |
| +<h2 id="vocabulary">The vocabulary this guide uses everywhere</h2> |
| +<p>Nine words carry most of the weight in this guide. They are defined properly in the <a href="reference.html#glossary">glossary</a>, but you will get much further if you internalise them now:</p> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Word</th><th>Meaning</th></tr> |
| + <tr><td><strong>space</strong></td><td>One chat group, as members experience it. Server-side, a space is one instance (next row).</td></tr> |
| + <tr><td><strong>instance</strong></td><td>The server-side unit behind a space: one process, one SQLite database file, one subdomain. This guide uses “instance” when talking about servers and “space” when talking about the product; they name the same thing from two sides.</td></tr> |
| + <tr><td><strong>host</strong></td><td>An instance’s full domain name, for example <code>chat-a.example.com</code>. The host is the identity of an instance: it names its database file, its systemd unit, its socket, and its certificate.</td></tr> |
| + <tr><td><strong>box</strong></td><td>One Linux server in the fleet. A box runs the router, many instances, and the supporting services.</td></tr> |
| + <tr><td><strong>the router</strong></td><td>The one process on a box that listens on the public ports (80 and 443). It holds the TLS certificates and forwards each request to the right instance. See <a href="router.html">part 02</a>.</td></tr> |
| + <tr><td><strong>the registry</strong></td><td>A small SQLite database (<code>/data/registry.db</code>) on each box listing every host the box serves and how to reach it. The router reads its routing table from here.</td></tr> |
| + <tr><td><strong>unit</strong></td><td>A systemd service definition. Instances run under a template unit called <code>lchat@<host>.service</code>; the router runs under <code>lchat-router.service</code>. See <a href="processes.html">part 03</a>.</td></tr> |
| + <tr><td><strong>the bucket</strong></td><td>The Amazon S3 bucket used for hibernation: parked space databases, per-space lock objects, and mesh heartbeats all live here. See <a href="hibernation.html">part 06</a>.</td></tr> |
| + <tr><td><strong>the lease</strong></td><td>The per-space lock object in the bucket. Whichever process holds a space’s lease is the only process allowed to write that space’s data. The lease is the ultimate tie-breaker in the whole design.</td></tr> |
| +</table> |
| +</div> |
| + |
| +<h2 id="one-binary">One program, many roles</h2> |
| +<p>Almost everything in this system is a single, statically compiled Go program called <code>lchat</code>. There is no interpreter to install, no runtime to patch, and no package manager involved: deploying software to a box means copying one file. The same binary behaves differently depending on the subcommand it is started with:</p> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Invocation</th><th>Role</th><th>Explained in</th></tr> |
| + <tr><td><code>lchat router</code></td><td>The front door: TLS, routing, and (when configured) the hibernation activator and the mesh.</td><td><a href="router.html">Part 02</a>, <a href="hibernation.html">06</a>, <a href="mesh.html">07</a></td></tr> |
| + <tr><td><code>lchat --socket … --db …</code></td><td>One messenger instance (this is what each <code>lchat@<host></code> unit runs).</td><td><a href="processes.html">Part 03</a></td></tr> |
| + <tr><td><code>lchat add-instance</code>, <code>deploy</code>, <code>instances</code>, <code>hibernate</code>, …</td><td>The operator’s command-line tools, run over <code>ssh</code> on the box.</td><td><a href="reference.html#commands">Part 09</a></td></tr> |
| + <tr><td><code>lchat controld</code></td><td>The root-only local control daemon that creates and deletes instances on request.</td><td><a href="spaces.html">Part 05</a></td></tr> |
| + <tr><td><code>lchat cloud</code></td><td>The public, unprivileged self-serve provisioner (lets signed-in users create their own spaces).</td><td><a href="spaces.html">Part 05</a></td></tr> |
| + <tr><td><code>lchat update run</code></td><td>One pass of the auto-updater (normally fired hourly by a systemd timer).</td><td><a href="deploys.html">Part 04</a></td></tr> |
| + <tr><td><code>lchat bot run</code></td><td>A bot sidecar: an ordinary encrypted chat member that happens to run on a machine.</td><td><a href="processes.html#bot-sidecars">Part 03</a></td></tr> |
| +</table> |
| +</div> |
| +<p>Two sibling services are separate small binaries from the same repository, deployed next to the messenger: the <strong>home server</strong> (<code>home</code>, the anonymous passkey-gated backup store) and the <strong>push relay</strong> (<code>push</code>, the central Apple push-notification signer). To the router they are just two more names in the routing table; they are covered in <a href="processes.html#sibling-services">part 03</a>.</p> |
| + |
| +<h2 id="a-request">Following one request through the system</h2> |
| +<p>To make the map concrete, here is what happens when a member opens <code>https://chat-a.example.com</code> in a browser:</p> |
| +<ol> |
| + <li>DNS resolves <code>chat-a.example.com</code> to a box’s public address. (A wildcard DNS record usually covers every subdomain at once, so new spaces need no DNS work.)</li> |
| + <li>The <strong>router</strong> on that box accepts the connection on port 443 and presents the TLS certificate for that hostname — either a per-host certificate it obtained automatically from Let’s Encrypt, or a wildcard certificate.</li> |
| + <li>The router looks the hostname up in the <strong>registry</strong> and finds the instance’s unix socket, <code>/run/lchat/chat-a.example.com.sock</code>.</li> |
| + <li>If the instance is awake, the router forwards the request over that socket and the instance answers. If the instance is <strong>hibernating</strong>, the router’s built-in activator first claims the space’s lease in S3, restores the database if needed, starts the instance process as a child, and then forwards the request (this adds a few seconds, once).</li> |
| + <li>If this box is part of a <strong>mesh</strong> and the hashing rule says a different box owns this space, the router transparently forwards the whole request to that box instead.</li> |
| +</ol> |
| +<p>Every step of that journey has its own page in this guide, and each page walks the same journey from its own component’s point of view.</p> |
| + |
| +<h2 id="what-is-real">What is running today, and what is aspirational</h2> |
| +<p>The planet-scale design (<code>docs/planet-scale.md</code>) was built incrementally, and parts of it are still design rather than code. Throughout this guide, three badges mark the difference, and you should trust them: <span class="status built">running today</span> means the behaviour is in production code you can operate now; <span class="status partial">partially built</span> means a seam or a first half exists but the full behaviour does not; <span class="status planned">designed, not built</span> means the repository contains a written design and <em>no</em> operable code — nothing for an operator to run, configure, or monitor.</p> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Capability</th><th>Status</th><th>What exists</th></tr> |
| + <tr><td>The router, registry, per-instance units, versioned deploys, release channels, auto-updates</td><td><span class="status built">running today</span></td><td>The whole of <a href="router.html">parts 02</a>–<a href="deploys.html">04</a>.</td></tr> |
| + <tr><td>Self-serve space creation (control daemon + cloud provisioner) and the reaper</td><td><span class="status built">running today</span></td><td>The whole of <a href="spaces.html">part 05</a>.</td></tr> |
| + <tr><td>Instance hibernation (activator, lease, replication to S3, scheduled wakes, sidecar co-spawn)</td><td><span class="status built">running today</span></td><td>The whole of <a href="hibernation.html">part 06</a>. Opt-in per host.</td></tr> |
| + <tr><td>The mesh (multiple boxes, S3 heartbeats, hash placement, proxy to owner)</td><td><span class="status partial">partially built</span></td><td><a href="mesh.html">Part 07</a>. The placement code is complete and tested, but it is <em>dormant in production</em> (no box sets <code>LCHAT_MESH_SELF</code> yet), and one prerequisite — copying routing entries between boxes — was withdrawn for rework and is not on the main branch. See <a href="mesh.html#mesh-limits">the honest limits section</a>.</td></tr> |
| + <tr><td>Fleet backup (continuous replication of every database to a backup bucket)</td><td><span class="status built">running today</span></td><td><a href="operations.html#backups">Part 08</a>.</td></tr> |
| + <tr><td>Fully deferred space creation (a create becomes only a registry row, no files touched)</td><td><span class="status partial">partially built</span></td><td>Creation is split into register + materialize steps, and hibernating creates skip the systemd half — but they still write the env file, stage the binary, and wake once to mint the setup invite. See <a href="spaces.html#hibernating-creates">part 05</a>.</td></tr> |
| + <tr><td>Attachments in an object store (instead of on the box’s disk)</td><td><span class="status partial">partially built</span></td><td>The storage interface (<code>blobstore.go</code>) exists; only the on-disk implementation is real. The object-store implementation is roadmap.</td></tr> |
| + <tr><td>Certificate cache in S3 (needed the day two boxes share certificate issuance)</td><td><span class="status partial">partially built</span></td><td><code>LCHAT_CERT_CACHE=s3</code> is reserved and deliberately <em>refuses to start</em>; only the on-disk cache works. See <a href="mesh.html#mesh-limits">part 07</a>.</td></tr> |
| + <tr><td>Billing rails (Stripe pointers and entitlement flags in the cloud ledger)</td><td><span class="status partial">partially built</span></td><td>The schema and code exist and are dormant until the Stripe environment variables are set.</td></tr> |
| + <tr><td>Geographic placement (<code>home_region</code>) and the co-op provider seam (<code>provider</code>)</td><td><span class="status partial">partially built</span></td><td>Both are columns in the registry, written but never read. No behaviour keys on them yet.</td></tr> |
| + <tr><td>Multi-region operation; relocating a space between regions while it sleeps</td><td><span class="status planned">designed, not built</span></td><td>Design only, in <code>docs/planet-scale.md</code>.</td></tr> |
| + <tr><td>An autoscaling cloud front plane (load balancer, autoscaling group, golden machine image)</td><td><span class="status planned">designed, not built</span></td><td>Design only. Production is a small number of hand-provisioned boxes.</td></tr> |
| + <tr><td>Sealed bucket (encrypted object names and metadata in the home store)</td><td><span class="status planned">designed, not built</span></td><td>Design only; object keys are plain user ids today, and the bucket stays private.</td></tr> |
| + <tr><td>Proper metrics (wake latency, lease conflicts, replication lag as gauges)</td><td><span class="status planned">designed, not built</span></td><td>These exist only as log lines today. See <a href="operations.html#observability">part 08</a> for what you <em>can</em> observe.</td></tr> |
| +</table> |
| +</div> |
| +<div class="warn"> |
| + <span class="label">Please note</span> |
| + <p>When a page in this guide describes aspirational behaviour, it is framed in a dashed “designed, not built” panel like the ones you will see later, so that nothing aspirational can be mistaken for something you could run today. If you find yourself looking for a command this guide has not given you, check the status badges first.</p> |
| +</div> |
| + |
| +<h2 id="reading-order">How to read this guide</h2> |
| +<p>The parts are ordered so that each one only depends on the ones before it. If you are new to the system, read parts 02 and 03 before anything else — they establish the router and the process model that everything else builds on. If you are here to do a specific task, <a href="operations.html">part 08</a> is a task-oriented runbook with step-by-step procedures, each of which links back to the explanatory pages.</p> |
| +<ol> |
| + <li><a href="index.html">Overview and orientation</a> — this page.</li> |
| + <li><a href="router.html">The router</a> — the front door of every box: TLS certificates, the routing table, the registry.</li> |
| + <li><a href="processes.html">Everything that runs on a box</a> — each process, its systemd unit, how it is started, supervised, configured, and inspected; the filesystem layout.</li> |
| + <li><a href="deploys.html">Installing and updating the software</a> — the version store, staged deploys, health checks, rollback, release channels, the hourly auto-updater, and pinning.</li> |
| + <li><a href="spaces.html">Creating and removing chat spaces</a> — <code>add-instance</code> and <code>remove</code> by hand; the control daemon, the cloud provisioner, quotas, and the reaper.</li> |
| + <li><a href="hibernation.html">Hibernation</a> — how an idle space’s process exits and its data parks in object storage, and how a request (or a schedule) wakes it again.</li> |
| + <li><a href="mesh.html">The mesh</a> — how several boxes share one fleet: heartbeats, placement by hashing, proxying, and draining a box.</li> |
| + <li><a href="operations.html">Day-to-day operations</a> — the runbook: checking what is running, reading logs, health checks, backups and restore, migrating an instance between boxes.</li> |
| + <li><a href="reference.html">Complete reference</a> — every command with every flag, every environment variable, every file path, every systemd unit, and the glossary.</li> |
| +</ol> |
| + |
| +<h2 id="conventions">Conventions used throughout</h2> |
| +<ul> |
| + <li><strong>Command blocks say where to run them.</strong> Every terminal example carries a label such as “Run on the box, as root” or “Run on your own machine”. Lines beginning with <code>#</code> inside a block are explanatory comments, not commands.</li> |
| + <li><strong>Placeholders are angle-bracketed.</strong> <code><host></code> always means an instance’s full domain name, such as <code>chat-a.example.com</code>. <code><ver></code> means a build version identifier (a short git commit hash). Replace them, brackets and all.</li> |
| + <li><strong>Example names are consistent.</strong> The guide uses <code>chat-a.example.com</code> as its running example instance and <code>box-1</code> / <code>box-2</code> as example servers, so you can follow the same imaginary fleet across pages.</li> |
| + <li><strong>“In plain words” panels restate the essential idea</strong> of a section without any terminology, on purpose, so you can check your understanding as you go.</li> |
| + <li><strong>“Why it is this way” panels explain a design decision</strong> where knowing the reason makes the behaviour easier to remember.</li> |
| +</ul> |
| + |
| +</div> |
| +</main> |
| +</div> |
| +</body> |
| +</html> |
diff --git a/docs/planet-scale-guide/mesh.html b/docs/planet-scale-guide/mesh.html| new file mode 100644 |
| index 0000000..8ecf8b9 |
| --- /dev/null |
| +++ b/docs/planet-scale-guide/mesh.html |
| @@ -0,0 +1,122 @@ |
| +<!doctype html> |
| +<html lang="en"> |
| +<head> |
| +<meta charset="utf-8" /> |
| +<meta name="viewport" content="width=device-width, initial-scale=1" /> |
| +<title>The mesh — Running the fleet</title> |
| +<link rel="stylesheet" href="guide.css" /> |
| +<script src="guide.js" defer></script> |
| +</head> |
| +<body data-page="mesh.html"> |
| +<div class="frame"> |
| +<nav class="side"></nav> |
| +<main class="content"> |
| +<div class="wrap"> |
| + |
| +<header class="top"> |
| + <p class="eyebrow"><span class="robot">🤖</span> Running the fleet · Part 07 of 09</p> |
| + <h1>The mesh: running more than one box</h1> |
| + <p class="thesis">The mesh lets several boxes serve one fleet of hibernating spaces <strong>with no coordinator, no shared database, and no new network protocol between them</strong>: boxes announce themselves with heartbeat objects in the shared S3 bucket, a hashing rule decides which box should own which space, and the per-space lease guarantees correctness even when boxes disagree.</p> |
| +</header> |
| + |
| +<div class="sysmap" data-highlight="peer bucket"></div> |
| +<p class="sysmap-caption">You are here: the peer boxes, and the bucket they coordinate through. Note that peers never need to discover each other directly — everything flows through the bucket they already share for hibernation.</p> |
| + |
| +<div class="warn"> |
| + <span class="label">Read this first: the mesh is built, but dormant</span> |
| + <p>Be clear about what you are reading. The mesh code on this page is complete, merged, and covered by live kill-a-box tests — but as of this writing <strong>no production box enables it</strong> (every box runs with <code>LCHAT_MESH_SELF</code> unset, which means zero mesh code paths execute), and one prerequisite for convenient multi-box operation — automatically copying routing entries between boxes — was merged, then withdrawn for rework, and is <em>not</em> on the main branch. The <a href="#mesh-limits">honest limits</a> section at the end spells out exactly what that means for an operator today. Everything else on this page describes code as it exists and behaves when enabled.</p> |
| +</div> |
| + |
| +<h2 id="mesh-idea">The idea: ownership as a rule, correctness as a lock</h2> |
| +<p>Hibernation (<a href="hibernation.html">part 06</a>) already moved every space’s durable state into the bucket, and made the per-space <a href="hibernation.html#the-lease">lease</a> the single arbiter of who may write it. That does most of the multi-box work by itself: any box <em>could</em> wake any space, and the lease already prevents two from doing so at once. What remains is efficiency and manners — spaces should not ping-pong between boxes, and a request that lands on the “wrong” box should still be answered. The mesh adds exactly three small mechanisms on top of the bucket:</p> |
| +<ol> |
| + <li><strong>Membership:</strong> each box periodically writes a tiny heartbeat object; listing them yields the set of live boxes.</li> |
| + <li><strong>Placement:</strong> a deterministic hashing rule (“rendezvous hashing”) maps each hostname to one preferred owner among the live boxes.</li> |
| + <li><strong>Forwarding:</strong> a box that receives a request for a space it does not own transparently proxies the request to the owner.</li> |
| +</ol> |
| +<div class="plain"> |
| + <span class="label">In plain words</span> |
| + <p>The boxes never talk to each other to decide who does what. Each box signs a guest book in the shared storage room every fifteen seconds, everyone applies the same arithmetic to the guest book to work out who should host which space, and the storage room’s key hooks (the leases) make it impossible for two boxes to act on a disagreement. A box that stops signing the book is presumed gone within forty-five seconds, and its spaces re-home themselves automatically.</p> |
| +</div> |
| + |
| +<h2 id="mesh-config">Enabling the mesh on a box</h2> |
| +<p>Two environment variables in <code>/data/router.env</code>, on top of an already-configured hibernation bucket:</p> |
| +<figure class="cmd"> |
| +<figcaption>Contents of /data/router.env (then: systemctl restart lchat-router)</figcaption> |
| +<pre><code>LCHAT_HIBERNATE_BUCKET=my-hibernation-bucket # prerequisite: the mesh rides on the hibernation bucket |
| +LCHAT_MESH_SELF=https://box-1.example.net # this box’s identity: a base URL its PEERS can reach it at |
| +LCHAT_MESH_DOMAINS=11ch.example.com # comma-separated; ONLY hosts under these domains participate</code></pre> |
| +</figure> |
| +<dl> |
| + <dt><code>LCHAT_MESH_SELF</code> — the master switch and the box’s identity</dt> |
| + <dd>A URL with a scheme (<code>https</code> or <code>http</code>) and a host, no path. Its <em>presence</em> turns the mesh on; absent means a single box and zero mesh code. It must be reachable by the <em>other boxes</em> (it is what they proxy to), which may be a private network address (<code>http://10.0.0.5</code>) or a public one.</dd> |
| + <dt><code>LCHAT_MESH_DOMAINS</code> — the scope, and it is required</dt> |
| + <dd>The router refuses to start with <code>LCHAT_MESH_SELF</code> set but no domains. Only hostnames equal to or directly under a listed domain participate in mesh placement; every other host is always served locally, exactly as without a mesh. Two reasons for the scoping, both practical: a routing entry for a host only exists on the boxes that have it in their registry (see <a href="#mesh-limits">the limits</a>), so placement must be confined to hosts every member can serve; and every mesh member must be able to obtain a TLS certificate for any host that might land on it, which per-host issuance cannot promise (it ties a certificate to whichever box the public DNS points at) but a shared <a href="router.html#dns01">wildcard certificate</a> can. In practice: <strong>mesh domains should be wildcard-certificate domains.</strong></dd> |
| +</dl> |
| + |
| +<h2 id="membership">Membership: heartbeats in the bucket</h2> |
| +<p>Every fifteen seconds, each mesh box writes one object under <code><prefix>/mesh/members/</code> in the hibernation bucket and lists the others. The scheme has three properties worth appreciating:</p> |
| +<ul> |
| + <li><strong>The object’s key <em>is</em> the record</strong> — it is an encoding of the box’s own <code>LCHAT_MESH_SELF</code> URL — so a single cheap LIST operation yields the entire membership, with no per-member reads.</li> |
| + <li><strong>Liveness is judged by the bucket’s clock</strong>, not the boxes’ clocks: freshness is the object’s last-modified time as recorded by S3 itself, so a box with a wrong clock cannot poison anyone’s view. A heartbeat older than forty-five seconds (three missed beats) marks its box dead.</li> |
| + <li><strong>Failures are soft.</strong> A box that fails to write its beat is simply taken over by its peers (the lease arbitrates any overlap); a box that fails to <em>read</em> keeps its last known view and at worst misroutes — which, again, the lease absorbs. Neither failure is fatal, and both are logged.</li> |
| +</ul> |
| +<figure class="cmd"> |
| +<figcaption>Run anywhere with bucket credentials — who is in the mesh right now?</figcaption> |
| +<pre><code>aws s3 ls s3://my-hibernation-bucket/prod/mesh/members/ |
| +<span class="out">2026-07-13 14:02:11 38 aHR0cHM6Ly9ib3gtMS5leGFtcGxlLm5ldA |
| +2026-07-13 14:02:09 38 aHR0cHM6Ly9ib3gtMi5leGFtcGxlLm5ldA</span> |
| +# the timestamps are the liveness signal; the key decodes to the peer URL: |
| +echo aHR0cHM6Ly9ib3gtMS5leGFtcGxlLm5ldA | base64 -d |
| +<span class="out">https://box-1.example.net</span></code></pre> |
| +</figure> |
| + |
| +<h2 id="placement">Placement: rendezvous hashing, and why the lease still decides</h2> |
| +<p>For each in-scope hostname, every box independently computes a score for every live member (a hash of the pair “this box, this host”) and the highest score wins ownership. This is <strong>rendezvous hashing</strong>, and its virtue is minimal disruption: because each (box, host) pair’s score never changes, adding or removing one box only reassigns the hosts whose winning box appeared or vanished — roughly <em>1/N</em> of them — while everything else stays put. There is no table to distribute, no leader to elect: ownership is arithmetic on the guest book.</p> |
| +<p>But membership views can momentarily disagree (one box has seen a fresh heartbeat the other has not). The design’s answer is that <strong>hashing is only a hint, and the lease is the truth</strong>: a box that wrongly believes it owns a space simply loses the lease race at wake time, loudly, and the requester retries onto the right answer. Transient disagreement structurally cannot corrupt anything — it can only cost a retry.</p> |
| +<p>Two refinements keep behaviour calm: a space that is <em>currently awake</em> on some box stays there until its natural sleep, even if the hash now prefers another box — its state is already in the bucket, so “migration” is nothing more than waking on the new owner next time. And <a href="hibernation.html#due-time-wakes">scheduled wakes</a> fire only on the box that owns the host, so a due reminder wakes one box, not all of them.</p> |
| + |
| +<h2 id="one-hop">Forwarding: wrong-box requests proxy to the owner, once</h2> |
| +<p>When a request arrives at a box that is not the computed owner, the box reverse-proxies the entire request to the owner’s <code>LCHAT_MESH_SELF</code> URL — preserving the original hostname, verifying the owner’s TLS certificate against <em>the instance’s hostname</em> (this is why mesh domains want the shared wildcard certificate), and pinning the underlying connection to the peer’s address. There is no certificate-verification bypass anywhere. A forwarded request is marked with an internal header (<code>X-Lchat-Mesh-Forwarded</code>, stripped from all outside traffic), and a marked request is <strong>never forwarded again</strong>: if two boxes disagree about ownership, the second box serves the request locally — a wrong-box wake the lease then resolves — instead of the pair bouncing the request forever.</p> |
| + |
| +<h2 id="box-lifecycle">Adding, draining, and losing a box</h2> |
| +<h3 id="add-box">Adding a box to the mesh</h3> |
| +<ol class="procedure"> |
| + <li><p class="step-title">Provision the box normally.</p> |
| + <p>Install the binary and run <code>lchat bootstrap</code>, exactly as in <a href="deploys.html#bootstrap">part 04</a>. Give it the <em>same</em> hibernation bucket, prefix, and credentials as its peers in <code>/data/router.env</code> — the shared bucket is the entire coordination fabric.</p></li> |
| + <li><p class="step-title">Give it the shared wildcard certificate capability.</p> |
| + <p>Configure <code>LCHAT_WILDCARD_DOMAINS</code> (and the DNS API token) for the mesh domains, so this box can present valid certificates for any space that lands on it.</p></li> |
| + <li><p class="step-title">Ensure it has registry rows for the mesh-scoped hosts.</p> |
| + <p>This is the manual step that automation was supposed to remove (see <a href="#mesh-limits">the limits</a>): a box can only serve hosts present in its own registry. Today that means registering the hibernating hosts on the new box yourself — for each host, <code>lchat add-instance <host> --hibernate --admin-token <the host’s existing token></code>, reusing the token from the host’s environment file on the original box so nothing is re-minted.</p></li> |
| + <li><p class="step-title">Set its mesh identity and restart the router.</p> |
| + <figure class="cmd"><figcaption>Run on the new box — /data/router.env additions, then restart</figcaption> |
| + <pre><code>LCHAT_MESH_SELF=https://box-2.example.net |
| +LCHAT_MESH_DOMAINS=11ch.example.com |
| +# then: systemctl restart lchat-router</code></pre></figure> |
| + <p>Within one heartbeat interval the peers see the new member, and roughly <em>1/N</em> of the in-scope spaces re-home to it at their next sleep/wake boundary. Nothing needs announcing; the guest book does it.</p></li> |
| +</ol> |
| +<h3 id="drain-box">Draining a box (deploys, maintenance, scale-down)</h3> |
| +<p>Stopping the router <em>is</em> the drain: its shutdown sequence <strong>deletes this box’s heartbeat first</strong> — so peers take ownership immediately — and then checkpoints every awake space to the bucket and releases its leases as usual. Deploying a new router build, retiring a box, and reacting to failure are deliberately the same code path.</p> |
| +<figure class="cmd"> |
| +<figcaption>Run on the box being drained</figcaption> |
| +<pre><code>systemctl stop lchat-router # heartbeat deleted, spaces checkpointed, leases released</code></pre> |
| +</figure> |
| +<p>The recovery numbers, verified by live tests in both directions: after a <em>graceful</em> drain, the survivor is serving the drained box’s spaces about <strong>two seconds</strong> later; after a <em>hard kill</em> (no drain, heartbeat left to rot), within about <strong>thirty seconds</strong> — the lease lifetime — with data intact in both cases, because every write had been streaming to the bucket all along.</p> |
| + |
| +<h2 id="mesh-limits">The honest limits: what stands between here and a true multi-box fleet</h2> |
| +<p>Three gaps, in decreasing order of consequence. None of them is speculative — each corresponds to a concrete piece of code that exists, is reserved, or was withdrawn:</p> |
| +<ul> |
| + <li><span class="status planned">not on main</span> <strong>Registries do not replicate.</strong> Each box’s registry is its own SQLite file, and a request for a host the box has no row for is answered 404 <em>before</em> any mesh logic runs. A change that fanned new routes out to every live peer (plus a <code>mesh-seed</code> command for a joining box’s initial catch-up, authenticated by a new shared secret) was merged and then <strong>reverted for rework</strong>; it currently lives on side branches only. Until it, or a successor, lands: keeping registries aligned across mesh boxes is <em>manual</em> (step 3 of “adding a box” above), and route changes after creation — renames, removals, hibernation flips — must be repeated on every box by hand.</li> |
| + <li><span class="status partial">partially built</span> <strong>The certificate cache is per box.</strong> The wildcard certificate makes mesh domains workable (each box independently obtains its own copy), but per-host certificates for domains outside the wildcard cannot follow a space between boxes; the S3-backed shared cache that would fix this is reserved (<code>LCHAT_CERT_CACHE=s3</code>) and deliberately refuses to start. See <a href="router.html#tls">part 02</a>.</li> |
| + <li><span class="status planned">not on main</span> <strong>Fleet-wide status aggregation.</strong> The public fleet dashboard reports the local box only; the change that lets each mesh member publish its counts as a sibling object next to its heartbeat, for the status publisher to merge, sits on an unmerged branch (and its bucket-permission change has not been applied). See <a href="operations.html#observability">part 08</a>.</li> |
| +</ul> |
| +<div class="roadmap"> |
| + <span class="label">Designed, not built: what the mesh grows into</span> |
| + <p>The written design (<code>docs/planet-scale.md</code>) carries the mesh further than any current code: per-space <em>geographic</em> placement using the registry’s <code>home_region</code> field (relocating a space between regions while it sleeps, by copying its bucket prefix and flipping the field); an autoscaling group of boxes behind a network load balancer, sized on warm-set memory; and a co-operative “provider” seam in which spaces could be delegated to operators other than the product’s own. Each of these has a schema field or an interface already cut for it, and none has operable behaviour today.</p> |
| +</div> |
| + |
| +</div> |
| +</main> |
| +</div> |
| +</body> |
| +</html> |
diff --git a/docs/planet-scale-guide/operations.html b/docs/planet-scale-guide/operations.html| new file mode 100644 |
| index 0000000..2990de0 |
| --- /dev/null |
| +++ b/docs/planet-scale-guide/operations.html |
| @@ -0,0 +1,153 @@ |
| +<!doctype html> |
| +<html lang="en"> |
| +<head> |
| +<meta charset="utf-8" /> |
| +<meta name="viewport" content="width=device-width, initial-scale=1" /> |
| +<title>Day-to-day operations — Running the fleet</title> |
| +<link rel="stylesheet" href="guide.css" /> |
| +<script src="guide.js" defer></script> |
| +</head> |
| +<body data-page="operations.html"> |
| +<div class="frame"> |
| +<nav class="side"></nav> |
| +<main class="content"> |
| +<div class="wrap"> |
| + |
| +<header class="top"> |
| + <p class="eyebrow"><span class="robot">🤖</span> Running the fleet · Part 08 of 09</p> |
| + <h1>Day-to-day operations and checks</h1> |
| + <p class="thesis">This page is the runbook: <strong>the checks worth doing routinely, the log lines worth knowing, backups and how to actually restore one, moving an instance between boxes, and a table of common problems with their diagnoses.</strong> Every procedure links back to the page that explains why it works.</p> |
| +</header> |
| + |
| +<div class="sysmap" data-highlight="router instances services control updater backup"></div> |
| +<p class="sysmap-caption">You are here: everywhere. Operations cut across every component, which is why this page leans on the earlier ones for depth.</p> |
| + |
| +<h2 id="routine">The routine health check, in ninety seconds</h2> |
| +<figure class="cmd"> |
| +<figcaption>Run on each box — the five questions and the commands that answer them</figcaption> |
| +<pre><code># 1. Are all supervised services up, and has anything been crash-looping? |
| +systemctl list-units 'lchat*' --failed |
| +systemctl status lchat-router lchat-control lchat-litestream --no-pager | grep -E 'Active|since' |
| + |
| +# 2. Does the registry agree — is every intended host ready? |
| +lchat instances # look for STATUS=failed rows and read their last error |
| + |
| +# 3. Is the fleet current on its release channels, and whose canaries are pinned? |
| +lchat update status # AUTO=pinned rows are someone’s live experiment — leave them be |
| + |
| +# 4. Is disk comfortable? (disk, not memory, is the binding resource on these boxes) |
| +df -h /data |
| + |
| +# 5. Does one real space answer end to end, through DNS, TLS, router, and instance? |
| +curl -s https://chat-a.example.com/api/version</code></pre> |
| +</figure> |
| +<p>For the fleet-wide view without logging into boxes, the public status dashboard (<a href="#observability">below</a>) publishes instance counts, awake/asleep split, capacity bands, version spread, and replication health every five minutes.</p> |
| + |
| +<h2 id="reading-logs">Reading the logs: the lines worth knowing</h2> |
| +<p>Everything logs to the systemd journal. The single most informative journal is the router’s, because the activator and mesh live inside it (<a href="hibernation.html#children">part 06</a>):</p> |
| +<figure class="cmd"> |
| +<figcaption>Run on the box</figcaption> |
| +<pre><code>journalctl -u lchat-router -f # follow everything: routing, certificates, wakes, sleeps |
| +journalctl -u lchat@chat-a.example.com -n 200 # one systemd-managed instance |
| +journalctl -u lchat-update.service -n 100 # what the last hourly update pass did |
| +journalctl -u lchat-litestream -n 50 # backup replication health</code></pre> |
| +</figure> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Log line (pattern)</th><th>Meaning</th><th>Action</th></tr> |
| + <tr><td><code><host> waking</code> / <code>awake</code> / <code>hibernated</code></td><td>The activator’s normal lifecycle.</td><td>None — this is the system working.</td></tr> |
| + <tr><td><code>LEASE LOST … standing down</code></td><td>This box could not renew a space’s <a href="hibernation.html#the-lease">lease</a>; it stopped serving the space, which will re-wake on the next request (possibly on another box, in a mesh).</td><td>Occasional: fine. Frequent: investigate bucket reachability and credentials.</td></tr> |
| + <tr><td><code>exited unexpectedly — recycling</code></td><td>A hibernating instance’s child process died; the activator cleaned up and the next request re-wakes it.</td><td>Check the surrounding lines for the child’s own crash output; repeated recycling of one host means its build or data needs attention.</td></tr> |
| + <tr><td><code>another deploy/rollback/update is running</code></td><td>The <a href="deploys.html#manual-deploy">deploy lock</a> refused a second concurrent operation.</td><td>Wait for the running one; if none is visibly running, look for a stuck updater pass.</td></tr> |
| + <tr><td><code>poisoned</code> (in an update pass)</td><td>A channel build failed its health check on one instance, so the pass declined to try it elsewhere.</td><td>Investigate that build before the next pass; pin anything sensitive meanwhile.</td></tr> |
| + <tr><td>Certificate errors mentioning <code>acme</code></td><td>Let’s Encrypt issuance failed — usually DNS not pointing at the box, port 80 blocked, or a rate limit.</td><td>Verify DNS and port 80; for rate limits, prefer the <a href="router.html#dns01">wildcard certificate</a>.</td></tr> |
| +</table> |
| +</div> |
| + |
| +<h2 id="backups">Backups: continuous replication, and the restore you should rehearse</h2> |
| +<p>Every database on the box — instance databases, the registry, the home, push, and cloud databases — is <strong>continuously replicated</strong> to a dedicated backup bucket by a stock Litestream process (<code>lchat-litestream.service</code>, configuration <code>/etc/litestream.yml</code>). This is replication as a property of normal operation, not nightly snapshots: recovery-point loss is measured in seconds, and point-in-time restore is available. Three properties of the arrangement matter operationally:</p> |
| +<ul> |
| + <li><strong>The configuration is generated, not written.</strong> <code>deploy/fleet-backup.sh</code> regenerates <code>/etc/litestream.yml</code> from what is actually in <code>/data</code>; run it after adding instances so new databases get picked up. It deliberately <strong>skips hibernating hosts</strong> — their replication belongs to the activator, and two replicators on one database corrupt each other (<a href="hibernation.html#hibernation-gotchas">part 06</a>).</li> |
| + <li><strong>The bucket is the ransomware posture.</strong> It keeps old versions of every object for fourteen days, and the box’s credentials can write objects but can never change versioning, lifecycle, or the bucket itself — so even an attacker with the box’s own keys cannot destroy the history they overwrite.</li> |
| + <li><strong>Blobs and the certificate cache ride separately.</strong> Attachment files (<code>/data/<host>-blobs/</code>) are immutable and content-addressed; the certificate cache (<code>/data/acme/</code>) is key material. Both belong in whatever file-level backup wraps the box.</li> |
| +</ul> |
| +<h3 id="restore">Restoring a database (rehearse this before you need it)</h3> |
| +<ol class="procedure"> |
| + <li><p class="step-title">Always restore to a scratch path — never over a live file.</p> |
| + <figure class="cmd"><figcaption>Run on the box</figcaption> |
| + <pre><code>litestream restore -config /etc/litestream.yml \ |
| + -o /tmp/restore-check/chat-a.example.com.db /data/chat-a.example.com.db |
| +sqlite3 /tmp/restore-check/chat-a.example.com.db "PRAGMA integrity_check" |
| +<span class="out">ok</span></code></pre></figure></li> |
| + <li><p class="step-title">Only after verifying, swap it into place with the instance stopped.</p> |
| + <figure class="cmd"><figcaption>Run on the box, as root</figcaption> |
| + <pre><code>systemctl stop lchat@chat-a.example.com |
| +mv /data/chat-a.example.com.db /data/chat-a.example.com.db.broken |
| +mv /tmp/restore-check/chat-a.example.com.db /data/chat-a.example.com.db |
| +chown lchat:lchat /data/chat-a.example.com.db |
| +systemctl start lchat@chat-a.example.com</code></pre></figure></li> |
| +</ol> |
| +<p>For a <em>hibernating</em> host the parked copy in the hibernation bucket is usually the freshest state; a cold wake restores it automatically. The backup bucket is your answer when the hibernation copy itself is what was damaged.</p> |
| + |
| +<h2 id="migration">Moving an instance between boxes</h2> |
| +<p>Migration is data movement plus a DNS change — nothing about members’ keys or history is affected, <em>provided the hostname never changes</em> (browser encryption keys and passkeys are bound to the origin). The procedure, condensed from the repository’s validated runbook (<code>docs/instance-migration.md</code>):</p> |
| +<ol class="procedure"> |
| + <li><p class="step-title">Recover the admin token and per-instance configuration from the source box.</p> |
| + <p>Both live in <code>/data/<host>.env</code>. The token must be <em>preserved</em> — importing it on the destination (rather than minting fresh) is what keeps every saved admin link working.</p></li> |
| + <li><p class="step-title">Snapshot the database cleanly, on the live system.</p> |
| + <figure class="cmd"><figcaption>Run on the source box</figcaption> |
| + <pre><code>sqlite3 /data/chat-a.example.com.db "VACUUM INTO '/tmp/chat-a-snapshot.db'" |
| +sqlite3 /tmp/chat-a-snapshot.db "PRAGMA integrity_check" |
| +<span class="out">ok</span></code></pre></figure> |
| + <p><code>VACUUM INTO</code> produces a single, transaction-consistent, compacted copy and is safe against a running instance — no stop required for the snapshot itself.</p></li> |
| + <li><p class="step-title">Copy three things, not one.</p> |
| + <p>The snapshot to <code>/data/<host>.db</code> on the destination; the blob directory (<code>rsync -a /data/<host>-blobs/ …</code> — immutable files, safe to copy live); and the web-push key file <code>/data/<host>.vapid</code>, without which every browser push subscription silently breaks.</p></li> |
| + <li><p class="step-title">Import on the destination, preserving the token.</p> |
| + <figure class="cmd"><figcaption>Run on the destination box, with the database already in place</figcaption> |
| + <pre><code>lchat add-instance chat-a.example.com \ |
| + --admin-token "<the existing token>" \ |
| + --push-relay-url https://push.example.com --push-relay-token "<existing relay token>"</code></pre></figure> |
| + <p><code>add-instance</code> adopts the pre-placed database, starts the unit against it, health-checks, and reloads the router (<a href="spaces.html#adding-manual">part 05</a>). Verify over the local socket <em>before</em> touching DNS.</p></li> |
| + <li><p class="step-title">Check the gap, then flip DNS.</p> |
| + <p>Compare message counts between source-now and the snapshot; if the space is busy, freeze writes and re-snapshot at cutover. Lower the DNS record’s TTL to about sixty seconds a while <em>before</em> the flip. The TLS certificate cannot be pre-issued (issuance validates over public DNS), so it issues on the first request after the flip — a few seconds, once.</p></li> |
| + <li><p class="step-title">Verify, and keep the source as the instant rollback.</p> |
| + <p>Leave the source instance running until you are satisfied: flipping DNS back is a complete rollback. Bot sidecars are their own step — their identity files (<code>/data/<host>-bot/</code>) must be copied, <em>never</em> re-provisioned with <code>add-bot</code> (that would mint duplicate bot members).</p></li> |
| +</ol> |
| + |
| +<h2 id="hardening">The hardening checklist (one-time per box)</h2> |
| +<ul> |
| + <li><strong>Firewall:</strong> allow inbound 22, 80, 443 only; default-deny the rest. Everything internal is unix sockets, so nothing else needs a port. (Add the allows <em>before</em> enabling the firewall, so you do not lock out your own SSH session.)</li> |
| + <li><strong>SSH:</strong> key-only (<code>PasswordAuthentication no</code>).</li> |
| + <li><strong>Automatic security updates</strong> for the operating system (<code>unattended-upgrades</code>) — the application updates itself (<a href="deploys.html#auto-updates">part 04</a>), but the kernel and OpenSSL do not.</li> |
| + <li><strong>Pin the box’s own hostnames to loopback</strong> in <code>/etc/hosts</code>, so on-box clients (bot sidecars, calls to the push relay) reach the local router instead of the public internet — and disable any cloud-image behaviour that regenerates that file on boot. The subtle failure this prevents is described in <a href="processes.html#bot-sidecars">part 03</a>.</li> |
| + <li><strong>The unit files already sandbox every service</strong> (<a href="processes.html#security-model">part 03</a>) — nothing to do, but worth knowing when something cannot write where you expected: <code>ProtectSystem=strict</code> means the whole filesystem is read-only to services except <code>/data</code> and <code>/run/lchat</code>.</li> |
| +</ul> |
| + |
| +<h2 id="observability">What you can observe, and what you cannot yet</h2> |
| +<p>A status publisher on the box regenerates a set of public dashboards every five minutes, including a fleet page whose data file (<code>fleet.json</code>) reports: total, systemd-managed, and hibernating instance counts; how many hibernators are awake versus asleep at this moment; the software version spread; up/down booleans for the sibling services; backup-replication health with its lag in seconds; and a coarse capacity band. The capacity band — <code>plenty</code>, <code>getting-full</code>, or <code>at-capacity</code> — is the worst of several internal signals (instance count against the cap, registered rows against the row ceiling, disk against the floor, memory), and deliberately <em>only the band word is published</em>: never counts of slots, gigabytes, percentages, hostnames, or addresses. The same discipline runs through everything public: aggregate numbers and band words, nothing that identifies a machine or a member.</p> |
| +<div class="roadmap"> |
| + <span class="label">Designed, not built: real metrics</span> |
| + <p>Wake latency, lease conflicts, and replication lag exist today as <em>log lines</em>, not as queryable metrics with alerting. The design owes an end-to-end-encryption-safe metrics discipline; until it lands, trend questions (“are wakes getting slower?”) are answered by reading journals, and the five-minute dashboard is the only aggregation. Budget your expectations accordingly.</p> |
| + <p>Likewise, in a mesh the dashboard currently reports <em>the local box only</em> — the multi-box aggregation exists on an unmerged branch (<a href="mesh.html#mesh-limits">part 07</a>).</p> |
| +</div> |
| + |
| +<h2 id="troubleshooting">Common problems and their diagnoses</h2> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Symptom</th><th>Likely cause</th><th>Diagnosis and fix</th></tr> |
| + <tr><td><code>lchat instances</code> shows a host as <code>failed</code></td><td>Its creation or last restart did not become healthy in time.</td><td>Read the row’s <code>last error</code> line, then the instance’s journal. The files are left in place on failure; fixing the cause and re-running the operation is safe.</td></tr> |
| + <tr><td>A hibernating host’s first wake fails with a permission error</td><td>Its environment file is root-owned (the activator reads it unprivileged).</td><td><code>chown lchat:lchat /data/<host>.env</code> — and note the flip commands do this themselves now; hand-copied files are the remaining way to hit it. See <a href="hibernation.html#hibernation-gotchas">part 06</a>.</td></tr> |
| + <tr><td>A burst of 503 responses on one host</td><td>Wake contention: a lease held elsewhere, or a wake exceeding its budget.</td><td>Check the router journal for lease messages. In a mesh, confirm both boxes see the same membership (<a href="mesh.html#membership">list the heartbeats</a>). Occasional 503s during wakes are by design; sustained ones are not.</td></tr> |
| + <tr><td>A bot stops answering after a migration or DNS change</td><td>The sidecar dials the public hostname, and the box resolves it to the wrong place.</td><td>Verify the <code>/etc/hosts</code> loopback pin on the box that runs the sidecar; restart the sidecar unit. See <a href="processes.html#bot-sidecars">part 03</a>.</td></tr> |
| + <tr><td><code>deploy</code> or <code>update</code> refuses: “another deploy… is running”</td><td>The deploy lock is held.</td><td>Usually a genuinely running hourly pass — check <code>systemctl status lchat-update.service</code>; a timer pass that lost the lock simply retries next hour.</td></tr> |
| + <tr><td>A new hostname will not get a certificate</td><td>DNS not pointing at the box, port 80 unreachable, or the host missing from the registry (the router refuses to request certificates for unknown hosts).</td><td><code>dig <host></code>, check the firewall, check <code>lchat instances</code>. See <a href="router.html#tls">part 02</a>.</td></tr> |
| + <tr><td>Web push notifications silently stopped for one instance</td><td>Its <code>/data/<host>.vapid</code> key file was lost in a move, so it re-minted keys and orphaned subscriptions.</td><td>Restore the original file from backup if possible; otherwise members’ browsers re-subscribe over time. Prevent it by always copying the file during <a href="#migration">migration</a>.</td></tr> |
| + <tr><td>Disk filling up</td><td>Attachments (blobs) — the one thing that grows with use.</td><td><code>du -sh /data/*-blobs</code> to find the heavy spaces; storage quotas per space are settable through the control daemon (<a href="spaces.html#controld">part 05</a>). Old builds are garbage-collected automatically and are rarely the cause.</td></tr> |
| +</table> |
| +</div> |
| + |
| +</div> |
| +</main> |
| +</div> |
| +</body> |
| +</html> |
diff --git a/docs/planet-scale-guide/processes.html b/docs/planet-scale-guide/processes.html| new file mode 100644 |
| index 0000000..fa355c1 |
| --- /dev/null |
| +++ b/docs/planet-scale-guide/processes.html |
| @@ -0,0 +1,193 @@ |
| +<!doctype html> |
| +<html lang="en"> |
| +<head> |
| +<meta charset="utf-8" /> |
| +<meta name="viewport" content="width=device-width, initial-scale=1" /> |
| +<title>Everything that runs on a box — Running the fleet</title> |
| +<link rel="stylesheet" href="guide.css" /> |
| +<script src="guide.js" defer></script> |
| +</head> |
| +<body data-page="processes.html"> |
| +<div class="frame"> |
| +<nav class="side"></nav> |
| +<main class="content"> |
| +<div class="wrap"> |
| + |
| +<header class="top"> |
| + <p class="eyebrow"><span class="robot">🤖</span> Running the fleet · Part 03 of 09</p> |
| + <h1>Everything that runs on a box</h1> |
| + <p class="thesis">This page is a complete census of the processes on a fleet box: <strong>what each one is, how it is started, how it is configured, how it is supervised, and how you check on it.</strong> It also maps the whole filesystem layout, because every process is defined by the files it reads and writes.</p> |
| +</header> |
| + |
| +<div class="sysmap" data-highlight="router instances services control updater"></div> |
| +<p class="sysmap-caption">You are here: everything inside the “one box” zone. Each highlighted node is one or more of the processes catalogued below.</p> |
| + |
| +<h2 id="systemd-primer">A one-minute systemd primer</h2> |
| +<p>Every long-running process on a box is supervised by <strong>systemd</strong>, the standard Linux service manager. Three ideas are enough to follow everything on this page:</p> |
| +<ul> |
| + <li>A <strong>unit</strong> is a small text file in <code>/etc/systemd/system/</code> describing one service: what command to run, as which user, with what restrictions, and what to do if it exits. All of this system’s unit files are written for you by <code>lchat bootstrap</code> (described in <a href="deploys.html#bootstrap">part 04</a>); their sources are version-controlled in the repository under <code>deploy/systemd/</code>.</li> |
| + <li>A <strong>template unit</strong> has an <code>@</code> in its name and is instantiated per name after the <code>@</code>. So <code>lchat@.service</code> is a template, and <code>lchat@chat-a.example.com.service</code> is the concrete service for one instance — systemd substitutes the part after <code>@</code> (written <code>%i</code> inside the unit file) wherever it appears. This one mechanism is how a single unit file serves any number of instances.</li> |
| + <li>The verbs you will use constantly: <code>systemctl status <unit></code> (is it running, since when, last log lines), <code>systemctl restart <unit></code>, <code>systemctl enable --now <unit></code> (start it and also start it at every boot), <code>systemctl disable --now <unit></code> (stop it and stop starting it at boot), and <code>journalctl -u <unit></code> (its full log).</li> |
| +</ul> |
| +<p>Nearly every unit here sets <code>Restart=always</code>, meaning systemd restarts the process within a second or two if it ever exits. Crash recovery is therefore automatic; your job is to notice <em>repeated</em> crashing (see <a href="operations.html#reading-logs">part 08</a>), not to restart things by hand.</p> |
| + |
| +<h2 id="census">The census: every process, at a glance</h2> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>systemd unit</th><th>What it is</th><th>Command it runs</th><th>Runs as</th></tr> |
| + <tr><td><code>lchat-router.service</code></td><td>The <a href="router.html">router</a>: TLS + routing; hosts the <a href="hibernation.html">activator</a> and <a href="mesh.html">mesh</a> when configured.</td><td><code>/opt/lchat/lchat router --registry /data/registry.db --acme-cache /data/acme</code></td><td><code>lchat</code></td></tr> |
| + <tr><td><code>lchat@<host>.service</code></td><td>One chat space instance (template unit; one per host).</td><td><code>/opt/lchat/live/<host> --socket /run/lchat/<host>.sock --db /data/<host>.db</code></td><td><code>lchat</code></td></tr> |
| + <tr><td><code>lchat-bot@<host>.service</code></td><td>The optional bot sidecar for one instance (template unit).</td><td><code>/opt/lchat/live/<host> bot run</code></td><td><code>lchat</code></td></tr> |
| + <tr><td><code>lchat-home.service</code></td><td>The home server: the anonymous, passkey-gated backup store (a separate binary from the same repository).</td><td><code>/opt/lchat/home --addr /run/lchat/home.sock --db /data/home.db</code></td><td><code>lchat</code></td></tr> |
| + <tr><td><code>lchat-push.service</code></td><td>The push relay: the central Apple push-notification signer (separate binary).</td><td><code>/opt/lchat/push --addr /run/lchat/push.sock --db /data/push.db</code></td><td><code>lchat</code></td></tr> |
| + <tr><td><code>lchat-control.service</code></td><td>The control daemon: the root-only local API that creates and deletes instances. See <a href="spaces.html#controld">part 05</a>.</td><td><code>/opt/lchat/lchat controld</code></td><td>root (deliberately)</td></tr> |
| + <tr><td><code>lchat-cloud.service</code></td><td>The cloud provisioner: the public, unprivileged self-serve space-creation service. See <a href="spaces.html#cloud">part 05</a>.</td><td><code>/opt/lchat/lchat cloud --socket /run/lchat/cloud.sock --db /data/cloud.db</code></td><td><code>lchat</code></td></tr> |
| + <tr><td><code>lchat-update.timer</code> + <code>.service</code></td><td>The hourly auto-updater (a timer that fires a one-shot pass). See <a href="deploys.html#auto-updates">part 04</a>.</td><td><code>/opt/lchat/lchat update run</code></td><td>root (deliberately)</td></tr> |
| + <tr><td><code>lchat-litestream.service</code></td><td>The fleet-backup replicator: streams every database on the box to the backup bucket, continuously. See <a href="operations.html#backups">part 08</a>.</td><td><code>litestream replicate</code> (a separate, stock Litestream binary with config <code>/etc/litestream.yml</code>)</td><td><code>lchat</code></td></tr> |
| + <tr><td><code>lchat-tunnel.service</code></td><td>An optional try-it tunnel service for self-hosters; not part of normal fleet operation and not covered further in this guide.</td><td><code>/opt/lchat/lchat tunnel serve</code></td><td><code>lchat</code></td></tr> |
| +</table> |
| +</div> |
| +<p>One process is deliberately <em>absent</em> from this census when hibernation is in use: a hibernating instance has <strong>no systemd unit running at all</strong>. Its process, when awake, is a direct child of the router, invisible to <code>systemctl</code> but visible in the process tree — <a href="hibernation.html#children">part 06</a> explains how to see and manage those.</p> |
| + |
| +<h2 id="see-everything">Seeing everything at once</h2> |
| +<figure class="cmd"> |
| +<figcaption>Run on the box — three views of what is running</figcaption> |
| +<pre><code># 1. Every lchat-related unit systemd knows about, with its state: |
| +systemctl list-units 'lchat*' |
| +<span class="out">UNIT LOAD ACTIVE SUB DESCRIPTION |
| +lchat-router.service loaded active running lchat router (TLS terminator + host router) |
| +lchat@chat-a.example.com.service loaded active running lchat instance chat-a.example.com |
| +lchat-bot@chat-a.example.com.service loaded active running lchat bot sidecar for chat-a.example.com |
| +lchat-control.service loaded active running lchat control daemon (local provisioning API) |
| +lchat-update.timer loaded active waiting hourly lchat release-channel update check</span> |
| + |
| +# 2. The fleet view from lchat itself — every registered host, whether systemd-managed |
| +# or hibernating, with status and version: |
| +lchat instances |
| +<span class="out">HOST KIND STATUS VERSION HIBERNATE SOCKET |
| +chat-a.example.com instance ready 1b4bd808 no /run/lchat/chat-a.example.com.sock |
| +chat-b.example.com instance ready 1b4bd808 yes /run/lchat/chat-b.example.com.sock |
| +home.example.com service ready - no /run/lchat/home.sock</span> |
| + |
| +# 3. Which unix sockets are actually bound right now (a hibernating host that is |
| +# asleep has a registry row but no bound socket): |
| +ss -xlH | grep /run/lchat</code></pre> |
| +</figure> |
| +<p>These three views answer different questions, and it is worth internalising the difference. <code>systemctl</code> shows <em>supervised processes</em>. <code>lchat instances</code> shows <em>the registry</em> — the intended state of the box, including hosts that are asleep on purpose. <code>ss</code> shows <em>reality at this instant</em> — which sockets have a live listener. When the three disagree, the disagreement is itself the diagnostic: a <code>ready</code> registry row with no bound socket and no unit is a sleeping hibernator (normal); a running unit with no bound socket is a process failing to start (investigate its journal).</p> |
| + |
| +<h2 id="filesystem">The filesystem layout: what lives where</h2> |
| +<p>Every process above is defined by files in three directories. This layout is worth memorising, because nearly every operational question (“where is this instance’s data?”, “where is its admin token?”, “which binary is it running?”) is answered by a path.</p> |
| +<h3 id="opt"><code>/opt/lchat/</code> — the software</h3> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Path</th><th>Contents</th></tr> |
| + <tr><td><code>/opt/lchat/lchat</code></td><td>The control binary: what the router, control daemon, updater, and your own command-line invocations run.</td></tr> |
| + <tr><td><code>/opt/lchat/versions/<ver>/lchat</code></td><td>The <strong>version store</strong>: every staged build, one directory per build identifier (a short git commit hash). Builds accumulate here and are garbage-collected when nothing references them. See <a href="deploys.html#version-store">part 04</a>.</td></tr> |
| + <tr><td><code>/opt/lchat/live/<host></code></td><td>One symbolic link per instance, pointing into the version store. <strong>This is the single most important path on the box</strong>: each instance’s unit runs <code>/opt/lchat/live/<host></code>, so <em>which build an instance runs is decided entirely by where its link points</em>. Deploying is repointing a link; rollback is repointing it back.</td></tr> |
| + <tr><td><code>/opt/lchat/incoming/</code></td><td>A staging area for uploaded and downloaded binaries before they are verified and moved into the version store.</td></tr> |
| + <tr><td><code>/opt/lchat/.deploy.lock</code></td><td>A lock file that makes deploys, rollbacks, and updater passes take turns instead of interleaving. If two are attempted at once, the second fails immediately with a clear message.</td></tr> |
| + <tr><td><code>/opt/lchat/manifest-stamp</code></td><td>The timestamp of the newest release manifest the box has ever accepted — the auto-updater refuses anything older, so the release feed cannot be replayed backwards. See <a href="deploys.html#signing">part 04</a>.</td></tr> |
| + <tr><td><code>/opt/lchat/home</code>, <code>/opt/lchat/push</code></td><td>The sibling-service binaries.</td></tr> |
| +</table> |
| +</div> |
| +<h3 id="data"><code>/data/</code> — the state</h3> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Path</th><th>Contents</th></tr> |
| + <tr><td><code>/data/<host>.db</code> (+ <code>-wal</code>, <code>-shm</code>)</td><td>One instance’s entire SQLite database. The <code>-wal</code> and <code>-shm</code> files are SQLite’s write-ahead log machinery and travel with the database. Contents are end-to-end ciphertext plus routing metadata.</td></tr> |
| + <tr><td><code>/data/<host>.env</code></td><td>The instance’s <strong>environment file</strong> (mode 0600): its configuration as <code>KEY=value</code> lines, read by its unit at start. Critically, it holds <code>LCHAT_ADMIN_TOKEN</code> — the database stores only a hash of the admin token, so <strong>this file is the only recoverable copy</strong>. <code>lchat admin-link <host></code> reads it for you.</td></tr> |
| + <tr><td><code>/data/<host>-blobs/</code></td><td>The instance’s encrypted attachment files (photos, videos), content-addressed, stored outside the database to keep the database small.</td></tr> |
| + <tr><td><code>/data/<host>.vapid</code></td><td>The instance’s web-push signing key. Kept out of the database on purpose (so database replicas in object storage carry no private key). If you ever move an instance between boxes, copy this file too, or every browser push subscription silently breaks — see <a href="operations.html#migration">part 08</a>.</td></tr> |
| + <tr><td><code>/data/<host>-bot.env</code> + <code>/data/<host>-bot/</code></td><td>The bot sidecar’s environment file and its persisted bot identities (session files). The existence of <code><host>-bot.env</code> is, literally, what “this instance has a sidecar” means to the system.</td></tr> |
| + <tr><td><code>/data/registry.db</code></td><td>The <a href="router.html#the-registry">registry</a> — the box’s routing table and instance ledger.</td></tr> |
| + <tr><td><code>/data/acme/</code></td><td>The TLS certificate cache (private keys inside — see <a href="router.html#tls">part 02</a>).</td></tr> |
| + <tr><td><code>/data/router.env</code>, <code>control.env</code>, <code>cloud.env</code>, <code>home.env</code>, <code>push.env</code>, <code>update.env</code></td><td>Per-service environment files, one per unit, same <code>KEY=value</code> format. Each unit declares its own; most are optional.</td></tr> |
| + <tr><td><code>/data/home.db</code>, <code>push.db</code>, <code>cloud.db</code></td><td>The sibling services’ own databases.</td></tr> |
| + <tr><td><code>/data/trash/<host>-<timestamp>/</code></td><td>Archived data of deleted spaces. Deletion in this system <strong>archives, never destroys</strong> — see <a href="spaces.html#removing">part 05</a>.</td></tr> |
| +</table> |
| +</div> |
| +<h3 id="run"><code>/run/lchat/</code> — the sockets</h3> |
| +<p>A memory-backed directory (recreated at every boot) holding the unix sockets: <code><host>.sock</code> for each awake instance, plus <code>home.sock</code>, <code>push.sock</code>, <code>cloud.sock</code>, and <code>control.sock</code>. Sockets are transient by nature; nothing here needs backing up.</p> |
| +<p>Two more paths complete the picture: <code>/etc/lchat/control.token</code> (a root-only secret gating the control daemon — <a href="spaces.html#controld">part 05</a>) and <code>/etc/litestream.yml</code> (the fleet-backup configuration — <a href="operations.html#backups">part 08</a>).</p> |
| + |
| +<h2 id="instances">The instance process, in detail</h2> |
| +<p>Each chat space runs as one instance of the template unit <code>lchat@.service</code>. Reading its key lines teaches you most of the process model:</p> |
| +<figure class="cmd"> |
| +<figcaption>The heart of /etc/systemd/system/lchat@.service (%i = the host)</figcaption> |
| +<pre><code>EnvironmentFile=-/data/%i.env |
| +ExecStart=/opt/lchat/live/%i --socket /run/lchat/%i.sock --db /data/%i.db |
| +Restart=always |
| +RestartSec=1 |
| +User=lchat |
| +MemoryMax=512M</code></pre> |
| +</figure> |
| +<ul> |
| + <li><strong>The host is the identity.</strong> The name after <code>@</code> flows into every path: the binary link, the socket, the database, the environment file. Knowing a host, you can find everything about it.</li> |
| + <li><strong>Configuration is the environment file.</strong> The instance reads ordinary environment variables (<code>LCHAT_ADMIN_TOKEN</code>, <code>LCHAT_PUSH_RELAY_URL</code>, and so on); the unit points them at <code>/data/<host>.env</code>. Changing an instance’s configuration means editing that file and running <code>systemctl restart lchat@<host></code>.</li> |
| + <li><strong>The version is the symlink.</strong> <code>ExecStart</code> runs <code>/opt/lchat/live/<host></code>, so a deploy changes the link and restarts the unit — nothing about the unit itself changes. <code>readlink /opt/lchat/live/<host></code> tells you exactly which build an instance runs.</li> |
| + <li><strong>Resource limits are per instance.</strong> <code>MemoryMax=512M</code> means one runaway instance is killed by the kernel before it can starve the others — isolation is enforced, not hoped for.</li> |
| + <li><strong>Graceful shutdown is built in.</strong> On being told to stop, the instance finishes in-flight requests for up to ten seconds and closes its WebSocket connections cleanly (clients auto-reconnect), then exits. Combined with the router’s <a href="router.html#restart-tolerance">thirteen-second dial retry</a>, this is why restarts do not drop requests.</li> |
| +</ul> |
| +<figure class="cmd"> |
| +<figcaption>Run on the box — the routine per-instance checks</figcaption> |
| +<pre><code>systemctl status lchat@chat-a.example.com # running? since when? recent log lines |
| +journalctl -u lchat@chat-a.example.com -n 200 # its full journal (add -f to follow) |
| +readlink /opt/lchat/live/chat-a.example.com # which build does it run? |
| +<span class="out">/opt/lchat/versions/1b4bd808/lchat</span> |
| +curl -s --unix-socket /run/lchat/chat-a.example.com.sock http://localhost/api/version |
| +<span class="out">{"version":"a41c9f2","build":"1b4bd808"}</span> # answers even if DNS/TLS are broken — isolates a problem to the instance itself</code></pre> |
| +</figure> |
| + |
| +<h2 id="security-model">The security posture every unit shares</h2> |
| +<p>All the units are heavily sandboxed with systemd’s confinement features: they run as the unprivileged system user <code>lchat</code>, see a read-only filesystem except <code>/data</code> and <code>/run/lchat</code>, cannot gain new privileges, cannot load kernel modules, and are memory-capped. Two deliberate exceptions run as root: the <strong>control daemon</strong> and the <strong>auto-updater</strong>, because their whole job is to do the two privileged things (start and stop units via <code>systemctl</code>, and write the version store). Both are reachable only locally — the control daemon on a local socket gated by a root-only token, the updater only via its timer. The router is the one unit with a special capability (<code>CAP_NET_BIND_SERVICE</code>) so it can bind ports 80 and 443 <em>without</em> being root.</p> |
| +<div class="plain"> |
| + <span class="label">In plain words</span> |
| + <p>Everything runs as a locked-down, unprivileged user except the two processes whose job is administration itself, and those two never talk to the internet. If an instance is somehow compromised, the sandbox means it cannot read other instances’ secrets, cannot touch the software store, and cannot escalate.</p> |
| +</div> |
| + |
| +<h2 id="sibling-services">The sibling services: home and push</h2> |
| +<p>Two services deploy next to the messenger but are not chat instances, and the router treats them as plain <code>service</code> routes — names it forwards traffic to, whose lifecycle it does not manage:</p> |
| +<ul> |
| + <li><strong>The home server</strong> (<code>lchat-home.service</code>) is the account-recovery backup store: members’ encrypted key backups, gated by passkeys. Its one operational constraint is that <strong>its hostname must never change</strong> — the hostname is part of the passkey security model (the WebAuthn “relying party” origin), and changing it orphans every passkey. Its environment file <code>/data/home.env</code> is <em>required</em> (no leading <code>-</code> in the unit) because a misconfigured home server is worse than a stopped one.</li> |
| + <li><strong>The push relay</strong> (<code>lchat-push.service</code>) holds the one Apple push-notification signing key for the whole fleet, so that individual instances never do. Instances authenticate to it with per-instance revocable tokens. Its environment file <code>/data/push.env</code> is likewise required.</li> |
| +</ul> |
| +<p>Both are registered with the router by <code>lchat add-service <host> --socket <path></code>, which simply adds a <code>service</code> row to the registry after checking the backend answers. Deploying either is manual and rare: copy the new binary over the old one and <code>systemctl restart</code> the unit (<a href="deploys.html#sibling-deploys">part 04</a>).</p> |
| + |
| +<h2 id="bot-sidecars">Bot sidecars: a member that runs on a server</h2> |
| +<p>An instance can have a <strong>bot sidecar</strong>: a process hosting the built-in helper personas (Claude and Eleven). Architecturally, a bot is an ordinary end-to-end encrypted <em>member</em> of the chat whose “device” happens to be a server process: it holds its own private keys (in the session files under <code>/data/<host>-bot/</code>) and connects to its instance as a WebSocket <em>client</em>, over the instance’s public hostname, exactly like a browser would. It serves no inbound traffic at all — no socket, no port, no route.</p> |
| +<div class="why"> |
| + <span class="label">Why it is this way</span> |
| + <p>The product’s core rule is that the server process never holds a usable decryption key. A bot must decrypt the threads it participates in, so its keys must live in a <em>different</em> process — one the operator can place anywhere (the same box, a different machine, behind a home router), because it only ever dials out. Compromising the server must not yield bot keys; this separation is what guarantees that.</p> |
| +</div> |
| +<p>Operationally: <code>lchat add-bot <host></code> provisions the identities and writes <code>/data/<host>-bot.env</code>; the unit <code>lchat-bot@<host>.service</code> runs <code>/opt/lchat/live/<host> bot run</code> — note that it execs <em>the same versioned symlink as its instance</em>, so the bot always runs the same build as its app, by construction, and every deploy restarts it onto the matching version. Give it intelligence by adding <code>ANTHROPIC_API_KEY=…</code> to its environment file and restarting; without a key the bots still run with deterministic (non-AI) behaviour.</p> |
| +<p>One subtlety worth knowing even if you never touch it: because the sidecar dials its instance’s <em>public</em> URL, the box must resolve its own hostnames to itself. Every hosted hostname is pinned to <code>127.0.0.1</code> in <code>/etc/hosts</code>, so on-box clients reach the local router directly instead of going out to the internet and back. Cloud images that regenerate <code>/etc/hosts</code> on boot have that behaviour disabled. If a sidecar logs connection failures after a migration, stale DNS or a missing hosts-pin is the first thing to check.</p> |
| +<p>On a <em>hibernating</em> instance the sidecar unit is disabled and the activator co-spawns the sidecar only while the instance is awake — that story is told in <a href="hibernation.html#sidecar-co-spawn">part 06</a>.</p> |
| + |
| +<h2 id="config-changes">How to change any process’s configuration</h2> |
| +<p>The pattern is identical everywhere, and worth stating once as a rule: <strong>configuration lives in an environment file under <code>/data/</code>, and takes effect on restart.</strong></p> |
| +<ol class="procedure"> |
| + <li><p class="step-title">Edit the environment file for the process.</p> |
| + <p>For an instance: <code>/data/<host>.env</code>. For the router: <code>/data/router.env</code>. For a sidecar: <code>/data/<host>-bot.env</code>. And so on per the <a href="#data">table above</a>. Keep the file’s ownership and 0600 mode — it may hold secrets.</p></li> |
| + <li><p class="step-title">Restart (or reload) the corresponding unit.</p> |
| + <figure class="cmd"> |
| + <figcaption>Run on the box</figcaption> |
| + <pre><code>systemctl restart lchat@chat-a.example.com # instance config change |
| +systemctl restart lchat-router # router config change (or reload, for route changes only)</code></pre> |
| + </figure></li> |
| + <li><p class="step-title">Confirm the process came back healthy.</p> |
| + <figure class="cmd"> |
| + <figcaption>Run on the box</figcaption> |
| + <pre><code>systemctl status lchat@chat-a.example.com |
| +curl -s https://chat-a.example.com/api/version</code></pre> |
| + </figure></li> |
| +</ol> |
| +<div class="warn"> |
| + <span class="label">One exception</span> |
| + <p>A <em>hibernating</em> instance has no unit to restart. Its environment file is read fresh by the activator at every wake, so an edit takes effect at the next wake with no action from you — or force the matter by making any request to the host. See <a href="hibernation.html#children">part 06</a>.</p> |
| +</div> |
| + |
| +</div> |
| +</main> |
| +</div> |
| +</body> |
| +</html> |
diff --git a/docs/planet-scale-guide/reference.html b/docs/planet-scale-guide/reference.html| new file mode 100644 |
| index 0000000..aa2e3c9 |
| --- /dev/null |
| +++ b/docs/planet-scale-guide/reference.html |
| @@ -0,0 +1,236 @@ |
| +<!doctype html> |
| +<html lang="en"> |
| +<head> |
| +<meta charset="utf-8" /> |
| +<meta name="viewport" content="width=device-width, initial-scale=1" /> |
| +<title>Complete reference — Running the fleet</title> |
| +<link rel="stylesheet" href="guide.css" /> |
| +<script src="guide.js" defer></script> |
| +</head> |
| +<body data-page="reference.html"> |
| +<div class="frame"> |
| +<nav class="side"></nav> |
| +<main class="content"> |
| +<div class="wrap"> |
| + |
| +<header class="top"> |
| + <p class="eyebrow"><span class="robot">🤖</span> Running the fleet · Part 09 of 09</p> |
| + <h1>Complete reference: commands, files, variables</h1> |
| + <p class="thesis">Everything in one place, for looking up rather than reading through: <strong>every operator command with its flags, every environment variable with its default, every file path, every systemd unit, and the glossary.</strong> Each entry links back to the page that explains it.</p> |
| +</header> |
| + |
| +<div class="sysmap" data-highlight=""></div> |
| +<p class="sysmap-caption">The map one last time, unhighlighted. If a term below is unfamiliar, the <a href="#glossary">glossary</a> at the bottom of this page defines it.</p> |
| + |
| +<h2 id="commands">Command reference: the <code>lchat</code> control commands</h2> |
| +<p>All of these run on a box (over <code>ssh</code>), as root, invoking <code>/opt/lchat/lchat</code>. Common conventions: <code><host></code> is always a full domain name; flags may come before or after the host; most commands accept <code>--registry <path></code> to point at a non-default registry (you will essentially never need it — the default is <code>/data/registry.db</code>).</p> |
| + |
| +<h3 id="cmd-lifecycle">Box and instance lifecycle</h3> |
| +<dl> |
| + <dt><code>lchat bootstrap</code></dt> |
| + <dd>One-time box setup: writes every systemd unit file, creates the <code>lchat</code> system user and the <code>/data</code>, <code>/run/lchat</code>, and certificate-cache directories, mints the root-only control token, enables and starts the router and the hourly update timer. Idempotent — safe to re-run to refresh unit files. Explained in <a href="deploys.html#bootstrap">part 04</a>.</dd> |
| + <dt><code>lchat add-instance <host> [--admin-token <tok>] [--hibernate] [--app-store-url <url>] [--home-origin <url>] [--push-relay-url <url>] [--push-relay-token <tok>]</code></dt> |
| + <dd>Creates a space: mints (or, with <code>--admin-token</code>, imports) the admin token, writes <code>/data/<host>.env</code>, registers the route, points the version link, starts and health-checks the unit (skipped with <code>--hibernate</code>), reloads the router, warms the certificate, and prints the one-time setup link and the admin link. Explained in <a href="spaces.html#adding-manual">part 05</a>.</dd> |
| + <dt><code>lchat add-service <host> --socket <path></code></dt> |
| + <dd>Registers a <code>service</code>-kind route (home server, push relay, cloud provisioner, or any third-party backend) after verifying the socket answers. No unit is created and no version is tracked — the router only forwards. Explained in <a href="processes.html#sibling-services">part 03</a>.</dd> |
| + <dt><code>lchat add-bot <host> [--model <model>]</code></dt> |
| + <dd>Provisions the bot sidecar for an instance: mints one agent invite per identity, writes <code>/data/<host>-bot.env</code> and the session directory, and enables <code>lchat-bot@<host></code> (on a hibernating host the unit is skipped; the activator co-spawns instead). Reads <code>ANTHROPIC_API_KEY</code> from the calling environment if AI behaviour is wanted. Explained in <a href="processes.html#bot-sidecars">part 03</a>.</dd> |
| + <dt><code>lchat remove <host> [--stop]</code></dt> |
| + <dd>Deletes the host’s registry row and reloads the router (traffic stops); with <code>--stop</code> also stops and disables the instance and sidecar units. <strong>Never touches data.</strong> Explained in <a href="spaces.html#removing">part 05</a>.</dd> |
| + <dt><code>lchat hibernate <host> --on | --off</code></dt> |
| + <dd>Moves one host between systemd management and activator management, in either direction, safely ordered so the host is never unserveable. Requires the router to be running with a hibernation bucket for <code>--on</code>. Explained in <a href="hibernation.html#host-switch">part 06</a>.</dd> |
| + <dt><code>lchat instances</code></dt> |
| + <dd>Prints the registry: HOST, KIND, STATUS, VERSION, HIBERNATE, SOCKET, plus a <code>last error</code> line for any failed row. The first stop for “what does this box think it is serving?”.</dd> |
| + <dt><code>lchat admin-link <host></code> / <code>lchat setup-invite <host></code></dt> |
| + <dd>Recover the admin link (from the environment file — the only plaintext copy of the token) / mint a fresh one-time setup invite.</dd> |
| +</dl> |
| + |
| +<h3 id="cmd-deploy">Versions and deploys</h3> |
| +<dl> |
| + <dt><code>lchat stage --file <binary> [--version <ver>]</code></dt> |
| + <dd>Copies a built binary into the version store under its build identifier (asked of the binary itself unless overridden). Inert and always safe. <a href="deploys.html#manual-deploy">Part 04</a>.</dd> |
| + <dt><code>lchat deploy <host>|--all [--version <ver>] [--pin] [--include-pinned]</code></dt> |
| + <dd>With <code>--version</code>: repoint the host’s live link, restart, poll <code>/api/version</code> for the exact build, auto-roll-back on mismatch, restart the bot sidecar. Without: restart on the current build. <code>--all</code> rolls one host at a time, stops on first failure, and skips pinned hosts unless <code>--include-pinned</code>. <code>--pin</code> (single host + version only) freezes the host there. Hibernating hosts: pure repoint, effective at next wake. <a href="deploys.html#manual-deploy">Part 04</a>.</dd> |
| + <dt><code>lchat rollback <host> | --all</code></dt> |
| + <dd>Repoints to the recorded previous version, restarts, health-checks, and swaps current/previous (so rollback twice returns forward). <a href="deploys.html#rollback">Part 04</a>.</dd> |
| + <dt><code>lchat init-versions [--from <binary>]</code></dt> |
| + <dd>One-time adoption when moving an existing box onto versioned deploys: stages a seed build and creates missing live links. Idempotent; never repoints an existing link.</dd> |
| + <dt><code>lchat update run [--dry-run]</code> / <code>status</code> / <code>pin <host> [<ver>]</code> / <code>unpin <host></code></dt> |
| + <dd>One updater pass (what the hourly timer fires) / the fleet update table (HOST, CHANNEL, AUTO, CURRENT, AVAILABLE) / freeze a host on a version / resume its channel. <a href="deploys.html#auto-updates">Part 04</a>.</dd> |
| + <dt><code>lchat version</code></dt> |
| + <dd>Prints the binary’s own build identifier (also served by every instance at <code>GET /api/version</code> as <code>build</code>).</dd> |
| +</dl> |
| + |
| +<h3 id="cmd-daemons">Daemons and utilities</h3> |
| +<dl> |
| + <dt><code>lchat router</code></dt> |
| + <dd>The router process itself — run by <code>lchat-router.service</code>, not by hand. Flags <code>--registry</code>, <code>--acme-cache</code>, <code>--https</code>, <code>--http</code>, <code>--fallback-url</code>, <code>--wildcard-domains</code>. <a href="router.html">Part 02</a>.</dd> |
| + <dt><code>lchat controld</code></dt> |
| + <dd>The root-only control daemon — run by <code>lchat-control.service</code>. Flags/env for socket, token file, and capacity policy are under <a href="#env-control">environment variables</a>. <a href="spaces.html#controld">Part 05</a>.</dd> |
| + <dt><code>lchat cloud --socket <path> --db <path></code></dt> |
| + <dd>The self-serve provisioner — run by <code>lchat-cloud.service</code>. <a href="spaces.html#cloud">Part 05</a>.</dd> |
| + <dt><code>lchat bot run [--base <url>]</code> / <code>lchat bot provision</code></dt> |
| + <dd>Run a bot sidecar (with no arguments, configuration comes from the unit’s environment file) / claim an agent invite into a session file. <a href="processes.html#bot-sidecars">Part 03</a>.</dd> |
| + <dt><code>lchat release keygen|sign|verify|payload</code></dt> |
| + <dd>The release-feed signing plumbing — used by the publishing pipeline, not in box operation. <a href="deploys.html#signing">Part 04</a>.</dd> |
| +</dl> |
| + |
| +<h2 id="env">Environment variable reference</h2> |
| +<p>Grouped by the file each is read from. All files are plain <code>KEY=value</code> lines, mode 0600, under <code>/data/</code> unless noted.</p> |
| + |
| +<h3 id="env-router">The router — <code>/data/router.env</code></h3> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Variable</th><th>Default</th><th>Meaning</th></tr> |
| + <tr><td><code>LCHAT_REGISTRY</code></td><td><code>/data/registry.db</code></td><td>The registry database.</td></tr> |
| + <tr><td><code>LCHAT_ACME_CACHE</code></td><td><code>/data/acme</code></td><td>Certificate cache directory.</td></tr> |
| + <tr><td><code>LCHAT_HTTPS_ADDR</code> / <code>LCHAT_HTTP_ADDR</code></td><td><code>:443</code> / <code>:80</code></td><td>Public listen addresses.</td></tr> |
| + <tr><td><code>LCHAT_FALLBACK_URL</code></td><td>unset (404)</td><td>Redirect target for unknown hostnames.</td></tr> |
| + <tr><td><code>LCHAT_WILDCARD_DOMAINS</code></td><td>unset</td><td>Comma list of domains to hold wildcard certificates for (needs the next variable). <a href="router.html#dns01">Part 02</a>.</td></tr> |
| + <tr><td><code>LCHAT_DNSIMPLE_TOKEN</code></td><td>unset</td><td>DNS-provider API token used for wildcard issuance.</td></tr> |
| + <tr><td><code>LCHAT_HIBERNATE_BUCKET</code></td><td>unset (activator off)</td><td>The hibernation bucket — the activator’s master switch. <a href="hibernation.html#activator-config">Part 06</a>.</td></tr> |
| + <tr><td><code>LCHAT_HIBERNATE_REGION</code> / <code>LCHAT_HIBERNATE_PREFIX</code></td><td>empty</td><td>Bucket region / key prefix.</td></tr> |
| + <tr><td><code>LCHAT_HIBERNATE_IDLE</code></td><td><code>15m</code></td><td>Idle window before sleep (a duration such as <code>15m</code> or <code>90s</code>; must be positive).</td></tr> |
| + <tr><td><code>LCHAT_MESH_SELF</code></td><td>unset (mesh off)</td><td>This box’s peer-reachable base URL — the mesh’s master switch. <a href="mesh.html#mesh-config">Part 07</a>.</td></tr> |
| + <tr><td><code>LCHAT_MESH_DOMAINS</code></td><td>unset</td><td>Comma list of domains whose hosts participate in mesh placement. Required whenever <code>LCHAT_MESH_SELF</code> is set.</td></tr> |
| +</table> |
| +</div> |
| + |
| +<h3 id="env-instance">Each instance — <code>/data/<host>.env</code></h3> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Variable</th><th>Meaning</th></tr> |
| + <tr><td><code>LCHAT_ADMIN_TOKEN</code></td><td>The instance’s admin token. The database stores only its hash; <strong>this file is the only recoverable copy</strong>.</td></tr> |
| + <tr><td><code>LCHAT_APP_STORE_URL</code>, <code>LCHAT_HOME_ORIGIN</code>, <code>LCHAT_PUSH_RELAY_URL</code>, <code>LCHAT_PUSH_RELAY_TOKEN</code></td><td>Per-instance pointers to the app-store listing, the home server, and the push relay (with its per-instance bearer token).</td></tr> |
| + <tr><td><code>LCHAT_BLOB_QUOTA_BYTES</code></td><td>Attachment-storage cap for the space (written by the control daemon’s quota endpoint; uploads beyond it are refused).</td></tr> |
| +</table> |
| +</div> |
| + |
| +<h3 id="env-control">The control daemon — <code>/data/control.env</code></h3> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Variable</th><th>Default</th><th>Meaning</th></tr> |
| + <tr><td><code>LCHAT_MAX_INSTANCES</code></td><td>40</td><td>Ceiling on running (materialised) instances.</td></tr> |
| + <tr><td><code>LCHAT_MIN_FREE_GB</code></td><td>5</td><td>Disk floor; creations refuse below it.</td></tr> |
| + <tr><td><code>LCHAT_CONTROL_HIBERNATE_CREATES</code></td><td>off</td><td><code>1</code> = every provisioner-created space is born hibernating. <a href="spaces.html#hibernating-creates">Part 05</a>.</td></tr> |
| + <tr><td><code>LCHAT_MAX_REGISTERED</code></td><td>0 (= 10 × max instances)</td><td>Row ceiling counting hibernating registrations.</td></tr> |
| + <tr><td><code>LCHAT_TRASH_KEEP_DAYS</code></td><td>0 (keep forever)</td><td>Age at which archived (deleted) spaces in <code>/data/trash</code> are purged.</td></tr> |
| + <tr><td><code>LCHAT_CONTROL_SOCKET</code> / <code>LCHAT_CONTROL_TOKEN_FILE</code></td><td><code>/run/lchat/control.sock</code> / <code>/etc/lchat/control.token</code></td><td>Where it listens / the root-only bearer token gating it.</td></tr> |
| +</table> |
| +</div> |
| + |
| +<h3 id="env-cloud">The cloud provisioner — <code>/data/cloud.env</code></h3> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Variable</th><th>Default</th><th>Meaning</th></tr> |
| + <tr><td><code>LCHAT_ADMIN_TOKEN</code></td><td>required</td><td>Gates the caller-token minting API.</td></tr> |
| + <tr><td><code>LCHAT_CLOUD_BASE_DOMAIN</code></td><td>required</td><td>Domain new spaces are created under (needs wildcard DNS).</td></tr> |
| + <tr><td><code>LCHAT_CLOUD_QUOTA</code></td><td>3</td><td>Free spaces each requesting user may hold at once.</td></tr> |
| + <tr><td><code>LCHAT_CLOUD_MAX_CREATES_PER_DAY</code></td><td>0 (uncapped)</td><td>Fleet-wide daily creation circuit breaker.</td></tr> |
| + <tr><td><code>LCHAT_CLOUD_REAP_AFTER_DAYS</code></td><td>0 (reaper off)</td><td>Age before a never-claimed space is reclaimable. <a href="spaces.html#reaper">Part 05</a>.</td></tr> |
| + <tr><td><code>LCHAT_CLOUD_REAP_INTERVAL_HOURS</code> / <code>LCHAT_CLOUD_REAP_MAX_PER_SWEEP</code> / <code>LCHAT_CLOUD_REAP_DRY_RUN</code></td><td>6 / 5 / off</td><td>Reaper sweep cadence / per-sweep cap / log-only mode.</td></tr> |
| + <tr><td><code>LCHAT_CLOUD_STORAGE_CAPS</code>, <code>LCHAT_STRIPE_*</code></td><td>unset</td><td>Per-tier storage caps and the dormant billing rails. <span class="status partial">partially built</span></td></tr> |
| +</table> |
| +</div> |
| + |
| +<h3 id="env-other">Other files</h3> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>File</th><th>Notable variables</th></tr> |
| + <tr><td><code>/data/<host>-bot.env</code></td><td><code>LCHAT_BASE</code> (the instance’s public URL), the per-identity session paths, optional <code>ANTHROPIC_API_KEY</code> and <code>LCHAT_MODEL</code>. <a href="processes.html#bot-sidecars">Part 03</a>.</td></tr> |
| + <tr><td><code>/data/update.env</code></td><td><code>LCHAT_UPDATE_MANIFEST_URL</code> (point a box at a different release feed); mirror fallback via <code>LCHAT_RELEASE_MIRROR_URL</code>. <a href="deploys.html#signing">Part 04</a>.</td></tr> |
| + <tr><td><code>/data/home.env</code> (required)</td><td><code>LCHAT_HOME_ORIGIN</code> (the passkey origin — must never change), <code>LCHAT_CLOUD_URL</code> + <code>LCHAT_CLOUD_TOKEN</code> (enables self-serve creation), <code>LCHAT_HOME_STORE</code> (<code>sqlite</code> default, or <code>s3</code> with <code>LCHAT_HOME_S3_BUCKET</code>/<code>_REGION</code>).</td></tr> |
| + <tr><td><code>/data/push.env</code> (required)</td><td>The Apple push signing key and its identifiers — the one copy in the fleet, plus the relay’s admin token.</td></tr> |
| +</table> |
| +</div> |
| + |
| +<h2 id="paths">File path reference</h2> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Path</th><th>What it is</th><th>Back up?</th></tr> |
| + <tr><td><code>/opt/lchat/lchat</code>, <code>/opt/lchat/home</code>, <code>/opt/lchat/push</code></td><td>The binaries.</td><td>No — rebuildable.</td></tr> |
| + <tr><td><code>/opt/lchat/versions/<ver>/lchat</code></td><td>The version store (every staged build).</td><td>No.</td></tr> |
| + <tr><td><code>/opt/lchat/live/<host></code></td><td>The per-instance version pointer (a symlink). <em>Which build a host runs.</em></td><td>No (recorded in the registry).</td></tr> |
| + <tr><td><code>/data/<host>.db</code> (+ <code>-wal</code>, <code>-shm</code>)</td><td>A space’s entire database (ciphertext).</td><td><strong>Yes</strong> — continuously, via the backup replicator.</td></tr> |
| + <tr><td><code>/data/<host>.env</code></td><td>A space’s configuration and its only plaintext admin token.</td><td><strong>Yes.</strong></td></tr> |
| + <tr><td><code>/data/<host>-blobs/</code></td><td>A space’s encrypted attachments.</td><td><strong>Yes</strong> (file-level).</td></tr> |
| + <tr><td><code>/data/<host>.vapid</code></td><td>A space’s web-push key. Losing it breaks push subscriptions.</td><td><strong>Yes.</strong></td></tr> |
| + <tr><td><code>/data/<host>-bot.env</code>, <code>/data/<host>-bot/</code></td><td>Bot sidecar configuration and identities.</td><td><strong>Yes.</strong></td></tr> |
| + <tr><td><code>/data/registry.db</code></td><td>The box’s routing table and instance ledger.</td><td><strong>Yes.</strong></td></tr> |
| + <tr><td><code>/data/acme/</code></td><td>TLS certificates and their private keys.</td><td>Yes (or accept re-issuance).</td></tr> |
| + <tr><td><code>/data/trash/<host>-<timestamp>/</code></td><td>Archived deleted spaces.</td><td>Optional.</td></tr> |
| + <tr><td><code>/run/lchat/*.sock</code></td><td>The unix sockets (transient).</td><td>No.</td></tr> |
| + <tr><td><code>/etc/lchat/control.token</code></td><td>The root-only control-daemon secret.</td><td><strong>Yes</strong> (secretly).</td></tr> |
| + <tr><td><code>/etc/litestream.yml</code></td><td>Generated backup-replication configuration.</td><td>No — regenerate with <code>deploy/fleet-backup.sh</code>.</td></tr> |
| + <tr><td><code>s3://<bucket>/<prefix>/<host>/</code></td><td>A hibernating space’s parked database and its <code>lock.json</code> lease.</td><td>The bucket’s own versioning covers it.</td></tr> |
| + <tr><td><code>s3://<bucket>/<prefix>/mesh/members/</code></td><td>The mesh membership heartbeats.</td><td>No — transient by design.</td></tr> |
| +</table> |
| +</div> |
| + |
| +<h2 id="units">systemd unit reference</h2> |
| +<p>Covered in narrative form in <a href="processes.html#census">part 03</a>; the lookup table:</p> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Unit</th><th>Runs</th><th>Enabled by</th><th>Journal</th></tr> |
| + <tr><td><code>lchat-router.service</code></td><td>The router (+ activator + mesh)</td><td><code>bootstrap</code></td><td><code>journalctl -u lchat-router</code></td></tr> |
| + <tr><td><code>lchat@<host>.service</code></td><td>One instance</td><td><code>add-instance</code></td><td><code>journalctl -u 'lchat@<host>'</code></td></tr> |
| + <tr><td><code>lchat-bot@<host>.service</code></td><td>One bot sidecar</td><td><code>add-bot</code></td><td><code>journalctl -u 'lchat-bot@<host>'</code></td></tr> |
| + <tr><td><code>lchat-home.service</code> / <code>lchat-push.service</code></td><td>Home server / push relay</td><td>manually, once</td><td><code>journalctl -u lchat-home</code> etc.</td></tr> |
| + <tr><td><code>lchat-control.service</code></td><td>Control daemon (root)</td><td>manually, once</td><td><code>journalctl -u lchat-control</code></td></tr> |
| + <tr><td><code>lchat-cloud.service</code></td><td>Cloud provisioner</td><td>manually, once</td><td><code>journalctl -u lchat-cloud</code></td></tr> |
| + <tr><td><code>lchat-update.timer</code> → <code>.service</code></td><td>Hourly update pass (root)</td><td><code>bootstrap</code></td><td><code>journalctl -u lchat-update.service</code></td></tr> |
| + <tr><td><code>lchat-litestream.service</code></td><td>Fleet backup replicator</td><td><code>deploy/fleet-backup.sh</code></td><td><code>journalctl -u lchat-litestream</code></td></tr> |
| +</table> |
| +</div> |
| + |
| +<h2 id="endpoints">Health and inspection endpoints</h2> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Endpoint</th><th>On</th><th>Returns</th></tr> |
| + <tr><td><code>GET /api/version</code></td><td>Every instance (no authentication)</td><td><code>{"version": <frontend build>, "build": <binary build id>}</code> — the health check deploys poll.</td></tr> |
| + <tr><td><code>GET /healthz</code></td><td>The cloud provisioner</td><td><code>ok</code>.</td></tr> |
| + <tr><td><code>GET /api/next-due</code> / <code>POST /api/charter-next-due</code></td><td>Every instance (token-gated, used by the activator and sidecar)</td><td>The earliest scheduled delivery — the machinery behind <a href="hibernation.html#due-time-wakes">scheduled wakes</a>. Not an operator surface.</td></tr> |
| +</table> |
| +</div> |
| + |
| +<h2 id="glossary">Glossary</h2> |
| +<dl> |
| + <dt id="gloss-space">space / instance / host</dt> |
| + <dd>One chat group; its server-side unit (one process + one SQLite database + one subdomain); and that subdomain itself, which names everything belonging to the instance. See <a href="index.html#vocabulary">the vocabulary table</a>.</dd> |
| + <dt id="gloss-box">box</dt> |
| + <dd>One Linux server in the fleet, running the router and everything in <a href="processes.html">part 03</a>.</dd> |
| + <dt id="gloss-systemd">systemd</dt> |
| + <dd>The standard Linux service manager: it starts processes at boot, restarts them when they crash, applies sandboxing, and keeps their logs (the “journal”). Controlled with <code>systemctl</code>, read with <code>journalctl</code>.</dd> |
| + <dt id="gloss-unit">unit / template unit</dt> |
| + <dd>A systemd service definition; a template unit (name containing <code>@</code>) is stamped out per instance name, which is how one file defines the service for every chat space.</dd> |
| + <dt id="gloss-unix-socket">unix socket</dt> |
| + <dd>A file (here under <code>/run/lchat/</code>) that local processes connect to like a network address. Invisible to the network — the reason instances need no firewall rules of their own.</dd> |
| + <dt id="gloss-registry">the registry</dt> |
| + <dd>The SQLite file <code>/data/registry.db</code>: one row per hostname a box serves. The router’s routing table and the fleet commands’ ledger. <a href="router.html#the-registry">Part 02</a>.</dd> |
| + <dt id="gloss-dns">DNS / wildcard DNS</dt> |
| + <dd>The system translating names to network addresses. A <em>wildcard</em> record (<code>*.example.com</code>) points every subdomain at once, which is why creating a space needs no DNS step.</dd> |
| + <dt id="gloss-le">Let’s Encrypt / ACME / HTTP-01 / DNS-01</dt> |
| + <dd>The free certificate authority and its automated issuance protocol. HTTP-01 proves control of a hostname by answering a challenge on port 80 (used per host); DNS-01 proves control of a whole domain via a temporary DNS record (used for wildcard certificates). <a href="router.html#tls">Part 02</a>.</dd> |
| + <dt id="gloss-s3">object storage / S3 / bucket / prefix</dt> |
| + <dd>Storage addressed as named objects in a “bucket” rather than files on a disk — durable, replicated, and shared. A “prefix” is the leading part of object names, used like a directory. Amazon S3 is the implementation used here.</dd> |
| + <dt id="gloss-litestream">Litestream</dt> |
| + <dd>A tool that continuously streams a SQLite database’s changes into object storage, allowing restore to any recent moment. Used twice in this system, in two forms: a stock copy for <a href="operations.html#backups">fleet backups</a>, and embedded inside the router for <a href="hibernation.html#the-lease">hibernation</a>.</dd> |
| + <dt id="gloss-lease">the lease</dt> |
| + <dd>The per-space lock object (<code>lock.json</code>) in the hibernation bucket, held while a process serves the space and renewed every few seconds. The system’s single source of truth about who may write a space’s data. <a href="hibernation.html#the-lease">Part 06</a>.</dd> |
| + <dt id="gloss-activator">the activator</dt> |
| + <dd>The component inside the router that wakes sleeping spaces on demand and puts idle ones to sleep. Exists only when <code>LCHAT_HIBERNATE_BUCKET</code> is set. <a href="hibernation.html">Part 06</a>.</dd> |
| + <dt id="gloss-hrw">rendezvous hashing</dt> |
| + <dd>The rule mesh boxes use to agree which of them owns each space without communicating: every box scores every (box, space) pair the same way, and the highest score wins. Adding or removing a box only moves the spaces whose winner changed. <a href="mesh.html#placement">Part 07</a>.</dd> |
| + <dt id="gloss-sidecar">bot sidecar</dt> |
| + <dd>The separate process hosting an instance’s bot members. It holds the bots’ encryption keys (which must never live in the server process) and connects outward as an ordinary client. <a href="processes.html#bot-sidecars">Part 03</a>.</dd> |
| + <dt id="gloss-canary">canary / pin</dt> |
| + <dd>Deploying a new build to one instance to watch it before promoting it everywhere; and the mark that freezes an instance on a version so the hourly auto-updater leaves the experiment alone. <a href="deploys.html#pinning">Part 04</a>.</dd> |
| + <dt id="gloss-vapid">web push / VAPID key</dt> |
| + <dd>Browser push notifications, signed per instance with a key pair (the <code>.vapid</code> file). Distinct from Apple device push, which goes through the push relay.</dd> |
| +</dl> |
| + |
| +</div> |
| +</main> |
| +</div> |
| +</body> |
| +</html> |
diff --git a/docs/planet-scale-guide/router.html b/docs/planet-scale-guide/router.html| new file mode 100644 |
| index 0000000..dde5fba |
| --- /dev/null |
| +++ b/docs/planet-scale-guide/router.html |
| @@ -0,0 +1,159 @@ |
| +<!doctype html> |
| +<html lang="en"> |
| +<head> |
| +<meta charset="utf-8" /> |
| +<meta name="viewport" content="width=device-width, initial-scale=1" /> |
| +<title>The router — Running the fleet</title> |
| +<link rel="stylesheet" href="guide.css" /> |
| +<script src="guide.js" defer></script> |
| +</head> |
| +<body data-page="router.html"> |
| +<div class="frame"> |
| +<nav class="side"></nav> |
| +<main class="content"> |
| +<div class="wrap"> |
| + |
| +<header class="top"> |
| + <p class="eyebrow"><span class="robot">🤖</span> Running the fleet · Part 02 of 09</p> |
| + <h1>The router: the front door of every box</h1> |
| + <p class="thesis">Exactly one process on each box listens on the public internet. It terminates TLS, looks each request’s hostname up in a small database, and forwards the request to the right local process over a unix socket. <strong>Everything else on the box hides behind it.</strong></p> |
| +</header> |
| + |
| +<div class="sysmap" data-highlight="router"></div> |
| +<p class="sysmap-caption">You are here: the router is the single public-facing process on a box. The hibernation activator (<a href="hibernation.html">part 06</a>) and the mesh (<a href="mesh.html">part 07</a>) both live inside this same process, switched on by configuration.</p> |
| + |
| +<h2 id="what-it-is">What the router is and why it exists</h2> |
| +<p>A box hosts many chat space instances, plus a few supporting services. Each of those is its own process, and none of them listens on a public network port — every one of them listens on a <strong>unix socket</strong>, which is a special file (in this system, under <code>/run/lchat/</code>) that processes on the same machine can connect to like a network address. Unix sockets are invisible to the internet, which is the point: the only way traffic reaches an instance is through the router.</p> |
| +<p>The router is started by systemd as the service <code>lchat-router.service</code>, and it is the same single <code>lchat</code> binary as everything else, started with the <code>router</code> subcommand. Its full start command, exactly as the systemd unit runs it, is:</p> |
| +<figure class="cmd"> |
| +<figcaption>What systemd runs (from /etc/systemd/system/lchat-router.service)</figcaption> |
| +<pre><code>/opt/lchat/lchat router --registry /data/registry.db --acme-cache /data/acme</code></pre> |
| +</figure> |
| +<p>The two arguments name the two files the router depends on: the <a href="#the-registry">registry</a> (its routing table) and the <a href="#tls">certificate cache</a> (its TLS material). The router has three jobs, taken in turn below:</p> |
| +<ol> |
| + <li><strong>TLS:</strong> obtain and present the right certificate for every hostname it serves.</li> |
| + <li><strong>Routing:</strong> map each request’s hostname to the right unix socket and forward the request.</li> |
| + <li><strong>Hosting the activator and the mesh:</strong> when configured, wake sleeping instances (<a href="hibernation.html">part 06</a>) and cooperate with peer boxes (<a href="mesh.html">part 07</a>). This page covers the first two; the other two have their own pages.</li> |
| +</ol> |
| + |
| +<div class="plain"> |
| + <span class="label">In plain words</span> |
| + <p>Think of the box as an office building. The router is the reception desk: it is the only door to the street, it checks which company (hostname) a visitor is asking for, and it walks the visitor to the right office (unix socket). The offices have no street doors of their own.</p> |
| +</div> |
| + |
| +<h2 id="the-registry">The registry: the routing table on disk</h2> |
| +<p>The router’s routing table lives in a small SQLite database, by default <code>/data/registry.db</code>, called <strong>the registry</strong> throughout this guide. It has one table, <code>routes</code>, with one row per hostname the box serves. The columns you will care about as an operator:</p> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Column</th><th>Meaning</th></tr> |
| + <tr><td><code>host</code></td><td>The full domain name, for example <code>chat-a.example.com</code>. This is the primary key.</td></tr> |
| + <tr><td><code>kind</code></td><td><code>instance</code> for a messenger space the box lifecycle-manages, or <code>service</code> for a sibling service (the home server, the push relay, the cloud provisioner) that deploys on its own cadence.</td></tr> |
| + <tr><td><code>socket</code></td><td>The unix socket the router forwards this host’s traffic to, for example <code>/run/lchat/chat-a.example.com.sock</code>.</td></tr> |
| + <tr><td><code>status</code></td><td><code>creating</code>, <code>ready</code>, or <code>failed</code>. Only <code>ready</code> rows are served; a <code>failed</code> row keeps its error message in <code>last_error</code>.</td></tr> |
| + <tr><td><code>version</code> / <code>prev_version</code></td><td>Which build of the software this instance currently runs, and which it ran before — the basis for <a href="deploys.html#rollback">rollback</a>.</td></tr> |
| + <tr><td><code>hibernate</code></td><td>Whether this host is managed by the <a href="hibernation.html">activator</a> (put to sleep when idle) instead of by systemd.</td></tr> |
| + <tr><td><code>next_due</code></td><td>For a hibernating host: the time of its earliest scheduled delivery (for example a reminder), so the activator can wake it on time with no request. See <a href="hibernation.html#due-time-wakes">part 06</a>.</td></tr> |
| + <tr><td><code>db_path</code>, <code>app_store_url</code>, <code>home_origin</code>, <code>push_relay_url</code>, <code>push_relay_token</code>, <code>home_region</code>, <code>provider</code></td><td>Per-instance configuration recorded at creation time. <code>home_region</code> and <code>provider</code> are written but not yet acted on — they are seams for future geographic and co-operative placement. <span class="status partial">partially built</span></td></tr> |
| +</table> |
| +</div> |
| +<p>You rarely edit the registry directly — the command-line tools (<code>lchat add-instance</code>, <code>lchat remove</code>, and so on) do it for you, and <code>lchat instances</code> prints it in a readable form. But it is useful to know that the whole routing state of a box is one ordinary SQLite file you can inspect:</p> |
| +<figure class="cmd"> |
| +<figcaption>Run on the box — inspect the routing table directly (read-only)</figcaption> |
| +<pre><code>sqlite3 "file:/data/registry.db?mode=ro" "SELECT host, kind, status, version, hibernate FROM routes" |
| +<span class="out">chat-a.example.com|instance|ready|1b4bd808|0 |
| +chat-b.example.com|instance|ready|1b4bd808|1 |
| +home.example.com|service|ready||0</span></code></pre> |
| +</figure> |
| + |
| +<h3 id="reloading">How the router learns about changes</h3> |
| +<p>The router keeps the routing table in memory (swapped atomically, so a reload never drops an in-flight connection) and re-reads the registry in two ways:</p> |
| +<ul> |
| + <li><strong>On a reload signal.</strong> <code>systemctl reload lchat-router</code> sends the process a HUP signal, and the unit’s <code>ExecReload</code> line is exactly that. The provisioning commands (<code>add-instance</code>, <code>remove</code>, <code>hibernate</code>) trigger this themselves, so you rarely type it.</li> |
| + <li><strong>On a two-second poll.</strong> The router also fingerprints the registry file every two seconds and reloads if it changed. This exists because a change made by another process (for example the control daemon) must be picked up even if nobody sends a signal.</li> |
| +</ul> |
| + |
| +<h2 id="tls">TLS certificates: automatic, two ways</h2> |
| +<p>The router obtains its certificates from <a href="reference.html#gloss-le">Let’s Encrypt</a> automatically. There are two mechanisms, and they coexist:</p> |
| +<h3 id="http01">Per-host certificates, issued on first contact</h3> |
| +<p>By default the router uses the ACME <strong>HTTP-01</strong> method: when a browser first connects for a hostname the router has never served, the router asks Let’s Encrypt for a certificate for exactly that hostname, answers the validation request Let’s Encrypt makes to port 80, and receives a certificate within a few seconds. Certificates (and their private keys) are cached on disk in <code>/data/acme/</code> — one file per host — so this happens once per host, not once per connection.</p> |
| +<p>Crucially, the router will only request a certificate for a hostname that exists in the registry with a serveable status. This rule (the “host policy”) prevents strangers from pointing arbitrary DNS names at your box and burning through Let’s Encrypt’s rate limits on your behalf.</p> |
| +<h3 id="dns01">A wildcard certificate for a whole domain</h3> |
| +<p>Per-host issuance rides Let’s Encrypt’s rate limit of roughly fifty new certificates per registered domain per week — a hard ceiling on how fast new spaces can be created. So the router can additionally hold a <strong>wildcard</strong> certificate (<code>*.example.com</code>), obtained with the ACME <strong>DNS-01</strong> method, which covers every direct subdomain at once. Enable it by setting two variables in the router’s environment file, <code>/data/router.env</code>, then restarting the router:</p> |
| +<figure class="cmd"> |
| +<figcaption>Contents of /data/router.env (edit on the box, then: systemctl restart lchat-router)</figcaption> |
| +<pre><code>LCHAT_WILDCARD_DOMAINS=example.com |
| +LCHAT_DNSIMPLE_TOKEN=<a DNSimple account API token that can edit the example.com zone></code></pre> |
| +</figure> |
| +<p>With this set, the router re-checks the wildcard certificate at startup and every twelve hours, renewing when it is missing or within thirty days of expiry. Issuance works by publishing a temporary DNS record (<code>_acme-challenge.example.com</code>) through the DNSimple API, which is why the token is needed. The certificate persists in the same <code>/data/acme/</code> cache as <code>wildcard-example.com.pem</code>. Hostnames on other domains, and the apex domain itself, keep using the per-host path — and if wildcard issuance ever fails, per-host issuance quietly keeps covering those hosts too, so a DNS-provider outage degrades gracefully instead of taking the box down.</p> |
| +<div class="warn"> |
| + <span class="label">Treat the certificate cache as secret material</span> |
| + <p>The files in <code>/data/acme/</code> contain private keys. They are mode 0600 for a reason. Include the directory in backups (losing it only forces re-issuance, but re-issuance is rate-limited), and never copy it anywhere world-readable.</p> |
| +</div> |
| +<div class="roadmap"> |
| + <span class="label">Designed, not built: a shared certificate cache in S3</span> |
| + <p>Today each box keeps its own certificate cache on its own disk, which is fine while each hostname is served by exactly one box. The day a <em>pool</em> of boxes must be able to answer for the same hostname, the cache needs to live somewhere they all can see — object storage. The configuration value for this, <code>LCHAT_CERT_CACHE=s3</code>, is already reserved in the code and deliberately <em>refuses to start</em>, so nobody can half-enable an unfinished mode. Until it exists, certificate issuance ties a hostname to one box. See also <a href="mesh.html#mesh-limits">the mesh’s honest limits</a>.</p> |
| +</div> |
| + |
| +<h2 id="ports">What the router does on each port</h2> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Port</th><th>Behaviour</th></tr> |
| + <tr><td><strong>443 (HTTPS)</strong></td><td>Terminates TLS with the certificate for the requested hostname, then routes by hostname (below). WebSocket connections — which the chat clients hold open continuously — pass straight through.</td></tr> |
| + <tr><td><strong>80 (HTTP)</strong></td><td>Answers Let’s Encrypt validation requests (<code>/.well-known/acme-challenge/…</code>), and permanently redirects everything else to HTTPS.</td></tr> |
| +</table> |
| +</div> |
| +<p>These are the only two public ports on a box. The firewall guidance in <a href="operations.html#hardening">part 08</a> allows exactly 22 (SSH), 80, and 443 inbound, and nothing else.</p> |
| + |
| +<h2 id="routing">How a request is routed, step by step</h2> |
| +<ol> |
| + <li>A request arrives for, say, <code>https://chat-a.example.com/api/version</code>. The router strips any port from the hostname and looks <code>chat-a.example.com</code> up in its in-memory copy of the registry.</li> |
| + <li><strong>If the host is unknown</strong> (no <code>ready</code> row), the router answers 404 — or, if <code>LCHAT_FALLBACK_URL</code> is set in <code>/data/router.env</code>, redirects to that URL instead. The fallback pairs well with wildcard DNS: a mistyped or retired subdomain lands on your marketing site rather than an error page.</li> |
| + <li><strong>If the host is a normal (systemd-managed) route</strong>, the router opens a connection to the row’s unix socket and forwards the request, preserving the original hostname so the instance knows who it is.</li> |
| + <li><strong>If the host is flagged for hibernation</strong>, the request is handed to the activator inside the router, which may need to wake the instance first — the whole story is <a href="hibernation.html">part 06</a>.</li> |
| + <li><strong>If the box is in a mesh</strong> and the placement rule says another box owns this host, the request is forwarded to that box instead — <a href="mesh.html">part 07</a>.</li> |
| +</ol> |
| + |
| +<h3 id="restart-tolerance">Why deploys do not drop requests</h3> |
| +<p>When an instance restarts (during a <a href="deploys.html">deploy</a>), there is a brief moment when the old process has exited and the new one has not yet bound its socket. During that gap, the router’s connection attempts fail — and the router responds by <strong>retrying the connection for up to thirteen seconds</strong> before giving up. Thirteen seconds comfortably exceeds an instance’s shutdown drain (ten seconds) plus startup, so in practice a request arriving mid-deploy just waits a beat and then succeeds. This retry window is a load-bearing detail: it is what makes “restart the instance” a zero-downtime operation without any blue-green complexity.</p> |
| +<p>The router also quietly strips two internal headers from every request before forwarding it (<code>X-Lchat-Sidecar</code> and <code>X-Lchat-Mesh-Forwarded</code>), so no outside client can forge the markers that <a href="hibernation.html#sidecar-co-spawn">bot sidecars</a> and <a href="mesh.html#one-hop">mesh peers</a> use.</p> |
| + |
| +<h2 id="router-config">Configuring the router</h2> |
| +<p>The router reads optional configuration from <code>/data/router.env</code> (a plain <code>KEY=value</code> file; the unit declares it with a leading <code>-</code>, meaning the file may be absent). Everything it understands:</p> |
| +<div class="tablewrap"> |
| +<table> |
| + <tr><th>Variable</th><th>Default</th><th>Meaning</th></tr> |
| + <tr><td><code>LCHAT_REGISTRY</code></td><td><code>/data/registry.db</code></td><td>Path to the registry database (also a <code>--registry</code> flag).</td></tr> |
| + <tr><td><code>LCHAT_ACME_CACHE</code></td><td><code>/data/acme</code></td><td>Certificate cache directory (also <code>--acme-cache</code>).</td></tr> |
| + <tr><td><code>LCHAT_HTTPS_ADDR</code> / <code>LCHAT_HTTP_ADDR</code></td><td><code>:443</code> / <code>:80</code></td><td>Listen addresses.</td></tr> |
| + <tr><td><code>LCHAT_FALLBACK_URL</code></td><td>unset (404)</td><td>Where unknown hostnames are redirected.</td></tr> |
| + <tr><td><code>LCHAT_WILDCARD_DOMAINS</code>, <code>LCHAT_DNSIMPLE_TOKEN</code></td><td>unset</td><td>Wildcard certificate issuance, described <a href="#dns01">above</a>.</td></tr> |
| + <tr><td><code>LCHAT_HIBERNATE_BUCKET</code>, <code>LCHAT_HIBERNATE_REGION</code>, <code>LCHAT_HIBERNATE_PREFIX</code>, <code>LCHAT_HIBERNATE_IDLE</code></td><td>unset</td><td>The hibernation activator’s master switch and tuning — explained fully in <a href="hibernation.html#activator-config">part 06</a>. Unset means the activator does not exist.</td></tr> |
| + <tr><td><code>LCHAT_MESH_SELF</code>, <code>LCHAT_MESH_DOMAINS</code></td><td>unset</td><td>The mesh’s master switch — explained fully in <a href="mesh.html#mesh-config">part 07</a>. Unset means zero mesh code runs.</td></tr> |
| +</table> |
| +</div> |
| + |
| +<h2 id="checking">Checking on the router</h2> |
| +<figure class="cmd"> |
| +<figcaption>Run on the box — the three routine checks</figcaption> |
| +<pre><code># Is it running, and since when? |
| +systemctl status lchat-router |
| + |
| +# What has it logged recently? (-f follows live) |
| +journalctl -u lchat-router -n 100 |
| + |
| +# Does it serve a known host end to end? (checks TLS, routing, and the instance) |
| +curl -s https://chat-a.example.com/api/version |
| +<span class="out">{"version":"a41c9f2","build":"1b4bd808"}</span></code></pre> |
| +</figure> |
| +<p>Because the activator runs inside the router, a hibernating fleet’s wake/sleep log lines also appear in <code>journalctl -u lchat-router</code> — that journal is the single most informative log on the box. <a href="operations.html#reading-logs">Part 08</a> catalogues the log lines worth knowing.</p> |
| + |
| +<div class="why"> |
| + <span class="label">Why it is this way</span> |
| + <p>The router is deliberately tiny and stateless: it holds no per-request state, its routing table is rebuilt from a file on disk, and systemd restarts it within a second if it crashes. It is a single point of failure for the box, but a much smaller and more stable one than any alternative — and every stateful, complicated thing (the instances) lives outside it in separate processes it merely connects to.</p> |
| +</div> |
| + |
| +</div> |
| +</main> |
| +</div> |
| +</body> |
| +</html> |
diff --git a/docs/planet-scale-guide/spaces.html b/docs/planet-scale-guide/spaces.html| new file mode 100644 |
| index 0000000..ce7b48c |
| --- /dev/null |
| +++ b/docs/planet-scale-guide/spaces.html |
| @@ -0,0 +1,156 @@ |
| +<!doctype html> |
| +<html lang="en"> |
| +<head> |
| +<meta charset="utf-8" /> |
| +<meta name="viewport" content="width=device-width, initial-scale=1" /> |
| +<title>Creating and removing chat spaces — Running the fleet</title> |
| +<link rel="stylesheet" href="guide.css" /> |
| +<script src="guide.js" defer></script> |
| +</head> |
| +<body data-page="spaces.html"> |
| +<div class="frame"> |
| +<nav class="side"></nav> |
| +<main class="content"> |
| +<div class="wrap"> |
| + |
| +<header class="top"> |
| + <p class="eyebrow"><span class="robot">🤖</span> Running the fleet · Part 05 of 09</p> |
| + <h1>Creating and removing chat spaces</h1> |
| + <p class="thesis">A new chat space is a hostname, a minted admin token, an environment file, a registry row, and (usually) a started process. This page walks the manual commands first — <strong><code>add-instance</code> and <code>remove</code></strong> — and then the self-serve pipeline that lets signed-in users create their own spaces: <strong>the control daemon, the cloud provisioner, and the reaper</strong> that quietly reclaims spaces nobody ever claimed.</p> |
| +</header> |
| + |
| +<div class="sysmap" data-highlight="control"></div> |
| +<p class="sysmap-caption">You are here: the control daemon and cloud provisioner. The manual commands on this page do the same work the control daemon does, from your terminal instead of over its socket.</p> |
| + |
| +<h2 id="adding-manual">Adding a space by hand: <code>lchat add-instance</code></h2> |
| +<p>One command creates a fully working space. Before running it, the only prerequisite is DNS: the hostname must resolve to the box. With a wildcard DNS record (<code>*.example.com → the box’s address</code>) every subdomain already resolves, so there is no per-space DNS step at all.</p> |
| +<figure class="cmd"> |
| +<figcaption>Run on the box, as root</figcaption> |
| +<pre><code>lchat add-instance chat-c.example.com |
| +<span class="out">setup link: https://chat-c.example.com/#setup=Xk3… (one-time; give this to the space’s owner) |
| +admin link: https://chat-c.example.com/admin#token=9f2…</span></code></pre> |
| +</figure> |
| +<p>What just happened, in order — this sequence is worth reading once carefully, because <code>remove</code>, hibernation, and migration all make sense in terms of it:</p> |
| +<ol> |
| + <li><strong>Validate and register.</strong> The hostname is checked (it becomes a filename, a unit name, and a socket path, so it must be a clean domain name), an admin token is minted, and a row is written to the <a href="router.html#the-registry">registry</a> with status <code>creating</code>.</li> |
| + <li><strong>Write the environment file.</strong> <code>/data/chat-c.example.com.env</code> (mode 0600) receives <code>LCHAT_ADMIN_TOKEN=…</code> plus any per-instance settings passed as flags. Remember from <a href="processes.html#data">part 03</a>: the database will store only a <em>hash</em> of the admin token, so this file is the token’s only recoverable copy.</li> |
| + <li><strong>Point the version link.</strong> The currently running build is staged into the version store if it is not already there, and <code>/opt/lchat/live/chat-c.example.com</code> is created pointing at it — so the new instance starts on a known build (see <a href="deploys.html#mental-model">part 04</a>).</li> |
| + <li><strong>Start and health-check the process.</strong> <code>systemctl enable --now lchat@chat-c.example.com</code>, then wait up to fifteen seconds for the socket to answer. On failure the row is marked <code>failed</code> with the error recorded (visible in <code>lchat instances</code>), and the files are left in place for diagnosis.</li> |
| + <li><strong>Publish the route and warm the certificate.</strong> The registry row flips to <code>ready</code>, the router reloads, and the command makes one HTTPS request through the router with the real hostname — which triggers the Let’s Encrypt issuance (<a href="router.html#http01">part 02</a>) so the first member does not pay that wait.</li> |
| + <li><strong>Mint the setup invite.</strong> A one-time setup link is created (this also pre-seeds the space’s admin member). The setup link is what you hand to the person who asked for the space; the admin link is the operator-level console.</li> |
| +</ol> |
| +<p>Useful flags, all optional: <code>--admin-token <tok></code> imports an existing token instead of minting one (essential when <a href="operations.html#migration">migrating</a> a space between boxes, because a new token would invalidate every saved admin link); <code>--push-relay-url</code> / <code>--push-relay-token</code>, <code>--home-origin</code>, <code>--app-store-url</code> record per-instance service configuration into the environment file; and <code>--hibernate</code> registers the space as activator-managed from birth (<a href="#hibernating-creates">below</a>).</p> |
| +<div class="warn"> |
| + <span class="label">Store the admin token somewhere safe, immediately</span> |
| + <p>Treat the printed admin link like a password: store it in your secrets manager as soon as the command returns. If it is lost, <code>lchat admin-link chat-c.example.com</code> recovers it from the environment file — but only while that file exists on that box.</p> |
| +</div> |
| + |
| +<h2 id="removing">Removing a space: <code>lchat remove</code>, and why deletion archives</h2> |
| +<p>Removal is deliberately layered, because “stop serving this hostname” and “destroy this data” are different decisions:</p> |
| +<ol class="procedure"> |
| + <li><p class="step-title">Take the space off the internet (data untouched).</p> |
| + <figure class="cmd"> |
| + <figcaption>Run on the box, as root</figcaption> |
| + <pre><code>lchat remove chat-c.example.com --stop</code></pre> |
| + </figure> |
| + <p><code>remove</code> deletes the registry row and reloads the router, so the hostname stops being served; <code>--stop</code> also stops and disables the instance unit and its bot sidecar unit. Without <code>--stop</code>, the process keeps running but receives no traffic. Either way, <strong>the database, blobs, and environment file remain on disk</strong> — re-running <code>add-instance</code> for the same host (with <code>--admin-token</code> from the old environment file) brings the space straight back.</p></li> |
| + <li><p class="step-title">Only if the data should go: archive it into the trash.</p> |
| + <p>Full deletion — the path the self-serve pipeline uses — moves the database, environment file, write-ahead log, push key, and blob directory into <code>/data/trash/<host>-<timestamp>/</code>, and removes the certificate and version link. Nothing in this system unlinks user data directly; the trash directory is the safety net, and a mistaken deletion is recovered by moving the files back and re-running <code>add-instance</code>.</p></li> |
| + <li><p class="step-title">Eventually, the trash is purged (only if configured).</p> |
| + <p>The control daemon sweeps <code>/data/trash</code> daily and deletes archives older than <code>LCHAT_TRASH_KEEP_DAYS</code> days. The default is <code>0</code>, which means <strong>keep forever</strong> — purging is opt-in, and the retention window should comfortably exceed the reaper’s thresholds below so that any automated reclaim stays reversible for weeks.</p></li> |
| +</ol> |
| +<div class="plain"> |
| + <span class="label">In plain words</span> |
| + <p>Removing a space is like taking a folder out of the filing cabinet and putting it in a banker’s box in the basement, with the date written on it. Only a separately configured janitor ever shreds the boxes, and by default there is no janitor.</p> |
| +</div> |
| +<p>One deliberate exception: for a <a href="hibernation.html">hibernating</a> space, the parked copy of its database in the S3 bucket is <em>not</em> deleted by the deletion path — the box’s control daemon holds no bucket credentials on purpose. Cleaning old parked prefixes is left to the bucket’s lifecycle rules.</p> |
| + |
| +<h2 id="self-serve">The self-serve pipeline: how a member creates a space with no operator</h2> |
| +<p>Members of the product can create new spaces themselves (“Create a space” in the app). No human runs <code>add-instance</code> for those — instead a request flows through three services, each with a narrow job, and understanding the chain makes the failure modes obvious:</p> |
| +<figure class="diagram"> |
| +<svg viewBox="0 0 740 150" role="img" aria-label="Diagram: a member's device calls the home server, which calls the cloud provisioner, which calls the control daemon, which creates the instance."> |
| + <rect x="16" y="50" width="130" height="52" rx="7" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| + <text x="81" y="72" font-size="11.5" font-weight="600" text-anchor="middle">member’s device</text> |
| + <text x="81" y="88" font-size="9.5" fill="var(--muted)" text-anchor="middle">signed in to home</text> |
| + <rect x="196" y="50" width="130" height="52" rx="7" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| + <text x="261" y="72" font-size="11.5" font-weight="600" text-anchor="middle">home server</text> |
| + <text x="261" y="88" font-size="9.5" fill="var(--muted)" text-anchor="middle">eligibility gate</text> |
| + <rect x="376" y="50" width="130" height="52" rx="7" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| + <text x="441" y="72" font-size="11.5" font-weight="600" text-anchor="middle">cloud provisioner</text> |
| + <text x="441" y="88" font-size="9.5" fill="var(--muted)" text-anchor="middle">policy: quotas, names</text> |
| + <rect x="556" y="50" width="130" height="52" rx="7" fill="var(--diagram-box)" stroke="var(--rule)"></rect> |
| + <text x="621" y="72" font-size="11.5" font-weight="600" text-anchor="middle">control daemon</text> |
| + <text x="621" y="88" font-size="9.5" fill="var(--muted)" text-anchor="middle">capacity + creation</text> |
| + <line x1="146" y1="76" x2="196" y2="76" stroke="var(--muted)" stroke-width="1.2"></line> |
| + <line x1="326" y1="76" x2="376" y2="76" stroke="var(--muted)" stroke-width="1.2"></line> |
| + <line x1="506" y1="76" x2="556" y2="76" stroke="var(--muted)" stroke-width="1.2"></line> |
| + <text x="171" y="66" font-size="9" fill="var(--muted)" text-anchor="middle">HTTPS</text> |
| + <text x="351" y="66" font-size="9" fill="var(--muted)" text-anchor="middle">caller token</text> |
| + <text x="531" y="66" font-size="9" fill="var(--muted)" text-anchor="middle">root-only token,</text> |
| + <text x="531" y="76" font-size="9" fill="var(--muted)" text-anchor="middle">local socket</text> |
| + <text x="370" y="130" font-size="10" fill="var(--muted)" text-anchor="middle">Each hop narrows trust: the internet-facing services hold policy; only the local, root-only daemon can actually change the box.</text> |
| +</svg> |
| +</figure> |
| + |
| +<h3 id="controld">The control daemon (<code>lchat controld</code>): the only process allowed to change the box</h3> |
| +<p>The control daemon is the “god surface”: it is the one process permitted to run <code>systemctl</code> and write the software store, which is why it runs as root, listens <strong>only on a local unix socket</strong> (<code>/run/lchat/control.sock</code>), and is never given a public hostname. Its API is four endpoints — create an instance, delete an instance, report a space’s statistics, set a space’s storage quota — and every call must present a bearer token whose only copy lives in the root-only file <code>/etc/lchat/control.token</code>. (Socket file permissions alone cannot be the gate, because every sandboxed service on the box shares the same <code>lchat</code> user; the root-only token is what distinguishes the cloud provisioner, which is handed the token by systemd’s <code>LoadCredential</code> mechanism, from a hypothetically compromised instance, which cannot read it.)</p> |
| +<p>Its create endpoint enforces <strong>capacity</strong> before doing anything: at most <code>LCHAT_MAX_INSTANCES</code> running instances (default 40), and at least <code>LCHAT_MIN_FREE_GB</code> of free disk (default 5). At capacity it answers <code>503 at_capacity</code> and the refusal propagates up the chain as a polite “try again later” to the member. Configure these in <code>/data/control.env</code> and restart <code>lchat-control</code>.</p> |
| + |
| +<h3 id="cloud">The cloud provisioner (<code>lchat cloud</code>): policy, in public</h3> |
| +<p>The cloud provisioner is the internet-facing half, running unprivileged behind the router like any other service (registered with <code>lchat add-service</code>). It holds everything that is <em>policy</em> rather than <em>power</em>:</p> |
| +<ul> |
| + <li><strong>Caller tokens.</strong> Services that may request creations (in practice: the home server) authenticate with tokens the operator mints at runtime — <code>POST /api/admin/tokens</code> with the provisioner’s admin token; tokens are stored hashed and are revocable individually.</li> |
| + <li><strong>Quotas.</strong> Each requesting user may hold at most <code>LCHAT_CLOUD_QUOTA</code> (default 3) <em>free</em> spaces at a time; a deleted space frees its slot, and a paid space does not count. There is also an optional fleet-wide circuit breaker, <code>LCHAT_CLOUD_MAX_CREATES_PER_DAY</code>, whose count deliberately survives create-and-delete games.</li> |
| + <li><strong>Names.</strong> New spaces get short subdomains like <code>k3f.example.com</code> — an encoding (sqids) of the space’s ledger row number, so names are unique by construction, never reused, and carry no information. The base domain comes from <code>LCHAT_CLOUD_BASE_DOMAIN</code> and needs the wildcard DNS record.</li> |
| + <li><strong>The ledger.</strong> Its own small database (<code>/data/cloud.db</code>) records who created what, plus the dormant billing rails: an opaque Stripe customer pointer and an entitlement flag per space — deliberately no email, name, or card data anywhere. <span class="status partial">partially built</span> (dormant until the Stripe environment variables are set).</li> |
| +</ul> |
| +<p>Upstream of both sits the <strong>home server’s eligibility gate</strong>: it forwards a creation request only for a signed-in account that already holds at least one space backup, which keeps space creation invitation-shaped without the server reading anything encrypted. The member’s identity crosses to the provisioner only as an opaque random identifier — the quota is per that identifier, and no personal information flows down the chain.</p> |
| + |
| +<h3 id="enabling-self-serve">Enabling the pipeline on a box, once</h3> |
| +<ol class="procedure"> |
| + <li><p class="step-title">Start the control daemon.</p> |
| + <figure class="cmd"><figcaption>Run on the box, as root</figcaption> |
| + <pre><code>systemctl enable --now lchat-control # the token at /etc/lchat/control.token was minted by bootstrap</code></pre></figure></li> |
| + <li><p class="step-title">Configure and start the cloud provisioner.</p> |
| + <figure class="cmd"><figcaption>Run on the box, as root — /data/cloud.env, then start</figcaption> |
| + <pre><code>cat > /data/cloud.env <<'EOF' |
| +LCHAT_ADMIN_TOKEN=<a fresh secret for the provisioner’s own admin API> |
| +LCHAT_CLOUD_BASE_DOMAIN=example.com |
| +LCHAT_CLOUD_QUOTA=3 |
| +EOF |
| +chown lchat:lchat /data/cloud.env && chmod 600 /data/cloud.env |
| +systemctl enable --now lchat-cloud</code></pre></figure></li> |
| + <li><p class="step-title">Give the provisioner a public hostname through the router.</p> |
| + <figure class="cmd"><figcaption>Run on the box, as root</figcaption> |
| + <pre><code>lchat add-service cloud.example.com --socket /run/lchat/cloud.sock</code></pre></figure></li> |
| + <li><p class="step-title">Mint a caller token and hand it to the home server.</p> |
| + <figure class="cmd"><figcaption>Run on the box</figcaption> |
| + <pre><code>curl -s -X POST https://cloud.example.com/api/admin/tokens \ |
| + -H "Authorization: Bearer <the provisioner admin token>" -d '{"label":"home"}' |
| +<span class="out">{"id":1,"token":"<shown exactly once — copy it now>"}</span> |
| +# then add to /data/home.env: LCHAT_CLOUD_URL=https://cloud.example.com |
| +# LCHAT_CLOUD_TOKEN=<the minted token> |
| +systemctl restart lchat-home</code></pre></figure></li> |
| +</ol> |
| + |
| +<h2 id="reaper">The reaper: reclaiming spaces nobody ever claimed</h2> |
| +<p>Self-serve creation produces abandonment: someone creates a space, never finishes setting it up, and its slot counts against capacity forever. The reaper is the cloud provisioner’s answer, and it is engineered to be terrified of false positives:</p> |
| +<ul> |
| + <li><strong>Only ledger spaces are candidates.</strong> It considers only spaces the self-serve pipeline itself created (rows in <code>/data/cloud.db</code>) — operator-created instances, the home server, and the push relay are structurally not candidates. Paid spaces are excluded at the query level.</li> |
| + <li><strong>“Never claimed” requires three signals to agree.</strong> A space is reclaimed only if its setup was never completed, <em>and</em> it has at most its one pre-seeded admin member, <em>and</em> it contains no messages or threads at all. Any error reading a space’s statistics is a skip, never a reap. (A hibernating space that has never even woken has no database file; it is answered definitionally from its registry row rather than being treated as an error — otherwise register-only spaces could never be reclaimed.)</li> |
| + <li><strong>It is off by default and bounded when on.</strong> Enable by setting <code>LCHAT_CLOUD_REAP_AFTER_DAYS</code> (choose a value comfortably past the seven-day life of the setup invite — fourteen is sensible); it sweeps every <code>LCHAT_CLOUD_REAP_INTERVAL_HOURS</code> (default 6), reclaims at most <code>LCHAT_CLOUD_REAP_MAX_PER_SWEEP</code> (default 5) spaces per sweep, and honours <code>LCHAT_CLOUD_REAP_DRY_RUN=1</code>, which logs what it <em>would</em> reclaim. Run the dry run for a few days before trusting it.</li> |
| + <li><strong>Reaping is the archiving deletion</strong> described <a href="#removing">above</a> — into <code>/data/trash</code>, recoverable until the (separately configured, default-never) trash purge.</li> |
| +</ul> |
| + |
| +<h2 id="hibernating-creates">Spaces born asleep</h2> |
| +<p>On a box with <a href="hibernation.html">hibernation</a> enabled, creation can skip starting a process at all: <code>add-instance --hibernate</code> (or, for the whole self-serve pipeline, <code>LCHAT_CONTROL_HIBERNATE_CREATES=1</code> in <code>/data/control.env</code>) registers the space as activator-managed from birth. No systemd unit is created and no process runs; the first request to the hostname performs the first wake, which finds no parked database in the bucket and simply starts fresh. Capacity accounting changes accordingly: hibernating registrations are bounded by a row ceiling (<code>LCHAT_MAX_REGISTERED</code>, default ten times <code>LCHAT_MAX_INSTANCES</code>) rather than the running-process cap, because a sleeping space costs disk, not memory.</p> |
| +<div class="roadmap"> |
| + <span class="label">Partially built: creation is cheap, not yet free</span> |
| + <p>The design’s end state is that creating a space writes <em>only</em> a registry row and an object-storage prefix — microseconds, no files, with everything else materialising at first wake. Today’s hibernating create is close but honest about the gap: it still writes the environment file, stages the binary, reloads the router, and performs one immediate wake, because minting the setup invite needs a running instance to pre-seed the admin member. Fully deferred materialisation remains roadmap, as does the eventual dissolution of the control daemon into the registry-only model.</p> |
| +</div> |
| + |
| +</div> |
| +</main> |
| +</div> |
| +</body> |
| +</html> |