eleven / ocho Public

docs: planet-scale guide catches up to the live mesh (#872)

The guide still described the mesh as dormant (no box sets LCHAT_MESH_SELF),
registry replication as reverted-for-rework and living only on side
branches, and fleet-wide status aggregation as an unmerged branch. All three
shipped and are now live in production across two boxes.

- mesh.html: "dormant" banner replaced with a live-status one; new
  LCHAT_MESH_TOKEN env var documented; new "Route replication" section
  covering automatic fan-out + `lchat mesh-seed`, including the
  materialize-via-controld gotcha that caused the first live flip's
  outage; "adding a box" procedure rewritten around mesh-seed instead of
  manual per-host registration; honest-limits section trimmed to the two
  gaps that genuinely remain (route mutations don't replicate; cert cache
  is still per-box).
- index.html: mesh status badge promoted partial -> built.
- operations.html: observability section describes the real multi-host
  aggregation (per-box ownership partitioning, LCHAT_BOX_REGION) instead
  of pointing at an unmerged branch.
- reference.html: `lchat mesh-seed`, LCHAT_MESH_TOKEN, LCHAT_BOX_REGION
  added to the command/env-var tables.
- status.md: the "hosts v1 is the local box alone" sentence replaced with
  how multi-host aggregation actually works; region env vars disambiguated
  (STATUS_BOX_REGION for the local box vs. LCHAT_BOX_REGION for a peer).
- planet-scale.md: decision log #7 notes the live production date; the
  materialize-via-controld incident added to "corrections the build
  taught us."

Co-authored-by: Claude <noreply@anthropic.com>

James Adam 3 months ago 334ededb52b83e9d27fd9b8f21f0cf68c54cb541 parent 8a833cc

6 files changed, +71 −25

docs/planet-scale-guide/index.html +1 −1
docs/planet-scale-guide/mesh.html +35 −13
docs/planet-scale-guide/operations.html +2 −2
docs/planet-scale-guide/reference.html +4 −0
docs/planet-scale.md +8 −3
docs/status.md +21 −6

diff --git a/docs/planet-scale-guide/index.html b/docs/planet-scale-guide/index.html

index 7383826..9adaaad 100644

--- a/docs/planet-scale-guide/index.html

+++ b/docs/planet-scale-guide/index.html

@@ -94,7 +94,7 @@

<tr><td>The router, registry, per-instance units, versioned deploys, release channels, auto-updates</td><td><span class="status built">running today</span></td><td>The whole of <a href="router.html">parts 02</a>–<a href="deploys.html">04</a>.</td></tr>

<tr><td>Self-serve space creation (control daemon + cloud provisioner) and the reaper</td><td><span class="status built">running today</span></td><td>The whole of <a href="spaces.html">part 05</a>.</td></tr>

<tr><td>Instance hibernation (activator, lease, replication to S3, scheduled wakes, sidecar co-spawn)</td><td><span class="status built">running today</span></td><td>The whole of <a href="hibernation.html">part 06</a>. Opt-in per host.</td></tr>

- <tr><td>The mesh (multiple boxes, S3 heartbeats, hash placement, proxy to owner)</td><td><span class="status partial">partially built</span></td><td><a href="mesh.html">Part 07</a>. The placement code is complete and tested, but it is <em>dormant in production</em> (no box sets <code>LCHAT_MESH_SELF</code> yet), and one prerequisite — copying routing entries between boxes — was withdrawn for rework and is not on the main branch. See <a href="mesh.html#mesh-limits">the honest limits section</a>.</td></tr>

+ <tr><td>The mesh (multiple boxes, S3 heartbeats, hash placement, proxy to owner, route replication, fleet-wide status)</td><td><span class="status built">running today</span></td><td><a href="mesh.html">Part 07</a>. Production runs two boxes, both mesh members, splitting a scoped domain set. Route mutations after creation (renames, removals, hibernate flips) still don’t replicate — see <a href="mesh.html#mesh-limits">the honest limits section</a>.</td></tr>

<tr><td>Fleet backup (continuous replication of every database to a backup bucket)</td><td><span class="status built">running today</span></td><td><a href="operations.html#backups">Part 08</a>.</td></tr>

<tr><td>Fully deferred space creation (a create becomes only a registry row, no files touched)</td><td><span class="status partial">partially built</span></td><td>Creation is split into register + materialize steps, and hibernating creates skip the systemd half — but they still write the env file, stage the binary, and wake once to mint the setup invite. See <a href="spaces.html#hibernating-creates">part 05</a>.</td></tr>

<tr><td>Attachments in an object store (instead of on the box’s disk)</td><td><span class="status partial">partially built</span></td><td>The storage interface (<code>blobstore.go</code>) exists; only the on-disk implementation is real. The object-store implementation is roadmap.</td></tr>

diff --git a/docs/planet-scale-guide/mesh.html b/docs/planet-scale-guide/mesh.html

index e33626c..18e9de3 100644

--- a/docs/planet-scale-guide/mesh.html

+++ b/docs/planet-scale-guide/mesh.html

@@ -19,12 +19,12 @@

<p class="thesis">The mesh lets several boxes serve one fleet of hibernating spaces <strong>with no coordinator, no shared database, and no new network protocol between them</strong>: boxes announce themselves with heartbeat objects in the shared S3 bucket, a hashing rule decides which box should own which space, and the per-space lease guarantees correctness even when boxes disagree.</p>

</header>

-<div class="sysmap" data-highlight="peer bucket"></div>

+<div class="sysmap" data-highlight="peer bucket control"></div>

<p class="sysmap-caption">You are here: the peer boxes, and the bucket they coordinate through. Note that peers never need to discover each other directly — everything flows through the bucket they already share for hibernation.</p>

-<div class="warn">

- <span class="label">Read this first: the mesh is built, but dormant</span>

- <p>Be clear about what you are reading. The mesh code on this page is complete, merged, and covered by live kill-a-box tests — but as of this writing <strong>no production box enables it</strong> (every box runs with <code>LCHAT_MESH_SELF</code> unset, which means zero mesh code paths execute), and one prerequisite for convenient multi-box operation — automatically copying routing entries between boxes — was merged, then withdrawn for rework, and is <em>not</em> on the main branch. The <a href="#mesh-limits">honest limits</a> section at the end spells out exactly what that means for an operator today. Everything else on this page describes code as it exists and behaves when enabled.</p>

+<div class="plain">

+ <span class="label">Read this first: the mesh is live</span>

+ <p>Production runs two boxes today, both with <code>LCHAT_MESH_SELF</code> set, splitting a shared pool of hibernating spaces under a scoped set of domains (<code>LCHAT_MESH_DOMAINS</code>). Registry rows now replicate automatically between them (see <a href="#mesh-replication">route replication</a> below) — the manual step this section used to describe is gone. The <a href="#mesh-limits">honest limits</a> section at the end lists what genuinely remains unbuilt; everything else on this page describes code as it runs today.</p>

</div>

<h2 id="mesh-idea">The idea: ownership as a rule, correctness as a lock</h2>

@@ -40,18 +40,21 @@

</div>

<h2 id="mesh-config">Enabling the mesh on a box</h2>

-<p>Two environment variables in <code>/data/router.env</code>, on top of an already-configured hibernation bucket:</p>

+<p>Three environment variables in <code>/data/router.env</code>, on top of an already-configured hibernation bucket:</p>

<figcaption>Contents of /data/router.env (then: systemctl restart lchat-router)</figcaption>

<pre><code>LCHAT_HIBERNATE_BUCKET=my-hibernation-bucket # prerequisite: the mesh rides on the hibernation bucket

LCHAT_MESH_SELF=https://box-1.example.net # this box’s identity: a base URL its PEERS can reach it at

-LCHAT_MESH_DOMAINS=11ch.example.com # comma-separated; ONLY hosts under these domains participate</code></pre>

+LCHAT_MESH_DOMAINS=11ch.example.com # comma-separated; ONLY hosts under these domains participate

+LCHAT_MESH_TOKEN=<a shared secret, identical on every box> # authenticates route replication</code></pre>

</figure>

<dl>

<dt><code>LCHAT_MESH_SELF</code> — the master switch and the box’s identity</dt>

<dd>A URL with a scheme (<code>https</code> or <code>http</code>) and a host, no path. Its <em>presence</em> turns the mesh on; absent means a single box and zero mesh code. It must be reachable by the <em>other boxes</em> (it is what they proxy to), which may be a private network address (<code>http://10.0.0.5</code>) or a public one.</dd>

<dt><code>LCHAT_MESH_DOMAINS</code> — the scope, and it is required</dt>

- <dd>The router refuses to start with <code>LCHAT_MESH_SELF</code> set but no domains. Only hostnames equal to or directly under a listed domain participate in mesh placement; every other host is always served locally, exactly as without a mesh. Two reasons for the scoping, both practical: a routing entry for a host only exists on the boxes that have it in their registry (see <a href="#mesh-limits">the limits</a>), so placement must be confined to hosts every member can serve; and every mesh member must be able to obtain a TLS certificate for any host that might land on it, which per-host issuance cannot promise (it ties a certificate to whichever box the public DNS points at) but a shared <a href="router.html#dns01">wildcard certificate</a> can. In practice: <strong>mesh domains should be wildcard-certificate domains.</strong></dd>

+ <dd>The router refuses to start with <code>LCHAT_MESH_SELF</code> set but no domains. Only hostnames equal to or directly under a listed domain participate in mesh placement; every other host is always served locally, exactly as without a mesh. Two reasons for the scoping, both practical: a routing entry for a host only exists on the boxes it has been replicated to (see <a href="#mesh-replication">route replication</a>), so placement must be confined to hosts every member can serve; and every mesh member must be able to obtain a TLS certificate for any host that might land on it, which per-host issuance cannot promise (it ties a certificate to whichever box the public DNS points at) but a shared <a href="router.html#dns01">wildcard certificate</a> can. In practice: <strong>mesh domains should be wildcard-certificate domains.</strong></dd>

+ <dt><code>LCHAT_MESH_TOKEN</code> — the shared secret, and it is also required</dt>

+ <dd>The router refuses to start with <code>LCHAT_MESH_SELF</code> set but no token. A bearer secret, identical across every mesh member, that authenticates the route-replication endpoint (<a href="#mesh-replication">below</a>) — without it, any box that could reach another's mesh port could hand it arbitrary routing entries. Generate it once and copy it to every box’s <code>/data/router.env</code>; it is not derived from anything else and is not printed anywhere at runtime.</dd>

</dl>

<h2 id="membership">Membership: heartbeats in the bucket</h2>

@@ -79,6 +82,21 @@ echo aHR0cHM6Ly9ib3gtMS5leGFtcGxlLm5ldA | base64 -d

<h2 id="one-hop">Forwarding: wrong-box requests proxy to the owner, once</h2>

<p>When a request arrives at a box that is not the computed owner, the box reverse-proxies the entire request to the owner’s <code>LCHAT_MESH_SELF</code> URL — preserving the original hostname, verifying the owner’s TLS certificate against <em>the instance’s hostname</em> (this is why mesh domains want the shared wildcard certificate), and pinning the underlying connection to the peer’s address. There is no certificate-verification bypass anywhere. A forwarded request is marked with an internal header (<code>X-Lchat-Mesh-Forwarded</code>, stripped from all outside traffic), and a marked request is <strong>never forwarded again</strong>: if two boxes disagree about ownership, the second box serves the request locally — a wrong-box wake the lease then resolves — instead of the pair bouncing the request forever.</p>

+<h2 id="mesh-replication">Route replication: every box learns about every host</h2>

+<p>The router looks a request’s hostname up in its <strong>own local registry</strong> before any mesh logic runs at all, and answers <code>404</code> immediately on a miss. So placement (hashing, forwarding) is only half the story: a box can only ever serve — or even correctly forward — a host it has a registry row for. Two mechanisms keep every mesh member’s registry in sync, both authenticated with <code>LCHAT_MESH_TOKEN</code>:</p>

+<ul>

+ <li><strong>Automatic fan-out on creation.</strong> The moment a new in-scope hibernating host is registered anywhere, that box replicates the row to every <em>currently live</em> mesh peer (found the same way placement does — the membership heartbeats). Only a narrow, purpose-built payload crosses the wire — host, kind, version, hibernate flag, and a handful of routing-relevant fields — never the receiving box’s own socket path or database path, which each box always derives locally. A peer that is down at that moment simply misses the row; fan-out is best-effort, logged, and never blocks the create.</li>

+ <li><strong><code>lchat mesh-seed <peer-url></code> — the one-time catch-up.</strong> Run on an existing box and pointed at a newly joined one, this bulk-replicates every in-scope hibernating route the existing box already knows about, in one pass. This is the command that makes joining a box a single step instead of hand-registering every host (see <a href="#add-box">adding a box</a>, below).</li>

+</ul>

+<div class="why">

+ <span class="label">Why replicate rows instead of just answering for any host</span>

+ <p>The alternative — a box that does not recognise a hostname asking its peers before giving up — would need a new box-to-box RPC, adding a network dependency the S3-only design deliberately avoids everywhere else. Replicating the (small, non-secret) routing metadata ahead of time keeps every box able to answer a hit or a genuine 404 from its own disk, no peer round-trip required.</p>

+</div>

+<div class="warn">

+ <span class="label">A replicated route still needs materializing before its first wake</span>

+ <p>A route reaching a box through replication (or <code>mesh-seed</code>) is a registry row only — unlike <code>add-instance</code> on the box that originates it, replication never stages a binary or creates a <code>live</code> symlink for that host. The router process is sandboxed (<code>ProtectSystem=strict</code>) and cannot write to <code>/opt/lchat</code> itself, so its wake path materializes a first-time route by asking the unsandboxed <a href="spaces.html#controld">control daemon</a> to do it instead. This needs the router unit’s <code>LoadCredential=control-token:/etc/lchat/control.token</code> drop-in (<code>deploy/systemd/lchat-router-control-token.conf</code>) — without it, the router logs a warning at startup and a replicated route’s <em>first</em> wake on that box fails outright (a live production incident the first time this was tried; see <code>docs/planet-scale.md</code>’s decision log for the full account). Every mesh member needs this drop-in installed.</p>

+</div>

<h2 id="box-lifecycle">Adding, draining, and losing a box</h2>

<h3 id="add-box">Adding a box to the mesh</h3>

@@ -86,14 +104,19 @@ echo aHR0cHM6Ly9ib3gtMS5leGFtcGxlLm5ldA | base64 -d

<p>Install the binary and run <code>lchat bootstrap</code>, exactly as in <a href="deploys.html#bootstrap">part 04</a>. Give it the <em>same</em> hibernation bucket, prefix, and credentials as its peers in <code>/data/router.env</code> — the shared bucket is the entire coordination fabric.</p></li>

<li><p class="step-title">Give it the shared wildcard certificate capability.</p>

<p>Configure <code>LCHAT_WILDCARD_DOMAINS</code> (and the DNS API token) for the mesh domains, so this box can present valid certificates for any space that lands on it.</p></li>

- <li><p class="step-title">Ensure it has registry rows for the mesh-scoped hosts.</p>

- <p>This is the manual step that automation was supposed to remove (see <a href="#mesh-limits">the limits</a>): a box can only serve hosts present in its own registry. Today that means registering the hibernating hosts on the new box yourself — for each host, <code>lchat add-instance <host> --hibernate --admin-token <the host’s existing token></code>, reusing the token from the host’s environment file on the original box so nothing is re-minted.</p></li>

+ <li><p class="step-title">Install the control-token drop-in.</p>

+ <p>Copy <code>deploy/systemd/lchat-router-control-token.conf</code> in alongside <code>lchat-router.service</code> so the router can materialize a replicated route’s first wake through <a href="spaces.html#controld">controld</a> — see the warning above. Skipping this is the single most common way a fresh mesh box looks fine (heartbeats, membership) until the first real request lands on it.</p></li>

<li><p class="step-title">Set its mesh identity and restart the router.</p>

<figure class="cmd"><figcaption>Run on the new box — /data/router.env additions, then restart</figcaption>

<pre><code>LCHAT_MESH_SELF=https://box-2.example.net

LCHAT_MESH_DOMAINS=11ch.example.com

+LCHAT_MESH_TOKEN=<the same shared secret as every other box>

# then: systemctl restart lchat-router</code></pre></figure>

- <p>Within one heartbeat interval the peers see the new member, and roughly <em>1/N</em> of the in-scope spaces re-home to it at their next sleep/wake boundary. Nothing needs announcing; the guest book does it.</p></li>

+ <p>Within one heartbeat interval the peers see the new member.</p></li>

+ <li><p class="step-title">Seed its registry from an existing box.</p>

+ <figure class="cmd"><figcaption>Run on an EXISTING mesh box, pointed at the new one</figcaption>

+ <pre><code>lchat mesh-seed https://box-2.example.net</code></pre></figure>

+ <p>Bulk-replicates every in-scope hibernating route in one pass, so the new box can immediately answer for (or correctly proxy) any of them. After this, roughly <em>1/N</em> of the in-scope spaces re-home to the new box at their next sleep/wake boundary — nothing needs announcing beyond this one command; the guest book and the registry rows do the rest.</p></li>

</ol>

<h3 id="drain-box">Draining a box (deploys, maintenance, scale-down)</h3>

<p>Stopping the router <em>is</em> the drain: its shutdown sequence <strong>deletes this box’s heartbeat first</strong> — so peers take ownership immediately — and then checkpoints every awake space to the bucket and releases its leases as usual. Deploying a new router build, retiring a box, and reacting to failure are deliberately the same code path.</p>

@@ -104,11 +127,10 @@ LCHAT_MESH_DOMAINS=11ch.example.com

<p>The recovery numbers, verified by live tests in both directions: after a <em>graceful</em> drain, the survivor is serving the drained box’s spaces about <strong>two seconds</strong> later; after a <em>hard kill</em> (no drain, heartbeat left to rot), within about <strong>the lease lifetime</strong> — thirty seconds when those tests ran, <strong>fifteen minutes</strong> since #836 stretched the lease to tame its S3 request bill — with data intact in both cases (up to the space’s <a href="hibernation.html#replication-classes">replication class</a>’s window), because writes had been syncing to the bucket all along.</p>

<h2 id="mesh-limits">The honest limits: what stands between here and a true multi-box fleet</h2>

-<p>Three gaps, in decreasing order of consequence. None of them is speculative — each corresponds to a concrete piece of code that exists, is reserved, or was withdrawn:</p>

+<p>Two gaps remain. Neither is speculative — each corresponds to a concrete piece of code that is reserved or not yet written. (Two things this section used to list — registry replication and fleet-wide status aggregation — have since shipped and are covered above and in <a href="operations.html#observability">part 08</a>.)</p>

<ul>

- <li><span class="status planned">not on main</span> <strong>Registries do not replicate.</strong> Each box’s registry is its own SQLite file, and a request for a host the box has no row for is answered 404 <em>before</em> any mesh logic runs. A change that fanned new routes out to every live peer (plus a <code>mesh-seed</code> command for a joining box’s initial catch-up, authenticated by a new shared secret) was merged and then <strong>reverted for rework</strong>; it currently lives on side branches only. Until it, or a successor, lands: keeping registries aligned across mesh boxes is <em>manual</em> (step 3 of “adding a box” above), and route changes after creation — renames, removals, hibernation flips — must be repeated on every box by hand.</li>

+ <li><span class="status partial">partially built</span> <strong>Route mutations after creation do not replicate.</strong> The initial fan-out (<a href="#mesh-replication">above</a>) and <code>mesh-seed</code> cover creation and joining, but a later rename, removal, or hibernation-flag flip on one box is not currently propagated to its peers — those must still be repeated by hand on every mesh member. Fine for a pool of auto-provisioned, rarely-renamed spaces; worth closing before mesh placement carries more than that.</li>

<li><span class="status partial">partially built</span> <strong>The certificate cache is per box.</strong> The wildcard certificate makes mesh domains workable (each box independently obtains its own copy), but per-host certificates for domains outside the wildcard cannot follow a space between boxes; the S3-backed shared cache that would fix this is reserved (<code>LCHAT_CERT_CACHE=s3</code>) and deliberately refuses to start. See <a href="router.html#tls">part 02</a>.</li>

- <li><span class="status planned">not on main</span> <strong>Fleet-wide status aggregation.</strong> The public fleet dashboard reports the local box only; the change that lets each mesh member publish its counts as a sibling object next to its heartbeat, for the status publisher to merge, sits on an unmerged branch (and its bucket-permission change has not been applied). See <a href="operations.html#observability">part 08</a>.</li>

</ul>

<span class="label">Designed, not built: what the mesh grows into</span>

diff --git a/docs/planet-scale-guide/operations.html b/docs/planet-scale-guide/operations.html

index 2990de0..a88f9c3 100644

--- a/docs/planet-scale-guide/operations.html

+++ b/docs/planet-scale-guide/operations.html

@@ -124,11 +124,11 @@ sqlite3 /tmp/chat-a-snapshot.db "PRAGMA integrity_check"

</ul>

<h2 id="observability">What you can observe, and what you cannot yet</h2>

-<p>A status publisher on the box regenerates a set of public dashboards every five minutes, including a fleet page whose data file (<code>fleet.json</code>) reports: total, systemd-managed, and hibernating instance counts; how many hibernators are awake versus asleep at this moment; the software version spread; up/down booleans for the sibling services; backup-replication health with its lag in seconds; and a coarse capacity band. The capacity band — <code>plenty</code>, <code>getting-full</code>, or <code>at-capacity</code> — is the worst of several internal signals (instance count against the cap, registered rows against the row ceiling, disk against the floor, memory), and deliberately <em>only the band word is published</em>: never counts of slots, gigabytes, percentages, hostnames, or addresses. The same discipline runs through everything public: aggregate numbers and band words, nothing that identifies a machine or a member.</p>

+<p>A status publisher on one box regenerates a set of public dashboards every five minutes, including a fleet page whose data file (<code>fleet.json</code>) reports: total, systemd-managed, and hibernating instance counts; how many hibernators are awake versus asleep <em>fleet-wide</em> at this moment; the software version spread; up/down booleans for the sibling services; backup-replication health with its lag in seconds; and a coarse capacity band. The capacity band — <code>plenty</code>, <code>getting-full</code>, or <code>at-capacity</code> — is the worst of several internal signals (instance count against the cap, registered rows against the row ceiling, disk against the floor, memory), and deliberately <em>only the band word is published</em>: never counts of slots, gigabytes, percentages, hostnames, or addresses. The same discipline runs through everything public: aggregate numbers and band words, nothing that identifies a machine or a member.</p>

+<p>In a <a href="mesh.html">mesh</a>, the dashboard aggregates across every live box, not just the one running the publisher. Each mesh member — including that one — publishes its own capacity snapshot as a sibling object next to its membership heartbeat, on a slower (one-minute) cadence; the publisher reads every live peer’s snapshot back over the same S3 substrate the mesh already uses, no new box-to-box port involved. Each box’s own reported awake/asleep count is filtered to the hosts it currently owns by the mesh’s <a href="mesh.html#placement">placement hash</a> — so a host that has been re-homed to a peer is never reported as “asleep” by more than one box at once, and the fleet page’s <code>hosts</code> array lists every box exactly once, each with its own whimsical, hash-derived label (never a hostname or address) and, optionally, an operator-set coarse region (<code>LCHAT_BOX_REGION</code>, “EU · Finland”-shaped — continent and country only).</p>

<span class="label">Designed, not built: real metrics</span>

<p>Wake latency, lease conflicts, and replication lag exist today as <em>log lines</em>, not as queryable metrics with alerting. The design owes an end-to-end-encryption-safe metrics discipline; until it lands, trend questions (“are wakes getting slower?”) are answered by reading journals, and the five-minute dashboard is the only aggregation. Budget your expectations accordingly.</p>

- <p>Likewise, in a mesh the dashboard currently reports <em>the local box only</em> — the multi-box aggregation exists on an unmerged branch (<a href="mesh.html#mesh-limits">part 07</a>).</p>

</div>

<h2 id="troubleshooting">Common problems and their diagnoses</h2>

diff --git a/docs/planet-scale-guide/reference.html b/docs/planet-scale-guide/reference.html

index c4253c4..2822ac7 100644

--- a/docs/planet-scale-guide/reference.html

+++ b/docs/planet-scale-guide/reference.html

@@ -43,6 +43,8 @@

<dd>Reads (no argument) or sets a hibernating host’s <a href="hibernation.html#replication-classes">replication class</a> — how often awake writes sync to the bucket. A set reloads the router and applies at the host’s next wake; on provisioned fleets billing drives this instead. Explained in <a href="hibernation.html#replication-classes">part 06</a>.</dd>

<dt><code>lchat instances</code></dt>

<dd>Prints the registry: HOST, KIND, STATUS, VERSION, HIBERNATE, REPLICATION, SOCKET, plus a <code>last error</code> line for any failed row. The first stop for “what does this box think it is serving?”.</dd>

+ <dt><code>lchat mesh-seed <peer-url></code></dt>

+ <dd>Run on an existing mesh box, pointed at a newly joined one: bulk-replicates every in-scope hibernating route to the peer in one pass, authenticated by <code>LCHAT_MESH_TOKEN</code>. The one-time catch-up step of <a href="mesh.html#add-box">adding a box to the mesh</a>. <a href="mesh.html#mesh-replication">Part 07</a>.</dd>

<dt><code>lchat admin-link <host></code> / <code>lchat setup-invite <host></code></dt>

<dd>Recover the admin link (from the environment file — the only plaintext copy of the token) / mint a fresh one-time setup invite.</dd>

</dl>

@@ -95,6 +97,8 @@

<tr><td><code>LCHAT_HIBERNATE_IDLE</code></td><td><code>15m</code></td><td>Idle window before sleep (a duration such as <code>15m</code> or <code>90s</code>; must be positive).</td></tr>

<tr><td><code>LCHAT_MESH_SELF</code></td><td>unset (mesh off)</td><td>This box’s peer-reachable base URL — the mesh’s master switch. <a href="mesh.html#mesh-config">Part 07</a>.</td></tr>

<tr><td><code>LCHAT_MESH_DOMAINS</code></td><td>unset</td><td>Comma list of domains whose hosts participate in mesh placement. Required whenever <code>LCHAT_MESH_SELF</code> is set.</td></tr>

+ <tr><td><code>LCHAT_MESH_TOKEN</code></td><td>unset</td><td>Shared secret authenticating route replication between mesh members. Required whenever <code>LCHAT_MESH_SELF</code> is set; identical on every box. <a href="mesh.html#mesh-config">Part 07</a>.</td></tr>

+ <tr><td><code>LCHAT_BOX_REGION</code></td><td>unset (omitted)</td><td>This box’s coarse continent + country string, published to peers over the mesh stats channel for the Fleet dashboard’s <code>region</code> field (e.g. “EU · Finland”) — never a city, datacenter, IP, or hostname. <a href="operations.html#observability">Part 08</a>.</td></tr>

</table>

</div>

diff --git a/docs/planet-scale.md b/docs/planet-scale.md

index a5f0c10..52ddcb9 100644

--- a/docs/planet-scale.md

+++ b/docs/planet-scale.md

@@ -334,7 +334,7 @@ means the decision stands with nothing built against it yet.

| 6 | instance runtime | hibernating processes over S3: local SQLite + Litestream, lease = single-writer lock (over Lambda / S3-as-only-DB / cells / renting DO-Fly) | **Shipped** #630/#641 — with the correction: the lease is Go-library-only; the CLI silently absorbs a second writer, so the activator embeds litestream |

+| 7 | placement | rendezvous hash as hint, S3 lease as truth (over a routing table) | **Shipped** (#678) — S3-heartbeat membership + HRW + proxy-to-owner; kill-a-box passes live both ways. **Live in production 2026-07-13** across two boxes: domain-scoped placement (#820), automatic route replication + `mesh-seed` (#821), and fleet-wide status aggregation partitioned by ownership (#822/#849/#851/#852) |

| 10 | provisioning | create = register only; materialize at first wake; controld dissolves | Split **shipped** #625, hibernate-by-default creates #663; full deferral + controld dissolution roadmap |

@@ -353,8 +353,13 @@ finding (#6 above); env-file ownership must follow the reader (root-written

files, service-user activator — a wake died on it live); a never-woken

hibernating space has no DB, so reaper stats answer from the registry row or

-with the lifecycle as backstop, not a bucket guarantee; and `errBadVersion`

-is a SQLite-only nicety S3 cannot express.

+with the lifecycle as backstop, not a bucket guarantee; `errBadVersion` is a

+SQLite-only nicety S3 cannot express; and the first live mesh flip (#7,

+2026-07-13) taught that a replicated route is a registry row only — it never

+stages a binary or creates a `live` symlink, and the sandboxed router

+(`ProtectSystem=strict`) cannot do that itself, so a replicated route's first

+wake 503ed outright until the wake path was rerouted through the unsandboxed

+control daemon (#844) instead of writing `/opt/lchat` directly.

## The invariants — the review checklist

diff --git a/docs/status.md b/docs/status.md

index bc5b350..f806a88 100644

--- a/docs/status.md

+++ b/docs/status.md

@@ -86,8 +86,13 @@ aggregation boundary lives in the generator and holds by construction:

read internally to pick the word and die in the generator. Host labels are

whimsical name-action pairs hashed from the machine id (`ada-stargazing`) — stable per box; never peer URLs, IPs, hostnames, or the machine id itself.

A box's optional `region` is coarse by design: an operator-set continent +

- country pair (`STATUS_BOX_REGION`, e.g. `EU · Germany`) — never a city,

- datacenter, IP, or hostname; unset, the field is omitted (never guessed).

+ country pair (e.g. `EU · Germany`) — never a city, datacenter, IP, or

+ hostname; unset, the field is omitted (never guessed). The local box (the

+ one running the publisher) reads `STATUS_BOX_REGION`; a mesh peer publishes

+ its own via `LCHAT_BOX_REGION` (set in its own `/data/router.env`, read by

+ the router binary itself) — two different env vars on two different code

+ paths, since a peer's region rides the mesh stats side-channel, never the

+ publisher script.

- **Books: aggregated income only** (decision #19 in #550: income publishes

aggregated, never per-payer; invariant 12: no PII in our stores). Never

customer ids, emails, subscription ids, per-payer lines, or timestamps of

@@ -214,10 +219,20 @@ proximity to the `LCHAT_MIN_FREE_GB` floor (at or below the floor is

at-capacity outright), and memory utilization (`/proc/meminfo`). The caps are

read from `/data/control.env`. A box's `load` word bands memory utilization

the same way (`comfortable` / `busy` / `critical`, null when unknown). Only

-the words publish — never the ratios, GB, or caps behind them. `hosts` v1 is

-the local box alone; multi-box joins when the mesh goes live — per-box

-`region` will then ride the mesh heartbeats, with `STATUS_BOX_REGION` as the

-single-box seam until it does.

+the words publish — never the ratios, GB, or caps behind them. `hosts`

+aggregates across every live mesh box, not just the one running the

+publisher: each member self-publishes its own capacity snapshot as a sibling

+object next to its membership heartbeat (a slower, one-minute cadence — this

+is visibility, not coordination), and the publisher folds every live peer's

+snapshot into the array alongside its own. Each box's own awake/asleep count

+is filtered to the hosts it currently owns by the mesh's placement hash, so a

+host re-homed to a peer is never reported as "asleep" by more than one box —

+`instances.awake`/`asleep` at the top level sum every box's own count, which

+is exactly why that filtering matters. Single-box behavior (no live mesh

+peer) is unchanged: `hosts.count` stays `1`, byte-identical to before the

+mesh existed. Per-box `region` rides the same self-published snapshot

+(`LCHAT_BOX_REGION`, see the privacy rules above) — a peer with no region set

+simply omits the field, same as the local box's own `STATUS_BOX_REGION` seam.

`books.json` (or `{"available": false, "reason": "…"}` when billing isn't

configured / Stripe is unreachable):