| 1 | <!doctype html> |
| 2 | <html lang="en"> |
| 3 | <head> |
| 4 | <meta charset="utf-8" /> |
| 5 | <meta name="viewport" content="width=device-width, initial-scale=1" /> |
| 6 | <title>The mesh — Running the fleet</title> |
| 7 | <link rel="stylesheet" href="guide.css" /> |
| 8 | <script src="guide.js" defer></script> |
| 9 | </head> |
| 10 | <body data-page="mesh.html"> |
| 11 | <div class="frame"> |
| 12 | <nav class="side"></nav> |
| 13 | <main class="content"> |
| 14 | <div class="wrap"> |
| 15 | |
| 16 | <header class="top"> |
| 17 | <p class="eyebrow"><span class="robot">🤖</span> Running the fleet · Part 07 of 10</p> |
| 18 | <h1>The mesh: running more than one box</h1> |
| 19 | <p class="thesis">The mesh lets several boxes serve one fleet of hibernating spaces <strong>with no coordinator, no shared database, and no new network protocol between them</strong>: boxes announce themselves with heartbeat objects in the shared S3 bucket, a hashing rule decides which box should own which space, and the per-space lease guarantees correctness even when boxes disagree.</p> |
| 20 | </header> |
| 21 | |
| 22 | <div class="sysmap" data-highlight="peer bucket control"></div> |
| 23 | <p class="sysmap-caption">You are here: the peer boxes, and the bucket they coordinate through. Note that peers never need to discover each other directly — everything flows through the bucket they already share for hibernation.</p> |
| 24 | |
| 25 | <div class="plain"> |
| 26 | <span class="label">Read this first: the mesh is live</span> |
| 27 | <p>Production runs two boxes today, both with <code>LCHAT_MESH_SELF</code> set, splitting a shared pool of hibernating spaces under a scoped set of domains (<code>LCHAT_MESH_DOMAINS</code>). Registry rows now replicate automatically between them (see <a href="#mesh-replication">route replication</a> below) — the manual step this section used to describe is gone. The <a href="#mesh-limits">honest limits</a> section at the end lists what genuinely remains unbuilt; everything else on this page describes code as it runs today.</p> |
| 28 | </div> |
| 29 | |
| 30 | <h2 id="mesh-idea">The idea: ownership as a rule, correctness as a lock</h2> |
| 31 | <p>Hibernation (<a href="hibernation.html">part 06</a>) already moved every space’s durable state into the bucket, and made the per-space <a href="hibernation.html#the-lease">lease</a> the single arbiter of who may write it. That does most of the multi-box work by itself: any box <em>could</em> wake any space, and the lease already prevents two from doing so at once. What remains is efficiency and manners — spaces should not ping-pong between boxes, and a request that lands on the “wrong” box should still be answered. The mesh adds exactly three small mechanisms on top of the bucket:</p> |
| 32 | <ol> |
| 33 | <li><strong>Membership:</strong> each box periodically writes a tiny heartbeat object; listing them yields the set of live boxes.</li> |
| 34 | <li><strong>Placement:</strong> a deterministic hashing rule (“rendezvous hashing”) maps each hostname to one preferred owner among the live boxes.</li> |
| 35 | <li><strong>Forwarding:</strong> a box that receives a request for a space it does not own transparently proxies the request to the owner.</li> |
| 36 | </ol> |
| 37 | <div class="plain"> |
| 38 | <span class="label">In plain words</span> |
| 39 | <p>The boxes never talk to each other to decide who does what. Each box signs a guest book in the shared storage room every fifteen seconds, everyone applies the same arithmetic to the guest book to work out who should host which space, and the storage room’s key hooks (the leases) make it impossible for two boxes to act on a disagreement. A box that stops signing the book is presumed gone within forty-five seconds, and its spaces re-home themselves automatically.</p> |
| 40 | </div> |
| 41 | |
| 42 | <h2 id="mesh-config">Enabling the mesh on a box</h2> |
| 43 | <p>Three environment variables in <code>/data/router.env</code>, on top of an already-configured hibernation bucket:</p> |
| 44 | <figure class="cmd"> |
| 45 | <figcaption>Contents of /data/router.env (then: systemctl restart lchat-router)</figcaption> |
| 46 | <pre><code>LCHAT_HIBERNATE_BUCKET=my-hibernation-bucket # prerequisite: the mesh rides on the hibernation bucket |
| 47 | LCHAT_MESH_SELF=https://box-1.example.net # this box’s identity: a base URL its PEERS can reach it at |
| 48 | LCHAT_MESH_DOMAINS=11ch.example.com # comma-separated; ONLY hosts under these domains participate |
| 49 | LCHAT_MESH_TOKEN=<a shared secret, identical on every box> # authenticates route replication</code></pre> |
| 50 | </figure> |
| 51 | <dl> |
| 52 | <dt><code>LCHAT_MESH_SELF</code> — the master switch and the box’s identity</dt> |
| 53 | <dd>A URL with a scheme (<code>https</code> or <code>http</code>) and a host, no path. Its <em>presence</em> turns the mesh on; absent means a single box and zero mesh code. It must be reachable by the <em>other boxes</em> (it is what they proxy to), which may be a private network address (<code>http://10.0.0.5</code>) or a public one.</dd> |
| 54 | <dt><code>LCHAT_MESH_DOMAINS</code> — the scope, and it is required</dt> |
| 55 | <dd>The router refuses to start with <code>LCHAT_MESH_SELF</code> set but no domains. Only hostnames equal to or directly under a listed domain participate in mesh placement; every other host is always served locally, exactly as without a mesh. Two reasons for the scoping, both practical: a routing entry for a host only exists on the boxes it has been replicated to (see <a href="#mesh-replication">route replication</a>), so placement must be confined to hosts every member can serve; and every mesh member must be able to obtain a TLS certificate for any host that might land on it, which per-host issuance cannot promise (it ties a certificate to whichever box the public DNS points at) but a shared <a href="router.html#dns01">wildcard certificate</a> can. In practice: <strong>mesh domains should be wildcard-certificate domains.</strong></dd> |
| 56 | <dt><code>LCHAT_MESH_TOKEN</code> — the shared secret, and it is also required</dt> |
| 57 | <dd>The router refuses to start with <code>LCHAT_MESH_SELF</code> set but no token. A bearer secret, identical across every mesh member, that authenticates the route-replication endpoint (<a href="#mesh-replication">below</a>) — without it, any box that could reach another's mesh port could hand it arbitrary routing entries. Generate it once and copy it to every box’s <code>/data/router.env</code>; it is not derived from anything else and is not printed anywhere at runtime.</dd> |
| 58 | </dl> |
| 59 | |
| 60 | <h2 id="membership">Membership: heartbeats in the bucket</h2> |
| 61 | <p>Every fifteen seconds, each mesh box writes one object under <code><prefix>/mesh/members/</code> in the hibernation bucket and lists the others. The scheme has three properties worth appreciating:</p> |
| 62 | <ul> |
| 63 | <li><strong>The object’s key <em>is</em> the record</strong> — it is an encoding of the box’s own <code>LCHAT_MESH_SELF</code> URL — so a single cheap LIST operation yields the entire membership, with no per-member reads.</li> |
| 64 | <li><strong>Liveness is judged by the bucket’s clock</strong>, not the boxes’ clocks: freshness is the object’s last-modified time as recorded by S3 itself, so a box with a wrong clock cannot poison anyone’s view. A heartbeat older than forty-five seconds (three missed beats) marks its box dead.</li> |
| 65 | <li><strong>Failures are soft.</strong> A box that fails to write its beat is simply taken over by its peers (the lease arbitrates any overlap); a box that fails to <em>read</em> keeps its last known view and at worst misroutes — which, again, the lease absorbs. Neither failure is fatal, and both are logged.</li> |
| 66 | </ul> |
| 67 | <figure class="cmd"> |
| 68 | <figcaption>Run anywhere with bucket credentials — who is in the mesh right now?</figcaption> |
| 69 | <pre><code>aws s3 ls s3://my-hibernation-bucket/prod/mesh/members/ |
| 70 | <span class="out">2026-07-13 14:02:11 38 aHR0cHM6Ly9ib3gtMS5leGFtcGxlLm5ldA |
| 71 | 2026-07-13 14:02:09 38 aHR0cHM6Ly9ib3gtMi5leGFtcGxlLm5ldA</span> |
| 72 | # the timestamps are the liveness signal; the key decodes to the peer URL: |
| 73 | echo aHR0cHM6Ly9ib3gtMS5leGFtcGxlLm5ldA | base64 -d |
| 74 | <span class="out">https://box-1.example.net</span></code></pre> |
| 75 | </figure> |
| 76 | |
| 77 | <h2 id="placement">Placement: rendezvous hashing, and why the lease still decides</h2> |
| 78 | <p>For each in-scope hostname, every box independently computes a score for every live member (a hash of the pair “this box, this host”) and the highest score wins ownership. This is <strong>rendezvous hashing</strong>, and its virtue is minimal disruption: because each (box, host) pair’s score never changes, adding or removing one box only reassigns the hosts whose winning box appeared or vanished — roughly <em>1/N</em> of them — while everything else stays put. There is no table to distribute, no leader to elect: ownership is arithmetic on the guest book.</p> |
| 79 | <p>But membership views can momentarily disagree (one box has seen a fresh heartbeat the other has not). The design’s answer is that <strong>hashing is only a hint, and the lease is the truth</strong>: a box that wrongly believes it owns a space simply loses the lease race at wake time, loudly, and the requester retries onto the right answer. Transient disagreement structurally cannot corrupt anything — it can only cost a retry.</p> |
| 80 | <p>Two refinements keep behaviour calm: a space that is <em>currently awake</em> on some box stays there until its natural sleep, even if the hash now prefers another box — its state is already in the bucket, so “migration” is nothing more than waking on the new owner next time. And <a href="hibernation.html#due-time-wakes">scheduled wakes</a> fire only on the box that owns the host, so a due reminder wakes one box, not all of them.</p> |
| 81 | |
| 82 | <h2 id="one-hop">Forwarding: wrong-box requests proxy to the owner, once</h2> |
| 83 | <p>When a request arrives at a box that is not the computed owner, the box reverse-proxies the entire request to the owner’s <code>LCHAT_MESH_SELF</code> URL — preserving the original hostname, verifying the owner’s TLS certificate against <em>the instance’s hostname</em> (this is why mesh domains want the shared wildcard certificate), and pinning the underlying connection to the peer’s address. There is no certificate-verification bypass anywhere. A forwarded request is marked with an internal header (<code>X-Lchat-Mesh-Forwarded</code>, stripped from all outside traffic), and a marked request is <strong>never forwarded again</strong>: if two boxes disagree about ownership, the second box serves the request locally — a wrong-box wake the lease then resolves — instead of the pair bouncing the request forever.</p> |
| 84 | |
| 85 | <h2 id="mesh-replication">Route replication: every box learns about every host</h2> |
| 86 | <p>The router looks a request’s hostname up in its <strong>own local registry</strong> before any mesh logic runs at all, and answers <code>404</code> immediately on a miss. So placement (hashing, forwarding) is only half the story: a box can only ever serve — or even correctly forward — a host it has a registry row for. Two mechanisms keep every mesh member’s registry in sync, both authenticated with <code>LCHAT_MESH_TOKEN</code>:</p> |
| 87 | <ul> |
| 88 | <li><strong>Automatic fan-out on creation.</strong> The moment a new in-scope hibernating host is registered anywhere, that box replicates the row to every <em>currently live</em> mesh peer (found the same way placement does — the membership heartbeats). Only a narrow, purpose-built payload crosses the wire — host, kind, version, hibernate flag, a handful of routing-relevant fields, <em>and the instance’s own env values</em> (admin token, entitlement, seeded name) so a peer-woken instance boots with the token its creator minted — never the receiving box’s own socket path or database path, which each box always derives locally. A peer that is down at that moment simply misses the row; fan-out is best-effort, logged, and never blocks the create. On a region-scoped fleet the target set is a <em>union</em>: the boxes that cover the host by domain, <em>plus</em> any box that owns it by <code>home_region</code> — so a space placed into a region whose box sits outside the creating box’s domains still gets its row there. (Presenting a valid TLS certificate for such an out-of-domain host is a still-unbuilt follow-on; on the single-region production fleet the two sets coincide and nothing changes.) <strong>The daemon doing the creating must see the mesh env:</strong> <code>lchat-control.service</code> reads <code>/data/router.env</code> for exactly this (a controld without <code>LCHAT_MESH_SELF</code> silently skips fan-out, and any create HRW-placed on a peer is born dark — the peer answers <code>404 unknown host</code> for it; a live production incident, 2026-07-24). controld logs its replication posture at startup.</li> |
| 89 | <li><strong><code>lchat mesh-seed [--region <cc>] <peer-url></code> — the one-time catch-up.</strong> Run on an existing box and pointed at a newly joined one, this bulk-replicates every in-scope hibernating route the existing box already knows about, in one pass. This is the command that makes joining a box a single step instead of hand-registering every host (see <a href="#add-box">adding a box</a>, below).</li> |
| 90 | </ul> |
| 91 | <div class="why"> |
| 92 | <span class="label">Why replicate rows instead of just answering for any host</span> |
| 93 | <p>The alternative — a box that does not recognise a hostname asking its peers before giving up — would need a new box-to-box RPC, adding a network dependency the S3-only design deliberately avoids everywhere else. Replicating the (small, non-secret) routing metadata ahead of time keeps every box able to answer a hit or a genuine 404 from its own disk, no peer round-trip required.</p> |
| 94 | </div> |
| 95 | <div class="warn"> |
| 96 | <span class="label">A replicated route still needs materializing before its first wake</span> |
| 97 | <p>A route reaching a box through replication (or <code>mesh-seed</code>) is a registry row only — unlike <code>add-instance</code> on the box that originates it, replication never stages a binary or creates a <code>live</code> symlink for that host. The router process is sandboxed (<code>ProtectSystem=strict</code>) and cannot write to <code>/opt/lchat</code> itself, so its wake path materializes a first-time route by asking the unsandboxed <a href="spaces.html#controld">control daemon</a> to do it instead. This needs the router unit’s <code>LoadCredential=control-token:/etc/lchat/control.token</code> drop-in (<code>deploy/systemd/lchat-router-control-token.conf</code>) — without it, the router logs a warning at startup and a replicated route’s <em>first</em> wake on that box fails outright (a live production incident the first time this was tried; see <code>docs/planet-scale.md</code>’s decision log for the full account). Every mesh member needs this drop-in installed.</p> |
| 98 | </div> |
| 99 | |
| 100 | <h2 id="box-lifecycle">Adding, draining, and losing a box</h2> |
| 101 | <h3 id="add-box">Adding a box to the mesh</h3> |
| 102 | <ol class="procedure"> |
| 103 | <li><p class="step-title">Provision the box normally.</p> |
| 104 | <p>Install the binary and run <code>lchat bootstrap</code>, exactly as in <a href="deploys.html#bootstrap">part 04</a>. Give it the <em>same</em> hibernation bucket, prefix, and credentials as its peers in <code>/data/router.env</code> — the shared bucket is the entire coordination fabric.</p></li> |
| 105 | <li><p class="step-title">Give it the shared wildcard certificate capability.</p> |
| 106 | <p>Configure <code>LCHAT_WILDCARD_DOMAINS</code> (and the DNS API token) for the mesh domains, so this box can present valid certificates for any space that lands on it.</p></li> |
| 107 | <li><p class="step-title">Install the control-token drop-in.</p> |
| 108 | <p>Copy <code>deploy/systemd/lchat-router-control-token.conf</code> in alongside <code>lchat-router.service</code> so the router can materialize a replicated route’s first wake through <a href="spaces.html#controld">controld</a> — see the warning above. Skipping this is the single most common way a fresh mesh box looks fine (heartbeats, membership) until the first real request lands on it.</p></li> |
| 109 | <li><p class="step-title">Set its mesh identity and restart the router.</p> |
| 110 | <figure class="cmd"><figcaption>Run on the new box — /data/router.env additions, then restart</figcaption> |
| 111 | <pre><code>LCHAT_MESH_SELF=https://box-2.example.net |
| 112 | LCHAT_MESH_DOMAINS=11ch.example.com |
| 113 | LCHAT_MESH_TOKEN=<the same shared secret as every other box> |
| 114 | # then: systemctl restart lchat-router</code></pre></figure> |
| 115 | <p>Within one heartbeat interval the peers see the new member.</p></li> |
| 116 | <li><p class="step-title">Seed its registry from an existing box.</p> |
| 117 | <figure class="cmd"><figcaption>Run on an EXISTING mesh box, pointed at the new one</figcaption> |
| 118 | <pre><code>lchat mesh-seed --env-file /data/router.env https://box-2.example.net |
| 119 | # region-scoped fleet — also seed routes the new box owns by home_region: |
| 120 | lchat mesh-seed --env-file /data/router.env --region fi https://box-2.example.net</code></pre> |
| 121 | <p>(<code>--env-file</code> borrows the router’s <code>LCHAT_MESH_*</code> values — the seed can run under a sudo’d operator session that has no mesh env of its own. Add <code>--prune</code> to also tombstone peer rows this registry no longer has — the reconciliation half of deletion replication. The encoded operator path is <code>./deploy/ops mesh seed [--prune] <peer-url> [box]</code>.)</p></figure> |
| 122 | <p>Bulk-replicates every in-scope hibernating route in one pass, so the new box can immediately answer for (or correctly proxy) any of them. After this, roughly <em>1/N</em> of the in-scope spaces re-home to the new box at their next sleep/wake boundary — nothing needs announcing beyond this one command; the guest book and the registry rows do the rest. On a region-scoped fleet, pass <code>--region</code> with the new box’s <code>LCHAT_MESH_REGION</code> so any space pinned to that region — even one addressed outside the seed’s domains — is seeded to it as well.</p></li> |
| 123 | </ol> |
| 124 | <h3 id="drain-box">Draining a box (deploys, maintenance, scale-down)</h3> |
| 125 | <p>Stopping the router <em>is</em> the drain: its shutdown sequence <strong>deletes this box’s heartbeat first</strong> — so peers take ownership immediately — and then checkpoints every awake space to the bucket and releases its leases as usual. Deploying a new router build, retiring a box, and reacting to failure are deliberately the same code path.</p> |
| 126 | <figure class="cmd"> |
| 127 | <figcaption>Run on the box being drained</figcaption> |
| 128 | <pre><code>systemctl stop lchat-router # heartbeat deleted, spaces checkpointed, leases released</code></pre> |
| 129 | </figure> |
| 130 | <p>The recovery numbers, verified by live tests in both directions: after a <em>graceful</em> drain, the survivor is serving the drained box’s spaces about <strong>two seconds</strong> later; after a <em>hard kill</em> (no drain, heartbeat left to rot), within about <strong>the lease lifetime</strong> — thirty seconds when those tests ran, <strong>fifteen minutes</strong> since #836 stretched the lease to tame its S3 request bill — with data intact in both cases (up to the space’s <a href="hibernation.html#replication-classes">replication class</a>’s window), because writes had been syncing to the bucket all along.</p> |
| 131 | |
| 132 | <h2 id="region-placement">Region-scoped placement: pinning a space to a place, and moving it</h2> |
| 133 | <p>The mesh so far spreads spaces across boxes for capacity and failover, treating every box as interchangeable. <strong>Region placement</strong> adds a second axis: a space can declare where it wants to live, and the mesh honours it. This is the machinery behind data residency (“this space stays in the EU”) and, eventually, latency (“serve this space near its members”) — and it is <span class="status built">running on the dev mesh</span> today, proven across a two-region (Dublin + Helsinki) fleet. It is on <code>main</code> but not yet deployed to production, which is still a single box.</p> |
| 134 | <p>Two fields carry it, one on each side of the pairing:</p> |
| 135 | <dl> |
| 136 | <dt><code>LCHAT_MESH_REGION</code> — a box’s region</dt> |
| 137 | <dd>An opaque label (e.g. <code>ie</code>, <code>fi</code>) each box advertises in its heartbeat <em>body</em> (not the key, which stays the peer URL). Absent means the box is region-agnostic and takes part in placement exactly as before; setting it costs nothing until a space actually names a region.</dd> |
| 138 | <dt><code>home_region</code> — a space’s desired region</dt> |
| 139 | <dd>A replicated registry column on the route. When set, <a href="#placement">rendezvous hashing</a> is scoped to only the live boxes whose <code>LCHAT_MESH_REGION</code> matches it — so the space is owned by a box in its region, and a box in the wrong region forwards to the right one. Empty (every space today) means the whole fleet is eligible, byte-for-byte the pre-region behaviour. If a space names a region no live box serves, placement falls back to the whole fleet rather than stranding it — availability wins over residency.</dd> |
| 140 | </dl> |
| 141 | <div class="plain"> |
| 142 | <span class="label">In plain words</span> |
| 143 | <p>Each box wears a region badge; each space can carry a note saying which badge it prefers. The same hashing arithmetic runs, but only over the boxes wearing the right badge. A space with no preference is placed as before, and a space whose region has no box up right now is still served somewhere rather than going dark.</p> |
| 144 | </div> |
| 145 | |
| 146 | <h3 id="region-neutral">The address stays put: region is a property, not a place in the name</h3> |
| 147 | <p>Because a space’s <em>address is its identity</em> (invite links and per-origin browser keys bind to it), region cannot live in the hostname — changing region would change the name and break every member. So a region-placed fleet mints spaces at a single region-neutral domain (<code><id>.example.com</code>, one wildcard) and the region rides in the <code>home_region</code> field instead. Three variables make that safe at fleet scale:</p> |
| 148 | <dl> |
| 149 | <dt><code>LCHAT_CLOUD_REGION</code> + <code>add-instance --home-region <cc></code> — stamping the region at birth</dt> |
| 150 | <dd>The provisioner stamps each new space’s <code>home_region</code> from <code>LCHAT_CLOUD_REGION</code> (or the flag, for a hand-run create), so a space is placed in its region from its first wake.</dd> |
| 151 | <dt><code>LCHAT_CLOUD_ID_LEDGER</code> — one id space across regions</dt> |
| 152 | <dd>With a single region-neutral namespace, two regions minting at once must never collide on a hostname. The ledger is a small, shared, S3-backed monotonic counter (block-allocated, so roughly one S3 write per hundred creates) that every provisioner draws from — <code><id></code> is globally unique regardless of which region minted it. Off by default (a lone provisioner keeps its local ids); an explicit opt-in, because switching it on for a namespace that <em>already</em> has spaces needs the counter seeded above the existing maximum first, or fresh ids would re-mint live hostnames.</dd> |
| 153 | <dt><code>LCHAT_DNS_PROVIDER=route53</code> — a wildcard cert without a token on the box</dt> |
| 154 | <dd>Every region’s box must terminate TLS for any region-neutral host (it may own the host, or forward to the box that does), which needs a <em>shared</em> wildcard certificate — and a wildcard is DNS-01 only. The DNS-01 solver is pluggable: <a href="router.html#dns01">DNSimple</a> stays the default, and Route53 is an additive off-ramp that authenticates with the box’s own AWS role/user (scoped to one hosted zone) so no DNS token sits on disk. It never replaces DNSimple — it is chosen only by this variable.</dd> |
| 155 | </dl> |
| 156 | |
| 157 | <h3 id="relocate">Relocating a space between regions</h3> |
| 158 | <p><code>lchat relocate <host> --to <region></code> is the deliberate move — an admin changing where a space lives, without changing its name. It runs three steps, and the <a href="hibernation.html#the-lease">single-writer lease</a> is what makes it safe:</p> |
| 159 | <figure class="cmd"> |
| 160 | <figcaption>Run on a mesh box (with /data/router.env sourced), for a HIBERNATING space</figcaption> |
| 161 | <pre><code>lchat relocate 8oe.example.com --to fi |
| 162 | <span class="out">relocated 8oe.example.com: ie → fi |
| 163 | store: no copy — source and destination share the same bucket (the shared-bucket case) |
| 164 | home_region flipped + route re-fanned-out to peers — the mesh now places 8oe.example.com on a fi box</span></code></pre> |
| 165 | </figure> |
| 166 | <ol> |
| 167 | <li><strong>Store copy.</strong> The destination box wakes the space by restoring its DB from <em>its</em> region’s object store, so the replica must be there first. When both regions share one bucket the bytes are already visible and this is a no-op; with per-region buckets it is a real prefix-to-prefix copy (deliberately excluding the lease object, so the destination takes a fresh lease).</li> |
| 168 | <li><strong>Flip <code>home_region</code> + re-fan-out.</strong> The registry column is updated and the route re-replicated to peers, so every box now hashes the space onto a destination-region box.</li> |
| 169 | <li><strong>(Optional) move the A record.</strong> Repointing DNS at the destination avoids the forwarding hop, but is not required for correctness — the wildcard + the mesh’s forward-to-owner already route through the DNS-TTL window. Off unless <code>--move-dns</code> (Route53 only).</li> |
| 170 | </ol> |
| 171 | <div class="why"> |
| 172 | <span class="label">Why it can’t split-brain</span> |
| 173 | <p>Region and DNS are only <em>hints</em> that steer placement; the <a href="hibernation.html#the-lease">S3 lease</a> is the one truth of who may write. Relocation never creates a second writer: on a shared bucket both regions contend on the same lease object, so if the source is briefly still awake the destination’s wake loses the lease race and retries once the source sleeps; with per-region buckets the copy runs while the source sleeps (lease released, replica flushed) and the destination takes a fresh lease in its own bucket. Re-running a completed relocation is a no-op — once the space is already home the destination holds the authoritative copy, so it never re-copies stale source bytes over newer ones.</p> |
| 174 | </div> |
| 175 | <div class="warn"> |
| 176 | <span class="label">Only hibernating spaces relocate</span> |
| 177 | <p>A live, systemd-managed instance is pinned to its box — its database is on that box’s local disk, so no other box can restore it. Relocation (and the automatic failover the mesh already does) works only for <a href="hibernation.html">hibernating</a> spaces, whose state lives in the bucket. Convert a live instance first with <code>lchat hibernate <host> --on</code>.</p> |
| 178 | </div> |
| 179 | |
| 180 | <h2 id="mesh-limits">The honest limits: what stands between here and a true multi-box fleet</h2> |
| 181 | <p>Two gaps remain. Neither is speculative — each corresponds to a concrete piece of code that is reserved or not yet written. (Two things this section used to list — registry replication and fleet-wide status aggregation — have since shipped and are covered above and in <a href="operations.html#observability">part 08</a>.)</p> |
| 182 | <ul> |
| 183 | <li><span class="status partial">partially built</span> <strong>Route mutations: deletions now replicate; hibernate flips still don’t.</strong> Since #1346 both delete paths (controld’s <code>deleteInstance</code> and <code>lchat remove</code>) fan a <em>tombstone</em> to every live peer (<code>DELETE /api/mesh/route/{host}</code> — idempotent; removes the row, the <code>in.<host></code> ingest row, and the replicated env file, never member-data-shaped files), and <code>mesh-seed --prune</code> reconciles whatever a best-effort fan-out missed by diffing the peer’s in-scope listing (<code>GET /api/mesh/routes</code>) against the source registry — with a zero-source guard so a misread registry can never mass-delete a peer. Without this, a destroyed space lived on — and could <em>wake</em>, from a stale replica — on any peer holding its row (six such zombies were swept off a production box by hand on 2026-07-25). Still open: a <em>hibernate flip</em> doesn’t replicate, and must not naively — a peer holding <code>Hibernate=false</code> would proxy to a local socket that doesn’t exist instead of forwarding; a flipped-to-systemd host needs a “lives elsewhere” route shape peers don’t have yet. Renames don’t exist (the host is the key).</li> |
| 184 | <li><span class="status partial">built, not yet proven in production</span> <strong>The cross-region store copy is exercised only as a no-op.</strong> <a href="#relocate">Relocation</a>’s per-region-bucket copy path is written and unit-tested, but the dev mesh shares one bucket, so live proof only covers the no-op skip path. Before a real cross-region move in production it needs: the source quiesced (asleep) before the copy so a live-writing source can’t be copied mid-write; the copy client signed for the destination bucket’s region; and a download+reupload variant for cross-account or R2 destinations (<code>CopyObject</code> covers only same-account, same-provider). All three are noted in <code>docs/multi-edge.md</code>.</li> |
| 185 | <li><span class="status partial">partially built</span> <strong>The certificate cache is per box.</strong> The wildcard certificate makes mesh domains workable (each box independently obtains its own copy), but per-host certificates for domains outside the wildcard cannot follow a space between boxes; the S3-backed shared cache that would fix this is reserved (<code>LCHAT_CERT_CACHE=s3</code>) and deliberately refuses to start. See <a href="router.html#tls">part 02</a>.</li> |
| 186 | </ul> |
| 187 | <div class="roadmap"> |
| 188 | <span class="label">Designed, not built: what the mesh grows into</span> |
| 189 | <p>The written design (<code>docs/planet-scale.md</code>, <code>docs/multi-edge.md</code>) carries the mesh further than any current code. <a href="#region-placement">Region-scoped placement</a> and <a href="#relocate">relocation</a> have since shipped and moved up to their own section above; what remains designed-only is: <em>multi-edge within a region</em> — several boxes replicating one space’s log for latency and true failover, rather than one owner at a time; an autoscaling group of boxes behind a network load balancer, sized on warm-set memory; and a co-operative “provider” seam in which spaces could be delegated to operators other than the product’s own. Each has a schema field or an interface already cut for it, and none has operable behaviour today.</p> |
| 190 | </div> |
| 191 | |
| 192 | </div> |
| 193 | </main> |
| 194 | </div> |
| 195 | </body> |
| 196 | </html> |
| 197 | |
amadan