| index b670437..8eb76ab 100644 |
| --- a/docs/planet-scale-guide/mesh.html |
| +++ b/docs/planet-scale-guide/mesh.html |
| @@ -126,15 +126,64 @@ LCHAT_MESH_TOKEN=<the same shared secret as every other box> |
| </figure> |
| <p>The recovery numbers, verified by live tests in both directions: after a <em>graceful</em> drain, the survivor is serving the drained box’s spaces about <strong>two seconds</strong> later; after a <em>hard kill</em> (no drain, heartbeat left to rot), within about <strong>the lease lifetime</strong> — thirty seconds when those tests ran, <strong>fifteen minutes</strong> since #836 stretched the lease to tame its S3 request bill — with data intact in both cases (up to the space’s <a href="hibernation.html#replication-classes">replication class</a>’s window), because writes had been syncing to the bucket all along.</p> |
| |
| +<h2 id="region-placement">Region-scoped placement: pinning a space to a place, and moving it</h2> |
| +<p>The mesh so far spreads spaces across boxes for capacity and failover, treating every box as interchangeable. <strong>Region placement</strong> adds a second axis: a space can declare where it wants to live, and the mesh honours it. This is the machinery behind data residency (“this space stays in the EU”) and, eventually, latency (“serve this space near its members”) — and it is <span class="status built">running on the dev mesh</span> today, proven across a two-region (Dublin + Helsinki) fleet. It is on <code>main</code> but not yet deployed to production, which is still a single box.</p> |
| +<p>Two fields carry it, one on each side of the pairing:</p> |
| +<dl> |
| + <dt><code>LCHAT_MESH_REGION</code> — a box’s region</dt> |
| + <dd>An opaque label (e.g. <code>ie</code>, <code>fi</code>) each box advertises in its heartbeat <em>body</em> (not the key, which stays the peer URL). Absent means the box is region-agnostic and takes part in placement exactly as before; setting it costs nothing until a space actually names a region.</dd> |
| + <dt><code>home_region</code> — a space’s desired region</dt> |
| + <dd>A replicated registry column on the route. When set, <a href="#placement">rendezvous hashing</a> is scoped to only the live boxes whose <code>LCHAT_MESH_REGION</code> matches it — so the space is owned by a box in its region, and a box in the wrong region forwards to the right one. Empty (every space today) means the whole fleet is eligible, byte-for-byte the pre-region behaviour. If a space names a region no live box serves, placement falls back to the whole fleet rather than stranding it — availability wins over residency.</dd> |
| +</dl> |
| +<div class="plain"> |
| + <span class="label">In plain words</span> |
| + <p>Each box wears a region badge; each space can carry a note saying which badge it prefers. The same hashing arithmetic runs, but only over the boxes wearing the right badge. A space with no preference is placed as before, and a space whose region has no box up right now is still served somewhere rather than going dark.</p> |
| +</div> |
| + |
| +<h3 id="region-neutral">The address stays put: region is a property, not a place in the name</h3> |
| +<p>Because a space’s <em>address is its identity</em> (invite links and per-origin browser keys bind to it), region cannot live in the hostname — changing region would change the name and break every member. So a region-placed fleet mints spaces at a single region-neutral domain (<code><id>.example.com</code>, one wildcard) and the region rides in the <code>home_region</code> field instead. Three variables make that safe at fleet scale:</p> |
| +<dl> |
| + <dt><code>LCHAT_CLOUD_REGION</code> + <code>add-instance --home-region <cc></code> — stamping the region at birth</dt> |
| + <dd>The provisioner stamps each new space’s <code>home_region</code> from <code>LCHAT_CLOUD_REGION</code> (or the flag, for a hand-run create), so a space is placed in its region from its first wake.</dd> |
| + <dt><code>LCHAT_CLOUD_ID_LEDGER</code> — one id space across regions</dt> |
| + <dd>With a single region-neutral namespace, two regions minting at once must never collide on a hostname. The ledger is a small, shared, S3-backed monotonic counter (block-allocated, so roughly one S3 write per hundred creates) that every provisioner draws from — <code><id></code> is globally unique regardless of which region minted it. Off by default (a lone provisioner keeps its local ids); an explicit opt-in, because switching it on for a namespace that <em>already</em> has spaces needs the counter seeded above the existing maximum first, or fresh ids would re-mint live hostnames.</dd> |
| + <dt><code>LCHAT_DNS_PROVIDER=route53</code> — a wildcard cert without a token on the box</dt> |
| + <dd>Every region’s box must terminate TLS for any region-neutral host (it may own the host, or forward to the box that does), which needs a <em>shared</em> wildcard certificate — and a wildcard is DNS-01 only. The DNS-01 solver is pluggable: <a href="router.html#dns01">DNSimple</a> stays the default, and Route53 is an additive off-ramp that authenticates with the box’s own AWS role/user (scoped to one hosted zone) so no DNS token sits on disk. It never replaces DNSimple — it is chosen only by this variable.</dd> |
| +</dl> |
| + |
| +<h3 id="relocate">Relocating a space between regions</h3> |
| +<p><code>lchat relocate <host> --to <region></code> is the deliberate move — an admin changing where a space lives, without changing its name. It runs three steps, and the <a href="hibernation.html#the-lease">single-writer lease</a> is what makes it safe:</p> |
| +<figure class="cmd"> |
| +<figcaption>Run on a mesh box (with /data/router.env sourced), for a HIBERNATING space</figcaption> |
| +<pre><code>lchat relocate 8oe.example.com --to fi |
| +<span class="out">relocated 8oe.example.com: ie → fi |
| + store: no copy — source and destination share the same bucket (the shared-bucket case) |
| + home_region flipped + route re-fanned-out to peers — the mesh now places 8oe.example.com on a fi box</span></code></pre> |
| +</figure> |
| +<ol> |
| + <li><strong>Store copy.</strong> The destination box wakes the space by restoring its DB from <em>its</em> region’s object store, so the replica must be there first. When both regions share one bucket the bytes are already visible and this is a no-op; with per-region buckets it is a real prefix-to-prefix copy (deliberately excluding the lease object, so the destination takes a fresh lease).</li> |
| + <li><strong>Flip <code>home_region</code> + re-fan-out.</strong> The registry column is updated and the route re-replicated to peers, so every box now hashes the space onto a destination-region box.</li> |
| + <li><strong>(Optional) move the A record.</strong> Repointing DNS at the destination avoids the forwarding hop, but is not required for correctness — the wildcard + the mesh’s forward-to-owner already route through the DNS-TTL window. Off unless <code>--move-dns</code> (Route53 only).</li> |
| +</ol> |
| +<div class="why"> |
| + <span class="label">Why it can’t split-brain</span> |
| + <p>Region and DNS are only <em>hints</em> that steer placement; the <a href="hibernation.html#the-lease">S3 lease</a> is the one truth of who may write. Relocation never creates a second writer: on a shared bucket both regions contend on the same lease object, so if the source is briefly still awake the destination’s wake loses the lease race and retries once the source sleeps; with per-region buckets the copy runs while the source sleeps (lease released, replica flushed) and the destination takes a fresh lease in its own bucket. Re-running a completed relocation is a no-op — once the space is already home the destination holds the authoritative copy, so it never re-copies stale source bytes over newer ones.</p> |
| +</div> |
| +<div class="warn"> |
| + <span class="label">Only hibernating spaces relocate</span> |
| + <p>A live, systemd-managed instance is pinned to its box — its database is on that box’s local disk, so no other box can restore it. Relocation (and the automatic failover the mesh already does) works only for <a href="hibernation.html">hibernating</a> spaces, whose state lives in the bucket. Convert a live instance first with <code>lchat hibernate <host> --on</code>.</p> |
| +</div> |
| + |
| <h2 id="mesh-limits">The honest limits: what stands between here and a true multi-box fleet</h2> |
| <p>Two gaps remain. Neither is speculative — each corresponds to a concrete piece of code that is reserved or not yet written. (Two things this section used to list — registry replication and fleet-wide status aggregation — have since shipped and are covered above and in <a href="operations.html#observability">part 08</a>.)</p> |
| <ul> |
| - <li><span class="status partial">partially built</span> <strong>Route mutations after creation do not replicate.</strong> The initial fan-out (<a href="#mesh-replication">above</a>) and <code>mesh-seed</code> cover creation and joining, but a later rename, removal, or hibernation-flag flip on one box is not currently propagated to its peers — those must still be repeated by hand on every mesh member. Fine for a pool of auto-provisioned, rarely-renamed spaces; worth closing before mesh placement carries more than that.</li> |
| + <li><span class="status partial">partially built</span> <strong>Route mutations after creation do not fully replicate.</strong> The initial fan-out (<a href="#mesh-replication">above</a>) and <code>mesh-seed</code> cover creation and joining, and <a href="#relocate"><code>lchat relocate</code></a> now explicitly re-fans-out the one route field it changes (<code>home_region</code>). But a later rename, removal, or hibernation-flag flip on one box is still not propagated to its peers — those must be repeated by hand on every mesh member. Fine for a pool of auto-provisioned, rarely-renamed spaces; worth closing before mesh placement carries more than that.</li> |
| + <li><span class="status partial">built, not yet proven in production</span> <strong>The cross-region store copy is exercised only as a no-op.</strong> <a href="#relocate">Relocation</a>’s per-region-bucket copy path is written and unit-tested, but the dev mesh shares one bucket, so live proof only covers the no-op skip path. Before a real cross-region move in production it needs: the source quiesced (asleep) before the copy so a live-writing source can’t be copied mid-write; the copy client signed for the destination bucket’s region; and a download+reupload variant for cross-account or R2 destinations (<code>CopyObject</code> covers only same-account, same-provider). All three are noted in <code>docs/multi-edge.md</code>.</li> |
| <li><span class="status partial">partially built</span> <strong>The certificate cache is per box.</strong> The wildcard certificate makes mesh domains workable (each box independently obtains its own copy), but per-host certificates for domains outside the wildcard cannot follow a space between boxes; the S3-backed shared cache that would fix this is reserved (<code>LCHAT_CERT_CACHE=s3</code>) and deliberately refuses to start. See <a href="router.html#tls">part 02</a>.</li> |
| </ul> |
| <div class="roadmap"> |
| <span class="label">Designed, not built: what the mesh grows into</span> |
| - <p>The written design (<code>docs/planet-scale.md</code>) carries the mesh further than any current code: per-space <em>geographic</em> placement using the registry’s <code>home_region</code> field (relocating a space between regions while it sleeps, by copying its bucket prefix and flipping the field); an autoscaling group of boxes behind a network load balancer, sized on warm-set memory; and a co-operative “provider” seam in which spaces could be delegated to operators other than the product’s own. Each of these has a schema field or an interface already cut for it, and none has operable behaviour today.</p> |
| + <p>The written design (<code>docs/planet-scale.md</code>, <code>docs/multi-edge.md</code>) carries the mesh further than any current code. <a href="#region-placement">Region-scoped placement</a> and <a href="#relocate">relocation</a> have since shipped and moved up to their own section above; what remains designed-only is: <em>multi-edge within a region</em> — several boxes replicating one space’s log for latency and true failover, rather than one owner at a time; an autoscaling group of boxes behind a network load balancer, sized on warm-set memory; and a co-operative “provider” seam in which spaces could be delegated to operators other than the product’s own. Each has a schema field or an interface already cut for it, and none has operable behaviour today.</p> |
| </div> |
| |
| </div> |
amadan