eleven / messenger Public

server: make mesh route replication region-aware (multi-edge Phase 1) (#1361)

Route replication was scoped only by LCHAT_MESH_DOMAINS: a box replicated an
in-domain route to every live peer and out-of-domain routes to nobody. With
per-country domains that stranded region-placed spaces — a host whose
home_region points at a box outside the creating box's domains never got a
registry row there, so the owner 404'd when the DNS-target box proxied to it
(the live ie.dev host, home_region=fi gap).

Replication now targets the UNION of two sets (docs/multi-edge.md):

  - the boxes covering the host by DOMAIN (today's fleet-wide fan-out for an
    in-domain host), and
  - the boxes owning the host by REGION (those advertising route.HomeRegion),
    read from slice 1's peer-region view (m.regions + m.region — no second
    advertisement mechanism).

New meshSvc.replicationTargets(route) computes the deduped union (self
excluded — registerInstance already upserted locally); boxesInRegion(region)
is candidatesFor's region filter keyed on a region string. fanOutMeshRoute
uses it; mesh-seed gains --region <cc> (the joining box's LCHAT_MESH_REGION)
so a joining box also receives the routes it owns by region, not only
domain-covered ones — the same union rule.

Back-compat is exact: with no LCHAT_MESH_REGION and no home_region anywhere
the region arm is always empty, so the target set is the pre-region,
domain-only fan-out — an agnostic box still early-returns (no beat, no sends)
for an out-of-domain host, and mesh-seed without --region filters by domain
exactly as before.

Scope is replication only — placement (mesh.go candidatesFor), the
provisioner, addressing, and the S3 lease are untouched. Cross-region TLS (a
box presenting a valid cert for an out-of-domain host it now has a row for)
is still unsolved and is the next slice; meshReplicateRoute's comment and the
mesh guide note it explicitly.

Tests (meshreplicate_test.go): region-owned-out-of-domain reaches the region
box; empty home_region stays domain-scoped both ways; domain+region union
dedups the doubly-qualified box; an agnostic box never region-places; and
mesh-seed --region carries a region-owned out-of-domain route to a joining
box while leaving a non-owned/non-hibernating route behind. go build/test
./... green.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Paul Campbell 2 months ago f8e17ff69d989857b5fa8b1645b4d76129b1a179 parent a6e20df

4 files changed, +297 −37

docs/planet-scale-guide/mesh.html +6 −4
docs/planet-scale-guide/reference.html +2 −2
meshreplicate.go +139 −31
meshreplicate_test.go +150 −0

diff --git a/docs/planet-scale-guide/mesh.html b/docs/planet-scale-guide/mesh.html

index 6f2713a..edf2d91 100644

--- a/docs/planet-scale-guide/mesh.html

+++ b/docs/planet-scale-guide/mesh.html

@@ -85,8 +85,8 @@ echo aHR0cHM6Ly9ib3gtMS5leGFtcGxlLm5ldA | base64 -d

<h2 id="mesh-replication">Route replication: every box learns about every host</h2>

The router looks a request’s hostname up in its own local registry before any mesh logic runs at all, and answers <code>404</code> immediately on a miss. So placement (hashing, forwarding) is only half the story: a box can only ever serve — or even correctly forward — a host it has a registry row for. Two mechanisms keep every mesh member’s registry in sync, both authenticated with <code>LCHAT_MESH_TOKEN</code>:

<ul>

- <li>Automatic fan-out on creation. The moment a new in-scope hibernating host is registered anywhere, that box replicates the row to every currently live mesh peer (found the same way placement does — the membership heartbeats). Only a narrow, purpose-built payload crosses the wire — host, kind, version, hibernate flag, a handful of routing-relevant fields, and the instance’s own env values (admin token, entitlement, seeded name) so a peer-woken instance boots with the token its creator minted — never the receiving box’s own socket path or database path, which each box always derives locally. A peer that is down at that moment simply misses the row; fan-out is best-effort, logged, and never blocks the create. The daemon doing the creating must see the mesh env: <code>lchat-control.service</code> reads <code>/data/router.env</code> for exactly this (a controld without <code>LCHAT_MESH_SELF</code> silently skips fan-out, and any create HRW-placed on a peer is born dark — the peer answers <code>404 unknown host</code> for it; a live production incident, 2026-07-24). controld logs its replication posture at startup.</li>

- <li><code>lchat mesh-seed <peer-url></code> — the one-time catch-up. Run on an existing box and pointed at a newly joined one, this bulk-replicates every in-scope hibernating route the existing box already knows about, in one pass. This is the command that makes joining a box a single step instead of hand-registering every host (see <a href="#add-box">adding a box</a>, below).</li>

+ <li>Automatic fan-out on creation. The moment a new in-scope hibernating host is registered anywhere, that box replicates the row to every currently live mesh peer (found the same way placement does — the membership heartbeats). Only a narrow, purpose-built payload crosses the wire — host, kind, version, hibernate flag, a handful of routing-relevant fields, and the instance’s own env values (admin token, entitlement, seeded name) so a peer-woken instance boots with the token its creator minted — never the receiving box’s own socket path or database path, which each box always derives locally. A peer that is down at that moment simply misses the row; fan-out is best-effort, logged, and never blocks the create. On a region-scoped fleet the target set is a union: the boxes that cover the host by domain, plus any box that owns it by <code>home_region</code> — so a space placed into a region whose box sits outside the creating box’s domains still gets its row there. (Presenting a valid TLS certificate for such an out-of-domain host is a still-unbuilt follow-on; on the single-region production fleet the two sets coincide and nothing changes.) The daemon doing the creating must see the mesh env: <code>lchat-control.service</code> reads <code>/data/router.env</code> for exactly this (a controld without <code>LCHAT_MESH_SELF</code> silently skips fan-out, and any create HRW-placed on a peer is born dark — the peer answers <code>404 unknown host</code> for it; a live production incident, 2026-07-24). controld logs its replication posture at startup.</li>

+ <li><code>lchat mesh-seed [--region <cc>] <peer-url></code> — the one-time catch-up. Run on an existing box and pointed at a newly joined one, this bulk-replicates every in-scope hibernating route the existing box already knows about, in one pass. This is the command that makes joining a box a single step instead of hand-registering every host (see <a href="#add-box">adding a box</a>, below).</li>

</ul>

Why replicate rows instead of just answering for any host

@@ -115,9 +115,11 @@ LCHAT_MESH_TOKEN=<the same shared secret as every other box>

Within one heartbeat interval the peers see the new member.</li>

<li>Seed its registry from an existing box.

<figure class="cmd"><figcaption>Run on an EXISTING mesh box, pointed at the new one</figcaption>

- <pre><code>lchat mesh-seed --env-file /data/router.env https://box-2.example.net</code></pre>

+ <pre><code>lchat mesh-seed --env-file /data/router.env https://box-2.example.net

+# region-scoped fleet — also seed routes the new box owns by home_region:

+lchat mesh-seed --env-file /data/router.env --region fi https://box-2.example.net</code></pre>

(<code>--env-file</code> borrows the router’s <code>LCHAT_MESH_*</code> values — the seed can run under a sudo’d operator session that has no mesh env of its own. Add <code>--prune</code> to also tombstone peer rows this registry no longer has — the reconciliation half of deletion replication. The encoded operator path is <code>./deploy/ops mesh seed [--prune] <peer-url> [box]</code>.)</figure>

- Bulk-replicates every in-scope hibernating route in one pass, so the new box can immediately answer for (or correctly proxy) any of them. After this, roughly 1/N of the in-scope spaces re-home to the new box at their next sleep/wake boundary — nothing needs announcing beyond this one command; the guest book and the registry rows do the rest.</li>

+ Bulk-replicates every in-scope hibernating route in one pass, so the new box can immediately answer for (or correctly proxy) any of them. After this, roughly 1/N of the in-scope spaces re-home to the new box at their next sleep/wake boundary — nothing needs announcing beyond this one command; the guest book and the registry rows do the rest. On a region-scoped fleet, pass <code>--region</code> with the new box’s <code>LCHAT_MESH_REGION</code> so any space pinned to that region — even one addressed outside the seed’s domains — is seeded to it as well.</li>

</ol>

<h3 id="drain-box">Draining a box (deploys, maintenance, scale-down)</h3>

Stopping the router is the drain: its shutdown sequence deletes this box’s heartbeat first — so peers take ownership immediately — and then checkpoints every awake space to the bucket and releases its leases as usual. Deploying a new router build, retiring a box, and reacting to failure are deliberately the same code path.

diff --git a/docs/planet-scale-guide/reference.html b/docs/planet-scale-guide/reference.html

index a06b073..cf93282 100644

--- a/docs/planet-scale-guide/reference.html

+++ b/docs/planet-scale-guide/reference.html

@@ -45,8 +45,8 @@

<dd>Reads (no argument) or sets a hibernating host’s <a href="hibernation.html#replication-classes">replication class</a> — how often awake writes sync to the bucket. A set reloads the router and applies at the host’s next wake; on provisioned fleets billing drives this instead. Explained in <a href="hibernation.html#replication-classes">part 06</a>.</dd>

<dt><code>lchat instances</code></dt>

<dd>Prints the registry: HOST, KIND, STATUS, VERSION, HIBERNATE, REPLICATION, SOCKET, plus a <code>last error</code> line for any failed row. The first stop for “what does this box think it is serving?”.</dd>

- <dt><code>lchat mesh-seed <peer-url></code></dt>

- <dd>Run on an existing mesh box, pointed at a newly joined one: bulk-replicates every in-scope hibernating route to the peer in one pass, authenticated by <code>LCHAT_MESH_TOKEN</code>. The one-time catch-up step of <a href="mesh.html#add-box">adding a box to the mesh</a>. <a href="mesh.html#mesh-replication">Part 07</a>.</dd>

+ <dt><code>lchat mesh-seed [--region <cc>] <peer-url></code></dt>

+ <dd>Run on an existing mesh box, pointed at a newly joined one: bulk-replicates every in-scope hibernating route to the peer in one pass, authenticated by <code>LCHAT_MESH_TOKEN</code>. The one-time catch-up step of <a href="mesh.html#add-box">adding a box to the mesh</a>. On a region-scoped fleet, pass <code>--region</code> with the joining box’s <code>LCHAT_MESH_REGION</code> so routes it owns by <code>home_region</code> are seeded too, even ones outside this seed’s <code>LCHAT_MESH_DOMAINS</code> (the multi-edge region-owner union); omitting it seeds by domain exactly as before. <a href="mesh.html#mesh-replication">Part 07</a>.</dd>

<dt><code>lchat relocate <host> --to <region> [--dst-bucket ] [--move-dns --dns-target <ipv4>]</code></dt>

<dd>Move a hibernating space to another region without changing its address: copies its store to the destination region (a no-op on a shared bucket), flips <code>home_region</code>, and re-fans-out the route so the mesh re-homes it. The <a href="hibernation.html#the-lease">lease</a> keeps a single writer throughout. Run on a mesh box with <code>/data/router.env</code> sourced. <a href="mesh.html#relocate">Part 07</a>.</dd>

<dt><code>lchat admin-link <host></code> / <code>lchat setup-invite <host></code></dt>

diff --git a/meshreplicate.go b/meshreplicate.go

index aa61f8a..0eb90dc 100644

--- a/meshreplicate.go

+++ b/meshreplicate.go

@@ -10,9 +10,22 @@ package main

// only when the mesh is on), inserts or updates one route.

// - registerInstance calls fanOutMeshRoute right after its local upsert,

// best-effort against whichever peers are live right now.

-// - `lchat mesh-seed <peer>` (runMeshSeed) bulk-replicates every existing

-// in-scope route once — the catch-up a freshly joined box needs before it

-// can serve anything, since it joins with an empty registry.

+// - `lchat mesh-seed [--region <cc>] <peer>` (runMeshSeed) bulk-replicates

+// every existing route the joining box owns once — the catch-up a freshly

+// joined box needs before it can serve anything, since it joins with an

+// empty registry.

+//

+// Which peers get a route is the UNION of two sets (docs/multi-edge.md, Phase 1):

+// the boxes covering the host by DOMAIN (its LCHAT_MESH_DOMAINS — today's

+// fleet-wide fan-out for an in-domain host) and the boxes owning it by REGION

+// (those advertising the route's home_region). The region arm is the multi-edge

+// fix: a space placed by home_region onto a box whose per-country domain does NOT

+// cover the host (mesh.go candidatesFor) still needs a registry row THERE to

+// serve when the DNS-target box proxies to it — domain scope alone would strand

+// it. With no LCHAT_MESH_REGION and no home_region anywhere the region arm is

+// always empty, so this stays byte-identical to the pre-region, domain-only

+// fan-out. (Completing TLS to a region-owner box for a host outside ITS domain

+// is a separate, still-unbuilt slice — see meshReplicateRoute.)

// Both paths send only the fields a route's CONTENT actually needs (host,

// kind, hibernate flag, per-instance config) — never Socket/DBPath, which are

@@ -26,7 +39,10 @@ package main

// `mesh-seed --prune` reconciles whatever a best-effort fan-out missed —

// without this, a destroyed space lived on (and WOKE, from a stale replica)

// on any peer that held its row, violating the destroy promise. Live case:

-// six zombies swept off kepler 2026-07-25.

+// six zombies swept off kepler 2026-07-25. Deletion fan-out and --prune are

+// still DOMAIN-scoped only: a region-owned, out-of-domain row's tombstone is

+// the next multi-edge slice (alongside out-of-domain TLS, below) — inert today

+// because nothing region-places out-of-domain hosts until that TLS lands.

// Honest gap, narrowed: a HIBERNATE FLIP still doesn't replicate — and must

// not naively: a peer holding Hibernate=false would proxy to a local socket

@@ -138,9 +154,15 @@ func (p meshRoutePayload) toLocalRoute() (Route, error) {

// address but performs TLS (and sends Host) as route.Host: a wildcard-covered

// hostname the receiving box's wildcard manager hands back a valid cert for

// regardless of whether that host is registered THERE yet (the "unknown

-// subdomain still completes TLS" property wildcard.go relies on). Callers

-// only ever pass an in-scope route (fanOutMeshRoute/runMeshSeed both check

-// first), so this is never asked to authenticate as an out-of-scope host.

+// subdomain still completes TLS" property wildcard.go relies on). For a

+// DOMAIN-covered target that cert is the box's own wildcard, so TLS completes

+// exactly as before. Region replication (fanOutMeshRoute's union with the

+// route's home_region owners) can now also target a box whose per-country

+// LCHAT_MESH_DOMAINS does NOT cover payload.Host — an out-of-domain host that

+// box may hold no wildcard for, so this handshake can fail there. Making a box

+// present a valid cert for an out-of-domain host it now owns by region is the

+// NEXT multi-edge slice (docs/multi-edge.md), deliberately not built here; a

+// same-domain fleet (production today) is unaffected.

func meshReplicateRoute(ctx context.Context, peer, token string, payload meshRoutePayload) error {

body, err := json.Marshal(payload)

if err != nil {

@@ -157,24 +179,91 @@ func meshReplicateRoute(ctx context.Context, peer, token string, payload meshRou

return nil

}

-// fanOutMeshRoute replicates route to every OTHER live mesh member,

-// best-effort: a peer that's down or errors just can't yet serve this host

-// (see the file doc comment's honest gap on mutations — creation retries only

-// via a later add-instance or an operator's mesh-seed rerun). A no-op — byte-

-// identical to today — whenever the mesh is off or the host is out of

-// LCHAT_MESH_DOMAINS scope, so self-hosters and non-mesh boxes are untouched.

-// Called right after registerInstance's local reg.upsert.

+// replicationTargets is the set of OTHER live boxes that must hold route's

+// registry row: the boxes covering host by DOMAIN (this box's

+// LCHAT_MESH_DOMAINS — today's fleet-wide fan-out for an in-domain host) UNION

+// the boxes owning host by REGION (those advertising route.HomeRegion). The

+// union is the multi-edge fix (docs/multi-edge.md): a space placed by home_region

+// onto a box whose per-country domain does NOT cover the host still needs a row

+// there to serve when the DNS-target box proxies to it — domain scope alone would

+// strand it. Self is excluded (registerInstance already upserted the local row),

+// and a box qualifying by BOTH domain and region is sent to once.

+//

+// Region targets are non-empty only on a region-aware box (one that fetched peer

+// regions; m.regions is empty otherwise) and only for a route carrying a

+// home_region, so with no LCHAT_MESH_REGION and no home_region anywhere this is

+// exactly the domain-scoped set — byte-identical to pre-region replication.

+func (m *meshSvc) replicationTargets(route Route) []string {

+ seen := map[string]bool{m.self: true} // never replicate to ourselves

+ var targets []string

+ add := func(boxes []string) {

+ for _, b := range boxes {

+ if !seen[b] {

+ seen[b] = true

+ targets = append(targets, b)

+ }

+ if m.inScope(route.Host) {

+ add(m.liveBoxes()) // domain-scoped: the whole live fleet, as before regions

+ }

+ add(m.boxesInRegion(route.HomeRegion)) // region-owned: the owner box(es)

+ return targets

+// boxesInRegion returns the live boxes whose advertised region matches region —

+// the mesh members that own a route with this home_region. It is candidatesFor's

+// region filter keyed on a region string (rather than a host's registry row),

+// reusing slice 1's peer-region view (m.regions, self via m.region — never a

+// second advertisement mechanism). An empty region, or a region-agnostic box

+// (m.regions empty because it never fetched peer regions), yields no matches,

+// which is what keeps region replication inert until the fleet opts into regions.

+func (m *meshSvc) boxesInRegion(region string) []string {

+ if region == "" {

+ return nil

+ }

+ live := m.liveBoxes()

+ m.mu.Lock()

+ regions := m.regions

+ m.mu.Unlock()

+ var matched []string

+ for _, b := range live {

+ r := regions[b]

+ if b == m.self {

+ r = m.region // our own region is known locally, never via S3

+ }

+ if r == region {

+ matched = append(matched, b)

+ }

+ return matched

+// fanOutMeshRoute replicates route to every OTHER box that must hold its

+// registry row — the DOMAIN ∪ REGION union replicationTargets computes

+// (docs/multi-edge.md) — best-effort: a peer that's down or errors just can't

+// yet serve this host (see the file doc comment's honest gap on mutations —

+// creation retries only via a later add-instance or an operator's mesh-seed

+// rerun). A no-op — byte-identical to today — whenever the mesh is off, or the

+// host is out of LCHAT_MESH_DOMAINS scope AND this box can't region-place it

+// (no LCHAT_MESH_REGION, or the route carries no home_region), so self-hosters,

+// non-mesh boxes, and every region-agnostic box are untouched. Called right

+// after registerInstance's local reg.upsert.

func fanOutMeshRoute(hc *hibernateConfig, route Route, env map[string]string) {

mesh, err := meshFromEnv(hc)

- if err != nil || mesh == nil || !mesh.inScope(route.Host) {

+ if err != nil || mesh == nil {

+ return

+ }

+ // Pre-region early-out, kept byte-identical: an out-of-domain host on a box

+ // that can't region-place it (region unset, or the route has no home_region)

+ // replicates nowhere, and — crucially — doesn't even beat. An agnostic box

+ // always lands here for an out-of-scope host, exactly as before regions.

+ if !mesh.inScope(route.Host) && (mesh.region == "" || route.HomeRegion == "") {

return

}

payload := payloadFromRouteEnv(route, env)

- mesh.beat() // one synchronous PUT+LIST for a fresh membership view

- for _, peer := range mesh.liveBoxes() {

- if peer == mesh.self {

- continue

- }

+ mesh.beat() // one synchronous PUT+LIST(+region refresh) for a fresh view

+ for _, peer := range mesh.replicationTargets(route) {

ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)

err := meshReplicateRoute(ctx, peer, mesh.token, payload)

cancel()

@@ -432,19 +521,29 @@ func pruneCandidates(sourceHosts, peerHosts, domains []string, sourceInScope int

return out

}

-// runMeshSeed is `lchat mesh-seed <peer-base-url>`: bulk-replicates every

-// existing in-scope, hibernating route to one peer — the one-time catch-up a

-// freshly joined box needs (it starts with an empty registry) before HRW can

-// honestly place any host there. Reads LCHAT_MESH_DOMAINS/LCHAT_MESH_TOKEN

-// directly rather than through meshFromEnv: the CALLER (the already-running

-// box doing the seeding) need not have LCHAT_MESH_SELF set yet — seeding is

-// an operator step that runs before the mesh itself is flipped on (see

-// docs/instance-router.md).

+// runMeshSeed is `lchat mesh-seed [--region <cc>] <peer-base-url>`:

+// bulk-replicates the hibernating routes one peer must hold — the one-time

+// catch-up a freshly joined box needs (it starts with an empty registry) before

+// HRW can honestly place any host there. It seeds by the same DOMAIN ∪ REGION

+// union fanOutMeshRoute uses (docs/multi-edge.md): a route whose host is under

+// this seed's LCHAT_MESH_DOMAINS, PLUS — when --region names the joining box's

+// placement region (its LCHAT_MESH_REGION) — a route whose home_region matches

+// it, even if out of domain. Without --region the filter is exactly domain scope,

+// so an existing same-domain fleet seeds byte-identically to before.

+//

+// Reads LCHAT_MESH_DOMAINS/LCHAT_MESH_TOKEN directly rather than through

+// meshFromEnv: the CALLER (the already-running box doing the seeding) need not

+// have LCHAT_MESH_SELF set yet — seeding is an operator step that runs before

+// the mesh itself is flipped on (see docs/instance-router.md). The joining box's

+// region is passed explicitly rather than looked up, so the seed needs no

+// bucket/heartbeat wiring — the operator who just set LCHAT_MESH_REGION on the

+// new box supplies the same value here.

func runMeshSeed(args []string) {

fs := flag.NewFlagSet("mesh-seed", flag.ExitOnError)

registryPath := fs.String("registry", envOr("LCHAT_REGISTRY", "/data/registry.db"), "router registry SQLite path")

envFile := fs.String("env-file", "", "load LCHAT_MESH_* (and anything else) from this KEY=value file first — lets a sudo'd operator run borrow the router's env (e.g. /data/router.env) without a root shell")

prune := fs.Bool("prune", false, "after seeding, tombstone the peer's in-scope hibernating rows this registry doesn't know — reconciles deletions a best-effort fan-out missed (#1346); refuses on an empty source view")

+ peerRegion := fs.String("region", "", "the JOINING box's placement region (its LCHAT_MESH_REGION); when set, routes whose home_region matches are seeded too, even if out of LCHAT_MESH_DOMAINS")

_ = fs.Parse(reorderFlagsFirst(fs, args))

if *envFile != "" {

kv, err := readEnvFile(*envFile)

@@ -459,7 +558,7 @@ func runMeshSeed(args []string) {

}

peer := fs.Arg(0)

if peer == "" {

- die("mesh-seed: usage: lchat mesh-seed [--env-file /data/router.env] <peer-base-url>")

+ die("mesh-seed: usage: lchat mesh-seed [--env-file /data/router.env] [--region <cc>] [--prune] <peer-base-url>")

}

domains := splitDomains(os.Getenv("LCHAT_MESH_DOMAINS"))

if len(domains) == 0 {

@@ -469,6 +568,7 @@ func runMeshSeed(args []string) {

if token == "" {

die("mesh-seed: LCHAT_MESH_TOKEN must be set")

}

+ region := strings.TrimSpace(*peerRegion)

reg, err := openRegistry(*registryPath)

if err != nil {

@@ -480,7 +580,15 @@ func runMeshSeed(args []string) {

}

seeded := 0

for _, route := range routes {

- if !route.Hibernate || !hostInDomains(route.Host, domains) {

+ if !route.Hibernate {

+ continue

+ }

+ // DOMAIN ∪ REGION, mirroring fanOutMeshRoute/replicationTargets: seed a

+ // route the joining box covers by domain, or (when --region is given) one

+ // it owns by home_region even though it's out of domain.

+ domainCovered := hostInDomains(route.Host, domains)

+ regionOwned := region != "" && route.HomeRegion == region

+ if !domainCovered && !regionOwned {

continue

}

// The instance env rides along (meshRoutePayload doc) — without it the

@@ -500,7 +608,7 @@ func runMeshSeed(args []string) {

fmt.Printf(" %s: seeded\n", route.Host)

seeded++

}

- fmt.Printf("seeded %d/%d in-scope hibernating route(s) to %s\n", seeded, len(routes), peer)

+ fmt.Printf("seeded %d/%d route(s) the joining box owns to %s\n", seeded, len(routes), peer)

if *prune {

// Reconcile: anything the peer serves in-scope that this registry has

diff --git a/meshreplicate_test.go b/meshreplicate_test.go

index c6cb9f0..6c7e904 100644

--- a/meshreplicate_test.go

+++ b/meshreplicate_test.go

@@ -17,6 +17,7 @@ import (

"os"

"path/filepath"

"testing"

+ "time"

)

// toLocalRoute must never trust a peer's Socket/DBPath (box-local filesystem

@@ -323,3 +324,152 @@ func TestPruneCandidates(t *testing.T) {

t.Errorf("zero-guard failed: %v", got)

}

+// --- region-aware replication (docs/multi-edge.md, Phase 1) ---------------------

+// replicationTargets is the DOMAIN ∪ REGION union that decides which peers get a

+// route's registry row. Four rules the multi-edge slice pins:

+//

+// (a) a route pinned to a region reaches that region's box even when this box's

+// LCHAT_MESH_DOMAINS does NOT cover the host — the gap this slice closes;

+// (b) an empty home_region is domain-scoped only (== pre-region): out of domain

+// ⇒ nobody, in domain ⇒ the whole live fleet, unchanged;

+// (c) a route that is both domain-covered and region-owned goes to the union,

+// with the region-owner box sent to exactly once (no duplicate);

+// (d) self is never a target — even when self advertises the route's region —

+// because registerInstance already upserted the local row.

+func TestMeshReplicationTargets(t *testing.T) {

+ clk := &fakeClock{t: time.Unix(1_700_000_000, 0)}

+ store := newFakeMeshStore(clk.now)

+ // self "http://a" is an ie box covering only ie.dev.11ch.at; peer "http://b"

+ // is an fi box; peer "http://c" is a second ie box.

+ m := newMesh("http://a", store, []string{"ie.dev.11ch.at"})

+ m.now = clk.now

+ m.region = "ie"

+ store.setBeat("http://b", clk.now())

+ store.setRegion("http://b", "fi")

+ store.setBeat("http://c", clk.now())

+ store.setRegion("http://c", "ie")

+ m.beat() // load membership + peer regions

+ // (a) region-owned but OUT of domain → only the fi box, which domain scope

+ // would have skipped entirely.

+ outOfDomain := Route{Host: "s.fi.dev.11ch.at", Kind: "instance", Hibernate: true, HomeRegion: "fi"}

+ if got := m.replicationTargets(outOfDomain); !sameSet(got, []string{"http://b"}) {

+ t.Fatalf("region-owned out-of-domain: targets = %v, want [http://b]", got)

+ }

+ // (b) empty home_region → domain scope only, both ways.

+ if got := m.replicationTargets(Route{Host: "s.fi.dev.11ch.at", Kind: "instance", Hibernate: true}); len(got) != 0 {

+ t.Fatalf("empty home_region out of domain: targets = %v, want none", got)

+ }

+ if got := m.replicationTargets(Route{Host: "s.ie.dev.11ch.at", Kind: "instance", Hibernate: true}); !sameSet(got, []string{"http://b", "http://c"}) {

+ t.Fatalf("empty home_region in domain: targets = %v, want the whole live fleet [b c]", got)

+ }

+ // (c) domain-covered AND region-owned by http://b → union {b,c}, b exactly once.

+ both := Route{Host: "s.ie.dev.11ch.at", Kind: "instance", Hibernate: true, HomeRegion: "fi"}

+ got := m.replicationTargets(both)

+ if !sameSet(got, []string{"http://b", "http://c"}) {

+ t.Fatalf("domain+region union: targets = %v, want [b c]", got)

+ }

+ if n := countOccurrences(got, "http://b"); n != 1 {

+ t.Fatalf("union sent to http://b %d time(s), want exactly 1 (dedup)", n)

+ }

+ // (d) region-owned by our OWN region, out of domain → the other ie box only;

+ // self is excluded despite matching the region.

+ ownRegion := Route{Host: "s.11ch.at", Kind: "instance", Hibernate: true, HomeRegion: "ie"}

+ if got := m.replicationTargets(ownRegion); !sameSet(got, []string{"http://c"}) {

+ t.Fatalf("own-region out-of-domain: targets = %v, want [http://c] (self excluded)", got)

+ }

+// A region-agnostic box (no LCHAT_MESH_REGION) never fetches peer regions, so its

+// replication is domain-scoped for EVERY route, home_region or not — the

+// byte-for-byte pre-region promise. Only the domain arm ever contributes.

+func TestMeshReplicationTargetsAgnostic(t *testing.T) {

+ clk := &fakeClock{t: time.Unix(1_700_000_000, 0)}

+ store := newFakeMeshStore(clk.now)

+ m := newMesh("http://a", store, []string{"ie.dev.11ch.at"}) // no region set

+ m.now = clk.now

+ store.setBeat("http://b", clk.now())

+ store.setRegion("http://b", "fi") // advertised, but an agnostic box never reads it

+ m.beat()

+ // A route carrying a home_region an agnostic box can't act on: out of domain ⇒

+ // nobody (not the fi box), in domain ⇒ domain scope only.

+ if got := m.replicationTargets(Route{Host: "s.fi.dev.11ch.at", Hibernate: true, HomeRegion: "fi"}); len(got) != 0 {

+ t.Fatalf("agnostic box, out-of-domain region route: targets = %v, want none", got)

+ }

+ if got := m.replicationTargets(Route{Host: "s.ie.dev.11ch.at", Hibernate: true, HomeRegion: "fi"}); !sameSet(got, []string{"http://b"}) {

+ t.Fatalf("agnostic box, in-domain route: targets = %v, want [http://b] (domain only)", got)

+ }

+// mesh-seed carries the routes a joining box owns by the same DOMAIN ∪ REGION

+// union: domain-covered routes always, plus (with --region) routes the joining

+// box owns by home_region even when out of this seed's domains. A non-hibernating

+// route and an out-of-domain route the joining box does NOT own are left behind.

+func TestMeshSeedRegionUnion(t *testing.T) {

+ // The registry the seeding (existing) box reads from.

+ regPath := filepath.Join(t.TempDir(), "registry.db")

+ src, err := openRegistry(regPath)

+ if err != nil {

+ t.Fatalf("open source registry: %v", err)

+ }

+ seed := func(host, region string, hibernate bool) {

+ if err := src.upsert(Route{Host: host, Kind: "instance", Socket: "/x.sock", Hibernate: hibernate, HomeRegion: region}); err != nil {

+ t.Fatalf("upsert %s: %v", host, err)

+ }

+ seed("dom.ie.dev.11ch.at", "ie", true) // domain-covered → seeded

+ seed("own.fi.dev.11ch.at", "fi", true) // out of domain but region-owned by the fi joiner → seeded

+ seed("other.fi.dev.11ch.at", "de", true) // out of domain, not the joiner's region → left behind

+ seed("live.ie.dev.11ch.at", "ie", false) // domain-covered but not hibernating → left behind

+ src.sql.Close() // release the file before the seed re-opens it

+ // The joining box: a fresh registry + router behind the receive handler.

+ dst, err := openRegistry(filepath.Join(t.TempDir(), "dst.db"))

+ if err != nil {

+ t.Fatalf("open dst registry: %v", err)

+ }

+ rt, err := newRouter(dst, nil)

+ if err != nil {

+ t.Fatalf("new router: %v", err)

+ }

+ mux := http.NewServeMux()

+ mux.Handle("POST /api/mesh/route", meshRouteHandler(dst, rt, "seed-secret"))

+ srv := httptest.NewServer(mux)

+ defer srv.Close()

+ t.Setenv("LCHAT_MESH_DOMAINS", "ie.dev.11ch.at")

+ t.Setenv("LCHAT_MESH_TOKEN", "seed-secret")

+ runMeshSeed([]string{"--registry", regPath, "--region", "fi", srv.URL})

+ if !rt.allowed("dom.ie.dev.11ch.at") {

+ t.Fatal("domain-covered route was not seeded")

+ }

+ if !rt.allowed("own.fi.dev.11ch.at") {

+ t.Fatal("region-owned out-of-domain route was not seeded to the box that owns it — the multi-edge gap")

+ }

+ if rt.allowed("other.fi.dev.11ch.at") {

+ t.Fatal("an out-of-domain route the joining box does not own by region was wrongly seeded")

+ }

+ if rt.allowed("live.ie.dev.11ch.at") {

+ t.Fatal("a non-hibernating route was wrongly seeded")

+ }

+// countOccurrences counts how many times want appears in xs — used to prove the

+// domain ∪ region union sends to a doubly-qualified box exactly once.

+func countOccurrences(xs []string, want string) int {

+ n := 0

+ for _, x := range xs {

+ if x == want {

+ n++

+ }

+ return n