🤖 Multi-edge — region as a property, relocation, and replicated spaces

Status: direction agreed 2026-07-18 (Paul). This is the plan of record for how a space is placed (which box serves it), relocated (moved between regions), and eventually replicated across edges (served from several boxes at once). It builds on docs/planet-scale.md (the running single-owner system) and supersedes the "country in the hostname" scheme the dev mesh trialled (docs/dev-mesh.md, PR #1043; described today by the docs/planet-scale-guide/ proving-ground page); see What this supersedes.

The north star is multi-edge: a space live on several edges at once, so every member reaches a nearby one. Two interim steps stand on their own and lay the groundwork: stable-address relocation and multi-edge within a single region.

The one idea: region is a property, not an identity

A space's address is its identity — the name members type, the host its invite links carry, the origin its per-thread encryption keys are bound to. Identity must never change, because changing it re-keys every member.

Where a space lives — its region — must be free to change: auto-assigned at creation, movable later by the space's admin, and (the endgame) spread across several places at once. So region cannot be encoded in the address. It is a separate, mutable property of the space — a home_region field — and the address stays region-neutral and permanent (<sqid>.11ch.at, one wildcard, no country label).

This is the home_region model docs/planet-scale.md always pointed at: placement and residency are properties the infrastructure moves a space to, never facts baked into its name.

The enabling property: append-only + content-blind is a CRDT

Everything ambitious below is tractable for one reason, and it is worth stating plainly because it is load-bearing: our data model is an append-only log of immutable, opaque, id'd events. Messages, edits, deletes, and reactions are all events — a delete is a new event, never an in-place row mutation; a reaction rides inside an event. The server stores and relays only ciphertext it cannot read.

An append-only set of id'd events is, formally, a CRDT (a grow-only set). Merging two boxes' logs is union → dedup by id → sort by (timestamp, id). There is no conflict to resolve, no data to lose, no coordination to run, and it converges regardless of the order events arrive in. The server was never going to do content-based merging — it cannot see the content — and it does not need to.

This is what removes the hardest categories of distributed-systems work from the table entirely:

• no consensus / leader election for writes,
• no write-conflict resolution,
• no content merges or last-writer-wins data loss.

What is left is eventual-consistency plumbing — a much lower tier of difficulty. That is the whole reason multi-edge is a road we can walk rather than a moonshot.

Phase 1 — stable-address relocation

An admin moves a space to another region; its address, invite links, and keys do not change. Members notice nothing.

The mechanism reuses parts that already exist:

1. Copy the space's store to the destination region's object store. (In the dev mesh there is one shared bucket, so this is free; in production each country has its own store, so this is a prefix-to-prefix copy — done while the space sleeps, per docs/planet-scale.md.)
2. Flip home_region, which replicates to every box the same way routing rows already do (meshreplicate.go).
3. Update DNS — the space's A record now points at the destination box (see Dynamic DNS).

Dynamic DNS and the propagation window

Keeping the name fixed while the serving box changes is exactly what API-driven dynamic DNS is for (we already use it for DNS-01 wildcard certs): the host stays <sqid>.11ch.at, only its A record's target moves.

The DNS TTL means a propagation window where some clients still resolve to the old box. No live replication is needed to cover it — the single-writer lease plus the mesh's proxy-to-owner already do. A request that lands on the old box finds home_region now points elsewhere and proxies to the new owner, exactly as it already does for any wrong-box request. The lease guarantees only the new box ever writes, so there is no dual-writer and no split-brain — the window costs at most a proxy hop, not correctness.

So Phase 1 is mostly existing machinery (hibernation + lease + proxy + the route-replication plumbing) plus the home_region field, region-neutral addressing, and placement-by-region. It is the smallest change that delivers the stable-address promise.

The lchat relocate command

The relocate action is a subcommand of the one binary — lchat relocate <host> --to <region> (relocate.go, a reusable relocate() core a future controld RPC or member-facing "change region" button can call). It runs the three steps above in order:

1. Store copy (or the dev no-op). A pure decision helper, relocationStorePlan(src, dst), returns skip=true exactly when source and destination name the same bytes — same bucket, same per-host prefix, same endpoint. That is the dev mesh's one-shared-bucket case, so the copy is a no-op there. In production each region has its own store, so it is a real prefix-to-prefix copy of s3://<bucket>/<host>/… (copyS3Prefix), run while the space sleeps. The copy deliberately excludes the lease object (lock.json): a copied, still-valid lease would make the destination's first AcquireLease see a held lock and 503 until the TTL expired — the destination must take its own fresh lease.
2. Flip home_region + re-fan-out. setHomeRegion writes the replicated routes column and fanOutMeshRoute pushes the updated route to peers, so every box's candidatesFor (mesh.go) re-homes the space onto a box in the destination region. Route edits aren't auto-replicated by the normal path, so this is the deliberate targeted re-fan-out of the single updated route. The local router needs no reload — its mesh reads home_region live from the shared registry on each placement decision.
3. Optional A-record move. --move-dns --dns-target <ip> repoints the host's A record at the destination box to avoid the proxy hop. Off by default — the *.<domain> wildcard + proxy-to-owner already route any request to the new owner through the DNS-TTL window, so relocation is correct with zero DNS write perms. Only the Route53 provider implements the A upsert today (route53dns.go); DNSimple refuses --move-dns with a clear message.

Preconditions the command enforces: the space must be an in-scope, ready, hibernating instance (a live systemd instance is box-pinned — the message tells you to lchat hibernate <host> --on first), and a live mesh box must advertise the target region (else the flip re-homes onto nothing). Run it on a box whose registry has the route (the source region's box) with the mesh env sourced (/data/router.env) — sudo strips LCHAT_MESH_*, the same gotcha the dev-cutover runbook's Step 5 calls out.

Why the single-writer lease makes it data-safe. Relocation never split-brains because the S3 lease (internal/activator, lock.json under the space's prefix) is the one truth of who may write; home_region and DNS are only placement hints:

• Shared bucket (dev): both regions contend on the same lock.json, so the lease is globally singular. After the flip the destination is the HRW owner; if the source is momentarily still awake it holds the lease, the destination's AcquireLease fails loudly (LeaseExistsError → 503 + Retry-After), and the client's retry lands once the source sleeps and releases. Never two writers.
• Per-region buckets (prod): the copy runs while the source is asleep (lease released, replica fully flushed by the sleep-time SyncAndWait), so the destination restores a complete copy and takes a fresh lease in its own bucket. Once home_region is flipped the source's owns() returns false, so it never wakes there again — a stray request proxies to the new owner instead.

Idempotency + the copy-clobber guard. Once home_region already equals the target, the space's authoritative copy is the destination's, so a re-run must not re-copy source→dest (that would overwrite newer destination writes with stale source bytes). The copy + flip therefore run only on the forward move (current != target); an already-home re-run just heals peers by re-fanning-out the route. A crash between copy and flip converges safely — the source is still authoritative (not yet flipped), so the next run re-copies (an idempotent overwrite) and flips.

Tested vs. needs live proof. Unit tests cover the pure store-plan decision (skip vs copy), copyS3Prefix (key layout preserved, lock.json excluded), the home_region flip + idempotency guard, target-region validation, and the Route53 A upsert. The dev shared-bucket path (skip) is what a live dev relocation exercises; the prod cross-bucket copy is unproven in production — it needs a per-region-bucket dev/prod trial, and two design points remain open: region→bucket auto-discovery (today the operator names the destination with --dst-bucket) and cross-account/cross-endpoint (R2) copies, which need a download+reupload variant rather than server-side CopyObject. Both are follow-ons.

Phase 2 — multi-edge within a region

Replicate a space's log across the boxes of one region. Same-jurisdiction, so no residency promise is affected — this is pure resilience and latency, and it is where the merge/gossip layer gets validated on the easy case (all edges in one country, no cross-border concerns).

Each edge accepts appends locally and propagates them to its peers; the CRDT merge means every edge converges without coordination. The single-writer lease — which today exists to stop two processes corrupting one SQLite file — relaxes into per-edge logs that merge, rather than one file one writer. A box failing no longer strands its spaces mid-write; its peers already hold the log.

Phase 3 — cross-region multi-edge (the goal)

The same replication, now spanning regions: a space served from edges in several countries, each member reaching the nearest. This is the north star — and the one place a real trade-off surfaces, below.

What append-only does not solve

Honest remaining work — the eventual-consistency plumbing the CRDT leaves us:

• Ordering / causal consistency. Union+sort converges, but "a reply appears after its parent" and "read-your-writes" want causal metadata (a per-sender sequence or a parent ref) carried inside the encrypted event. Today there is a real server-assigned monotonic sequence (the subscription watermark); multi-edge moves that to a partial order (timestamp+id, or Lamport clocks) instead of one counter. Chat tolerates weak ordering well, so this is manageable, not scary.
• Ephemeral fan-out. Typing, presence, and relay-only frames are not log entries — they are live signals, and the WebSocket hub is per-box today. Multi-edge needs a cross-edge pub/sub for them. New, but small.
• Real-time delivery. A message appended at edge A must reach a subscriber on edge B now, not at anti-entropy time — so edges want fast push to each other, not only periodic reconciliation.
• Server-side watermarks. Read-marks are already forward-only max-wins (CRDT-friendly). Scheduled future-messages (the sweep goroutine, miniapps.go) need one edge to own the timer or idempotent delivery. A handful of these, each solvable.
• Cost. N copies of the data and gossip traffic — a real but bounded price.

Residency ⟂ multi-edge — two modes, not both

Multi-edge and data residency pull in opposite directions: multi-edge spreads the ciphertext across edges (and countries) for reach and resilience; residency pins it to one country. They cannot both hold for the same space at once.

So they are modes a space's owner chooses, not a global setting:

• Residency — one region, data stays put; the guarantee a country subdomain can make.
• Multi-edge within a jurisdiction — the sweet spot: resilience and lower latency with no residency promise broken.
• Cross-region multi-edge — maximum reach, explicitly trading the residency guarantee for it.

Relocation (Phase 1) is available in every mode — it is just "change where the one copy lives" in the residency/single-edge case.

What this supersedes

The dev mesh trialled country in the hostname (<sqid>.<cc>.dev.11ch.at, per-country wildcards, placement scoped by the host's country label — the domain-scoped mesh in mesh.go/heartbeat.go/meshreplicate.go). That made region an immutable identity, which is precisely what the stable-address requirement rules out. It was a useful stepping stone; this model replaces it:

• Addresses go region-neutral (<sqid>.11ch.at, one wildcard). The country leaves the name.
• Placement keys on home_region, not the host label. Each box advertises its region in its mesh heartbeat body; every box learns its peers' regions and scopes placement to the box(es) whose region matches a space's home_region (mesh.go candidatesFor, keying on the home_region the registry already replicates onto every box). The country-in-host trial's domain-scoping lived in an unmerged branch, so there was no prior per-box advertisement to reuse — the region rides the heartbeat every box already reads for membership, and the fetch is skipped entirely on a box that declares no region (zero cost when a mesh has not opted into regions).
• docs/dev-mesh.md and the docs/planet-scale-guide/ "proving ground" page describe the country-in-host scheme that runs on the dev mesh today; they stay accurate until the phases below land, then are reworked in step.

Status and build order

Phase 1 is now largely built; the pieces it reuses were always running.

• The append-only, content-blind data model it rests on: running today — it is how every space already works.
• Hibernation + single-writer lease + proxy-to-owner (the machinery Phase 1 reuses): running today (docs/planet-scale.md).
• Region-neutral addressing + home_region placement (Phase 1): built + proven live on the dev mesh — one shared dev.11ch.at namespace, a shared wildcard, cross-region serve returning 200 (the dev cutover runbook, docs/multi-edge-dev-cutover.md, lands with the cutover PR).
• The lchat relocate action (Phase 1's last slice): built; the dev shared-bucket path (skip-copy) lands on the proven cross-region serve; the prod per-region-bucket copy is implemented + unit-tested but unproven in production (see The lchat relocate command).
• Multi-edge within a region (Phase 2) and cross-region multi-edge (Phase 3): designed, not built.

Build order, each its own reviewable slice:

1. Phase 1 — stable-address relocation. home_region field, region-neutral addressing, mesh places by region, relocation via lease + proxy + cross-region copy + dynamic DNS. Mostly existing parts; the foundation everything else sits on.
2. Phase 2 — multi-edge within a region. The log-merge / gossip layer, validated where there are no residency concerns.
3. Phase 3 — cross-region multi-edge. The same, spanning regions, as an explicit owner mode.