| 1 | # 🤖 Multi-edge — region as a property, relocation, and replicated spaces |
| 2 | |
| 3 | Status: **direction agreed 2026-07-18** (Paul). This is the plan of record for |
| 4 | how a space is *placed* (which box serves it), *relocated* (moved between |
| 5 | regions), and eventually *replicated across edges* (served from several boxes at |
| 6 | once). It builds on `docs/planet-scale.md` (the running single-owner system) and |
| 7 | supersedes the "country in the hostname" scheme the dev mesh trialled |
| 8 | (`docs/dev-mesh.md`, PR #1043; described today by the `docs/planet-scale-guide/` |
| 9 | proving-ground page); see [What this supersedes](#what-this-supersedes). |
| 10 | |
| 11 | The north star is **multi-edge**: a space live on several edges at once, so every |
| 12 | member reaches a nearby one. Two interim steps stand on their own and lay the |
| 13 | groundwork: **stable-address relocation** and **multi-edge within a single |
| 14 | region**. |
| 15 | |
| 16 | ## The one idea: region is a property, not an identity |
| 17 | |
| 18 | A space's **address is its identity** — the name members type, the host its |
| 19 | invite links carry, the origin its per-thread encryption keys are bound to. |
| 20 | Identity must never change, because changing it re-keys every member. |
| 21 | |
| 22 | Where a space *lives* — its **region** — must be free to change: auto-assigned at |
| 23 | creation, movable later by the space's admin, and (the endgame) spread across |
| 24 | several places at once. So region **cannot be encoded in the address**. It is a |
| 25 | separate, mutable property of the space — a `home_region` field — and the address |
| 26 | stays region-neutral and permanent (`<sqid>.11ch.at`, one wildcard, no country |
| 27 | label). |
| 28 | |
| 29 | This is the `home_region` model `docs/planet-scale.md` always pointed at: |
| 30 | placement and residency are properties the infrastructure moves a space *to*, |
| 31 | never facts baked into its name. |
| 32 | |
| 33 | ## The enabling property: append-only + content-blind is a CRDT |
| 34 | |
| 35 | Everything ambitious below is tractable for one reason, and it is worth stating |
| 36 | plainly because it is load-bearing: **our data model is an append-only log of |
| 37 | immutable, opaque, id'd events.** Messages, edits, deletes, and reactions are all |
| 38 | *events* — a delete is a new event, never an in-place row mutation; a reaction |
| 39 | rides inside an event. The server stores and relays only ciphertext it cannot |
| 40 | read. |
| 41 | |
| 42 | An append-only set of id'd events is, formally, a **CRDT** (a grow-only set). |
| 43 | Merging two boxes' logs is *union → dedup by id → sort by (timestamp, id)*. There |
| 44 | is no conflict to resolve, no data to lose, no coordination to run, and it |
| 45 | converges regardless of the order events arrive in. The server was never going to |
| 46 | do content-based merging — it cannot see the content — and it does not need to. |
| 47 | |
| 48 | This is what removes the **hardest categories** of distributed-systems work from |
| 49 | the table entirely: |
| 50 | |
| 51 | - no consensus / leader election for writes, |
| 52 | - no write-conflict resolution, |
| 53 | - no content merges or last-writer-wins data loss. |
| 54 | |
| 55 | What is left is *eventual-consistency plumbing* — a much lower tier of |
| 56 | difficulty. That is the whole reason multi-edge is a road we can walk rather than |
| 57 | a moonshot. |
| 58 | |
| 59 | ## Phase 1 — stable-address relocation |
| 60 | |
| 61 | An admin moves a space to another region; its address, invite links, and keys do |
| 62 | not change. Members notice nothing. |
| 63 | |
| 64 | The mechanism reuses parts that already exist: |
| 65 | |
| 66 | 1. **Copy** the space's store to the destination region's object store. (In the |
| 67 | dev mesh there is one shared bucket, so this is free; in production each |
| 68 | country has its own store, so this is a prefix-to-prefix copy — done while the |
| 69 | space sleeps, per `docs/planet-scale.md`.) |
| 70 | 2. **Flip** `home_region`, which replicates to every box the same way routing |
| 71 | rows already do (`meshreplicate.go`). |
| 72 | 3. **Update DNS** — the space's A record now points at the destination box (see |
| 73 | [Dynamic DNS](#dynamic-dns-and-the-propagation-window)). |
| 74 | |
| 75 | ### Dynamic DNS and the propagation window |
| 76 | |
| 77 | Keeping the *name* fixed while the *serving box* changes is exactly what |
| 78 | API-driven dynamic DNS is for (we already use it for DNS-01 wildcard certs): the |
| 79 | host stays `<sqid>.11ch.at`, only its A record's target moves. |
| 80 | |
| 81 | The DNS TTL means a propagation window where some clients still resolve to the |
| 82 | old box. **No live replication is needed to cover it** — the |
| 83 | [single-writer lease](planet-scale.md) plus the mesh's proxy-to-owner already do. |
| 84 | A request that lands on the old box finds `home_region` now points elsewhere and |
| 85 | **proxies to the new owner**, exactly as it already does for any wrong-box |
| 86 | request. The lease guarantees only the new box ever writes, so there is no |
| 87 | dual-writer and no split-brain — the window costs at most a proxy hop, not |
| 88 | correctness. |
| 89 | |
| 90 | So Phase 1 is mostly existing machinery (hibernation + lease + proxy + the |
| 91 | route-replication plumbing) plus the `home_region` field, region-neutral |
| 92 | addressing, and placement-by-region. It is the smallest change that delivers the |
| 93 | stable-address promise. |
| 94 | |
| 95 | ### The `lchat relocate` command |
| 96 | |
| 97 | The relocate action is a subcommand of the one binary — `lchat relocate <host> |
| 98 | --to <region>` (`relocate.go`, a reusable `relocate()` core a future controld |
| 99 | RPC or member-facing "change region" button can call). It runs the three steps |
| 100 | above in order: |
| 101 | |
| 102 | 1. **Store copy (or the dev no-op).** A pure decision helper, |
| 103 | `relocationStorePlan(src, dst)`, returns `skip=true` exactly when source and |
| 104 | destination name the same bytes — same bucket, same per-host prefix, same |
| 105 | endpoint. That is the **dev mesh's one-shared-bucket case**, so the copy is a |
| 106 | no-op there. In production each region has its own store, so it is a real |
| 107 | prefix-to-prefix copy of `s3://<bucket>/<host>/…` (`copyS3Prefix`), run while |
| 108 | the space sleeps. The copy deliberately **excludes the lease object** |
| 109 | (`lock.json`): a copied, still-valid lease would make the destination's first |
| 110 | `AcquireLease` see a held lock and 503 until the TTL expired — the destination |
| 111 | must take its own fresh lease. |
| 112 | 2. **Flip `home_region` + re-fan-out.** `setHomeRegion` writes the replicated |
| 113 | routes column and `fanOutMeshRoute` pushes the updated route to peers, so every |
| 114 | box's `candidatesFor` (`mesh.go`) re-homes the space onto a box in the |
| 115 | destination region. Route *edits* aren't auto-replicated by the normal path, so |
| 116 | this is the deliberate targeted re-fan-out of the single updated route. The |
| 117 | local router needs **no reload** — its mesh reads `home_region` live from the |
| 118 | shared registry on each placement decision. |
| 119 | 3. **Optional A-record move.** `--move-dns --dns-target <ip>` repoints the host's |
| 120 | A record at the destination box to avoid the proxy hop. **Off by default** — the |
| 121 | `*.<domain>` wildcard + proxy-to-owner already route any request to the new |
| 122 | owner through the DNS-TTL window, so relocation is correct with zero DNS write |
| 123 | perms. Only the Route53 provider implements the A upsert today |
| 124 | (`route53dns.go`); DNSimple refuses `--move-dns` with a clear message. |
| 125 | |
| 126 | **Preconditions the command enforces:** the space must be an in-scope, `ready`, |
| 127 | **hibernating** instance (a live systemd instance is box-pinned — the message |
| 128 | tells you to `lchat hibernate <host> --on` first), and a **live mesh box must |
| 129 | advertise the target region** (else the flip re-homes onto nothing). Run it on a |
| 130 | box whose registry has the route (the source region's box) with the mesh env |
| 131 | sourced (`/data/router.env`) — `sudo` strips `LCHAT_MESH_*`, the same gotcha the |
| 132 | dev-cutover runbook's Step 5 calls out. |
| 133 | |
| 134 | **Why the single-writer lease makes it data-safe.** Relocation never split-brains |
| 135 | because the S3 lease (`internal/activator`, `lock.json` under the space's prefix) |
| 136 | is the one truth of who may write; `home_region` and DNS are only placement hints: |
| 137 | |
| 138 | - *Shared bucket (dev):* both regions contend on the **same** `lock.json`, so the |
| 139 | lease is globally singular. After the flip the destination is the HRW owner; if |
| 140 | the source is momentarily still awake it holds the lease, the destination's |
| 141 | `AcquireLease` fails loudly (`LeaseExistsError` → 503 + Retry-After), and the |
| 142 | client's retry lands once the source sleeps and releases. Never two writers. |
| 143 | - *Per-region buckets (prod):* the copy runs while the source is **asleep** (lease |
| 144 | released, replica fully flushed by the sleep-time `SyncAndWait`), so the |
| 145 | destination restores a complete copy and takes a fresh lease in its own bucket. |
| 146 | Once `home_region` is flipped the source's `owns()` returns false, so it never |
| 147 | wakes there again — a stray request proxies to the new owner instead. |
| 148 | |
| 149 | **Idempotency + the copy-clobber guard.** Once `home_region` already equals the |
| 150 | target, the space's authoritative copy is the **destination's**, so a re-run must |
| 151 | **not** re-copy source→dest (that would overwrite newer destination writes with |
| 152 | stale source bytes). The copy + flip therefore run **only on the forward move** |
| 153 | (`current != target`); an already-home re-run just heals peers by re-fanning-out |
| 154 | the route. A crash between copy and flip converges safely — the source is still |
| 155 | authoritative (not yet flipped), so the next run re-copies (an idempotent |
| 156 | overwrite) and flips. |
| 157 | |
| 158 | **Tested vs. needs live proof.** Unit tests cover the pure store-plan decision |
| 159 | (skip vs copy), `copyS3Prefix` (key layout preserved, `lock.json` excluded), the |
| 160 | `home_region` flip + idempotency guard, target-region validation, and the Route53 |
| 161 | A upsert. The **dev shared-bucket path (skip) is what a live dev relocation |
| 162 | exercises**; the **prod cross-bucket copy is unproven in production** — it needs a |
| 163 | per-region-bucket dev/prod trial, and two design points remain open: region→bucket |
| 164 | auto-discovery (today the operator names the destination with `--dst-bucket`) and |
| 165 | cross-*account*/cross-*endpoint* (R2) copies, which need a download+reupload |
| 166 | variant rather than server-side `CopyObject`. Both are follow-ons. |
| 167 | |
| 168 | ## Phase 2 — multi-edge within a region |
| 169 | |
| 170 | Replicate a space's log across the boxes of *one* region. Same-jurisdiction, so |
| 171 | **no residency promise is affected** — this is pure resilience and latency, and |
| 172 | it is where the merge/gossip layer gets validated on the easy case (all edges in |
| 173 | one country, no cross-border concerns). |
| 174 | |
| 175 | Each edge accepts appends locally and propagates them to its peers; the CRDT |
| 176 | merge means every edge converges without coordination. The single-writer lease — |
| 177 | which today exists to stop two processes corrupting one SQLite file — relaxes |
| 178 | into **per-edge logs that merge**, rather than one file one writer. A box failing |
| 179 | no longer strands its spaces mid-write; its peers already hold the log. |
| 180 | |
| 181 | ## Phase 3 — cross-region multi-edge (the goal) |
| 182 | |
| 183 | The same replication, now spanning regions: a space served from edges in several |
| 184 | countries, each member reaching the nearest. This is the north star — and the one |
| 185 | place a real trade-off surfaces, below. |
| 186 | |
| 187 | ## What append-only does *not* solve |
| 188 | |
| 189 | Honest remaining work — the eventual-consistency plumbing the CRDT leaves us: |
| 190 | |
| 191 | - **Ordering / causal consistency.** Union+sort converges, but "a reply appears |
| 192 | after its parent" and "read-your-writes" want causal metadata (a per-sender |
| 193 | sequence or a parent ref) carried *inside* the encrypted event. Today there is a |
| 194 | real server-assigned monotonic sequence (the subscription watermark); multi-edge |
| 195 | moves that to a partial order (timestamp+id, or Lamport clocks) instead of one |
| 196 | counter. Chat tolerates weak ordering well, so this is manageable, not scary. |
| 197 | - **Ephemeral fan-out.** Typing, presence, and relay-only frames are *not* log |
| 198 | entries — they are live signals, and the WebSocket hub is per-box today. |
| 199 | Multi-edge needs a cross-edge pub/sub for them. New, but small. |
| 200 | - **Real-time delivery.** A message appended at edge A must reach a subscriber on |
| 201 | edge B *now*, not at anti-entropy time — so edges want fast push to each other, |
| 202 | not only periodic reconciliation. |
| 203 | - **Server-side watermarks.** Read-marks are already forward-only max-wins |
| 204 | (CRDT-friendly). Scheduled future-messages (the sweep goroutine, `miniapps.go`) |
| 205 | need one edge to own the timer or idempotent delivery. A handful of these, each |
| 206 | solvable. |
| 207 | - **Cost.** N copies of the data and gossip traffic — a real but bounded price. |
| 208 | |
| 209 | ## Residency ⟂ multi-edge — two modes, not both |
| 210 | |
| 211 | Multi-edge and data residency pull in opposite directions: multi-edge **spreads** |
| 212 | the ciphertext across edges (and countries) for reach and resilience; residency |
| 213 | **pins** it to one country. They cannot both hold for the same space at once. |
| 214 | |
| 215 | So they are **modes a space's owner chooses**, not a global setting: |
| 216 | |
| 217 | - **Residency** — one region, data stays put; the guarantee a country subdomain |
| 218 | can make. |
| 219 | - **Multi-edge within a jurisdiction** — the sweet spot: resilience and lower |
| 220 | latency with *no* residency promise broken. |
| 221 | - **Cross-region multi-edge** — maximum reach, explicitly trading the residency |
| 222 | guarantee for it. |
| 223 | |
| 224 | Relocation (Phase 1) is available in every mode — it is just "change where the |
| 225 | one copy lives" in the residency/single-edge case. |
| 226 | |
| 227 | ## What this supersedes |
| 228 | |
| 229 | The dev mesh trialled **country in the hostname** (`<sqid>.<cc>.dev.11ch.at`, |
| 230 | per-country wildcards, placement scoped by the host's country label — the |
| 231 | domain-scoped mesh in `mesh.go`/`heartbeat.go`/`meshreplicate.go`). That made |
| 232 | region an *immutable identity*, which is precisely what the stable-address |
| 233 | requirement rules out. It was a useful stepping stone; this model replaces it: |
| 234 | |
| 235 | - **Addresses go region-neutral** (`<sqid>.11ch.at`, one wildcard). The country |
| 236 | leaves the name. |
| 237 | - **Placement keys on `home_region`, not the host label.** Each box advertises |
| 238 | its region in its mesh heartbeat body; every box learns its peers' regions and |
| 239 | scopes placement to the box(es) whose region matches a space's `home_region` |
| 240 | (`mesh.go` `candidatesFor`, keying on the `home_region` the registry already |
| 241 | replicates onto every box). The country-in-host trial's domain-scoping lived |
| 242 | in an unmerged branch, so there was no prior per-box advertisement to reuse — |
| 243 | the region rides the heartbeat every box already reads for membership, and the |
| 244 | fetch is skipped entirely on a box that declares no region (zero cost when a |
| 245 | mesh has not opted into regions). |
| 246 | - `docs/dev-mesh.md` and the `docs/planet-scale-guide/` "proving ground" page |
| 247 | describe the country-in-host scheme that runs on the dev mesh *today*; they stay |
| 248 | accurate until the phases below land, then are reworked in step. |
| 249 | |
| 250 | ## Status and build order |
| 251 | |
| 252 | Phase 1 is now largely built; the pieces it reuses were always running. |
| 253 | |
| 254 | - The **append-only, content-blind data model** it rests on: *running today* — it |
| 255 | is how every space already works. |
| 256 | - **Hibernation + single-writer lease + proxy-to-owner** (the machinery Phase 1 |
| 257 | reuses): *running today* (`docs/planet-scale.md`). |
| 258 | - **Region-neutral addressing + `home_region` placement** (Phase 1): *built + |
| 259 | proven live on the dev mesh* — one shared `dev.11ch.at` namespace, a shared |
| 260 | wildcard, cross-region serve returning 200 (the dev cutover runbook, |
| 261 | `docs/multi-edge-dev-cutover.md`, lands with the cutover PR). |
| 262 | - **The `lchat relocate` action** (Phase 1's last slice): *built; the dev |
| 263 | shared-bucket path (skip-copy) lands on the proven cross-region serve; the prod |
| 264 | per-region-bucket copy is implemented + unit-tested but unproven in production* |
| 265 | (see [The `lchat relocate` command](#the-lchat-relocate-command)). |
| 266 | - **Multi-edge within a region** (Phase 2) and **cross-region multi-edge** |
| 267 | (Phase 3): *designed, not built*. |
| 268 | |
| 269 | Build order, each its own reviewable slice: |
| 270 | |
| 271 | 1. **Phase 1 — stable-address relocation.** `home_region` field, region-neutral |
| 272 | addressing, mesh places by region, relocation via lease + proxy + cross-region |
| 273 | copy + dynamic DNS. Mostly existing parts; the foundation everything else sits |
| 274 | on. |
| 275 | 2. **Phase 2 — multi-edge within a region.** The log-merge / gossip layer, |
| 276 | validated where there are no residency concerns. |
| 277 | 3. **Phase 3 — cross-region multi-edge.** The same, spanning regions, as an |
| 278 | explicit owner mode. |
| 279 | |
amadan