| new file mode 100644 |
| index 0000000..5e539a8 |
| --- /dev/null |
| +++ b/docs/multi-edge.md |
| @@ -0,0 +1,195 @@ |
| +# 🤖 Multi-edge — region as a property, relocation, and replicated spaces |
| + |
| +Status: **direction agreed 2026-07-18** (Paul). This is the plan of record for |
| +how a space is *placed* (which box serves it), *relocated* (moved between |
| +regions), and eventually *replicated across edges* (served from several boxes at |
| +once). It builds on `docs/planet-scale.md` (the running single-owner system) and |
| +supersedes the "country in the hostname" scheme the dev mesh trialled |
| +(`docs/dev-mesh.md`, PR #1043; described today by the `docs/planet-scale-guide/` |
| +proving-ground page); see [What this supersedes](#what-this-supersedes). |
| + |
| +The north star is **multi-edge**: a space live on several edges at once, so every |
| +member reaches a nearby one. Two interim steps stand on their own and lay the |
| +groundwork: **stable-address relocation** and **multi-edge within a single |
| +region**. |
| + |
| +## The one idea: region is a property, not an identity |
| + |
| +A space's **address is its identity** — the name members type, the host its |
| +invite links carry, the origin its per-thread encryption keys are bound to. |
| +Identity must never change, because changing it re-keys every member. |
| + |
| +Where a space *lives* — its **region** — must be free to change: auto-assigned at |
| +creation, movable later by the space's admin, and (the endgame) spread across |
| +several places at once. So region **cannot be encoded in the address**. It is a |
| +separate, mutable property of the space — a `home_region` field — and the address |
| +stays region-neutral and permanent (`<sqid>.11ch.at`, one wildcard, no country |
| +label). |
| + |
| +This is the `home_region` model `docs/planet-scale.md` always pointed at: |
| +placement and residency are properties the infrastructure moves a space *to*, |
| +never facts baked into its name. |
| + |
| +## The enabling property: append-only + content-blind is a CRDT |
| + |
| +Everything ambitious below is tractable for one reason, and it is worth stating |
| +plainly because it is load-bearing: **our data model is an append-only log of |
| +immutable, opaque, id'd events.** Messages, edits, deletes, and reactions are all |
| +*events* — a delete is a new event, never an in-place row mutation; a reaction |
| +rides inside an event. The server stores and relays only ciphertext it cannot |
| +read. |
| + |
| +An append-only set of id'd events is, formally, a **CRDT** (a grow-only set). |
| +Merging two boxes' logs is *union → dedup by id → sort by (timestamp, id)*. There |
| +is no conflict to resolve, no data to lose, no coordination to run, and it |
| +converges regardless of the order events arrive in. The server was never going to |
| +do content-based merging — it cannot see the content — and it does not need to. |
| + |
| +This is what removes the **hardest categories** of distributed-systems work from |
| +the table entirely: |
| + |
| +- no consensus / leader election for writes, |
| +- no write-conflict resolution, |
| +- no content merges or last-writer-wins data loss. |
| + |
| +What is left is *eventual-consistency plumbing* — a much lower tier of |
| +difficulty. That is the whole reason multi-edge is a road we can walk rather than |
| +a moonshot. |
| + |
| +## Phase 1 — stable-address relocation |
| + |
| +An admin moves a space to another region; its address, invite links, and keys do |
| +not change. Members notice nothing. |
| + |
| +The mechanism reuses parts that already exist: |
| + |
| +1. **Copy** the space's store to the destination region's object store. (In the |
| + dev mesh there is one shared bucket, so this is free; in production each |
| + country has its own store, so this is a prefix-to-prefix copy — done while the |
| + space sleeps, per `docs/planet-scale.md`.) |
| +2. **Flip** `home_region`, which replicates to every box the same way routing |
| + rows already do (`meshreplicate.go`). |
| +3. **Update DNS** — the space's A record now points at the destination box (see |
| + [Dynamic DNS](#dynamic-dns-and-the-propagation-window)). |
| + |
| +### Dynamic DNS and the propagation window |
| + |
| +Keeping the *name* fixed while the *serving box* changes is exactly what |
| +API-driven dynamic DNS is for (we already use it for DNS-01 wildcard certs): the |
| +host stays `<sqid>.11ch.at`, only its A record's target moves. |
| + |
| +The DNS TTL means a propagation window where some clients still resolve to the |
| +old box. **No live replication is needed to cover it** — the |
| +[single-writer lease](planet-scale.md) plus the mesh's proxy-to-owner already do. |
| +A request that lands on the old box finds `home_region` now points elsewhere and |
| +**proxies to the new owner**, exactly as it already does for any wrong-box |
| +request. The lease guarantees only the new box ever writes, so there is no |
| +dual-writer and no split-brain — the window costs at most a proxy hop, not |
| +correctness. |
| + |
| +So Phase 1 is mostly existing machinery (hibernation + lease + proxy + the |
| +route-replication plumbing) plus the `home_region` field, region-neutral |
| +addressing, and placement-by-region. It is the smallest change that delivers the |
| +stable-address promise. |
| + |
| +## Phase 2 — multi-edge within a region |
| + |
| +Replicate a space's log across the boxes of *one* region. Same-jurisdiction, so |
| +**no residency promise is affected** — this is pure resilience and latency, and |
| +it is where the merge/gossip layer gets validated on the easy case (all edges in |
| +one country, no cross-border concerns). |
| + |
| +Each edge accepts appends locally and propagates them to its peers; the CRDT |
| +merge means every edge converges without coordination. The single-writer lease — |
| +which today exists to stop two processes corrupting one SQLite file — relaxes |
| +into **per-edge logs that merge**, rather than one file one writer. A box failing |
| +no longer strands its spaces mid-write; its peers already hold the log. |
| + |
| +## Phase 3 — cross-region multi-edge (the goal) |
| + |
| +The same replication, now spanning regions: a space served from edges in several |
| +countries, each member reaching the nearest. This is the north star — and the one |
| +place a real trade-off surfaces, below. |
| + |
| +## What append-only does *not* solve |
| + |
| +Honest remaining work — the eventual-consistency plumbing the CRDT leaves us: |
| + |
| +- **Ordering / causal consistency.** Union+sort converges, but "a reply appears |
| + after its parent" and "read-your-writes" want causal metadata (a per-sender |
| + sequence or a parent ref) carried *inside* the encrypted event. Today there is a |
| + real server-assigned monotonic sequence (the subscription watermark); multi-edge |
| + moves that to a partial order (timestamp+id, or Lamport clocks) instead of one |
| + counter. Chat tolerates weak ordering well, so this is manageable, not scary. |
| +- **Ephemeral fan-out.** Typing, presence, and relay-only frames are *not* log |
| + entries — they are live signals, and the WebSocket hub is per-box today. |
| + Multi-edge needs a cross-edge pub/sub for them. New, but small. |
| +- **Real-time delivery.** A message appended at edge A must reach a subscriber on |
| + edge B *now*, not at anti-entropy time — so edges want fast push to each other, |
| + not only periodic reconciliation. |
| +- **Server-side watermarks.** Read-marks are already forward-only max-wins |
| + (CRDT-friendly). Scheduled future-messages (the sweep goroutine, `miniapps.go`) |
| + need one edge to own the timer or idempotent delivery. A handful of these, each |
| + solvable. |
| +- **Cost.** N copies of the data and gossip traffic — a real but bounded price. |
| + |
| +## Residency ⟂ multi-edge — two modes, not both |
| + |
| +Multi-edge and data residency pull in opposite directions: multi-edge **spreads** |
| +the ciphertext across edges (and countries) for reach and resilience; residency |
| +**pins** it to one country. They cannot both hold for the same space at once. |
| + |
| +So they are **modes a space's owner chooses**, not a global setting: |
| + |
| +- **Residency** — one region, data stays put; the guarantee a country subdomain |
| + can make. |
| +- **Multi-edge within a jurisdiction** — the sweet spot: resilience and lower |
| + latency with *no* residency promise broken. |
| +- **Cross-region multi-edge** — maximum reach, explicitly trading the residency |
| + guarantee for it. |
| + |
| +Relocation (Phase 1) is available in every mode — it is just "change where the |
| +one copy lives" in the residency/single-edge case. |
| + |
| +## What this supersedes |
| + |
| +The dev mesh trialled **country in the hostname** (`<sqid>.<cc>.dev.11ch.at`, |
| +per-country wildcards, placement scoped by the host's country label — the |
| +domain-scoped mesh in `mesh.go`/`heartbeat.go`/`meshreplicate.go`). That made |
| +region an *immutable identity*, which is precisely what the stable-address |
| +requirement rules out. It was a useful stepping stone; this model replaces it: |
| + |
| +- **Addresses go region-neutral** (`<sqid>.11ch.at`, one wildcard). The country |
| + leaves the name. |
| +- **Placement keys on `home_region`, not the host label.** The per-box |
| + advertisement plumbing that domain-scoping added (each box publishing its |
| + capability via a sibling S3 object) is **reused** — a box now advertises its |
| + *region* instead of its host-domains, and placement/replication scope to the |
| + box(es) whose region matches the space's `home_region`. |
| +- `docs/dev-mesh.md` and the `docs/planet-scale-guide/` "proving ground" page |
| + describe the country-in-host scheme that runs on the dev mesh *today*; they stay |
| + accurate until the phases below land, then are reworked in step. |
| + |
| +## Status and build order |
| + |
| +Nothing here is operable yet; the pieces it reuses are. |
| + |
| +- The **append-only, content-blind data model** it rests on: *running today* — it |
| + is how every space already works. |
| +- **Hibernation + single-writer lease + proxy-to-owner** (the machinery Phase 1 |
| + reuses): *running today* (`docs/planet-scale.md`). |
| +- **Region-neutral addressing + `home_region` placement + relocation** (Phase 1): |
| + *designed, not built*. |
| +- **Multi-edge within a region** (Phase 2) and **cross-region multi-edge** |
| + (Phase 3): *designed, not built*. |
| + |
| +Build order, each its own reviewable slice: |
| + |
| +1. **Phase 1 — stable-address relocation.** `home_region` field, region-neutral |
| + addressing, mesh places by region, relocation via lease + proxy + cross-region |
| + copy + dynamic DNS. Mostly existing parts; the foundation everything else sits |
| + on. |
| +2. **Phase 2 — multi-edge within a region.** The log-merge / gossip layer, |
| + validated where there are no residency concerns. |
| +3. **Phase 3 — cross-region multi-edge.** The same, spanning regions, as an |
| + explicit owner mode. |
amadan