amadan
docs: planet-scale.md — the mesh moved from roadmap to running (#678, #681) (#683)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+34
−6
docs/planet-scale.md+34 −6
diff --git a/docs/planet-scale.md b/docs/planet-scale.md| index d6bc78c..b18110c 100644 |
| --- a/docs/planet-scale.md |
| +++ b/docs/planet-scale.md |
| @@ -230,6 +230,39 @@ immortal. The stats handler now answers those **definitionally from the |
| registry row** (0 members, never set up, no content); a missing DB on a |
| *non*-hibernating row stays a 404, meaning a failed create, skipped as before. |
| +### The mesh: multiple boxes, one owner per instance |
| + |
| +The multi-box mechanism from Part 4 v2 shipped as PR #678, gated on |
| +`LCHAT_MESH_SELF` (a peer-reachable base URL); absent means zero mesh |
| +codepaths, and the current single-box deployment runs exactly as before. |
| + |
| +- **Membership is S3 heartbeats**, not gossip: each box PUTs one object per |
| + 15 seconds under the hibernation prefix's `mesh/members/`, and the object |
| + *key* is the record (base64url of the peer URL) so a single LIST yields the |
| + whole membership. Liveness reads S3's own `LastModified`, so a box with a |
| + wrong clock can't poison the staleness judgment. Stale after 45s. |
| +- **Rendezvous (HRW) hashing is the placement hint** — `owner(host)` is a |
| + pure function of the live member set; adding a box moves only the hosts |
| + that land on it. **The lease stays the truth**: a box that wrongly believes |
| + it owns a host loses the CAS race loudly and proxies instead, so transient |
| + membership disagreement structurally cannot split-brain. |
| +- **Wrong-box requests reverse-proxy to the owner**, TLS verified against the |
| + instance hostname with the connection pinned to the peer's address (no |
| + `InsecureSkipVerify` anywhere); a one-hop marker turns disagreement loops |
| + into a wrong-box wake the lease resolves. |
| +- **Drain deletes the heartbeat first**, then checkpoints and releases as |
| + usual — so deploys, scale-down, and box death are the same code path. The |
| + kill-a-box property is a live test, both ways: a drained owner's instances |
| + are serving on the survivor ~2 seconds later; a SIGKILLed owner's within |
| + ~30 (the lease TTL), data intact in both. |
| + |
| +Hardening that same week (PR #681, both lessons from the first real |
| +production flip): flipping a host refuses while the box-wide backup |
| +replicator still tracks it — two litestream processes on one DB share a |
| +derived meta directory and corrupt each other — and the lease release runs |
| +under its own budget so a wedged final checkpoint can't strand the lease |
| +until TTL. |
| + |
| ### The seams cut for what's next |
| Small PRs that exist so later phases are config, not surgery: |
| @@ -281,7 +314,7 @@ means the decision stands with nothing built against it yet. |
| | 4 | metadata at rest | sealed bucket: HMAC(pepper) keys + pepper-encrypted metadata | **Not built** — object keys are plain uids; bucket stays private (Block Public Access) | |
| | 5 | public bucket | design as if public; the flip is its own reviewed decision (takedown process prerequisite) | Held — posture unchanged | |
| | 6 | instance runtime | hibernating processes over S3: local SQLite + Litestream, lease = single-writer lock (over Lambda / S3-as-only-DB / cells / renting DO-Fly) | **Shipped** #630/#641 — with the correction: the lease is Go-library-only; the CLI silently absorbs a second writer, so the activator embeds litestream | |
| -| 7 | placement | rendezvous hash as hint, S3 lease as truth (over a routing table) | Lease half **shipped**; gossip/HRW is roadmap — one box, hash picks self | |
| +| 7 | placement | rendezvous hash as hint, S3 lease as truth (over a routing table) | **Shipped** (#678) — S3-heartbeat membership + HRW + proxy-to-owner; kill-a-box passes live both ways | |
| | 8 | instance data | one SQLite per space, forever (shared DB tier is the anti-goal) | Held, load-bearing | |
| | 9 | attachments | blob interface → R2 from day one | Seam **shipped** #636; disk impl only, R2 roadmap | |
| | 10 | provisioning | create = register only; materialize at first wake; controld dissolves | Split **shipped** #625, hibernate-by-default creates #663; full deferral + controld dissolution roadmap | |
| @@ -339,11 +372,6 @@ PR that touches this territory gets checked against these: |
| Everything below is settled design from #550 with no code behind it (or only |
| a seam). Build triggers are events, not dates. |
| -- **The activator mesh** — the "second box" work, in progress: membership |
| - (gossip or S3 heartbeats), rendezvous hashing as the placement hint, peer |
| - proxying to the hash-owner. The lease already guarantees split-brain can't |
| - happen; the mesh only decides who *tries* first. Today's single router box |
| - is the degenerate case — membership of one, hash picks self. |
| - **Multi-region / `home_region`** — one write region per space, wakes |
| deterministic, relocation *while hibernated* (copy the prefix, flip the |
| field, next wake claims the lease elsewhere). The field exists; nothing |