amadan
server: mesh places a space by its home_region (multi-edge Phase 1) (#1093)
* server: mesh places a space by its home_region (multi-edge Phase 1) The activator mesh now scopes HRW placement to the live boxes whose region matches a space's home_region, instead of hashing over the whole live set. This is the "mesh places by region" foundation from docs/multi-edge.md, Phase 1 — placement only (no addressing changes, no relocation, the S3 single-writer lease untouched). How: - A box declares its region via LCHAT_MESH_REGION (empty = region-agnostic, today's behaviour), read in meshFromEnv beside SELF/DOMAINS/TOKEN. - Boxes advertise their region in the heartbeat BODY (never the key — the membership key stays base64url(peer) so List and the operator tools that decode it, deploy/*-posture.sh, are untouched). Peers learn each other's regions via a new HeartbeatStore.Regions, read off the hot path on the beat into an in-memory cache; membership/List keep their single-LIST, no-GET invariant. - New meshSvc.candidatesFor(host) looks up the host's replicated home_region from the local registry and returns the region-matching live boxes; owns and remoteOwner hash over candidatesFor instead of liveBoxes. Back-compat is load-bearing and byte-for-byte: an empty home_region, OR a region with no live box, OR no LCHAT_MESH_REGION anywhere falls back to the whole live set — exactly pre-region placement. Direct-newMesh unit tests (nil registry) are agnostic, so every existing mesh test passes unchanged. Tests: TestMeshRegionPlacement (region match routes there; empty and unmatched home_region both fall back to all-boxes; a no-region peer is agnostic), TestMeshRegionAgnosticUnchanged (candidatesFor == liveBoxes with no region/registry), TestMeshFromEnvRegion (env read + trim). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * server: gate the per-beat region fetch on this box declaring a region Region placement is opt-in per mesh, so an agnostic box (no LCHAT_MESH_REGION — prod today) must add zero S3 requests over pre-region behaviour. beat() now skips the Regions() fetch entirely when m.region == "", leaving m.regions empty; candidatesFor already falls back to all-boxes HRW when the map is empty, so behaviour is unchanged and correct. A mixed fleet stays safe — an agnostic box just hashes across everyone and the S3 single-writer lease absorbs any placement disagreement, exactly as for ordinary membership disagreement. Put stays unconditional (the heartbeat is membership, region-or-not). Added a note that the per-beat re-fetch could later narrow to fetch-on-new-peer since a box's region is static — a planet-scale optimisation, out of scope here. Test: TestMeshBeatRegionFetchGate — an agnostic box makes 0 Regions calls across two beats, a region-scoped box makes one per beat (fakeMeshStore now counts invocations). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: multi-edge — correct how region placement is advertised The "what this supersedes" note claimed placement reuses the domain-scoping per-box S3 advertisement. That plumbing lived in an unmerged branch, never on main — there was nothing to reuse. Region rides the mesh heartbeat body (every box already reads it for membership), fetched only by boxes that declare a region, matching what the placement slice actually builds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> ---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
5 files changed,
+426
−27
docs/multi-edge.md+9 −5internal/activator/heartbeat.go+72 −3mesh.go+122 −11mesh_test.go+218 −6router.go+5 −2
diff --git a/docs/multi-edge.md b/docs/multi-edge.md| index 5e539a8..82708ae 100644 |
| --- a/docs/multi-edge.md |
| +++ b/docs/multi-edge.md |
| @@ -161,11 +161,15 @@ requirement rules out. It was a useful stepping stone; this model replaces it: |
| - **Addresses go region-neutral** (`<sqid>.11ch.at`, one wildcard). The country |
| leaves the name. |
| -- **Placement keys on `home_region`, not the host label.** The per-box |
| - advertisement plumbing that domain-scoping added (each box publishing its |
| - capability via a sibling S3 object) is **reused** — a box now advertises its |
| - *region* instead of its host-domains, and placement/replication scope to the |
| - box(es) whose region matches the space's `home_region`. |
| +- **Placement keys on `home_region`, not the host label.** Each box advertises |
| + its region in its mesh heartbeat body; every box learns its peers' regions and |
| + scopes placement to the box(es) whose region matches a space's `home_region` |
| + (`mesh.go` `candidatesFor`, keying on the `home_region` the registry already |
| + replicates onto every box). The country-in-host trial's domain-scoping lived |
| + in an unmerged branch, so there was no prior per-box advertisement to reuse — |
| + the region rides the heartbeat every box already reads for membership, and the |
| + fetch is skipped entirely on a box that declares no region (zero cost when a |
| + mesh has not opted into regions). |
| - `docs/dev-mesh.md` and the `docs/planet-scale-guide/` "proving ground" page |
| describe the country-in-host scheme that runs on the dev mesh *today*; they stay |
| accurate until the phases below land, then are reworked in step. |
diff --git a/internal/activator/heartbeat.go b/internal/activator/heartbeat.go| index 281479d..5d2a92e 100644 |
| --- a/internal/activator/heartbeat.go |
| +++ b/internal/activator/heartbeat.go |
| @@ -24,6 +24,7 @@ import ( |
| "encoding/base64" |
| "encoding/json" |
| "fmt" |
| + "io" |
| "path" |
| "strings" |
| "time" |
| @@ -133,9 +134,19 @@ func (s *HeartbeatStore) PutStats(ctx context.Context, peer string, stats BoxSta |
| } |
| // Put writes (or refreshes) this box's heartbeat. The write itself is the |
| -// timestamp — LastModified advances on every beat. |
| -func (s *HeartbeatStore) Put(ctx context.Context, peer string) error { |
| - body, err := json.Marshal(map[string]string{"peer": peer}) |
| +// timestamp — LastModified advances on every beat. The body also carries the |
| +// box's placement region (LCHAT_MESH_REGION), so peers can learn it via Regions |
| +// and scope placement to a region (mesh.go's candidatesFor). region is omitted |
| +// when empty, so a region-agnostic box's body stays byte-identical to before |
| +// regions existed. The KEY is unchanged either way (base64url(peer)), so |
| +// membership/List and every operator tool that decodes the key |
| +// (deploy/*-posture.sh) are untouched — region rides the body, never the key. |
| +func (s *HeartbeatStore) Put(ctx context.Context, peer, region string) error { |
| + rec := map[string]string{"peer": peer} |
| + if region != "" { |
| + rec["region"] = region |
| + } |
| + body, err := json.Marshal(rec) |
| if err != nil { |
| return err |
| } |
| @@ -176,6 +187,64 @@ func (s *HeartbeatStore) List(ctx context.Context) ([]Heartbeat, error) { |
| return out, nil |
| } |
| +// Regions returns each peer's advertised placement region (the "region" field |
| +// its heartbeat body carries — see Put), keyed by peer URL. Deliberately |
| +// SEPARATE from List: List reads membership + liveness from keys + LastModified |
| +// alone (no per-member GET — the property the whole membership design rests |
| +// on), whereas region lives in the body, so this GETs each small object. The |
| +// mesh calls it off the hot request path, on its beat, into an in-memory cache |
| +// — region changes rarely (only on relocation) and a box only needs peers' |
| +// regions to scope placement. Best-effort per object: a body that won't GET or |
| +// decode, or that advertises no region (an older box, or a region-agnostic |
| +// one), is simply absent from the map, which the caller treats as agnostic. |
| +func (s *HeartbeatStore) Regions(ctx context.Context) (map[string]string, error) { |
| + out := map[string]string{} |
| + p := awss3.NewListObjectsV2Paginator(s.client, &awss3.ListObjectsV2Input{ |
| + Bucket: aws.String(s.bucket), |
| + Prefix: aws.String(s.prefix + "/"), |
| + }) |
| + for p.HasMorePages() { |
| + page, err := p.NextPage(ctx) |
| + if err != nil { |
| + return nil, err |
| + } |
| + for _, obj := range page.Contents { |
| + key := aws.ToString(obj.Key) |
| + peer, err := decodeHeartbeatKey(path.Base(key)) |
| + if err != nil { |
| + continue // a foreign object must never poison the region view |
| + } |
| + region := s.getRegion(ctx, key) |
| + if region == "" { |
| + continue // no region advertised — leave the peer agnostic |
| + } |
| + out[peer] = region |
| + } |
| + } |
| + return out, nil |
| +} |
| + |
| +// getRegion GETs one heartbeat object and pulls its advertised region from the |
| +// body, "" on any error — a soft miss, never fatal to the region refresh (a |
| +// stale region view at worst scopes a space to the whole fleet for one beat). |
| +func (s *HeartbeatStore) getRegion(ctx context.Context, key string) string { |
| + obj, err := s.client.GetObject(ctx, &awss3.GetObjectInput{ |
| + Bucket: aws.String(s.bucket), |
| + Key: aws.String(key), |
| + }) |
| + if err != nil { |
| + return "" |
| + } |
| + defer obj.Body.Close() |
| + var rec struct { |
| + Region string `json:"region"` |
| + } |
| + if err := json.NewDecoder(io.LimitReader(obj.Body, 1<<12)).Decode(&rec); err != nil { |
| + return "" |
| + } |
| + return rec.Region |
| +} |
| + |
| // Delete removes this box's heartbeat — the FIRST step of a drain, before any |
| // checkpoint or lease release, so peers take over immediately. |
| func (s *HeartbeatStore) Delete(ctx context.Context, peer string) error { |
diff --git a/mesh.go b/mesh.go| index adb0c9f..52368ef 100644 |
| --- a/mesh.go |
| +++ b/mesh.go |
| @@ -40,6 +40,18 @@ package main |
| // LCHAT_WILDCARD_DOMAINS's ACME DNS-01 cert, which — unlike per-host |
| // HTTP-01 — any box can obtain independently of which box DNS currently |
| // points at). |
| +// 6. Region placement (LCHAT_MESH_REGION + a space's home_region) — the |
| +// multi-edge foundation (docs/multi-edge.md, Phase 1). Within the in-scope |
| +// set, HRW picks the owner from only the live boxes whose region matches |
| +// the space's home_region (candidatesFor), so a space lives on a box in its |
| +// region and relocating it is just a home_region flip (the hash re-homes it |
| +// onto the destination box, the lease still guaranteeing one writer). A |
| +// space with no home_region, OR whose region currently has no live box, |
| +// falls back to hashing across the WHOLE live set — byte-identical to |
| +// pre-region placement, which is what keeps today's single-region fleet |
| +// (and every self-hoster) unchanged. Boxes learn each other's region from |
| +// the heartbeat body (heartbeat.go's Regions); the home_region itself is |
| +// replicated into every box's registry alongside the route (meshreplicate.go). |
| // |
| // Drain (activatorSvc.shutdown) deletes the heartbeat FIRST, then checkpoints |
| // and releases as today — peers take over immediately, and deploys, box |
| @@ -75,9 +87,11 @@ const meshForwardedHeader = "X-Lchat-Mesh-Forwarded" |
| const ( |
| // meshHeartbeatInterval paces the PUT + LIST beat. Membership changes are |
| - // rare events; this is liveness, not coordination — and each beat is two |
| - // S3 Tier-1 requests, so 15s cost ~11.5k requests/day per box (~$21/yr) |
| - // for nothing membership-shaped in return. Slower still would work, but |
| + // rare events; this is liveness, not coordination — and each beat is a PUT |
| + // plus a membership LIST, and now a region refresh (heartbeat.go's Regions: |
| + // a second LIST plus one small GET per live peer — a handful of requests at |
| + // fleet scale), so it stays cheap for nothing membership-shaped in return. |
| + // Slower still would work, but |
| // staleness (3 beats) rides on this, and 3m dead-box detection already |
| // sits well inside the lease expiry that actually gates takeover. |
| meshHeartbeatInterval = 60 * time.Second |
| @@ -95,8 +109,12 @@ const ( |
| // production), an interface so the mesh's unit tests share an in-memory |
| // "bucket" between boxes. |
| type meshStore interface { |
| - Put(ctx context.Context, peer string) error |
| + Put(ctx context.Context, peer, region string) error |
| List(ctx context.Context) ([]activator.Heartbeat, error) |
| + // Regions returns each peer's advertised placement region (peer URL -> |
| + // region); a peer absent from the map advertises none (agnostic). Read off |
| + // the hot path, on the beat — see heartbeat.go's Regions. |
| + Regions(ctx context.Context) (map[string]string, error) |
| Delete(ctx context.Context, peer string) error |
| } |
| @@ -105,12 +123,15 @@ type meshSvc struct { |
| store meshStore |
| domains []string // LCHAT_MESH_DOMAINS, lowercased — see inScope. Empty = unrestricted (test convenience only; meshFromEnv always supplies a non-empty list) |
| token string // LCHAT_MESH_TOKEN — shared secret authenticating box-to-box route replication (meshreplicate.go); empty = unrestricted (test convenience only) |
| + region string // LCHAT_MESH_REGION — this box's placement region (e.g. "ie"); "" = region-agnostic (today's behaviour). Advertised in the heartbeat body; peers read it via Regions. See candidatesFor. |
| + reg *Registry // local registry handle, for reading a host's replicated home_region (candidatesFor). nil = no lookup (direct-newMesh unit tests) = every space region-agnostic. |
| interval time.Duration |
| staleAfter time.Duration |
| now func() time.Time // a seam for the unit tests' clock |
| mu sync.Mutex |
| members []activator.Heartbeat // last LIST; staleness applied at read time |
| + regions map[string]string // last Regions view: peer URL -> advertised region (self excluded — known locally via m.region) |
| proxies map[string]*httputil.ReverseProxy |
| started bool |
| @@ -126,6 +147,7 @@ func newMesh(self string, store meshStore, domains []string) *meshSvc { |
| interval: meshHeartbeatInterval, |
| staleAfter: meshStaleAfter, |
| now: time.Now, |
| + regions: make(map[string]string), |
| proxies: make(map[string]*httputil.ReverseProxy), |
| stopCh: make(chan struct{}), |
| done: make(chan struct{}), |
| @@ -145,6 +167,10 @@ func newMesh(self string, store meshStore, domains []string) *meshSvc { |
| // LCHAT_MESH_TOKEN, the shared secret every mesh member authenticates |
| // box-to-box route replication with (meshreplicate.go) — without it any box |
| // could hand any other box arbitrary routes. |
| +// |
| +// LCHAT_MESH_REGION is optional: this box's placement region (e.g. "ie"). Set, |
| +// it opts the box into region-scoped placement (candidatesFor); unset, the box |
| +// is region-agnostic and everything behaves exactly as before regions existed. |
| func meshFromEnv(hc *hibernateConfig) (*meshSvc, error) { |
| self := strings.TrimSpace(os.Getenv("LCHAT_MESH_SELF")) |
| if self == "" { |
| @@ -172,6 +198,7 @@ func meshFromEnv(hc *hibernateConfig) (*meshSvc, error) { |
| } |
| m := newMesh(self, store, domains) |
| m.token = token |
| + m.region = strings.TrimSpace(os.Getenv("LCHAT_MESH_REGION")) |
| return m, nil |
| } |
| @@ -197,10 +224,23 @@ func (m *meshSvc) start() { |
| }() |
| } |
| -// beat PUTs our heartbeat and refreshes the member view. Both failures are |
| -// soft by design: a missed PUT makes peers see us stale (they take over; the |
| -// lease arbitrates), a failed LIST keeps the last known view (a stale view |
| -// misroutes at worst — same arbitration). |
| +// beat PUTs our heartbeat (with our region) and refreshes the member view, plus |
| +// the peer-region view WHEN this box has opted into region placement. Every |
| +// failure is soft by design: a missed PUT makes peers see us stale (they take |
| +// over; the lease arbitrates), a failed LIST keeps the last known membership (a |
| +// stale view misroutes at worst — same arbitration), and a failed region |
| +// refresh keeps the last regions (at worst a space scopes to the whole fleet |
| +// for one beat, never wrong — candidatesFor's fallback IS all-boxes). |
| +// |
| +// Region placement is opt-in PER MESH: a box that declares no region |
| +// (LCHAT_MESH_REGION unset — prod today) skips the peer-region fetch entirely, |
| +// so it adds ZERO S3 requests over pre-region behaviour — an inert feature |
| +// costs nothing. Such a box leaves m.regions empty and candidatesFor always |
| +// falls back to all-boxes HRW. A mixed fleet (some boxes with regions, some |
| +// without) is safe too: an agnostic box just can't region-scope, so it hashes |
| +// across everyone, and any resulting placement disagreement is absorbed by the |
| +// S3 single-writer lease exactly as ordinary membership disagreement is — a |
| +// wrong-box wake, never corruption. |
| func (m *meshSvc) beat() { |
| timeout := m.interval |
| if timeout < 5*time.Second { |
| @@ -208,7 +248,9 @@ func (m *meshSvc) beat() { |
| } |
| ctx, cancel := context.WithTimeout(context.Background(), timeout) |
| defer cancel() |
| - if err := m.store.Put(ctx, m.self); err != nil { |
| + // Put stays unconditional — the heartbeat is membership (its key + the write |
| + // timestamp), region-or-not; an unset region just writes an empty-region body. |
| + if err := m.store.Put(ctx, m.self, m.region); err != nil { |
| log.Printf("mesh: heartbeat put failed: %v — peers will see this box stale after %s", err, m.staleAfter) |
| } |
| members, err := m.store.List(ctx) |
| @@ -219,6 +261,21 @@ func (m *meshSvc) beat() { |
| m.mu.Lock() |
| m.members = members |
| m.mu.Unlock() |
| + if m.region == "" { |
| + return // not region-scoped — no peer regions to fetch, zero extra S3 |
| + } |
| + // Region-scoped: refresh peers' regions. A box's region is static, so this |
| + // per-beat re-fetch could later narrow to fetch-on-new-peer only (a |
| + // planet-scale optimisation — out of scope for this slice, where the fleet |
| + // is a handful of boxes and a per-beat GET each is noise). |
| + regions, err := m.store.Regions(ctx) |
| + if err != nil { |
| + log.Printf("mesh: region refresh failed: %v — keeping the last known regions", err) |
| + return |
| + } |
| + m.mu.Lock() |
| + m.regions = regions |
| + m.mu.Unlock() |
| } |
| // halt stops the heartbeat loop without touching the heartbeat object — the |
| @@ -273,6 +330,60 @@ func (m *meshSvc) liveBoxes() []string { |
| return boxes |
| } |
| +// candidatesFor returns the live boxes eligible to own host — the member set |
| +// hrwOwner picks from. Placement keys on the space's home_region (registry.go, |
| +// replicated to every in-scope box by meshreplicate.go), so a space pinned to a |
| +// region is owned by a live box IN that region; relocation is a home_region flip |
| +// and the hash re-homes it onto the destination box (docs/multi-edge.md). |
| +// |
| +// The fallback to ALL live boxes is load-bearing: a space with no home_region |
| +// (every space today), or one whose region has no live box right now, hashes |
| +// across the whole fleet exactly as before regions existed. So with no |
| +// LCHAT_MESH_REGION set anywhere and empty home_region everywhere, this returns |
| +// liveBoxes() and owns/remoteOwner are byte-identical to pre-region placement. |
| +func (m *meshSvc) candidatesFor(host string) []string { |
| + live := m.liveBoxes() |
| + home := m.homeRegionOf(host) |
| + if home == "" { |
| + return live // region-agnostic space — the whole fleet, as before regions |
| + } |
| + m.mu.Lock() |
| + regions := m.regions |
| + m.mu.Unlock() |
| + var matched []string |
| + for _, b := range live { |
| + region := regions[b] |
| + if b == m.self { |
| + region = m.region // our own region is known locally, never via S3 |
| + } |
| + if region == home { |
| + matched = append(matched, b) |
| + } |
| + } |
| + if len(matched) == 0 { |
| + // No live box advertises the space's region (none configured for it yet, |
| + // or that region's box is down): fall back to the whole fleet so the |
| + // space is still served somewhere — availability, and exact back-compat. |
| + return live |
| + } |
| + return matched |
| +} |
| + |
| +// homeRegionOf reads host's replicated home_region from the LOCAL registry |
| +// (meshreplicate.go copies it onto every in-scope box, so no network). "" when |
| +// the mesh has no registry handle (direct-newMesh unit tests), the host has no |
| +// row, or its region is unset — all of which candidatesFor treats as agnostic. |
| +func (m *meshSvc) homeRegionOf(host string) string { |
| + if m.reg == nil { |
| + return "" |
| + } |
| + r, err := m.reg.get(host) |
| + if err != nil || r == nil { |
| + return "" |
| + } |
| + return r.HomeRegion |
| +} |
| + |
| // inScope reports whether host participates in HRW placement at all. Only |
| // hosts under a configured LCHAT_MESH_DOMAINS domain are ever hashed across |
| // boxes; everything else always resolves to whichever box is asked — correct |
| @@ -310,7 +421,7 @@ func (m *meshSvc) owns(host string) bool { |
| if !m.inScope(host) { |
| return true |
| } |
| - return hrwOwner(host, m.liveBoxes()) == m.self |
| + return hrwOwner(host, m.candidatesFor(host)) == m.self |
| } |
| // remoteOwner returns the peer to proxy host to when its HRW owner is another |
| @@ -322,7 +433,7 @@ func (m *meshSvc) remoteOwner(host string) (peer string, ok bool) { |
| if !m.inScope(host) { |
| return "", false |
| } |
| - if o := hrwOwner(host, m.liveBoxes()); o != m.self { |
| + if o := hrwOwner(host, m.candidatesFor(host)); o != m.self { |
| return o, true |
| } |
| return "", false |
diff --git a/mesh_test.go b/mesh_test.go| index 95df128..25f6b80 100644 |
| --- a/mesh_test.go |
| +++ b/mesh_test.go |
| @@ -34,19 +34,24 @@ import ( |
| // between boxes it plays the bucket; rec (optional) feeds a fakeBackend's |
| // order log so drain tests can pin event ordering. |
| type fakeMeshStore struct { |
| - mu sync.Mutex |
| - beats map[string]time.Time |
| - now func() time.Time |
| - rec func(ev string) |
| + mu sync.Mutex |
| + beats map[string]time.Time |
| + regions map[string]string // peer -> advertised region (like the real body's "region") |
| + regionsCall int // how many times Regions was invoked — proves the beat gate |
| + now func() time.Time |
| + rec func(ev string) |
| } |
| func newFakeMeshStore(now func() time.Time) *fakeMeshStore { |
| - return &fakeMeshStore{beats: make(map[string]time.Time), now: now} |
| + return &fakeMeshStore{beats: make(map[string]time.Time), regions: make(map[string]string), now: now} |
| } |
| -func (s *fakeMeshStore) Put(_ context.Context, peer string) error { |
| +func (s *fakeMeshStore) Put(_ context.Context, peer, region string) error { |
| s.mu.Lock() |
| s.beats[peer] = s.now() |
| + if region != "" { |
| + s.regions[peer] = region // mirror the real store: region omitted when empty |
| + } |
| rec := s.rec |
| s.mu.Unlock() |
| if rec != nil { |
| @@ -65,9 +70,27 @@ func (s *fakeMeshStore) List(context.Context) ([]activator.Heartbeat, error) { |
| return out, nil |
| } |
| +func (s *fakeMeshStore) Regions(context.Context) (map[string]string, error) { |
| + s.mu.Lock() |
| + defer s.mu.Unlock() |
| + s.regionsCall++ |
| + out := make(map[string]string, len(s.regions)) |
| + for peer, region := range s.regions { |
| + out[peer] = region |
| + } |
| + return out, nil |
| +} |
| + |
| +func (s *fakeMeshStore) regionsCalls() int { |
| + s.mu.Lock() |
| + defer s.mu.Unlock() |
| + return s.regionsCall |
| +} |
| + |
| func (s *fakeMeshStore) Delete(_ context.Context, peer string) error { |
| s.mu.Lock() |
| delete(s.beats, peer) |
| + delete(s.regions, peer) |
| rec := s.rec |
| s.mu.Unlock() |
| if rec != nil { |
| @@ -82,6 +105,14 @@ func (s *fakeMeshStore) setBeat(peer string, at time.Time) { |
| s.beats[peer] = at |
| } |
| +// setRegion advertises a peer's placement region, as if its heartbeat body |
| +// carried it — the fake counterpart of the real box PUTting with a region. |
| +func (s *fakeMeshStore) setRegion(peer, region string) { |
| + s.mu.Lock() |
| + defer s.mu.Unlock() |
| + s.regions[peer] = region |
| +} |
| + |
| func (s *fakeMeshStore) hasBeat(peer string) bool { |
| s.mu.Lock() |
| defer s.mu.Unlock() |
| @@ -156,6 +187,7 @@ func newMeshTestBox(t *testing.T, name string, store meshStore, clk *fakeClock, |
| box.mesh = newMesh(box.front.URL, store, domains) |
| box.mesh.now = clk.now |
| + box.mesh.reg = reg // region-scoped placement reads home_region from the registry |
| box.act.mesh = box.mesh |
| return box |
| } |
| @@ -419,6 +451,186 @@ func TestMeshDomainScopingBypassesHRW(t *testing.T) { |
| } |
| } |
| +// --- region placement (docs/multi-edge.md, Phase 1) ----------------------------- |
| + |
| +// candidatesFor scopes HRW to the live boxes whose region matches a space's |
| +// home_region, with an all-boxes fallback that keeps pre-region behaviour |
| +// byte-identical. One setup, four rules the task pins: |
| +// (a) a space pinned to a region routes to that region's box; |
| +// (b) a space with empty home_region falls back to all-boxes HRW (== today); |
| +// (c) a home_region no live box advertises falls back to all-boxes; |
| +// (d) a peer that advertises no region is treated as agnostic (only ever a |
| +// fallback candidate, never a region match). |
| +func TestMeshRegionPlacement(t *testing.T) { |
| + clk := &fakeClock{t: time.Unix(1_700_000_000, 0)} |
| + store := newFakeMeshStore(clk.now) |
| + reg, err := openRegistry(filepath.Join(t.TempDir(), "registry.db")) |
| + if err != nil { |
| + t.Fatalf("open registry: %v", err) |
| + } |
| + |
| + // Three live boxes: self in "ie", a peer in "fi", and an agnostic peer that |
| + // advertises no region at all (rule d). |
| + m := newMesh("http://a", store, []string{"11ch.at"}) |
| + m.now = clk.now |
| + m.reg = reg |
| + m.region = "ie" |
| + store.setBeat("http://b", clk.now()) |
| + store.setRegion("http://b", "fi") |
| + store.setBeat("http://c", clk.now()) // agnostic peer — no setRegion |
| + m.beat() // our PUT records self's region; List+Regions load the view |
| + |
| + all := []string{"http://a", "http://b", "http://c"} |
| + if got := m.liveBoxes(); !sameSet(got, all) { |
| + t.Fatalf("liveBoxes = %v, want %v", got, all) |
| + } |
| + |
| + // A helper to register an in-scope space with a given home_region. |
| + pin := func(host, region string) { |
| + if err := reg.upsert(Route{Host: host, Kind: "instance", Socket: "/x.sock", Hibernate: true, HomeRegion: region}); err != nil { |
| + t.Fatalf("upsert %s: %v", host, err) |
| + } |
| + } |
| + |
| + // (a) Pinned to "fi" → only the fi box is a candidate, so it owns and every |
| + // other box proxies to it, regardless of what bare HRW over all three says. |
| + pin("fi-space.11ch.at", "fi") |
| + if got := m.candidatesFor("fi-space.11ch.at"); !sameSet(got, []string{"http://b"}) { |
| + t.Fatalf("home_region fi: candidates = %v, want [http://b]", got) |
| + } |
| + if m.owns("fi-space.11ch.at") { |
| + t.Fatal("box a wrongly owns an fi-pinned space") |
| + } |
| + if peer, remote := m.remoteOwner("fi-space.11ch.at"); !remote || peer != "http://b" { |
| + t.Fatalf("remoteOwner(fi-space) = %q,%v; want http://b,true", peer, remote) |
| + } |
| + |
| + // Self-region match: a space pinned to our own "ie" is owned locally. |
| + pin("ie-space.11ch.at", "ie") |
| + if got := m.candidatesFor("ie-space.11ch.at"); !sameSet(got, []string{"http://a"}) { |
| + t.Fatalf("home_region ie: candidates = %v, want [http://a]", got) |
| + } |
| + if !m.owns("ie-space.11ch.at") { |
| + t.Fatal("box a does not own an ie-pinned space it is the region for") |
| + } |
| + |
| + // (b) Empty home_region (a host with no row at all is empty too) → all boxes, |
| + // and the chosen owner is identical to bare HRW over the whole live set. |
| + noRegion := "plain.11ch.at" // no reg row → homeRegionOf "" |
| + if got := m.candidatesFor(noRegion); !sameSet(got, all) { |
| + t.Fatalf("empty home_region: candidates = %v, want all %v", got, all) |
| + } |
| + if hrwOwner(noRegion, m.candidatesFor(noRegion)) != hrwOwner(noRegion, m.liveBoxes()) { |
| + t.Fatal("empty home_region must place identically to pre-region HRW") |
| + } |
| + |
| + // (c) A home_region no live box advertises → fall back to all boxes. |
| + pin("de-space.11ch.at", "de") |
| + if got := m.candidatesFor("de-space.11ch.at"); !sameSet(got, all) { |
| + t.Fatalf("unmatched home_region de: candidates = %v, want all %v", got, all) |
| + } |
| + |
| + // (d) The agnostic peer "http://c" is never a region match (it advertised |
| + // none), yet is present in every all-boxes fallback above — already asserted |
| + // by the sameSet(all) checks that include it. |
| +} |
| + |
| +// TestMeshRegionAgnosticUnchanged pins the core back-compat promise directly: a |
| +// mesh with no region set and no registry (exactly how the existing unit tests |
| +// and today's fleet run) has candidatesFor == liveBoxes for every host, so |
| +// owns/remoteOwner cannot diverge from pre-region behaviour. |
| +func TestMeshRegionAgnosticUnchanged(t *testing.T) { |
| + clk := &fakeClock{t: time.Unix(1_700_000_000, 0)} |
| + store := newFakeMeshStore(clk.now) |
| + m := newMesh("http://a", store, []string{"11ch.at"}) |
| + m.now = clk.now // no region, no reg — the pre-region shape |
| + store.setBeat("http://b", clk.now()) |
| + m.beat() |
| + |
| + for i := 0; i < 50; i++ { |
| + host := fmt.Sprintf("space-%d.11ch.at", i) |
| + if !sameSet(m.candidatesFor(host), m.liveBoxes()) { |
| + t.Fatalf("region-agnostic mesh scoped %s away from the full live set", host) |
| + } |
| + } |
| +} |
| + |
| +// TestMeshBeatRegionFetchGate proves the zero-cost-when-inert promise: a box |
| +// with no region declared (prod today) fetches NO peer regions on a beat, so |
| +// region placement adds zero S3 requests over pre-region behaviour; a box that |
| +// declares a region does fetch them. |
| +func TestMeshBeatRegionFetchGate(t *testing.T) { |
| + clk := &fakeClock{t: time.Unix(1_700_000_000, 0)} |
| + |
| + // Agnostic box: beats never touch Regions. |
| + store := newFakeMeshStore(clk.now) |
| + agnostic := newMesh("http://a", store, []string{"11ch.at"}) |
| + agnostic.now = clk.now // no region |
| + agnostic.beat() |
| + agnostic.beat() |
| + if got := store.regionsCalls(); got != 0 { |
| + t.Fatalf("agnostic box made %d Regions call(s) across two beats, want 0 — an inert feature must cost nothing", got) |
| + } |
| + |
| + // Region-scoped box: each beat fetches peer regions. |
| + store2 := newFakeMeshStore(clk.now) |
| + scoped := newMesh("http://b", store2, []string{"11ch.at"}) |
| + scoped.now = clk.now |
| + scoped.region = "ie" |
| + scoped.beat() |
| + scoped.beat() |
| + if got := store2.regionsCalls(); got != 2 { |
| + t.Fatalf("region-scoped box made %d Regions call(s) across two beats, want 2", got) |
| + } |
| +} |
| + |
| +// sameSet compares two peer slices ignoring order (liveBoxes/candidatesFor draw |
| +// from a map, so order is not defined). |
| +func sameSet(a, b []string) bool { |
| + if len(a) != len(b) { |
| + return false |
| + } |
| + seen := make(map[string]int, len(a)) |
| + for _, x := range a { |
| + seen[x]++ |
| + } |
| + for _, y := range b { |
| + seen[y]-- |
| + } |
| + for _, n := range seen { |
| + if n != 0 { |
| + return false |
| + } |
| + } |
| + return true |
| +} |
| + |
| +// meshFromEnv reads LCHAT_MESH_REGION (optional): set = this box's placement |
| +// region, unset = region-agnostic (the empty string, today's behaviour). |
| +func TestMeshFromEnvRegion(t *testing.T) { |
| + t.Setenv("LCHAT_MESH_SELF", "https://box-a.internal") |
| + t.Setenv("LCHAT_MESH_DOMAINS", "11ch.at") |
| + t.Setenv("LCHAT_MESH_TOKEN", "shared-secret") |
| + |
| + t.Setenv("LCHAT_MESH_REGION", "") |
| + m, err := meshFromEnv(&hibernateConfig{Bucket: "b", Region: "eu-west-1"}) |
| + if err != nil || m == nil { |
| + t.Fatalf("unset region: %v, %v", m, err) |
| + } |
| + if m.region != "" { |
| + t.Fatalf("region = %q, want \"\" (agnostic) when LCHAT_MESH_REGION is unset", m.region) |
| + } |
| + |
| + t.Setenv("LCHAT_MESH_REGION", " ie ") |
| + m, err = meshFromEnv(&hibernateConfig{Bucket: "b", Region: "eu-west-1"}) |
| + if err != nil || m == nil { |
| + t.Fatalf("set region: %v, %v", m, err) |
| + } |
| + if m.region != "ie" { |
| + t.Fatalf("region = %q, want the trimmed \"ie\"", m.region) |
| + } |
| +} |
| + |
| // meshFromEnv refuses to start the mesh without an explicit domain allowlist — |
| // failing loudly beats silently letting HRW assign a host to a box that has |
| // no registry row for it (see the inScope doc comment). |
diff --git a/router.go b/router.go| index f37dbea..490a684 100644 |
| --- a/router.go |
| +++ b/router.go |
| @@ -267,9 +267,12 @@ func runRouter(args []string) { |
| // mesh from its first tick, and the first request deserves a |
| // membership view. |
| act.mesh = mesh |
| + // The registry is how the mesh reads a host's replicated |
| + // home_region for region-scoped placement (candidatesFor). |
| + mesh.reg = reg |
| mesh.start() |
| - log.Printf("router: activator mesh on — this box is %s (heartbeats s3://%s/%s every %s, peers stale after %s, placement scoped to %v)", |
| - mesh.self, hc.Bucket, hc.meshPrefix(), mesh.interval, mesh.staleAfter, mesh.domains) |
| + log.Printf("router: activator mesh on — this box is %s (region %q, heartbeats s3://%s/%s every %s, peers stale after %s, placement scoped to %v)", |
| + mesh.self, mesh.region, hc.Bucket, hc.meshPrefix(), mesh.interval, mesh.staleAfter, mesh.domains) |
| // This box's own capacity snapshot for the Fleet status page's |
| // multi-host aggregation (meshstats.go) — a separate S3 client on |