amadan
server: lchat relocate — change a space's home region (multi-edge P1 slice 5) (#1129)
The relocate action moves a space to another region while keeping its stable address: copy the store to the destination region (a no-op on dev's shared bucket, a real prefix copy for prod's per-region stores), flip the replicated home_region, re-fan-out the route so every box re-homes the space, and optionally move the A record (off by default — the wildcard + proxy already route to the new owner). Data-safety rests on the single-writer S3 lease: home_region/DNS only steer placement, the lease is the one truth of who writes, so a wrong-box request 502/503s and retries onto the owner — never two writers. Idempotent, with a copy-clobber guard: once a space is already home the destination is authoritative, so a re-run never re-copies stale source bytes. - relocate.go: relocationStorePlan (pure skip-vs-copy decision), copyS3Prefix (excludes lock.json), relocate() core, runRelocate CLI, optional A-record path. - registry.go: setHomeRegion (narrow UPDATE). - mesh.go: regionLive (target-region liveness precondition). - route53dns.go: upsertA + aRecordUpserter (optional, Route53-only). - Tests: store-plan decision, copyS3Prefix, home_region flip + idempotency, target-region validation, Route53 A upsert. - docs/multi-edge.md: the relocate command, data-safety, dev-vs-prod, follow-ons.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
8 files changed,
+968
−6
docs/multi-edge.md+82 −3main.go+3 −0mesh.go+32 −3registry.go+15 −0relocate.go+462 −0relocate_test.go+304 −0route53dns.go+34 −0route53dns_test.go+36 −0
diff --git a/docs/multi-edge.md b/docs/multi-edge.md| index 82708ae..0bb57cf 100644 |
| --- a/docs/multi-edge.md |
| +++ b/docs/multi-edge.md |
| @@ -92,6 +92,79 @@ route-replication plumbing) plus the `home_region` field, region-neutral |
| addressing, and placement-by-region. It is the smallest change that delivers the |
| stable-address promise. |
| +### The `lchat relocate` command |
| + |
| +The relocate action is a subcommand of the one binary — `lchat relocate <host> |
| +--to <region>` (`relocate.go`, a reusable `relocate()` core a future controld |
| +RPC or member-facing "change region" button can call). It runs the three steps |
| +above in order: |
| + |
| +1. **Store copy (or the dev no-op).** A pure decision helper, |
| + `relocationStorePlan(src, dst)`, returns `skip=true` exactly when source and |
| + destination name the same bytes — same bucket, same per-host prefix, same |
| + endpoint. That is the **dev mesh's one-shared-bucket case**, so the copy is a |
| + no-op there. In production each region has its own store, so it is a real |
| + prefix-to-prefix copy of `s3://<bucket>/<host>/…` (`copyS3Prefix`), run while |
| + the space sleeps. The copy deliberately **excludes the lease object** |
| + (`lock.json`): a copied, still-valid lease would make the destination's first |
| + `AcquireLease` see a held lock and 503 until the TTL expired — the destination |
| + must take its own fresh lease. |
| +2. **Flip `home_region` + re-fan-out.** `setHomeRegion` writes the replicated |
| + routes column and `fanOutMeshRoute` pushes the updated route to peers, so every |
| + box's `candidatesFor` (`mesh.go`) re-homes the space onto a box in the |
| + destination region. Route *edits* aren't auto-replicated by the normal path, so |
| + this is the deliberate targeted re-fan-out of the single updated route. The |
| + local router needs **no reload** — its mesh reads `home_region` live from the |
| + shared registry on each placement decision. |
| +3. **Optional A-record move.** `--move-dns --dns-target <ip>` repoints the host's |
| + A record at the destination box to avoid the proxy hop. **Off by default** — the |
| + `*.<domain>` wildcard + proxy-to-owner already route any request to the new |
| + owner through the DNS-TTL window, so relocation is correct with zero DNS write |
| + perms. Only the Route53 provider implements the A upsert today |
| + (`route53dns.go`); DNSimple refuses `--move-dns` with a clear message. |
| + |
| +**Preconditions the command enforces:** the space must be an in-scope, `ready`, |
| +**hibernating** instance (a live systemd instance is box-pinned — the message |
| +tells you to `lchat hibernate <host> --on` first), and a **live mesh box must |
| +advertise the target region** (else the flip re-homes onto nothing). Run it on a |
| +box whose registry has the route (the source region's box) with the mesh env |
| +sourced (`/data/router.env`) — `sudo` strips `LCHAT_MESH_*`, the same gotcha the |
| +dev-cutover runbook's Step 5 calls out. |
| + |
| +**Why the single-writer lease makes it data-safe.** Relocation never split-brains |
| +because the S3 lease (`internal/activator`, `lock.json` under the space's prefix) |
| +is the one truth of who may write; `home_region` and DNS are only placement hints: |
| + |
| +- *Shared bucket (dev):* both regions contend on the **same** `lock.json`, so the |
| + lease is globally singular. After the flip the destination is the HRW owner; if |
| + the source is momentarily still awake it holds the lease, the destination's |
| + `AcquireLease` fails loudly (`LeaseExistsError` → 503 + Retry-After), and the |
| + client's retry lands once the source sleeps and releases. Never two writers. |
| +- *Per-region buckets (prod):* the copy runs while the source is **asleep** (lease |
| + released, replica fully flushed by the sleep-time `SyncAndWait`), so the |
| + destination restores a complete copy and takes a fresh lease in its own bucket. |
| + Once `home_region` is flipped the source's `owns()` returns false, so it never |
| + wakes there again — a stray request proxies to the new owner instead. |
| + |
| +**Idempotency + the copy-clobber guard.** Once `home_region` already equals the |
| +target, the space's authoritative copy is the **destination's**, so a re-run must |
| +**not** re-copy source→dest (that would overwrite newer destination writes with |
| +stale source bytes). The copy + flip therefore run **only on the forward move** |
| +(`current != target`); an already-home re-run just heals peers by re-fanning-out |
| +the route. A crash between copy and flip converges safely — the source is still |
| +authoritative (not yet flipped), so the next run re-copies (an idempotent |
| +overwrite) and flips. |
| + |
| +**Tested vs. needs live proof.** Unit tests cover the pure store-plan decision |
| +(skip vs copy), `copyS3Prefix` (key layout preserved, `lock.json` excluded), the |
| +`home_region` flip + idempotency guard, target-region validation, and the Route53 |
| +A upsert. The **dev shared-bucket path (skip) is what a live dev relocation |
| +exercises**; the **prod cross-bucket copy is unproven in production** — it needs a |
| +per-region-bucket dev/prod trial, and two design points remain open: region→bucket |
| +auto-discovery (today the operator names the destination with `--dst-bucket`) and |
| +cross-*account*/cross-*endpoint* (R2) copies, which need a download+reupload |
| +variant rather than server-side `CopyObject`. Both are follow-ons. |
| + |
| ## Phase 2 — multi-edge within a region |
| Replicate a space's log across the boxes of *one* region. Same-jurisdiction, so |
| @@ -176,14 +249,20 @@ requirement rules out. It was a useful stepping stone; this model replaces it: |
| ## Status and build order |
| -Nothing here is operable yet; the pieces it reuses are. |
| +Phase 1 is now largely built; the pieces it reuses were always running. |
| - The **append-only, content-blind data model** it rests on: *running today* — it |
| is how every space already works. |
| - **Hibernation + single-writer lease + proxy-to-owner** (the machinery Phase 1 |
| reuses): *running today* (`docs/planet-scale.md`). |
| -- **Region-neutral addressing + `home_region` placement + relocation** (Phase 1): |
| - *designed, not built*. |
| +- **Region-neutral addressing + `home_region` placement** (Phase 1): *built + |
| + proven live on the dev mesh* — one shared `dev.11ch.at` namespace, a shared |
| + wildcard, cross-region serve returning 200 (the dev cutover runbook, |
| + `docs/multi-edge-dev-cutover.md`, lands with the cutover PR). |
| +- **The `lchat relocate` action** (Phase 1's last slice): *built; the dev |
| + shared-bucket path (skip-copy) lands on the proven cross-region serve; the prod |
| + per-region-bucket copy is implemented + unit-tested but unproven in production* |
| + (see [The `lchat relocate` command](#the-lchat-relocate-command)). |
| - **Multi-edge within a region** (Phase 2) and **cross-region multi-edge** |
| (Phase 3): *designed, not built*. |
diff --git a/main.go b/main.go| index 46e6d91..b27269c 100644 |
| --- a/main.go |
| +++ b/main.go |
| @@ -211,6 +211,9 @@ func main() { |
| case "replication": |
| runReplication(os.Args[2:]) |
| return |
| + case "relocate": |
| + runRelocate(os.Args[2:]) |
| + return |
| case "admin-link": |
| runAdminLink(os.Args[2:]) |
| return |
diff --git a/mesh.go b/mesh.go| index 52368ef..31d504f 100644 |
| --- a/mesh.go |
| +++ b/mesh.go |
| @@ -121,9 +121,9 @@ type meshStore interface { |
| type meshSvc struct { |
| self string // this box's peer base URL — its mesh identity |
| store meshStore |
| - domains []string // LCHAT_MESH_DOMAINS, lowercased — see inScope. Empty = unrestricted (test convenience only; meshFromEnv always supplies a non-empty list) |
| - token string // LCHAT_MESH_TOKEN — shared secret authenticating box-to-box route replication (meshreplicate.go); empty = unrestricted (test convenience only) |
| - region string // LCHAT_MESH_REGION — this box's placement region (e.g. "ie"); "" = region-agnostic (today's behaviour). Advertised in the heartbeat body; peers read it via Regions. See candidatesFor. |
| + domains []string // LCHAT_MESH_DOMAINS, lowercased — see inScope. Empty = unrestricted (test convenience only; meshFromEnv always supplies a non-empty list) |
| + token string // LCHAT_MESH_TOKEN — shared secret authenticating box-to-box route replication (meshreplicate.go); empty = unrestricted (test convenience only) |
| + region string // LCHAT_MESH_REGION — this box's placement region (e.g. "ie"); "" = region-agnostic (today's behaviour). Advertised in the heartbeat body; peers read it via Regions. See candidatesFor. |
| reg *Registry // local registry handle, for reading a host's replicated home_region (candidatesFor). nil = no lookup (direct-newMesh unit tests) = every space region-agnostic. |
| interval time.Duration |
| staleAfter time.Duration |
| @@ -384,6 +384,35 @@ func (m *meshSvc) homeRegionOf(host string) string { |
| return r.HomeRegion |
| } |
| +// regionLive reports whether any live box currently advertises region — the |
| +// precondition `lchat relocate` checks before flipping a space's home_region |
| +// onto a destination region: there is no point re-homing a space to a region |
| +// no box serves (candidatesFor would just fall back to the whole fleet and the |
| +// space would never actually move). It reads the same cached membership + |
| +// peer-region view HRW placement uses, so the caller must beat() first to |
| +// refresh it (fanOutMeshRoute's pattern). This box's own region is known |
| +// locally (m.region), never via S3. |
| +func (m *meshSvc) regionLive(region string) bool { |
| + if region == "" { |
| + return false |
| + } |
| + if m.region == region { |
| + return true |
| + } |
| + m.mu.Lock() |
| + regions := m.regions |
| + m.mu.Unlock() |
| + for _, b := range m.liveBoxes() { |
| + if b == m.self { |
| + continue // handled by m.region above |
| + } |
| + if regions[b] == region { |
| + return true |
| + } |
| + } |
| + return false |
| +} |
| + |
| // inScope reports whether host participates in HRW placement at all. Only |
| // hosts under a configured LCHAT_MESH_DOMAINS domain are ever hashed across |
| // boxes; everything else always resolves to whichever box is asked — correct |
diff --git a/registry.go b/registry.go| index f7d4a4b..a19371a 100644 |
| --- a/registry.go |
| +++ b/registry.go |
| @@ -289,6 +289,21 @@ func (r *Registry) setReplication(host, class string) error { |
| return err |
| } |
| +// setHomeRegion records host's placement region (multi-edge, docs/multi-edge.md) |
| +// without touching the rest of the route — the same narrow-UPDATE contract as |
| +// setHibernate/setReplication, so `lchat relocate` can re-home a space onto a |
| +// new region without clobbering a concurrent next_due write or a deploy's |
| +// read-modify-write (and vice versa). The mesh reads this replicated column to |
| +// place the space on a box in the matching region (mesh.go's candidatesFor), so |
| +// flipping it IS the whole mechanic of relocation — the hash re-homes the space |
| +// and the single-writer lease still guarantees one writer through the move. An |
| +// unknown host is a quiet no-op, matching setHibernate; region "" means |
| +// region-agnostic (today's default, hashed across the whole fleet). |
| +func (r *Registry) setHomeRegion(host, region string) error { |
| + _, err := r.sql.Exec(`UPDATE routes SET home_region = ? WHERE host = ?`, region, host) |
| + return err |
| +} |
| + |
| // setVersions records a host's current + previous version, same narrow-UPDATE |
| // contract as setHibernate/setNextDue. Deploys and update passes hold their |
| // route reads for a while (minutes, when a download runs) — a full upsert |
diff --git a/relocate.go b/relocate.go| new file mode 100644 |
| index 0000000..1d57ae1 |
| --- /dev/null |
| +++ b/relocate.go |
| @@ -0,0 +1,462 @@ |
| +package main |
| + |
| +// 🤖 `lchat relocate <host> --to <region>` — move a space to another region |
| +// while keeping its STABLE ADDRESS (multi-edge Phase 1, docs/multi-edge.md). |
| +// |
| +// A space's address is its identity (invite links, per-origin keys bind to it), |
| +// so it must never change; its region is a mutable property (`home_region`), so |
| +// it must be free to move. Relocation is the operator/admin action that changes |
| +// where the one copy lives without changing its name. Three moving parts, in the |
| +// order they run: |
| +// |
| +// 1. Store copy. The destination region's box wakes the space by RESTORING its |
| +// DB from that region's object store, so the replica must exist there first. |
| +// On the dev mesh both regions share ONE bucket, so the bytes are already |
| +// visible — the copy is a NO-OP (relocationStorePlan detects same |
| +// bucket+prefix+endpoint → skip). In production each region has its own |
| +// store, so it is a real prefix-to-prefix copy of s3://<bucket>/<host>/… |
| +// (copyS3Prefix), done while the space sleeps. |
| +// 2. Flip home_region + re-fan-out. setHomeRegion writes the replicated routes |
| +// column; fanOutMeshRoute pushes the updated route to peers so every box's |
| +// candidatesFor (mesh.go) re-homes the space onto a box in the destination |
| +// region. The local router needs no reload — its mesh reads home_region live |
| +// from the shared registry each placement decision. |
| +// 3. (Optional) A-record move. Repointing <host>'s A record at the destination |
| +// box avoids the proxy hop, but is NOT required for correctness: the |
| +// *.<domain> wildcard + the mesh's proxy-to-owner already route any request |
| +// to the new owner through the DNS-TTL window. Off by default; --move-dns |
| +// opts in, and only Route53 implements it today (route53dns.go). |
| +// |
| +// WHY THIS IS DATA-SAFE — the single-writer lease. Relocation never |
| +// split-brains because the S3 lease (internal/activator: lock.json under the |
| +// space's prefix, acquired via a conditional write) is the ONE truth of who may |
| +// write, and home_region/DNS are only hints that steer placement: |
| +// |
| +// - Shared bucket (dev): source and destination contend on the SAME lock.json, |
| +// so the lease is globally singular. After the flip the destination becomes |
| +// the HRW owner; if the source is momentarily still awake it holds the lease, |
| +// the destination's AcquireLease fails loudly (LeaseExistsError → 503 + |
| +// Retry-After), and the client's retry lands once the source sleeps and |
| +// releases. Never two writers, ever. |
| +// - Per-region buckets (prod): the copy must run while the source is ASLEEP |
| +// (lease released, replica fully flushed by the sleep-time SyncAndWait), so |
| +// the destination restores a complete copy and acquires a FRESH lease in its |
| +// own bucket. Once home_region is flipped the source's owns() returns false, |
| +// so it never wakes there again — a stray request proxies to the new owner |
| +// instead. copyS3Prefix deliberately skips lock.json so a copied, live lease |
| +// can't stall the destination's first wake. |
| +// |
| +// IDEMPOTENCY + the copy-clobber guard. Re-running a completed relocation is a |
| +// no-op: once home_region already equals the target the space's authoritative |
| +// copy is the DESTINATION's, so relocate must NOT re-copy source→dest (that |
| +// would overwrite newer destination writes with stale source bytes). So the |
| +// copy + flip run ONLY on the forward move (current != target); an |
| +// already-home re-run just heals peers by re-fanning-out the route. A crash |
| +// between copy and flip converges safely: the source is still authoritative |
| +// (not yet flipped), so the next run re-copies (idempotent overwrite) and flips. |
| + |
| +import ( |
| + "context" |
| + "flag" |
| + "fmt" |
| + "log" |
| + "os" |
| + "path" |
| + "strings" |
| + |
| + "github.com/aws/aws-sdk-go-v2/aws" |
| + awsconfig "github.com/aws/aws-sdk-go-v2/config" |
| + "github.com/aws/aws-sdk-go-v2/credentials" |
| + awss3 "github.com/aws/aws-sdk-go-v2/service/s3" |
| +) |
| + |
| +// storeLocation names one region's object store for a space's replica prefix. |
| +// Endpoint distinguishes same-named buckets across accounts/providers (AWS S3 |
| +// vs an R2 endpoint) so the skip test in relocationStorePlan is never fooled by |
| +// a bucket-name coincidence. |
| +type storeLocation struct { |
| + Bucket string |
| + Prefix string // the per-host key prefix, e.g. "<hibernate-prefix>/<host>" |
| + Endpoint string // "" = AWS default endpoint; set for R2 / custom endpoints |
| +} |
| + |
| +// storePlan is relocationStorePlan's decision: whether the cross-region store |
| +// copy is a no-op (the dev shared bucket — the bytes are already there) or a |
| +// real prefix-to-prefix copy (prod's per-region stores). |
| +type storePlan struct { |
| + Skip bool |
| + Reason string |
| + Src storeLocation |
| + Dst storeLocation |
| +} |
| + |
| +// relocationStorePlan is the PURE decision at the heart of the store-copy step, |
| +// split out so it is unit-testable with no S3. It returns skip=true exactly when |
| +// the source and destination name the same bytes — same bucket, same per-host |
| +// prefix, and same endpoint — which is the dev mesh's one-shared-bucket case: |
| +// both regions already read the same replica, so there is nothing to copy. |
| +// Anything else is a real copy plan the caller executes with copyS3Prefix. |
| +func relocationStorePlan(src, dst storeLocation) storePlan { |
| + if src.Bucket == dst.Bucket && src.Prefix == dst.Prefix && src.Endpoint == dst.Endpoint { |
| + return storePlan{ |
| + Skip: true, |
| + Reason: "source and destination share the same bucket, prefix and endpoint — the replica bytes are already visible to the destination region (the dev shared-bucket case)", |
| + Src: src, Dst: dst, |
| + } |
| + } |
| + return storePlan{ |
| + Skip: false, |
| + Reason: fmt.Sprintf("cross-region prefix copy: s3://%s/%s/ → s3://%s/%s/", src.Bucket, src.Prefix, dst.Bucket, dst.Prefix), |
| + Src: src, Dst: dst, |
| + } |
| +} |
| + |
| +// s3CopyAPI is the slice of the S3 client copyS3Prefix needs — an interface so |
| +// the copy is unit-tested against a fake without reaching AWS (the same shape |
| +// s3blobstore.go and route53dns.go use for their clients). |
| +type s3CopyAPI interface { |
| + ListObjectsV2(ctx context.Context, in *awss3.ListObjectsV2Input, optFns ...func(*awss3.Options)) (*awss3.ListObjectsV2Output, error) |
| + CopyObject(ctx context.Context, in *awss3.CopyObjectInput, optFns ...func(*awss3.Options)) (*awss3.CopyObjectOutput, error) |
| +} |
| + |
| +// copyS3Prefix copies every object under src's per-host prefix to dst's, |
| +// preserving the relative key layout, via server-side CopyObject (no bytes |
| +// transit this process). It EXCLUDES the lease object (lock.json): a copied, |
| +// still-valid lease would make the destination's first AcquireLease see a held |
| +// lock and 503 until the TTL expires — the destination must start with a clean |
| +// lease and take its own. Returns the count copied. |
| +// |
| +// Same-account, same-credential copy is the realistic prod shape (regional |
| +// buckets in one AWS account; CopyObject works cross-region). Cross-ACCOUNT or |
| +// cross-ENDPOINT (e.g. an R2 destination) needs a download+reupload variant — |
| +// a documented follow-on (docs/multi-edge.md), not this MVP. |
| +func copyS3Prefix(ctx context.Context, client s3CopyAPI, srcBucket, srcPrefix, dstBucket, dstPrefix string) (int, error) { |
| + srcPrefix = strings.Trim(srcPrefix, "/") |
| + dstPrefix = strings.Trim(dstPrefix, "/") |
| + listPrefix := srcPrefix + "/" |
| + p := awss3.NewListObjectsV2Paginator(client, &awss3.ListObjectsV2Input{ |
| + Bucket: aws.String(srcBucket), |
| + Prefix: aws.String(listPrefix), |
| + }) |
| + copied := 0 |
| + for p.HasMorePages() { |
| + page, err := p.NextPage(ctx) |
| + if err != nil { |
| + return copied, fmt.Errorf("relocate: list s3://%s/%s: %w", srcBucket, listPrefix, err) |
| + } |
| + for _, obj := range page.Contents { |
| + srcKey := aws.ToString(obj.Key) |
| + rel := strings.TrimPrefix(srcKey, listPrefix) |
| + if rel == "" || path.Base(rel) == "lock.json" { |
| + continue // never copy the lease — the destination must take a fresh one |
| + } |
| + dstKey := dstPrefix + "/" + rel |
| + // CopySource is "<bucket>/<key>"; our keys are ASCII-safe (host FQDN |
| + // prefixes + litestream LTX / base64url leaves), so no URL-escaping |
| + // is needed. A cross-account/cross-endpoint copier would encode here. |
| + copySource := srcBucket + "/" + srcKey |
| + if _, err := client.CopyObject(ctx, &awss3.CopyObjectInput{ |
| + Bucket: aws.String(dstBucket), |
| + Key: aws.String(dstKey), |
| + CopySource: aws.String(copySource), |
| + }); err != nil { |
| + return copied, fmt.Errorf("relocate: copy %s → s3://%s/%s: %w", copySource, dstBucket, dstKey, err) |
| + } |
| + copied++ |
| + } |
| + } |
| + return copied, nil |
| +} |
| + |
| +// aRecordUpserter is the OPTIONAL A-record half of relocation — a sibling to the |
| +// dnsProvider seam (wildcard.go), kept separate so the ACME-TXT path stays |
| +// untouched and a provider can support DNS-01 without supporting A moves. |
| +// route53DNS implements it (route53dns.go); DNSimple does not yet, so --move-dns |
| +// refuses on DNSimple with a clear message. Relocation is correct WITHOUT it |
| +// (the wildcard + proxy route to the new owner), so this is never on the MVP's |
| +// critical path. |
| +type aRecordUpserter interface { |
| + upsertA(ctx context.Context, zone, host, ipv4 string) error |
| +} |
| + |
| +// relocateOpts parameterizes one relocation. Reusable so a future controld RPC |
| +// or member-facing "change region" button can call relocate() with the same |
| +// core (the CLI runRelocate is one caller). |
| +type relocateOpts struct { |
| + Host string |
| + ToRegion string |
| + // Dst names the destination region's object store for the cross-region copy. |
| + // The zero value means "the same store as the source" — the dev shared-bucket |
| + // case, where the copy is a no-op. Prefix defaults to the source's per-host |
| + // prefix when empty. |
| + Dst storeLocation |
| + // MoveDNS opts into repointing <host>'s A record at DNSTarget (off by |
| + // default). DNSZone overrides the hosted zone; empty derives it from the |
| + // wildcard/mesh domains the host falls under. |
| + MoveDNS bool |
| + DNSTarget string |
| + DNSZone string |
| +} |
| + |
| +// relocateResult reports what happened, for the CLI to print. |
| +type relocateResult struct { |
| + Host string |
| + FromRegion string |
| + ToRegion string |
| + Plan storePlan |
| + Copied int // objects copied (0 on the skip path) |
| + AlreadyHome bool // home_region already == ToRegion — a converging no-op |
| + DNSMoved bool |
| +} |
| + |
| +// relocate is the shared relocation core behind `lchat relocate` (and any future |
| +// RPC/button). It validates, does the store copy (or skips it), flips |
| +// home_region, re-fans-out the route, and optionally moves the A record. mesh |
| +// answers the "is the destination region live?" precondition and is beaten once |
| +// here for a fresh view; the actual route replication reuses fanOutMeshRoute. |
| +func relocate(reg *Registry, mesh *meshSvc, hc *hibernateConfig, opts relocateOpts) (*relocateResult, error) { |
| + host := strings.ToLower(strings.TrimSpace(opts.Host)) |
| + if !validHost(host) { |
| + return nil, fmt.Errorf("%w: %q", errInvalidHost, host) |
| + } |
| + region := strings.TrimSpace(opts.ToRegion) |
| + if region == "" { |
| + return nil, fmt.Errorf("relocate: --to <region> is required (clearing a space's region back to agnostic is a separate operation, not relocate)") |
| + } |
| + if hc == nil { |
| + return nil, fmt.Errorf("relocate: hibernation is not configured on this box (LCHAT_HIBERNATE_BUCKET unset) — relocation is hibernation-backed: the space's state must live in S3 for the destination region to restore it") |
| + } |
| + if mesh == nil { |
| + return nil, fmt.Errorf("relocate: the activator mesh is not configured (LCHAT_MESH_SELF/_DOMAINS/_TOKEN/_REGION) — placement-by-region needs it; run on a mesh box with /data/router.env sourced") |
| + } |
| + |
| + rt, err := reg.get(host) |
| + if err != nil { |
| + return nil, err |
| + } |
| + if rt == nil { |
| + return nil, fmt.Errorf("%w: %q (run relocate on a box whose registry has the route — typically the source region's box)", errUnknownHost, host) |
| + } |
| + if rt.Kind != "instance" { |
| + return nil, fmt.Errorf("relocate: %q is a %s route, not an instance", host, rt.Kind) |
| + } |
| + if rt.Status != "ready" { |
| + return nil, fmt.Errorf("relocate: %s is %s, not ready — fix the instance first", host, rt.Status) |
| + } |
| + if !rt.Hibernate { |
| + return nil, fmt.Errorf("relocate: %s is systemd-managed, not hibernating — a live systemd instance is pinned to this box; run `lchat hibernate %s --on` first so its state lives in S3 and can be restored in the destination region", host, host) |
| + } |
| + if !mesh.inScope(host) { |
| + return nil, fmt.Errorf("relocate: %s is not under a configured mesh domain (%s) — only in-scope hosts are placed by region", host, strings.Join(mesh.domains, ", ")) |
| + } |
| + |
| + res := &relocateResult{Host: host, FromRegion: rt.HomeRegion, ToRegion: region} |
| + |
| + // Idempotent + copy-clobber guard: once the space is already homed to the |
| + // target, the DESTINATION holds the authoritative copy. Re-copying source→dest |
| + // would clobber newer destination writes, so DON'T. Just heal any peer that |
| + // missed the earlier flip by re-fanning-out the route, and return. |
| + if rt.HomeRegion == region { |
| + res.AlreadyHome = true |
| + here := storeLocation{Bucket: hc.Bucket, Prefix: hc.hibernatePrefix(host)} |
| + res.Plan = relocationStorePlan(here, here) // same store → Skip |
| + fanOutMeshRoute(hc, *rt) // best-effort heal; a no-op when the mesh is off |
| + return res, nil |
| + } |
| + |
| + // The destination region must have a live box, or the flip re-homes the space |
| + // onto nothing (candidatesFor would just fall back to the whole fleet and the |
| + // space would not actually move). beat() first, exactly like fanOutMeshRoute, |
| + // so the membership + peer-region view is fresh. |
| + mesh.beat() |
| + if !mesh.regionLive(region) { |
| + return nil, fmt.Errorf("relocate: no live mesh box advertises region %q — bring up a box with LCHAT_MESH_REGION=%s (and confirm this box has its own LCHAT_MESH_REGION set so it can read peers' regions)", region, region) |
| + } |
| + |
| + // Step 1 — the store copy (or the dev no-op). |
| + src := storeLocation{Bucket: hc.Bucket, Prefix: hc.hibernatePrefix(host)} |
| + dst := opts.Dst |
| + if dst.Bucket == "" { |
| + dst = src // no destination store named → same store (dev shared bucket) |
| + } |
| + if dst.Prefix == "" { |
| + dst.Prefix = hc.hibernatePrefix(host) |
| + } |
| + res.Plan = relocationStorePlan(src, dst) |
| + if !res.Plan.Skip { |
| + copied, err := runStoreCopy(context.Background(), hc.Region, res.Plan) |
| + if err != nil { |
| + return nil, err |
| + } |
| + res.Copied = copied |
| + } |
| + |
| + // Step 2 — flip home_region and re-fan-out the updated route to peers. |
| + if err := reg.setHomeRegion(host, region); err != nil { |
| + return nil, fmt.Errorf("relocate: set home_region: %w", err) |
| + } |
| + updated := *rt |
| + updated.HomeRegion = region |
| + fanOutMeshRoute(hc, updated) |
| + |
| + // Step 3 — optional A-record move. |
| + if opts.MoveDNS { |
| + if err := moveARecord(context.Background(), mesh, host, opts); err != nil { |
| + // The flip already succeeded, so relocation is correct via the proxy; |
| + // surface the DNS failure but don't unwind the (correct) move. |
| + return res, fmt.Errorf("relocate: home_region flipped to %s, but the optional A-record move failed: %w (relocation is still correct — the wildcard + proxy route to the new owner)", region, err) |
| + } |
| + res.DNSMoved = true |
| + } |
| + return res, nil |
| +} |
| + |
| +// runStoreCopy builds one S3 client and runs copyS3Prefix. Only reached on the |
| +// cross-bucket (prod) path — the dev shared-bucket case skips before here. Uses |
| +// the source region as the signing region (a same-account client copies |
| +// cross-region via CopyObject regardless) and, when the destination names a |
| +// custom endpoint (e.g. R2), addresses it path-style. Optional static |
| +// credentials for the destination store ride LCHAT_RELOCATE_DST_ACCESS_KEY / |
| +// _SECRET_KEY, mirroring s3blobstore.go's mirror-store pattern; unset = the |
| +// default AWS chain. |
| +func runStoreCopy(ctx context.Context, srcRegion string, plan storePlan) (int, error) { |
| + var opts []func(*awsconfig.LoadOptions) error |
| + if srcRegion != "" { |
| + opts = append(opts, awsconfig.WithRegion(srcRegion)) |
| + } |
| + if ak := strings.TrimSpace(os.Getenv("LCHAT_RELOCATE_DST_ACCESS_KEY")); ak != "" { |
| + opts = append(opts, awsconfig.WithCredentialsProvider( |
| + credentials.NewStaticCredentialsProvider(ak, strings.TrimSpace(os.Getenv("LCHAT_RELOCATE_DST_SECRET_KEY")), ""))) |
| + } |
| + cfg, err := awsconfig.LoadDefaultConfig(ctx, opts...) |
| + if err != nil { |
| + return 0, fmt.Errorf("relocate: load aws config for copy: %w", err) |
| + } |
| + client := awss3.NewFromConfig(cfg, func(o *awss3.Options) { |
| + if plan.Dst.Endpoint != "" { |
| + o.BaseEndpoint = aws.String(plan.Dst.Endpoint) |
| + o.UsePathStyle = true |
| + } |
| + }) |
| + log.Printf("relocate: %s", plan.Reason) |
| + return copyS3Prefix(ctx, client, plan.Src.Bucket, plan.Src.Prefix, plan.Dst.Bucket, plan.Dst.Prefix) |
| +} |
| + |
| +// moveARecord repoints host's A record via whichever DNS provider is configured, |
| +// if it supports A upserts. Off unless the caller passed --move-dns. |
| +func moveARecord(ctx context.Context, mesh *meshSvc, host string, opts relocateOpts) error { |
| + if strings.TrimSpace(opts.DNSTarget) == "" { |
| + return fmt.Errorf("--move-dns needs --dns-target <ipv4> (the destination box's address)") |
| + } |
| + prov, reason := dnsProviderFromEnv() |
| + if prov == nil { |
| + return fmt.Errorf("no DNS provider configured: %s", reason) |
| + } |
| + up, ok := prov.(aRecordUpserter) |
| + if !ok { |
| + return fmt.Errorf("the configured DNS provider can't move A records yet (only LCHAT_DNS_PROVIDER=route53 does) — omit --move-dns: relocation is correct without it (the wildcard + proxy route to the new owner)") |
| + } |
| + zone := strings.TrimSpace(opts.DNSZone) |
| + if zone == "" { |
| + zone = zoneForHost(host, mesh.domains) |
| + } |
| + if zone == "" { |
| + return fmt.Errorf("could not derive a hosted zone for %s from the mesh domains — pass --dns-zone", host) |
| + } |
| + return up.upsertA(ctx, zone, host, strings.TrimSpace(opts.DNSTarget)) |
| +} |
| + |
| +// zoneForHost picks the hosted zone for host from the configured domains: the |
| +// longest domain that host is a subdomain of (so a delegated sub-zone wins over |
| +// its parent). "" when none matches. |
| +func zoneForHost(host string, domains []string) string { |
| + host = strings.ToLower(strings.TrimSuffix(host, ".")) |
| + best := "" |
| + for _, d := range domains { |
| + d = strings.ToLower(strings.TrimSpace(d)) |
| + if d == "" { |
| + continue |
| + } |
| + if strings.HasSuffix(host, "."+d) && len(d) > len(best) { |
| + best = d |
| + } |
| + } |
| + return best |
| +} |
| + |
| +// runRelocate: `lchat relocate <host> --to <region> [flags]`. |
| +func runRelocate(args []string) { |
| + fs := flag.NewFlagSet("relocate", flag.ExitOnError) |
| + registryPath := fs.String("registry", envOr("LCHAT_REGISTRY", "/data/registry.db"), "router registry SQLite path") |
| + to := fs.String("to", "", "destination region (matches a live box's LCHAT_MESH_REGION) — required") |
| + dstBucket := fs.String("dst-bucket", "", "destination region's object-store bucket (prod per-region stores); empty = same store as source (the dev shared-bucket no-op)") |
| + dstPrefix := fs.String("dst-prefix", "", "destination per-host key prefix; empty = the same prefix the source uses") |
| + dstEndpoint := fs.String("dst-endpoint", "", "destination S3 endpoint (e.g. an R2 URL); empty = AWS default") |
| + moveDNS := fs.Bool("move-dns", false, "also repoint the host's A record at the destination box (OPTIONAL — off by default; relocation is correct without it, the wildcard + proxy route to the new owner)") |
| + dnsTarget := fs.String("dns-target", "", "destination box IPv4 for the A record (required with --move-dns)") |
| + dnsZone := fs.String("dns-zone", "", "hosted zone for the A record; empty = derived from the mesh domains") |
| + _ = fs.Parse(reorderFlagsFirst(fs, args)) |
| + |
| + host := strings.ToLower(strings.TrimSpace(fs.Arg(0))) |
| + if !validHost(host) { |
| + die("relocate: invalid host %q (usage: lchat relocate <host> --to <region>)", host) |
| + } |
| + region := strings.TrimSpace(*to) |
| + if region == "" { |
| + die("relocate: --to <region> is required") |
| + } |
| + |
| + hc, err := hibernateConfigFromEnv() |
| + if err != nil { |
| + die("relocate: %v", err) |
| + } |
| + mesh, err := meshFromEnv(hc) |
| + if err != nil { |
| + die("relocate: mesh config: %v", err) |
| + } |
| + |
| + reg, err := openRegistry(*registryPath) |
| + if err != nil { |
| + die("relocate: open registry: %v", err) |
| + } |
| + if mesh != nil { |
| + mesh.reg = reg // so any region read resolves against the local registry |
| + } |
| + |
| + res, err := relocate(reg, mesh, hc, relocateOpts{ |
| + Host: host, |
| + ToRegion: region, |
| + Dst: storeLocation{ |
| + Bucket: strings.TrimSpace(*dstBucket), |
| + Prefix: strings.TrimSpace(*dstPrefix), |
| + Endpoint: strings.TrimSpace(*dstEndpoint), |
| + }, |
| + MoveDNS: *moveDNS, |
| + DNSTarget: strings.TrimSpace(*dnsTarget), |
| + DNSZone: strings.TrimSpace(*dnsZone), |
| + }) |
| + if err != nil { |
| + die("relocate: %v", err) |
| + } |
| + |
| + if res.AlreadyHome { |
| + fmt.Printf("%s is already homed to %s — re-fanned the route out to heal any peer that missed it (no copy, no re-flip)\n", res.Host, res.ToRegion) |
| + return |
| + } |
| + from := res.FromRegion |
| + if from == "" { |
| + from = "(agnostic)" |
| + } |
| + fmt.Printf("relocated %s: %s → %s\n", res.Host, from, res.ToRegion) |
| + if res.Plan.Skip { |
| + fmt.Printf(" store: no copy — %s\n", res.Plan.Reason) |
| + } else { |
| + fmt.Printf(" store: copied %d object(s) — %s\n", res.Copied, res.Plan.Reason) |
| + } |
| + fmt.Printf(" home_region flipped + route re-fanned-out to peers — the mesh now places %s on a %s box (the single-writer lease keeps one writer through the move)\n", res.Host, res.ToRegion) |
| + if res.DNSMoved { |
| + fmt.Printf(" A record moved to the destination box (%s)\n", strings.TrimSpace(*dnsTarget)) |
| + } else { |
| + fmt.Printf(" A record unchanged — the wildcard + mesh proxy route to the new owner through the DNS-TTL window (use --move-dns --dns-target to avoid the proxy hop)\n") |
| + } |
| +} |
diff --git a/relocate_test.go b/relocate_test.go| new file mode 100644 |
| index 0000000..bd109d1 |
| --- /dev/null |
| +++ b/relocate_test.go |
| @@ -0,0 +1,304 @@ |
| +package main |
| + |
| +// Tests for relocate.go — the multi-edge Phase 1 "relocate" action. Three |
| +// concerns, matching the load-bearing decisions: |
| +// |
| +// - relocationStorePlan: the PURE skip-vs-copy decision (dev's shared bucket |
| +// is a no-op; anything differing is a real copy). |
| +// - copyS3Prefix: the cross-region prefix copy — key layout preserved, the |
| +// lease (lock.json) EXCLUDED, tested against a fake S3 client. |
| +// - relocate: target-region validation, the home_region flip, and the |
| +// idempotent + copy-clobber guard (an already-home re-run never re-copies). |
| + |
| +import ( |
| + "context" |
| + "path/filepath" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/aws/aws-sdk-go-v2/aws" |
| + awss3 "github.com/aws/aws-sdk-go-v2/service/s3" |
| + s3types "github.com/aws/aws-sdk-go-v2/service/s3/types" |
| +) |
| + |
| +// --- relocationStorePlan (pure) ------------------------------------------------ |
| + |
| +func TestRelocationStorePlan(t *testing.T) { |
| + base := storeLocation{Bucket: "shared", Prefix: "hib/space1.dev.11ch.at", Endpoint: ""} |
| + |
| + // Same bucket + prefix + endpoint = the dev shared-bucket case: the bytes are |
| + // already visible to both regions, so the copy is a no-op. |
| + if p := relocationStorePlan(base, base); !p.Skip { |
| + t.Fatalf("identical src/dst should skip, got copy: %+v", p) |
| + } |
| + |
| + // A different bucket is a real copy (prod's per-region stores). |
| + dstBucket := base |
| + dstBucket.Bucket = "fi-bucket" |
| + if p := relocationStorePlan(base, dstBucket); p.Skip { |
| + t.Fatal("different bucket must NOT skip — it needs a real copy") |
| + } |
| + |
| + // A different prefix is a real copy. |
| + dstPrefix := base |
| + dstPrefix.Prefix = "hib2/space1.dev.11ch.at" |
| + if p := relocationStorePlan(base, dstPrefix); p.Skip { |
| + t.Fatal("different prefix must NOT skip") |
| + } |
| + |
| + // A same-named bucket at a DIFFERENT endpoint (S3 vs R2) is different bytes — |
| + // the endpoint must be part of the identity test. |
| + dstEndpoint := base |
| + dstEndpoint.Endpoint = "https://acct.r2.cloudflarestorage.com" |
| + if p := relocationStorePlan(base, dstEndpoint); p.Skip { |
| + t.Fatal("different endpoint must NOT skip — a bucket-name coincidence is not the same bytes") |
| + } |
| +} |
| + |
| +// --- copyS3Prefix -------------------------------------------------------------- |
| + |
| +// fakeS3Copy is an in-memory stand-in for the S3 client copyS3Prefix drives: |
| +// ListObjectsV2 returns the seeded keys under the requested prefix (one page), |
| +// CopyObject records every call. |
| +type fakeS3Copy struct { |
| + objects []string |
| + copies []*awss3.CopyObjectInput |
| +} |
| + |
| +func (f *fakeS3Copy) ListObjectsV2(_ context.Context, in *awss3.ListObjectsV2Input, _ ...func(*awss3.Options)) (*awss3.ListObjectsV2Output, error) { |
| + prefix := aws.ToString(in.Prefix) |
| + var contents []s3types.Object |
| + for _, k := range f.objects { |
| + if strings.HasPrefix(k, prefix) { |
| + contents = append(contents, s3types.Object{Key: aws.String(k)}) |
| + } |
| + } |
| + return &awss3.ListObjectsV2Output{Contents: contents}, nil |
| +} |
| + |
| +func (f *fakeS3Copy) CopyObject(_ context.Context, in *awss3.CopyObjectInput, _ ...func(*awss3.Options)) (*awss3.CopyObjectOutput, error) { |
| + f.copies = append(f.copies, in) |
| + return &awss3.CopyObjectOutput{}, nil |
| +} |
| + |
| +func TestCopyS3Prefix(t *testing.T) { |
| + src := "hib/space1.dev.11ch.at" |
| + dst := "hib/space1.dev.11ch.at" // same per-host prefix, different bucket |
| + fake := &fakeS3Copy{objects: []string{ |
| + src + "/lock.json", // the lease — MUST be skipped |
| + src + "/ltx/0001.ltx", // replica data |
| + src + "/snapshots/000.snap", // replica data |
| + "other/space2.dev.11ch.at/ltx/0001.ltx", // a different host — out of prefix |
| + }} |
| + |
| + n, err := copyS3Prefix(context.Background(), fake, "src-bucket", src, "fi-bucket", dst) |
| + if err != nil { |
| + t.Fatalf("copyS3Prefix: %v", err) |
| + } |
| + if n != 2 { |
| + t.Fatalf("copied %d objects, want 2 (lock.json excluded, other host out of prefix)", n) |
| + } |
| + if len(fake.copies) != 2 { |
| + t.Fatalf("recorded %d CopyObject calls, want 2", len(fake.copies)) |
| + } |
| + got := map[string]string{} // dst key -> CopySource |
| + for _, c := range fake.copies { |
| + if aws.ToString(c.Bucket) != "fi-bucket" { |
| + t.Fatalf("copy dst bucket = %q, want fi-bucket", aws.ToString(c.Bucket)) |
| + } |
| + got[aws.ToString(c.Key)] = aws.ToString(c.CopySource) |
| + } |
| + // The lease was never copied. |
| + for k := range got { |
| + if strings.HasSuffix(k, "lock.json") { |
| + t.Fatalf("lock.json was copied (%q) — a copied lease stalls the destination's first wake", k) |
| + } |
| + } |
| + // Keys map into the destination prefix, CopySource points at the source. |
| + if cs, ok := got["hib/space1.dev.11ch.at/ltx/0001.ltx"]; !ok || cs != "src-bucket/hib/space1.dev.11ch.at/ltx/0001.ltx" { |
| + t.Fatalf("ltx object: dstKey/CopySource wrong (%q -> %q)", "hib/space1.dev.11ch.at/ltx/0001.ltx", cs) |
| + } |
| + if _, ok := got["hib/space1.dev.11ch.at/snapshots/000.snap"]; !ok { |
| + t.Fatalf("snapshot object was not copied to the right dst key; got %v", got) |
| + } |
| +} |
| + |
| +// --- relocate: validation + flip + idempotency --------------------------------- |
| + |
| +// relocTestReg seeds an in-memory registry with one hibernating, ready, |
| +// in-scope instance route homed to fromRegion. |
| +func relocTestReg(t *testing.T, host, fromRegion string, hibernate bool, kind string) *Registry { |
| + t.Helper() |
| + reg, err := openRegistry(filepath.Join(t.TempDir(), "registry.db")) |
| + if err != nil { |
| + t.Fatalf("open registry: %v", err) |
| + } |
| + if err := reg.upsert(Route{ |
| + Host: host, Kind: kind, Status: "ready", |
| + Hibernate: hibernate, HomeRegion: fromRegion, |
| + }); err != nil { |
| + t.Fatalf("seed route: %v", err) |
| + } |
| + return reg |
| +} |
| + |
| +// relocTestMesh builds a mesh over a fake heartbeat store: this box is |
| +// selfRegion, and each peer in peers advertises its region. relocate.beat() |
| +// refreshes membership + regions off this store. |
| +func relocTestMesh(reg *Registry, self, selfRegion string, peers map[string]string) *meshSvc { |
| + now := time.Now() |
| + store := newFakeMeshStore(func() time.Time { return now }) |
| + for peer, region := range peers { |
| + store.setBeat(peer, now) |
| + store.setRegion(peer, region) |
| + } |
| + m := newMesh(self, store, []string{"dev.11ch.at"}) |
| + m.region = selfRegion |
| + m.reg = reg |
| + m.now = func() time.Time { return now } |
| + return m |
| +} |
| + |
| +func TestRelocateFlipAndIdempotency(t *testing.T) { |
| + host := "space1.dev.11ch.at" |
| + reg := relocTestReg(t, host, "ie", true, "instance") |
| + mesh := relocTestMesh(reg, "https://ie-box", "ie", map[string]string{"https://fi-box": "fi"}) |
| + hc := &hibernateConfig{Bucket: "shared", Prefix: "hib"} // one shared bucket → skip path |
| + |
| + // Forward move ie → fi: flips home_region, skip copy (shared bucket). |
| + res, err := relocate(reg, mesh, hc, relocateOpts{Host: host, ToRegion: "fi"}) |
| + if err != nil { |
| + t.Fatalf("relocate: %v", err) |
| + } |
| + if res.AlreadyHome { |
| + t.Fatal("first move should not be AlreadyHome") |
| + } |
| + if !res.Plan.Skip || res.Copied != 0 { |
| + t.Fatalf("shared bucket should skip the copy, got plan=%+v copied=%d", res.Plan, res.Copied) |
| + } |
| + if rt, _ := reg.get(host); rt == nil || rt.HomeRegion != "fi" { |
| + t.Fatalf("home_region not flipped to fi: %+v", rt) |
| + } |
| + |
| + // Re-run to the same region: AlreadyHome no-op, home_region unchanged, and — |
| + // critically — NO copy even though we now point at a DIFFERENT bucket (the |
| + // copy-clobber guard: the destination is authoritative once home, so |
| + // re-copying stale source bytes would lose newer writes). |
| + res2, err := relocate(reg, mesh, hc, relocateOpts{ |
| + Host: host, ToRegion: "fi", |
| + Dst: storeLocation{Bucket: "some-other-bucket"}, // would force a copy if reached |
| + }) |
| + if err != nil { |
| + t.Fatalf("idempotent relocate: %v", err) |
| + } |
| + if !res2.AlreadyHome { |
| + t.Fatal("re-run to the same region must be AlreadyHome") |
| + } |
| + if res2.Copied != 0 || !res2.Plan.Skip { |
| + t.Fatalf("already-home re-run must NOT copy (copy-clobber guard), got copied=%d plan=%+v", res2.Copied, res2.Plan) |
| + } |
| + if rt, _ := reg.get(host); rt.HomeRegion != "fi" { |
| + t.Fatalf("home_region changed on idempotent re-run: %q", rt.HomeRegion) |
| + } |
| +} |
| + |
| +func TestRelocateValidation(t *testing.T) { |
| + host := "space1.dev.11ch.at" |
| + hc := &hibernateConfig{Bucket: "shared", Prefix: "hib"} |
| + liveFi := map[string]string{"https://fi-box": "fi"} |
| + |
| + cases := []struct { |
| + name string |
| + reg *Registry |
| + mesh *meshSvc |
| + hc *hibernateConfig |
| + opts relocateOpts |
| + wantSub string |
| + }{ |
| + { |
| + name: "missing target region", |
| + reg: relocTestReg(t, host, "ie", true, "instance"), |
| + hc: hc, |
| + opts: relocateOpts{Host: host, ToRegion: ""}, |
| + wantSub: "--to <region> is required", |
| + }, |
| + { |
| + name: "hibernation not configured", |
| + reg: relocTestReg(t, host, "ie", true, "instance"), |
| + hc: nil, |
| + opts: relocateOpts{Host: host, ToRegion: "fi"}, |
| + wantSub: "hibernation is not configured", |
| + }, |
| + { |
| + name: "mesh not configured", |
| + reg: relocTestReg(t, host, "ie", true, "instance"), |
| + hc: hc, |
| + mesh: nil, |
| + opts: relocateOpts{Host: host, ToRegion: "fi"}, |
| + wantSub: "activator mesh is not configured", |
| + }, |
| + { |
| + name: "unknown host", |
| + reg: relocTestReg(t, host, "ie", true, "instance"), |
| + hc: hc, |
| + opts: relocateOpts{Host: "nope.dev.11ch.at", ToRegion: "fi"}, |
| + wantSub: "unknown host", |
| + }, |
| + { |
| + name: "not an instance", |
| + reg: relocTestReg(t, host, "ie", true, "service"), |
| + hc: hc, |
| + opts: relocateOpts{Host: host, ToRegion: "fi"}, |
| + wantSub: "not an instance", |
| + }, |
| + { |
| + name: "not hibernating (box-pinned)", |
| + reg: relocTestReg(t, host, "ie", false, "instance"), |
| + hc: hc, |
| + opts: relocateOpts{Host: host, ToRegion: "fi"}, |
| + wantSub: "systemd-managed", |
| + }, |
| + { |
| + name: "no live box in target region", |
| + reg: relocTestReg(t, host, "ie", true, "instance"), |
| + hc: hc, |
| + opts: relocateOpts{Host: host, ToRegion: "zz"}, |
| + wantSub: "no live mesh box advertises region", |
| + }, |
| + } |
| + |
| + for _, tc := range cases { |
| + t.Run(tc.name, func(t *testing.T) { |
| + mesh := tc.mesh |
| + // A nil mesh in the case means "test the nil-mesh guard"; otherwise |
| + // build a live-fi mesh. The "mesh not configured" case sets mesh:nil |
| + // explicitly and we honour it. |
| + if mesh == nil && tc.name != "mesh not configured" { |
| + mesh = relocTestMesh(tc.reg, "https://ie-box", "ie", liveFi) |
| + } |
| + _, err := relocate(tc.reg, mesh, tc.hc, tc.opts) |
| + if err == nil { |
| + t.Fatalf("want error containing %q, got nil", tc.wantSub) |
| + } |
| + if !strings.Contains(err.Error(), tc.wantSub) { |
| + t.Fatalf("error %q does not contain %q", err.Error(), tc.wantSub) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// zoneForHost picks the longest matching configured domain (a delegated |
| +// sub-zone wins over its parent). |
| +func TestZoneForHost(t *testing.T) { |
| + domains := []string{"11ch.at", "dev.11ch.at"} |
| + if z := zoneForHost("space1.dev.11ch.at", domains); z != "dev.11ch.at" { |
| + t.Fatalf("zone = %q, want dev.11ch.at (most specific)", z) |
| + } |
| + if z := zoneForHost("chat.11ch.at", domains); z != "11ch.at" { |
| + t.Fatalf("zone = %q, want 11ch.at", z) |
| + } |
| + if z := zoneForHost("space1.example.com", domains); z != "" { |
| + t.Fatalf("zone = %q, want empty (no configured domain matches)", z) |
| + } |
| +} |
diff --git a/route53dns.go b/route53dns.go| index 0abf637..1799e6e 100644 |
| --- a/route53dns.go |
| +++ b/route53dns.go |
| @@ -110,6 +110,40 @@ func (r *route53DNS) present(ctx context.Context, zone, name, value string) (fun |
| return cleanup, nil |
| } |
| +// upsertA satisfies aRecordUpserter (relocate.go): point host's A record at |
| +// ipv4. It is the OPTIONAL, off-by-default half of relocation — moving the A |
| +// record to the destination box avoids the mesh's proxy hop, but relocation is |
| +// correct without it (the *.dev wildcard + proxy-to-owner already route any |
| +// request to the new owner; docs/multi-edge.md). UPSERT so a re-run is |
| +// idempotent (overwrites in place, never a duplicate-record error), TTL 60 so |
| +// the propagation window the proxy covers stays short, and we wait for INSYNC |
| +// like present so the caller can trust the record is live before returning. |
| +func (r *route53DNS) upsertA(ctx context.Context, zone, host, ipv4 string) error { |
| + zoneID, err := r.hostedZoneID(ctx, zone) |
| + if err != nil { |
| + return err |
| + } |
| + rrset := &r53types.ResourceRecordSet{ |
| + Name: aws.String(dnsFQDN(host)), |
| + Type: r53types.RRTypeA, |
| + TTL: aws.Int64(60), |
| + ResourceRecords: []r53types.ResourceRecord{{Value: aws.String(ipv4)}}, |
| + } |
| + out, err := r.client.ChangeResourceRecordSets(ctx, &route53.ChangeResourceRecordSetsInput{ |
| + HostedZoneId: aws.String(zoneID), |
| + ChangeBatch: &r53types.ChangeBatch{ |
| + Changes: []r53types.Change{{Action: r53types.ChangeActionUpsert, ResourceRecordSet: rrset}}, |
| + }, |
| + }) |
| + if err != nil { |
| + return fmt.Errorf("route53: upsert A %s → %s: %w", host, ipv4, err) |
| + } |
| + if out.ChangeInfo != nil { |
| + r.waitInSync(ctx, aws.ToString(out.ChangeInfo.Id)) |
| + } |
| + return nil |
| +} |
| + |
| // hostedZoneID resolves a zone to its Route53 hosted-zone id, caching the |
| // lookup. It expects the wildcard domain to have its OWN hosted zone (which is |
| // how the box's IAM role is scoped — one zone, e.g. dev.11ch.at), so it takes |
diff --git a/route53dns_test.go b/route53dns_test.go| index 606f590..2bc1ddc 100644 |
| --- a/route53dns_test.go |
| +++ b/route53dns_test.go |
| @@ -90,6 +90,42 @@ func TestRoute53Present(t *testing.T) { |
| } |
| } |
| +// upsertA (the optional relocation A-record move) UPSERTs an A record at the |
| +// host's FQDN with a short TTL and waits for INSYNC. |
| +func TestRoute53UpsertA(t *testing.T) { |
| + fake := &fakeRoute53{hostedZones: []r53types.HostedZone{ |
| + {Name: aws.String("dev.11ch.at."), Id: aws.String("/hostedzone/DEV")}, |
| + }} |
| + r := &route53DNS{client: fake, zoneIDs: map[string]string{}} |
| + |
| + if err := r.upsertA(context.Background(), "dev.11ch.at", "8oe.dev.11ch.at", "157.180.123.183"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(fake.changes) != 1 { |
| + t.Fatalf("want 1 change, got %d", len(fake.changes)) |
| + } |
| + ch := fake.changes[0].ChangeBatch.Changes[0] |
| + if ch.Action != r53types.ChangeActionUpsert { |
| + t.Fatalf("action = %v, want UPSERT (idempotent re-run)", ch.Action) |
| + } |
| + rr := ch.ResourceRecordSet |
| + if got := aws.ToString(rr.Name); got != "8oe.dev.11ch.at." { |
| + t.Fatalf("record name = %q, want the host FQDN with trailing dot", got) |
| + } |
| + if rr.Type != r53types.RRTypeA { |
| + t.Fatalf("record type = %v, want A", rr.Type) |
| + } |
| + if got := aws.ToInt64(rr.TTL); got != 60 { |
| + t.Fatalf("ttl = %d, want 60 (short — the proxy covers the window)", got) |
| + } |
| + if got := aws.ToString(rr.ResourceRecords[0].Value); got != "157.180.123.183" { |
| + t.Fatalf("record value = %q, want the target IP (unquoted — A, not TXT)", got) |
| + } |
| + if fake.getChangeN == 0 { |
| + t.Fatal("upsertA did not wait for INSYNC") |
| + } |
| +} |
| + |
| func TestRoute53HostedZoneMatch(t *testing.T) { |
| // Both the parent and the delegated sub-zone exist: the record for the |
| // sub-domain must resolve to the most specific (longest) match. |