amadan

carlos / platform Public

self-serve moves: the holding page end to end on a third edge and a custom domain

Plan 3 Task 7: nine witnesses in internal/e2e/selfserve over plan 2's
two-box world, each box with a real edge (its registry, its offline view's
holds, its home map, TLS on its own loopback address) and a third edge in
the test process that reads only the edges files, with a custom domain
attached as an explicit alias through the real alias pass.

internal/edge/e2e_bridge.go (new, e2e build tag only): arm a home map
without its loop, run one poll, set the map's clock, point the backhaul
dial at loopback, list a name's candidates. internal/host/e2e_bridge.go: a
per-box published address, the box edge's hold subscription (renewed at
every agent restart), an opt-in alias pass, a lifetimes-off publication.
Census: the internal/edge and internal/host envelopes re-pinned, the seal
unchanged; the proposal note gains Task 7 and the overlay summary.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
claude 408279cefa5175a5b397e8a43732aa0964d9f4d1 parent 27b630b
6 files changed, +1822 −3
  • docs/superpowers/notes/census-proposal-self-serve-moves-3.md +38 −0
  • internal/e2e/selfserve/holding_test.go +800 −0
  • internal/e2e/selfserve/holding_world_test.go +770 −0
  • internal/edge/e2e_bridge.go +106 −0
  • internal/host/e2e_bridge.go +105 −1
  • internal/provision/testdata/plan14-root-reads-self-serve-moves-3.tsv +3 −2

CI failed — run details

CI log 
=== gate ===
sigwho: pid 2 ppid 1 pgid 0 sid 0; child pid 3
.amadan/test-gate.sh
gate: 2026-10-24T06:14:29Z on aarch64, 32 cpus, 61367 MB available, go1.26.8 linux/arm64
gate: +0s start launchctx
gate: +0s start vet
gate: +0s start gofmt
gate: +1s start rest
gate: +10s start internal_console@0
gate: +10s start internal_console@1
gate: +10s start internal_console@2
gate: +10s start internal_console@3
gate: +10s start internal_console@4
gate: +10s start internal_console@5
gate: +10s start internal_console@6
gate: +10s start internal_console@7
gate: +10s waiting for internal_console's shards before the next split package
gate: +179s start root@0
gate: +179s start root@1
gate: +179s start root@2
gate: +179s waiting for root's shards before the next split package

amadan: step gate failed: signal: terminated

amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/docs/superpowers/notes/census-proposal-self-serve-moves-3.md b/docs/superpowers/notes/census-proposal-self-serve-moves-3.md
index 6ebee04a7..20fbcb6f1 100644
--- a/docs/superpowers/notes/census-proposal-self-serve-moves-3.md
+++ b/docs/superpowers/notes/census-proposal-self-serve-moves-3.md
@@ -113,3 +113,41 @@ Both re-pins move the seal to `f512b220…`. The `internal/edge` and `internal/c
### Task 6 review round 1
The two "not served here" bodies now share one constant, `notServedHere` in `edge/servedby.go`: the probe refusal and `homeMap.staleHopRefusal`, which reads no row. `Edge.ServeHTTP` gains a comment on why a probe may be served from the local catch-all, and is re-pinned in place. Its read sites are unchanged and the offsets moved. The `internal/edge` envelope is re-pinned, and the seal moves to `e070316b…`. No root read, class or task is added.
+
+## Task 7: the end-to-end witnesses
+
+Task 7 is tests in `internal/e2e/selfserve` (`holding_test.go`, `holding_world_test.go`), plus two seams for them. No record envelope changes. No root read, record read, class or task is added, no unit's fingerprint moves, and the seal stays `e070316b…`.
+
+- `internal/edge/e2e_bridge.go` is new and carries `//go:build e2e`, so no production binary has it. It arms an `Edge`'s home map without its loop (`E2EArmHomes`), runs one poll (`E2ERefreshHomes`), points every backhaul dial at one loopback port trusting a test CA (`E2EBackhaulDial`), and lists a name's admitted remote candidates (`E2ERemoteCandidates`). It reads no row. The census scans the file by name because the file is inactive without the tag, so its candidate type's field is named `Type` and not `Kind`, a tracked field name that would read as a row read.
+- `internal/host/e2e_bridge.go` gains a per-box published address (`BoxHandle.SetEIP`), the box edge's subscription to the box's offline view, renewed at every agent restart (`BoxHandle.SubscribeHolds`), an opt-in alias pass in each tick (`E2E.AliasPass`: `domains.ReconcileAliases`, accepted legacy, over the deployment route fold the witness hands in), and a publication as a lifetimes-off writer makes it (`BoxHandle.PublishWithoutPairs`, through `publishErr`). None of these is a unit with a read site. The bridge adds no edges-file read site either, so `internal/edgesfile`'s audit list is unchanged.
+
+The `internal/edge` and `internal/host` envelopes are re-pinned. The record-write census is unchanged: the witnesses' writes are the bridge's and the console's own.
+
+## The overlay after Task 7
+
+The loader block in `TestPlan14RootReadCensus`, after every earlier overlay block and before `compareRootReads`, verifies plan 2's seal, loads the overlay and pins this plan's seal:
+
+```go
+seal = "f7f1a9f9e33fac5247c181ac88a6888a33be2793fbb68ca5392f6bae1afffeb7"
+if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 {
+ t.Fatal(errs)
+}
+if err = loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-self-serve-moves-3.tsv", &inv, true); err != nil {
+ t.Fatal(err)
+}
+seal = "e070316bc7a483a790121f378c4d25e3868307fa58825af2126aa0611d29b2ac"
+errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...)
+```
+
+The envelopes the plan changes, each re-pinned in the overlay's `@coverage` rows:
+
+- `internal/console`: Task 6's probe (`selfserve_health.go`).
+- `internal/domains`: Task 1's provenance writers (`aliases.go`).
+- `internal/edge`: Tasks 1, 2, 3, 5 and 6 (`holding.go`, `rank.go`, `homemap.go`, `edge.go`, `certwarm.go`, `servedby.go`, `importgate.go`), and Task 7's `e2e_bridge.go`.
+- `internal/edgesfile`: Task 1's `eligible.go`.
+- `internal/host`: Tasks 1 and 2 (`edgesfile.go`, `movesoffline.go`, `movespass.go`, `agent.go`), and the bridge (`e2e_bridge.go`) in Tasks 1 and 7.
+- `internal/registry`: Task 1's column.
+
+`internal/steering` is not an envelope here. Its only change is Task 4's test file, and the census reads no test file.
+
+The registry field is `route_aliases.explicit` (`RouteAlias.Explicit`), tracked as `Explicit` in the overlay's `@fields` row. `TestPlan9B8RecordWriteCensus` takes `plan14SelfServeMoves3WriteCensusInventory`, which adds and removes nothing in this plan, and stays green.
diff --git a/internal/e2e/selfserve/holding_test.go b/internal/e2e/selfserve/holding_test.go
new file mode 100644
index 000000000..f5143bb8c
--- /dev/null
+++ b/internal/e2e/selfserve/holding_test.go
@@ -0,0 +1,800 @@
+//go:build e2e
+
+package selfserve
+
+import (
+ "net/http"
+ "net/http/httptest"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/console"
+ "github.com/carlosframework/platform/internal/domains"
+ "github.com/carlosframework/platform/internal/edge"
+ "github.com/carlosframework/platform/internal/host"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// The holding page end to end (self-serve moves plan 3 Task 7, design
+// §6.4 and §11's tests): the two real boxes, their real edges and a third
+// edge (holding_world_test.go), through the console's real job.
+
+const (
+ src = host.E2ESourceBox
+ tgt = host.E2ETargetBox
+)
+
+// held is whether box's row for the instance is held by a move.
+func (w *holdWorld) held(box string) bool {
+ rt := w.rowOf(box, host.E2EHost)
+ return rt != nil && registry.IsHold(rt.Status)
+}
+
+// pair is box's row's (incarnation, serving_seq), or false with no row.
+func (w *holdWorld) pair(box string) (uint64, uint64, bool) {
+ rt := w.rowOf(box, host.E2EHost)
+ if rt == nil {
+ return 0, 0, false
+ }
+ return rt.Incarnation, rt.ServingSeq, true
+}
+
+// requireCandidate fails unless be's map admits, for name, a candidate of
+// kind from box at the pair, live (or not).
+func (w *holdWorld) requireCandidate(what string, be *boxEdge, name, kind, box string, inc, seq uint64, live bool) {
+ w.t.Helper()
+ cs := w.candidates(be, name)
+ for _, c := range cs {
+ if c.Type == kind && c.Box == box && c.Incarnation == inc && c.Seq == seq && c.Live == live {
+ return
+ }
+ }
+ w.t.Fatalf("%s: %s's candidates for %s are %s; want %s %s (%d,%d) live=%v", what, be.label, name, describeCandidates(cs), box, kind, inc, seq, live)
+}
+
+// askNoHop is ask, failing when the answer came over a backhaul to box.
+func (w *holdWorld) askNoHop(be *boxEdge, name string, box *boxEdge) answer {
+ w.t.Helper()
+ before := box.hops.Load()
+ a := w.ask(be, name)
+ if n := box.hops.Load() - before; n != 0 {
+ w.t.Fatalf("%s's answer for %s (%s) was backhauled to %s", be.label, name, a, box.label)
+ }
+ return a
+}
+
+// TestHoldingPageOnAThirdEdgeAndACustomDomain: from the start (the source's
+// row held) until the target's opened publication, an edge on an
+// unrelated box answers the instance's host and its explicitly attached
+// custom domain with the holding page (503, its headers and a recorded
+// sentence naming the account), and so does the source box's own edge for
+// its held row and its alias; once the target's ready publication has
+// reached it, the third edge backhauls both to the target, before the
+// health phase passes (two probe-mode passes of the target itself); an app
+// that then fails on the target answers as the app does through the rest
+// of the move and after it, and no hold comes back.
+func TestHoldingPageOnAThirdEdgeAndACustomDomain(t *testing.T) {
+ w := newHoldWorld(t)
+ w.attach()
+ if a := w.aliasRow(src, customDomain); a == nil || !a.Explicit || a.DNSStatus != "pointing" {
+ t.Fatalf("the custom domain's row on the source: %+v", a)
+ }
+ names := []string{host.E2EHost, customDomain}
+ for _, name := range names {
+ w.requireApp("before the move, the third edge's "+name, w.ask(w.third, name), src, http.StatusOK)
+ }
+ row := w.toCut()
+ var before, holding, opened, failing int
+ copySeen := map[string]bool{}
+ w.drive(300, "done", row.ID, func(r *console.SelfServeMove) {
+ started := holding > 0 || w.held(src)
+ targetOpen := w.published(tgt, host.E2EHost).ready
+ switch {
+ case targetOpen:
+ if opened == 0 && (r.HealthConsecutiveOK >= 2 || jobPhaseOrder[r.Phase] > jobPhaseOrder["health"]) {
+ t.Fatalf("the third edge first backhauled to the target with the row %s", describe(r))
+ }
+ if !w.published(tgt, customDomain).ready {
+ t.Fatalf("the target published the host without its custom domain")
+ }
+ opened++
+ code := http.StatusOK
+ if failing > 0 {
+ code = http.StatusInternalServerError
+ failing++
+ }
+ for _, name := range names {
+ w.requireApp(r.Phase+": the third edge's "+name, w.ask(w.third, name), tgt, code)
+ w.requireApp(r.Phase+": the source edge's "+name, w.ask(w.src, name), tgt, code)
+ }
+ // The app fails once the health phase has passed.
+ if failing == 0 && jobPhaseOrder[r.Phase] > jobPhaseOrder["health"] {
+ w.tgt.app.fail(http.StatusInternalServerError)
+ failing = 1
+ }
+ case started:
+ holding++
+ for _, name := range names {
+ copySeen[w.requirePage(r.Phase+": the third edge's "+name, w.ask(w.third, name))] = true
+ if w.held(src) {
+ w.requirePage(r.Phase+": the source edge's "+name, w.ask(w.src, name))
+ }
+ }
+ default:
+ before++
+ for _, name := range names {
+ w.requireApp(r.Phase+": the third edge's "+name, w.ask(w.third, name), src, http.StatusOK)
+ }
+ }
+ }, moveFinished)
+ // Done with no warning: the target's own edge passed both probe-mode
+ // probes (holdWorld.probeTarget) before its app failed.
+ if r := w.row(row.ID); r.Outcome != "done" || r.Warning != "" || r.HealthConsecutiveOK < 2 || w.probes < 2 {
+ t.Fatalf("finished %s, warning %q, %d probes", describe(r), r.Warning, w.probes)
+ }
+ // After the move: the target's failing app, no hold anywhere.
+ w.pointAliases()
+ w.boxes()
+ w.converge()
+ for _, name := range names {
+ w.requireApp("after the move, the third edge's "+name, w.ask(w.third, name), tgt, http.StatusInternalServerError)
+ for _, c := range w.candidates(w.third, name) {
+ if c.Type == "holding" {
+ t.Fatalf("a hold is back for %s: %s", name, describeCandidates(w.candidates(w.third, name)))
+ }
+ }
+ }
+ if before == 0 || holding == 0 || opened == 0 || failing < 2 {
+ t.Fatalf("rounds: %d before the start, %d holding, %d open (%d failing)", before, holding, opened, failing)
+ }
+ if !copySeen["MV-O259"] {
+ t.Fatalf("the page never named the account with an estimate: %v", copySeen)
+ }
+}
+
+// abortAfterLanding cuts a fresh move, runs it to landed, takes the target
+// box down (its last edges file, carrying the landing row's holding
+// entry, stays in the store) and lets the open's deadline abort the move;
+// it returns the row once the abort is asked for.
+func (w *holdWorld) abortAfterLanding() *console.SelfServeMove {
+ w.t.Helper()
+ r := w.cutTo("landed")
+ w.converge()
+ w.requireCandidate("landed", w.third, host.E2EHost, "holding", tgt, 1, 1, true)
+ w.e.Box(tgt).Down()
+ w.failKinds["open"] = true
+ w.tick()
+ w.pastDeadline(w.row(r.ID))
+ w.consoleUntil(4, "the abort", func() bool { return !w.row(r.ID).AbortRequestedAt.IsZero() })
+ return w.row(r.ID)
+}
+
+// TestAbortAfterLandingThenSourceReconstructionWithRetainedTargetHold
+// (design §11): the move holds the source at (1, 1), flips (the target's
+// landing row publishes its hold at (1, 1)), and aborts with the target
+// down, its holding publication left in the store; the source reopens at
+// (1, 2) and the abort finishes with move_seq 3. After the finish the
+// source's row is deleted; reconcile mints it again at (1, 3) from the
+// record, and the edges file publishes that; the third edge still holds
+// the target's holding publication at (1, 1), live, and the reconstructed
+// route wins the request.
+func TestAbortAfterLandingThenSourceReconstructionWithRetainedTargetHold(t *testing.T) {
+ w := newHoldWorld(t)
+ if inc, seq, _ := w.pair(src); inc != 1 || seq != 0 {
+ t.Fatalf("the source's row before the move: (%d,%d)", inc, seq)
+ }
+ r := w.abortAfterLanding()
+ if inc, seq, _ := w.pair(src); !w.held(src) || inc != 1 || seq != 1 {
+ t.Fatalf("the source's row at the abort: (%d,%d) held %v", inc, seq, w.held(src))
+ }
+ reopened := false
+ r = w.drive(300, "finished", r.ID, func(*console.SelfServeMove) {
+ if w.held(src) {
+ w.requirePage("the abort, the source still held", w.ask(w.third, host.E2EHost))
+ return
+ }
+ inc, seq, ok := w.pair(src)
+ if !ok {
+ t.Fatal("the source has no row after the abort")
+ }
+ if !reopened && (inc != 1 || seq != 2) {
+ t.Fatalf("the source reopened at (%d,%d), want (1,2)", inc, seq)
+ }
+ reopened = true
+ if p := w.published(src, host.E2EHost); p.ready {
+ w.requireCandidate("reopened", w.third, host.E2EHost, "holding", tgt, 1, 1, true)
+ w.requireApp("reopened at (1, 2)", w.ask(w.third, host.E2EHost), src, http.StatusOK)
+ }
+ }, moveFinished)
+ if r.Outcome != "aborted" || !reopened {
+ t.Fatalf("finished %s, reopened %v", describe(r), reopened)
+ }
+ if rec := w.record(); rec.Move != nil || rec.MoveSeq != 3 {
+ t.Fatalf("the record after the abort: move %+v move_seq %d", rec.Move, rec.MoveSeq)
+ }
+
+ // The reconstruction: the source's row is gone (a rebuilt registry);
+ // reconcile mints it from the record at the record's move_seq.
+ if err := w.e.Box(src).Registry().Delete(host.E2EHost); err != nil {
+ t.Fatal(err)
+ }
+ w.boxes()
+ w.converge()
+ if inc, seq, ok := w.pair(src); !ok || inc != 1 || seq != 3 || w.held(src) {
+ t.Fatalf("the reconstructed row: (%d,%d) %v", inc, seq, ok)
+ }
+ if p := w.published(src, host.E2EHost); !p.ready || p.inc != 1 || p.seq != 3 {
+ t.Fatalf("the source's publication of the reconstructed row: %+v", p)
+ }
+ if p := w.published(tgt, host.E2EHost); !p.held || p.holdInc != 1 || p.holdSeq != 1 {
+ t.Fatalf("the target's retained publication: %+v", p)
+ }
+ w.requireCandidate("reconstructed", w.third, host.E2EHost, "holding", tgt, 1, 1, true)
+ w.requireCandidate("reconstructed", w.third, host.E2EHost, "ready", src, 1, 3, true)
+ w.requireApp("the reconstructed route", w.ask(w.third, host.E2EHost), src, http.StatusOK)
+ w.requireApp("the source's own edge", w.ask(w.src, host.E2EHost), src, http.StatusOK)
+}
+
+// TestReadyRouteWinsOverStaleHoldAndHoldOverStaleRoute (design §11): the
+// third edge keeps the source's ready publication at (1, 0) it read before
+// the move (the source's file then unreadable to it), and the target's
+// landing hold at (1, 1) wins over that stale route: the page, answered by
+// the third edge with no dial to the source. Then it keeps the source's
+// holding publication at (1, 1) it read during the move, and the target's
+// ready route at (1, 2) wins over that stale hold through the open and
+// after the move, when the source publishes no hold at all.
+func TestReadyRouteWinsOverStaleHoldAndHoldOverStaleRoute(t *testing.T) {
+ w := newHoldWorld(t)
+ w.requireCandidate("before the move", w.third, host.E2EHost, "ready", src, 1, 0, true)
+ w.third.store.retain(src, true)
+ row := w.toCut()
+ w.drive(300, "the target's landing hold", row.ID, func(r *console.SelfServeMove) {
+ if !w.published(tgt, host.E2EHost).held {
+ // Only the stale route: the third edge dials the source,
+ // whose own edge answers (its app, or its own hold's page).
+ if a := w.ask(w.third, host.E2EHost); w.held(src) {
+ w.requirePage(r.Phase+": backhauled to the held source", a)
+ } else {
+ w.requireApp(r.Phase+": the stale route", a, src, http.StatusOK)
+ }
+ }
+ }, func(*console.SelfServeMove) bool {
+ return w.published(tgt, host.E2EHost).held && w.thirdSees(tgt)
+ })
+ w.requireCandidate("the hold over a stale route", w.third, host.E2EHost, "ready", src, 1, 0, true)
+ w.requireCandidate("the hold over a stale route", w.third, host.E2EHost, "holding", tgt, 1, 1, true)
+ w.requirePage("the hold over a stale route", w.askNoHop(w.third, host.E2EHost, w.src))
+
+ // The source's file read once more (its hold at (1, 1)), then kept.
+ w.third.store.retain(src, false)
+ if err := w.pollThird(); err != nil {
+ t.Fatal(err)
+ }
+ w.requireCandidate("the source's hold read", w.third, host.E2EHost, "holding", src, 1, 1, true)
+ w.third.store.retain(src, true)
+ opened := 0
+ r := w.drive(300, "done", row.ID, func(r *console.SelfServeMove) {
+ if !w.published(tgt, host.E2EHost).ready {
+ w.requirePage(r.Phase+": before the open", w.ask(w.third, host.E2EHost))
+ return
+ }
+ opened++
+ w.requireCandidate(r.Phase+": the stale hold", w.third, host.E2EHost, "holding", src, 1, 1, true)
+ w.requireApp(r.Phase+": the ready route over a stale hold", w.ask(w.third, host.E2EHost), tgt, http.StatusOK)
+ }, moveFinished)
+ if r.Outcome != "done" || opened == 0 {
+ t.Fatalf("finished %s after %d open rounds", describe(r), opened)
+ }
+ // After the move the source publishes no hold; the third edge still
+ // keeps the one it read, and the ready route still wins.
+ w.boxes()
+ w.converge()
+ if p := w.published(src, host.E2EHost); p.held || p.ready {
+ t.Fatalf("the source still publishes the host: %+v", p)
+ }
+ w.requireCandidate("after the move", w.third, host.E2EHost, "holding", src, 1, 1, true)
+ w.requireApp("after the move", w.ask(w.third, host.E2EHost), tgt, http.StatusOK)
+}
+
+// thirdSees is whether the third edge's map admits a candidate of box for
+// the instance's host.
+func (w *holdWorld) thirdSees(box string) bool {
+ for _, c := range w.candidates(w.third, host.E2EHost) {
+ if c.Box == box {
+ return true
+ }
+ }
+ return false
+}
+
+// TestFirstMoveWithPairlessRetainedPublication (design §6.4, §11): the
+// source's publication from before lifetimes (no pair: (0, 0)) is what
+// the third edge keeps for the source through the first move. Until the
+// target publishes, it dials the source (its app, then its own hold's
+// page); the target's landing hold at (1, 1) beats the pair-less route
+// (the page, no dial); the open's (1, 2) beats both.
+func TestFirstMoveWithPairlessRetainedPublication(t *testing.T) {
+ w := newHoldWorld(t)
+ if err := w.e.Box(src).PublishWithoutPairs(w.ctx); err != nil {
+ t.Fatal(err)
+ }
+ if err := w.pollThird(); err != nil {
+ t.Fatal(err)
+ }
+ if p := w.published(src, host.E2EHost); !p.ready || p.inc != 0 || p.seq != 0 {
+ t.Fatalf("the pair-less publication: %+v", p)
+ }
+ w.requireCandidate("the pair-less route", w.third, host.E2EHost, "ready", src, 0, 0, true)
+ w.third.store.retain(src, true)
+ row := w.toCut()
+ var dialled, held, opened int
+ r := w.drive(300, "done", row.ID, func(r *console.SelfServeMove) {
+ w.requireCandidate(r.Phase+": retained", w.third, host.E2EHost, "ready", src, 0, 0, true)
+ tp := w.published(tgt, host.E2EHost)
+ switch {
+ case tp.ready:
+ opened++
+ w.requireApp(r.Phase+": the open's route", w.askNoHop(w.third, host.E2EHost, w.src), tgt, http.StatusOK)
+ case tp.held:
+ held++
+ w.requirePage(r.Phase+": the landing hold over the pair-less route", w.askNoHop(w.third, host.E2EHost, w.src))
+ default:
+ dialled++
+ before := w.src.hops.Load()
+ a := w.ask(w.third, host.E2EHost)
+ if w.src.hops.Load() == before {
+ t.Fatalf("%s: the pair-less route was not dialled: %s", r.Phase, a)
+ }
+ if w.held(src) {
+ w.requirePage(r.Phase+": the source's own hold, backhauled", a)
+ } else {
+ w.requireApp(r.Phase+": the pair-less route", a, src, http.StatusOK)
+ }
+ }
+ }, moveFinished)
+ if r.Outcome != "done" || dialled == 0 || held == 0 || opened == 0 {
+ t.Fatalf("finished %s; rounds dialled %d, held %d, open %d", describe(r), dialled, held, opened)
+ }
+}
+
+// TestRecreatedHostOutranksOldHoldingAndReadyPublications (design §6.4,
+// §11): a first move takes the instance to the target; the third edge
+// keeps the target's landing hold of lifetime 1 at (1, 1). The host is
+// deleted and created again (lifetime 2) at home, with the target box
+// down, so its last publication (lifetime 1's ready route at (1, 2)) stays
+// in the store; the source mints lifetime 2's first row at (2, 0). It
+// outranks the retained hold, and then the retained ready route once the
+// third edge reads the target's file again.
+func TestRecreatedHostOutranksOldHoldingAndReadyPublications(t *testing.T) {
+ w := newHoldWorld(t)
+ row := w.toCut()
+ w.drive(300, "the target's landing hold", row.ID, nil, func(*console.SelfServeMove) bool { return w.published(tgt, host.E2EHost).held })
+ w.converge()
+ w.requireCandidate("landing", w.third, host.E2EHost, "holding", tgt, 1, 1, true)
+ w.third.store.retain(tgt, true)
+ r := w.drive(300, "done", row.ID, nil, moveFinished)
+ if r.Outcome != "done" {
+ t.Fatalf("finished %s", describe(r))
+ }
+ if inc, seq, _ := w.pair(tgt); inc != 1 || seq != 2 {
+ t.Fatalf("the target's row after the move: (%d,%d)", inc, seq)
+ }
+ w.requirePage("the old lifetime's hold, retained", w.ask(w.third, host.E2EHost))
+
+ w.e.Box(tgt).Down()
+ w.recreate(host.E2EHost, 2)
+ w.boxes()
+ w.converge()
+ if inc, seq, ok := w.pair(src); !ok || inc != 2 || seq != 0 || w.held(src) {
+ t.Fatalf("lifetime 2's first row: (%d,%d) %v", inc, seq, ok)
+ }
+ if p := w.published(src, host.E2EHost); !p.ready || p.inc != 2 || p.seq != 0 {
+ t.Fatalf("lifetime 2's publication: %+v", p)
+ }
+ w.requireCandidate("recreated", w.third, host.E2EHost, "holding", tgt, 1, 1, true)
+ w.requireApp("lifetime 2 over lifetime 1's hold", w.ask(w.third, host.E2EHost), src, http.StatusOK)
+
+ w.third.store.retain(tgt, false)
+ if err := w.pollThird(); err != nil {
+ t.Fatal(err)
+ }
+ if p := w.published(tgt, host.E2EHost); !p.ready || p.inc != 1 || p.seq != 2 {
+ t.Fatalf("the down target's last publication: %+v", p)
+ }
+ w.requireCandidate("recreated", w.third, host.E2EHost, "ready", tgt, 1, 2, true)
+ w.requireApp("lifetime 2 over lifetime 1's ready route", w.ask(w.third, host.E2EHost), src, http.StatusOK)
+ w.requireApp("the source's own edge", w.ask(w.src, host.E2EHost), src, http.StatusOK)
+}
+
+// TestCurrentHoldBeatsPreviousLifetimesReadyRoute (design §6.4, §11): the
+// third edge keeps lifetime 1's ready route at (1, 0) from the source's
+// file; the host is deleted and created again (lifetime 2), and lifetime
+// 2's move holds it: once the target's landing hold at (2, 1) reaches the
+// third edge, that hold beats the previous lifetime's route (the page, no
+// dial), and the open's (2, 2) serves.
+func TestCurrentHoldBeatsPreviousLifetimesReadyRoute(t *testing.T) {
+ w := newHoldWorld(t)
+ w.requireCandidate("lifetime 1", w.third, host.E2EHost, "ready", src, 1, 0, true)
+ w.third.store.retain(src, true)
+ w.recreate(host.E2EHost, 2)
+ w.boxes()
+ w.converge()
+ if inc, seq, _ := w.pair(src); inc != 2 || seq != 0 {
+ t.Fatalf("lifetime 2's first row: (%d,%d)", inc, seq)
+ }
+ row := w.toCut()
+ if row.Incarnation != 2 {
+ t.Fatalf("the move is of lifetime %d", row.Incarnation)
+ }
+ var held, opened int
+ r := w.drive(300, "done", row.ID, func(r *console.SelfServeMove) {
+ w.requireCandidate(r.Phase+": retained", w.third, host.E2EHost, "ready", src, 1, 0, true)
+ switch tp := w.published(tgt, host.E2EHost); {
+ case tp.ready:
+ opened++
+ w.requireApp(r.Phase+": the open", w.askNoHop(w.third, host.E2EHost, w.src), tgt, http.StatusOK)
+ case tp.held:
+ if tp.holdInc != 2 || tp.holdSeq != 1 {
+ t.Fatalf("the target's hold: %+v", tp)
+ }
+ held++
+ w.requirePage(r.Phase+": the current hold over lifetime 1's route", w.askNoHop(w.third, host.E2EHost, w.src))
+ w.requireCandidate(r.Phase+": the current hold", w.third, host.E2EHost, "holding", tgt, 2, 1, true)
+ }
+ }, moveFinished)
+ if r.Outcome != "done" || held == 0 || opened == 0 {
+ t.Fatalf("finished %s; rounds held %d, open %d", describe(r), held, opened)
+ }
+}
+
+// restartedThird is a third edge started now (a restarted edge process: no
+// map of its own until its first poll), polled once.
+func (w *holdWorld) restartedThird() (*boxEdge, error) {
+ w.t.Helper()
+ be := w.newThirdEdge(w.t)
+ return be, w.pollThirdEdge(be)
+}
+
+func (w *holdWorld) pollThirdEdge(be *boxEdge) error {
+ if err := be.e.Reload(); err != nil {
+ return err
+ }
+ return edge.E2ERefreshHomes(w.ctx, be.e)
+}
+
+// TestEdgeRestartedIntoMixedPublications (design §11): at every round of
+// a move to done (the source's ready route; its hold; both boxes' holds;
+// the target's ready route beside the source's hold; the target alone) and
+// of a move aborted after landing with the target down (the target's
+// landing hold left beside the source's hold, then beside the source
+// reopened), an edge restarted into the store answers the host and the
+// custom domain exactly as the edge that has run throughout. A restarted
+// edge whose first poll cannot list the store serves no other box's host
+// until a poll lands.
+func TestEdgeRestartedIntoMixedPublications(t *testing.T) {
+ compare := func(w *holdWorld, phase string, seen map[string]bool) {
+ w.t.Helper()
+ fresh, err := w.restartedThird()
+ if err != nil {
+ w.t.Fatalf("%s: the restarted edge's first poll: %v", phase, err)
+ }
+ for _, name := range []string{host.E2EHost, customDomain} {
+ want, got := w.ask(w.third, name), w.ask(fresh, name)
+ if want.String() != got.String() {
+ w.t.Fatalf("%s: %s: the restarted edge answered %s, the running edge %s (candidates %s)", phase, name, got, want, describeCandidates(w.candidates(fresh, name)))
+ }
+ seen[want.kind] = true
+ }
+ }
+ t.Run("forward", func(t *testing.T) {
+ w := newHoldWorld(t)
+ w.attach()
+ row := w.toCut()
+ seen := map[string]bool{}
+ r := w.drive(300, "done", row.ID, func(r *console.SelfServeMove) { compare(w, r.Phase, seen) }, moveFinished)
+ if r.Outcome != "done" || !seen["page"] || !seen["app:"+src] || !seen["app:"+tgt] {
+ t.Fatalf("finished %s; answers seen %v", describe(r), seen)
+ }
+ })
+ t.Run("aborted after landing", func(t *testing.T) {
+ w := newHoldWorld(t)
+ w.attach()
+ r := w.abortAfterLanding()
+ seen := map[string]bool{}
+ compare(w, "aborting", seen)
+ r = w.drive(300, "finished", r.ID, func(r *console.SelfServeMove) { compare(w, r.Phase, seen) }, moveFinished)
+ if r.Outcome != "aborted" || !seen["page"] || !seen["app:"+src] {
+ t.Fatalf("finished %s; answers seen %v", describe(r), seen)
+ }
+ w.requireCandidate("after the abort", w.third, host.E2EHost, "holding", tgt, 1, 1, true)
+ })
+ t.Run("a first poll that cannot list", func(t *testing.T) {
+ w := newHoldWorld(t)
+ row := w.toCut()
+ w.drive(300, "the source held", row.ID, nil, func(*console.SelfServeMove) bool { return w.published(src, host.E2EHost).held })
+ w.converge()
+ fresh := w.newThirdEdge(t)
+ fresh.store.failList(true)
+ if err := w.pollThirdEdge(fresh); err == nil {
+ t.Fatal("the restarted edge's poll listed a store that fails")
+ }
+ if a := w.ask(fresh, host.E2EHost); a.kind != "unknown" {
+ t.Fatalf("the restarted edge with no map answered %s", a)
+ }
+ w.requirePage("the running edge", w.ask(w.third, host.E2EHost))
+ fresh.store.failList(false)
+ if err := w.pollThirdEdge(fresh); err != nil {
+ t.Fatal(err)
+ }
+ w.requirePage("the restarted edge's first good poll", w.ask(fresh, host.E2EHost))
+ })
+}
+
+// TestExpiredHoldingEntryFallsBackIncludingAfterAWholeRefreshFails
+// (design §6.4, §11): the source's holding entry, fetched at T by the
+// third edge and by a second edge that serves a catch-all for the host's
+// parent locally, is believed until T + 5 m: with every refresh after T
+// failing (the listing fails), both answer the page at 4 m 59 s (the
+// estimate passed: the taking-longer sentence), and at 5 m 0 s and 5 m 1 s
+// the fallback: no answer for the host on the third edge (the plain unknown
+// host), the local catch-all's app on the other, and the custom domain
+// unknown on both. A good poll fetches the entry afresh and the page
+// returns. Then every object read failing (the other way a whole refresh
+// fails) ages it out the same way.
+//
+// The clocks: the edges' map clock (edgeClock) is the only one moved here,
+// with the boxes and the console still, so the entry's age is the advance.
+func TestExpiredHoldingEntryFallsBackIncludingAfterAWholeRefreshFails(t *testing.T) {
+ w := newHoldWorld(t)
+ w.attach()
+ wild := w.catchAllThird(t)
+ row := w.toCut()
+ // Held, with the offline time on the entry (the first holding entry
+ // can precede it by a pass) so the page counts down from it.
+ w.drive(300, "the source held", row.ID, nil, func(*console.SelfServeMove) bool {
+ p := w.published(src, host.E2EHost)
+ return p.held && p.offlineAt != 0 && w.published(src, customDomain).held
+ })
+ w.converge()
+ if err := w.pollThirdEdge(wild); err != nil {
+ t.Fatal(err)
+ }
+ edges := []*boxEdge{w.third, wild}
+ for _, be := range edges {
+ w.requirePage(be.label+" at the fetch", w.ask(be, host.E2EHost))
+ w.requirePage(be.label+" at the fetch, the custom domain", w.ask(be, customDomain))
+ }
+ poll := func(what string, wantErr bool) {
+ t.Helper()
+ for _, be := range edges {
+ err := w.pollThirdEdge(be)
+ if (err != nil) != wantErr {
+ t.Fatalf("%s: %s's poll: %v", what, be.label, err)
+ }
+ }
+ }
+ expire := func(shape string) {
+ t.Helper()
+ w.clock.Advance(5*time.Minute - time.Second)
+ poll(shape+" at 4 m 59 s", true)
+ for _, be := range edges {
+ if id := w.requirePage(shape+" at 4 m 59 s: "+be.label, w.ask(be, host.E2EHost)); id != "MV-O262" {
+ t.Fatalf("%s at 4 m 59 s: the page says %s, want the estimate passed", shape, id)
+ }
+ w.requirePage(shape+" at 4 m 59 s: "+be.label+", the custom domain", w.ask(be, customDomain))
+ }
+ for _, step := range []string{"5 m 0 s", "5 m 1 s"} {
+ w.clock.Advance(time.Second)
+ poll(shape+" at "+step, true)
+ if a := w.ask(w.third, host.E2EHost); a.kind != "unknown" {
+ t.Fatalf("%s at %s: the third edge answered %s, want the plain unknown host", shape, step, a)
+ }
+ if a := w.ask(wild, host.E2EHost); a.kind != "app:"+thirdBox || a.code != http.StatusOK {
+ t.Fatalf("%s at %s: the catch-all edge answered %s, want its catch-all's app", shape, step, a)
+ }
+ for _, be := range edges {
+ if a := w.ask(be, customDomain); a.kind != "unknown" {
+ t.Fatalf("%s at %s: %s answered the custom domain %s", shape, step, be.label, a)
+ }
+ w.requireCandidate(shape+" at "+step, be, host.E2EHost, "holding", src, 1, 1, false)
+ }
+ }
+ }
+
+ for _, be := range edges {
+ be.store.failList(true)
+ }
+ expire("the listing failing")
+ for _, be := range edges {
+ be.store.failList(false)
+ }
+ poll("a good poll", false)
+ for _, be := range edges {
+ w.requirePage("a good poll: "+be.label, w.ask(be, host.E2EHost))
+ }
+
+ for _, be := range edges {
+ be.store.retain(src, true)
+ be.store.retain(tgt, true)
+ }
+ expire("every read failing")
+}
+
+// catchAllThird is a second unrelated edge, labelled as the third, whose
+// own registry serves a catch-all for the instance's parent domain
+// (*.example.com, an alias of a route of its own) from an app of its own.
+func (w *holdWorld) catchAllThird(t *testing.T) *boxEdge {
+ t.Helper()
+ reg, err := registry.Open(filepath.Join(t.TempDir(), "registry.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { reg.Close() })
+ app := &testApp{box: thirdBox}
+ srv := httptest.NewServer(app) // 127.0.0.1
+ t.Cleanup(srv.Close)
+ if err := reg.Upsert(registry.Route{Host: "www.example.com", Kind: "service", Addr: srv.Listener.Addr().String(), Status: "ready"}); err != nil {
+ t.Fatal(err)
+ }
+ if err := reg.UpsertAlias(registry.RouteAlias{Alias: "*.example.com", Host: "www.example.com", CreatedAt: time.Now().Unix()}); err != nil {
+ t.Fatal(err)
+ }
+ be := &boxEdge{label: thirdBox + " (catch-all)", app: app, store: &holdStore{Store: w.raw}}
+ be.e = w.armEdge(t, reg, thirdBox, thirdBox, nil, be.store)
+ return be
+}
+
+// otherInstance creates a second app of the account, "other" (domain
+// other.test), with an instance shop.other.test homed on the target box:
+// its manifest, a promoted release, its incarnation slot and its record.
+func (w *holdWorld) otherInstance() string {
+ w.t.Helper()
+ const other, otherHost = "other", "shop.other.test"
+ if err := provision.PutManifest(w.ctx, w.raw, w.acct, other, provision.Manifest{Schema: 1, App: other, Domain: "other.test"}); err != nil {
+ w.t.Fatal(err)
+ }
+ m := release.Manifest{Schema: 1, App: other, Version: "v1", Kind: "binary", Artifacts: map[string]release.Artifact{"linux-amd64": {File: other}}}
+ if err := release.Ship(w.ctx, w.raw, w.acct, m, map[string][]byte{"linux-amd64": []byte("bin")}); err != nil {
+ w.t.Fatal(err)
+ }
+ if err := release.Promote(w.ctx, w.raw, w.e.ReleaseKey(), w.acct, other, "stable", "v1"); err != nil {
+ w.t.Fatal(err)
+ }
+ if _, err := provision.WriteHostIncarnation(w.ctx, w.raw, otherHost, provision.HostIncarnation{Account: w.acct, App: other,
+ Incarnation: 1, State: provision.SlotLive, Intent: "create-e2e"}, ""); err != nil {
+ w.t.Fatal(err)
+ }
+ if err := provision.CreateRecord(w.ctx, w.raw, provision.Record{Schema: 1, Host: otherHost, Slug: "shop", App: other, Account: w.acct,
+ Channel: "stable", Region: host.E2ETargetRegion, CreatedAt: time.Now().UTC(), Incarnation: 1}); err != nil {
+ w.t.Fatal(err)
+ }
+ return otherHost
+}
+
+// catchAll claims the instance catch-all *.custom.test for app (each of
+// app's instances is named <slug>.custom.test unless an exact claim takes
+// the name).
+func (w *holdWorld) catchAll(app string) {
+ w.t.Helper()
+ if _, _, err := domains.AttachClaim(w.ctx, w.raw, domains.Claim{Host: "*." + customParent, Account: w.acct, App: app, Instances: true}); err != nil {
+ w.t.Fatal(err)
+ }
+}
+
+// detach removes the custom domain's exact claim, as the detach door does.
+func (w *holdWorld) detach() {
+ w.t.Helper()
+ if err := w.raw.Delete(w.ctx, domains.ClaimKey(customDomain)); err != nil {
+ w.t.Fatal(err)
+ }
+}
+
+// TestAliasWitnessesEndToEnd (design §6.4, §11): the two alias
+// conversions through the real alias pass, with the move holding the
+// instance.
+//
+// Explicit to derived, a different target: the custom domain is attached
+// to the moving instance and the third edge keeps the source's holding
+// entry that lists it; the domain is detached, and the account's instance
+// catch-all for another app names it after that app's instance on the
+// target box (target_host shop.other.test): that ready route wins over the
+// retained hold, fresh and as a last-good publication, and the other
+// instance answers.
+//
+// Explicit to derived, the same target (Task 1's provenance rule): the
+// domain is detached and the catch-all of the moving instance's own app
+// names it after the moving instance: the row turns derived, the next
+// holding entry leaves it out, and the domain answers as a derived alias
+// of a held row does (the plain unknown host on the third edge and on the
+// source's own edge) while the host keeps the page; after the open the
+// target's derived alias serves it.
+func TestAliasWitnessesEndToEnd(t *testing.T) {
+ t.Run("explicit to derived, another target", func(t *testing.T) {
+ w := newHoldWorld(t)
+ otherHost := w.otherInstance()
+ w.attach()
+ w.catchAll("other")
+ row := w.toCut()
+ w.drive(300, "the source holding the domain", row.ID, nil, func(*console.SelfServeMove) bool {
+ return w.published(src, customDomain).held
+ })
+ w.converge()
+ w.requirePage("held", w.ask(w.third, customDomain))
+ w.third.store.retain(src, true)
+ w.detach()
+ w.pointAliases()
+ w.boxes()
+ w.pointAliases()
+ w.boxes()
+ w.converge()
+ if a := w.aliasRow(tgt, customDomain); a == nil || a.Explicit || a.Host != otherHost {
+ t.Fatalf("the target's row for the domain: %+v", a)
+ }
+ if p := w.published(tgt, customDomain); !p.ready || p.targetHost != otherHost {
+ t.Fatalf("the target's publication of the domain: %+v", p)
+ }
+ w.requireCandidate("retained", w.third, customDomain, "holding", src, 1, 1, true)
+ a := w.ask(w.third, customDomain)
+ w.requireApp("the other target's fresh route", a, tgt, http.StatusOK)
+ if a.route != otherHost {
+ t.Fatalf("the domain was served for %q, want %q", a.route, otherHost)
+ }
+ w.third.store.retain(tgt, true)
+ w.boxes()
+ w.converge()
+ a = w.ask(w.third, customDomain)
+ w.requireApp("the other target's last-good route", a, tgt, http.StatusOK)
+ if a.route != otherHost {
+ t.Fatalf("the domain was served for %q, want %q", a.route, otherHost)
+ }
+ w.requirePage("the host", w.ask(w.third, host.E2EHost))
+ })
+ t.Run("explicit to derived, the same target", func(t *testing.T) {
+ w := newHoldWorld(t)
+ w.attach()
+ w.catchAll(host.E2EApp)
+ row := w.toCut()
+ w.drive(300, "the source holding the domain", row.ID, nil, func(*console.SelfServeMove) bool {
+ return w.published(src, customDomain).held
+ })
+ w.converge()
+ w.requirePage("held, the third edge", w.ask(w.third, customDomain))
+ w.requirePage("held, the source's edge", w.ask(w.src, customDomain))
+ w.detach()
+ w.boxes()
+ w.converge()
+ if a := w.aliasRow(src, customDomain); a == nil || a.Explicit || a.Host != host.E2EHost {
+ t.Fatalf("the source's row for the domain: %+v", a)
+ }
+ if p := w.published(src, customDomain); p.held || p.ready {
+ t.Fatalf("the source still publishes the domain: %+v", p)
+ }
+ if p := w.published(src, host.E2EHost); !p.held {
+ t.Fatalf("the source no longer holds the host: %+v", p)
+ }
+ for _, be := range []*boxEdge{w.third, w.src} {
+ if a := w.ask(be, customDomain); a.kind != "unknown" {
+ t.Fatalf("%s answered the derived domain of a held row %s", be.label, a)
+ }
+ w.requirePage(be.label+": the host", w.ask(be, host.E2EHost))
+ }
+ r := w.drive(300, "done", row.ID, nil, moveFinished)
+ if r.Outcome != "done" {
+ t.Fatalf("finished %s", describe(r))
+ }
+ w.pointAliases()
+ w.boxes()
+ w.converge()
+ if a := w.aliasRow(tgt, customDomain); a == nil || a.Explicit || a.Host != host.E2EHost {
+ t.Fatalf("the target's row for the domain: %+v", a)
+ }
+ a := w.ask(w.third, customDomain)
+ w.requireApp("after the move", a, tgt, http.StatusOK)
+ if a.route != host.E2EHost {
+ t.Fatalf("the domain was served for %q", a.route)
+ }
+ })
+}
diff --git a/internal/e2e/selfserve/holding_world_test.go b/internal/e2e/selfserve/holding_world_test.go
new file mode 100644
index 000000000..e69f4bb8b
--- /dev/null
+++ b/internal/e2e/selfserve/holding_world_test.go
@@ -0,0 +1,770 @@
+//go:build e2e
+
+package selfserve
+
+import (
+ "context"
+ "crypto/ecdsa"
+ "crypto/elliptic"
+ "crypto/rand"
+ "crypto/tls"
+ "crypto/x509"
+ "crypto/x509/pkix"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "math/big"
+ "net"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "regexp"
+ "runtime"
+ "strings"
+ "sync"
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/console"
+ "github.com/carlosframework/platform/internal/domains"
+ "github.com/carlosframework/platform/internal/edge"
+ "github.com/carlosframework/platform/internal/edgesfile"
+ "github.com/carlosframework/platform/internal/host"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// The holding page's world (self-serve moves plan 3 Task 7): the job's
+// world (two real boxes, a real console) with a real edge on each box and
+// a third edge that stands for an unrelated box's.
+//
+// Each box's edge is an edge.Edge over the box's own registry, subscribed
+// to the box's offline view (the bridge's SubscribeHolds, as the agent
+// subscribes its edge), with its home map armed over the deployment store
+// and the mesh key minted there, serving TLS on its own loopback address
+// (the address the box's edges file publishes) so another edge's backhaul
+// dials it through a real handshake against a test CA. The app behind each
+// box is a handler the edge's activator seam calls for the box's instance
+// row (the activator stands in for the instance's process).
+//
+// The third edge is an edge.Edge in the test process, with its own label
+// and box id and an empty registry of its own: it serves nothing locally
+// and sees the move only through the edges files the two boxes publish to
+// the store. A witness's request is handed to its ServeHTTP on the test's
+// goroutine; its backhaul is a real TLS dial to a box's listener. Its reads
+// go through holdStore, so a witness can make one box's file unreadable
+// (the edge keeps the parse it had: a retained publication) or fail a whole
+// refresh. It is in process and not a subprocess because a witness drives
+// every poll and moves the map's clock (the five-minute validity of a
+// holding entry is judged at it), and a subprocess's 30 s loop and wall
+// clock cannot be driven (the edge package's seams for this,
+// internal/edge/e2e_bridge.go, are compiled only under the e2e tag).
+//
+// The boxes' own edges and the third edge do not run their loops: each
+// round, after the boxes tick, every edge reloads its registry (the 2 s
+// watch) and polls the store once (the 30 s home map loop), so each
+// publication reaches every edge within the round it is written.
+//
+// The clocks. Every edge's home map reads edgeClock: the fetch time a poll
+// stamps on each file and the time each request is ranked and its page is
+// rendered at. It starts at the wall clock and moves only when a witness
+// advances it. The boxes' moves pass reads the bridge's Clock, which
+// follows the wall clock here, and the console ticks at the wall clock
+// plus its skew (jobWorld.now). The expiry witness advances edgeClock alone
+// between polls, with the boxes and the console still, so a hold's age is
+// exactly what the witness set.
+
+const (
+ // thirdBox is the third edge's label and box id: a box that serves no
+ // part of the move.
+ thirdBox = "eu-box"
+ // customDomain is the account's custom domain, attached to the moving
+ // instance by an exact claim (an explicit alias).
+ customDomain = "shop.custom.test"
+ // customParent is its parent, claimed by the account too (an instance
+ // catch-all under it needs the parent's claim).
+ customParent = "custom.test"
+ // The edges' loopback addresses: each box's edge listens on its own,
+ // on the one port every backhaul dials.
+ sourceIP = "127.0.0.2"
+ targetIP = "127.0.0.3"
+)
+
+// edgeClock is every edge's home-map clock.
+type edgeClock struct {
+ mu sync.Mutex
+ t time.Time
+}
+
+func (c *edgeClock) Now() time.Time {
+ c.mu.Lock()
+ defer c.mu.Unlock()
+ return c.t
+}
+
+func (c *edgeClock) Advance(d time.Duration) {
+ c.mu.Lock()
+ c.t = c.t.Add(d)
+ c.mu.Unlock()
+}
+
+// holdStore is one edge's view of the deployment store: Get of a key in
+// unreadable fails (the edge carries that file's last parse forward), and
+// with listFailing every List fails (a whole refresh fails).
+type holdStore struct {
+ release.Store
+ mu sync.Mutex
+ unreadable map[string]bool
+ listFailing bool
+}
+
+var errEdgeRead = errors.New("holdStore: the edge could not read the store")
+
+func (s *holdStore) Get(ctx context.Context, key string) ([]byte, string, error) {
+ s.mu.Lock()
+ bad := s.unreadable[key]
+ s.mu.Unlock()
+ if bad {
+ return nil, "", errEdgeRead
+ }
+ return s.Store.Get(ctx, key)
+}
+
+func (s *holdStore) List(ctx context.Context, prefix string) ([]string, error) {
+ s.mu.Lock()
+ bad := s.listFailing
+ s.mu.Unlock()
+ if bad {
+ return nil, errEdgeRead
+ }
+ return s.Store.List(ctx, prefix)
+}
+
+// retain makes box's edges file unreadable to this edge (on) or readable
+// again.
+func (s *holdStore) retain(box string, on bool) {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ if s.unreadable == nil {
+ s.unreadable = map[string]bool{}
+ }
+ s.unreadable[edgesKey(box)] = on
+}
+
+func (s *holdStore) failList(on bool) {
+ s.mu.Lock()
+ s.listFailing = on
+ s.mu.Unlock()
+}
+
+func edgesKey(box string) string { return "control/edges/" + box + ".json" }
+
+// testApp is the app behind one box's edge: it answers with its status and
+// a body naming the box whose app it is, the host asked for and the route
+// the edge served it from (routeHeader, set by appActivator).
+type testApp struct {
+ box string
+ status atomic.Int64
+ hits atomic.Int64
+}
+
+func (a *testApp) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+ a.hits.Add(1)
+ code := int(a.status.Load())
+ if code == 0 {
+ code = http.StatusOK
+ }
+ w.Header().Set("Content-Type", "text/plain")
+ w.WriteHeader(code)
+ fmt.Fprintf(w, "app on %s answered %s for %s", a.box, r.Host, r.Header.Get(routeHeader))
+}
+
+// routeHeader carries the served route's host from the activator seam to
+// the app.
+const routeHeader = "X-Test-Route"
+
+// fail makes the app answer code from now on (0: 200 again).
+func (a *testApp) fail(code int) { a.status.Store(int64(code)) }
+
+// appActivator is the edge's activator seam over the box's app: it serves
+// every hibernating instance row from the app, as the instance's process
+// would once woken.
+type appActivator struct{ app http.Handler }
+
+func (a appActivator) ServeHTTP(w http.ResponseWriter, r *http.Request, rt registry.Route, _ http.Handler) bool {
+ r.Header.Set(routeHeader, rt.Host)
+ a.app.ServeHTTP(w, r)
+ return true
+}
+
+func (appActivator) RetireExcept(map[string]bool) {}
+
+// testCA signs each edge's certificate for the name a handshake asks for.
+type testCA struct {
+ root *x509.Certificate
+ key *ecdsa.PrivateKey
+ pool *x509.CertPool
+ mu sync.Mutex
+ leaf map[string]*tls.Certificate
+}
+
+func newTestCA(t *testing.T) *testCA {
+ t.Helper()
+ key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ tmpl := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "CARLOS holding page test CA"},
+ NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(24 * time.Hour),
+ KeyUsage: x509.KeyUsageCertSign, BasicConstraintsValid: true, IsCA: true}
+ der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ root, err := x509.ParseCertificate(der)
+ if err != nil {
+ t.Fatal(err)
+ }
+ pool := x509.NewCertPool()
+ pool.AddCert(root)
+ return &testCA{root: root, key: key, pool: pool, leaf: map[string]*tls.Certificate{}}
+}
+
+func (ca *testCA) certFor(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
+ name := hello.ServerName
+ ca.mu.Lock()
+ defer ca.mu.Unlock()
+ if c := ca.leaf[name]; c != nil {
+ return c, nil
+ }
+ key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+ if err != nil {
+ return nil, err
+ }
+ serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 96))
+ if err != nil {
+ return nil, err
+ }
+ tmpl := &x509.Certificate{SerialNumber: serial, Subject: pkix.Name{CommonName: name}, DNSNames: []string{name},
+ NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(24 * time.Hour),
+ KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}}
+ der, err := x509.CreateCertificate(rand.Reader, tmpl, ca.root, &key.PublicKey, ca.key)
+ if err != nil {
+ return nil, err
+ }
+ c := &tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key}
+ ca.leaf[name] = c
+ return c, nil
+}
+
+// boxEdge is one edge: a box's (with its app and listener) or the third.
+type boxEdge struct {
+ label string
+ e *edge.Edge
+ store *holdStore
+ app *testApp // nil on the third edge
+ // hops counts the requests the box's listener received: another
+ // edge's backhaul dials.
+ hops atomic.Int64
+}
+
+// holdWorld is the job's world with the edges.
+type holdWorld struct {
+ *jobWorld
+ ca *testCA
+ clock *edgeClock
+ src, tgt *boxEdge
+ third *boxEdge
+ port string
+ // thirdReg is the third edge's own registry (empty unless a witness
+ // gives it a route).
+ thirdReg *registry.Registry
+}
+
+// newHoldWorld builds the job's world, the two boxes' edges and the third
+// edge, with the alias pass on in every tick.
+func newHoldWorld(t *testing.T) *holdWorld {
+ t.Helper()
+ if runtime.GOOS != "linux" {
+ t.Skip("the edges listen on 127.0.0.2 and 127.0.0.3, which only Linux routes by default")
+ }
+ w := &holdWorld{jobWorld: newJobWorld(t), ca: newTestCA(t), clock: &edgeClock{t: time.Now()}}
+ w.e.AliasPass(func(ctx context.Context) map[string]bool { return edgesfile.DeploymentRoutes(ctx, w.raw, time.Now()) })
+ if err := edge.MintMeshKey(w.ctx, w.raw); err != nil {
+ t.Fatal(err)
+ }
+ src, tgt := w.e.Box(host.E2ESourceBox), w.e.Box(host.E2ETargetBox)
+ lnS, lnT := w.listenPair(t)
+ w.src = w.boxEdge(t, src, lnS)
+ w.tgt = w.boxEdge(t, tgt, lnT)
+ t.Cleanup(edge.E2EBackhaulDial(w.port, w.ca.pool))
+ src.SetEIP(sourceIP)
+ tgt.SetEIP(targetIP)
+ w.third = w.newThirdEdge(t)
+ // The job's health probe is the console's: a probe-mode request to
+ // the target box's own listener, passing on a 2xx it says it served.
+ w.probe = w.probeTarget
+ w.boxes()
+ w.converge()
+ return w
+}
+
+// listenPair listens on sourceIP and targetIP on one port.
+func (w *holdWorld) listenPair(t *testing.T) (net.Listener, net.Listener) {
+ t.Helper()
+ for range 20 {
+ a, err := net.Listen("tcp", sourceIP+":0")
+ if err != nil {
+ t.Skipf("no second loopback address: %v", err)
+ }
+ _, port, _ := net.SplitHostPort(a.Addr().String())
+ b, err := net.Listen("tcp", targetIP+":"+port)
+ if err != nil {
+ a.Close()
+ continue
+ }
+ w.port = port
+ return a, b
+ }
+ t.Fatal("no port free on both loopback addresses")
+ return nil, nil
+}
+
+// boxEdge builds the box's edge over its registry, serving ln.
+func (w *holdWorld) boxEdge(t *testing.T, b host.BoxHandle, ln net.Listener) *boxEdge {
+ t.Helper()
+ app := &testApp{box: b.Label()}
+ be := &boxEdge{label: b.Label(), app: app, store: &holdStore{Store: w.raw}}
+ be.e = w.armEdge(t, b.Registry(), b.Label(), b.BoxID(), appActivator{app: app}, be.store)
+ b.SubscribeHolds(be.e.SetHoldView)
+ srv := httptest.NewUnstartedServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
+ be.hops.Add(1)
+ be.e.ServeHTTP(rw, r)
+ }))
+ srv.Listener.Close()
+ srv.Listener = ln
+ srv.TLS = &tls.Config{GetCertificate: w.ca.certFor, MinVersion: tls.VersionTLS12}
+ srv.StartTLS()
+ t.Cleanup(srv.Close)
+ return be
+}
+
+// armEdge is an edge as RunWithActivatorReady arms one: over reg, labelled
+// and identified, its home map over store at the world's edge clock.
+func (w *holdWorld) armEdge(t *testing.T, reg *registry.Registry, label, id string, act edge.Activator, store release.Store) *edge.Edge {
+ t.Helper()
+ e, err := edge.New(reg, act)
+ if err != nil {
+ t.Fatal(err)
+ }
+ edge.WithBoxLabel(label)(e)
+ edge.WithBoxID(id)(e)
+ e.EnableStatic(store, w.e.ReleasePub(), t.TempDir())
+ if err := edge.E2EArmHomes(e, w.clock.Now); err != nil {
+ t.Fatal(err)
+ }
+ return e
+}
+
+// newThirdEdge is a third edge over its own empty registry, reading the
+// store through a holdStore of its own, its map polled once.
+func (w *holdWorld) newThirdEdge(t *testing.T) *boxEdge {
+ t.Helper()
+ if w.thirdReg == nil {
+ reg, err := registry.Open(filepath.Join(t.TempDir(), "registry.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { reg.Close() })
+ w.thirdReg = reg
+ }
+ be := &boxEdge{label: thirdBox, store: &holdStore{Store: w.raw}}
+ be.e = w.armEdge(t, w.thirdReg, thirdBox, thirdBox, nil, be.store)
+ return be
+}
+
+// converge is what the edges' own loops would have done since the boxes'
+// last tick: each edge reloads its registry (the 2 s watch) and polls the
+// store (the 30 s home map loop). The third edge's poll error is returned;
+// a box edge's is fatal.
+func (w *holdWorld) converge() error {
+ w.t.Helper()
+ for _, be := range []*boxEdge{w.src, w.tgt} {
+ if err := be.e.Reload(); err != nil {
+ w.t.Fatalf("%s: reload: %v", be.label, err)
+ }
+ if err := edge.E2ERefreshHomes(w.ctx, be.e); err != nil {
+ w.t.Fatalf("%s: home map: %v", be.label, err)
+ }
+ }
+ return w.pollThird()
+}
+
+// pollThird is one poll of the third edge's home map.
+func (w *holdWorld) pollThird() error {
+ if err := w.third.e.Reload(); err != nil {
+ return err
+ }
+ return edge.E2ERefreshHomes(w.ctx, w.third.e)
+}
+
+// round is one round of the world: the DNS health stamp, the boxes tick,
+// the edges converge,
+// observe sees the world, and the console ticks; in health the console's
+// clock moves on 31 s first, as the job's probes are 30 s apart.
+func (w *holdWorld) round(id string, observe func(r *console.SelfServeMove)) {
+ w.t.Helper()
+ w.pointAliases()
+ w.boxes()
+ w.converge()
+ r := w.row(id)
+ if observe != nil {
+ observe(r)
+ }
+ if r.Phase == "health" && r.Outcome == "" {
+ w.advance(31 * time.Second)
+ }
+ w.tick()
+}
+
+// drive runs rounds until done holds (checked before each round).
+func (w *holdWorld) drive(max int, what, id string, observe func(r *console.SelfServeMove), done func(r *console.SelfServeMove) bool) *console.SelfServeMove {
+ w.t.Helper()
+ for i := 0; i < max; i++ {
+ if done(w.row(id)) {
+ return w.row(id)
+ }
+ w.round(id, observe)
+ }
+ r := w.row(id)
+ if !done(r) {
+ w.t.Fatalf("after %d rounds, still not %s: %s", max, what, describe(r))
+ }
+ return r
+}
+
+// moveFinished is a row with its outcome and nothing owed.
+func moveFinished(r *console.SelfServeMove) bool {
+ return r.Outcome != "" && !r.ReleasePending && r.EffectsDone
+}
+
+// probeTarget is the job's health probe, made the way the console makes
+// it: to the target box's own address with the host as SNI and Host, in
+// probe mode, no redirects followed; it passes on a 2xx the target says it
+// served itself.
+func (w *holdWorld) probeTarget() (bool, string) {
+ tgt := w.e.Box(host.E2ETargetBox)
+ client := &http.Client{
+ Timeout: 10 * time.Second,
+ Transport: &http.Transport{
+ TLSClientConfig: &tls.Config{RootCAs: w.ca.pool, MinVersion: tls.VersionTLS12},
+ DialContext: func(ctx context.Context, network, _ string) (net.Conn, error) {
+ return (&net.Dialer{}).DialContext(ctx, network, net.JoinHostPort(targetIP, w.port))
+ },
+ },
+ CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
+ }
+ defer client.CloseIdleConnections()
+ req, _ := http.NewRequest(http.MethodGet, "https://"+host.E2EHost+"/?carlos-probe=1", nil)
+ req.Header.Set("X-Carlos-Probe", "local")
+ req.Header.Set("Cache-Control", "no-cache")
+ resp, err := client.Do(req)
+ if err != nil {
+ return false, "the probe did not get an answer"
+ }
+ io.Copy(io.Discard, resp.Body)
+ resp.Body.Close()
+ switch {
+ case resp.StatusCode < 200 || resp.StatusCode > 299:
+ return false, fmt.Sprintf("the probe was answered %d", resp.StatusCode)
+ case resp.Header.Get("X-Carlos-Served-By") != tgt.BoxID():
+ return false, fmt.Sprintf("the probe was served by box %q", resp.Header.Get("X-Carlos-Served-By"))
+ }
+ return true, ""
+}
+
+// answer is what an edge answered one request.
+type answer struct {
+ code int
+ // kind is "page" (the holding page), "app:<box>" (that box's app),
+ // "unknown" (the plain unknown-host answer) or "other".
+ kind string
+ // route is the route an app answer was served from.
+ route string
+ sentence string
+ body string
+ hdr http.Header
+}
+
+func (a answer) String() string {
+ if a.kind == "page" {
+ return fmt.Sprintf("%d page %q", a.code, a.sentence)
+ }
+ return fmt.Sprintf("%d %s %q", a.code, a.kind, strings.TrimSpace(a.body))
+}
+
+// ask is a GET of name through be.
+func (w *holdWorld) ask(be *boxEdge, name string) answer {
+ w.t.Helper()
+ req := httptest.NewRequest(http.MethodGet, "https://"+name+"/", nil)
+ rec := httptest.NewRecorder()
+ be.e.ServeHTTP(rec, req)
+ body := rec.Body.String()
+ a := answer{code: rec.Code, body: body, hdr: rec.Header(), kind: "other"}
+ switch {
+ case rec.Code == http.StatusServiceUnavailable && strings.Contains(body, `<p id="s">`):
+ a.kind = "page"
+ a.sentence = pageSentence(body)
+ case strings.HasPrefix(body, "app on "):
+ f := strings.Fields(body)
+ a.kind = "app:" + f[2]
+ if len(f) == 7 {
+ a.route = f[6]
+ }
+ case rec.Code == http.StatusNotFound && strings.TrimSpace(body) == "unknown host":
+ a.kind = "unknown"
+ }
+ return a
+}
+
+var pageSentenceRe = regexp.MustCompile(`<p id="s">([^<]*)</p>`)
+
+func pageSentence(body string) string {
+ m := pageSentenceRe.FindStringSubmatch(body)
+ if m == nil {
+ return ""
+ }
+ return m[1]
+}
+
+// holdingCopy is the holding page's sentences as the copy record states
+// them (docs/superpowers/plans/2026-10-06-moves-phase1-copy.md, MV-O259 to
+// MV-O262), each as a pattern over the page's escaped sentence: the
+// display name and N are the copy's placeholders.
+var (
+ holdingCopyOnce sync.Once
+ holdingCopy map[string]*regexp.Regexp
+ holdingCopyErr error
+)
+
+func loadHoldingCopy(t *testing.T) map[string]*regexp.Regexp {
+ t.Helper()
+ holdingCopyOnce.Do(func() {
+ raw, err := os.ReadFile(filepath.Join("..", "..", "..", "docs", "superpowers", "plans", "2026-10-06-moves-phase1-copy.md"))
+ if err != nil {
+ holdingCopyErr = err
+ return
+ }
+ holdingCopy = map[string]*regexp.Regexp{}
+ for _, line := range strings.Split(string(raw), "\n") {
+ cols := strings.Split(line, "|")
+ if len(cols) < 5 {
+ continue
+ }
+ id, text := strings.TrimSpace(cols[1]), strings.TrimSpace(cols[3])
+ switch id {
+ case "MV-O259", "MV-O260", "MV-O261", "MV-O262":
+ default:
+ continue
+ }
+ pat := regexp.QuoteMeta(text)
+ pat = strings.ReplaceAll(pat, regexp.QuoteMeta("<display name>"), `(?P<name>.+)`)
+ pat = strings.ReplaceAll(pat, regexp.QuoteMeta("<N>"), `(?P<n>[1-9][0-9]*)`)
+ holdingCopy[id] = regexp.MustCompile("^" + pat + "$")
+ }
+ if len(holdingCopy) != 4 {
+ holdingCopyErr = fmt.Errorf("the copy record has %d of the four holding page rows", len(holdingCopy))
+ }
+ })
+ if holdingCopyErr != nil {
+ t.Fatal(holdingCopyErr)
+ }
+ return holdingCopy
+}
+
+// displayNameHTML is the job world's display name as the page renders it
+// (html/template escapes the apostrophe).
+const displayNameHTML = "Shop&#39;s account"
+
+// requirePage fails unless a is the holding page, with its headers, and
+// its sentence one of the copy rows; it answers which.
+func (w *holdWorld) requirePage(what string, a answer) string {
+ w.t.Helper()
+ switch {
+ case a.kind != "page" || a.code != http.StatusServiceUnavailable:
+ w.t.Fatalf("%s: %s, want the holding page", what, a)
+ case a.hdr.Get("Retry-After") != "30" || a.hdr.Get("Cache-Control") != "no-store" ||
+ a.hdr.Get("X-Robots-Tag") != "noindex" || a.hdr.Get("Content-Type") != "text/html; charset=utf-8":
+ w.t.Fatalf("%s: the page's headers %v", what, a.hdr)
+ }
+ for id, re := range loadHoldingCopy(w.t) {
+ m := re.FindStringSubmatch(a.sentence)
+ if m == nil {
+ continue
+ }
+ if i := re.SubexpIndex("name"); i >= 0 && m[i] != displayNameHTML {
+ w.t.Fatalf("%s: the page names %q, want %q", what, m[i], displayNameHTML)
+ }
+ return id
+ }
+ w.t.Fatalf("%s: the page's sentence %q is none of the recorded copy", what, a.sentence)
+ return ""
+}
+
+// requireApp fails unless a is box's app's own answer with code.
+func (w *holdWorld) requireApp(what string, a answer, box string, code int) {
+ w.t.Helper()
+ if a.kind != "app:"+box || a.code != code {
+ w.t.Fatalf("%s: %s, want %d from the app on %s", what, a, code, box)
+ }
+}
+
+// edgesEntry is one box's publication of a name as its edges file in the
+// store carries it: ready (a route or an alias under hosts, with its
+// pair), held (a holding entry for the host, or one listing it as an
+// alias), or neither.
+type edgesEntry struct {
+ ready, held bool
+ inc, seq uint64
+ holdInc, holdSeq uint64
+ // offlineAt is the holding entry's offline time (unix seconds, 0 for
+ // none yet).
+ offlineAt int64
+ targetHost string
+}
+
+func (w *holdWorld) published(box, name string) edgesEntry {
+ w.t.Helper()
+ raw, _, err := w.raw.Get(w.ctx, edgesKey(box))
+ if err != nil {
+ w.t.Fatalf("edges file of %s: %v", box, err)
+ }
+ var f struct {
+ Hosts map[string]struct {
+ Role string `json:"role"`
+ Incarnation uint64 `json:"incarnation"`
+ ServingSeq uint64 `json:"serving_seq"`
+ TargetHost string `json:"target_host"`
+ } `json:"hosts"`
+ OfflineMoves []struct {
+ Host string `json:"host"`
+ Aliases []string `json:"aliases"`
+ Incarnation uint64 `json:"incarnation"`
+ ServingSeq uint64 `json:"serving_seq"`
+ OfflineAt int64 `json:"offline_at"`
+ } `json:"offline_moves"`
+ }
+ if err := json.Unmarshal(raw, &f); err != nil {
+ w.t.Fatal(err)
+ }
+ var out edgesEntry
+ if h, ok := f.Hosts[name]; ok && h.Role == "home" {
+ out.ready, out.inc, out.seq, out.targetHost = true, h.Incarnation, h.ServingSeq, h.TargetHost
+ }
+ for _, m := range f.OfflineMoves {
+ if m.Host == name || containsString(m.Aliases, name) {
+ out.held, out.holdInc, out.holdSeq, out.offlineAt = true, m.Incarnation, m.ServingSeq, m.OfflineAt
+ }
+ }
+ return out
+}
+
+func containsString(list []string, s string) bool {
+ for _, x := range list {
+ if x == s {
+ return true
+ }
+ }
+ return false
+}
+
+// rowOf is box's registry row for name (nil when it has none).
+func (w *holdWorld) rowOf(box, name string) *registry.Route {
+ w.t.Helper()
+ rt, err := w.e.Box(box).Registry().Get(name)
+ if err != nil {
+ w.t.Fatal(err)
+ }
+ return rt
+}
+
+// attach claims customDomain for the moving instance by an exact claim
+// (the attach door's write) and its parent for the account, so the alias
+// pass makes it an explicit alias on the box serving the instance; then
+// stamps it pointing on each box once its row exists, as the DNS health
+// pass would for a domain whose record names the edges.
+func (w *holdWorld) attach() {
+ w.t.Helper()
+ for _, c := range []domains.Claim{
+ {Host: customParent, Account: w.acct, App: host.E2EApp},
+ {Host: customDomain, Route: host.E2EHost, Account: w.acct, App: host.E2EApp},
+ } {
+ if _, _, err := domains.AttachClaim(w.ctx, w.raw, c); err != nil {
+ w.t.Fatal(err)
+ }
+ }
+ w.boxes()
+ w.pointAliases()
+ w.boxes()
+ w.converge()
+}
+
+// pointAliases stamps every alias row on both boxes pointing (the DNS
+// health pass's verdict for a record that names the edges).
+func (w *holdWorld) pointAliases() {
+ w.t.Helper()
+ for _, label := range []string{host.E2ESourceBox, host.E2ETargetBox} {
+ reg := w.e.Box(label).Registry()
+ as, err := reg.ListAliases()
+ if err != nil {
+ w.t.Fatal(err)
+ }
+ for _, a := range as {
+ if a.DNSStatus != "pointing" {
+ if err := reg.SetAliasDNSStatus(a.Alias, "pointing", time.Now().Unix()); err != nil {
+ w.t.Fatal(err)
+ }
+ }
+ }
+ }
+}
+
+// aliasRow is box's alias row for name (nil when it has none).
+func (w *holdWorld) aliasRow(box, name string) *registry.RouteAlias {
+ w.t.Helper()
+ as, err := w.e.Box(box).Registry().ListAliases()
+ if err != nil {
+ w.t.Fatal(err)
+ }
+ for _, a := range as {
+ if a.Alias == name {
+ return &a
+ }
+ }
+ return nil
+}
+
+// candidates is the third edge's admitted remote candidates for name, as
+// a string for a failure message.
+func (w *holdWorld) candidates(be *boxEdge, name string) []edge.E2ECandidate {
+ return edge.E2ERemoteCandidates(be.e, name)
+}
+
+func describeCandidates(cs []edge.E2ECandidate) string {
+ var parts []string
+ for _, c := range cs {
+ live := ""
+ if !c.Live {
+ live = " expired"
+ }
+ alias := ""
+ if c.Alias {
+ alias = " alias"
+ }
+ parts = append(parts, fmt.Sprintf("%s %s%s (%d,%d)%s", c.Box, c.Type, alias, c.Incarnation, c.Seq, live))
+ }
+ return "[" + strings.Join(parts, "; ") + "]"
+}
diff --git a/internal/edge/e2e_bridge.go b/internal/edge/e2e_bridge.go
new file mode 100644
index 000000000..52f5c10c1
--- /dev/null
+++ b/internal/edge/e2e_bridge.go
@@ -0,0 +1,106 @@
+//go:build e2e
+
+package edge
+
+// The edge's seams for the end-to-end witnesses of self-serve moves plan 3
+// (Task 7, internal/e2e/selfserve, under the same e2e build tag): what an
+// Edge built in another package needs to stand for a box's edge in a test
+// process, and nothing else. The file is compiled only with -tags e2e, so
+// no production binary carries it.
+//
+// RunWithActivatorReady arms the home map, starts its 30 s loop and dials
+// the backhaul on port 443 against the system roots. A witness cannot reach
+// any of that from outside this package, and must drive every poll itself
+// at a clock it controls (the holding entries expire on the map's clock),
+// so these seams arm the same map without its loop, run one poll, set the
+// map's clock, point the backhaul dial at loopback listeners trusting a
+// test CA, and list a name's admitted candidates so a witness can show that
+// a publication is still retained.
+
+import (
+ "context"
+ "crypto/x509"
+ "fmt"
+ "sort"
+ "time"
+)
+
+// E2EArmHomes arms e's home map as RunWithActivatorReady arms it (newHomeMap:
+// over the deployment store EnableStatic gave e, excluding the box
+// WithBoxLabel named, with the region HA switch from the environment), with
+// now as the map's clock (nil: the wall clock), and does not start its loop:
+// the witness runs each poll with E2ERefreshHomes. It fails when e has no
+// deployment store.
+func E2EArmHomes(e *Edge, now func() time.Time) error {
+ hm := e.newHomeMap()
+ if hm == nil {
+ return fmt.Errorf("edge: no deployment store to arm the home map over")
+ }
+ hm.now = now
+ e.homes.Store(hm)
+ return nil
+}
+
+// E2ERefreshHomes is one poll of e's home map (the loop's Refresh): the
+// edges files and the mesh keys, each keeping its last good on failure.
+func E2ERefreshHomes(ctx context.Context, e *Edge) error {
+ hm := e.homes.Load()
+ if hm == nil {
+ return fmt.Errorf("edge: no home map armed")
+ }
+ return hm.Refresh(ctx)
+}
+
+// E2EBackhaulDial points every edge's backhaul dial in this process at port
+// on the published address, trusting roots, until restore is called. Every
+// box is dialled on the one port, so each box listens on its own loopback
+// address.
+func E2EBackhaulDial(port string, roots *x509.CertPool) (restore func()) {
+ oldPort, oldRoots := backhaulPort, backhaulRootCAs
+ backhaulPort, backhaulRootCAs = port, roots
+ return func() { backhaulPort, backhaulRootCAs = oldPort, oldRoots }
+}
+
+// E2ECandidate is one candidate e's home map admitted for a name from
+// another box's edges file: its type ("ready", "holding" or "cert"), the
+// box that published it, whether it is an alias, its pair, and whether a
+// holding candidate is still believed at the map's clock (a ready or cert
+// candidate is always Live).
+type E2ECandidate struct {
+ Type string
+ Box string
+ Alias bool
+ Incarnation, Seq uint64
+ Live bool
+}
+
+// E2ERemoteCandidates is name's admitted remote candidates at the map's
+// clock, sorted by box then type.
+func E2ERemoteCandidates(e *Edge, name string) []E2ECandidate {
+ hm := e.homes.Load()
+ if hm == nil {
+ return nil
+ }
+ now := hm.clock()
+ var out []E2ECandidate
+ for _, c := range hm.snapshot().cands[name] {
+ x := E2ECandidate{Box: c.box, Alias: c.alias, Incarnation: c.incarnation, Seq: c.servingSeq, Live: true}
+ switch c.kind {
+ case candReady:
+ x.Type = "ready"
+ case candHolding:
+ x.Type = "holding"
+ x.Live = now.Sub(c.fetchedAt) < holdingValidity
+ default:
+ x.Type = "cert"
+ }
+ out = append(out, x)
+ }
+ sort.Slice(out, func(i, j int) bool {
+ if out[i].Box != out[j].Box {
+ return out[i].Box < out[j].Box
+ }
+ return out[i].Type < out[j].Type
+ })
+ return out
+}
diff --git a/internal/host/e2e_bridge.go b/internal/host/e2e_bridge.go
index f239bd638..c34a041f8 100644
--- a/internal/host/e2e_bridge.go
+++ b/internal/host/e2e_bridge.go
@@ -49,6 +49,7 @@ import (
"github.com/carlosframework/platform/internal/appconfig"
"github.com/carlosframework/platform/internal/cgroup"
"github.com/carlosframework/platform/internal/domains"
+ "github.com/carlosframework/platform/internal/edge"
"github.com/carlosframework/platform/internal/hostwrite"
"github.com/carlosframework/platform/internal/instancekey"
"github.com/carlosframework/platform/internal/lease"
@@ -118,6 +119,13 @@ type offBox struct {
idSeen string
stagePath func() *bool
diskFree func(dir string) (total, avail uint64, err error)
+ // eip is the public address the box's edges file publishes (the
+ // bridge's BoxHandle.SetEIP); "" publishes the documentation address
+ // e2eDefaultEIP, as before.
+ eip string
+ // pairless makes the next publication a lifetimes-off writer's
+ // (BoxHandle.PublishWithoutPairs).
+ pairless bool
}
// instanceIDOf is the box's metadata service: the instance id it answers,
@@ -937,7 +945,21 @@ func (lw *landWorld) publishErr(ctx context.Context, b *offBox) error {
if b.w.cfg.OfflineCapable {
view = b.w.cfg.Offline
}
- return edgesFileWriteClaims(ctx, lw.wd.s, b.reg, ident, "192.0.2.1", &b.last, false, nil, b.w.cfg.Judge != nil, holdsOf(view))
+ // pairless (BoxHandle.PublishWithoutPairs) writes as a box that judges
+ // no lifetimes: no pair and no holding entry.
+ lifetimes := b.w.cfg.Judge != nil && !b.pairless
+ return edgesFileWriteClaims(ctx, lw.wd.s, b.reg, ident, b.publicIP(), &b.last, false, nil, lifetimes, holdsOf(view))
+}
+
+// e2eDefaultEIP is the address a box's edges file publishes until the
+// bridge names another (TEST-NET-1: nothing answers it).
+const e2eDefaultEIP = "192.0.2.1"
+
+func (b *offBox) publicIP() string {
+ if b.eip != "" {
+ return b.eip
+ }
+ return e2eDefaultEIP
}
// tick is the agent's order on one box: reconcile, the edges file, the
@@ -1029,11 +1051,20 @@ type E2E struct {
boxes []*e2eBox
mu sync.Mutex
faults map[string]func()
+ // aliasRoutes, when set, turns on the alias pass in every tick and is
+ // its deployment route fold (AliasPass).
+ aliasMu sync.Mutex
+ aliasRoutes func(context.Context) map[string]bool
}
type e2eBox struct {
b *offBox
down atomic.Bool
+ // holds is the box's own edge's hold callback (BoxHandle.SubscribeHolds),
+ // subscribed to the box's offline view again at every agent restart, as
+ // subscribeArmedHolds subscribes the ready edge to a newly armed view.
+ holdsMu sync.Mutex
+ holds func(edge.HoldSnapshot)
// The knobs (BoxHandle.SetStagePathArmed, SetDiskFree), read on every
// use: the stage path's answer (nil: unknown) and the disk's (nil: the
// box's real statfs, diskBytes).
@@ -1161,6 +1192,19 @@ func (e *E2E) rearm(x *e2eBox) {
return carry(ctx, host, bucket, prefix)
}
}
+ x.subscribeHolds()
+}
+
+// subscribeHolds subscribes the box's edge callback, when the witness gave
+// one, to the armed worker's offline view, on the terms subscribeHoldsLocked
+// subscribes the agent's edge: only a capable box's view publishes holds.
+func (x *e2eBox) subscribeHolds() {
+ x.holdsMu.Lock()
+ defer x.holdsMu.Unlock()
+ if x.holds == nil || x.b.w == nil || !x.b.w.cfg.OfflineCapable || x.b.w.cfg.Offline == nil {
+ return
+ }
+ x.b.w.cfg.Offline.Subscribe(x.holds)
}
// Store is the deployment store the bridge was built over.
@@ -1294,6 +1338,14 @@ func (e *E2E) tick(ctx context.Context, x *e2eBox) error {
if err := b.reconcileErr(ctx); err != nil {
errs = append(errs, fmt.Errorf("%s: reconcile: %w", b.name, err))
}
+ e.aliasMu.Lock()
+ routes := e.aliasRoutes
+ e.aliasMu.Unlock()
+ if routes != nil {
+ if err := e.aliasPass(ctx, b, routes); err != nil {
+ errs = append(errs, fmt.Errorf("%s: aliases: %w", b.name, err))
+ }
+ }
if err := e.lw.publishErr(ctx, b); err != nil {
errs = append(errs, fmt.Errorf("%s: edges file: %w", b.name, err))
}
@@ -1303,6 +1355,31 @@ func (e *E2E) tick(ctx context.Context, x *e2eBox) error {
return errors.Join(errs...)
}
+// aliasPass is the agent's fast-converge alias pass on b
+// (domains.ReconcileAliases over the deployment's routes, under the box's
+// lifetime judge), run here after the reconcile and before the edges file,
+// so a tick's publication carries the aliases its reconcile allowed. The
+// claim cache is left out (nil: the uncached fold, legal by contract).
+func (e *E2E) aliasPass(ctx context.Context, b *offBox, deploymentRoutes func(context.Context) map[string]bool) error {
+ ctx = provision.WithLifetimes(ctx, b.wd.judge)
+ return domains.ReconcileAliases(ctx, e.lw.wd.s, b.reg, time.Now, deploymentRoutes(ctx), nil)
+}
+
+// AliasPass turns the alias pass on in every tick from now on, folding the
+// deployment's real routes for it with routes each pass, as the agent folds
+// them (edgesfile.DeploymentRoutes over the store); nil turns it off, which
+// is the default every witness before plan 3 ran with. With it on, a claim
+// attached in the deployment store (domains.AttachClaim) becomes an alias
+// row on the box that serves its route, explicit when an exact claim backs
+// it, through the reconciler the agent runs. The witness hands the fold in
+// so the bridge itself reads no edges file (the edges-file read sites are
+// inventoried, internal/edgesfile's audit).
+func (e *E2E) AliasPass(routes func(context.Context) map[string]bool) {
+ e.aliasMu.Lock()
+ e.aliasRoutes = routes
+ e.aliasMu.Unlock()
+}
+
func (e *E2E) status(ctx context.Context, x *e2eBox) error {
b := x.b
now := e.clock.follow()
@@ -1394,6 +1471,33 @@ func (h BoxHandle) InstanceID() string { return h.x.b.w.cfg.instance() }
// and then ArmOffline (the new machine's agent process).
func (h BoxHandle) SetInstanceID(id string) { h.x.b.instanceID.Store(&id) }
+// SetEIP sets the public address the box's edges file publishes from its
+// next publication on (the bridge publishes e2eDefaultEIP until then), so a
+// witness's edge can dial the box at a listener of its own.
+func (h BoxHandle) SetEIP(ip string) { h.x.b.eip = ip }
+
+// SubscribeHolds subscribes fn, the box's own edge's edge.SetHoldView, to
+// the box's offline view as the agent subscribes its ready edge
+// (subscribeEdgeHolds): the view delivers its current snapshot at once and
+// every publication after. The subscription is renewed at every agent
+// restart (ArmOffline), whose worker arms a new view.
+func (h BoxHandle) SubscribeHolds(fn func(edge.HoldSnapshot)) {
+ h.x.holdsMu.Lock()
+ h.x.holds = fn
+ h.x.holdsMu.Unlock()
+ h.x.subscribeHolds()
+}
+
+// PublishWithoutPairs is one pass of the box's edges file as a writer that
+// judges no lifetimes writes it (lifetimes off: no pair on any entry and no
+// holding entry), the publication a box from before lifetimes left in the
+// bucket.
+func (h BoxHandle) PublishWithoutPairs(ctx context.Context) error {
+ h.x.b.pairless = true
+ defer func() { h.x.b.pairless = false }()
+ return h.e.lw.publishErr(ctx, h.x.b)
+}
+
// Region is the box's placement ("" is the home).
func (h BoxHandle) Region() string { return h.x.b.region }
diff --git a/internal/provision/testdata/plan14-root-reads-self-serve-moves-3.tsv b/internal/provision/testdata/plan14-root-reads-self-serve-moves-3.tsv
index 8a9cd065a..9d3140fdc 100644
--- a/internal/provision/testdata/plan14-root-reads-self-serve-moves-3.tsv
+++ b/internal/provision/testdata/plan14-root-reads-self-serve-moves-3.tsv
@@ -7,6 +7,7 @@
# Task 5 (freshness), rev3=PROPOSED-self-serve-moves-3: the home map's wake compares the platform holds live at each poll's time (wakeOnMove, livePlatformHolds in homemap.go), so a remote platform hold's expiry and its revival by a fresh fetch wake the wildcard manager; Hosts derives from the same live set. homeMap.Refresh, refreshHosts and Hosts read no row and are no census units. Only the internal/edge envelope is re-pinned; no unit moves, so the seal is unchanged. No root read, class or task is added.
# Task 6 (probe mode), rev3=PROPOSED-self-serve-moves-3: a probe (X-Carlos-Probe: local) is refused in backhaulTo before the key and the cap (503 "not served here", no dial), and a probe served locally is answered under servedByWriter (servedby.go, new; reads no row) and kept out of the response cache. Edge.ServeHTTP is re-pinned in place (read sites unchanged, offsets moved). The console's health probe asks in probe mode and requires X-Carlos-Served-By to be the pinned target's box id: Server.probeTarget, accepted legacy since self-serve moves 2 Task 8, is re-pinned here (read site unchanged). Both re-pins move the seal; the internal/edge and internal/console envelopes are re-pinned. No root read, class or task is added.
# Task 6 review round 1, rev3=PROPOSED-self-serve-moves-3: the two "not served here" bodies (the probe refusal and homeMap.staleHopRefusal) share one constant, notServedHere in servedby.go, and ServeHTTP gains a comment on why a probe may be served from the local catch-all. Edge.ServeHTTP is re-pinned in place (read sites unchanged, offsets moved), which moves the seal; the internal/edge envelope is re-pinned. No root read, class or task is added.
+# Task 7 (the end-to-end witnesses), rev3=PROPOSED-self-serve-moves-3: tests in internal/e2e/selfserve, plus two seams. internal/edge/e2e_bridge.go (new, //go:build e2e, so no production binary carries it) arms an Edge's home map without its loop, runs one poll, sets the map's clock, points the backhaul dial at loopback listeners and lists a name's admitted candidates: it reads no row, and the census's name match over the inactive file is kept clear of tracked field names (E2ECandidate.Type, not Kind). internal/host/e2e_bridge.go gains a per-box published address, the box edge's hold subscription (renewed at every agent restart), an opt-in alias pass in each tick (domains.ReconcileAliases, already accepted legacy, over the deployment route fold the witness hands in, so the bridge adds no edges-file read site to internal/edgesfile's audit) and a pair-less publication through the same writer (publishErr with lifetimes off): no unit with a read site moves. The internal/edge and internal/host envelopes are re-pinned; no root read, class or task is added, and the seal is unchanged.
@legacy 652c416a0cd27a153fa259d76ab20538480e2c3afd98cbec6af083902b17f5ff internal/domains/aliases.go ReconcileAliases legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-3;basis=root-route; the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its fast-converge and instances loops: self-serve moves 1's body (rev3=J1, the catch-all's labels counted through domains.InstanceLabelCountsJudged), which now writes each alias row's provenance: an exact claim on the row's own name upserts it explicit (every pass, so a legacy row is upgraded on the first), and the synthetic "*.<host>" catch-all row not explicit; registry read sites unchanged, offsets moved; a report the edges file lists holding aliases by, not a decision registry:Account@15605,registry:Account@15618,registry:Alias@15220,registry:Alias@15260,registry:Alias@15330,registry:App@12165,registry:App@12365,registry:App@12521,registry:App@12722,registry:App@15631,registry:App@15640,registry:App@2026,registry:App@2222,registry:App@6036,registry:App@6235,registry:Host@15590,registry:Host@15879,registry:Host@15932,registry:Host@16004,registry:Host@16516,registry:Host@16608,registry:Host@16667,registry:Host@16748,registry:Host@406,registry:Prefix@16524,registry:Prefix@16622,registry:Prefix@16675,registry:Prefix@16756,registry:Status@8899,whole:Route@12134,whole:Route@12209,whole:Route@12356,whole:Route@12418,whole:Route@12561,whole:Route@12717,whole:Route@13613,whole:Route@1369,whole:Route@14825,whole:Route@14833,whole:Route@2066,whole:Route@2217,whole:Route@2273,whole:Route@359,whole:Route@388,whole:Route@399,whole:Route@417,whole:Route@5382,whole:Route@6080,whole:Route@6226,whole:Route@8229,whole:Route@8681,whole:Route@8843,whole:RouteAlias@14841,whole:RouteAlias@15195,whole:RoutePath@10426,whole:RoutePath@16409,whole:RoutePath@16451,whole:RoutePath@16565,whole:RoutePath@16595,whole:WildcardDomain@12933,whole:WildcardDomain@15520,whole:WildcardDomain@15569,whole:WildcardDomain@15732,whole:WildcardDomain@15838,whole:WildcardDomain@15867 edef8a1bf06cab5666a52ec19840c54f02913ddc67a32da7f6197db1606556a4
internal/domains/aliases.go ReconcileAliases legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-3;basis=root-route; the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its fast-converge and instances loops: self-serve moves 1's body (rev3=J1, the catch-all's labels counted through domains.InstanceLabelCountsJudged), which now writes each alias row's provenance: an exact claim on the row's own name upserts it explicit (every pass, so a legacy row is upgraded on the first), and the synthetic "*.<host>" catch-all row not explicit; registry read sites unchanged, offsets moved; a report the edges file lists holding aliases by, not a decision registry:Account@15605,registry:Account@15618,registry:Alias@15220,registry:Alias@15260,registry:Alias@15330,registry:App@12165,registry:App@12365,registry:App@12521,registry:App@12722,registry:App@15631,registry:App@15640,registry:App@2026,registry:App@2222,registry:App@6036,registry:App@6235,registry:Host@15590,registry:Host@15879,registry:Host@15932,registry:Host@16004,registry:Host@16516,registry:Host@16608,registry:Host@16667,registry:Host@16748,registry:Host@406,registry:Prefix@16524,registry:Prefix@16622,registry:Prefix@16675,registry:Prefix@16756,registry:Status@8899,whole:Route@12134,whole:Route@12209,whole:Route@12356,whole:Route@12418,whole:Route@12561,whole:Route@12717,whole:Route@13613,whole:Route@1369,whole:Route@14825,whole:Route@14833,whole:Route@2066,whole:Route@2217,whole:Route@2273,whole:Route@359,whole:Route@388,whole:Route@399,whole:Route@417,whole:Route@5382,whole:Route@6080,whole:Route@6226,whole:Route@8229,whole:Route@8681,whole:Route@8843,whole:RouteAlias@14841,whole:RouteAlias@15195,whole:RoutePath@10426,whole:RoutePath@16409,whole:RoutePath@16451,whole:RoutePath@16565,whole:RoutePath@16595,whole:WildcardDomain@12933,whole:WildcardDomain@15520,whole:WildcardDomain@15569,whole:WildcardDomain@15732,whole:WildcardDomain@15838,whole:WildcardDomain@15867 edef8a1bf06cab5666a52ec19840c54f02913ddc67a32da7f6197db1606556a4
@legacy 887eab39c9584891268b9afedcd81c995849a97545258b3eacc506b5c0165b23 internal/domains/aliases.go deriveInstanceAliases legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-3;basis=root-route; the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): self-serve moves 1's body, whose already-the-row-we-want skip now also compares the existing row's provenance (registry:Explicit, one new site): a derived name left explicit by a detached exact claim on the same target and owner is rewritten not explicit, and both keep-as-it-is paths (a listing failure, an unjudged label) hand the kept row to keepDerived; the other registry read sites unchanged, offsets moved; no new route registry:Account@2163,registry:Account@3516,registry:Account@4587,registry:Alias@2225,registry:Alias@2293,registry:Alias@2337,registry:Alias@557,registry:App@1686,registry:App@2189,registry:App@3542,registry:App@4613,registry:Explicit@4632,registry:Host@1843,registry:Host@4566,registry:Host@4576,registry:Host@4787,registry:Provisioned@1636,registry:Replica@1672,registry:Slug@1775,whole:Route@1620,whole:Route@1726,whole:Route@3058,whole:RouteAlias@2143,whole:RouteAlias@2377,whole:RouteAlias@3498,whole:RouteAlias@3867,whole:RouteAlias@4548,whole:RouteAlias@509,whole:RouteAlias@539,whole:RouteAlias@552,whole:RouteAlias@568 42d5a40cccb38c31f306b49c00fde7482e4ee0b3cbf6a6a2dfbaac06592ef883
@@ -46,7 +47,7 @@ internal/console/selfserve_health.go Server.probeTarget legacy accepted-legacy a
@fields A,Account,Accounts,AddedAt,Addr,Addrs,Alias,Aliases,Alive,AllowAbsent,App,Archives,Args,At,Attempts,B,Backing,Bucket,BuildState,Callback,CallbackError,CallbackFailed,Canary,Caps,CertExpiresAt,Changed,Channel,CheckedAt,ClaimIncarnation,ClaimOwner,ClaimOwnership,ClaimRowID,ClaimStamp,Class,Contents,CreatedAt,DBPath,DNSCheckedAt,DNSStatus,Deadline,Demand,Digest,DoneBefore,Due,DueAt,Env,EnvBefore,Environment,Error,ErrorKind,ExcludeHosts,ExpectGood,ExpectOp,ExpectPair,Expected,Explicit,F,Failed,FenceGen,Fleet,FleetBox,FleetBoxes,Fleets,From,FromDB,FromOp,FromSock,Gate,GoodDemand,GoodDigest,GoodVersion,Hash,Health,Held,Hibernate,HoldOp,Host,Hosts,ID,Id,IdValue,Idle,IdleSeconds,ImportGate,Incarnation,Journal,Keep,Kind,Label,LastAt,LastError,LastMs,LastStatus,Limit,MaxRun,MaxRunSeconds,Method,Ms,Name,Next,NextAt,NextDue,NextGood,NextValue,Now,Number,Number64,Observed,ObservedValue,Old,On,Op,Original,Owner,Ownership,P,Pair,Path,Paths,Platform,Pool,Prefix,Prev,PrevVersion,Provisioned,RecordKey,Region,Registration,Replica,Replicas,Replication,Retirement,Route,Routes,Row,Rows,Seed,SeedID,SeedSource,ServingSeq,ServingVersion,Slug,Socket,Spec,Sqid,State,Status,StatusNamed,Steering,StringMap,T,TValue,Tags,Target,Targets,Text,Timer,Timers,Times,To,ToDB,ToOp,ToSock,TokenHash,Tunnel,Unit,UpdateSettings,Ver,Version,W,WakeIncarnation,Wildcards,closed,conn,connect,ctx,d,dec,done,enc,full,input,mu,output,path,remote,sql,wait
@coverage internal/console bc88b0e12e2a48b9a22951b01cf4e10f65e475079bce2a6cf076942ac1808ef6
@coverage internal/domains 0cf7748c97e4caa7d0daa73e251fdd2f2f03e425321599bf046292b0a86ebde8
-@coverage internal/edge 5ef637212c8c1cc68016f3bef3f0ab214d9cfa5b23a1f58dac16f4990ed07255
+@coverage internal/edge 9da3d71f792b907f50fb8a8c57384ede585bf652ec82e9631741916272753efd
@coverage internal/edgesfile ee82e43c7d6363297f63abcde9c3e828a53d377573649f0d26a9853056aad4be
-@coverage internal/host 27d077c9f63838516ad29fb69ea48ebbf85a7c17ade258a3b8ce2d57eb3cc21d
+@coverage internal/host 30ad675ecba1fc67bc34028c91c4da4e8b83ffa730f6e172fdd73ab7f7fa40f9
@coverage internal/registry ae19eaec5c4dde2c400fcb2fc6afd23c6b129e77306d6916fe55b2091b0fb5ed