amadan
provision: serving classes — static and bake adopt on every edge
The multi-region ingress design's Part 3 table, stated in one place and
asked by both halves of the pass:
Everywhere static (bake-served) — the edge resolves the app's
channel pointer out of the bucket every box reads
Everywhere, probed addr routes whose upstreams are ALL publicly
dialable — each edge probes and dials them itself
Home-boxed exec, unit-owned, socket-activated, tunnel- and
fleet-backed — the process exists on ONE box, and a
request landing elsewhere backhauls to it
Only the REGION half of the scope gate widens, and only between platform
regions. Pool stays a hard tenancy boundary (it names the boxes a tenant
is allowed on, not a geography), and a fleet placement ("fleet/<name>")
is refused from both sides: the platform's boxes do not pick up a
customer fleet's routes, and a customer's box does not pick up the
platform's. Unit-owned routes stay home-boxed because adoption CYCLES
the named unit — that exclusion is what keeps a far box from ever
running systemctl for a unit name that means something else there.
The removal phase widens with it, through the same inScope with the
class read off the row: a far box that mints a static route under a
widened gate and cannot then see it as one would serve a retired site
forever. Anything the row cannot be positively classified from —
a name-based upstream, a private or loopback address, a row that has
grown Backing/Tunnel/Caps since it was minted — reads as home-boxed,
which is the direction a wrong answer has to fail in.
Single-box inertness: 'every box' is the one box, every record is in
scope for exactly the reason it was before, and the widened half of the
gate never fires. Nothing writes control/serial.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 files changed,
+382
−3
internal/provision/reconcile.go+169 −3internal/provision/reconcile_test.go+213 −0
diff --git a/internal/provision/reconcile.go b/internal/provision/reconcile.go| index 71ee5a3b7..56b0b32c0 100644 |
| --- a/internal/provision/reconcile.go |
| +++ b/internal/provision/reconcile.go |
| @@ -1,7 +1,9 @@ |
| package provision |
| // Reconcile turns bucket instance records into registry routes, for THIS |
| -// agent's own region and pool set. Spec: |
| +// agent's own region and pool set — plus, since the multi-region ingress |
| +// design's Part 3, the serving classes every platform box materializes |
| +// wherever they are homed (see "serving classes" below). Spec: |
| // docs/superpowers/specs/2026-08-04-instance-provisioning-design.md §4. |
| // |
| // DNS seam (spec §10): multi-region deployments write the per-slug CNAME at |
| @@ -14,6 +16,7 @@ import ( |
| "errors" |
| "fmt" |
| "log" |
| + "net" |
| "os" |
| "slices" |
| "strings" |
| @@ -52,6 +55,160 @@ type ReconcileConfig struct { |
| // only within an account, never globally. |
| type acctApp struct{ acct, app string } |
| +// ---- serving classes ------------------------------------------------------- |
| +// |
| +// Which boxes materialize a record. The multi-region ingress design's Part 3 |
| +// table, verbatim, and the ONE place this pass states it: |
| +// |
| +// Everywhere static (bake-served): the edge resolves the app's |
| +// channel pointer and serves the shipped tar out of the |
| +// deployment store every box already reads, so no box |
| +// needs the home box to serve it. |
| +// Everywhere, probed addr routes whose upstreams are ALL publicly dialable: |
| +// each edge runs its own prober and dials the backends |
| +// directly, so per-edge live sets are per-edge by design. |
| +// Home-boxed exec, unit-owned, socket-activated, tunnel- and |
| +// fleet-backed: the process exists on exactly ONE box, |
| +// and a request that lands elsewhere backhauls to it |
| +// (internal/edge's hop). Unchanged invariant. |
| +// |
| +// Everything here decides SCOPE only. Nothing about the route a record mints |
| +// changes with the box that mints it — routeFor stays the one renderer, so a |
| +// far box's copy and the home box's copy are the same route, column for |
| +// column, including the record's own Region. |
| +// |
| +// Single-box inertness: with one box there is no far box, every record is in |
| +// scope for the same reason it was before, and the widened half of the gate |
| +// never fires. |
| + |
| +// everywhereClass reports whether rec's serving CLASS is one every platform |
| +// box materializes. It answers the class question only — the tenancy |
| +// questions (pool, fleet placement) are inScope's, below. |
| +// |
| +// Conservative by construction: anything this function cannot positively |
| +// classify as servable from a box that holds nothing but the bucket is |
| +// home-boxed. That is the direction a wrong answer must fail in — a route that |
| +// stays home-boxed is served exactly as it is today (and backhauled from |
| +// elsewhere), while one wrongly promoted to everywhere is a box proxying to an |
| +// upstream it cannot reach. |
| +func everywhereClass(rec Record) bool { |
| + // Unit names the systemd unit that owns this route's process (issue |
| + // #110); adoption CYCLES it. That unit exists on the home box, so a far |
| + // box adopting the row would be stamping authority over a unit name that |
| + // means something else there, or nothing at all. |
| + if rec.Unit != "" { |
| + return false |
| + } |
| + // Belt and braces: ValidateShape already refuses hibernation on both |
| + // classes below (a static route has no process to wake, an addr-backed one |
| + // has no socket), and this pass validates every record before the gate. A |
| + // hibernating route needs the activator and the socket only its home box |
| + // has, so if that door ever moved, this must not be the hole it opens. |
| + if rec.HibernateOrDefault() { |
| + return false |
| + } |
| + if rec.KindOrDefault() == "static" { |
| + return true |
| + } |
| + ups := rec.AddrsOrDefault() |
| + if len(ups) == 0 { |
| + return false // socket-backed: the exec child lives on one box |
| + } |
| + for _, a := range ups { |
| + if !publiclyDialable(a) { |
| + return false |
| + } |
| + } |
| + return true |
| +} |
| + |
| +// everywhereRoute is everywhereClass read off a registry ROW instead of a |
| +// record — the removal phase's half of the same question, asked when the |
| +// record is gone and the row is all that is left to ask. |
| +// |
| +// The two must agree, or the widened mint gate leaves immortal routes: a far |
| +// box that mints an everywhere-class route and then cannot see it as one is a |
| +// box serving a retired site forever. They are separate functions rather than |
| +// one over a shared shape because the row carries columns no record can state |
| +// (Backing, Tunnel, Caps) — and a row that has GROWN one of those since it was |
| +// minted is deliberately read as home-boxed here, which costs a stale row on a |
| +// far box and never a surprise deletion. |
| +func everywhereRoute(rt registry.Route) bool { |
| + if rt.Unit != "" || rt.Hibernate || rt.Backing != "" || rt.Tunnel != "" || rt.Caps != "" { |
| + return false |
| + } |
| + if rt.Kind == "static" { |
| + return true |
| + } |
| + ups := rt.AddrList() |
| + if len(ups) == 0 { |
| + return false |
| + } |
| + for _, a := range ups { |
| + if !publiclyDialable(a) { |
| + return false |
| + } |
| + } |
| + return true |
| +} |
| + |
| +// publiclyDialable reports whether one upstream is an address every edge in |
| +// the deployment can reach — the "publicly-reachable upstreams" half of the |
| +// spec's table. |
| +// |
| +// A NAME is not dialable for this purpose even if it resolves to a public |
| +// address today: validAddrsElement's doc comment already records why a name |
| +// cannot be judged at all here (it is free to move after the check), and the |
| +// legacy Addr column is full of names pointing at box-local backends. It stays |
| +// home-boxed, which is where those routes are serving from today. |
| +// |
| +// Loopback and link-local are the spec's own examples (another box's |
| +// 127.0.0.1 is not this box's). RFC1918 and ULA are refused for exactly the |
| +// same reason one layer out: a private address is private to ONE network, and |
| +// two boxes in two regions are two networks. IsGlobalUnicast covers the rest |
| +// (unspecified, multicast, loopback, link-local). |
| +func publiclyDialable(addr string) bool { |
| + h, _, err := net.SplitHostPort(addr) |
| + if err != nil { |
| + return false |
| + } |
| + ip := net.ParseIP(h) |
| + if ip == nil { |
| + return false |
| + } |
| + return ip.IsGlobalUnicast() && !ip.IsPrivate() |
| +} |
| + |
| +// inScope answers "is this region/pool pair this pass's business" — the one |
| +// gate the record scan and the removal phase both ask, so a record this box |
| +// mints is a record this box can retire. |
| +// |
| +// Pool is compared FIRST and never widened. It is a TENANCY boundary, not a |
| +// geography: a pool names the boxes a tenant's routes are allowed to land on |
| +// (customer fleets, dedicated boxes), and "everywhere" in the spec's table |
| +// means every box that already serves this tenant, not every box in the world. |
| +// |
| +// Region equality is the unchanged home-box test. It is widened for the |
| +// everywhere classes, and only between PLATFORM regions: a fleet placement |
| +// ("fleet/<name>", registry.FleetPlacement) is the other tenancy boundary and |
| +// is refused from both sides — the platform's boxes do not pick up a |
| +// customer fleet's routes, and a customer's box does not pick up the |
| +// platform's. |
| +func (cfg ReconcileConfig) inScope(region, pool string, everywhere bool) bool { |
| + if !poolServed(cfg.Pools, pool) { |
| + return false |
| + } |
| + if region == cfg.Region { |
| + return true |
| + } |
| + if !everywhere { |
| + return false |
| + } |
| + _, recordOnFleet := registry.FleetPlacement(region) |
| + _, boxOnFleet := registry.FleetPlacement(cfg.Region) |
| + return !recordOnFleet && !boxOnFleet |
| +} |
| + |
| // routeFor is the one answer to "what route does this record describe". The |
| // mint path builds from it and the adoption match test (design §4.2) |
| // compares against it — one function, so a shape the reconciler would mint |
| @@ -418,7 +575,10 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg |
| protect[host] = true |
| continue |
| } |
| - if rec.Region != cfg.Region || !poolServed(cfg.Pools, rec.Pool) { |
| + // Scope. For a home-boxed record this is the region/pool test it |
| + // always was; for a static or public-addr one the region half widens |
| + // to every platform box in the deployment (everywhereClass, above). |
| + if !cfg.inScope(rec.Region, rec.Pool, everywhereClass(rec)) { |
| continue // another agent's jurisdiction, not this pass's business |
| } |
| // Two accounts writing the same host is a collision this pass |
| @@ -640,6 +800,12 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg |
| // absence from the record set this pass just read (the bucket's own |
| // decision to retire the instance, not a guess this pass made). |
| // |
| + // Scope is asked through the SAME inScope as the record scan above, with |
| + // the class read off the row (everywhereRoute) because the record whose |
| + // class it was is exactly what is gone. Both halves must widen together: |
| + // a far box that mints a static route under the widened mint gate and then |
| + // judges it out of scope here would serve a retired site forever. |
| + // |
| // What reaches reg.Delete here may carry Backing, Tunnel or Caps even |
| // though shapeMismatch refuses to ADOPT a row carrying any of them: those |
| // refusals cover the adoption instant only. `carlos route -backing/-unit/ |
| @@ -650,7 +816,7 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg |
| // the adoption guard is not mistaken for a standing guarantee. |
| if !readFailed { |
| for _, rt := range routes { |
| - if !rt.Provisioned || rt.Region != cfg.Region || !poolServed(cfg.Pools, rt.Pool) { |
| + if !rt.Provisioned || !cfg.inScope(rt.Region, rt.Pool, everywhereRoute(rt)) { |
| continue |
| } |
| if _, ok := recs[rt.Host]; ok { |
diff --git a/internal/provision/reconcile_test.go b/internal/provision/reconcile_test.go| index f9193a80d..9048d09d4 100644 |
| --- a/internal/provision/reconcile_test.go |
| +++ b/internal/provision/reconcile_test.go |
| @@ -1826,6 +1826,219 @@ func TestReconcileRefusesAdoptInstanceRouteForStaticRecord(t *testing.T) { |
| } |
| } |
| +// ---- serving classes (multi-region ingress design, Part 3's table) --------- |
| +// |
| +// The same deployment seen from two PLATFORM boxes: the record's home region |
| +// ("ie") and another one ("us"), both serving the tito pool. Everything below |
| +// asks the same question of one record — "does the far box materialize it?" — |
| +// and the answer is the record's serving class. |
| + |
| +func homeBoxCfg() ReconcileConfig { |
| + return ReconcileConfig{Region: "ie", Pools: []string{"tito"}, DataDir: "/data"} |
| +} |
| + |
| +func farBoxCfg() ReconcileConfig { |
| + return ReconcileConfig{Region: "us", Pools: []string{"tito"}, DataDir: "/data"} |
| +} |
| + |
| +// titoRecord is the tito-app fixture every serving-class case below mutates: |
| +// homed in "ie", in the "tito" pool, on the stable channel. |
| +func titoRecord(sqid, host string) Record { |
| + return Record{Schema: 1, Host: host + ".go.tito.io", Slug: host, App: "tito", |
| + Account: sqid, Pool: "tito", Region: "ie", Channel: "stable", CreatedAt: time.Now()} |
| +} |
| + |
| +// A static (bake-served) record is materialized by EVERY platform box in the |
| +// deployment, not only its home: the edge resolves the app's channel pointer |
| +// out of the same bucket every box already reads, so no box needs the home box |
| +// to serve it. Spec Part 3, class "Everywhere". |
| +func TestReconcileStaticIsMaterializedOnEveryPlatformBox(t *testing.T) { |
| + s, sqid := setup(t) |
| + rec := titoRecord(sqid, "site") |
| + rec.Kind = "static" |
| + if err := CreateRecord(ctx, s, rec); err != nil { |
| + t.Fatal(err) |
| + } |
| + reg := regWith(t) |
| + if err := Reconcile(ctx, s, reg, farBoxCfg()); err != nil { |
| + t.Fatal(err) |
| + } |
| + got := mustGet(t, reg, "site.go.tito.io") |
| + if !got.Provisioned || got.Kind != "static" { |
| + t.Fatalf("far box did not materialize the static record: %+v", *got) |
| + } |
| + // The row still carries the RECORD's region — the record is the desired |
| + // state and routeFor is its one renderer, so a far box's copy and the home |
| + // box's copy are the same route, byte for byte. |
| + if got.Region != "ie" { |
| + t.Errorf("Region = %q, want the record's own %q", got.Region, "ie") |
| + } |
| + if got.Socket != "" || got.DBPath != "" || got.Hibernate { |
| + t.Errorf("a static route grew a process shape on the far box: %+v", *got) |
| + } |
| +} |
| + |
| +// An addr route whose upstreams are all publicly dialable is materialized |
| +// everywhere too — every edge probes and dials the same public backends. Spec |
| +// Part 3, class "Everywhere, probed". |
| +func TestReconcilePublicAddrIsMaterializedOnEveryPlatformBox(t *testing.T) { |
| + s, sqid := setup(t) |
| + one := titoRecord(sqid, "pub") |
| + one.Kind = "service" |
| + one.Addr = "203.0.113.10:443" |
| + many := titoRecord(sqid, "pubs") |
| + many.Kind = "service" |
| + many.Addrs = []string{"203.0.113.11:443", "198.51.100.7:443"} |
| + for _, rec := range []Record{one, many} { |
| + if err := CreateRecord(ctx, s, rec); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + reg := regWith(t) |
| + if err := Reconcile(ctx, s, reg, farBoxCfg()); err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, host := range []string{"pub.go.tito.io", "pubs.go.tito.io"} { |
| + if got := mustGet(t, reg, host); !got.Provisioned { |
| + t.Errorf("%s: far box did not materialize a public-upstream record: %+v", host, *got) |
| + } |
| + } |
| +} |
| + |
| +// The home-boxed classes, unchanged: the process backing them exists on |
| +// exactly ONE box, so a far box must still refuse every one of them. This is |
| +// the refusal test the class change is only safe with. |
| +func TestReconcileHomeBoxedClassesStayHomeBoxed(t *testing.T) { |
| + s, sqid := setup(t) |
| + socketed := titoRecord(sqid, "exec") // no kind, no addr: the socket/exec shape |
| + loopback := titoRecord(sqid, "loop") |
| + loopback.Kind = "service" |
| + loopback.Addr = "127.0.0.1:8090" |
| + private := titoRecord(sqid, "priv") |
| + private.Kind = "service" |
| + private.Addrs = []string{"10.0.0.5:3000"} |
| + unitOwned := titoRecord(sqid, "unit") |
| + unitOwned.Addr = "203.0.113.20:8090" |
| + unitOwned.Unit = "console.service" |
| + named := titoRecord(sqid, "named") |
| + named.Kind = "service" |
| + named.Addr = "backend.example.com:443" |
| + for _, rec := range []Record{socketed, loopback, private, unitOwned, named} { |
| + if err := CreateRecord(ctx, s, rec); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + reg := regWith(t) |
| + if err := Reconcile(ctx, s, reg, farBoxCfg()); err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, host := range []string{"exec.go.tito.io", "loop.go.tito.io", "priv.go.tito.io", |
| + "unit.go.tito.io", "named.go.tito.io"} { |
| + if rt, err := reg.Get(host); err != nil || rt != nil { |
| + t.Errorf("%s: a home-boxed record materialized on a foreign box: %+v %v", host, rt, err) |
| + } |
| + } |
| + // ...and every one of them still materializes on its OWN box. |
| + if err := Reconcile(ctx, s, reg, homeBoxCfg()); err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, host := range []string{"exec.go.tito.io", "loop.go.tito.io", "priv.go.tito.io", |
| + "unit.go.tito.io", "named.go.tito.io"} { |
| + mustGet(t, reg, host) |
| + } |
| +} |
| + |
| +// Pool is a TENANCY boundary, not a geography: widening the region half of the |
| +// gate must not widen the pool half. A static record in the tito pool stays |
| +// off a box that does not serve that pool, however far it travels. |
| +func TestReconcileEverywhereClassStillRespectsThePool(t *testing.T) { |
| + s, sqid := setup(t) |
| + rec := titoRecord(sqid, "site") |
| + rec.Kind = "static" |
| + if err := CreateRecord(ctx, s, rec); err != nil { |
| + t.Fatal(err) |
| + } |
| + reg := regWith(t) |
| + // A far box serving only the shared "" pool (nil Pools), and one serving |
| + // somebody else's pool. |
| + for _, cfg := range []ReconcileConfig{ |
| + {Region: "us", DataDir: "/data"}, |
| + {Region: "us", Pools: []string{"eleven"}, DataDir: "/data"}, |
| + } { |
| + if err := Reconcile(ctx, s, reg, cfg); err != nil { |
| + t.Fatal(err) |
| + } |
| + if rt, err := reg.Get("site.go.tito.io"); err != nil || rt != nil { |
| + t.Fatalf("pools %v: a static record crossed a pool boundary: %+v %v", cfg.Pools, rt, err) |
| + } |
| + } |
| +} |
| + |
| +// Fleet placement is the other tenancy boundary, and it is refused from BOTH |
| +// sides: a customer fleet's static site is not served by the platform's boxes, |
| +// and a customer's fleet box does not pick up the platform's static records. |
| +// (registry.FleetPlacement is the grammar — "fleet/<name>" in the Region.) |
| +func TestReconcileFleetPlacementIsNeverEverywhere(t *testing.T) { |
| + s, sqid := setup(t) |
| + onFleet := titoRecord(sqid, "pisite") |
| + onFleet.Kind = "static" |
| + onFleet.Region = "fleet/pi" |
| + platform := titoRecord(sqid, "site") |
| + platform.Kind = "static" |
| + for _, rec := range []Record{onFleet, platform} { |
| + if err := CreateRecord(ctx, s, rec); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + reg := regWith(t) |
| + |
| + // A platform box must not serve the fleet's site... |
| + if err := Reconcile(ctx, s, reg, farBoxCfg()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if rt, err := reg.Get("pisite.go.tito.io"); err != nil || rt != nil { |
| + t.Errorf("a fleet-placed static record materialized on a platform box: %+v %v", rt, err) |
| + } |
| + // ...and the fleet's own box must not pick up the platform's. |
| + fleetReg := regWith(t) |
| + if err := Reconcile(ctx, s, fleetReg, ReconcileConfig{Region: "fleet/pi", Pools: []string{"tito"}, DataDir: "/data"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if rt, err := fleetReg.Get("site.go.tito.io"); err != nil || rt != nil { |
| + t.Errorf("a platform static record materialized on a customer fleet box: %+v %v", rt, err) |
| + } |
| + mustGet(t, fleetReg, "pisite.go.tito.io") // its own placement, unchanged |
| +} |
| + |
| +// The other half of adopting everywhere: retiring everywhere. A far box that |
| +// mints an everywhere-class route from a record must remove it when that |
| +// record goes, or the widened mint gate leaves an immortal route serving a |
| +// retired site on every box but its home. |
| +func TestReconcileRemovesEverywhereRouteWhenItsRecordGoes(t *testing.T) { |
| + s, sqid := setup(t) |
| + rec := titoRecord(sqid, "site") |
| + rec.Kind = "static" |
| + if err := CreateRecord(ctx, s, rec); err != nil { |
| + t.Fatal(err) |
| + } |
| + reg := regWith(t) |
| + cfg := farBoxCfg() |
| + if err := Reconcile(ctx, s, reg, cfg); err != nil { |
| + t.Fatal(err) |
| + } |
| + mustGet(t, reg, "site.go.tito.io") |
| + |
| + if err := DeleteRecord(ctx, s, sqid, "tito", "site.go.tito.io"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := Reconcile(ctx, s, reg, cfg); err != nil { |
| + t.Fatal(err) |
| + } |
| + if rt, err := reg.Get("site.go.tito.io"); err != nil || rt != nil { |
| + t.Errorf("a far box's everywhere-class route outlived its record: %+v %v", rt, err) |
| + } |
| +} |
| + |
| // TestRouteForCarriesAddrsAndHealth is Task 3's mint-side proof: a record |
| // with an explicit multi-upstream Addrs set mints a route whose Addrs is the |
| // comma-joined string and whose Health carries a fully-materialized default |