amadan

carlos / platform Public

replication: the rung cannot outlive the sleeping, and the comments stop lying

Review fixes on top of the replication door.

The rail was point-in-time. `carlos route --replication` refuses a route
that does not hibernate — the rung is read on the activator's wake path and
nowhere else — but nothing held that afterwards. Three writers flip
hibernation later: this verb's own `--hibernate=false`, `carlos add` with no
--hibernate, and the provisioning converge reading an instance record. A
`daily` route could become a non-hibernating route that `carlos routes` still
reported as `daily`, while box-wide /etc/litestream.yml replicated its
database at its own interval. That is the lying column the door was built to
end, relocated to a later moment.

Closed as a registry invariant rather than a check in each caller, because
the callers are what keep changing: registry.SetHibernate clears the rung
when hibernation goes off (both narrow-setter doors pass through it), and
Upsert refuses to carry one onto a non-hibernating row on INSERT and on
conflict. A non-empty `replication` now implies a hibernating row, from every
door there is and every door added later. The two commands that spend the
value say so: `carlos route --hibernate=false` names what it cleared and why,
and the converge logs it beside the flip.

The rail also missed half its own premise. ops.FleetDBs excludes a route on
`r.Hibernate && r.DBPath != ""` — hibernation ALONE excludes nothing — so a
hibernating route whose row names no database was swept into box-wide
litestream anyway and a rung on it bought nothing. It needs both halves, and
routeFor's comment already recorded the trap from the other side.

The comments claimed the provisioning reconciler "re-Upserts every owned row
on every pass". It does not: its only reg.Upsert is the mint branch, the
owned branch converges through narrow setters, and the domains reconciler's
two Upserts are mint-only as well. The caller that really Upserts over an
existing row is `carlos add`, which has no replication flag — that is the gap
the preserved column stands in, and it is worth saying truly, because a
reader who checks the old claim finds it false and reverts the protection.

`replication` is box-scoped and inexpressible in a record exactly like
Backing/Tunnel/Caps, and is deliberately NOT added to shapeMismatch. Those
three are refused at adoption because each grants or relocates authority, and
adoption is an authority transfer. A rung grants nothing, and it survives
adoption unchanged. Listing it would mean that tuning a large sleeping
tenant's rung silently opted that route out of the backfill forever — the
opposite trade, stranding precisely the tenants the ladder exists for.

And the wire hop had no assertion: the console emits the field and the CLI
parses it, so the struct field could have been dropped and every route would
have printed `default` with a green suite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XxxmnFjdNxLc3M16ooFG95
Paul Campbell pushed by paul@keymail.dev 9de8ab55834ed012a0ed3a716f3f08329ea46ffc parent 85b7621
8 files changed, +476 −50
  • cli.go +49 −9
  • docs/site/cli.md +14 −5
  • internal/console/apiroutes_test.go +36 −1
  • internal/provision/reconcile.go +45 −2
  • internal/provision/reconcile_test.go +47 −0
  • internal/registry/registry.go +73 −13
  • internal/registry/replication_test.go +95 −12
  • route_replication_test.go +117 −8

CI passed — run details

diff --git a/cli.go b/cli.go
index 6e73df976..8b3d7eb88 100644
--- a/cli.go
+++ b/cli.go
@@ -1348,6 +1348,19 @@ func runRouteHibernate(r *registry.Registry, rt registry.Route, on, dryRun bool)
}
}
fmt.Printf("%s%s: hibernate %v -> %v\n", prefix, rt.Host, rt.Hibernate, on)
+ // Said at the moment of typing, because the write does two things and
+ // only one of them was asked for. registry.SetHibernate clears the
+ // replication class when hibernation goes off — a rung is only read on
+ // the activator's wake path, and a non-hibernating database is covered
+ // by box-wide /etc/litestream.yml at its own interval, so keeping the
+ // column would leave `carlos routes` printing a rung nothing honours.
+ // An operator who turns hibernation back on has to name the rung again,
+ // and being told that here is the difference between a decision and a
+ // discovery.
+ if !on && rt.Replication != "" {
+ fmt.Printf(" replication %s cleared — the rung is read only on the activator's wake path, and this route no longer sleeps.\n", replicationWord(rt.Replication))
+ fmt.Printf(" its database is now covered by box-wide /etc/litestream.yml at that unit's own interval; re-`--hibernate` then set the rung again if you want it back\n")
+ }
}
// runRouteBacking stamps who owns a route's process — the door design §5b
@@ -1543,12 +1556,20 @@ func unitWord(unit string) string {
// or whether it belongs to a plan tier, is a design question #10 filed
// rather than answered. Nothing here forecloses either answer.
//
-// One rail, structural: a route that does not hibernate is refused. The
-// class is read in exactly one place — the activator's wake path — and
-// /etc/litestream.yml deliberately EXCLUDES hibernating databases and covers
-// everything else at its own interval (internal/ops/litestream.go). So on a
-// non-hibernating route the column would be written, reported as set, and
-// read by nothing, which is the same silent no-op the missing door was.
+// Two rails, both structural, and they are one premise read whole. The class
+// is read in exactly one place — the activator's wake path — and
+// /etc/litestream.yml covers everything box-wide litestream is NOT told to
+// leave alone, at its own interval. What it is told to leave alone is
+// `r.Hibernate && r.DBPath != ""` (internal/ops/litestream.go), so BOTH
+// halves are required: a non-hibernating route, and a hibernating route
+// whose row names no database, are each swept into the box-wide config and
+// each would have the column written, reported as set, and read by nothing —
+// the same silent no-op the missing door was.
+//
+// The column also cannot outlive the first half. registry.SetHibernate
+// clears the rung when hibernation goes off, and Upsert refuses to carry one
+// onto a non-hibernating row, so `carlos routes` cannot come to print a rung
+// this rail would no longer let anyone set.
//
// The value's shape is checked at the flag, before the registry is opened.
func runRouteReplication(r *registry.Registry, rt registry.Route, class string, dryRun bool) {
@@ -1576,14 +1597,33 @@ func runRouteReplication(r *registry.Registry, rt registry.Route, class string,
fmt.Printf(" takes effect on this instance's next wake — the sync interval is handed to litestream at wake, so an instance awake now keeps the old one until it sleeps\n")
}
-// replicationRail is runRouteReplication's one refusal, split out as a pure
-// function for the reason unitRail and backingRail are: a rail nobody can
-// exercise without running the process is a rail nobody tests.
+// replicationRail is runRouteReplication's pair of refusals, split out as a
+// pure function for the reason unitRail and backingRail are: a rail nobody
+// can exercise without running the process is a rail nobody tests.
func replicationRail(rt registry.Route) error {
if !rt.Hibernate {
return fmt.Errorf("route does not hibernate — the replication class is read only where the activator replicates, on wake, and /etc/litestream.yml (which covers every NON-hibernating database on this box) has its own interval. "+
"Setting a rung here would write a column nothing reads. Use `carlos route --host %s --hibernate` first if this route should be activator-managed", rt.Host)
}
+ // Hibernation alone was never the premise — the exclusion it stands on
+ // is `r.Hibernate && r.DBPath != ""` (ops.FleetDBsWithSkipped), and
+ // provision.routeFor's comment already records this trap from the other
+ // side. A hibernating route whose row names no database is NOT excluded
+ // from box-wide litestream: the dataDir glob claims its file anyway and
+ // replicates it at that unit's interval, so a rung set here would be
+ // written, reported by `carlos routes`, and honoured by nothing — the
+ // same silent no-op the missing door was, reached through the door.
+ //
+ // Empty db_path on a hibernating row is not hypothetical: `carlos add`
+ // defaults the path only for a single-upstream `instance` that is not
+ // unit-backed, so a static or service route, or an instance record
+ // carrying an upstream SET (provision.Record.DerivesPaths is false
+ // there), hibernates with the column empty.
+ if rt.DBPath == "" {
+ return fmt.Errorf("route hibernates but its row names no database (db_path is empty) — box-wide litestream excludes a hibernating route only when it knows WHERE its database is (`r.Hibernate && r.DBPath != \"\"`, ops.FleetDBs), "+
+ "so this route's file is still being replicated at /etc/litestream.yml's own interval and a rung set here would be honoured by nothing. "+
+ "Give the row its path first (`carlos add --host %s --db <absolute path>`, which rewrites the existing route), then set the rung", rt.Host)
+ }
return nil
}
diff --git a/docs/site/cli.md b/docs/site/cli.md
index 754ee0636..8cee31745 100644
--- a/docs/site/cli.md
+++ b/docs/site/cli.md
@@ -757,11 +757,20 @@ carlos route --host hello.example.com --replication batched
```
`--replication -` puts the route back on the default, which behaves exactly
-like `stream`. It only applies to routes that hibernate: those are the ones
-the activator replicates, and everything else on the box is covered by the
-box's own replicator at its own interval. The command refuses a route that
-does not hibernate rather than writing a setting nothing would read. A change
-takes effect the next time the instance wakes.
+like `stream`. It only applies to routes that hibernate and whose row names a
+database: those are the ones the activator replicates, and everything else on
+the box is covered by the box's own replicator at its own interval. The
+command refuses both cases rather than writing a setting nothing would read —
+a route that does not hibernate, and one that hibernates with no `--db` path
+on it, which the box replicator cannot tell apart from any other file and so
+sweeps up anyway. A change takes effect the next time the instance wakes.
+
+The rung lives only as long as the sleeping does. Stopping a route from
+hibernating clears it — with `carlos route --hibernate=false`, and equally
+when an instance record turns hibernation off and the box converges — because
+from that moment the box replicator owns the database and the rung would name
+an interval nothing honours. The command says so when it clears one. Turning
+hibernation back on starts from the default again.
Repointing the channel is the half to be careful with. Adoption converges a
route onto whatever its channel currently points at, so moving the channel
diff --git a/internal/console/apiroutes_test.go b/internal/console/apiroutes_test.go
index e2a9a6a33..cdc036f45 100644
--- a/internal/console/apiroutes_test.go
+++ b/internal/console/apiroutes_test.go
@@ -38,8 +38,13 @@ func routesFixture(t *testing.T, code string) (*consoleFixture, string) {
// exactly why `carlos instances list` could not answer for it.
{Host: "connect.go.tito.io", Kind: "instance", Status: "ready", App: "titogo",
Account: "bab", Addr: "127.0.0.1:9001", Channel: "canary/relay", Environment: "relay"},
+ // Hibernating, with a database path and a rung on it: the two halves
+ // `carlos route --replication` requires, so this row is shaped the
+ // way a real tuned tenant is.
{Host: "router-a.go.tito.io", Kind: "instance", Status: "ready", App: "titogo",
- Account: "bab", Addr: "127.0.0.1:9002", Channel: "stable"},
+ Account: "bab", Addr: "127.0.0.1:9002", Channel: "stable",
+ Hibernate: true, DBPath: "/data/router-a.go.tito.io.db",
+ Replication: registry.ReplicationDaily},
{Host: "neighbour.example.com", Kind: "instance", Status: "ready", App: "kass",
Account: "bab", Addr: "127.0.0.1:9003"},
{Host: "foreign.example.com", Kind: "instance", Status: "ready", App: "titogo",
@@ -137,6 +142,36 @@ func TestAPIRoutesReportsTheEnvironmentBinding(t *testing.T) {
}
}
+// TestAPIRoutesReportsTheReplicationRung: the rung is set on the box, by an
+// operator, with `carlos route --replication` — and the member reading
+// `carlos routes` is the only one who can tell whether the tenant they are
+// paying for is on the rung somebody intended. The field is `omitempty` on
+// the wire and nothing else in this repo reads it back out of JSON, so an
+// untested hop is one where the struct field could be deleted and every
+// route would print "default" with a green suite.
+func TestAPIRoutesReportsTheReplicationRung(t *testing.T) {
+ fx, token := routesFixture(t, "code-routes-repl")
+ status, out := getJSON(t, routesURL(fx, "titogo"), token)
+ if status != http.StatusOK {
+ t.Fatalf("GET routes = %d %v", status, out)
+ }
+ byHost := hostsOf(t, out)
+ tuned, ok := byHost["router-a.go.tito.io"]
+ if !ok {
+ t.Fatalf("routes = %v, want the tuned route listed", out["routes"])
+ }
+ if got := str(t, tuned, "replication"); got != registry.ReplicationDaily {
+ t.Fatalf("replication = %q, want %q — the rung an operator set is unreadable to the member whose bill it is", got, registry.ReplicationDaily)
+ }
+ // A route on the pre-ladder default reports nothing rather than a rung it
+ // is not on: the field is omitempty, and "" is what every route carried
+ // before the column had a door. The CLI gives that absence its name
+ // ("default", routes_cli.go's replicationCell) — the wire does not.
+ if v, present := byHost["connect.go.tito.io"]["replication"]; present {
+ t.Fatalf("an unset rung was sent as %v, want the field omitted", v)
+ }
+}
+
// TestAPIRoutesListsOnlyThisAppsRoutes: the registry holds every tenant's
// rows, and the tenant boundary is the only thing standing between a
// caller and a neighbour's upstream addresses and unit names.
diff --git a/internal/provision/reconcile.go b/internal/provision/reconcile.go
index 7dcaa45a8..7f620aeae 100644
--- a/internal/provision/reconcile.go
+++ b/internal/provision/reconcile.go
@@ -420,6 +420,11 @@ func shapeMatches(existing, want registry.Route) bool {
// columns there (the original message did app/kind/addr) leaves every
// refusal over any OTHER column looking identical and undiagnosable to the
// operator the refusal message exists for.
+//
+// `replication` is box-scoped and inexpressible in a record exactly like
+// Backing/Tunnel/Caps, and is DELIBERATELY not listed here — see the
+// asymmetry note above the removal loop below for why refusing adoption over
+// it would cost the fleet the backfill and buy nothing.
func shapeMismatch(existing, want registry.Route) string {
switch {
case existing.Backing != "":
@@ -808,13 +813,28 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
convergeOwned(reg, existing, rec, cfg.DryRun)
// Absent means "this record does not speak to hibernation" —
// never a derivation stamped over an operator's own setting.
+ //
+ // Converging hibernation OFF also drops the route's replication
+ // rung, inside registry.SetHibernate — a rung is read only on
+ // the activator's wake path, so on a route the record has just
+ // stopped hibernating it would describe nothing that happens
+ // while `carlos routes` went on printing it. A record cannot
+ // express the rung (it is box-scoped, like caps and tunnel), so
+ // this pass cannot restore one either; the drop is LOGGED with
+ // the flip rather than left for an operator to notice, because
+ // it is the one operator-typed value a converge tick can
+ // silently spend.
if rec.Hibernate != nil && existing.Hibernate != *rec.Hibernate {
+ dropped := ""
+ if !*rec.Hibernate && existing.Replication != "" {
+ dropped = fmt.Sprintf(" (replication %q cleared: only a hibernating route's rung is read)", existing.Replication)
+ }
if cfg.DryRun {
- log.Printf("provision: dry-run: would converge %s hibernate %v -> %v", rec.Host, existing.Hibernate, *rec.Hibernate)
+ log.Printf("provision: dry-run: would converge %s hibernate %v -> %v%s", rec.Host, existing.Hibernate, *rec.Hibernate, dropped)
} else if err := reg.SetHibernate(rec.Host, *rec.Hibernate); err != nil {
log.Printf("provision: converge hibernate on %s: %v", rec.Host, err)
} else {
- log.Printf("provision: converged %s hibernate %v -> %v", rec.Host, existing.Hibernate, *rec.Hibernate)
+ log.Printf("provision: converged %s hibernate %v -> %v%s", rec.Host, existing.Hibernate, *rec.Hibernate, dropped)
}
}
// seedIsKnown guards against stamping a transient manifest-read
@@ -919,6 +939,29 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
// one unattended record-deletion away from registry.Delete's grant purge.
// That asymmetry predates this pass's adoption step; it is stated here so
// the adoption guard is not mistaken for a standing guarantee.
+ //
+ // `replication` (discussion #10) is the same KIND of column — box-scoped,
+ // operator-typed, inexpressible in a record — and is deliberately exempt
+ // from shapeMismatch rather than added to it. The three that are listed
+ // there are listed for a reason that does not hold for this one: each
+ // grants or relocates AUTHORITY over a route's process (a capability, a
+ // process owner, an ingress path), and adoption is itself an authority
+ // transfer, so handing bucket control a row that already carries one is
+ // the thing worth stopping to look at. A replication rung grants nothing.
+ // It names how often a database is copied off the box, and it survives
+ // adoption unchanged: adoption's own write is SetProvisioned, the owned
+ // branch converges through narrow setters that never touch this column,
+ // and the one path that DOES clear it — registry.SetHibernate, when a
+ // record turns hibernation off — is the honest answer rather than a loss,
+ // since a non-hibernating route's rung is read by nobody.
+ //
+ // Listing it would mean an operator who tuned a large sleeping tenant's
+ // rung had, by that act, opted the route out of the backfill this pass
+ // exists to perform — permanently and invisibly, until someone thought to
+ // clear a column to let a record adopt a row. That is the exact opposite
+ // trade from the other three: it would strand precisely the hibernating
+ // tenants the ladder was built for, in exchange for guarding an authority
+ // this column does not carry.
if !readFailed {
for _, rt := range routes {
if !rt.Provisioned || !cfg.inScope(rt.Region, rt.Pool, everywhereRoute(rt)) {
diff --git a/internal/provision/reconcile_test.go b/internal/provision/reconcile_test.go
index b23f39392..d71e9b35d 100644
--- a/internal/provision/reconcile_test.go
+++ b/internal/provision/reconcile_test.go
@@ -602,6 +602,53 @@ func TestReconcileConvergesHibernateOnlyWhenExplicit(t *testing.T) {
}
}
+// Converging hibernation OFF drops the route's replication rung with it.
+// This is the quietest of the three doors that flip the column — no operator
+// types it, a record edit somewhere else does — and it is the one that made
+// `carlos route --replication`'s rail point-in-time: the rung is read only on
+// the activator's wake path, so the moment this pass stops a route sleeping,
+// box-wide /etc/litestream.yml owns that database and a stored `daily` is a
+// column `carlos routes` prints and nothing honours (discussion #10).
+//
+// A record cannot express a rung — it is box-scoped, like caps and tunnel —
+// so this pass can only drop one, never restore it. That is why the drop is
+// logged beside the flip rather than left to be discovered.
+func TestReconcileConvergingHibernateOffClearsTheReplicationRung(t *testing.T) {
+ s, sqid := setup(t)
+ rec := acmeRecord(sqid)
+ on := true
+ rec.Hibernate = &on
+ if err := CreateRecord(ctx, s, rec); err != nil {
+ t.Fatal(err)
+ }
+ reg := regWith(t)
+ cfg := ReconcileConfig{Region: "ie", Pools: []string{"tito"}, DataDir: "/data"}
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if !mustGet(t, reg, "acme.go.tito.io").Hibernate {
+ t.Fatal("setup: the minted route does not hibernate")
+ }
+ // The operator's own write, on the box, through the only door there is.
+ if err := reg.SetReplication("acme.go.tito.io", registry.ReplicationDaily); err != nil {
+ t.Fatal(err)
+ }
+
+ off := false
+ rec.Hibernate = &off
+ putReplace(t, s, RecordKey(sqid, "tito", rec.Host), mustJSON(t, rec))
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ rt := mustGet(t, reg, "acme.go.tito.io")
+ if rt.Hibernate {
+ t.Fatal("setup: hibernate did not converge off")
+ }
+ if rt.Replication != "" {
+ t.Errorf("replication = %q after the record stopped the route hibernating — the column now names an interval nothing reads", rt.Replication)
+ }
+}
+
// StatusNamed is converged, so it carries Hibernate's tri-state for
// Hibernate's reason: explicit true reaches the row, explicit false CLEARS
// it, and absent is not an answer at all.
diff --git a/internal/registry/registry.go b/internal/registry/registry.go
index 5e4dcf188..05ede7c7f 100644
--- a/internal/registry/registry.go
+++ b/internal/registry/registry.go
@@ -760,7 +760,7 @@ func (r *Registry) Delete(host string) error {
// `seed`, `unit`, `fleet_box` and `replication` all keep the existing row's
// value when the incoming Route carries the zero one.
//
-// The reason is the same for all seven, and it is structural: a full-row
+// The reason is the same for all eight, and it is structural: a full-row
// Upsert caller (`carlos add`, the domains reconciler's two mint paths, the
// provision pass's mint) builds its Route from what IT knows, and none of
// them knows about the columns another feature owns. `environment` was the
@@ -785,13 +785,32 @@ func (r *Registry) Delete(host string) error {
// the next promote would stamp X-Carlos-Version onto a console still serving
// the old build, which is issue #110 returning silently.
//
-// `replication` joined them when the ladder got a door (discussion #10). It
-// is the same shape as `environment` and the reason is the same sentence:
-// the provisioning reconciler mints routes from instance records, records
-// carry no replication class, and a plain excluded.replication would return
-// an operator's `daily` route to the every-second default within seconds of
-// the next serial bump — silently, and at roughly 100x the S3 traffic the
-// setting was chosen to avoid.
+// `replication` joined them when the ladder got a door (discussion #10) —
+// the same CASE as `environment`, but NOT, as an earlier draft of this
+// paragraph claimed, because the provisioning reconciler re-Upserts owned
+// rows. It does not: its only reg.Upsert is the mint branch (reconcile.go,
+// `case !ok`), and the owned branch converges through narrow setters
+// (convergeOwned, SetHibernate, SetSeed), exactly as the addrs/health
+// paragraph below already says. The domains reconciler's two Upserts are
+// mint-only too, guarded by byHost/hasApex. A reader who checks the old
+// claim finds it false and reasonably concludes the protection is
+// unnecessary, so it is worth stating what is actually true.
+//
+// The caller that really does Upsert over an EXISTING row is `carlos add`
+// (cli.go's runAdd — it prints "updated <host> (existing route)" when it
+// does). It builds its Route literal from flags alone, and it has no
+// replication flag: without this CASE, a re-add to restamp a unit or
+// repoint an addr would return an operator's `daily` route to the
+// every-second default, silently, at roughly 100x the S3 traffic the
+// setting was chosen to avoid. That is the same gap `environment` was
+// caught in and the same gap Addrs/Health/Steering are closed against at
+// the CLI's own door below.
+//
+// Against the two mint-only reconcilers the CASE is defensive rather than
+// load-bearing, and it stays that way deliberately: the rule for this
+// column is then identical to the seven beside it, so the NEXT full-row
+// Upsert caller — which will know nothing about replication either — is
+// covered before it is written rather than after it is diagnosed.
//
// An explicit non-zero value still wins in every case; clearing keeps
// exactly one door per column, none of which goes through Upsert:
@@ -800,6 +819,13 @@ func (r *Registry) Delete(host string) error {
// -unit -`), SetReplication (`carlos route -replication -`), SetSeed,
// SetFleetBox, and for `provisioned` the row's own deletion.
//
+// `replication` is the one column with a SECOND clearing door, and it is not
+// an operator's: SetHibernate clears it when a route stops hibernating,
+// because a rung is read only on the activator's wake path and describes
+// nothing on a route that no longer sleeps. This statement's own
+// `excluded.hibernate = 0` arm is that same rule reaching the full-row
+// callers. See SetHibernate for the whole of it.
+//
// `addrs` and `health` (multi-upstream routing, issue #152) are, by
// contrast, PLAIN overwrites — see their ON CONFLICT clause below — same as
// `addr`, `status` and the rest of the un-preserved majority. That is safe
@@ -843,6 +869,16 @@ func (r *Registry) Upsert(t Route) error {
if t.Kind == "" {
t.Kind = "instance"
}
+ // The replication invariant, INSERT half (the ON CONFLICT half is the
+ // `excluded.hibernate = 0` arm below): a non-empty class on a row that
+ // does not hibernate is a column that lies. Nothing reads it there —
+ // the activator's wake path is its only reader, and box-wide
+ // /etc/litestream.yml covers every non-hibernating database at its own
+ // interval — so a stored `daily` would be reported by `carlos routes`
+ // while the database was streamed every second. See SetHibernate.
+ if !t.Hibernate {
+ t.Replication = ""
+ }
b2i := func(b bool) int {
if b {
return 1
@@ -865,7 +901,7 @@ ON CONFLICT(host) DO UPDATE SET
dns_checked_at=excluded.dns_checked_at,
cert_expires_at=excluded.cert_expires_at, account=excluded.account,
environment=CASE WHEN excluded.environment = '' THEN routes.environment ELSE excluded.environment END,
- replication=CASE WHEN excluded.replication = '' THEN routes.replication ELSE excluded.replication END,
+ replication=CASE WHEN excluded.hibernate = 0 THEN '' WHEN excluded.replication = '' THEN routes.replication ELSE excluded.replication END,
caps=CASE WHEN excluded.caps = '' THEN routes.caps ELSE excluded.caps END,
tunnel=CASE WHEN excluded.tunnel = '' THEN routes.tunnel ELSE excluded.tunnel END,
provisioned=CASE WHEN excluded.provisioned = 0 THEN routes.provisioned ELSE excluded.provisioned END,
@@ -1471,12 +1507,36 @@ func (r *Registry) SetVersions(host, ver, prev string) error {
}
// SetHibernate flips the per-host hibernation opt-in as a narrow UPDATE.
+// Turning hibernation OFF also clears `replication`, and that is the point
+// of doing it here rather than in each caller. The replication class is read
+// in exactly one place — the activator's wake path — and box-wide
+// /etc/litestream.yml deliberately covers every NON-hibernating database at
+// its own interval (internal/ops/litestream.go, ops.FleetDBsWithSkipped).
+// So the moment a route stops hibernating, a stored `daily` describes
+// nothing that happens: `carlos routes` would keep printing it while
+// litestream streamed the database every second, which is the lying column
+// the `carlos route --replication` rail (cli.go's replicationRail) exists to
+// prevent, merely relocated to a later moment.
+//
+// Both doors that turn hibernation off come through here — `carlos route
+// --hibernate=false` (cli.go's runRouteHibernate) and the provisioning
+// converge, which flips the column from the instance record on every pass —
+// and any door added later will too. The third writer, a full-row Upsert
+// carrying hibernate=0 (`carlos add` with no --hibernate), is covered by the
+// matching arm of that statement's ON CONFLICT clause. Between them the
+// invariant is total: a non-empty `replication` implies a hibernating row.
+//
+// Clearing rather than preserving-for-later is deliberate. A preserved class
+// is invisible state that changes behaviour weeks after it was typed, and
+// the operator turning hibernation back on is already at a prompt where
+// naming the rung again costs one command; runRouteHibernate prints what was
+// dropped so it is a fact they are told, not one they discover.
func (r *Registry) SetHibernate(host string, on bool) error {
- v := 0
- if on {
- v = 1
+ if !on {
+ _, err := r.sql.Exec(`UPDATE routes SET hibernate = 0, replication = '' WHERE host = ?`, host)
+ return err
}
- _, err := r.sql.Exec(`UPDATE routes SET hibernate = ? WHERE host = ?`, v, host)
+ _, err := r.sql.Exec(`UPDATE routes SET hibernate = 1 WHERE host = ?`, host)
return err
}
diff --git a/internal/registry/replication_test.go b/internal/registry/replication_test.go
index a904aa9aa..ca459dbb1 100644
--- a/internal/registry/replication_test.go
+++ b/internal/registry/replication_test.go
@@ -6,12 +6,22 @@ import (
)
// TestUpsertPreservesReplication is the fence that makes the new door worth
-// anything. The provisioning reconciler mints routes from instance records,
-// records carry no replication class, and it re-Upserts every owned row on
-// every pass. With a plain excluded.replication an operator's `daily` route
-// would be back on the every-second default within seconds of the next serial
-// bump — silently, and at roughly 100x the S3 traffic the setting was chosen
-// to avoid (discussion #10). Same shape, same reason, as `environment`.
+// anything. The caller it stands against is `carlos add` re-run over a host
+// that already exists (cli.go's runAdd, which prints "updated ... (existing
+// route)"): it builds its Route literal from flags alone, and it has no
+// replication flag, so with a plain excluded.replication a re-add to restamp
+// a unit or repoint an addr would return an operator's `daily` route to the
+// every-second default — silently, and at roughly 100x the S3 traffic the
+// setting was chosen to avoid (discussion #10). Same shape, same reason, as
+// `environment`.
+//
+// NOT the provisioning reconciler, whatever the first draft of this comment
+// said: its only reg.Upsert is the mint branch (reconcile.go, `case !ok`,
+// where there is no existing row to clobber), and the owned branch converges
+// through narrow setters. The domains reconciler's two Upserts are mint-only
+// as well. The re-Upsert below is therefore a stand-in for `carlos add`'s
+// shape — a full-row write that knows nothing about this column — which is
+// the shape any future full-row caller will have too.
func TestUpsertPreservesReplication(t *testing.T) {
r := openTestRegistry(t)
host := "seapointish.example.com"
@@ -22,8 +32,8 @@ func TestUpsertPreservesReplication(t *testing.T) {
t.Fatalf("SetReplication: %v", err)
}
- // The reconciler's mint: a full-row Upsert built from an instance record,
- // which knows nothing about this column.
+ // A full-row Upsert built by a caller that knows nothing about this
+ // column — `carlos add`'s shape, and the reconciler's mint's shape.
if err := r.Upsert(Route{Host: host, Kind: "instance", App: "seapointish", Hibernate: true, Channel: "edge"}); err != nil {
t.Fatalf("re-upsert: %v", err)
}
@@ -32,22 +42,24 @@ func TestUpsertPreservesReplication(t *testing.T) {
t.Fatalf("get: %+v %v", rt, err)
}
if rt.Replication != ReplicationDaily {
- t.Fatalf("replication = %q after a re-mint, want %q — a reconcile pass silently returned the route to the most expensive rung", rt.Replication, ReplicationDaily)
+ t.Fatalf("replication = %q after a full-row re-Upsert, want %q — a re-add silently returned the route to the most expensive rung", rt.Replication, ReplicationDaily)
}
if rt.Channel != "edge" {
- t.Fatalf("channel = %q, want the re-mint's own value: preserving replication must not freeze the columns the reconciler DOES own", rt.Channel)
+ t.Fatalf("channel = %q, want the re-Upsert's own value: preserving replication must not freeze the columns a full-row caller DOES own", rt.Channel)
}
}
// TestUpsertHonoursAnExplicitReplication: preservation is for the zero value
// only. A caller that carries a class still wins, exactly as with environment.
+// Both rows hibernate, because a rung on a row that does not is a state this
+// table refuses to hold — see TestReplicationImpliesHibernation.
func TestUpsertHonoursAnExplicitReplication(t *testing.T) {
r := openTestRegistry(t)
host := "a.example.com"
- if err := r.Upsert(Route{Host: host, Replication: ReplicationDaily}); err != nil {
+ if err := r.Upsert(Route{Host: host, Hibernate: true, Replication: ReplicationDaily}); err != nil {
t.Fatalf("upsert: %v", err)
}
- if err := r.Upsert(Route{Host: host, Replication: ReplicationBatched}); err != nil {
+ if err := r.Upsert(Route{Host: host, Hibernate: true, Replication: ReplicationBatched}); err != nil {
t.Fatalf("re-upsert: %v", err)
}
rt, _ := r.Get(host)
@@ -56,6 +68,77 @@ func TestUpsertHonoursAnExplicitReplication(t *testing.T) {
}
}
+// TestReplicationImpliesHibernation is the invariant that stops the column
+// from lying after the fact. `carlos route --replication` refuses a route
+// that does not hibernate, because the rung is read on the activator's wake
+// path and nowhere else — but that rail is point-in-time, and hibernation is
+// flipped afterwards by three writers: `carlos route --hibernate=false`, the
+// provisioning converge reading an instance record, and a full-row Upsert
+// from `carlos add`. Without this invariant a `daily` route becomes a
+// non-hibernating route that `carlos routes` still reports as `daily`, while
+// box-wide /etc/litestream.yml replicates its database at its own interval:
+// the lying column the rail exists to prevent, relocated to a later moment.
+//
+// Both halves are pinned here because they are two different statements —
+// one in SetHibernate's UPDATE, one in Upsert's INSERT and ON CONFLICT — and
+// a fix to either alone leaves a door open.
+func TestReplicationImpliesHibernation(t *testing.T) {
+ r := openTestRegistry(t)
+
+ // The narrow setter: `carlos route --hibernate=false` and the converge
+ // pass both arrive here.
+ host := "sleeper.example.com"
+ if err := r.Upsert(Route{Host: host, Kind: "instance", App: "sleeper", Hibernate: true,
+ DBPath: "/data/sleeper.example.com.db"}); err != nil {
+ t.Fatalf("upsert: %v", err)
+ }
+ if err := r.SetReplication(host, ReplicationDaily); err != nil {
+ t.Fatalf("SetReplication: %v", err)
+ }
+ if err := r.SetHibernate(host, false); err != nil {
+ t.Fatalf("SetHibernate off: %v", err)
+ }
+ rt, err := r.Get(host)
+ if err != nil || rt == nil {
+ t.Fatalf("get: %+v %v", rt, err)
+ }
+ if rt.Replication != "" {
+ t.Fatalf("replication = %q on a route that stopped hibernating — box-wide litestream owns this database now, and the column claims otherwise", rt.Replication)
+ }
+ // Turning hibernation back ON does not resurrect the rung: a preserved
+ // class is invisible state that changes behaviour weeks later, and the
+ // operator is at a prompt where naming it again costs one command.
+ if err := r.SetHibernate(host, true); err != nil {
+ t.Fatalf("SetHibernate on: %v", err)
+ }
+ if rt, _ = r.Get(host); rt.Replication != "" {
+ t.Fatalf("re-hibernating restored %q — the rung is set, not remembered", rt.Replication)
+ }
+
+ // The full-row write: `carlos add` with no --hibernate over an existing
+ // tuned row. It turns hibernation off (hibernate is a plain overwrite),
+ // so the preserved-when-empty CASE must not hand the new row a rung.
+ if err := r.SetReplication(host, ReplicationBatched); err != nil {
+ t.Fatalf("SetReplication batched: %v", err)
+ }
+ if err := r.Upsert(Route{Host: host, Kind: "instance", App: "sleeper",
+ DBPath: "/data/sleeper.example.com.db"}); err != nil {
+ t.Fatalf("re-upsert awake: %v", err)
+ }
+ if rt, _ = r.Get(host); rt.Replication != "" {
+ t.Fatalf("replication = %q after an Upsert that turned hibernation off", rt.Replication)
+ }
+
+ // And on INSERT, where there is no ON CONFLICT clause to do it.
+ fresh := "awake.example.com"
+ if err := r.Upsert(Route{Host: fresh, Kind: "instance", App: "awake", Replication: ReplicationDaily}); err != nil {
+ t.Fatalf("insert: %v", err)
+ }
+ if rt, _ = r.Get(fresh); rt.Replication != "" {
+ t.Fatalf("a fresh non-hibernating row was inserted carrying %q", rt.Replication)
+ }
+}
+
// TestSetReplicationValidates: the value reaches internal/activator's
// replicationSyncInterval, which answers anything it does not recognise with
// litestream's default — so an unvalidated write is a route that quietly
diff --git a/route_replication_test.go b/route_replication_test.go
index 3837f3e04..ec6fc74e0 100644
--- a/route_replication_test.go
+++ b/route_replication_test.go
@@ -14,17 +14,17 @@ import (
"github.com/carlosframework/platform/internal/registry"
)
-// TestReplicationRailRefusesNonHibernatingRoutes pins the one refusal this
-// verb has, as a pure function for the reason unitRail and backingRail are:
-// the class is read in exactly one place — the activator's wake path — and
-// /etc/litestream.yml deliberately EXCLUDES hibernating databases and covers
-// everything else at its own interval. On a route that does not hibernate the
-// column would be written, reported as set, and read by nothing, which is the
-// same silent no-op the missing door was.
+// TestReplicationRailRefusesNonHibernatingRoutes pins the first of this
+// verb's two refusals, as a pure function for the reason unitRail and
+// backingRail are: the class is read in exactly one place — the activator's
+// wake path — and /etc/litestream.yml covers everything the box-wide
+// replicator is not told to leave alone, at its own interval. On a route that
+// does not hibernate the column would be written, reported as set, and read
+// by nothing, which is the same silent no-op the missing door was.
func TestReplicationRailRefusesNonHibernatingRoutes(t *testing.T) {
awake := registry.Route{
Host: "console.carloku.com", Kind: "instance", App: "console",
- Addr: "127.0.0.1:8090",
+ Addr: "127.0.0.1:8090", DBPath: "/data/console.carloku.com.db",
}
err := replicationRail(awake)
if err == nil {
@@ -43,6 +43,36 @@ func TestReplicationRailRefusesNonHibernatingRoutes(t *testing.T) {
}
}
+// TestReplicationRailRefusesAHibernatingRouteWithNoDatabase pins the second
+// half of the same premise, which the first version of this rail missed.
+// Box-wide litestream excludes a route on `r.Hibernate && r.DBPath != ""`
+// (ops.FleetDBsWithSkipped) — hibernation ALONE excludes nothing. A
+// hibernating route whose row names no database is claimed by the dataDir
+// glob like any other file and replicated at /etc/litestream.yml's interval,
+// so a rung set on it would be written, reported by `carlos routes`, and
+// honoured by nobody: the exact failure the rail exists to refuse, reached
+// through the rail.
+//
+// Not hypothetical: `carlos add` defaults db_path only for a single-upstream
+// non-unit `instance`, so a static or service route, or an instance record
+// carrying an upstream set (provision.Record.DerivesPaths is false there),
+// hibernates with the column empty.
+func TestReplicationRailRefusesAHibernatingRouteWithNoDatabase(t *testing.T) {
+ pathless := registry.Route{
+ Host: "static.example.com", Kind: "static", App: "site",
+ Hibernate: true,
+ }
+ err := replicationRail(pathless)
+ if err == nil {
+ t.Fatal("replicationRail(hibernating route with no db_path) accepted; want a refusal")
+ }
+ for _, want := range []string{"db_path", "--db"} {
+ if !strings.Contains(err.Error(), want) {
+ t.Errorf("refusal %q does not mention %q — the operator has to be told what is missing and which door supplies it", err, want)
+ }
+ }
+}
+
// TestReplicationWordNamesTheDefault: "" is this column's value on every
// route that predates the door, it behaves exactly like stream, and printing
// it as an empty string reads as a bug rather than as the answer it is.
@@ -222,3 +252,82 @@ func TestRouteReplicationRefusesCombinationWithOtherAxes(t *testing.T) {
t.Errorf("route --app --replication: want a refusal, got 0\n%s", out)
}
}
+
+// TestRouteHibernateOffClearsTheRung: the rail is point-in-time, and until
+// this landed nothing held it afterwards. A route set to `daily` could stop
+// hibernating — by this verb, by `carlos add` with no --hibernate, or by a
+// converge pass reading an instance record — and `carlos routes` went on
+// printing `daily` while box-wide /etc/litestream.yml replicated the database
+// at its own interval. A column that lies is the failure the door was built
+// to end, so it must not survive the flip that makes it false.
+//
+// The clearing itself lives in registry.SetHibernate, where every door that
+// turns hibernation off passes; what is pinned here is the operator-facing
+// half — the flip works through the real binary, and it SAYS what it spent,
+// because the rung is the one operator-typed value this command drops
+// without being asked to.
+func TestRouteHibernateOffClearsTheRung(t *testing.T) {
+ bin := buildCarlos(t)
+ tmp := t.TempDir()
+ scratch := t.TempDir()
+ regPath := filepath.Join(tmp, "registry.db")
+
+ run := func(args ...string) (string, error) {
+ t.Helper()
+ cmd := exec.Command(bin, args...)
+ cmd.Env = carlosTestEnv(t)
+ cmd.Dir = scratch
+ out, err := cmd.CombinedOutput()
+ return string(out), err
+ }
+ rowOf := func(host string) (string, bool) {
+ t.Helper()
+ reg, err := registry.Open(regPath)
+ if err != nil {
+ t.Fatalf("open registry: %v", err)
+ }
+ defer reg.Close()
+ rt, err := reg.Get(host)
+ if err != nil || rt == nil {
+ t.Fatalf("get %s: %+v, %v", host, rt, err)
+ }
+ return rt.Replication, rt.Hibernate
+ }
+
+ host := "seapointish.example.com"
+ if out, err := run("add", "-registry", regPath, "-app", "seapointish",
+ "-socket", filepath.Join(tmp, "s.sock"), "-hibernate", host); err != nil {
+ t.Fatalf("add route: %v\n%s", err, out)
+ }
+ if out, err := run("route", "-registry", regPath, "-host", host, "-replication", "daily"); err != nil {
+ t.Fatalf("route --replication daily: %v\n%s", err, out)
+ }
+ if class, _ := rowOf(host); class != registry.ReplicationDaily {
+ t.Fatalf("setup: replication = %q, want %q", class, registry.ReplicationDaily)
+ }
+
+ out, err := run("route", "-registry", regPath, "-host", host, "-hibernate=false")
+ if err != nil {
+ t.Fatalf("route --hibernate=false: %v\n%s", err, out)
+ }
+ class, hib := rowOf(host)
+ if hib {
+ t.Fatalf("setup: the route still hibernates\n%s", out)
+ }
+ if class != "" {
+ t.Fatalf("replication = %q on a route that no longer hibernates — `carlos routes` would print a rung the box replicator does not honour", class)
+ }
+ // Told, not discovered.
+ for _, want := range []string{"replication", "cleared"} {
+ if !strings.Contains(out, want) {
+ t.Errorf("--hibernate=false output does not mention %q:\n%s", want, out)
+ }
+ }
+
+ // And the door is shut behind it: the rung cannot be re-set while the
+ // route is awake, which is what makes the clear an honest state rather
+ // than a gap between two commands.
+ if out, err := run("route", "-registry", regPath, "-host", host, "-replication", "daily"); err == nil {
+ t.Errorf("route --replication on a woken route: want a refusal, got 0\n%s", out)
+ }
+}