amadan

carlos / platform Public

durability: drill the replicas nobody drilled, check what we restore

The question was "would Turso be a good fit?", reframed as "how are we
defending against corrupted SQLite?". The answer to the first is no (recorded
in the design note); the second found three holes.

1. The activator-owned replicas were never restore-drilled. FleetDBs excludes
   hibernating exec routes from /etc/litestream.yml — correctly, since a
   second replicator on the same WAL corrupts the replica — so the fleet's
   LARGEST set of app databases belonged to neither pass. ExecBackend.Restore
   never integrity-checked what it pulled back either, and a corrupt restored
   file was left where the next wake's warm-cache branch would boot it
   unchecked.

   - lease.CheckDB: the integrity check moves beside Restore so the wake path
     and the drill share one definition of "restores clean";
     ops.CheckRestoredDB delegates to it.
   - The wake proves the restore before anything serves it. On failure: fresh
     stays false (a fresh verdict would run the route's seed OVER recoverable
     data), the file is quarantined at <db>.corrupt, and the wake fails loudly.
   - carlos ops hibernate-verify + its unit/timer: the same drill for the set
     the config omits, restoring each replica into a throwaway copy. It takes
     no lease and needs none — what the lease forbids is a second WRITER, and
     the box-wide drill has always restored databases its own replicator was
     streaming. Capped (-max 25) and ordered stalest-first, so a large fleet is
     covered over days rather than in one unbounded nightly bill.
   - control/restore-verify.json (internal/dbverify): the verdicts, published
     to the deployment bucket so the result is readable without a session on
     the box. Three-valued after usage.StoreUnswept — ok, failed, or
     UNVERIFIED, never a silent absence. Unvisited hosts keep their old verdict
     AND its old timestamp; retired hosts are dropped. Exit status carries
     corruption only: an unreachable replica must not train an operator to
     ignore a red run.

2. console.db sat one directory below FleetDBs' flat <data>/*.db glob, so the
   deployment's identity database — every account, ownership row, session and
   the Keymail credentials — was replicated only if its route happened to be a
   BackingUnit row carrying -db. Included explicitly now, via CONSOLE_DB or
   <data>/console/console.db.

3. The recovery window was a library default. litestream 0.5's
   DefaultSnapshotRetention is 24h and nothing overrode it, so a bad migration
   noticed on Tuesday was unrecoverable if it landed on Sunday. The generated
   config now states the policy: 24h snapshots, retained 7 days.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Paul Campbell d573678a06628bd6d5ee6d0fbfc9e5640a847c53 parent 8371cf8
16 files changed, +1487 −35
  • docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md +131 −0
  • internal/activator/activator.go +10 −1
  • internal/activator/activator_test.go +70 −1
  • internal/activator/backend_exec.go +35 −0
  • internal/dbverify/dbverify.go +151 −0
  • internal/dbverify/dbverify_test.go +124 −0
  • internal/host/host_test.go +2 −0
  • internal/host/units/carlos-hibernate-verify.service +46 −0
  • internal/host/units/carlos-hibernate-verify.timer +15 −0
  • internal/lease/check.go +100 −0
  • internal/lease/check_test.go +192 −0
  • internal/ops/hibernate_verify.go +132 −0
  • internal/ops/hibernate_verify_test.go +187 −0
  • internal/ops/litestream.go +59 −33
  • internal/ops/ops.go +146 −0
  • internal/ops/ops_test.go +87 −0
diff --git a/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md b/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md
new file mode 100644
index 000000000..08608c091
--- /dev/null
+++ b/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md
@@ -0,0 +1,131 @@
+# SQLite corruption defence — design
+
+Status: built 2026-08-15. Origin: "would Turso be a good fit for CARLOS?",
+reframed by Paul as "how are we defending against corrupted SQLite?".
+
+## 1. What "corrupted" means here
+
+Three failure modes, and they want different defences:
+
+1. **Physical corruption** — torn pages, a bad disk, two writers on one WAL, a
+ kill -9 mid-checkpoint. `PRAGMA integrity_check` sees it.
+2. **Logical corruption** — a bad migration, an app writing nonsense, an
+ errant DELETE. `integrity_check` says `ok`; the data is simply wrong. Only
+ an OLDER COPY helps.
+3. **Loss** — the box dies, the file is gone. Replication helps.
+
+Replication alone defends against (3) and, faithfully, replicates (2).
+
+## 2. Posture before this branch
+
+| Database | Defence |
+| --- | --- |
+| `registry.db` | WAL + `busy_timeout` + `synchronous=NORMAL`; `VACUUM INTO` snapshot to the deployment bucket on every fingerprint change, 8 generations (`internal/host/snapshot_loop.go`); box-wide litestream |
+| Tenant/box DBs under `<data>/*.db`, and `BackingUnit` routes' `-db` paths | box-wide litestream + **`carlos ops restore-verify`** daily: a real restore from S3, then `integrity_check` and a non-empty-schema check |
+| Hibernating exec tenants | embedded litestream under the **S3 lease** — single-writer lock, typed `*HeldError` on a second claimant; deterministic checkpoint at sleep; stale `-wal`/`-shm`/meta dirs cleared before restore |
+
+The strongest part of that is the daily drill: not "we have backups" but "we
+prove they restore". The lease is the strongest part of the *prevention* side
+— the classic two-writers-one-WAL corruption cannot happen while it holds.
+
+### The three holes it left
+
+1. **The activator-owned replicas were never drilled.** `FleetDBs` excludes
+ hibernating exec routes from `/etc/litestream.yml` — correctly, since a
+ second replicator on the same WAL corrupts the replica (Eleven's lesson).
+ The consequence was that the fleet's LARGEST set of app databases belonged
+ to neither pass, and `ExecBackend.Restore` never integrity-checked what it
+ pulled back. The first thing to learn whether a hibernating tenant's
+ replica restores clean was a real user's wake — and a corrupt restored file
+ was then left on disk, where the next wake's warm-cache branch would boot
+ it with no check at all.
+2. **`console.db` was outside the glob.** It lives at
+ `/data/console/console.db`, one directory below `FleetDBs`' flat
+ `<data>/*.db` glob, so the deployment's identity database — every account,
+ ownership row, session, and the Keymail credentials — was replicated only
+ if its route happened to be a `BackingUnit` row carrying `-db`.
+3. **The recovery window was a library default.** litestream 0.5's
+ `DefaultSnapshotRetention` is **24h**, and neither the generated config nor
+ the activator's embedded store overrode it. Against failure mode (2) that
+ is close to no defence: a bad migration noticed on Tuesday is unrecoverable
+ if it landed on Sunday.
+
+## 3. What this branch changed
+
+- **`lease.CheckDB`** — the integrity check moved beside `Restore` so both the
+ wake path and the drill run one definition of "restores clean".
+ `ops.CheckRestoredDB` delegates to it.
+- **Wake-path check** — `ExecBackend.Restore` proves a restored replica before
+ anything serves it. A failure returns `fresh == false` (a fresh verdict would
+ run the route's seed OVER recoverable data), quarantines the file at
+ `<db>.corrupt` so the warm-cache branch cannot boot it next time, and fails
+ the wake loudly. A down route beats a corrupt one.
+- **`carlos ops hibernate-verify`** — the drill for the set the config omits.
+ It restores each replica into a throwaway file and integrity-checks it. It
+ takes NO lease and needs none: what the lease forbids is a second *writer*,
+ and the box-wide drill has always restored databases its own replicator was
+ actively streaming. Capped (`-max`, default 25) and ordered stalest-first, so
+ a large fleet is covered over days rather than in one unbounded nightly bill.
+- **`control/restore-verify.json`** (`internal/dbverify`) — the verdicts,
+ published to the deployment bucket so the result is readable without a
+ session on the box. Three-valued, following `usage.AppStorage.StoreUnswept`:
+ `ok`, failed, or **`unverified`** — never a silent absence, never a 0 that
+ reads as fine. Unvisited hosts keep their old verdict AND its old timestamp
+ (staleness must stay visible); retired hosts are dropped (a dead instance's
+ last "ok" must not pad coverage).
+- **Exit status carries corruption only.** A damaged replica fails the unit; a
+ replica the pass could not reach is `unverified` and does not. Conflating
+ them would either cry wolf on every S3 blip or train an operator to ignore a
+ red run.
+- **`console.db` joins the replication set**, via `CONSOLE_DB` or
+ `<data>/console/console.db`.
+- **Snapshot policy is stated, not inherited**: `SnapshotInterval` 24h,
+ `SnapshotRetention` **7 days**, written into the generated config. Seven
+ daily snapshots of megabyte-scale files against S3 storage at ~$0.023/GB-mo
+ is noise next to the request bill this fleet watches; the week is the
+ deliberate answer to "we noticed the bad migration on Thursday".
+
+## 4. Why not Turso
+
+Turso is a good product aimed at a different shape. Its answer to corruption
+is page-level sync to its cloud (the production engine is still the libSQL C
+fork; the Rust rewrite was beta as of mid-2026). It is no better than
+litestream generations against logical corruption, and it cannot fix
+verification coverage — that was ours to fix, and it is what actually
+mattered.
+
+Against that it would have cost:
+
+- a third-party dependency in every tenant's **wake path**;
+- "databases never sleep" against hibernation economics — €5/yr spaces versus
+ $4.99–$24.92/mo org tiers with per-GB sync overage, and no per-tenant
+ metering to bill through;
+- customer data off the customer's own box and account, which is the Carloku
+ ladder's proposition;
+- a second control plane with its own tokens and DB-creation API to converge,
+ against bucket-is-the-control-plane;
+- the embedded Go path (`tursogo`: purego + prebuilt native libraries)
+ displacing a pure-Go `modernc` binary that CI cross-compiles to darwin.
+
+Self-hosted `sqld` keeps the data local but puts a beta daemon in front of
+files that today need no process, and gains nothing on corruption.
+
+Revisit only if the shape changes: multi-region reads of one tenant database,
+or multi-writer becoming a real constraint rather than a deliberate one.
+
+## 5. Known follow-ups
+
+- **Console surface.** `control/restore-verify.json` is read from the bucket
+ today, like `control/registry-snapshot.json`. A console/API view of "last
+ proven restorable" per instance is the obvious next step.
+- **Fleet-box publication.** `hibernate-verify` resolves the deployment store
+ from `CARLOS_DEPLOYMENT_BUCKET`/`_DIR` only. A fleet box that reaches its
+ deployment through the console door publishes nothing — and says so loudly
+ rather than quietly.
+- **The `replicas:` key is deprecated** in litestream 0.5 (`replica:` is the
+ singular successor; both parse today). Deliberately not changed here — a
+ config-shape change to live replication does not belong in the same branch
+ as a durability fix.
+- **Unit tenants without `-db`** remain invisible to every pass, by design
+ (the platform enforces no filename convention inside a unit tenant's state
+ dir). `cli.go` requires `-db` for hibernating unit routes for this reason.
diff --git a/internal/activator/activator.go b/internal/activator/activator.go
index ce345e4ae..f44b976fc 100644
--- a/internal/activator/activator.go
+++ b/internal/activator/activator.go
@@ -102,7 +102,16 @@ func ConfigFromEnv() (*Config, error) {
// hibernatePrefix is the per-instance S3 key prefix: <Prefix>/<host>/ (the
// lease's lock.json is colocated under it by the lease package).
func (c Config) hibernatePrefix(host string) string {
- return path.Join(c.Prefix, host)
+ return HibernatePrefix(c.Prefix, host)
+}
+
+// HibernatePrefix is that layout as a free function, exported for the
+// verification pass (internal/ops) that restore-drills these replicas. It is
+// a function rather than a comment because a drill that guessed the prefix
+// wrong would report "no replica" for a perfectly good backup — coverage
+// theatre of exactly the kind the drill exists to end.
+func HibernatePrefix(prefix, host string) string {
+ return path.Join(prefix, host)
}
// mesh seam: Eleven also derived meshPrefix() here — <Prefix>/mesh/members,
diff --git a/internal/activator/activator_test.go b/internal/activator/activator_test.go
index 1416eb235..86d696090 100644
--- a/internal/activator/activator_test.go
+++ b/internal/activator/activator_test.go
@@ -11,6 +11,7 @@ package activator
import (
"context"
+ "database/sql"
"errors"
"fmt"
"io"
@@ -30,6 +31,8 @@ import (
"github.com/carlosframework/platform/internal/edge"
"github.com/carlosframework/platform/internal/registry"
+
+ _ "modernc.org/sqlite"
)
// --- fakes -------------------------------------------------------------------
@@ -2788,7 +2791,8 @@ func TestExecBackendRestoreWarmCacheIsNotFresh(t *testing.T) {
// restore, not a first boot.
func TestExecBackendRestoreFromReplicaIsNotFresh(t *testing.T) {
b := testExecBackend(t)
- swapRestoreReplica(t, func(context.Context, string, string, string, string) (bool, error) {
+ swapRestoreReplica(t, func(_ context.Context, dbPath, _, _, _ string) (bool, error) {
+ writeUsableDB(t, dbPath)
return true, nil
})
@@ -2801,6 +2805,71 @@ func TestExecBackendRestoreFromReplicaIsNotFresh(t *testing.T) {
}
}
+// A replica that comes back DAMAGED must fail the wake, not boot. Two things
+// are on the line: `fresh` must stay false (a fresh verdict runs the route's
+// seed, writing an empty database over recoverable data), and the bad file
+// must not be left where the next wake's warm-cache branch will serve it
+// unchecked.
+func TestExecBackendRestoreCorruptReplicaFailsAndQuarantines(t *testing.T) {
+ b := testExecBackend(t)
+ host := "damaged.example.com"
+ dbPath := b.defaultDBPath(host)
+
+ swapRestoreReplica(t, func(_ context.Context, out, _, _, _ string) (bool, error) {
+ if err := os.WriteFile(out, []byte("this is not a sqlite database at all"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ return true, nil
+ })
+
+ fresh, err := b.Restore(context.Background(), registry.Route{Host: host, Kind: "instance"})
+ if err == nil {
+ t.Fatal("Restore accepted a corrupt replica")
+ }
+ if fresh {
+ t.Fatal("a corrupt restore reported fresh — the seed would run over recoverable data")
+ }
+ if _, statErr := os.Stat(dbPath); statErr == nil {
+ t.Fatal("the corrupt file was left at the live path — the next wake would serve it as a warm cache")
+ }
+ if _, statErr := os.Stat(dbPath + quarantineSuffix); statErr != nil {
+ t.Fatalf("no quarantined copy for the operator to look at: %v", statErr)
+ }
+
+ // And the next wake must genuinely re-attempt the restore rather than
+ // short-circuit on leftover state.
+ called := false
+ swapRestoreReplica(t, func(_ context.Context, out, _, _, _ string) (bool, error) {
+ called = true
+ writeUsableDB(t, out)
+ return true, nil
+ })
+ fresh, err = b.Restore(context.Background(), registry.Route{Host: host, Kind: "instance"})
+ if err != nil {
+ t.Fatalf("second Restore: %v", err)
+ }
+ if !called {
+ t.Fatal("the second wake never consulted the bucket")
+ }
+ if fresh {
+ t.Fatal("the recovered instance reported fresh")
+ }
+}
+
+// writeUsableDB creates the kind of file a healthy restore produces: a real
+// SQLite database with a schema, which is what CheckDB demands.
+func writeUsableDB(t *testing.T, path string) {
+ t.Helper()
+ db, err := sql.Open("sqlite", path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer db.Close()
+ if _, err := db.Exec("CREATE TABLE t (id INTEGER PRIMARY KEY); INSERT INTO t VALUES (1)"); err != nil {
+ t.Fatal(err)
+ }
+}
+
// DB absent AND no replica in the bucket: the one true first boot.
func TestExecBackendRestoreNoReplicaIsFresh(t *testing.T) {
b := testExecBackend(t)
diff --git a/internal/activator/backend_exec.go b/internal/activator/backend_exec.go
index 22c8d30ce..e0679120e 100644
--- a/internal/activator/backend_exec.go
+++ b/internal/activator/backend_exec.go
@@ -32,6 +32,12 @@ import (
// socket and answer. A var so tests can shrink the failure path.
var spawnHealthTimeout = 30 * time.Second
+// quarantineSuffix names where a restore that failed its integrity check is
+// parked. It deliberately does NOT end in ".db": the box-wide litestream
+// generator globs <data>/*.db, and a quarantined corpse must never be picked
+// up for replication.
+const quarantineSuffix = ".corrupt"
+
// ExecBackend is the production Backend: leases + restore via the lease
// package's litestream primitives, processes via exec of the same versioned
// binary a systemd unit would run — `<OptDir>/live/<host>` with the
@@ -194,6 +200,35 @@ func (b *ExecBackend) Restore(ctx context.Context, rt registry.Route) (bool, err
return false, err
}
if restored {
+ // Prove the restored file before anything serves from it. A replica
+ // can come back damaged — a torn page, a truncated object, a lineage
+ // that never had a snapshot — and the alternatives to failing here are
+ // both worse than a down route: serving a corrupt database, or (if
+ // this returned fresh) running the route's seed over recoverable data.
+ if err := lease.CheckDB(dbPath); err != nil {
+ q := dbPath + quarantineSuffix
+ _ = os.Remove(q) // keep exactly one: the newest failure is the evidence
+ if mvErr := os.Rename(dbPath, q); mvErr != nil {
+ // Quarantine failed, so the live path may still hold the bad
+ // file — remove it rather than leave a warm cache that the
+ // next wake would boot with no check at all.
+ _ = os.Remove(dbPath)
+ log.Printf("activator: %s restored CORRUPT (%v) and could not be quarantined (%v) — removed", host, err, mvErr)
+ } else {
+ log.Printf("activator: %s restored CORRUPT from s3://%s/%s: %v — quarantined at %s, instance NOT started",
+ host, b.Cfg.Bucket, b.Cfg.hibernatePrefix(host), err, q)
+ }
+ for _, p := range []string{dbPath + "-wal", dbPath + "-shm"} {
+ _ = os.Remove(p)
+ }
+ // The next wake re-attempts the restore rather than refusing on
+ // sight of the quarantined file: a replaced or repaired replica
+ // must recover with no operator action. The price is a repeated
+ // restore for as long as the replica stays broken, which is the
+ // right way round — a permanently corrupt replica is an alarm
+ // someone is answering, not a steady state to optimise for.
+ return false, fmt.Errorf("restored replica for %s is corrupt: %w", host, err)
+ }
log.Printf("activator: restored %s from s3://%s/%s", host, b.Cfg.Bucket, b.Cfg.hibernatePrefix(host))
} else {
// Never replicated: a brand-new instance boots fresh and creates its
diff --git a/internal/dbverify/dbverify.go b/internal/dbverify/dbverify.go
new file mode 100644
index 000000000..10247c0ae
--- /dev/null
+++ b/internal/dbverify/dbverify.go
@@ -0,0 +1,151 @@
+// Package dbverify is the fleet's restore-drill record: which databases have
+// been PROVEN restorable, when, and which could not be decided.
+//
+// It exists as its own package for the reason internal/usage does — the
+// writer (the box-side `carlos ops restore-verify` pass) and any future
+// reader (the console) must agree on one shape and one key without either
+// importing the other's machinery.
+//
+// The vocabulary is deliberately three-valued, following
+// usage.AppStorage.StoreUnswept: a replica is `ok`, `failed`, or
+// `unverified` — never a silent absence and never a 0/false that reads as
+// "fine". A pass that could not reach S3 has learned NOTHING about the
+// replica, and saying so is the whole point of keeping this record.
+package dbverify
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// Key is where the record lives in the deployment bucket, beside
+// control/serial and control/registry-snapshot.json.
+const Key = "control/restore-verify.json"
+
+// Verdict is one database's standing.
+type Verdict struct {
+ // OK is true only when the replica was actually restored and passed its
+ // integrity check on this box, at VerifiedAt.
+ OK bool `json:"ok"`
+ // Unverified marks "we could not tell" — an S3 error, a prefix that has
+ // never been replicated to, or a host the run's cap did not reach. It is
+ // never a failure and never a pass.
+ Unverified bool `json:"unverified,omitempty"`
+ // VerifiedAt is when this verdict was reached — NOT when the replica was
+ // written. An old timestamp on an ok verdict is itself a finding.
+ VerifiedAt time.Time `json:"verified_at"`
+ // Detail carries the failure or the reason it could not be decided.
+ Detail string `json:"detail,omitempty"`
+}
+
+// Record is the whole box's standing at CheckedAt.
+type Record struct {
+ V int `json:"v"` // 1
+ Box string `json:"box"`
+ CheckedAt time.Time `json:"checked_at"`
+ Hosts map[string]Verdict `json:"hosts"`
+}
+
+// Merge folds this run's verdicts into the prior record. prior may be nil.
+//
+// Two rules, both about not overstating coverage:
+//
+// - A capped run visits only some hosts, so untouched hosts KEEP their
+// previous verdict — with its old timestamp, which is what makes staleness
+// visible rather than hidden behind a fresh CheckedAt.
+// - A host no longer in `fleet` is DROPPED. A retired instance's last "ok"
+// would otherwise sit in the record forever, padding the coverage of a
+// database that no longer exists.
+func Merge(prior *Record, box string, now time.Time, fleet []string, results map[string]Verdict) *Record {
+ rec := &Record{V: 1, Box: box, CheckedAt: now.UTC(), Hosts: map[string]Verdict{}}
+ inFleet := map[string]bool{}
+ for _, h := range fleet {
+ inFleet[h] = true
+ }
+ if prior != nil {
+ for host, v := range prior.Hosts {
+ if inFleet[host] {
+ rec.Hosts[host] = v
+ }
+ }
+ }
+ for host, v := range results {
+ if !inFleet[host] {
+ continue
+ }
+ rec.Hosts[host] = v
+ }
+ return rec
+}
+
+// Failures counts the verdicts that are an alarm — restored and damaged.
+// Unverified rows are deliberately not counted here: they are a coverage
+// problem, not a corruption one, and conflating them would either cry wolf or
+// train operators to ignore a red run.
+func (r *Record) Failures() int {
+ n := 0
+ for _, v := range r.Hosts {
+ if !v.OK && !v.Unverified {
+ n++
+ }
+ }
+ return n
+}
+
+// Unverified counts the verdicts the pass could not decide.
+func (r *Record) Unverified() int {
+ n := 0
+ for _, v := range r.Hosts {
+ if v.Unverified {
+ n++
+ }
+ }
+ return n
+}
+
+// Load reads the current record. A missing key is (nil, nil) — a deployment
+// that has never run the drill, not an error.
+func Load(ctx context.Context, s release.Store) (*Record, error) {
+ body, _, err := s.Get(ctx, Key)
+ if errors.Is(err, os.ErrNotExist) {
+ return nil, nil
+ }
+ if err != nil {
+ return nil, err
+ }
+ var rec Record
+ if err := json.Unmarshal(body, &rec); err != nil {
+ return nil, fmt.Errorf("parse %s: %w", Key, err)
+ }
+ return &rec, nil
+}
+
+// Publish writes rec, replacing whatever is there. Three attempts on a lost
+// CAS race, the recipe usage.UpsertEdgeDay and host.putReplacing both use.
+func Publish(ctx context.Context, s release.Store, rec *Record) error {
+ body, err := json.Marshal(rec)
+ if err != nil {
+ return err
+ }
+ for attempt := 0; attempt < 3; attempt++ {
+ _, etag, err := s.Get(ctx, Key)
+ switch {
+ case errors.Is(err, os.ErrNotExist):
+ etag = ""
+ case err != nil:
+ return fmt.Errorf("get %s: %w", Key, err)
+ }
+ if _, err := s.Put(ctx, Key, body, etag); err == nil {
+ return nil
+ } else if !errors.Is(err, release.ErrCASConflict) {
+ return fmt.Errorf("put %s: %w", Key, err)
+ }
+ }
+ return fmt.Errorf("put %s: lost three CAS races", Key)
+}
diff --git a/internal/dbverify/dbverify_test.go b/internal/dbverify/dbverify_test.go
new file mode 100644
index 000000000..9c6dd544d
--- /dev/null
+++ b/internal/dbverify/dbverify_test.go
@@ -0,0 +1,124 @@
+package dbverify
+
+import (
+ "context"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func at(s string) time.Time {
+ t, err := time.Parse(time.RFC3339, s)
+ if err != nil {
+ panic(err)
+ }
+ return t
+}
+
+// A capped run must not erase the hosts it did not reach — and must not
+// restamp them either. The old timestamp is what makes thin coverage visible.
+func TestMergeKeepsUnvisitedHostsWithTheirOldTimestamp(t *testing.T) {
+ old := at("2026-08-01T00:00:00Z")
+ prior := &Record{V: 1, Box: "box-a", CheckedAt: old, Hosts: map[string]Verdict{
+ "a.example.com": {OK: true, VerifiedAt: old},
+ "b.example.com": {OK: true, VerifiedAt: old},
+ }}
+ now := at("2026-08-15T09:00:00Z")
+
+ rec := Merge(prior, "box-a", now, []string{"a.example.com", "b.example.com"},
+ map[string]Verdict{"a.example.com": {OK: true, VerifiedAt: now}})
+
+ if got := rec.Hosts["a.example.com"].VerifiedAt; !got.Equal(now) {
+ t.Fatalf("visited host kept the old stamp: %v", got)
+ }
+ b, ok := rec.Hosts["b.example.com"]
+ if !ok {
+ t.Fatal("a host the capped run never reached was dropped from the record")
+ }
+ if !b.VerifiedAt.Equal(old) {
+ t.Fatalf("an unvisited host was restamped %v — its coverage would read as fresh", b.VerifiedAt)
+ }
+}
+
+// A retired instance's last "ok" must not linger and pad the coverage of a
+// database that no longer exists.
+func TestMergeDropsHostsNoLongerInTheFleet(t *testing.T) {
+ old := at("2026-08-01T00:00:00Z")
+ prior := &Record{V: 1, Hosts: map[string]Verdict{
+ "gone.example.com": {OK: true, VerifiedAt: old},
+ "live.example.com": {OK: true, VerifiedAt: old},
+ }}
+ rec := Merge(prior, "box-a", at("2026-08-15T09:00:00Z"), []string{"live.example.com"}, nil)
+ if _, ok := rec.Hosts["gone.example.com"]; ok {
+ t.Fatal("a retired host kept its verdict")
+ }
+ if _, ok := rec.Hosts["live.example.com"]; !ok {
+ t.Fatal("a live host lost its verdict")
+ }
+}
+
+// The three-valued vocabulary is the point: unreachable is not a pass and not
+// an alarm.
+func TestUnverifiedIsNeitherPassNorFailure(t *testing.T) {
+ rec := &Record{Hosts: map[string]Verdict{
+ "ok.example.com": {OK: true},
+ "bad.example.com": {OK: false, Detail: "integrity_check: page 4 corrupt"},
+ "unknown.example.com": {Unverified: true, Detail: "s3: connection refused"},
+ }}
+ if got := rec.Failures(); got != 1 {
+ t.Fatalf("Failures() = %d, want 1 — only the damaged replica is an alarm", got)
+ }
+ if got := rec.Unverified(); got != 1 {
+ t.Fatalf("Unverified() = %d, want 1", got)
+ }
+ if rec.Hosts["unknown.example.com"].OK {
+ t.Fatal("an undecided verdict reported OK")
+ }
+}
+
+func TestLoadPublishRoundTrip(t *testing.T) {
+ ctx := context.Background()
+ store := release.NewDirStore(t.TempDir())
+
+ // Never run: absent is not an error.
+ got, err := Load(ctx, store)
+ if err != nil {
+ t.Fatalf("Load on a fresh deployment: %v", err)
+ }
+ if got != nil {
+ t.Fatalf("Load invented a record: %+v", got)
+ }
+
+ now := at("2026-08-15T09:00:00Z")
+ rec := Merge(nil, "box-a", now, []string{"a.example.com"},
+ map[string]Verdict{"a.example.com": {OK: true, VerifiedAt: now}})
+ if err := Publish(ctx, store, rec); err != nil {
+ t.Fatalf("Publish: %v", err)
+ }
+
+ got, err = Load(ctx, store)
+ if err != nil {
+ t.Fatalf("Load: %v", err)
+ }
+ if got == nil || !got.Hosts["a.example.com"].OK {
+ t.Fatalf("round-trip lost the verdict: %+v", got)
+ }
+ if got.Box != "box-a" || !got.CheckedAt.Equal(now) {
+ t.Fatalf("round-trip lost the provenance: %+v", got)
+ }
+
+ // And a second publish replaces rather than conflicting.
+ rec2 := Merge(got, "box-a", now.Add(time.Hour), []string{"a.example.com"},
+ map[string]Verdict{"a.example.com": {OK: false, Detail: "integrity_check: bad"}})
+ if err := Publish(ctx, store, rec2); err != nil {
+ t.Fatalf("second Publish: %v", err)
+ }
+ got, err = Load(ctx, store)
+ if err != nil {
+ t.Fatalf("Load after replace: %v", err)
+ }
+ if got.Failures() != 1 {
+ t.Fatalf("replace did not land: %+v", got)
+ }
+}
diff --git a/internal/host/host_test.go b/internal/host/host_test.go
index 09cd6a18e..81387cf9c 100644
--- a/internal/host/host_test.go
+++ b/internal/host/host_test.go
@@ -32,6 +32,8 @@ var expectedUnits = []string{
"carlos-app@.socket",
"carlos-config-sync.service",
"carlos-edge.service",
+ "carlos-hibernate-verify.service",
+ "carlos-hibernate-verify.timer",
"carlos-litestream.service",
"carlos-restart.path",
"carlos-restart.service",
diff --git a/internal/host/units/carlos-hibernate-verify.service b/internal/host/units/carlos-hibernate-verify.service
new file mode 100644
index 000000000..dbd37a716
--- /dev/null
+++ b/internal/host/units/carlos-hibernate-verify.service
@@ -0,0 +1,46 @@
+# The other half of the restore drill. carlos-restore-verify.service proves
+# every database named in /etc/litestream.yml; that config deliberately OMITS
+# hibernating instances, because the activator's embedded litestream owns
+# their replication under its lease and a second replicator on the same WAL
+# would corrupt the replica. The consequence, until this unit existed, was
+# that the fleet's largest set of app databases was the one set nobody ever
+# drilled — the first thing to learn whether a hibernating tenant's replica
+# restores clean was a real user's wake.
+#
+# This restores each of those replicas into a private tmp, integrity-checks
+# it, and publishes a per-host verdict to the deployment bucket
+# (control/restore-verify.json) so the result is readable WITHOUT a session on
+# this box. A damaged replica leaves the unit 'failed'; a replica the pass
+# could not reach is recorded 'unverified' and does not fail the run.
+#
+# Inert until /etc/carlos/host.env carries CARLOS_HIBERNATE_BUCKET — a box
+# with no hibernating instances prints that and exits 0.
+[Unit]
+Description=carlos hibernate-verify (restore-drill the activator-owned replicas)
+After=network-online.target carlos-edge.service
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+# Same host-wide config the edge reads: CARLOS_HIBERNATE_* names the replica
+# set, CARLOS_DEPLOYMENT_BUCKET/_DIR is where the verdict is published.
+# Deliberately NOT /data/litestream.env — those credentials belong to the
+# box-wide replication bucket, and borrowing them here would sign for the
+# wrong store.
+EnvironmentFile=-/etc/carlos/host.env
+ExecStart=/opt/carlos/carlos ops hibernate-verify
+User=carlos
+Group=carlos
+CapabilityBoundingSet=
+NoNewPrivileges=yes
+ProtectSystem=strict
+# The registry is opened (read-only in effect, but SQLite wants the file and
+# its directory writable to attach WAL) — the same grant carlos-edge.service
+# holds, for a unit running as the user that already owns /data.
+ReadWritePaths=/data
+ProtectHome=yes
+PrivateTmp=yes
+PrivateDevices=yes
+RestrictAddressFamilies=AF_INET AF_INET6
+MemoryDenyWriteExecute=yes
+MemoryMax=256M
diff --git a/internal/host/units/carlos-hibernate-verify.timer b/internal/host/units/carlos-hibernate-verify.timer
new file mode 100644
index 000000000..031332494
--- /dev/null
+++ b/internal/host/units/carlos-hibernate-verify.timer
@@ -0,0 +1,15 @@
+# Daily restore drill for the activator-owned replicas — see
+# carlos-hibernate-verify.service. Deliberately at 03:00 rather than
+# carlos-restore-verify.timer's midnight: both drills pull whole databases
+# back from S3, and running them in the same window would double the box's
+# restore load for no gain.
+[Unit]
+Description=carlos hibernate-verify daily timer
+
+[Timer]
+OnCalendar=*-*-* 03:00:00
+RandomizedDelaySec=30min
+Persistent=true
+
+[Install]
+WantedBy=timers.target
diff --git a/internal/lease/check.go b/internal/lease/check.go
new file mode 100644
index 000000000..9902489f0
--- /dev/null
+++ b/internal/lease/check.go
@@ -0,0 +1,100 @@
+package lease
+
+import (
+ "context"
+ "database/sql"
+ "errors"
+ "fmt"
+ "os"
+ "path/filepath"
+
+ _ "modernc.org/sqlite"
+)
+
+// restoreForVerify is Restore behind a package var so the drill's three
+// outcomes (clean, damaged, never-replicated) are testable without S3.
+var restoreForVerify = Restore
+
+// ErrNoReplica reports that a prefix holds no backup at all — a brand-new
+// instance that has never replicated, not a damaged one. Verification callers
+// must keep the two apart: "nothing to restore yet" is not a failure, and
+// reporting it as one would train operators to ignore the alarm.
+var ErrNoReplica = errors.New("no replica in prefix")
+
+// CheckDB opens path read-only and proves it is a sane SQLite database:
+// integrity_check == ok and at least one table present. It is the one check a
+// disaster recovery leans on, so it lives here — beside Restore — where both
+// the wake path (internal/activator) and the verification pass (internal/ops)
+// can reach it without either importing the other.
+//
+// The schema check is not pedantry: a restore that yields a valid but empty
+// database passes integrity_check and would be useless in the recovery it was
+// meant for.
+func CheckDB(path string) error {
+ db, err := sql.Open("sqlite", "file:"+path+"?mode=ro")
+ if err != nil {
+ return err
+ }
+ defer db.Close()
+ db.SetMaxOpenConns(1)
+ // busy_timeout first, always (Global Constraints) — and on a garbage file
+ // this first statement is what fails loudly instead of a later query.
+ if _, err := db.Exec("PRAGMA busy_timeout=5000"); err != nil {
+ return fmt.Errorf("open: %v", err)
+ }
+ var res string
+ if err := db.QueryRow("PRAGMA integrity_check").Scan(&res); err != nil {
+ return fmt.Errorf("integrity_check: %v", err)
+ }
+ if res != "ok" {
+ return fmt.Errorf("integrity_check: %s", res)
+ }
+ var tables int
+ if err := db.QueryRow("SELECT count(*) FROM sqlite_master WHERE type='table'").Scan(&tables); err != nil {
+ return fmt.Errorf("schema count: %v", err)
+ }
+ if tables == 0 {
+ return errors.New("restored database has no tables")
+ }
+ return nil
+}
+
+// VerifyReplica restores s3://bucket/prefix into a throwaway file under dir
+// and proves it opens clean, then deletes it. It returns ErrNoReplica when
+// the prefix has never been replicated to, and a describing error when the
+// restore or the check fails.
+//
+// IT DELIBERATELY DOES NOT TAKE THE LEASE, and that is safe in a way Restore
+// itself is not. Restore's contract requires the lease because it materializes
+// the LIVE database: an unleased restore there can resurrect a stale copy that
+// later replicates over the true holder's newer state. Here the output path is
+// a temp file that is never replicated from and is removed before return, so
+// there is nothing that could shadow anything. What the lease forbids is a
+// second WRITER; a reader costs the holder nothing — the box-wide
+// `carlos ops restore-verify` has always restored databases that its own
+// replicator was actively streaming, and litestream restores a committed
+// position rather than a live tail.
+func VerifyReplica(ctx context.Context, bucket, prefix, region, dir string) error {
+ out := filepath.Join(dir, "verify-"+filepath.Base(prefix)+".db")
+ // A retry must not trip over a previous attempt's file: litestream refuses
+ // to restore over an existing output path.
+ _ = os.Remove(out)
+ // Sweep on EVERY path, including a failed check. The sidecars are swept
+ // too: CheckDB's "mode=ro" does not stop the driver attaching a WAL, and a
+ // drill that left a copy of every tenant's database on the box would be
+ // both a disk bug and a data-exposure one.
+ defer func() {
+ _ = os.Remove(out)
+ _ = os.Remove(out + "-wal")
+ _ = os.Remove(out + "-shm")
+ }()
+
+ restored, err := restoreForVerify(ctx, out, bucket, prefix, region)
+ if err != nil {
+ return fmt.Errorf("restore: %w", err)
+ }
+ if !restored {
+ return ErrNoReplica
+ }
+ return CheckDB(out)
+}
diff --git a/internal/lease/check_test.go b/internal/lease/check_test.go
new file mode 100644
index 000000000..851b320b3
--- /dev/null
+++ b/internal/lease/check_test.go
@@ -0,0 +1,192 @@
+package lease
+
+import (
+ "context"
+ "database/sql"
+ "errors"
+ "os"
+ "path/filepath"
+ "testing"
+
+ _ "modernc.org/sqlite"
+)
+
+// TestCheckDB pins the three verdicts a restored file can earn. Ported from
+// ops.TestCheckRestoredDBRejectsEmptySchema, which now runs against this
+// implementation through the delegating wrapper.
+func TestCheckDB(t *testing.T) {
+ dir := t.TempDir()
+
+ good := filepath.Join(dir, "good.db")
+ db, err := sql.Open("sqlite", good)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := db.Exec("CREATE TABLE t (id INTEGER PRIMARY KEY); INSERT INTO t VALUES (1)"); err != nil {
+ t.Fatal(err)
+ }
+ db.Close()
+ if err := CheckDB(good); err != nil {
+ t.Fatalf("CheckDB(good) = %v, want nil", err)
+ }
+
+ // A schema-less database restores "successfully" but would be useless in a
+ // disaster — the check must refuse it.
+ empty := filepath.Join(dir, "empty.db")
+ db, err = sql.Open("sqlite", empty)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := db.Exec("PRAGMA user_version=1"); err != nil { // force file creation
+ t.Fatal(err)
+ }
+ db.Close()
+ if err := CheckDB(empty); err == nil {
+ t.Fatal("CheckDB accepted a database with no tables")
+ }
+
+ // Garbage bytes must fail loudly, not pass as an open-able file.
+ junk := filepath.Join(dir, "junk.db")
+ if err := os.WriteFile(junk, []byte("this is not a sqlite database at all"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ if err := CheckDB(junk); err == nil {
+ t.Fatal("CheckDB accepted a non-SQLite file")
+ }
+
+ // A file that is not there at all is a failure, never a silent pass.
+ if err := CheckDB(filepath.Join(dir, "absent.db")); err == nil {
+ t.Fatal("CheckDB accepted a missing file")
+ }
+}
+
+// TestCheckDBDetectsPageCorruption is the case the whole verification pass
+// exists for: a file that opens fine and has a schema, but whose pages are
+// damaged. integrity_check is the only thing that catches it.
+func TestCheckDBDetectsPageCorruption(t *testing.T) {
+ dir := t.TempDir()
+ path := filepath.Join(dir, "corrupt.db")
+
+ db, err := sql.Open("sqlite", path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // No WAL: the damage below must land in the main database file, not in a
+ // sidecar the reopen would replay over.
+ if _, err := db.Exec("PRAGMA journal_mode=DELETE"); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := db.Exec("CREATE TABLE t (id INTEGER PRIMARY KEY, body TEXT)"); err != nil {
+ t.Fatal(err)
+ }
+ for i := 0; i < 2000; i++ {
+ if _, err := db.Exec("INSERT INTO t (body) VALUES (?)", "row padding to force several pages"); err != nil {
+ t.Fatal(err)
+ }
+ }
+ db.Close()
+
+ if fi, err := os.Stat(path); err != nil {
+ t.Fatal(err)
+ } else if fi.Size() < 3*4096 {
+ t.Fatalf("test fixture is only %d bytes — too few pages to corrupt one meaningfully", fi.Size())
+ }
+
+ if err := CheckDB(path); err != nil {
+ t.Fatalf("CheckDB on the intact file = %v, want nil", err)
+ }
+
+ // Scribble over page 2's header — its page type and cell-pointer array —
+ // leaving the file length and page 1 (the schema) intact so the open still
+ // succeeds and only integrity_check can tell.
+ f, err := os.OpenFile(path, os.O_WRONLY, 0o644)
+ if err != nil {
+ t.Fatal(err)
+ }
+ junk := make([]byte, 512)
+ for i := range junk {
+ junk[i] = 0xA5
+ }
+ if _, err := f.WriteAt(junk, 4096); err != nil {
+ t.Fatal(err)
+ }
+ f.Close()
+
+ if err := CheckDB(path); err == nil {
+ t.Fatal("CheckDB passed a file with a scribbled-over page")
+ }
+}
+
+// The restored file is WAL-journaled — that is what instances write and what
+// litestream restores — so the check must pass one. This is the shape both
+// callers actually hand it, and the box-wide drill has been opening exactly
+// this way in production daily.
+//
+// It also pins what the "mode=ro" in the DSN does NOT buy: the driver still
+// attaches WAL sidecars. That is harmless at both call sites and neither may
+// come to depend on the opposite — the wake path calls this holding the
+// lease, on a file it has just restored with the sidecars already cleared
+// (nothing else has the database open), and VerifyReplica works on a
+// throwaway copy it deletes, sidecars included.
+func TestCheckDBAcceptsWALDatabase(t *testing.T) {
+ dir := t.TempDir()
+ path := filepath.Join(dir, "wal.db")
+ seedDB(t, path) // WAL mode, 50 rows — the lease package's own fixture
+
+ // litestream restores a checkpointed file with no live -wal beside it;
+ // mimic that so this tests the shape the wake path actually sees.
+ for _, p := range []string{path + "-wal", path + "-shm"} {
+ _ = os.Remove(p)
+ }
+ if err := CheckDB(path); err != nil {
+ t.Fatalf("CheckDB on a WAL-mode restore = %v, want nil", err)
+ }
+}
+
+// VerifyReplica must leave nothing behind — including the WAL sidecars the
+// check attaches. A drill that littered the box with restored copies of every
+// tenant database would be a disk-usage bug and a data-exposure one.
+func TestVerifyReplicaCleansUpEvenOnFailure(t *testing.T) {
+ dir := t.TempDir()
+
+ // A restore that "succeeds" into a corrupt file: the check fails, and the
+ // wreckage must still be swept.
+ prev := restoreForVerify
+ restoreForVerify = func(_ context.Context, out, _, _, _ string) (bool, error) {
+ if err := os.WriteFile(out, []byte("not a database"), 0o600); err != nil {
+ return false, err
+ }
+ return true, nil
+ }
+ t.Cleanup(func() { restoreForVerify = prev })
+
+ if err := VerifyReplica(context.Background(), "bucket", "hib/x.example.com", "", dir); err == nil {
+ t.Fatal("VerifyReplica accepted a corrupt restore")
+ }
+ left, err := os.ReadDir(dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(left) != 0 {
+ names := []string{}
+ for _, e := range left {
+ names = append(names, e.Name())
+ }
+ t.Fatalf("VerifyReplica left %v behind", names)
+ }
+}
+
+// A prefix that has never been replicated to is ErrNoReplica — "nothing to
+// restore yet", which callers must not report as a failure.
+func TestVerifyReplicaReportsNoReplica(t *testing.T) {
+ prev := restoreForVerify
+ restoreForVerify = func(context.Context, string, string, string, string) (bool, error) {
+ return false, nil
+ }
+ t.Cleanup(func() { restoreForVerify = prev })
+
+ err := VerifyReplica(context.Background(), "bucket", "hib/new.example.com", "", t.TempDir())
+ if !errors.Is(err, ErrNoReplica) {
+ t.Fatalf("VerifyReplica on a never-replicated prefix = %v, want ErrNoReplica", err)
+ }
+}
diff --git a/internal/ops/hibernate_verify.go b/internal/ops/hibernate_verify.go
new file mode 100644
index 000000000..985adfbcd
--- /dev/null
+++ b/internal/ops/hibernate_verify.go
@@ -0,0 +1,132 @@
+package ops
+
+// The hibernation half of the restore drill.
+//
+// FleetDBs deliberately EXCLUDES hibernating exec routes from
+// /etc/litestream.yml — the activator's embedded litestream owns their
+// replication under the lease, and a second replicator on the same WAL would
+// corrupt the replica (Eleven's lesson). The unintended consequence was that
+// the fleet's LARGEST set of app databases was the one set nobody ever
+// restore-drilled: their replicas live in the instances bucket under
+// <CARLOS_HIBERNATE_PREFIX>/<host>/, and the first thing to find out whether
+// one restores clean was a real user's wake.
+//
+// This pass closes that. It is a READER: it restores each replica into a
+// throwaway file and integrity-checks it (see lease.VerifyReplica for why
+// that needs no lease and cannot shadow the true holder's state), so it can
+// run against instances that are asleep, awake and actively replicating, or
+// anywhere in between.
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "sort"
+ "time"
+
+ "github.com/carlosframework/platform/internal/activator"
+ "github.com/carlosframework/platform/internal/dbverify"
+ "github.com/carlosframework/platform/internal/lease"
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// HibernatingHosts lists the routes whose replication the activator owns —
+// exactly the set FleetDBs filters out, derived with the SAME conditions so
+// the two cannot drift into a gap that belongs to neither pass.
+func HibernatingHosts(reg *registry.Registry) ([]string, error) {
+ routes, err := reg.List()
+ if err != nil {
+ return nil, err
+ }
+ var hosts []string
+ for _, r := range routes {
+ // BackingUnit tenants are the box-wide replicator's job (their wake is
+ // systemctl and the activator never replicates them) — FleetDBs adds
+ // them, so this pass must not claim them.
+ if r.Backing == registry.BackingUnit {
+ continue
+ }
+ if r.Hibernate && r.DBPath != "" {
+ hosts = append(hosts, r.Host)
+ }
+ }
+ sort.Strings(hosts)
+ return hosts, nil
+}
+
+// verifyOne is the seam the tests drive; production points it at S3.
+var verifyOne = lease.VerifyReplica
+
+// VerifyHibernating restore-drills up to max of the given hosts and returns a
+// verdict per host it VISITED. Hosts beyond the cap are simply absent from
+// the result — Merge keeps their previous verdict and its old timestamp
+// rather than inventing coverage for them.
+//
+// Order is the caller's: pass hosts oldest-verified-first so a fleet larger
+// than the cap is still covered completely, just over several days.
+//
+// The cap is not squeamishness — it is the same request-cost discipline that
+// set LeaseTTL. Each verification is a full restore: in-region S3→EC2
+// transfer is free, but the GETs and the restore CPU are not, and an
+// unbounded nightly drill over a growing fleet is a bill that grows with it.
+func VerifyHibernating(ctx context.Context, cfg *activator.Config, hosts []string, max int, now time.Time, tmpDir string) map[string]dbverify.Verdict {
+ out := map[string]dbverify.Verdict{}
+ for i, host := range hosts {
+ if max > 0 && i >= max {
+ break
+ }
+ prefix := activator.HibernatePrefix(cfg.Prefix, host)
+ err := verifyOne(ctx, cfg.Bucket, prefix, cfg.Region, tmpDir)
+ switch {
+ case err == nil:
+ out[host] = dbverify.Verdict{OK: true, VerifiedAt: now.UTC()}
+ fmt.Printf("ok %s\n", host)
+ case errors.Is(err, lease.ErrNoReplica):
+ // Never replicated: a registered instance that has not woken yet.
+ // Not a failure — but not a pass either, because nothing was
+ // proven about anything.
+ out[host] = dbverify.Verdict{Unverified: true, VerifiedAt: now.UTC(), Detail: "no replica yet"}
+ fmt.Printf("unverified %s: no replica yet (never woken)\n", host)
+ case ctx.Err() != nil:
+ // The run is being torn down; what we could not reach is
+ // unverified, never failed.
+ out[host] = dbverify.Verdict{Unverified: true, VerifiedAt: now.UTC(), Detail: err.Error()}
+ fmt.Printf("unverified %s: %v\n", host, err)
+ default:
+ out[host] = dbverify.Verdict{VerifiedAt: now.UTC(), Detail: err.Error()}
+ fmt.Printf("FAIL %s: %v\n", host, err)
+ }
+ }
+ if max > 0 && len(hosts) > max {
+ // A silent cap reads as full coverage — say what was left.
+ fmt.Printf("hibernate-verify: %d of %d hosts this run (-max %d); the rest keep their previous verdict\n",
+ max, len(hosts), max)
+ }
+ return out
+}
+
+// OldestFirst orders hosts by how long ago they were last verified, most
+// stale first, so a capped run works through the fleet instead of drilling
+// the same alphabetical prefix every night. Hosts with no verdict at all sort
+// first — never verified is staler than any timestamp.
+func OldestFirst(hosts []string, prior *dbverify.Record) []string {
+ stamp := map[string]time.Time{}
+ if prior != nil {
+ for h, v := range prior.Hosts {
+ stamp[h] = v.VerifiedAt
+ }
+ }
+ ordered := append([]string(nil), hosts...)
+ sort.SliceStable(ordered, func(i, j int) bool {
+ a, aok := stamp[ordered[i]]
+ b, bok := stamp[ordered[j]]
+ if aok != bok {
+ return !aok // never-verified first
+ }
+ if a.Equal(b) {
+ return ordered[i] < ordered[j] // stable, and deterministic in tests
+ }
+ return a.Before(b)
+ })
+ return ordered
+}
diff --git a/internal/ops/hibernate_verify_test.go b/internal/ops/hibernate_verify_test.go
new file mode 100644
index 000000000..1c88d2bb3
--- /dev/null
+++ b/internal/ops/hibernate_verify_test.go
@@ -0,0 +1,187 @@
+package ops
+
+import (
+ "context"
+ "errors"
+ "os"
+ "path/filepath"
+ "slices"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/activator"
+ "github.com/carlosframework/platform/internal/dbverify"
+ "github.com/carlosframework/platform/internal/lease"
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+func at(s string) time.Time {
+ t, err := time.Parse(time.RFC3339, s)
+ if err != nil {
+ panic(err)
+ }
+ return t
+}
+
+// The two passes must partition the fleet: every database is owned by exactly
+// one of them. This test is the mirror of TestConfigGenSkipsHibernatingRoutes
+// — what that one proves is EXCLUDED here must be INCLUDED, or it is drilled
+// by nobody.
+func TestHibernatingHostsIsTheComplementOfFleetDBs(t *testing.T) {
+ dir := t.TempDir()
+ reg, err := registry.Open(filepath.Join(dir, "registry.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ unitDir := t.TempDir()
+
+ awake := filepath.Join(dir, "a.example.com.db")
+ asleep := filepath.Join(dir, "b.example.com.db")
+ unitTenant := filepath.Join(unitDir, "state.db")
+ for _, p := range []string{awake, asleep, unitTenant} {
+ if err := os.WriteFile(p, []byte("x"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if err := reg.Upsert(registry.Route{Host: "a.example.com", DBPath: awake}); err != nil {
+ t.Fatal(err)
+ }
+ if err := reg.Upsert(registry.Route{Host: "b.example.com", DBPath: asleep, Hibernate: true}); err != nil {
+ t.Fatal(err)
+ }
+ if err := reg.Upsert(registry.Route{Host: "c.example.com", DBPath: unitTenant,
+ Hibernate: true, Backing: registry.BackingUnit}); err != nil {
+ t.Fatal(err)
+ }
+
+ hosts, err := HibernatingHosts(reg)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !slices.Contains(hosts, "b.example.com") {
+ t.Fatalf("the hibernating exec route is drilled by nobody: %v", hosts)
+ }
+ if slices.Contains(hosts, "a.example.com") {
+ t.Fatalf("an awake route is claimed by both passes: %v", hosts)
+ }
+ if slices.Contains(hosts, "c.example.com") {
+ t.Fatalf("a unit tenant (box-wide replicator's job) is claimed by both passes: %v", hosts)
+ }
+
+ // And nothing this pass claims may appear in the box-wide config.
+ dbs, err := FleetDBs(reg, dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ cfg := LitestreamConfig(dbs, "b", "p", "eu-west-1")
+ for _, h := range hosts {
+ if strings.Contains(cfg, h+".db") {
+ t.Fatalf("%s is replicated by BOTH passes — two replicators on one WAL:\n%s", h, cfg)
+ }
+ }
+}
+
+// swapVerifyOne points the restore seam at a stub.
+func swapVerifyOne(t *testing.T, fn func(ctx context.Context, bucket, prefix, region, dir string) error) {
+ t.Helper()
+ prev := verifyOne
+ verifyOne = fn
+ t.Cleanup(func() { verifyOne = prev })
+}
+
+// The three verdicts, and — most importantly — that an S3 failure is
+// "unverified", not a corruption alarm, and a never-woken instance is neither.
+func TestVerifyHibernatingVerdicts(t *testing.T) {
+ now := at("2026-08-15T09:00:00Z")
+ swapVerifyOne(t, func(_ context.Context, _, prefix, _, _ string) error {
+ switch {
+ case strings.Contains(prefix, "good"):
+ return nil
+ case strings.Contains(prefix, "damaged"):
+ return errors.New("integrity_check: page 4 corrupt")
+ case strings.Contains(prefix, "new"):
+ return lease.ErrNoReplica
+ }
+ return errors.New("unexpected host")
+ })
+
+ cfg := &activator.Config{Bucket: "instances", Prefix: "hib", Region: "eu-west-1"}
+ got := VerifyHibernating(context.Background(), cfg,
+ []string{"good.example.com", "damaged.example.com", "new.example.com"}, 0, now, t.TempDir())
+
+ if v := got["good.example.com"]; !v.OK || v.Unverified {
+ t.Fatalf("clean replica verdict = %+v", v)
+ }
+ if v := got["damaged.example.com"]; v.OK || v.Unverified {
+ t.Fatalf("damaged replica must be a failure, got %+v", v)
+ }
+ if v := got["new.example.com"]; v.OK || !v.Unverified {
+ t.Fatalf("never-woken instance must be unverified (nothing was proven), got %+v", v)
+ }
+
+ rec := dbverify.Merge(nil, "box-a", now,
+ []string{"good.example.com", "damaged.example.com", "new.example.com"}, got)
+ if rec.Failures() != 1 {
+ t.Fatalf("Failures() = %d, want 1 — only the damaged replica", rec.Failures())
+ }
+ if rec.Unverified() != 1 {
+ t.Fatalf("Unverified() = %d, want 1", rec.Unverified())
+ }
+}
+
+// The per-host prefix must be the one the activator actually writes to, or
+// every drill reports "no replica" over perfectly good backups.
+func TestVerifyHibernatingUsesTheActivatorsPrefix(t *testing.T) {
+ var seen string
+ swapVerifyOne(t, func(_ context.Context, _, prefix, _, _ string) error {
+ seen = prefix
+ return nil
+ })
+ cfg := &activator.Config{Bucket: "instances", Prefix: "hib"}
+ VerifyHibernating(context.Background(), cfg, []string{"x.example.com"}, 0, time.Now(), t.TempDir())
+
+ want := activator.HibernatePrefix("hib", "x.example.com")
+ if seen != want {
+ t.Fatalf("drilled prefix %q, activator writes %q", seen, want)
+ }
+}
+
+// A capped run visits the cap and no more — the rest keep their old verdict
+// via Merge, which is what stops a cap from reading as full coverage.
+func TestVerifyHibernatingHonoursTheCap(t *testing.T) {
+ calls := 0
+ swapVerifyOne(t, func(context.Context, string, string, string, string) error {
+ calls++
+ return nil
+ })
+ cfg := &activator.Config{Bucket: "instances"}
+ got := VerifyHibernating(context.Background(), cfg,
+ []string{"a.example.com", "b.example.com", "c.example.com"}, 2, time.Now(), t.TempDir())
+
+ if calls != 2 {
+ t.Fatalf("restored %d replicas under -max 2", calls)
+ }
+ if len(got) != 2 {
+ t.Fatalf("returned %d verdicts under -max 2: %v", len(got), got)
+ }
+}
+
+// Oldest-first is what makes a capped drill cover the whole fleet over time
+// instead of re-drilling the same alphabetical head every night.
+func TestOldestFirstPutsNeverVerifiedThenStalest(t *testing.T) {
+ prior := &dbverify.Record{Hosts: map[string]dbverify.Verdict{
+ "recent.example.com": {OK: true, VerifiedAt: at("2026-08-14T00:00:00Z")},
+ "stale.example.com": {OK: true, VerifiedAt: at("2026-08-01T00:00:00Z")},
+ }}
+ got := OldestFirst([]string{"recent.example.com", "stale.example.com", "never.example.com"}, prior)
+ want := []string{"never.example.com", "stale.example.com", "recent.example.com"}
+ if !slices.Equal(got, want) {
+ t.Fatalf("OldestFirst = %v, want %v", got, want)
+ }
+
+ // With no prior record at all, the order must still be deterministic.
+ if got := OldestFirst([]string{"b.example.com", "a.example.com"}, nil); !slices.Equal(got, []string{"a.example.com", "b.example.com"}) {
+ t.Fatalf("OldestFirst with no prior record = %v", got)
+ }
+}
diff --git a/internal/ops/litestream.go b/internal/ops/litestream.go
index fc9275121..05c65c862 100644
--- a/internal/ops/litestream.go
+++ b/internal/ops/litestream.go
@@ -22,7 +22,6 @@ package ops
import (
"bytes"
- "database/sql"
"fmt"
"os"
"os/exec"
@@ -31,9 +30,8 @@ import (
"strings"
"time"
+ "github.com/carlosframework/platform/internal/lease"
"github.com/carlosframework/platform/internal/registry"
-
- _ "modernc.org/sqlite"
)
// litestreamVersion is the pinned release provisioned on the box when the
@@ -45,9 +43,33 @@ const litestreamVersion = "0.5.15"
// can shrink it.
var retryDelay = 10 * time.Second
+// SnapshotInterval and SnapshotRetention pin the fleet's point-in-time
+// recovery window instead of inheriting it.
+//
+// This is the one setting whose DEFAULT was a real hole: litestream 0.5's
+// DefaultSnapshotRetention is 24h, so a database this replicator owns could
+// only ever be restored to roughly the last day. That covers a dead box; it
+// does not cover the failure mode replication is otherwise useless against —
+// LOGICAL corruption, where integrity_check says "ok" and the data is simply
+// wrong. A bad migration or an errant DELETE is typically noticed days later,
+// and by then the only copy is the faithfully-replicated bad one.
+//
+// A week is the deliberate choice: seven daily snapshots per database, of
+// files measured in megabytes, against S3 storage at ~$0.023/GB-mo — noise
+// next to the request bill this fleet actually watches (see internal/lease's
+// LeaseTTL for that story). Lengthen it if a restore is ever wanted from
+// further back than "we noticed within the week".
+const (
+ SnapshotInterval = 24 * time.Hour
+ SnapshotRetention = 7 * 24 * time.Hour
+)
+
// LitestreamConfig renders the generated /etc/litestream.yml. Deliberately
-// minimal — path + replica URL + region only — so it can't rot against
-// litestream's config surface; tuning belongs in flags on the unit, not here.
+// minimal — path + replica URL + region, plus the snapshot policy above — so
+// it can't rot against litestream's config surface; other tuning belongs in
+// flags on the unit, not here. The snapshot stanza is the considered
+// exception to that minimalism: a recovery window is a decision, and leaving
+// it to a library default meant nobody had made it.
// Paths are sorted so re-runs with an unchanged fleet produce a byte-identical
// file (the `litestream` verb's changed/unchanged report depends on it).
func LitestreamConfig(dbPaths []string, bucket, prefix, region string) string {
@@ -56,6 +78,10 @@ func LitestreamConfig(dbPaths []string, bucket, prefix, region string) string {
var b strings.Builder
b.WriteString("# GENERATED by `carlos ops litestream` — do not hand-edit; re-run the verb\n")
b.WriteString("# after `carlos add`/`remove` so the replication set follows the fleet.\n")
+ // Global snapshot policy: applies to every database below.
+ b.WriteString("snapshot:\n")
+ fmt.Fprintf(&b, " interval: %s\n", SnapshotInterval)
+ fmt.Fprintf(&b, " retention: %s\n", SnapshotRetention)
b.WriteString("dbs:\n")
for _, p := range sorted {
fmt.Fprintf(&b, " - path: %s\n", p)
@@ -113,6 +139,19 @@ func ParseLitestreamDBs(cfg string) []string {
// carried verbatim.) That exclusion never applies to BackingUnit routes —
// their replication has no other owner at all, so they are added here
// rather than filtered.
+// consoleDBPath resolves where this box's console keeps its database:
+// CONSOLE_DB when set (the same env var cmd/console reads, so a relocated
+// console cannot silently drop out of replication), else the production
+// layout <data>/console/console.db — expressed against dataDir rather than
+// hardcoded to /data so a test box's console is found under its own root and
+// never the real one.
+func consoleDBPath(dataDir string) string {
+ if p := strings.TrimSpace(os.Getenv("CONSOLE_DB")); p != "" {
+ return filepath.Clean(p)
+ }
+ return filepath.Join(dataDir, "console", "console.db")
+}
+
func FleetDBs(reg *registry.Registry, dataDir string) ([]string, error) {
routes, err := reg.List()
if err != nil {
@@ -145,6 +184,14 @@ func FleetDBs(reg *registry.Registry, dataDir string) ([]string, error) {
for p := range unitPaths {
matches = append(matches, p)
}
+ // And the console's own database, for the same reason in miniature: it
+ // sits at <data>/console/console.db — ONE DIRECTORY BELOW the flat glob,
+ // which is exactly why it went unreplicated. It holds no platform state
+ // (routes and channels are read live) but it holds every account,
+ // ownership row, session and the deployment's Keymail credentials: losing
+ // it loses who everyone is. Boxes without a console simply have no file
+ // here and the loop below drops it.
+ matches = append(matches, consoleDBPath(dataDir))
seen := map[string]bool{}
var dbs []string
@@ -216,34 +263,13 @@ func restoreAndCheck(lsBin, cfgPath, dbPath, tmpDir string) error {
// CheckRestoredDB opens a restored file read-only and proves it is a sane
// SQLite database: integrity_check == ok and at least one table present.
-func CheckRestoredDB(path string) error {
- db, err := sql.Open("sqlite", "file:"+path+"?mode=ro")
- if err != nil {
- return err
- }
- defer db.Close()
- db.SetMaxOpenConns(1)
- // busy_timeout first, always (Global Constraints) — and on a garbage file
- // this first statement is what fails loudly instead of a later query.
- if _, err := db.Exec("PRAGMA busy_timeout=5000"); err != nil {
- return fmt.Errorf("open: %v", err)
- }
- var res string
- if err := db.QueryRow("PRAGMA integrity_check").Scan(&res); err != nil {
- return fmt.Errorf("integrity_check: %v", err)
- }
- if res != "ok" {
- return fmt.Errorf("integrity_check: %s", res)
- }
- var tables int
- if err := db.QueryRow("SELECT count(*) FROM sqlite_master WHERE type='table'").Scan(&tables); err != nil {
- return fmt.Errorf("schema count: %v", err)
- }
- if tables == 0 {
- return fmt.Errorf("restored database has no tables")
- }
- return nil
-}
+//
+// The implementation moved to internal/lease so the WAKE path can run the
+// same check without importing this package — one definition of "restores
+// clean", used by both the daily sweep and the moment an instance is about to
+// serve a restored file. The name stays here because the ops verb and its
+// tests are its documented door.
+func CheckRestoredDB(path string) error { return lease.CheckDB(path) }
// writeIfChanged writes content to path only when it differs, and reports
// whether it wrote. The byte-identical no-op is what lets a timer re-run the
diff --git a/internal/ops/ops.go b/internal/ops/ops.go
index bf7d6668e..80842d064 100644
--- a/internal/ops/ops.go
+++ b/internal/ops/ops.go
@@ -5,12 +5,18 @@
package ops
import (
+ "context"
+ "errors"
"flag"
"fmt"
"log"
"os"
+ "time"
+ "github.com/carlosframework/platform/internal/activator"
+ "github.com/carlosframework/platform/internal/dbverify"
"github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
)
const (
@@ -30,6 +36,8 @@ func Run(args []string) {
runLitestream(args)
case "restore-verify":
runRestoreVerify(args)
+ case "hibernate-verify":
+ runHibernateVerify(args)
case "", "-h", "--help", "help":
usage()
if verb == "" {
@@ -50,6 +58,12 @@ func usage() {
so the caller knows whether to restart replication
carlos ops restore-verify restore every configured replica from S3 and
integrity-check each (run by the daily timer)
+ carlos ops hibernate-verify
+ the same drill for the replicas the config does
+ NOT name: hibernating instances, whose
+ replication the activator owns under its lease.
+ Publishes a per-host verdict to the deployment
+ bucket so the result is readable off-box.
No verb for your task? Extend ops in a small PR — never a hand-run sequence
reconstructed from docs.
@@ -137,3 +151,135 @@ func runRestoreVerify(args []string) {
log.Fatal(err)
}
}
+
+// defaultHibernateMax caps one drill run. Each verification is a full restore
+// from S3, so an uncapped nightly pass over a growing fleet is a bill that
+// grows with it (the request-cost discipline internal/lease's LeaseTTL comment
+// records). 25 covers today's fleets in a single run and keeps a large one
+// fully covered within days, oldest-first.
+const defaultHibernateMax = 25
+
+// runHibernateVerify — restore-drill the hibernating instances' replicas, the
+// set /etc/litestream.yml deliberately omits, and publish the verdicts.
+//
+// Exit status carries corruption only: a damaged replica fails the run (the
+// oneshot unit lands in 'failed' where fleet sweeps see it), while replicas
+// the pass could not decide are recorded as unverified and do NOT fail it.
+// Conflating the two would either cry wolf on every S3 blip or — worse —
+// train an operator to ignore a red run.
+func runHibernateVerify(args []string) {
+ fs := flag.NewFlagSet("ops hibernate-verify", flag.ExitOnError)
+ regPath := fs.String("registry", "", "registry database (default <data>/registry.db)")
+ data := fs.String("data", "", "data directory (default $CARLOS_DATA_DIR or "+defaultDataDir+")")
+ max := fs.Int("max", defaultHibernateMax, "most replicas to restore in one run (0 = no cap)")
+ _ = fs.Parse(args)
+
+ ctx := context.Background()
+ cfg, err := activator.ConfigFromEnv()
+ if err != nil {
+ log.Fatalf("hibernate-verify: %v", err)
+ }
+ if cfg == nil {
+ // Hibernation disabled on this box: there is no such replica set, so
+ // there is nothing to drill and nothing is wrong.
+ fmt.Println("hibernate-verify: CARLOS_HIBERNATE_BUCKET unset — no hibernating replicas on this box")
+ return
+ }
+
+ dir := dataDir(*data)
+ rp := *regPath
+ if rp == "" {
+ rp = dir + "/registry.db"
+ }
+ reg, err := registry.Open(rp)
+ if err != nil {
+ log.Fatalf("hibernate-verify: open registry %s: %v", rp, err)
+ }
+ hosts, err := HibernatingHosts(reg)
+ if err != nil {
+ log.Fatalf("hibernate-verify: %v", err)
+ }
+ if len(hosts) == 0 {
+ fmt.Println("hibernate-verify: no hibernating instances registered on this box")
+ return
+ }
+
+ // The prior record does two jobs: it orders this run (stalest first) and
+ // it carries the verdicts this run's cap will not revisit.
+ store, storeErr := openDeployStore(ctx)
+ var prior *dbverify.Record
+ if storeErr == nil {
+ if prior, err = dbverify.Load(ctx, store); err != nil {
+ log.Printf("hibernate-verify: reading %s: %v (continuing; this run starts the record over)", dbverify.Key, err)
+ prior = nil
+ }
+ }
+
+ tmp, err := os.MkdirTemp("", "carlos-hibernate-verify")
+ if err != nil {
+ log.Fatalf("hibernate-verify: %v", err)
+ }
+ defer os.RemoveAll(tmp)
+
+ now := time.Now()
+ results := VerifyHibernating(ctx, cfg, OldestFirst(hosts, prior), *max, now, tmp)
+ rec := dbverify.Merge(prior, boxLabel(), now, hosts, results)
+
+ // Publication is best-effort and LOUD when it fails: the drill's value is
+ // that someone can read the result without a session on this box, so a
+ // silent failure to publish would leave exactly the blind spot this verb
+ // exists to remove.
+ switch {
+ case storeErr != nil:
+ log.Printf("hibernate-verify: NOT PUBLISHED (%v) — the verdicts below exist only in this log", storeErr)
+ default:
+ if err := dbverify.Publish(ctx, store, rec); err != nil {
+ log.Printf("hibernate-verify: NOT PUBLISHED: %v — the verdicts below exist only in this log", err)
+ } else {
+ fmt.Printf("hibernate-verify: published %s (%d hosts, %d unverified)\n", dbverify.Key, len(rec.Hosts), rec.Unverified())
+ }
+ }
+
+ if n := rec.Failures(); n > 0 {
+ os.RemoveAll(tmp) // log.Fatal skips the defer
+ log.Fatalf("hibernate-verify: %d replica(s) did not restore clean", n)
+ }
+ // Count what was PROVEN, not what was visited: an unverified host is not
+ // coverage, and a summary that said otherwise would be the same comfortable
+ // lie this verb exists to remove.
+ proven := 0
+ for _, v := range results {
+ if v.OK {
+ proven++
+ }
+ }
+ fmt.Printf("hibernate-verify: %d of %d hibernating replicas proven this run (%d visited, %d in the fleet)\n",
+ proven, len(hosts), len(results), len(hosts))
+}
+
+// boxLabel names the writing box in the published record. The hostname is
+// enough — the record is informational, unlike the cold-standby snapshot
+// sidecar, whose Box field guards against a standby clobbering the dead box's
+// registry.
+func boxLabel() string {
+ h, err := os.Hostname()
+ if err != nil || h == "" {
+ return "unknown"
+ }
+ return h
+}
+
+// openDeployStore resolves the deployment bucket the same way the agent's
+// bucket/dir env vars do. A fleet box that reaches its deployment through the
+// console door instead (openDeployStoreEnv's third case in internal/host) is
+// NOT covered here and is told so out loud rather than publishing nothing
+// quietly.
+func openDeployStore(ctx context.Context) (release.Store, error) {
+ if b := os.Getenv("CARLOS_DEPLOYMENT_BUCKET"); b != "" {
+ return release.NewS3Store(ctx, b)
+ }
+ if d := os.Getenv("CARLOS_DEPLOYMENT_DIR"); d != "" {
+ return release.NewDirStore(d), nil
+ }
+ return nil, errors.New("neither CARLOS_DEPLOYMENT_BUCKET nor CARLOS_DEPLOYMENT_DIR is set")
+}
diff --git a/internal/ops/ops_test.go b/internal/ops/ops_test.go
index 7f5fc7739..97a3b6355 100644
--- a/internal/ops/ops_test.go
+++ b/internal/ops/ops_test.go
@@ -4,10 +4,14 @@ import (
"database/sql"
"os"
"path/filepath"
+ "slices"
"strings"
"testing"
+ "time"
"github.com/carlosframework/platform/internal/registry"
+
+ _ "modernc.org/sqlite"
)
// TestConfigGenSkipsHibernatingRoutes pins Eleven's two-replicators lesson in
@@ -222,3 +226,86 @@ func TestCheckRestoredDBRejectsEmptySchema(t *testing.T) {
t.Fatal("CheckRestoredDB accepted a non-SQLite file")
}
}
+
+// The console's own SQLite — accounts, ownership, sessions, the deployment's
+// Keymail credentials — lives one directory BELOW dataDir, where FleetDBs'
+// flat <data>/*.db glob cannot see it. Losing it loses who everyone is, so it
+// is included explicitly.
+func TestFleetDBsIncludesConsoleDB(t *testing.T) {
+ dir := t.TempDir()
+ reg, err := registry.Open(filepath.Join(dir, "registry.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ // Absent console: nothing invented, and no error.
+ dbs, err := FleetDBs(reg, dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, p := range dbs {
+ if strings.Contains(p, "console") {
+ t.Fatalf("FleetDBs listed a console DB on a box that has none: %v", dbs)
+ }
+ }
+
+ consoleDB := filepath.Join(dir, "console", "console.db")
+ if err := os.MkdirAll(filepath.Dir(consoleDB), 0o750); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(consoleDB, []byte("x"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+
+ dbs, err = FleetDBs(reg, dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !slices.Contains(dbs, consoleDB) {
+ t.Fatalf("console DB missing from the replication set: %v", dbs)
+ }
+}
+
+// CONSOLE_DB is the console's own env var for its database; a box that moved
+// it must not silently drop out of replication.
+func TestFleetDBsHonoursConsoleDBEnv(t *testing.T) {
+ dir := t.TempDir()
+ reg, err := registry.Open(filepath.Join(dir, "registry.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ elsewhere := filepath.Join(t.TempDir(), "moved-console.db")
+ if err := os.WriteFile(elsewhere, []byte("x"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ t.Setenv("CONSOLE_DB", elsewhere)
+
+ dbs, err := FleetDBs(reg, dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !slices.Contains(dbs, elsewhere) {
+ t.Fatalf("relocated console DB missing from the replication set: %v", dbs)
+ }
+}
+
+// The generated config must state the recovery window rather than inherit
+// litestream's 24h snapshot retention by silence. A day is the shortest
+// useful window there is: a bad migration noticed on Tuesday is unrecoverable
+// if it landed on Sunday.
+func TestConfigPinsSnapshotRetention(t *testing.T) {
+ cfg := LitestreamConfig([]string{"/data/registry.db"}, "b", "p", "eu-west-1")
+ if !strings.Contains(cfg, "snapshot:") {
+ t.Fatalf("generated config states no snapshot policy — it inherits a 24h window silently:\n%s", cfg)
+ }
+ if !strings.Contains(cfg, "retention: "+SnapshotRetention.String()) {
+ t.Fatalf("generated config does not pin retention to %s:\n%s", SnapshotRetention, cfg)
+ }
+ if SnapshotRetention <= 24*time.Hour {
+ t.Fatalf("SnapshotRetention = %s, which is no better than litestream's default", SnapshotRetention)
+ }
+ // And the round-trip the restore-verify verb depends on must survive it.
+ if got := ParseLitestreamDBs(cfg); len(got) != 1 || got[0] != "/data/registry.db" {
+ t.Fatalf("ParseLitestreamDBs broke on the retention stanza: %v", got)
+ }
+}