amadan
litestream: warn when the box's binary ignores the snapshot policy
Verified on the fleet: the flagship's litestream is v0.3.13 (Oct 2023), which parses the new snapshot stanza, exits 0, lists the databases — and drops the retention on the floor. 0.5.15 does the same with a misspelled key. So a successful parse is not evidence, and a config that STATES a recovery window nothing enforces is worse than one that states none: it reads as a decision someone made and checked. carlos ops litestream now reads the binary's version and says so, loudly, when the window it just wrote will not be honoured. Loud but not fatal — a box on an old litestream still wants its database SET kept current, and that set is what carries console.db; refusing would withhold the fix that works to protest the one that doesn't. Also records the fleet's real state in the design note. The headline is that `carlos ops litestream` is NOT the generator on either box (flagship: a hand-rolled kass-tenant config; tito: the legacy `titogo ops litestream`), so the config-side fixes here change nothing until the verb takes that job over — while hibernate-verify, which reads the registry and the hibernate env rather than any generated config, works the moment it ships.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4 files changed,
+142
−1
docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md+37 −1internal/ops/litestream.go+51 −0internal/ops/ops.go+26 −0internal/ops/ops_test.go+28 −0
diff --git a/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md b/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md| index 08608c091..54b5cdf00 100644 |
| --- a/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md |
| +++ b/docs/superpowers/specs/2026-08-15-sqlite-corruption-defence-design.md |
| @@ -113,7 +113,43 @@ files that today need no process, and gains nothing on corruption. |
| Revisit only if the shape changes: multi-region reads of one tenant database, |
| or multi-writer becoming a real constraint rather than a deliberate one. |
| -## 5. Known follow-ups |
| +## 5. What the fleet actually looks like (verified 2026-08-15) |
| + |
| +Read-only inspection of both platform boxes, before any of this is deployed: |
| + |
| +| | flagship (`carlos-flagship-ie`) | tito (`carlos-tito-ie`) | |
| +| --- | --- | --- | |
| +| litestream binary | **v0.3.13** (dated Oct 2023) | v0.5.15 | |
| +| `/etc/litestream.yml` generator | hand-rolled, "GENERATED for the kass tenant" | legacy **`titogo ops litestream`** | |
| +| databases configured | 5 | 54 | |
| +| `console.db` on disk | yes, 188 KB | yes, 188 KB | |
| +| `console.db` replicated | **NO** | **NO** | |
| +| snapshot policy | none → 24h default | none → 24h default | |
| +| `carlos-litestream.service` | running but **not enabled** (a reboot ends replication silently) | enabled | |
| +| `carlos-restore-verify.timer` | enabled | **disabled** — 54 replicas, drilled by nothing | |
| +| hibernating routes | 6 of 31 | 4 of 47 | |
| +| carlos binary | c42ec0c | c42ec0c | |
| + |
| +Three consequences worth stating plainly: |
| + |
| +1. **`carlos ops litestream` is not the generator on either box.** The config |
| + fixes in this branch — `console.db` and the snapshot policy — are correct |
| + code that changes nothing on the fleet until the verb takes that job over. |
| + The `console.db` hole is live on both boxes TODAY and merging this does not |
| + close it. |
| +2. **`hibernate-verify` is independent of all of that.** It reads the registry |
| + and `CARLOS_HIBERNATE_*` and uses the embedded litestream library — no |
| + generated config, no box binary. Both boxes have the env set and 6 + 4 |
| + hibernating routes, so that half works the moment it ships. |
| +3. **A stated policy is not an enforced one.** The flagship's v0.3.13 parses |
| + the snapshot stanza, exits 0, lists the databases — and ignores the |
| + retention. 0.5.15 does the same with a misspelled key. Parsing proves |
| + nothing, so `carlos ops litestream` now checks the BINARY VERSION and warns |
| + loudly when the window it just wrote will not be honoured |
| + (`SnapshotPolicySupported`). Loud, not fatal: an old box still wants its |
| + database set kept current, and that is what carries `console.db`. |
| + |
| +## 6. Known follow-ups |
| - **Console surface.** `control/restore-verify.json` is read from the bucket |
| today, like `control/registry-snapshot.json`. A console/API view of "last |
diff --git a/internal/ops/litestream.go b/internal/ops/litestream.go| index 05c65c862..bc703c486 100644 |
| --- a/internal/ops/litestream.go |
| +++ b/internal/ops/litestream.go |
| @@ -27,6 +27,7 @@ import ( |
| "os/exec" |
| "path/filepath" |
| "sort" |
| + "strconv" |
| "strings" |
| "time" |
| @@ -64,6 +65,56 @@ const ( |
| SnapshotRetention = 7 * 24 * time.Hour |
| ) |
| +// SnapshotPolicySupported reports whether `litestream version` output names a |
| +// binary that HONOURS the snapshot stanza LitestreamConfig writes. The policy |
| +// arrived with litestream 0.5; 0.3 and 0.4 parse the stanza and drop it on the |
| +// floor. |
| +// |
| +// This exists because parsing is not evidence. Verified on the fleet |
| +// 2026-08-15: the flagship's litestream v0.3.13 accepts the generated config, |
| +// exits 0, lists the databases — and ignores the retention entirely. So does |
| +// 0.5.15 when handed a misspelled key. A config that STATES a recovery window |
| +// nothing enforces is worse than one that states none, because it reads as a |
| +// decision someone made and checked. |
| +// |
| +// Unparseable output is UNSUPPORTED, never assumed-fine: the whole point is to |
| +// refuse to claim a window we cannot show is real. |
| +func SnapshotPolicySupported(versionOutput string) bool { |
| + f := strings.Fields(strings.TrimSpace(versionOutput)) |
| + if len(f) == 0 { |
| + return false |
| + } |
| + v := strings.TrimPrefix(f[len(f)-1], "v") // "litestream v0.5.15" or "0.5.15" |
| + major, rest, ok := strings.Cut(v, ".") |
| + if !ok { |
| + return false |
| + } |
| + minor, _, _ := strings.Cut(rest, ".") |
| + maj, err := strconv.Atoi(major) |
| + if err != nil { |
| + return false |
| + } |
| + min, err := strconv.Atoi(minor) |
| + if err != nil { |
| + return false |
| + } |
| + if maj > 0 { |
| + return true // a future 1.x must not regress to "unsupported" |
| + } |
| + return min >= 5 |
| +} |
| + |
| +// litestreamVersionOutput runs `<bin> version`. A binary that is missing or |
| +// will not answer yields "", which SnapshotPolicySupported treats as |
| +// unsupported. |
| +func litestreamVersionOutput(bin string) string { |
| + out, err := exec.Command(bin, "version").CombinedOutput() |
| + if err != nil { |
| + return "" |
| + } |
| + return string(out) |
| +} |
| + |
| // LitestreamConfig renders the generated /etc/litestream.yml. Deliberately |
| // minimal — path + replica URL + region, plus the snapshot policy above — so |
| // it can't rot against litestream's config surface; other tuning belongs in |
diff --git a/internal/ops/ops.go b/internal/ops/ops.go| index 80842d064..19806a7d4 100644 |
| --- a/internal/ops/ops.go |
| +++ b/internal/ops/ops.go |
| @@ -11,6 +11,7 @@ import ( |
| "fmt" |
| "log" |
| "os" |
| + "strings" |
| "time" |
| "github.com/carlosframework/platform/internal/activator" |
| @@ -94,6 +95,7 @@ func runLitestream(args []string) { |
| prefix := fs.String("prefix", "", "key prefix inside the bucket for this fleet (required)") |
| region := fs.String("region", "", "S3 region for the replica stanza") |
| out := fs.String("out", defaultConfigPath, "config file to write") |
| + lsBin := fs.String("litestream-bin", "litestream", "litestream binary to check the snapshot policy against (pinned release "+litestreamVersion+")") |
| _ = fs.Parse(args) |
| if *bucket == "" || *prefix == "" { |
| @@ -122,6 +124,20 @@ func runLitestream(args []string) { |
| log.Fatalf("litestream: write %s: %v", *out, err) |
| } |
| fmt.Fprintf(os.Stderr, "litestream: %d databases -> s3://%s/%s/ (%s)\n", len(dbs), *bucket, *prefix, *out) |
| + |
| + // The config now STATES a recovery window, so say plainly when the binary |
| + // on this box will ignore it. Loud, not fatal: a box on an old litestream |
| + // still wants its database SET kept current (that is what carries |
| + // console.db), and refusing here would withhold the fix that does work in |
| + // order to protest the one that does not. |
| + if v := litestreamVersionOutput(*lsBin); !SnapshotPolicySupported(v) { |
| + fmt.Fprintf(os.Stderr, `litestream: WARNING — this box's litestream (%s) predates the snapshot |
| + policy (0.5+), so the "snapshot: retention: %s" this file now states is |
| + IGNORED and the real recovery window stays at litestream's 24h default. |
| + The database set below is still correct and still replicated. Upgrade the |
| + binary to %s to make the stated window real. |
| +`, strings.TrimSpace(firstLine(v)), SnapshotRetention, litestreamVersion) |
| + } |
| // The one contract line on stdout: the calling unit restarts the |
| // replicator on "changed" and leaves it alone on "unchanged". |
| if wrote { |
| @@ -152,6 +168,16 @@ func runRestoreVerify(args []string) { |
| } |
| } |
| +// firstLine is the version banner's first line, or a stand-in when the binary |
| +// answered nothing at all. |
| +func firstLine(s string) string { |
| + if s = strings.TrimSpace(s); s == "" { |
| + return "version unknown — binary missing or unreadable" |
| + } |
| + line, _, _ := strings.Cut(s, "\n") |
| + return line |
| +} |
| + |
| // defaultHibernateMax caps one drill run. Each verification is a full restore |
| // from S3, so an uncapped nightly pass over a growing fleet is a bill that |
| // grows with it (the request-cost discipline internal/lease's LeaseTTL comment |
diff --git a/internal/ops/ops_test.go b/internal/ops/ops_test.go| index 97a3b6355..1a07842e5 100644 |
| --- a/internal/ops/ops_test.go |
| +++ b/internal/ops/ops_test.go |
| @@ -309,3 +309,31 @@ func TestConfigPinsSnapshotRetention(t *testing.T) { |
| t.Fatalf("ParseLitestreamDBs broke on the retention stanza: %v", got) |
| } |
| } |
| + |
| +// A config that STATES a snapshot policy the installed binary silently |
| +// ignores is worse than one that states nothing: it reads as a decision |
| +// someone made. litestream 0.3.13 (still on the flagship, verified 2026-08-15) |
| +// parses the stanza and drops it on the floor, and 0.5 accepts unknown keys |
| +// just as quietly — so parsing proves nothing and the VERSION is the only |
| +// signal there is. |
| +func TestSnapshotPolicySupported(t *testing.T) { |
| + cases := []struct { |
| + out string |
| + want bool |
| + }{ |
| + {"v0.3.13", false}, // the flagship's binary, dated 2023 |
| + {"0.3.13", false}, // some builds print no leading v |
| + {"v0.4.9", false}, // still pre-snapshot-policy |
| + {"0.5.15", true}, // tito's binary, and the repo's pin |
| + {"v0.5.14", true}, // the vendored library version |
| + {"v1.0.0", true}, // a future major must not regress to "unsupported" |
| + {"litestream v0.5.15\n", true}, |
| + {"", false}, // unknown is treated as unsupported: warn, never assume |
| + {"not-a-version", false}, |
| + } |
| + for _, c := range cases { |
| + if got := SnapshotPolicySupported(c.out); got != c.want { |
| + t.Errorf("SnapshotPolicySupported(%q) = %v, want %v", c.out, got, c.want) |
| + } |
| + } |
| +} |