carlos / platform Public

Astra (gpt-6-astra, high effort) read the design above against the worktree and litestream v0.5.14. Twelve findings, six P1. I checked the cited source for 1, 2, 3 and 8 and they hold. The findings are pasted unedited below my dispositions.

What this changes

The design above is withdrawn as written. Slice A (cap, backoff, restarts rule) is unaffected and goes ahead on litestream-oom-284 alone. Slice B moves to its own branch and gets a revised design before more is built. The restarts alert is the one that would have caught the #284 incident, within about a minute of the loop starting.

Dispositions

| # | Finding | Disposition | |---|---|---| | 1 | LastSuccessfulSyncAt advances while WAL capture fails | Accepted. litestream's own bookkeeping answers "has what was captured been shipped", not "is the replica current with the database". Rev 2 says so in its definitions, takes capture health from DB.SyncDiagnostic for the embedded replicator, and never calls the box-wide binary "level" on position equality alone. | | 2 | Idle daily-class replica pages after 24h15m | Accepted. The replica monitor waits two ticks when idle. Rev 2 measures outstanding captured changes (local position against replica position), not time since the last sync. | | 3 | A nil Close() is not proof of a final sync | Accepted. unitRep.Close is a no-op and claimOwnedRep.Close returns nil once writers-stop has begun. The Parked state I built from stopHistory.replicaClosed is wrong for both and is removed until the owning replicator gives an explicit receipt. | | 4 | Max L0 TXIDs prove neither initialisation nor lineage | Accepted. Zero or missing positions are "unknown", never "level". | | 5 | A healthy subset can clear an alert for a database that became unreadable | Accepted. Rev 2 tracks state per database; an overdue database with no fresh reading stays overdue until a reading says level or it leaves the config. A checker that stops reporting gets its own signal. | | 6 | Host-only joins report the previous owner after a move | Accepted. A box publishes only for hosts with a route row in its own registry; claims-on needs writer identity in the observation and is recorded as an activation item, not built here. | | 7 | The two-minute gate discards the known outage age | Accepted. The earliest unresolved outage is kept until catch-up is observed against the position at recovery. | | 8 | Claims-on passes interval 0 whatever the class | Accepted. The replicator reports the interval it actually runs with. | | 9 | The parser cannot establish coverage for handwritten configs | Accepted. Only a generated config is checked; anything else is "coverage unknown". | | 10 | Pagination and bounded execution | Accepted. Library iterator, no metadata HEADs, bounded concurrency, per-database and per-run deadlines, explicit partial failures. | | 11 | restore-verify's hardening would break the checker | Accepted. AF_UNIX, a writable monitor directory, atomic replace. | | 12 | Wants= does not arm a running box | Accepted. The timer is armed fleet-wide like the relay, with ConditionPathExists=/etc/litestream.yml on the service. |

One thing rev 2 will add that the review did not ask for

restore-verify's currency check never judges a database that is written continuously (the 15-minute quiet rule). Keeping the previous drill's change counter per database would close that: a source whose counter moved between two drills while the replica's did not is stale, quiet or not. It is independent of litestream's bookkeeping, which is the property finding 1 says the cheap signal lacks.

---

Findings, unedited

1. P1 — LastSuccessfulSyncAt() can advance while committed changes remain unshipped.

Replica.Sync reads a local LTX position once, uploads through that position, then timestamps completion (Litestream replica.go:156). That position represents LTX files, not the current SQLite WAL (db.go:559). WAL capture runs independently and backs off on errors (db.go:2705).

Failure: after reaching TXID 100, WAL capture fails persistently while the application continues writing. Uploads see local=remote=100 and keep refreshing the timestamp. The embedded source reports fresh indefinitely. A box-wide process surviving two minutes has the same blind spot. Even without errors, changes captured during a long upload remain pending when the completion timestamp advances.

Smallest fix: measure successful WAL capture and upload separately. Associate freshness with the captured watermark and its observation time; require healthy capture before asserting “level.” For the binary, uptime alone cannot provide that evidence: add capture-health evidence or report position equality with freshness unknown.

1. P1 — An idle daily-class replica can falsely page for almost a day.

The replica monitor waits for a tick at the top of its loop, then—when positions are equal—waits for another tick or notification (replica.go:338). The configured daily interval is 24 hours (activator.go:348).

Failure: a fully synchronized, unchanged daily database can refresh its successful-sync timestamp only every 48 hours. Subtracting one 24-hour interval makes it page after 24h15m, despite no missing changes, then clear at the next idle sync. Batched instances have the same doubled-idle-wait shape, though its five-minute excess alone does not cross the threshold.

Smallest fix: determine lag from outstanding captured changes, with zero lag when capture is healthy and positions match. Do not use time since the last uploader invocation as the sole lag measurement or compensate by widening the fixed alert thresholds.

1. P1 — A nil Close() result is not universally proof of a final sync.

The activator’s Rep abstraction has implementations with different semantics. unitRep.Close is an unconditional no-op (backend_unit.go:234). With claims enabled, claimOwnedRep.Close deliberately returns nil once writers-stop has begun, leaving replication cleanup to its owner (claimwakeassembly.go:821); the actual close completes separately (claimwakeassembly.go:504).

Failure: a claim-controlled stop records a clean parked replica before the final upload completes—or before that upload fails. A unit-backed tenant can similarly acquire a false embedded freshness reading merely by sleeping.

Smallest fix: require an explicit final-sync receipt from the owning replicator, scoped to its writer identity. Exclude unit-backed replication from the embedded source. Keep a proven parked replica at zero outstanding lag; retain its completion timestamp for display.

1. P1 — Maximum L0 TXIDs prove neither initialization nor a consistent replica lineage.

Litestream itself requires a nonzero, equal position for SyncStatus.InSync (db.go:656). When remote is ahead, initialization removes local L0 files and downloads a remote baseline; a subsequent capture must reconcile that baseline with the actual database (db.go:1400).

More seriously, the existing snapshot-only experiment demonstrates equal local/remote L0 positions while a cold restore loses new writes: numbering restarts below the existing snapshot (facts_test.go:366).

Failure: an uninitialized database, a restored database awaiting reconciliation, or the demonstrated snapshot-only lineage satisfies “no local files beyond remote” without having a fresh recoverable replica.

Smallest fix: define explicit uninitialized, reconciling, and unsupported-lineage states. Require positive positions and successful initialization/capture before equality means level. Validate the snapshot/L0 anchor when enrolling or resetting a lineage. Treat unexplained gaps as unknown; do not require contiguous retained L0 history, because legitimate compaction removes it.

1. P1 — A healthy subset can clear an alert for a database that became unreadable.

The design specifies the worst numeric reading but no coverage requirement ([design-290.md:28](design-290.md:28)). The existing evaluator clears on any supplied value below the clear line and leaves state untouched only for nil (alert.go:230).

Failure: database A is 30 minutes behind and fires. Its next LIST fails, while database B—or the other replicator—reports zero. Taking the maximum of available readings clears A’s alert without evidence of recovery. Conversely, a checker that never produces a reading never fires on a box without embedded replicas. Removing the last monitored database after an alert can leave the rule firing indefinitely.

Smallest fix: carry expected inventory and coverage alongside values. Known overdue evidence may fire with incomplete coverage; clearing requires recovery or explicit removal of the affected databases. Distinguish “not configured” from “configured but unreadable.” Give checker failure its own health signal while retaining the fixed nil-reading behavior.

1. P1 — Host-only joins can report the previous owner’s freshness after a move.

The proposed model copies route_db_bytes, whose join simply overwrites values by hostname as boxes are traversed (apiinstances.go:597). Claims distinguish incarnation, installation stamp, writer session, box, boot, and controller (claimstop_writers.go:28).

Failure: after a move, the source still publishes its clean parked observation while the destination has unshipped writes. Both box snapshots are current; traversal order decides which hostname entry wins. Host reuse has the same problem.

Smallest fix: include writer/replica identity in each observation and select against current ownership. Expire observations on ownership changes. Preserve them across a process-only zero-downtime swap when the replicator identity stays unchanged. Forward the observation through claimOwnedRep; adding a method only to lease.Replicator leaves that wrapper invisible.

1. P2 — The two-minute uptime gate measures restart timing, not recovery, and can discard the known outage age.

The proposed gate dates failure from its first observation and resumes L0 comparison after two minutes ([design-290.md:20](design-290.md:20)). LTX files receive their timestamps when capture writes them, potentially long after the application committed the changes (db.go:1941).

Failure: after an hour-long outage, the restarted replicator captures old WAL into newly dated L0 files. At two minutes, switching from outage age to those mtimes can clear the alert although those old changes remain unshipped. Separately, config restarts one minute before each five-minute sample can produce a continuous “not replicating” alarm even when replication catches up between restarts.

Smallest fix: use uptime as startup context, not proof. Preserve the earliest unresolved outage across restarts and capture until catch-up is established. Track boot/invocation identity and distinguish startup grace from failed recovery. Describe L0 mtime as capture time, not the exact oldest outstanding commit time.

1. P2 — Claims-on replication currently uses a different interval from the route’s declared class.

The legacy path passes the class-derived interval (activator.go:2384). The claims path explicitly passes 0, selecting stream behavior (claimwakeassembly.go:492).

Failure: a claims-on route marked daily actually streams, but subtracting its declared 24-hour interval hides a stalled stream for a day.

Smallest fix: report the interval actually used by the replicator alongside its observation. This preserves the decision not to change the claims-on replication path.

1. P2 — The generator’s parser cannot establish monitoring coverage for handwritten configurations.

ParseLitestreamDBs accepts only the literal prefix " - path: " and does not parse replica destinations (litestream.go:165). Litestream supports environment expansion, YAML parsing, directory entries, and custom metadata paths (main.go:622, main.go:705).

Failure: valid alternate indentation omits databases; quoted paths remain quoted; meta-path makes the hardcoded local directory wrong. A partial parse can appear to be complete coverage.

Smallest fix: parse the supported configuration semantically, including destination and metadata path, using the service’s environment. Explicitly reject unsupported configurations as unknown coverage rather than returning an empty or partial success.

1. P2 — “One LIST per database” needs pagination and bounded execution.

Litestream’s S3 iterator uses a paginator and checks errors while fetching subsequent pages (replica_client.go:1404, replica_client.go:1494).

Failure: with more than one page of retained L0 objects, a literal single request misses the newest TXID and reports false lag. Unbounded pagination/retries across 1,000 databases can prevent publication before the 15-minute expiry, leaving only “no reading.”

Smallest fix: require complete pagination, propagate iterator errors, disable metadata HEAD requests, bound concurrency and per-database/run deadlines, and publish explicit partial failures. The baseline is 28,800 listing requests/day at 100 databases and 288,000 at 1,000 only when each listing fits one page; retries and additional pages increase it.

1. P2 — Copying restore-verify’s hardening would prevent the checker from querying systemd or writing its result.

Restore-verify uses ProtectSystem=strict, has no writable /data/monitor exception, and permits only AF_INET AF_INET6 (carlos-restore-verify.service:35). The agent, which already invokes systemctl show, permits AF_UNIX (carlos-edge.service:51).

Failure: an otherwise reasonable copied unit cannot connect to the local system bus or publish replicas.json. The resulting missing readings do not page.

Smallest fix: specify AF_UNIX, a narrowly writable monitor directory with suitable ownership, and atomic result replacement. Keep database metadata read-only. Systemd can load the existing environment file before dropping privileges; granting the checker root is unnecessary.

1. P2 — Adding Wants=timer does not arm the timer on an already-running deployment.

Bootstrap explicitly starts selected timers and deliberately does not restart ordinary services (bootstrap.go:229). Litestream is outside its enabled/armed list (bootstrap.go:167).

Failure: an upgrade installs the new dependency and reloads systemd while litestream continues running. No start transaction pulls in the timer; monitoring remains absent until a later restart or reboot.

Smallest fix: add an idempotent rollout/adoption step that starts the timer when box-wide replication is configured. Keep the timer running through replicator failures; do not couple its lifetime to the failing service with BindsTo= or restart propagation.

Checked and found sound:

• Normal Litestream retention preserves the newest remote L0 file. Local cleanup targets files already represented remotely; ordinary compaction does not erase unshipped L0 evidence (db.go:2627, facts_test.go:466).
• The two-minute gate catches the reported four-second crash loop, provided its unresolved-failure timestamp survives successive starts.
• Upload errors do not advance the successful-sync timestamp. They reset the replica position and trigger retry/backoff; this correctly exposes upload failure, subject to the capture and idle-timing problems above.
• The 15-minute fire / 10-minute clear hysteresis is sound for complete, correctly calculated readings. An unconfigured source need not contribute a guessed zero.
• Clean sleeping replicas should retain zero outstanding lag. The Parked state added during this review addresses the design’s literal timestamp-aging problem, though final-sync proof still needs finding 3’s correction.
• The existing restore checks can remain unchanged.

This was source inspection only: no edits, builds, tests, or subagents. The worktree changed concurrently; I checked the arriving lease, activator, and alert changes before finishing.

Slice B design, rev 2

Rev 1 is withdrawn. This answers the twelve findings above; each change is marked (astra N). Nothing here is built yet. The branch replica-freshness-284 holds rev 1's code and will be reworked to this.

0. What the signal claims, and what it does not

The cheap signal answers one question per database: is everything this box has captured from the database in the replica, and is capture itself working? It rests on litestream's own bookkeeping. It does not restore anything, so it cannot prove a replica restores, or that litestream captured correctly. restore-verify and hibernate-verify stay the proof of that, and §6 closes their one blind spot. (astra 1)

Per database, exactly one state:

• shipped, as of T: capture last succeeded at T, the local position is non-zero, and the replica position equals it.
• behind, since T: the box holds captured changes the replica does not, or capture is failing. T is when the checker first saw the oldest outstanding thing, kept until it is seen resolved.
• empty: the database has no position and capture reports no error. Nothing exists to ship.
• unknown, with a reason: anything else. Zero or missing positions, an unreadable listing, a config the checker cannot read, a replicator that has not been up long enough to judge. (astra 4)

Lag is measured from outstanding changes, never from the time since the last upload attempt: a database whose positions match and whose capture is healthy has zero lag however long ago its last sync ran. That removes the idle daily-class false page. (astra 2) A database is overdue when it has been behind for longer than the interval its replicator actually runs with, plus 15 minutes.

1. The embedded replicator (hibernating instances, awake)

lease.Replicator gains one read-only method returning a snapshot, all from memory:

• local position (DB.Pos), replica position (Replica.Pos; zero before the first sync and after any upload error);
• capture health from DB.SyncDiagnostic: the last finished capture's time and error;
• the sync interval the replica was built with (Replica.SyncInterval), not the route's declared class. (astra 8)

The agent samples it each tick and keeps, per host, a short ring of (time, local position). State:

• capture's last finished run has an error, or no capture has finished for 3 monitor intervals plus litestream's backoff ceiling: behind since the first tick that saw it. (astra 1)
• local position zero, no capture error: empty.
• replica position ≥ local position, both non-zero: shipped as of the last capture.
• replica position known and below local: behind since the earliest ring sample whose local position exceeds the current replica position. A healthy stream replica that is one upload behind at every sample never accumulates lag, because each sample's backlog is a different position.
• replica position zero with a non-zero local position: behind since the first tick that saw it (litestream zeroes it on an upload error and recomputes it on the next attempt).

No parked state from Close(). unitRep.Close is a no-op and claimOwnedRep.Close returns nil once writers-stop has begun, so a nil close proves nothing in general. A sleeping host is published only when the agent itself saw that host's embedded replicator report shipped on the tick before its entry disappeared and the activator's stop history says the replica closed and the stop was certain. Otherwise a sleeping host has no entry: unknown. (astra 3)

Claims-on wakes use claimOwnedRep, which does not forward the method. Those hosts are unknown until it does; forwarding it, with the writer identity §4 needs, is recorded as an activation item on #250, not built here. (astra 3, 6)

2. The box-wide litestream binary

A oneshot carlos ops replica-check on a 5-minute timer, as carlos, with /data/litestream.env.

Coverage. It checks only a config with the generator's header, read with the generator's own parser (path, replica URL, region). Anything else is one result: unknown: config not generated, never a partial list. (astra 9)

Per database, through the litestream library's S3 client (the same one hibernate-verify uses), with the iterator run to completion, no metadata HEADs, 8 databases at a time, a 20-second deadline each and a 4-minute deadline for the run. A database that hits a deadline or an error is unknown with the error, and the run still publishes. (astra 10)

• replica position: the highest TXID in the replica's level 0 (litestream's calcPos rule);
• local position: the highest TXID among .<db>-litestream/ltx/0 filenames;
• either zero or missing: unknown. (astra 4)
• replica ahead of local: unknown: replica ahead (a restored box reconciling). (astra 4)
• local ahead: behind.
• equal: a candidate for shipped, subject to capture health below.

Capture health, without trusting uptime. Position equality says nothing if the replicator is not converting the WAL. Two independent checks, both required for shipped:

1. The WAL is not ahead of capture. If the database's -wal file holds frames (larger than its 32-byte header) and was modified more than 2 minutes after the newest local L0 file, changes are sitting uncaptured: behind. This predicate is only as good as two facts about litestream that I have read in its source but not tested: every WAL write is followed by a new L0 file within a monitor interval while capture works, and litestream's own truncating checkpoint leaves no frames behind without a following L0. Both get pinned as executable facts in internal/sim/litestreamfacts against the real library before this is built; if either fails, this check is dropped and equality reports unknown: capture not observed. (astra 1, 7)
2. The unit is steady. ActiveState=active, and NRestarts unchanged since the previous check. A unit seen restarting or inactive marks every database behind.

Sticky markers. The checker's state file keeps, per database, {since, need_txid} for any unresolved behind. (astra 7)

• A marker is created at the earliest evidence: the oldest unshipped L0 file's mtime (described as capture time, not commit time), or the check time for the WAL and unit predicates.
• While the cause is "not capturing", need_txid is open. At the first check where capture is healthy again it is set to the local position at that moment, so the alert covers everything written during the outage, including changes captured late into freshly dated L0 files.
• A marker is cleared only when the replica position reaches need_txid. A later backlog starts a new marker with its own evidence. An ordinary restart never clears one and never creates one by itself: a unit that is active with uptime under 2 minutes and an unchanged restart counter is "starting", and equality during it reports unknown: starting, which neither fires nor clears.

Result. /data/monitor/replicas.json, replaced atomically: checked_at, the unit's state, per-path state, positions, since and reason. (astra 11)

Units. carlos-replica-check.service has ConditionPathExists=/etc/litestream.yml, RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX, ProtectSystem=strict, ReadWritePaths=/data/monitor, EnvironmentFile=/data/litestream.env. Its timer joins enabledUnits, armed fleet-wide by bootstrap and by self-update like the relay; the condition makes it a no-op on a box with no box-wide replication. No Wants= from the replicator, and no lifetime coupling to it. (astra 11, 12)

Cost at one page per listing: 28,800 LISTs a day for 100 databases, 288,000 for 1,000 (about $1.44 a day). More pages or retries raise it. (astra 10)

3. From observations to one alert, without losing a database

The agent keeps a table of every database it has ever been told is behind, keyed by replicator and path or host. (astra 5)

• behind adds or keeps a row with its since.
• shipped or empty for that key removes the row.
• unknown, or no reading at all, keeps the row and lets its lag grow. An unreadable database cannot clear its own alert, and a healthy neighbour cannot clear it either.
• A row is dropped without a shipped only by explicit removal: the path leaves the generated config, or the host's route row leaves this box's registry.

The replica rule's reading is the worst overdue across the table, with the row's name as detail. An empty table reads 0, which clears. The reading is nil, leaving the rule untouched, only when the agent has no table yet (first ticks after start).

The checker has its own health signal. If /etc/litestream.yml exists and replicas.json is missing or its checked_at is older than three periods, the agent raises a second rule, replica-check, naming the last time it reported. Coverage that is absent is a page of its own, not a silent nil. A box with no box-wide config and no embedded replicas has an empty table and no replica-check expectation: nothing fires. (astra 5)

4. Publishing, and who may speak for a host

A box publishes a host's state in route_replicas only when that host has a route row in the box's own registry at publish time, and each entry carries the box's label and the replicator kind. The console takes a host's entry only from the box that serves it (the same answer ServedBy gives), never "last box read wins". That removes the stale source after a completed move. (astra 6)

It does not solve a move in flight under claims, where two boxes hold a row. Entries need the claim's writer identity for that; until claimOwnedRep forwards it, a claims-on host publishes nothing. (astra 6)

Wire shape, replacing rev 1's: {state, since?, as_of?, reason?, interval_secs, kind}.

5. Console

As rev 1, on the new shape: a replica object on the instance list API, a REPLICA column in carlos instances, a Replica column in the HTML table. unknown renders blank. The 15-minute "late" line is computed by the box and published as overdue, so the console and the alert cannot disagree.

6. Closing restore-verify's blind spot

Independent of everything above, and of litestream's bookkeeping. restore-verify judges currency by SQLite's change counter but skips any source modified in the last 15 minutes, so a database written continuously is never judged. It will keep each database's source and replica counters from its previous run in a small state file. A database whose source counter moved between two consecutive drills while the replica's did not is stale, quiet or not. A WAL-mode counter only advances at checkpoint, so this can miss, never false-alarm: the same property the existing check documents.

7. Order of work

1. The two litestream facts (§2 capture check) as tests. Their result decides whether the check exists.
2. restore-verify's previous-run comparison (§6). Small, independent, shippable alone.
3. The embedded snapshot and the agent's table (§1, §3).
4. The box-wide checker and its units (§2).
5. Publishing and the console (§4, §5).
6. Copy review.

Not in this branch

• Forwarding through claimOwnedRep and writer identity (activation item, #250).
• Acknowledging an alert. A per-database page.

Astra's second read of rev 2: NOT READY. Seven of the twelve round 1 findings are fixed or fixed for their original case; five are partly fixed; and rev 2 introduced seven new P1s and two P2s. Findings pasted below, local path prefixes shortened.

What the two rounds say, taken together

The findings are no longer about details. They say the cheap signal I designed cannot be made sound from where it sits:

• The box-wide binary gives an outside observer no positive evidence of capture. The WAL-versus-L0 predicate is wrong in both directions (new finding 3: a commit 30 seconds after the last L0 followed by a permanent capture failure never trips it; a long open transaction trips it on a healthy box). Astra's fix is the fallback rev 2 named: position equality is unknown: capture not observed. A checker that can say "behind" and "unknown" but never "shipped" is not the column #284 asked for.
• For the embedded replicator, sampling litestream's state from outside its own calls is incoherent. SyncDiagnostic is a current-operation slot that the next operation overwrites (new finding 1), and positions and completion times read separately can describe a capture that was never shipped (new finding 2).
• The bookkeeping I added to compensate (ring, sticky markers, a per-database table) is where the remaining P1s live (new findings 4, 5 and 7).

Per the lesson recorded from earlier specs: when each round's fixes produce the next round's findings, stop patching the mechanism and change it.

A different mechanism, for a ruling before any more design

Embedded replicator: prove it by doing it. The agent already owns Replicator.SyncAndWait, which captures and uploads and returns only when the replica has caught up. A sync the agent itself started at T and saw return nil is positive evidence that everything in the database at T is in the replica: no sampling, no positions, no diagnostic slot. One such "proof sync" per awake host per its class interval (5 minutes for stream) gives proven level at T, and the final sync at sleep is the same call. Overdue is now - T beyond the interval plus 15 minutes. An idle database costs no S3 request. This replaces rev 2 §1 and most of §3.

Box-wide binary: three options, and this is Paul's call.

1. Nothing cheap; the drill is the signal. Keep slice A's restarts alert (which catches the #284 loop in about a minute) and the daily restore-verify, fix its never-quiet blind spot the way finding 6 says (an aged obligation: does today's replica still trail the source watermark seen at the previous drill), and publish its per-database verdict so the console can show "verified at". Honest, small, but a day's resolution.
2. Ask litestream. Turn on its control socket and use sync -wait as the same proof sync. The socket also exposes stop and unregister to anything running as carlos, which on a box without isolated exec includes tenant code. Rejected in rev 1 for that reason; it only becomes reasonable once every box runs isolated exec.
3. Replace the binary with a platform replicator process built on the same library, so box-wide databases get the same proof sync as hibernating ones and the binary/library version skew (finding 9) disappears. The right end state, and a project of its own.

My recommendation: build the embedded proof sync and option 1 now; open option 3 as its own thread.

Nothing more will be built on replica-freshness-284 until this is ruled on.

---

Round 2 findings

Reviewed rev 2 against b110af29 and Litestream v0.5.14. No files edited, tests/builds run, or reviewers spawned. Platform references below are to the pinned commit; LS/ denotes the supplied Litestream module.

| Round 1 finding | Disposition | Rev 2 assessment | |---|---|---| | 1. Successful upload timestamp hides capture failure | Partly fixed | §§0–2 separate capture from upload, but the replacement diagnostic and WAL predicates cannot establish the promised capture health. New findings 1–3 below. | | 2. Idle daily replicas falsely page | Fixed | §§0–1 use outstanding positions instead of upload-call age. Equal, initialized positions with independently proven capture health need no recent upload. | | 3. Nil Close() is not final-sync proof | Fixed for the original false-positive paths | §1 requires an observed embedded snapshot and certain stop completion; unit-backed and claims-on wrappers cannot supply that snapshot. The new sleeping-host rule introduces a separate false alarm, finding 5. | | 4. Zero/equal L0 positions overclaim initialization and lineage | Partly fixed | §0’s narrower claim removes the restore-lineage promise; §2 rejects zero/ahead positions. §1 still calls zero local position empty without positively proving completed initialization, and accepts remote-ahead as shipped. LS/db.go:559 explicitly returns zero for “no replication yet”; reconciliation is described at LS/db.go:1400. Require completed capture before classifying empty, and handle reconciliation consistently across sources. | | 5. Healthy subset clears an unreadable database | Partly fixed | §3 retains previously observed failures. However, databases unknown from their first observation never enter the table; fresh checker output containing only errors satisfies replica-check while the empty table reads zero (design-rev2.md:72–81). Checker execution and coverage need separate verdicts. | | 6. Previous owner wins a hostname join | Partly fixed | §4’s claims-on suppression is sound. But ServedBy is a list, not a unique owner: internal/console/instancesplacement.go:48. Its reader deliberately includes any advertised role, not specifically the write home (internal/edgesfile/snapshot.go:23). Select a unique eligible source or return unknown on ambiguity. A host with no eligible source gets unknown. Ordinary legacy sleeping routes remain advertised (internal/host/edgesfile.go:378). | | 7. Uptime gate loses outage age | Partly fixed | §2 introduces persistent debt, but its recovery proof depends on the invalid WAL predicate. With the prescribed fallback to unknown, an open need_txid can remain open indefinitely. Recovery must explicitly resolve capture failure and the associated upload watermark. | | 8. Declared class differs from actual interval | Fixed | §1 reports Replica.SyncInterval; claims-on forwarding stays deferred. The real interval is assigned at internal/lease/lease.go:338. | | 9. Generator parser cannot establish coverage | Partly fixed | §2 refuses configurations without the generated header. The existing parser still returns paths only, silently accepting a partial match; it neither extracts destinations nor rejects unsupported syntax (internal/ops/litestream.go:165). Specify a strict parser for the complete generated format, not merely a header check. | | 10. Pagination and unbounded execution | Fixed | §2 requires complete iteration, error propagation, bounded concurrency and deadlines, and publication of partial failures. | | 11. Copied hardening blocks operation | Partly fixed | §2 adds AF_UNIX and a writable exception. It must also provision /data/monitor with suitable ownership; currently its creation depends on the agent’s alert persistence (internal/host/metrics_loop.go:541). Also account for supported unit databases inside inaccessible DynamicUser directories (internal/ops/litestream.go:211); checker access cannot be inferred from replicator access. | | 12. Wants= does not arm an installed timer | Fixed | §2 adds the timer to bootstrap’s enabled/armed plan. Timers are explicitly started, and the target build supplies that plan to self-update (internal/host/bootstrap.go:206–223). |

Only new findings follow.

1. P1 — SyncDiagnostic cannot supply “the last finished capture’s time and error.”

Evidence: §1, [`design-rev2.md:23`](design-rev2.md:23); LS/db.go:398, :431, :2143, :2193.

Starting an operation overwrites the timestamps and clears the previous error. Phase updates refresh UpdatedAt before completion. Checkpoints use the same slot; a checkpoint can return nil without copying anything when chkMu.TryLock() fails.

Failure: capture fails with local=remote=100. The next capture begins and clears the error, then stalls. Reading its recent UpdatedAt and empty error can report shipped despite no completed recovery. Merely ignoring active diagnostics is insufficient: periodic sampling can miss every completed operation and falsely conclude that capture has stopped.

Smallest change: obtain a durable-in-memory completion record containing capture sequence, completion time, resulting position and error. Starting another operation must not erase it. Record checkpoints separately; do not derive this record by periodically sampling the current-operation diagnostic.

1. P1 — Independently read positions and diagnostics can describe a capture that was never shipped.

Evidence: §1, [`design-rev2.md:20`](design-rev2.md:20); LS/db.go:1760 publishes capture state and position, LS/db.go:431 finishes the diagnostic separately, and LS/replica.go:175 independently advances upload position.

Failure: the snapshot reads local=100. Capture finishes TXID 101. The snapshot then reads remote=100 and capture completion time T101. It reports shipped as of T101, although 101 remains unuploaded. This affects stream, batched and daily classes.

Smallest change: define a coherent snapshot protocol. Associate completion with its captured position, and validate that capture did not change across the observation. Return unknown/retry on an overlapping operation or generation change. Scope cached observations to the replicator lifetime, including stop or lease-loss transitions.

1. P1 — The WAL-versus-L0 mtime predicate has permanent false negatives and healthy false positives.

Evidence: §2, [`design-rev2.md:55`](design-rev2.md:55); LS/wal_reader.go:189, LS/db.go:1912, :2229, :2241, :2274.

Failing scenarios:

- Missed failure: newest L0 is written at 12:00. An application commits at 12:00:30, and capture subsequently fails forever. WAL mtime is only 30 seconds later than L0 mtime. Waiting hours never makes the proposed two-minute difference true; equal positions continue qualifying as shipped. - False failure: after a quiet period, a large uncommitted transaction spills frames into WAL and remains open. PageMap intentionally excludes uncommitted transactions, so healthy capture creates no corresponding L0. WAL mtime satisfies the predicate and eventually produces an overdue alert.

The first proposed executable fact therefore does not hold as stated: every physical WAL write need not produce an L0. Nor is completion within one monitor interval a hard bound on capture execution.

The second fact holds only in a narrower form: after a successful checkpoint that changes the WAL header, Litestream writes _litestream_seq and performs another capture before returning. With an unchanged header—including a PASSIVE checkpoint—it returns after that internal write and relies on a later sync. A second process can also append frames immediately afterward. App checkpoints and WAL restarts must be interpreted using salts and committed frame boundaries; file length can retain obsolete frames (LS/db.go:1249). Litestream’s read lock constrains checkpoints, but does not prevent concurrent WAL writes (LS/db.go:1125). An idle WAL containing frames is not itself evidence of pending capture.

Smallest change: take the design’s stated fallback now: position equality is unknown: capture not observed until there is positive capture evidence. Do not use the mtime predicate to establish either failure or recovery.

1. P1 — The agent table can turn continuously healthy progress into permanent lag.

Evidence: §1, [`design-rev2.md:26`](design-rev2.md:26); §3, [`design-rev2.md:72`](design-rev2.md:72).

Failure: every sample sees local=N+1 and remote=N, with both advancing between samples. §1 correctly says the oldest outstanding sample advances. But §3 keeps the existing row and its since, removing it only on shipped/empty. If implemented literally, the first observation ages past 15 minutes even though every previously observed backlog has shipped.

Replacing since indiscriminately is not sufficient either. A bounded ring can discard the oldest unresolved sample during a long outage; switching to “first observed zero” after an upload error can discard older backlog age. Litestream explicitly resets its reported remote position on errors (LS/replica.go:138).

Smallest change: retain unresolved watermarks with their observation times, retire them on positive upload evidence, and let the oldest remaining watermark determine since. Keep the oldest unresolved boundary even when compacting the ring. A zero replica position must not erase previously established debt or acknowledgements.

1. P1 — A successful sleep can leave a healthy replica paging indefinitely.

Evidence: §1, [`design-rev2.md:34`](design-rev2.md:34); §3, design-rev2.md:75–77; internal/activator/activator.go:4111–4130; internal/lease/lease.go:419–445.

Failure: the last awake sample sees one outstanding upload and creates a row. Before the next sample, the application stops and the embedded replicator completes its final sync successfully. The stop is certain, but the preceding tick was behind, not shipped. §1 therefore publishes no sleeping entry. §3 retains the row forever while the route remains registered.

Smallest change: let successful final-sync evidence from the identified embedded replicator resolve its outstanding debt regardless of the previous sample. This need not infer anything from generic Rep.Close() implementations; unit-backed and claims-on hosts remain excluded as specified.

1. P1 — The previous-run counter comparison can fail a completely current restore.

Evidence: §6, [`design-rev2.md:97`](design-rev2.md:97); internal/lease/check.go:127–145; internal/ops/freshness.go:105; LS/replica.go:686; LTX v0.5.1 decoder.go:244–251.

Failure: run one reads source main-file counter 10 and restored counter 11: the replica already contains page one from WAL. A checkpoint later copies that page into the source main file. Run two reads source=11 and replica=11. The source counter moved and the replica counter did not, so §6 declares stale despite equality.

There is also a timing false alarm when the source advances immediately before the second drill and normal replication has not caught up yet. Two observations do not establish that this new change has been missing since the previous drill.

Normal LTX cold restore copies page bytes; the counter randomization in replica.go:909 belongs to follow-mode updates, not this restore path. Source replacement by restore/move nevertheless invalidates a path-keyed historical baseline. VACUUM does not make counter movement proof of lag. SQLite itself does not guarantee incrementing this counter on every WAL transaction. SQLite file-format documentation

Smallest change: judge an aged obligation: whether today’s replica still trails the source watermark already observed in the previous successful drill, within an unchanged database/replica identity. Current equality must pass. Rebaseline on replacement, regression or uncertain identity.

1. P1 — The new table needs persistence semantics or an agent restart can clear an unresolved alert.

Evidence: §3, [`design-rev2.md:72`](design-rev2.md:72); internal/host/metrics_loop.go:699, :808; internal/metrics/alert.go:138–145, :333.

Failure: database A is behind and the alert fires. Its readings become unknown. After an agent restart, the existing alert state reloads, but the proposed per-database table has no specified persisted representation. Once initialized empty, it supplies zero and clears A without recovery or removal evidence. The existing persisted alert record contains the aggregate rule, not every outstanding database.

Smallest change: persist unresolved table entries and their identity/watermark evidence, or specify an equivalent lossless reconstruction. Missing or unreadable debt state must not become an authoritative empty table.

1. P2 — replica-check can fire before the checker has had a legitimate first opportunity to report.

Evidence: §2’s five-minute timer and fleet-wide arming, design-rev2.md:40, :66; §3’s immediate missing-file condition, [`design-rev2.md:81`](design-rev2.md:81).

Failure: a deployment installs the checker where /etc/litestream.yml already exists. The next agent tick finds no result and raises the rule while the first timer invocation is still pending or running. The missing-file branch has no equivalent of the three-period age allowance.

Smallest change: persist when checker coverage became expected and allow the first three periods before declaring missing output. Keep using the last successful checked_at for established coverage; restarting must not reset an existing outage.

1. P2 — The proposed executable facts target a different version from the box-wide writer.

Evidence: §7, [`design-rev2.md:101`](design-rev2.md:101); internal/ops/litestream.go:43 pins the installed binary to 0.5.15, while the embedded library reviewed and proposed fact tests use 0.5.14.

Failure: passing library tests authorize a filesystem inference about another binary version’s checkpoint and capture behavior. The deployed binary may also predate the current provisioning pin.

Smallest change: gate any binary-specific capture inference on the actual writer version and verify it against that version. Unsupported versions retain unknown capture health.

NOT READY.

Another case for this thread, from Tito (PRO-6189), 2026-10-23: the hibernating replicas for apropos and fronteers-conference are stale (from 20 and 21 Oct). Titogo's import-sync writes those databases from outside the instance, so the instance's own replicator never sees the writes. A freshness signal has to catch "the database changed but its replica didn't", whoever wrote it. The durability design (design-carloku-durability, §5.2) depends on whatever this thread settles, and it uses restore-verify's change-counter check until then.