amadan
Sibling: hibernating instances' history window is filed as the next discussion (retention for the activator's embedded litestream). Without it, --at reaches back about a day on hibernating instances.
carlos db pull/query --at: read a database as of a past time Feature Request closed
Claimed by claude~paul@keymail.dev 2 weeks ago
What
Add --at <RFC3339 time> (and maybe --txid <n>) to carlos db pull and carlos db query. The member gets a read-only copy of the database as it was at that point, from the replica, the same way they get the latest copy today.
Why: a bad migration or a wrong DELETE is usually noticed days later. Today a member can only pull the latest state, which already has the damage in it. Recovering from that means an operator running litestream restore -timestamp by hand with S3 access.
Why it's small (checked on main 4a51a6da)
- The console's replica listing already carries each LTX file's
created_at(internal/console/apidb.go:283,internal/dbdoor/coords.go:41,internal/dbdoor/client.go:120). restoreOncebuildslitestream.NewRestoreOptions()(internal/dbdoor/restore.go:133); litestream supportsTimestamp/TXIDthere.- So: the flag in
db.go, thread it intoRestoreTo, set the option, and test againstdbdoortest.
Things to get right
- A time outside the kept history must fail with a plain sentence naming the oldest restorable point, not litestream's raw error.
dbAsOfoutput should say the copy is as of the requested time, not "latest".- Check the
created_atthe door serves is trustworthy enough for selection (it comes from the replica listing; litestream says it uses CreatedAt only for logs in the client's iterator, seeclient.go:81). If timestamp selection needs the LTX header timestamp instead, say so here before building. - How far back you can go depends on retention. Awake instances keep 7 days (
internal/ops/litestream.go:66-67, litestream ≥0.5 only). Hibernating instances keep about 24h until the sibling thread on retention lands.
Out of scope
Rolling a live instance back in place. That's a separate, much bigger spec (lease, new replica generation, backend swap, moves, replicas, console door). Paul agreed 2026-10-08 to ship this piece and the retention piece first.
That's #263.
Findings before building (litestream module v0.5.14, main 4a51a6da):
- Which timestamp drives the plan.
CalcRestorePlankeeps a file only ifinfo.CreatedAt.Before(T), and asks the client for "accurate" (S3 metadata, LTX header) timestamps. Our door client ignores that flag. The legacy door lists withuseMetadata=false, socreated_atis S3 LastModified, the upload time. That is always at or after the file's data, so a copy for T never includes changes after T. It can be older than T (a file uploaded just after T is left out with everything after it). The copy reports its real as-of point from the newest file's LTX header, whichrecorder.newest()already reads. One consequence: objects copied by a move get a new LastModified, so--atbefore a move finds nothing. That's safe, but worth a line in the docs. Switching the door to header timestamps (HEAD per file) costs S3 requests, so I'm not doing that now.
- Claim mode lists no times.
listDBVersions(apidbclaim.go:183,HostClaim) returns a pinned plan with nocreated_at. Zero time is "before" any T, so litestream would quietly restore the latest state and we'd label it as T.--atwill refuse when any listed file has no time, with a sentence saying so. Making--atwork under the claim is follow-up work for whoever activates it (#250).
- Dropping
--txid. TXID restore only lands on exact file boundaries (it skips files whose MaxTXID > N, then fails if it can't reach N), so most numbers a person would type fail.--atonly.
- Before the oldest history, litestream returns
ErrTxNotAvailable, whichdbRestoreErrorcurrently words as "nothing has been replicated yet". With--atit will say the time is earlier than the history kept.
Astra (gpt-6-astra, high) reviewed 4a51a6da..26c65c10 in two rounds.
Round 1:
- P2 — Explicit timestamps can disable the cutoff. In db.go:127,
--at 0001-01-01T00:00:00Zparses successfully as Go’s zero time, which means “restore latest.” Likewise,--at 0001-01-01T00:00:00.5Zbecomes zero when restore.go:184 truncates it. Litestream then skips timestamp filtering and restores current data, violating the required cutoff. Reject explicitly supplied timestamps whose value becomes zero after truncation, and guardRestoreAsOfagainst nonzero inputs that truncate to zero. Add regression coverage for both inputs and equivalent timezone representations. Fixed in 26c65c10. Round 2: - Round 1 — P2: Explicit timestamps can disable the cutoff — Fixed. db.go:137 maps explicit pre-epoch timestamps—including zero and timezone equivalents—to the Unix epoch. restore.go:108 also guards nonzero inputs that truncate to zero. These requests retain a cutoff instead of restoring latest.
New findings on 4a51a6da..26c65c10: none.
Source review included Litestream v0.5.14’s planner. Targeted tests could not run because the read-only sandbox prevents Go from creating its build directory. Also found while building (not by Astra): S3 LastModified is second-resolution, rounded down, so a write just after --at in the same second got in. Fixed in af256c32 by rounding --at down to its second; regression test plus an e2e through the real console door.
Merged to main as 1621d27 (squash of db-pull-at). Not released or rolled: it's CLI-only, so members get it with the next carlos CLI release. Sleeping instances keep about a day of history until #263.