carlos / platform Public
carlos db — reading an app's database from its replica
Design, 2026-09-02. Revised twice after review, same day (§14).
1. What this is for
There is no way to look at an app's data through the platform. The question "why does this row look wrong" currently has no carlos answer, which means it has an AWS answer — an SSM session onto the box and a sqlite3 prompt against the live file. That violates the standing rule that every carlos command is designed for a user with no access to the underlying infra (AGENTS.md, Paul 2026-08-08), and it puts a hand-typed shell next to a production database for what is almost always a read.
carlos db closes that gap by answering a deliberately narrower question. It does not connect you to the running database. It hands you a restored copy of the replica and lets you query that.
The narrowing is the feature, not a compromise. It makes the command safe for a class of caller we specifically want to serve: an agent debugging a production problem. An agent that can read the data and cannot possibly change it is a useful colleague; an agent with a live sqlite3 prompt on production is an incident waiting for a bad inference. Every design decision below is downstream of that.
Scope
In: reading an app's platform-replicated SQLite, from the CLI, by an account member, as a local file, for in-region, non-pooled, activator-replicated instances.
Out: writing to tenant data by any path; querying the live database; fleet-homed and cross-region instances (§13 — a v2 with a different transport); box-wide replicas; non-SQLite stores; the console web UI.
2. The two decisions behind the shape
Replica, not live. The alternative was a query door onto the running instance. That needs a synchronous console→instance path, which CARLOS does not have for ordinary boxes: the control plane is the bucket and boxes poll (schedule.TickEvery 15s, logship 10s, control/serial 2s). Building one would mean a new transport and a new door onto the tenant process, to deliver a capability whose main use is a read. The replica is already there, already off-box, and already proven restorable by the restore drills.
Client-side query, not console-side. The console serves replica bytes; the member's machine restores and queries. The console never executes tenant SQL, never holds a restored tenant database, and needs no query timeouts, row caps, or result-set memory budget.
> The first draft justified this as "the console serves objects it > already stores". That is false — replicas live in customer data > buckets (§6). The client-side argument survives on its own merits, but > the transport is a genuinely new door, and §6 is the hardest part of > this design rather than the easiest.
The client-side choice also buys the microcopy problem's best available answer (§5): the user is handed an actual file, so "this is a copy" is a true statement about an artifact in their hand rather than a warning against an illusion.
Cost accepted: you transfer the whole database rather than one query's rows, and freshness is replica freshness rather than live.
3. What already exists
Accurate as of 2026-09-02. There are two replicators and they are easy to conflate:
- The activator's embedded library. litestream v0.5.14 is a
go.moddependency (not vendored — there is novendor/), linked into thecarlosbinary. It replicates hibernating exec-backed instances under the lease.lease.Restore(lease/lease.go:378) restores from it via litestream's nil-*DBURL-restore pattern (lease.go:390). This is the setcarlos dbserves. - The box-wide binary. A separately pinned litestream 0.5.15 executable (
ops/litestream.go:43) driven by/etc/litestream.yml.carlos ops restore-verifyshells out to it.
ops.FleetDBs is the box-wide unit's set and excludes hibernating exec routes (ops/hibernate_verify.go:5-8) — the complement of the activator's set, not the whole. Box-wide replica keys are absolute paths (ops/litestream.go:170) and collide across boxes sharing a fleet prefix (/data/registry.db), which is one reason they stay out of v1.
Other parts this design uses:
lease.VerifyReplicarestores without taking the lease;check.go:66-79documents why that cannot shadow the true holder. §12 records how far that reasoning actually reaches.activator.HibernatePrefix(prefix, host)=path.Join(prefix, host)(activator.go:113), prefixinstancesin production. Hosts are unique FQDNs (activator.go:118-121), so prefixes cannot collide across tenants.modernc.org/sqlitev1.55.0 is a direct dependency and pure Go (go.mod:23); litestream imports it, so the CLI already links it. The query engine embeds — nosqlite3on the user's machine, no cgo.internal/dbverifyrecords restore-drill verdicts keyed by host (dbverify.go:45), at one deployment-wide key (dbverify.go:28), published only by the hibernation drill.Mergedrops hosts outside the writing box's fleet (dbverify.go:60-80), so the last box to run overwrites the others. A missing verdict rendersunverified.- Console gates:
instanceGate(apiinstances.go:70),s.audit(server.go:2692). The gate is membership viaAccountsFor(addr)(apicli.go:208) — people carry no owner/member role.
The console has no registry snapshot. It opens the local registry.db on its own box (console/view.go:532-538). The control/registry-snapshot/<box>.db objects boxes write (host/snapshot_loop.go:34) are read by nothing in internal/console. Fleet-wide instance knowledge is apps/<sqid>/<app>/instances/<host>.json. §6 and §8 both depend on this distinction.
4. Command surface
carlos db ls --app <a> [--account <a>]
carlos db pull --app <a> [--instance <host>] --out <path>
carlos db shell --app <a> [--instance <host>]
carlos db query --app <a> [--instance <host>] --sql <text>ls is the discovery verb and the one that makes the rest legible: per instance, the database the platform replicates, the age of the newest replicated write, whether the instance is replicating right now, its replication class or an honest "unknown" (§8), and its restore-drill verdict. It also names databases declared in data.dbs that the platform does not replicate (§7) — otherwise visible only in adopt logs a member cannot read.
pull writes the restored copy to a path you name and stops. --out is required and must not already exist.
shell restores into a private directory, opens an interactive prompt against it with the embedded engine, and removes the directory on exit. This is the closest thing to rails console a Go platform can honestly offer: a REPL over your data, which exists, rather than over your app, which cannot — Go is compiled ahead of time, and there is no interpreter in the running binary to evaluate against its live heap.
query is the non-interactive form — one statement, results to stdout, exit code reflecting the query. This is the shape an agent uses and should pipe cleanly.
Where the copy lands
litestream's Restore refuses an existing output path (replica.go:604) and creates <out>.tmp itself with os.Create, at the process umask (typically 0644). You cannot pre-create a 0600 file for it to fill.
So: always restore inside an os.MkdirTemp directory created 0700. That confines the .tmp and the -wal/-shm files a read-only open still needs (check.go:96-98). shell and query never leave that directory. pull restores there, then copies to --out with fsync and 0600 — a copy, not os.Rename, because the temp directory and the user's destination are routinely on different filesystems. Open question 2 is thereby answered: always 0600, and the user can loosen it.
Read-only is enforced, not assumed
Open with file:<path>?mode=ro — the form lease.CheckDB already uses (check.go:36) — plus PRAGMA query_only=1.
This matters most for the agent case, for a non-obvious reason. A copy is already harmless to write to, so enforcement is not about protecting production. It is about not letting a caller believe it changed production. An agent that runs an UPDATE against a writable copy sees 1 row affected and concludes it fixed the bug. Against a read-only copy it gets SQLite's attempt to write a readonly database, and the error is the truth. Silent success on a discarded write is the single worst outcome this command could produce.
§11 pins that error string in a test, so the claim stays true.
5. The microcopy contract
The safety property is a fact the user has to hold in their head — that this is a copy, of a stated age, and that nothing they do here reaches production. If they lose that fact, the command becomes a way to make confident wrong decisions. The copy is load-bearing.
Exact strings are not drafted here. They go through the copy-review gate at implementation time, before being written into any source file. What follows is what each string must accomplish.
R1 — Say it at handover, and name the source exactly. The line reporting a successful restore states that this is a copy, and names host, database, the newest replicated transaction id and its time. Two pulls must be distinguishable, and an agent must be able to cite an "as of".
R2 — Age is a number, and carries its expected bound — or admits it cannot. Report the age of the newest replicated write, never a "lag" (which implies a live comparison the client cannot make, §8) and never an adjective. Beside it, report the replication class bound: an awake daily instance is legitimately up to 24h behind, exactly the case where an old copy is most easily mistaken for a current one.
Where the class is unknown — which is the common case, because the console only knows it for routes on its own box (§8) — say so. It must never render as stream by default. A wrong bound is worse than no bound: it converts "I don't know" into "one second old".
R3 — The interactive prompt carries it too. A shell session can outlive its banner by an hour. The prompt itself carries both the word copy/replica and the as-of time.
R4 — Refuse loudly, explain briefly. Four refusals need their own voice: replicate: "app" (§7); the read-only write refusal (§4); the residency refusal (§6), which must name residency rather than read as a transient error; and the pooled-instance refusal (§6). Per the repo's rule, store errors are logged rather than returned (apischedule.go:145-147) — they carry bucket names and request ids.
R5 — Written for two readers. A human skims; an agent parses. No carlos verb has a --json flag today, so this introduces one, and the fields are fixed here so the human and machine forms cannot drift:
copy: true
as_of # time of the newest replicated write (§8)
replicated_txid
lease_held: bool # is the instance replicating right now (§6, §8)
sync_class # stream|batched|daily|unknown — never defaulted
verified # ok|failed|unverifiedR6 — Name what is not there. A database declared in data.dbs but not replicated (§7) is listed as such. "The platform has no copy of audit.db" must be said before someone goes looking for it.
6. Transport: the replica door
Decision: the console reads the region's data bucket directly. Option B — having the owning box mirror replicas into a staging prefix in the deployment bucket — was considered and rejected. It needs a new box-side pass plus a sweeper, which contradicts §9's "no path that runs anything on the box"; its first-mirror latency is a CopyObject per LTX file; it lands tenant database bytes in the deployment bucket, which the moves design deliberately routed around (staging goes to a sibling data bucket, never the deployment bucket); and it does not even remove the data-bucket read, which hibernate-verify still needs. Its one advantage over A — cross-region reach — is a residency violation rather than a feature.
Configuration
The console takes the same triple the agent reads — CARLOS_HIBERNATE_BUCKET / _REGION / _PREFIX — plus CARLOS_REGION for the placement compare, and opens one *s3.Client reusing lease/endpoint.go's config path including s3compat.Options() and the endpoint override (endpoint.go:110-131).
There is deliberately no edges-file lookup for another box's bucket. A bucket discovered that way is one IAM will refuse, so resolving it only converts a clear refusal into a confusing one.
Whether this needs an IAM change
Probably not. The console opens S3 through the default credential chain (cmd/console/main.go:394-399) — the box's instance role, the same role the activator on that box uses to write instances/<host>/. If the console's box hosts any hibernating instance, the read authority is already there. If not, it is one read-only, in-region grant, smaller than the iam:CreateUser/PutUserPolicy the console already holds via CARLOS_STORE_BACKEND=aws (objstore/aws.go:39-46).
Resolution, and what is refused before S3 is touched
apps/<sqid>/<app>/instances/<host>.json → ValidHostShape (provision.go:250-262, written as path-traversal defence — path.Join alone cleans ../x to x rather than refusing it) → then three refusals, none of which make an S3 call:
rec.Region != CARLOS_REGION(both""in a single-region deployment) → residency refusal naming the region.- a
fleet/<name>region (console/fleets.go:535) → residency refusal; the data is on the customer's own box. §13 records the v2. rec.Pool != ""→ refusal; a pooled record's data lives on the pool box, the same classmovespass.go:150-153already refuses.
An AccessDenied from S3 maps to the residency refusal as a backstop, never to a generic 502.
The door
Two handlers, one gate, one rate budget:
list(app, instance, level)→[{level, min_txid, max_txid, size, created_at}]get(app, instance, level, min_txid, max_txid)withRangesupport
Implement them as an HTTP adapter over litestream's own lss3.ReplicaClient read methods, with Path set per request. That way the server derives the key layout (<prefix>/%04x/%016x-%016x.ltx, s3/replica_client.go:655-656) from the same code the writer uses, and reader and writer cannot drift.
Addressing by coordinate rather than key is what keeps lock.json — which sits under the same prefix and is not a member's to fetch — unreachable.
Following the edge store door's F8 precedent (storedoor.go:55-59): bytes stream through the console, never a presigned URL, and nothing the request supplies may influence which bucket or client is used.
Six wire specifics the implementation must honour:
listaccepts levels 0–9 inclusive.CalcRestorePlanasks forSnapshotLevel(= 9,compaction_level.go:9) first, then 8 down to 0 (replica.go:1452, 1476-1485). The activator writes only 0–3 and 9; the door answers empty for the rest, never 404.- Listings are ordered by
min_txidascending —restoreLevelCursorassumes filename order, which S3 gives because the names are zero-padded hex. seekanduseMetadataare not on the wire. The client filtersseeklocally; nothing in a non-timestamp restore readsCreatedAtexcept debug logs.size,min_txidandmax_txidare mandatory.RestorerejectsSize < ltx.HeaderSize(replica.go:630) andResumableReaderuses size to detect a premature EOF (resumable_reader.go:96). Checksums are not needed.getRange semantics match the S3 client exactly:bytes=<offset>-whensize == 0, elsebytes=<offset>-<offset+size-1>(s3/replica_client.go:647-652). A missing object is a 404 the client maps toos.ErrNotExist.lsneeds a third, derived field:lease_held(plusexpires_at), computed console-side fromlock.json(leaser.go:31-35; a released lease is deleted,s3/leaser.go:180). The lock's body is never returned —Owneris the box's identity. This is how §8's "is it replicating now" is answered without exposing an object coordinate addressing deliberately hides.
Who may open it
PrincipalUser only. Every service principal — the legacy app token (role instance) and every named service credential (publish/instance/operate/admin) — gets the same http.NotFound as a stranger.
A 403 would be an oracle: it tells any token holder that a db door exists for this app. The gate ethos here is one byte-identical 404.
No legitimate caller loses anything — an app reads its own database live, and CI has no business pulling production data. admin is refused deliberately, despite already reaching store grant.
> Revision, 2026-10-01 (implementation; Paul may overrule). Service > principals get a 403, not the 404 above. The two door routes are > registered with the zero role Set, the repo's closed-door mechanism > since 2026-08-18, and the shared cliRoute gate answers every service > credential with one constant 403 ("closed to service credentials") > BEFORE any account or app lookup. The body is the same for every role, > account and app, so it reveals only that the door exists, which is > public in the code, and nothing about this app or account. Members who > are not in the account, and unknown apps and instances, still get the > byte-identical 404. A literal 404 would need either a role Set the > handler then overrides, or a change to the shared gate for every closed > door.
Every successful open writes an s.audit row naming account, app, host, database and the replicated TXID served.
The read-only replica client
The CLI implements litestream's ReplicaClient over this door, the way internal/fleetapi implements release.Store over the edge store door, and for the same reason: the member has no AWS credentials, by design.
The interface has eight methods (replica_client.go:19-50). This implementation supplies the five a restore needs — Type, Init, LTXFiles, OpenLTXFile, SetLogger — and the three mutating ones (WriteLTXFile, DeleteLTXFiles, DeleteAll) return an error unconditionally.
The client cannot corrupt the replica, because it has no code that writes to one. Not "is not supposed to" — has none. §12 records the verification and the three conditions it depends on.
7. Which database — one, not eight
caps.Data allows up to eight names (maxDBs = 8), but a route replicates exactly one database: Route.DBPath is single-valued and only DBs[0] is stamped (host/caps_pass.go:294, whose own adopt warning says so verbatim). The activator's Replicate takes one dbPath.
So there is no --db flag and no selection story:
- The replicated database is implied by the instance.
--instanceselects among an app's instances — inferred when there is one, required when there are several, because guessing hands someone the wrong data while telling them it is the right data.lsnames the declared-but-unreplicated entries (DBs[1:]), surfacingdataAdvice's warning to a member who cannot read adopt logs (R6).
replicate: "app" has no replica
An app declaring replicate: "app" streams its own WAL and the platform replicates nothing for it (caps.DataReplicateApp; caps.go:416 refuses data.dbs alongside it precisely to keep the two-replicators hazard unreachable).
This must fail naming that cause. The dangerous failure is a generic "no replica found", which reads as transient and invites a retry loop; the operator needs to know the platform was never going to have this data.
8. Age, and the three things that are not lag
The client cannot compute lag. ops.freshness gets the live change counter from the source file on the box (freshness.go:96-115), which neither console nor CLI can reach. Any claimed distance from live would be invented.
What the replica alone supports:
- Age of the newest replicated write. Take it from the LTX header timestamp, not S3
LastModified— a compacted file'sLastModifiedis compaction time, and theltxcompactor carriesmaxHdr.Timestampthrough (compactor.go:118). One ranged GET ofltx.HeaderSizebytes on the max-TXID file gives the true newest-write time at any level. UsingLastModifiedwould report an L3 compaction as a fresh write — a stale copy looking current, the exact failure this command exists to prevent. - Whether it is replicating now —
lease_held, derived fromlock.json(§6). - The expected bound — when it is knowable.
Replicationlives onregistry.Route(registry.go:126) in the box registry;provision.Recordhas no such field. So the console knows the class only for routes on its own box. For an in-region sibling it is genuinely unknown, and R2 requires saying so rather than defaulting. §13 records the additive follow-up. - The drill verdict —
dbverify, per host, hibernation drill only.
The first draft claimed "no explicit sync interval, so 1s default, therefore seconds behind live". That is wrong for the instances this targets, and wrong in the dangerous direction: classes are ""/stream ≈ 1s, batched 5m, daily 24h (activator.go:205-230, lease.go:245-252). An awake daily instance is legitimately up to a day behind, and reporting its age without its bound makes correct platform behaviour look like current data.
Retention differs between the two replicators too: SnapshotInterval 24h / SnapshotRetention 7d (ops/litestream.go:64-65) reach only /etc/litestream.yml. The activator's embedded store never sets Retention, so hibernating replicas run on litestream's DefaultRetention of 24h.
A sleeping instance is consistent, not necessarily complete
stop() logs and swallows a failed final checkpoint (activator.go:876-881) — the hole the instance-move spec had to add a re-checkpoint for. A parked replica is consistent at a committed boundary (LTX carries committed frames only) but may trail the sleep point.
So: "consistent as of its last successful sync, which on a clean sleep is the sleep point" — never "complete as of when it went to sleep". carlos restart and Bounce go through the same stop().
9. Security posture
The exposure is real and is the point of the gate. This command moves production data onto a laptop. Read-only does not reduce that; a copy leaks exactly as well as an original. Hence: members only, no service principals of any role, an audit row per open, --out required and refused if it exists, 0600 on anything kept, a 0700 private directory for anything temporary.
Nothing new is reachable on the box, and nothing new runs there. No door on the tenant process, no new agent verb, no pass in the converge loop. The sandbox (DynamicUser, ProtectSystem=strict, empty CapabilityBoundingSet) is untouched, and the same-uid sibling residual in activator/admintoken.go:8-12 is not widened, because no part of this design runs on a box at all. Option B would have forfeited this sentence; A keeps it.
One new read authority: the console gains in-region, read-only access to the data bucket's instances/*. Bounded by residency — it cannot reach another region's data even if asked, and §6 refuses before trying rather than surfacing an AccessDenied.
The failure mode to design against is not corruption, it is confidence. Every serious way this command hurts someone routes through a caller believing old data is current, or believing a write landed. §5 and §4's read-only enforcement exist for that; they are not decoration on the security story, they are most of it.
10. Failure matrix
Each row is a test.
| Condition | Result | |---|---| | Not a member of the account | Byte-identical 404, matching instanceGate | | Any service principal (app token, or publish/instance/operate/admin) | Same byte-identical 404 — never a 403 | | No such app | Byte-identical 404 | | replicate: "app" | Refusal naming that cause specifically (§7) | | Several instances, no --instance | Refusal listing the hosts | | Instance in another region | Residency refusal naming the region — no S3 call | | Instance region fleet/<name> | Residency refusal — no S3 call (§13 records the v2) | | rec.Pool != "" | Refusal — data lives on the pool box | | S3 AccessDenied | Mapped to the residency refusal, never a generic 502 | | Declared-but-unreplicated db named | Refusal naming it as never replicated (§7) | | Nothing replicated yet | "Nothing replicated yet" — match both ErrNoSnapshots and ErrTxNotAvailable (lease.go:394) | | list for a level the activator never writes | Empty list, not 404 (§6.1) | | Replica changed under the restore | One retry around the whole Restore (§12) | | Restore fails integrity check | Refuse to hand over the file; surface the dbverify verdict | | Instance asleep | Succeeds; reports age, lease_held: false, class or unknown | | Instance awake on daily class | Succeeds; age reported beside its 24h bound (R2) | | Class not knowable (sibling box) | sync_class: "unknown", never stream (R2) | | Write attempted in shell/query | SQLite attempt to write a readonly database, surfaced plainly | | --out omitted, or already exists | Usage error | | Host failing ValidHostShape | Refused before it becomes a key segment | | Client asked to write a replica | Unconditional error — no code path exists |
11. Testing shape
Following apischedule_test.go and internal/lease/check_test.go:
- Door: the full gate matrix, plus a coordinate-confinement test asserting
lock.jsonand any key outside the resolved prefix are unreachable. That test is what keeps this from becoming an object proxy as the code ages. - Wire conformance: levels 0–9 answer empty rather than 404; listings ascend by
min_txid;size/min_txid/max_txidalways present; Range semantics matchs3/replica_client.go:647-652for both thesize == 0and sized forms. - Replica client, both halves: the three mutating methods error without touching the network and a call-counting fake asserts they stay at zero across a full restore. The second half is what catches a litestream upgrade changing the restore path.
ReplicaClientV3negative assertion — §12 condition (i).- Compile-time
var _ litestream.ReplicaClient = (*client)(nil). - Freshness: a table over (age,
lease_held, class-or-unknown, verdict) asserting the wording distinguishes asleep from failing, always carries the bound or admits it is unknown, and never renders unknown asstream. - Age source: a compacted L3 file whose
LastModifiedis far newer than its LTX header timestamp reports the header time (§8.1). - Selection:
caps.Datafixtures forDBs[1:]unreplicated and thereplicate: "app"refusal. - Read-only: an
UPDATEinqueryfails with the pinned error string, and the file on disk is byte-identical afterwards. - Permissions: the temp directory is 0700,
--outlands 0600, andpullsucceeds when temp and destination are on different filesystems.
Definition of done includes a docs/site/cli.md heading — the internal/docsite gate fails without one (AGENTS.md).
12. Why the safety claim holds
Verified against litestream v0.5.14 during review, because §6's central claim depends on it.
Replica.Restore (replica.go:544-720) → CalcRestorePlan (replica.go:1485ff, LTXFiles per level) → NewResumableReader, whose client parameter is the narrow LTXFileOpener interface (internal/resumable_reader.go:51,72) — OpenLTXFile only. WriteLTXFile, DeleteLTXFiles and DeleteAll are reached only from Sync, EnforceRetention, and db.go's snapshot/retention paths (replica.go:192,312; db.go:2470,2526,2649), none of which run without a *DB/Store. With a nil *DB, they are unreachable.
Three conditions this depends on. Each is a test in §11:
- The client must not implement
ReplicaClientV3(v3.go:145).Restoretype-asserts for it (replica.go:612) and would take the v0.3 path instead. - Never set
RestoreOptions.Follow. It writes into the local output and a-txidsidecar, and keeps polling. - Run
lease.CheckDBafter the restore rather than relying onopt.IntegrityCheck, matchinglease.Restore's existing pattern.
What the claim does not cover
check.go's reasoning is about not shadowing the writer. It does not promise the reader a stable listing. CalcRestorePlan lists every level once and opens lazily, so the writer's L0 retention (DefaultL0Retention 5m, checked hourly by the activator, lease.go:78) can delete a planned file before it is opened.
The outcome is a failed restore, never a torn one: the output is <out>.tmp, renamed only on success (replica.go:661-663).
The retry must wrap the whole Restore, not an individual read: ResumableReader treats ErrNotExist as fatal and does not retry (resumable_reader.go:72-75), so a vanished planned file needs a fresh plan, which only a new Restore produces.
13. Open questions and follow-ups
- Fleet-homed instances (v2). §6 refuses them: their data is in the customer's bucket and IAM will not cross. The reverse tunnel (
edge/tunnel.go) is a real synchronous edge→box path for exactly the hosts that dial in — so the v2 is the box serving its own replica coordinates over its tunnel channel, with the console never touching the customer's bucket. Different transport, same wire shape. - An additive
replicationfield on the instance record. Closes §8'ssync_class: "unknown"for in-region siblings. Wire-additive and therefore allowed; worth doing regardless of this command. shell's prompt. An embedded REPL is small code but a real surface (history, formatting, dot commands). Shipls/pull/queryfirst, thenshell.- Console web view. Out of scope, but §6's door is what it would use; confirm the coordinate-addressed shape does not foreclose it.
- Past points in time. Not offered. Given the activator's effective 24h
DefaultRetention(§8), is "current replica only" the right permanent scope? - Rate limiting. Pulls are large; neither the token-keyed instance burst nor the address-keyed write budget fits. Needs its own budget.
Decomposition
Three units, in order: (1) door + read-only client + pull + query; (2) ls and its freshness fields; (3) shell.
14. What review changed
Round one. The safety claim survived verification (§12). The transport did not: "the console serves objects it already stores" was false. Also corrected — a route replicates exactly one database, not up to eight; daily class is legitimately 24h behind, not "seconds behind live"; a 403 for service principals was an oracle; litestream is a dependency, not vendored; FleetDBs is the complement of the activator's set; the console has no registry snapshot; dbverify is keyed by host; the mode-0600 temp file was not constructible.
Round two. Option A chosen and B rejected on its merits (§6), so §9 regains "nothing runs on the box". The edges-file lookup was dropped as buying nothing IAM will allow. Six wire specifics added, including levels 0–9 answering empty rather than 404 and mandatory size. Two new findings, both in the direction of overstating freshness:
sync_classis unknowable off-box —Replicationis onregistry.Route, notprovision.Record. R2 now requires an explicit "unknown" that must never render asstream.- Age must come from the LTX header timestamp, not S3
LastModified, or a compaction reads as a fresh write.
Plus lease_held as a derived ls field (answering "is it replicating" without exposing lock.json), the retry wrapping the whole Restore, and pull copying rather than renaming across filesystems.