| 1 | # `carlos db` — reading an app's database from its replica |
| 2 | |
| 3 | Design, 2026-09-02. Revised twice after review, same day (§14). |
| 4 | |
| 5 | ## 1. What this is for |
| 6 | |
| 7 | There is no way to look at an app's data through the platform. The |
| 8 | question "why does this row look wrong" currently has no `carlos` answer, |
| 9 | which means it has an AWS answer — an SSM session onto the box and a |
| 10 | `sqlite3` prompt against the live file. That violates the standing rule |
| 11 | that every `carlos` command is designed for a user with no access to the |
| 12 | underlying infra (AGENTS.md, Paul 2026-08-08), and it puts a hand-typed |
| 13 | shell next to a production database for what is almost always a read. |
| 14 | |
| 15 | `carlos db` closes that gap by answering a deliberately narrower |
| 16 | question. It does **not** connect you to the running database. It hands |
| 17 | you a **restored copy of the replica** and lets you query that. |
| 18 | |
| 19 | The narrowing is the feature, not a compromise. It makes the command |
| 20 | safe for a class of caller we specifically want to serve: an **agent** |
| 21 | debugging a production problem. An agent that can read the data and |
| 22 | cannot possibly change it is a useful colleague; an agent with a live |
| 23 | `sqlite3` prompt on production is an incident waiting for a bad |
| 24 | inference. Every design decision below is downstream of that. |
| 25 | |
| 26 | ### Scope |
| 27 | |
| 28 | In: reading an app's platform-replicated SQLite, from the CLI, by an |
| 29 | account member, as a local file, for **in-region, non-pooled, |
| 30 | activator-replicated instances**. |
| 31 | |
| 32 | Out: writing to tenant data by any path; querying the live database; |
| 33 | fleet-homed and cross-region instances (§13 — a v2 with a different |
| 34 | transport); box-wide replicas; non-SQLite stores; the console web UI. |
| 35 | |
| 36 | ## 2. The two decisions behind the shape |
| 37 | |
| 38 | **Replica, not live.** The alternative was a query door onto the running |
| 39 | instance. That needs a synchronous console→instance path, which CARLOS |
| 40 | does not have for ordinary boxes: the control plane is the bucket and |
| 41 | boxes poll (`schedule.TickEvery` 15s, `logship` 10s, `control/serial` |
| 42 | 2s). Building one would mean a new transport *and* a new door onto the |
| 43 | tenant process, to deliver a capability whose main use is a read. The |
| 44 | replica is already there, already off-box, and already proven restorable |
| 45 | by the restore drills. |
| 46 | |
| 47 | **Client-side query, not console-side.** The console serves replica |
| 48 | **bytes**; the member's machine restores and queries. The console never |
| 49 | executes tenant SQL, never holds a restored tenant database, and needs |
| 50 | no query timeouts, row caps, or result-set memory budget. |
| 51 | |
| 52 | > The first draft justified this as "the console serves objects it |
| 53 | > already stores". That is false — replicas live in customer data |
| 54 | > buckets (§6). The client-side argument survives on its own merits, but |
| 55 | > the transport is a genuinely new door, and §6 is the hardest part of |
| 56 | > this design rather than the easiest. |
| 57 | |
| 58 | The client-side choice also buys the microcopy problem's best available |
| 59 | answer (§5): the user is handed an actual file, so "this is a copy" is a |
| 60 | true statement about an artifact in their hand rather than a warning |
| 61 | against an illusion. |
| 62 | |
| 63 | Cost accepted: you transfer the whole database rather than one query's |
| 64 | rows, and freshness is replica freshness rather than live. |
| 65 | |
| 66 | ## 3. What already exists |
| 67 | |
| 68 | Accurate as of 2026-09-02. There are **two** replicators and they are |
| 69 | easy to conflate: |
| 70 | |
| 71 | - **The activator's embedded library.** litestream v0.5.14 is a `go.mod` |
| 72 | dependency (not vendored — there is no `vendor/`), linked into the |
| 73 | `carlos` binary. It replicates **hibernating exec-backed instances** |
| 74 | under the lease. `lease.Restore` (`lease/lease.go:378`) restores from |
| 75 | it via litestream's nil-`*DB` URL-restore pattern (`lease.go:390`). |
| 76 | This is the set `carlos db` serves. |
| 77 | - **The box-wide binary.** A separately pinned litestream **0.5.15** |
| 78 | executable (`ops/litestream.go:43`) driven by `/etc/litestream.yml`. |
| 79 | `carlos ops restore-verify` shells out to it. |
| 80 | |
| 81 | `ops.FleetDBs` is the box-wide unit's set and **excludes** hibernating |
| 82 | exec routes (`ops/hibernate_verify.go:5-8`) — the complement of the |
| 83 | activator's set, not the whole. Box-wide replica keys are absolute paths |
| 84 | (`ops/litestream.go:170`) and **collide across boxes sharing a fleet |
| 85 | prefix** (`/data/registry.db`), which is one reason they stay out of v1. |
| 86 | |
| 87 | Other parts this design uses: |
| 88 | |
| 89 | - `lease.VerifyReplica` restores without taking the lease; |
| 90 | `check.go:66-79` documents why that cannot shadow the true holder. |
| 91 | §12 records how far that reasoning actually reaches. |
| 92 | - `activator.HibernatePrefix(prefix, host)` = `path.Join(prefix, host)` |
| 93 | (`activator.go:113`), prefix `instances` in production. Hosts are |
| 94 | unique FQDNs (`activator.go:118-121`), so prefixes cannot collide |
| 95 | across tenants. |
| 96 | - `modernc.org/sqlite` v1.55.0 is a direct dependency and **pure Go** |
| 97 | (`go.mod:23`); litestream imports it, so the CLI already links it. The |
| 98 | query engine embeds — no `sqlite3` on the user's machine, no cgo. |
| 99 | - `internal/dbverify` records restore-drill verdicts keyed by **host** |
| 100 | (`dbverify.go:45`), at one deployment-wide key (`dbverify.go:28`), |
| 101 | published only by the hibernation drill. `Merge` drops hosts outside |
| 102 | the writing box's fleet (`dbverify.go:60-80`), so the last box to run |
| 103 | overwrites the others. A missing verdict renders `unverified`. |
| 104 | - Console gates: `instanceGate` (`apiinstances.go:70`), `s.audit` |
| 105 | (`server.go:2692`). The gate is **membership** via `AccountsFor(addr)` |
| 106 | (`apicli.go:208`) — people carry no owner/member role. |
| 107 | |
| 108 | **The console has no registry snapshot.** It opens the *local* |
| 109 | `registry.db` on its own box (`console/view.go:532-538`). The |
| 110 | `control/registry-snapshot/<box>.db` objects boxes write |
| 111 | (`host/snapshot_loop.go:34`) are read by nothing in `internal/console`. |
| 112 | Fleet-wide instance knowledge is `apps/<sqid>/<app>/instances/<host>.json`. |
| 113 | §6 and §8 both depend on this distinction. |
| 114 | |
| 115 | ## 4. Command surface |
| 116 | |
| 117 | ``` |
| 118 | carlos db ls --app <a> [--account <a>] |
| 119 | carlos db pull --app <a> [--instance <host>] --out <path> |
| 120 | carlos db shell --app <a> [--instance <host>] |
| 121 | carlos db query --app <a> [--instance <host>] --sql <text> |
| 122 | ``` |
| 123 | |
| 124 | `ls` is the discovery verb and the one that makes the rest legible: per |
| 125 | instance, the database the platform replicates, the age of the newest |
| 126 | replicated write, whether the instance is replicating right now, its |
| 127 | replication class *or* an honest "unknown" (§8), and its restore-drill |
| 128 | verdict. It **also names databases declared in `data.dbs` that the |
| 129 | platform does not replicate** (§7) — otherwise visible only in adopt |
| 130 | logs a member cannot read. |
| 131 | |
| 132 | `pull` writes the restored copy to a path you name and stops. `--out` is |
| 133 | **required** and must not already exist. |
| 134 | |
| 135 | `shell` restores into a private directory, opens an interactive prompt |
| 136 | against it with the embedded engine, and removes the directory on exit. |
| 137 | This is the closest thing to `rails console` a Go platform can honestly |
| 138 | offer: a REPL over your *data*, which exists, rather than over your |
| 139 | *app*, which cannot — Go is compiled ahead of time, and there is no |
| 140 | interpreter in the running binary to evaluate against its live heap. |
| 141 | |
| 142 | `query` is the non-interactive form — one statement, results to stdout, |
| 143 | exit code reflecting the query. This is the shape an agent uses and |
| 144 | should pipe cleanly. |
| 145 | |
| 146 | ### Where the copy lands |
| 147 | |
| 148 | litestream's `Restore` refuses an existing output path |
| 149 | (`replica.go:604`) and creates `<out>.tmp` itself with `os.Create`, at |
| 150 | the process umask (typically 0644). You **cannot** pre-create a 0600 |
| 151 | file for it to fill. |
| 152 | |
| 153 | So: always restore inside an `os.MkdirTemp` directory created 0700. That |
| 154 | confines the `.tmp` and the `-wal`/`-shm` files a read-only open still |
| 155 | needs (`check.go:96-98`). `shell` and `query` never leave that |
| 156 | directory. `pull` restores there, then **copies** to `--out` with fsync |
| 157 | and 0600 — a copy, not `os.Rename`, because the temp directory and the |
| 158 | user's destination are routinely on different filesystems. Open |
| 159 | question 2 is thereby answered: always 0600, and the user can loosen it. |
| 160 | |
| 161 | ### Read-only is enforced, not assumed |
| 162 | |
| 163 | Open with `file:<path>?mode=ro` — the form `lease.CheckDB` already uses |
| 164 | (`check.go:36`) — plus `PRAGMA query_only=1`. |
| 165 | |
| 166 | This matters most for the agent case, for a non-obvious reason. A copy |
| 167 | is already harmless to write to, so enforcement is not about protecting |
| 168 | production. It is about **not letting a caller believe it changed |
| 169 | production**. An agent that runs an `UPDATE` against a writable copy |
| 170 | sees `1 row affected` and concludes it fixed the bug. Against a |
| 171 | read-only copy it gets SQLite's `attempt to write a readonly database`, |
| 172 | and the error is the truth. Silent success on a discarded write is the |
| 173 | single worst outcome this command could produce. |
| 174 | |
| 175 | §11 pins that error string in a test, so the claim stays true. |
| 176 | |
| 177 | ## 5. The microcopy contract |
| 178 | |
| 179 | The safety property is a *fact the user has to hold in their head* — |
| 180 | that this is a copy, of a stated age, and that nothing they do here |
| 181 | reaches production. If they lose that fact, the command becomes a way to |
| 182 | make confident wrong decisions. The copy is load-bearing. |
| 183 | |
| 184 | Exact strings are **not drafted here**. They go through the |
| 185 | `copy-review` gate at implementation time, before being written into any |
| 186 | source file. What follows is what each string must accomplish. |
| 187 | |
| 188 | **R1 — Say it at handover, and name the source exactly.** The line |
| 189 | reporting a successful restore states that this is a copy, and names |
| 190 | host, database, the newest replicated transaction id and its time. Two |
| 191 | pulls must be distinguishable, and an agent must be able to cite an |
| 192 | "as of". |
| 193 | |
| 194 | **R2 — Age is a number, and carries its expected bound — or admits it |
| 195 | cannot.** Report the **age of the newest replicated write**, never a |
| 196 | "lag" (which implies a live comparison the client cannot make, §8) and |
| 197 | never an adjective. Beside it, report the replication class bound: an |
| 198 | awake `daily` instance is legitimately up to 24h behind, exactly the |
| 199 | case where an old copy is most easily mistaken for a current one. |
| 200 | |
| 201 | Where the class is **unknown** — which is the common case, because the |
| 202 | console only knows it for routes on its own box (§8) — say so. It must |
| 203 | never render as `stream` by default. A wrong bound is worse than no |
| 204 | bound: it converts "I don't know" into "one second old". |
| 205 | |
| 206 | **R3 — The interactive prompt carries it too.** A `shell` session can |
| 207 | outlive its banner by an hour. The prompt itself carries both the word |
| 208 | copy/replica and the as-of time. |
| 209 | |
| 210 | **R4 — Refuse loudly, explain briefly.** Four refusals need their own |
| 211 | voice: `replicate: "app"` (§7); the read-only write refusal (§4); the |
| 212 | residency refusal (§6), which must name residency rather than read as a |
| 213 | transient error; and the pooled-instance refusal (§6). Per the repo's |
| 214 | rule, store errors are logged rather than returned |
| 215 | (`apischedule.go:145-147`) — they carry bucket names and request ids. |
| 216 | |
| 217 | **R5 — Written for two readers.** A human skims; an agent parses. No |
| 218 | `carlos` verb has a `--json` flag today, so this **introduces** one, and |
| 219 | the fields are fixed here so the human and machine forms cannot drift: |
| 220 | |
| 221 | ``` |
| 222 | copy: true |
| 223 | as_of # time of the newest replicated write (§8) |
| 224 | replicated_txid |
| 225 | lease_held: bool # is the instance replicating right now (§6, §8) |
| 226 | sync_class # stream|batched|daily|unknown — never defaulted |
| 227 | verified # ok|failed|unverified |
| 228 | ``` |
| 229 | |
| 230 | **R6 — Name what is not there.** A database declared in `data.dbs` but |
| 231 | not replicated (§7) is listed as such. "The platform has no copy of |
| 232 | `audit.db`" must be said before someone goes looking for it. |
| 233 | |
| 234 | ## 6. Transport: the replica door |
| 235 | |
| 236 | **Decision: the console reads the region's data bucket directly.** |
| 237 | Option B — having the owning box mirror replicas into a staging prefix |
| 238 | in the deployment bucket — was considered and rejected. It needs a new |
| 239 | box-side pass plus a sweeper, which contradicts §9's "no path that runs |
| 240 | anything on the box"; its first-mirror latency is a CopyObject per LTX |
| 241 | file; it lands tenant database bytes in the deployment bucket, which the |
| 242 | moves design deliberately routed *around* (staging goes to a sibling |
| 243 | data bucket, never the deployment bucket); and it does not even remove |
| 244 | the data-bucket read, which `hibernate-verify` still needs. Its one |
| 245 | advantage over A — cross-region reach — is a residency violation rather |
| 246 | than a feature. |
| 247 | |
| 248 | ### Configuration |
| 249 | |
| 250 | The console takes the same triple the agent reads — |
| 251 | `CARLOS_HIBERNATE_BUCKET` / `_REGION` / `_PREFIX` — plus `CARLOS_REGION` |
| 252 | for the placement compare, and opens one `*s3.Client` reusing |
| 253 | `lease/endpoint.go`'s config path including `s3compat.Options()` and the |
| 254 | endpoint override (`endpoint.go:110-131`). |
| 255 | |
| 256 | There is deliberately **no edges-file lookup** for another box's bucket. |
| 257 | A bucket discovered that way is one IAM will refuse, so resolving it |
| 258 | only converts a clear refusal into a confusing one. |
| 259 | |
| 260 | ### Whether this needs an IAM change |
| 261 | |
| 262 | Probably not. The console opens S3 through the default credential chain |
| 263 | (`cmd/console/main.go:394-399`) — the box's instance role, the same role |
| 264 | the activator on that box uses to *write* `instances/<host>/`. If the |
| 265 | console's box hosts any hibernating instance, the read authority is |
| 266 | already there. If not, it is one read-only, in-region grant, smaller |
| 267 | than the `iam:CreateUser`/`PutUserPolicy` the console already holds via |
| 268 | `CARLOS_STORE_BACKEND=aws` (`objstore/aws.go:39-46`). |
| 269 | |
| 270 | ### Resolution, and what is refused before S3 is touched |
| 271 | |
| 272 | `apps/<sqid>/<app>/instances/<host>.json` → `ValidHostShape` |
| 273 | (`provision.go:250-262`, written as path-traversal defence — `path.Join` |
| 274 | alone cleans `../x` to `x` rather than refusing it) → then three |
| 275 | refusals, none of which make an S3 call: |
| 276 | |
| 277 | - `rec.Region != CARLOS_REGION` (both `""` in a single-region |
| 278 | deployment) → residency refusal naming the region. |
| 279 | - a `fleet/<name>` region (`console/fleets.go:535`) → residency refusal; |
| 280 | the data is on the customer's own box. §13 records the v2. |
| 281 | - `rec.Pool != ""` → refusal; a pooled record's data lives on the pool |
| 282 | box, the same class `movespass.go:150-153` already refuses. |
| 283 | |
| 284 | An `AccessDenied` from S3 maps to the residency refusal as a backstop, |
| 285 | never to a generic 502. |
| 286 | |
| 287 | ### The door |
| 288 | |
| 289 | Two handlers, one gate, one rate budget: |
| 290 | |
| 291 | - `list(app, instance, level)` → `[{level, min_txid, max_txid, size, created_at}]` |
| 292 | - `get(app, instance, level, min_txid, max_txid)` with `Range` support |
| 293 | |
| 294 | Implement them as an HTTP adapter over litestream's own |
| 295 | `lss3.ReplicaClient` **read methods**, with `Path` set per request. That |
| 296 | way the server derives the key layout |
| 297 | (`<prefix>/%04x/%016x-%016x.ltx`, `s3/replica_client.go:655-656`) from |
| 298 | the same code the writer uses, and reader and writer cannot drift. |
| 299 | |
| 300 | Addressing by coordinate rather than key is what keeps `lock.json` |
| 301 | — which sits under the same prefix and is not a member's to fetch — |
| 302 | unreachable. |
| 303 | |
| 304 | Following the edge store door's F8 precedent (`storedoor.go:55-59`): |
| 305 | bytes **stream through the console**, never a presigned URL, and nothing |
| 306 | the request supplies may influence which bucket or client is used. |
| 307 | |
| 308 | Six wire specifics the implementation must honour: |
| 309 | |
| 310 | 1. `list` accepts **levels 0–9 inclusive**. `CalcRestorePlan` asks for |
| 311 | `SnapshotLevel` (= 9, `compaction_level.go:9`) first, then 8 down to |
| 312 | 0 (`replica.go:1452, 1476-1485`). The activator writes only 0–3 and |
| 313 | 9; the door answers **empty** for the rest, never 404. |
| 314 | 2. Listings are ordered by `min_txid` **ascending** — |
| 315 | `restoreLevelCursor` assumes filename order, which S3 gives because |
| 316 | the names are zero-padded hex. |
| 317 | 3. `seek` and `useMetadata` are not on the wire. The client filters |
| 318 | `seek` locally; nothing in a non-timestamp restore reads `CreatedAt` |
| 319 | except debug logs. |
| 320 | 4. `size`, `min_txid` and `max_txid` are **mandatory**. `Restore` |
| 321 | rejects `Size < ltx.HeaderSize` (`replica.go:630`) and |
| 322 | `ResumableReader` uses size to detect a premature EOF |
| 323 | (`resumable_reader.go:96`). Checksums are not needed. |
| 324 | 5. `get` Range semantics match the S3 client exactly: |
| 325 | `bytes=<offset>-` when `size == 0`, else |
| 326 | `bytes=<offset>-<offset+size-1>` (`s3/replica_client.go:647-652`). |
| 327 | A missing object is a 404 the client maps to `os.ErrNotExist`. |
| 328 | 6. `ls` needs a third, **derived** field: `lease_held` (plus |
| 329 | `expires_at`), computed console-side from `lock.json` |
| 330 | (`leaser.go:31-35`; a released lease is deleted, `s3/leaser.go:180`). |
| 331 | The lock's **body is never returned** — `Owner` is the box's |
| 332 | identity. This is how §8's "is it replicating now" is answered |
| 333 | without exposing an object coordinate addressing deliberately hides. |
| 334 | |
| 335 | ### Who may open it |
| 336 | |
| 337 | `PrincipalUser` only. Every service principal — the legacy app token |
| 338 | (role `instance`) and every named service credential |
| 339 | (`publish`/`instance`/`operate`/`admin`) — gets the same |
| 340 | `http.NotFound` as a stranger. |
| 341 | |
| 342 | A 403 would be an oracle: it tells any token holder that a db door |
| 343 | exists for this app. The gate ethos here is one byte-identical 404. |
| 344 | |
| 345 | No legitimate caller loses anything — an app reads its own database |
| 346 | live, and CI has no business pulling production data. `admin` is refused |
| 347 | **deliberately**, despite already reaching `store grant`. |
| 348 | |
| 349 | > **Revision, 2026-10-01 (implementation; Paul may overrule).** Service |
| 350 | > principals get a 403, not the 404 above. The two door routes are |
| 351 | > registered with the zero role Set, the repo's closed-door mechanism |
| 352 | > since 2026-08-18, and the shared `cliRoute` gate answers every service |
| 353 | > credential with one constant 403 ("closed to service credentials") |
| 354 | > BEFORE any account or app lookup. The body is the same for every role, |
| 355 | > account and app, so it reveals only that the door exists, which is |
| 356 | > public in the code, and nothing about this app or account. Members who |
| 357 | > are not in the account, and unknown apps and instances, still get the |
| 358 | > byte-identical 404. A literal 404 would need either a role Set the |
| 359 | > handler then overrides, or a change to the shared gate for every closed |
| 360 | > door. |
| 361 | |
| 362 | Every successful open writes an `s.audit` row naming account, app, host, |
| 363 | database and the replicated TXID served. |
| 364 | |
| 365 | ### The read-only replica client |
| 366 | |
| 367 | The CLI implements litestream's `ReplicaClient` over this door, the way |
| 368 | `internal/fleetapi` implements `release.Store` over the edge store door, |
| 369 | and for the same reason: the member has no AWS credentials, by design. |
| 370 | |
| 371 | The interface has eight methods (`replica_client.go:19-50`). This |
| 372 | implementation supplies the five a restore needs — `Type`, `Init`, |
| 373 | `LTXFiles`, `OpenLTXFile`, `SetLogger` — and the three mutating ones |
| 374 | (`WriteLTXFile`, `DeleteLTXFiles`, `DeleteAll`) return an error |
| 375 | unconditionally. |
| 376 | |
| 377 | **The client cannot corrupt the replica, because it has no code that |
| 378 | writes to one.** Not "is not supposed to" — has none. §12 records the |
| 379 | verification and the three conditions it depends on. |
| 380 | |
| 381 | ## 7. Which database — one, not eight |
| 382 | |
| 383 | `caps.Data` allows up to eight names (`maxDBs = 8`), but **a route |
| 384 | replicates exactly one database**: `Route.DBPath` is single-valued and |
| 385 | only `DBs[0]` is stamped (`host/caps_pass.go:294`, whose own adopt |
| 386 | warning says so verbatim). The activator's `Replicate` takes one |
| 387 | `dbPath`. |
| 388 | |
| 389 | So there is no `--db` flag and no selection story: |
| 390 | |
| 391 | - The replicated database is implied by the instance. |
| 392 | - `--instance` selects among an app's instances — inferred when there is |
| 393 | one, required when there are several, because guessing hands someone |
| 394 | the wrong data while telling them it is the right data. |
| 395 | - `ls` names the **declared-but-unreplicated** entries (`DBs[1:]`), |
| 396 | surfacing `dataAdvice`'s warning to a member who cannot read adopt |
| 397 | logs (R6). |
| 398 | |
| 399 | ### `replicate: "app"` has no replica |
| 400 | |
| 401 | An app declaring `replicate: "app"` streams its own WAL and the platform |
| 402 | replicates **nothing** for it (`caps.DataReplicateApp`; `caps.go:416` |
| 403 | refuses `data.dbs` alongside it precisely to keep the two-replicators |
| 404 | hazard unreachable). |
| 405 | |
| 406 | This must fail naming that cause. The dangerous failure is a generic "no |
| 407 | replica found", which reads as transient and invites a retry loop; the |
| 408 | operator needs to know the platform was never going to have this data. |
| 409 | |
| 410 | ## 8. Age, and the three things that are not lag |
| 411 | |
| 412 | The client **cannot compute lag**. `ops.freshness` gets the live change |
| 413 | counter from the source file on the box (`freshness.go:96-115`), which |
| 414 | neither console nor CLI can reach. Any claimed distance from live would |
| 415 | be invented. |
| 416 | |
| 417 | What the replica alone supports: |
| 418 | |
| 419 | 1. **Age of the newest replicated write.** Take it from the **LTX |
| 420 | header timestamp**, not S3 `LastModified` — a compacted file's |
| 421 | `LastModified` is compaction time, and the `ltx` compactor carries |
| 422 | `maxHdr.Timestamp` through (`compactor.go:118`). One ranged GET of |
| 423 | `ltx.HeaderSize` bytes on the max-TXID file gives the true |
| 424 | newest-write time at any level. Using `LastModified` would report an |
| 425 | L3 compaction as a fresh write — a stale copy looking current, the |
| 426 | exact failure this command exists to prevent. |
| 427 | 2. **Whether it is replicating now** — `lease_held`, derived from |
| 428 | `lock.json` (§6). |
| 429 | 3. **The expected bound — when it is knowable.** `Replication` lives on |
| 430 | `registry.Route` (`registry.go:126`) in the **box** registry; |
| 431 | `provision.Record` has no such field. So the console knows the class |
| 432 | only for routes on its own box. For an in-region sibling it is |
| 433 | genuinely unknown, and R2 requires saying so rather than defaulting. |
| 434 | §13 records the additive follow-up. |
| 435 | 4. **The drill verdict** — `dbverify`, per host, hibernation drill only. |
| 436 | |
| 437 | The first draft claimed "no explicit sync interval, so 1s default, |
| 438 | therefore seconds behind live". That is wrong for the instances this |
| 439 | targets, and wrong in the dangerous direction: classes are `""`/`stream` |
| 440 | ≈ 1s, `batched` 5m, `daily` 24h (`activator.go:205-230`, |
| 441 | `lease.go:245-252`). An awake `daily` instance is **legitimately up to a |
| 442 | day behind**, and reporting its age without its bound makes correct |
| 443 | platform behaviour look like current data. |
| 444 | |
| 445 | Retention differs between the two replicators too: `SnapshotInterval` |
| 446 | 24h / `SnapshotRetention` 7d (`ops/litestream.go:64-65`) reach only |
| 447 | `/etc/litestream.yml`. The activator's embedded store never sets |
| 448 | `Retention`, so hibernating replicas run on litestream's |
| 449 | `DefaultRetention` of **24h**. |
| 450 | |
| 451 | ### A sleeping instance is consistent, not necessarily complete |
| 452 | |
| 453 | `stop()` logs and swallows a failed final checkpoint |
| 454 | (`activator.go:876-881`) — the hole the instance-move spec had to add a |
| 455 | re-checkpoint for. A parked replica is consistent at a *committed* |
| 456 | boundary (LTX carries committed frames only) but may trail the sleep |
| 457 | point. |
| 458 | |
| 459 | So: "consistent as of its last successful sync, which on a clean sleep |
| 460 | is the sleep point" — never "complete as of when it went to sleep". |
| 461 | `carlos restart` and Bounce go through the same `stop()`. |
| 462 | |
| 463 | ## 9. Security posture |
| 464 | |
| 465 | **The exposure is real and is the point of the gate.** This command |
| 466 | moves production data onto a laptop. Read-only does not reduce that; a |
| 467 | copy leaks exactly as well as an original. Hence: members only, no |
| 468 | service principals of any role, an audit row per open, `--out` required |
| 469 | and refused if it exists, 0600 on anything kept, a 0700 private |
| 470 | directory for anything temporary. |
| 471 | |
| 472 | **Nothing new is reachable on the box, and nothing new runs there.** No |
| 473 | door on the tenant process, no new agent verb, no pass in the converge |
| 474 | loop. The sandbox (`DynamicUser`, `ProtectSystem=strict`, empty |
| 475 | `CapabilityBoundingSet`) is untouched, and the same-uid sibling residual |
| 476 | in `activator/admintoken.go:8-12` is not widened, because no part of |
| 477 | this design runs on a box at all. Option B would have forfeited this |
| 478 | sentence; A keeps it. |
| 479 | |
| 480 | **One new read authority:** the console gains in-region, read-only |
| 481 | access to the data bucket's `instances/*`. Bounded by residency — it |
| 482 | cannot reach another region's data even if asked, and §6 refuses before |
| 483 | trying rather than surfacing an `AccessDenied`. |
| 484 | |
| 485 | **The failure mode to design against is not corruption, it is |
| 486 | confidence.** Every serious way this command hurts someone routes |
| 487 | through a caller believing old data is current, or believing a write |
| 488 | landed. §5 and §4's read-only enforcement exist for that; they are not |
| 489 | decoration on the security story, they are most of it. |
| 490 | |
| 491 | ## 10. Failure matrix |
| 492 | |
| 493 | Each row is a test. |
| 494 | |
| 495 | | Condition | Result | |
| 496 | |---|---| |
| 497 | | Not a member of the account | Byte-identical 404, matching `instanceGate` | |
| 498 | | Any service principal (app token, or `publish`/`instance`/`operate`/`admin`) | Same byte-identical 404 — never a 403 | |
| 499 | | No such app | Byte-identical 404 | |
| 500 | | `replicate: "app"` | Refusal naming that cause specifically (§7) | |
| 501 | | Several instances, no `--instance` | Refusal listing the hosts | |
| 502 | | Instance in another region | Residency refusal naming the region — no S3 call | |
| 503 | | Instance region `fleet/<name>` | Residency refusal — no S3 call (§13 records the v2) | |
| 504 | | `rec.Pool != ""` | Refusal — data lives on the pool box | |
| 505 | | S3 `AccessDenied` | Mapped to the residency refusal, never a generic 502 | |
| 506 | | Declared-but-unreplicated db named | Refusal naming it as never replicated (§7) | |
| 507 | | Nothing replicated yet | "Nothing replicated yet" — match **both** `ErrNoSnapshots` and `ErrTxNotAvailable` (`lease.go:394`) | |
| 508 | | `list` for a level the activator never writes | Empty list, **not** 404 (§6.1) | |
| 509 | | Replica changed under the restore | One retry around the whole `Restore` (§12) | |
| 510 | | Restore fails integrity check | Refuse to hand over the file; surface the `dbverify` verdict | |
| 511 | | Instance asleep | Succeeds; reports age, `lease_held: false`, class or unknown | |
| 512 | | Instance awake on `daily` class | Succeeds; age reported beside its 24h bound (R2) | |
| 513 | | Class not knowable (sibling box) | `sync_class: "unknown"`, never `stream` (R2) | |
| 514 | | Write attempted in `shell`/`query` | SQLite `attempt to write a readonly database`, surfaced plainly | |
| 515 | | `--out` omitted, or already exists | Usage error | |
| 516 | | Host failing `ValidHostShape` | Refused before it becomes a key segment | |
| 517 | | Client asked to write a replica | Unconditional error — no code path exists | |
| 518 | |
| 519 | ## 11. Testing shape |
| 520 | |
| 521 | Following `apischedule_test.go` and `internal/lease/check_test.go`: |
| 522 | |
| 523 | - **Door:** the full gate matrix, plus a coordinate-confinement test |
| 524 | asserting `lock.json` and any key outside the resolved prefix are |
| 525 | unreachable. That test is what keeps this from becoming an object |
| 526 | proxy as the code ages. |
| 527 | - **Wire conformance:** levels 0–9 answer empty rather than 404; |
| 528 | listings ascend by `min_txid`; `size`/`min_txid`/`max_txid` always |
| 529 | present; Range semantics match `s3/replica_client.go:647-652` for both |
| 530 | the `size == 0` and sized forms. |
| 531 | - **Replica client, both halves:** the three mutating methods error |
| 532 | without touching the network **and** a call-counting fake asserts they |
| 533 | stay at zero across a full restore. The second half is what catches a |
| 534 | litestream upgrade changing the restore path. |
| 535 | - **`ReplicaClientV3` negative assertion** — §12 condition (i). |
| 536 | - **Compile-time** `var _ litestream.ReplicaClient = (*client)(nil)`. |
| 537 | - **Freshness:** a table over (age, `lease_held`, class-or-unknown, |
| 538 | verdict) asserting the wording distinguishes asleep from failing, |
| 539 | always carries the bound or admits it is unknown, and **never** |
| 540 | renders unknown as `stream`. |
| 541 | - **Age source:** a compacted L3 file whose `LastModified` is far newer |
| 542 | than its LTX header timestamp reports the header time (§8.1). |
| 543 | - **Selection:** `caps.Data` fixtures for `DBs[1:]` unreplicated and the |
| 544 | `replicate: "app"` refusal. |
| 545 | - **Read-only:** an `UPDATE` in `query` fails with the pinned error |
| 546 | string, and the file on disk is byte-identical afterwards. |
| 547 | - **Permissions:** the temp directory is 0700, `--out` lands 0600, and |
| 548 | `pull` succeeds when temp and destination are on different |
| 549 | filesystems. |
| 550 | |
| 551 | Definition of done includes a `docs/site/cli.md` heading — the |
| 552 | `internal/docsite` gate fails without one (AGENTS.md). |
| 553 | |
| 554 | ## 12. Why the safety claim holds |
| 555 | |
| 556 | Verified against litestream v0.5.14 during review, because §6's central |
| 557 | claim depends on it. |
| 558 | |
| 559 | `Replica.Restore` (`replica.go:544-720`) → `CalcRestorePlan` |
| 560 | (`replica.go:1485ff`, `LTXFiles` per level) → `NewResumableReader`, |
| 561 | whose client parameter is the narrow `LTXFileOpener` interface |
| 562 | (`internal/resumable_reader.go:51,72`) — `OpenLTXFile` only. |
| 563 | `WriteLTXFile`, `DeleteLTXFiles` and `DeleteAll` are reached only from |
| 564 | `Sync`, `EnforceRetention`, and `db.go`'s snapshot/retention paths |
| 565 | (`replica.go:192,312`; `db.go:2470,2526,2649`), none of which run |
| 566 | without a `*DB`/`Store`. With a nil `*DB`, they are unreachable. |
| 567 | |
| 568 | Three conditions this depends on. Each is a test in §11: |
| 569 | |
| 570 | 1. **The client must not implement `ReplicaClientV3`** (`v3.go:145`). |
| 571 | `Restore` type-asserts for it (`replica.go:612`) and would take the |
| 572 | v0.3 path instead. |
| 573 | 2. **Never set `RestoreOptions.Follow`.** It writes into the local |
| 574 | output and a `-txid` sidecar, and keeps polling. |
| 575 | 3. **Run `lease.CheckDB` after the restore** rather than relying on |
| 576 | `opt.IntegrityCheck`, matching `lease.Restore`'s existing pattern. |
| 577 | |
| 578 | ### What the claim does *not* cover |
| 579 | |
| 580 | `check.go`'s reasoning is about not *shadowing the writer*. It does not |
| 581 | promise the reader a stable listing. `CalcRestorePlan` lists every level |
| 582 | once and opens lazily, so the writer's L0 retention |
| 583 | (`DefaultL0Retention` 5m, checked hourly by the activator, |
| 584 | `lease.go:78`) can delete a planned file before it is opened. |
| 585 | |
| 586 | The outcome is a **failed** restore, never a torn one: the output is |
| 587 | `<out>.tmp`, renamed only on success (`replica.go:661-663`). |
| 588 | |
| 589 | The retry must wrap the **whole `Restore`**, not an individual read: |
| 590 | `ResumableReader` treats `ErrNotExist` as fatal and does not retry |
| 591 | (`resumable_reader.go:72-75`), so a vanished planned file needs a fresh |
| 592 | plan, which only a new `Restore` produces. |
| 593 | |
| 594 | ## 13. Open questions and follow-ups |
| 595 | |
| 596 | 1. **Fleet-homed instances (v2).** §6 refuses them: their data is in the |
| 597 | customer's bucket and IAM will not cross. The reverse tunnel |
| 598 | (`edge/tunnel.go`) is a real synchronous edge→box path for exactly |
| 599 | the hosts that dial in — so the v2 is the box serving its own replica |
| 600 | coordinates over its tunnel channel, with the console never touching |
| 601 | the customer's bucket. Different transport, same wire shape. |
| 602 | 2. **An additive `replication` field on the instance record.** Closes |
| 603 | §8's `sync_class: "unknown"` for in-region siblings. Wire-additive |
| 604 | and therefore allowed; worth doing regardless of this command. |
| 605 | 3. **`shell`'s prompt.** An embedded REPL is small code but a real |
| 606 | surface (history, formatting, dot commands). Ship `ls`/`pull`/`query` |
| 607 | first, then `shell`. |
| 608 | 4. **Console web view.** Out of scope, but §6's door is what it would |
| 609 | use; confirm the coordinate-addressed shape does not foreclose it. |
| 610 | 5. **Past points in time.** Not offered. Given the activator's effective |
| 611 | 24h `DefaultRetention` (§8), is "current replica only" the right |
| 612 | permanent scope? |
| 613 | 6. **Rate limiting.** Pulls are large; neither the token-keyed instance |
| 614 | burst nor the address-keyed write budget fits. Needs its own budget. |
| 615 | |
| 616 | ### Decomposition |
| 617 | |
| 618 | Three units, in order: (1) door + read-only client + `pull` + `query`; |
| 619 | (2) `ls` and its freshness fields; (3) `shell`. |
| 620 | |
| 621 | ## 14. What review changed |
| 622 | |
| 623 | **Round one.** The safety claim survived verification (§12). The |
| 624 | transport did not: "the console serves objects it already stores" was |
| 625 | false. Also corrected — a route replicates exactly one database, not up |
| 626 | to eight; `daily` class is legitimately 24h behind, not "seconds behind |
| 627 | live"; a 403 for service principals was an oracle; litestream is a |
| 628 | dependency, not vendored; `FleetDBs` is the complement of the |
| 629 | activator's set; the console has no registry snapshot; `dbverify` is |
| 630 | keyed by host; the mode-0600 temp file was not constructible. |
| 631 | |
| 632 | **Round two.** Option A chosen and B rejected on its merits (§6), so §9 |
| 633 | regains "nothing runs on the box". The edges-file lookup was dropped as |
| 634 | buying nothing IAM will allow. Six wire specifics added, including |
| 635 | levels 0–9 answering empty rather than 404 and mandatory `size`. Two new |
| 636 | findings, both in the direction of overstating freshness: |
| 637 | |
| 638 | - **`sync_class` is unknowable off-box** — `Replication` is on |
| 639 | `registry.Route`, not `provision.Record`. R2 now requires an explicit |
| 640 | "unknown" that must never render as `stream`. |
| 641 | - **Age must come from the LTX header timestamp**, not S3 |
| 642 | `LastModified`, or a compaction reads as a fresh write. |
| 643 | |
| 644 | Plus `lease_held` as a derived `ls` field (answering "is it replicating" |
| 645 | without exposing `lock.json`), the retry wrapping the whole `Restore`, |
| 646 | and `pull` copying rather than renaming across filesystems. |
| 647 | |
amadan