amadan
spec: carlos db — rework transport and freshness after review
The safety claim survived verification: litestream v0.5.14's restore path with a nil *DB reaches only LTXFiles and OpenLTXFile, so a client with no write methods genuinely cannot corrupt a replica. Section 12 records that and the three conditions it rests on (no ReplicaClientV3, never Follow, CheckDB afterwards), each now a test. The transport did not survive. "The console serves objects it already stores" was false — replicas live in per-customer data buckets the console has no client for, and the instance-move design bars cross-region reads at IAM. Section 6 now states that as an open decision between console-reads-data-bucket and mirror-through-staging, which is a residency call rather than mine to make. Three further corrections, all in the direction of overstating safety: a route replicates exactly one database, not up to eight; a `daily` class instance is legitimately 24h behind, not "seconds behind live"; and a 403 for service principals was an oracle for the door's existence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PEKBoE5EyWTUJPU15ZoenD
1 file changed,
+425
−233
docs/superpowers/specs/2026-09-02-carlos-db-design.md+425 −233
diff --git a/docs/superpowers/specs/2026-09-02-carlos-db-design.md b/docs/superpowers/specs/2026-09-02-carlos-db-design.md| index 8d127b4bc..0a7d2f590 100644 |
| --- a/docs/superpowers/specs/2026-09-02-carlos-db-design.md |
| +++ b/docs/superpowers/specs/2026-09-02-carlos-db-design.md |
| @@ -1,6 +1,6 @@ |
| # `carlos db` — reading an app's database from its replica |
| -Design, 2026-09-02. |
| +Design, 2026-09-02. Revised after review, same day (§14 records what changed). |
| ## 1. What this is for |
| @@ -26,7 +26,7 @@ inference. Every design decision below is downstream of that. |
| ### Scope |
| In: reading an app's platform-replicated SQLite, from the CLI, by an |
| -account owner, as a local file. |
| +account member, as a local file. |
| Out: writing to tenant data by any path; querying the live database; |
| non-SQLite stores; the console web UI (this is CLI-first — a web view |
| @@ -39,18 +39,24 @@ Two forks were considered and settled before this spec. |
| **Replica, not live.** The alternative was a query door onto the running |
| instance. That needs a synchronous console→instance path, which CARLOS |
| does not have for ordinary boxes: the control plane is the bucket and |
| -boxes poll (`schedule.TickEvery` is 15s; `logship` drains every 10s). |
| -Building one would mean a new transport *and* a new door onto the tenant |
| -process, to deliver a capability whose main use is a read. The replica |
| -is already there, already off-box, and already proven restorable by the |
| -`carlos ops restore-verify` drill. |
| +boxes poll (`schedule.TickEvery` is 15s, `logship` drains every 10s, and |
| +the agent polls `control/serial` every 2s). Building one would mean a new |
| +transport *and* a new door onto the tenant process, to deliver a |
| +capability whose main use is a read. The replica is already there, |
| +already off-box, and already proven restorable by the restore drills. |
| **Client-side query, not console-side.** The console serves replica |
| **bytes**; the member's own machine restores and queries. The console |
| therefore never executes tenant SQL, never holds a restored tenant |
| database, and needs no query timeouts, row caps, or result-set memory |
| -budget. It serves objects it already stores — which is a thing it |
| -already knows how to do safely. |
| +budget. |
| + |
| +> **Note.** The first draft justified this as "the console serves objects |
| +> it already stores". That is false — see §6. The console holds the |
| +> deployment bucket; replicas live in customer data buckets. The |
| +> client-side argument survives on its own merits, but the transport it |
| +> rests on is a genuinely new door, and §6 is now the hardest part of |
| +> this design rather than the easiest. |
| The client-side choice also buys the microcopy problem's best available |
| answer, discussed in §5: the user is handed an actual file, so "this is |
| @@ -62,240 +68,375 @@ rows, and freshness is replica freshness rather than live. |
| ## 3. What already exists |
| -This design is mostly assembly. The parts: |
| - |
| -- `lease.Restore(ctx, dbPath, bucket, prefix, region)` restores a |
| - replica to a file. `lease.VerifyReplica` does it without taking the |
| - lease, and `internal/lease/check.go` documents at length why that is |
| - safe against a live replicator: litestream restores a committed |
| - snapshot, and the reader never writes. |
| -- litestream v0.5.14 is a **vendored Go library**, not a binary we |
| - shell out to. `litestream.NewReplicaWithClient(nil, client)` is its |
| - own URL-restore pattern, already used at `internal/lease/lease.go:390`. |
| -- `modernc.org/sqlite` v1.55.0 is already a direct dependency and is |
| - **pure Go**. The query engine embeds; `carlos db` needs no `sqlite3` |
| - on the user's machine and no cgo. |
| -- `activator.HibernatePrefix(prefix, host)` is the replica prefix layout |
| - for exec-backed instances (`<CARLOS_HIBERNATE_PREFIX>/<host>/`, prefix |
| - `instances` in production). |
| -- `ops.FleetDBs` is the box-wide replicator's set — the "glob proposes, |
| - registry disposes" rule for platform-replicated databases. |
| -- `internal/dbverify` already records, per database, whether its replica |
| - is `ok`, `failed`, or `unverified`, at `control/restore-verify.json`. |
| -- `internal/console` has the gate shape to copy: `instanceGate` |
| - (`apiinstances.go:70`) and `apiScheduleApp` (`apischedule.go:96`) — |
| - same principal proof, byte-identical 404, shared rate limiter — and |
| - `s.audit` (`server.go:2692`). |
| - |
| -Nothing here needs a new replication path, a new bucket, or a change to |
| -how any app runs. |
| +Accurate as of 2026-09-02. There are **two** replicators in this |
| +platform and they are easy to conflate: |
| + |
| +- **The activator's embedded library.** litestream v0.5.14 is a `go.mod` |
| + dependency (not vendored — there is no `vendor/`), linked into the |
| + `carlos` binary. It replicates **hibernating exec-backed instances** |
| + under the lease. `lease.Restore` (`lease/lease.go:378`) restores from |
| + it, using litestream's own nil-`*DB` URL-restore pattern |
| + (`lease.go:390`). This is the set `carlos db` primarily serves. |
| +- **The box-wide binary.** A separately pinned litestream **0.5.15** |
| + executable (`ops/litestream.go:43`), driven by a generated |
| + `/etc/litestream.yml`, replicating everything else. |
| + `carlos ops restore-verify` shells out to it. |
| + |
| +`ops.FleetDBs` is the box-wide unit's set, and it **excludes** |
| +hibernating exec routes (`ops/hibernate_verify.go:5-8`) — it is the |
| +complement of the activator's set, not the whole. |
| + |
| +Other parts this design uses: |
| + |
| +- `lease.VerifyReplica` restores without taking the lease; |
| + `internal/lease/check.go:66-79` documents why that cannot shadow the |
| + true holder. §12 records how far that reasoning actually reaches. |
| +- `activator.HibernatePrefix(prefix, host)` = `path.Join(prefix, host)` |
| + (`activator.go:113`), prefix `instances` in production. |
| +- `modernc.org/sqlite` v1.55.0 is a direct dependency and **pure Go** |
| + (`go.mod:23`). litestream itself imports it, so the CLI already links |
| + it; the no-cgo claim holds. The query engine embeds — `carlos db` |
| + needs no `sqlite3` on the user's machine. |
| +- `internal/dbverify` records restore-drill verdicts — but keyed by |
| + **host**, not by database (`dbverify.go:45`), at one deployment-wide |
| + key (`dbverify.go:28`), and only the hibernation drill publishes. |
| + `Merge` drops hosts outside the writing box's fleet |
| + (`dbverify.go:60-80`), so the last box to run overwrites the others. |
| + Treat a missing verdict as `unverified`, which is the record's own |
| + discipline. |
| +- `internal/console` gates: `instanceGate` (`apiinstances.go:70`), |
| + `apiScheduleApp` (`apischedule.go:96`), `s.audit` (`server.go:2692`). |
| + The gate is **membership** via `AccountsFor(addr)` (`apicli.go:208`) — |
| + people carry no owner/member role, so this spec says "member". |
| + |
| +**The console has no registry snapshot.** It opens the *local* |
| +`registry.db` on its own box (`console/view.go:532-538`). The |
| +`control/registry-snapshot/<box>.db` objects boxes write |
| +(`host/snapshot_loop.go:34`) are read by nothing in `internal/console`. |
| +Fleet-wide instance knowledge lives in |
| +`apps/<sqid>/<app>/instances/<host>.json`. Any claim that the console |
| +"resolves from its registry snapshot" is wrong; §6 resolves from the |
| +instances records. |
| ## 4. Command surface |
| ``` |
| carlos db ls --app <a> [--account <a>] |
| -carlos db pull --app <a> [--instance <host>] [--db <name>] --out <path> |
| -carlos db shell --app <a> [--instance <host>] [--db <name>] |
| -carlos db query --app <a> [--instance <host>] [--db <name>] --sql <text> |
| +carlos db pull --app <a> [--instance <host>] --out <path> |
| +carlos db shell --app <a> [--instance <host>] |
| +carlos db query --app <a> [--instance <host>] --sql <text> |
| ``` |
| `ls` is the discovery verb and the one that makes the rest legible: for |
| -each database the platform replicates for this app, it prints the |
| -instance, the database name, how far behind the replica is, and its |
| -last restore-drill verdict from `dbverify`. A member who has never used |
| -this command should be able to type `ls` first and understand what the |
| -other three will hand them. |
| +each instance, the database the platform replicates, how old the newest |
| +replicated write is, the instance's replication class, and its |
| +restore-drill verdict. It **also names databases the app declared in |
| +`data.dbs` that the platform does not replicate** (§7) — that fact is |
| +otherwise only visible in adopt logs a member cannot read. |
| `pull` writes the restored copy to a path you name and stops. `--out` is |
| -**required** — a command that leaves a production database somewhere you |
| -did not name is not a command we should ship. |
| - |
| -`shell` restores to a mode-0600 temp file, opens an interactive prompt |
| -against it with the embedded engine, and deletes the file on exit. This |
| -is the interactive verb, and it is the closest thing to `rails console` |
| -that a Go platform can honestly offer: a REPL over your *data*, which |
| -exists, rather than over your *app*, which cannot (Go is compiled ahead |
| -of time; there is no interpreter in the running binary and no way to |
| -evaluate an expression against its live heap). |
| +**required**, and must not already exist — a command that leaves a |
| +production database somewhere you did not name is not one we should |
| +ship. |
| + |
| +`shell` restores into a private directory, opens an interactive prompt |
| +against it with the embedded engine, and removes the directory on exit. |
| +This is the closest thing to `rails console` that a Go platform can |
| +honestly offer: a REPL over your *data*, which exists, rather than over |
| +your *app*, which cannot (Go is compiled ahead of time; there is no |
| +interpreter in the running binary and no way to evaluate an expression |
| +against its live heap). |
| `query` is the non-interactive form — one statement, results to stdout, |
| -exit code reflects the query. This is the shape an agent uses, and it |
| -should be the shape that is easy to pipe. |
| +exit code reflecting the query. This is the shape an agent uses and |
| +should be the shape that pipes cleanly. |
| + |
| +### Where the copy lands |
| + |
| +litestream's `Restore` refuses an existing output path |
| +(`replica.go:604`) and creates `<out>.tmp` itself with `os.Create`, |
| +so it lands at the process umask (typically 0644) and you **cannot** |
| +pre-create a mode-0600 file for it to fill. |
| + |
| +So: always restore inside an `os.MkdirTemp` directory created 0700. That |
| +confines the `.tmp`, and also the `-wal`/`-shm` files a read-only open |
| +still needs (`check.go:96-98`). `shell` and `query` never leave that |
| +directory. `pull` restores there and then moves the result to `--out`, |
| +chmod 0600. Q2 in §13 is thereby answered: always 0600, and the user can |
| +loosen it themselves. |
| ### Read-only is enforced, not assumed |
| -Every path opens the copy with SQLite in read-only mode |
| -(`query_only`, and the file opened `mode=ro`). This matters most for the |
| -agent case and it is worth being explicit about why: a copy is already |
| -harmless to write to, so enforcement is not about protecting production. |
| -It is about **not letting a caller believe it changed production**. An |
| -agent that runs an `UPDATE` against a writable copy sees `1 row |
| -affected` and concludes it has fixed the bug. The same agent against a |
| -read-only copy gets an error, and the error is the truth. Silent success |
| -on a discarded write is the single worst outcome this command could |
| -produce, and read-only mode is what forecloses it. |
| +Open with `file:<path>?mode=ro` — the form `lease.CheckDB` already uses |
| +(`check.go:36`) — plus `PRAGMA query_only=1`. |
| + |
| +This matters most for the agent case, for a non-obvious reason: a copy |
| +is already harmless to write to, so enforcement is not about protecting |
| +production. It is about **not letting a caller believe it changed |
| +production**. An agent that runs an `UPDATE` against a writable copy |
| +sees `1 row affected` and concludes it fixed the bug. Against a |
| +read-only copy it gets SQLite's `attempt to write a readonly database`, |
| +and the error is the truth. Silent success on a discarded write is the |
| +single worst outcome this command could produce. |
| + |
| +§11 pins that error string in a test, so the claim stays true. |
| ## 5. The microcopy contract |
| The safety property of this command is a *fact the user has to hold in |
| -their head* — that they are looking at a copy, seconds to minutes stale, |
| -and that nothing they do here reaches production. If they lose that |
| -fact, the command becomes a way to make confident wrong decisions. |
| -So the copy is load-bearing and is specified here as requirements. |
| +their head* — that they are looking at a copy, of a stated age, and that |
| +nothing they do here reaches production. If they lose that fact, the |
| +command becomes a way to make confident wrong decisions. The copy is |
| +load-bearing and is specified here as requirements. |
| Exact strings are **not drafted in this spec**. They go through the |
| -`copy-review` gate at implementation time, before they are written into |
| -any source file. What follows is what each string has to accomplish. |
| +`copy-review` gate at implementation time, before being written into any |
| +source file. What follows is what each string must accomplish. |
| + |
| +**R1 — Say it at handover, and name the source exactly.** The line |
| +reporting a successful restore states that this is a copy, and names |
| +host, database, the newest replicated transaction id and its time. Two |
| +pulls must be distinguishable, and an agent must be able to cite an |
| +"as of". Help text is read once; this line is read every time. |
| + |
| +**R2 — Age is a number, and carries its expected bound.** Report the |
| +**age of the newest replicated write**, never a "lag" (which implies a |
| +live comparison the client cannot make — §8), and never an adjective. |
| +Beside it, report the instance's replication class bound: an awake |
| +`daily` instance is legitimately up to 24h behind, and that is exactly |
| +the case where a member is most likely to mistake an old copy for a |
| +current one. Where age cannot be determined, say so rather than |
| +defaulting to a reassuring word — `dbverify`'s existing three-valued |
| +discipline (`ok`/`failed`/`unverified`, never a silent absence). |
| -**R1 — Say it at the moment of handover, not only in help.** The line |
| -that reports a successful restore must state both that this is a copy |
| -and how stale it is. Help text is read once; this line is read every |
| -time. |
| +**R3 — The interactive prompt carries it too.** A `shell` session can be |
| +open for an hour, long outliving its banner. The prompt itself carries |
| +both the word copy/replica and the as-of time. |
| + |
| +**R4 — Refuse loudly, explain briefly.** Three refusals need their own |
| +voice: `replicate: "app"` (§7), the read-only write refusal (§4), and |
| +the cross-region residency refusal (§6) — which must name residency, not |
| +read as a transient error. Per the repo's rule, store errors are logged |
| +rather than returned (`apischedule.go:145-147`): they carry bucket names |
| +and request ids. |
| + |
| +**R5 — Written for two readers.** A human skims; an agent parses. The |
| +stale-copy fact must survive both. No `carlos` verb has a `--json` flag |
| +today, so this **introduces** one, and the spec fixes its fields here so |
| +the human and machine forms cannot drift: `copy: true`, `as_of`, |
| +`replicated_txid`, `instance: awake|asleep|unknown`, `sync_class`, |
| +`verified: ok|failed|unverified`. |
| + |
| +**R6 — Name what is not there.** A database declared in `data.dbs` but |
| +not replicated (§7) is listed as such. "The platform has no copy of |
| +`audit.db`" must be said before someone goes looking for it. |
| -**R2 — Freshness is a number, never an adjective.** "Recent" and "up to |
| -date" are the failure mode. Report the replica's actual lag, and where |
| -it cannot be determined, say that it cannot be determined rather than |
| -defaulting to a reassuring word. This follows `dbverify`'s existing |
| -three-valued discipline: `ok`, `failed`, `unverified` — never a silent |
| -absence and never a `0`/`false` that reads as fine. |
| +## 6. Transport: the replica door |
| -**R3 — The interactive prompt carries it too.** A `shell` session can be |
| -open for an hour. The prompt itself, not just the banner, must be |
| -distinguishable from a live database prompt at a glance. |
| +**This section contains the design's one unresolved decision.** |
| -**R4 — Refuse loudly, explain briefly.** The `replicate: "app"` refusal |
| -(§7) and the read-only write refusal both need to say what happened and |
| -what the caller can do, in one line, without a lecture. |
| +### The problem |
| -**R5 — The copy is written for two readers.** A human skims; an agent |
| -parses. The stale-copy fact must survive both — present in the human |
| -line, and present as a machine-readable field in `--json` output rather |
| -than only in prose an agent may not be reading. |
| +The console cannot serve these bytes today, and the reason is a |
| +deliberate platform decision, not an oversight: |
| -## 6. Transport: the replica door |
| +- The console's only store is `CARLOS_DEPLOYMENT_BUCKET` |
| + (`cmd/console/main.go:395-405`). `internal/console` never reads |
| + `CARLOS_HIBERNATE_*`. |
| +- Hibernating instances' replicas live in `CARLOS_HIBERNATE_BUCKET` |
| + (`activator.go:73`) — the per-customer **data** bucket |
| + (`carlos fleets create --bucket`, edges-file `data_bucket`). |
| +- Box-wide replicas live in a third place, `s3://<bucket>/<prefix>/` |
| + keyed by absolute db path (`ops/litestream.go:170`), whose coordinates |
| + nothing publishes. |
| +- The instance-move design (2026-08-28, §149-157) makes cross-region |
| + reads of data buckets **impossible at IAM by design** — residency at |
| + rest. |
| + |
| +So the door needs a decision about buckets, credentials and residency |
| +before it can be built. |
| + |
| +### Option A — console reads the data bucket (recommended for v1) |
| + |
| +Scope v1 to activator-replicated exec instances. The console takes |
| +`CARLOS_HIBERNATE_BUCKET`/`_REGION`/`_PREFIX` (or reads `data_bucket` |
| +from edges files) and opens a second S3 client. An instance homed in |
| +another region gets a **specific residency refusal**, never a generic |
| +502. |
| + |
| +Smaller, and honest about what it can reach. Cost: the console gains |
| +read on a data bucket, which is a real widening of its blast radius, |
| +and it only works in-region. |
| + |
| +### Option B — mirror through the deployment bucket |
| + |
| +A request record; the owning box mirrors `instances/<host>/` into a |
| +staging prefix the console does serve, reusing the moves worker's |
| +CopyObject mirror (`host/movespass.go`), swept after N minutes. The |
| +console then genuinely serves what it stores, and residency stays a |
| +box-side decision. |
| + |
| +Cost: staging tenant data in the deployment bucket is *itself* a |
| +residency choice, and adds latency and a sweeper. |
| + |
| +**This is Paul's call, not mine.** It trades console blast radius |
| +against deployment-bucket residency, and both are decisions with |
| +precedent this week. |
| + |
| +### The door, once the bucket question is settled |
| + |
| +Not a general object proxy. It exposes LTX files by coordinate, not keys |
| +by name: |
| + |
| +- `list(app, instance, level)` → `[{level, min_txid, max_txid, size, created_at}]` |
| +- `get(app, instance, level, min_txid, max_txid)` with `Range` support — |
| + litestream's `ResumableReader` already resumes by offset. |
| + |
| +Addressing by coordinate rather than key is what keeps `lock.json` |
| +— which sits under the same prefix and is not a member's to fetch — |
| +unreachable. The host is resolved from |
| +`apps/<sqid>/<app>/instances/<host>.json` and **must** pass |
| +`provision.ValidHostShape` (`provision.go:250-262`, written as |
| +path-traversal defence) before becoming a key segment: `path.Join` alone |
| +cleans `../x` to `x` rather than refusing it. |
| + |
| +Hosts are unique FQDNs (`activator.go:118-121`), so hibernate prefixes |
| +cannot collide across tenants. Box-wide replica keys are absolute paths |
| +and **do** collide across boxes sharing a fleet prefix |
| +(`/data/registry.db`) — a further reason to keep them out of v1. |
| + |
| +### Who may open it |
| -A new owner-gated console endpoint serves replica objects for one app. |
| +`PrincipalUser` only. Every service principal — the legacy app token |
| +(role `instance`) and every named service credential |
| +(`publish`/`instance`/`operate`/`admin`) — gets the same |
| +`http.NotFound` as a stranger. |
| -It is **not** a general object proxy. The handler resolves |
| -`(account, app, instance, db)` to exactly one replica prefix using the |
| -console's registry snapshot, and will serve keys under that prefix and |
| -no other. The client cannot name a bucket, a prefix, or a key: it names |
| -an app and a database, and the console decides what that means. This is |
| -the same posture as `caps` — a closed vocabulary, no passthrough field, |
| -no escape hatch. |
| +A 403 would be an oracle: it tells any token holder that a db door |
| +exists for this app. The gate ethos here is one byte-identical 404. |
| -The gate reuses `instanceGate`'s shape: same principal proof, the same |
| -byte-identical 404 for "no such app" and "not yours", the same rate |
| -limiter. Owner-only. Unlike the logs door, an **app token may not read |
| -its own database** — logs are the app's own output, but the database |
| -may hold other tenants' or end users' data, and an app token is a |
| -weaker credential held in more places. |
| +No legitimate caller loses anything — an app reads its own database |
| +live, and CI has no business pulling production data. `admin` is |
| +refused **deliberately**, despite already reaching `store grant`. |
| -Every successful open writes an audit row via `s.audit`, naming the |
| -account, app, database, and caller address. Pulling a production |
| -database is a meaningful act and the record should show it. |
| +Every successful open writes an `s.audit` row naming account, app, host, |
| +database and the replicated TXID served. |
| ### The read-only replica client |
| -On the client, litestream needs a `ReplicaClient` to restore from. The |
| -S3 one (`lease.newReplicaClient`) is unusable — the member has no AWS |
| -credentials, by design. So `carlos db` implements the interface over the |
| -console door, the way `internal/fleetapi` implements `release.Store` |
| -over the edge store door for exactly the same reason. |
| +The client implements litestream's `ReplicaClient` over this door, the |
| +way `internal/fleetapi` implements `release.Store` over the edge store |
| +door for the same reason: the member has no AWS credentials, by design. |
| -The interface has eight methods. This implementation supplies the five a |
| -restore needs — `Type`, `Init`, `LTXFiles`, `OpenLTXFile`, `SetLogger` — |
| -and the three mutating ones (`WriteLTXFile`, `DeleteLTXFiles`, |
| -`DeleteAll`) return an error unconditionally. |
| +The interface has eight methods (`replica_client.go:19-50`). This |
| +implementation supplies the five a restore needs — `Type`, `Init`, |
| +`LTXFiles`, `OpenLTXFile`, `SetLogger` — and the three mutating ones |
| +(`WriteLTXFile`, `DeleteLTXFiles`, `DeleteAll`) return an error |
| +unconditionally. |
| -That is worth stating as a property rather than an implementation note: |
| -**the client cannot corrupt the replica, because it has no code that |
| -writes to one.** Not "is not supposed to" — has none. The two-replicators |
| -hazard that `internal/caps` and `internal/lease` both warn about at |
| -length cannot be reached from this path, and a future edit that tried to |
| -reach it would have to add the capability first. |
| +**The client cannot corrupt the replica, because it has no code that |
| +writes to one.** Not "is not supposed to" — has none. §12 records the |
| +verification of that claim and the three conditions it depends on. |
| -## 7. Which database |
| +## 7. Which database — one, not eight |
| -`caps.Data` allows up to eight databases per app (`maxDBs = 8`), so |
| -selection is real, not hypothetical. |
| +`caps.Data` allows up to eight names (`maxDBs = 8`), but **a route |
| +replicates exactly one database**: `Route.DBPath` is single-valued and |
| +only `DBs[0]` is stamped (`host/caps_pass.go:294`, whose own adopt |
| +warning says so verbatim). The activator's `Replicate` takes one |
| +`dbPath`. |
| -- One database, no `--db`: use it. |
| -- Several, no `--db`: refuse and list the names. Guessing here means |
| - handing someone the wrong data while telling them it is the right |
| - data, which is worse than a round trip. |
| -- `--db <name>`: must match a declared name. |
| +So there is no `--db` flag and no selection story. Instead: |
| -`--instance` selects among an app's instances. Exec-backed instances |
| -each have their own replica under `activator.HibernatePrefix`; with one |
| -instance it is inferred, with several it is required, on the same |
| -reasoning as `--db`. |
| +- The replicated database is implied by the instance. |
| +- `--instance` selects among an app's instances — inferred when there is |
| + one, required when there are several, because guessing means handing |
| + someone the wrong data while telling them it is the right data. |
| +- `ls` names the **declared-but-unreplicated** entries (`DBs[1:]`), |
| + surfacing `dataAdvice`'s warning to a member who cannot read adopt |
| + logs (R6). |
| ### `replicate: "app"` has no replica |
| -An app declaring `replicate: "app"` streams its own WAL, and the |
| -platform replicates **nothing** for it (`caps.DataReplicateApp`; |
| -`caps.go:416` refuses `data.dbs` alongside it precisely to keep the |
| -two-replicators hazard unreachable). There is no platform replica to |
| -restore. |
| - |
| -This must fail with a specific message naming that cause. The dangerous |
| -failure is a generic "no replica found", which reads as a transient |
| -problem and invites a retry loop; the app's operator needs to know the |
| -platform was never going to have this data and that their own |
| -replication is where to look. |
| - |
| -## 8. Freshness, and the two clocks |
| - |
| -Litestream sets no explicit sync interval in this deployment, so WAL |
| -shipping runs at its 1s default; `SnapshotInterval` is 24h, but those |
| -are full snapshots with WAL segments covering the gap. A restore |
| -therefore typically lands within seconds of live. |
| - |
| -"Typically" is doing work in that sentence, and the command must not |
| -launder it. Two clocks can disagree: |
| - |
| -1. **Replica lag** — how far behind the replica's newest committed |
| - transaction is. Normally seconds. |
| -2. **Replicator health** — whether anything is streaming at all. A |
| - hibernating instance holds no lease and streams nothing while |
| - asleep. Its replica is *complete as of when it went to sleep*, which |
| - may be days ago, and is perfectly restorable the whole time. This is |
| - correct behaviour that looks identical to a broken replicator if you |
| - only report a timestamp. |
| - |
| -`ls` and the restore line report both: how stale, and why. "Asleep since |
| -Tuesday" and "replicator failing since Tuesday" are the same number and |
| -opposite situations, and R2 forbids collapsing them into one adjective. |
| - |
| -`dbverify`'s `ok`/`failed`/`unverified` verdict rides along as the third |
| -signal — a replica the drill has never managed to restore should say so |
| -before you build an argument on its contents. |
| +An app declaring `replicate: "app"` streams its own WAL and the platform |
| +replicates **nothing** for it (`caps.DataReplicateApp`; `caps.go:416` |
| +refuses `data.dbs` alongside it precisely to keep the two-replicators |
| +hazard unreachable). |
| + |
| +This must fail naming that cause. The dangerous failure is a generic "no |
| +replica found", which reads as transient and invites a retry loop; the |
| +operator needs to know the platform was never going to have this data. |
| + |
| +## 8. Age, and the three things that are not lag |
| + |
| +The client **cannot compute lag**. `ops.freshness` gets the live change |
| +counter from the source file on the box (`freshness.go:96-115`); neither |
| +the console nor the CLI can reach it. Anything claiming a distance from |
| +live would be invented. |
| + |
| +What the replica alone supports, and what `ls` reports: |
| + |
| +1. **Age of the newest replicated write** — the newest LTX file's |
| + `MaxTXID` and timestamp. |
| +2. **Whether the instance is replicating now** — whether |
| + `<prefix>/lock.json` is currently held (`s3/leaser.go:23`): awake and |
| + streaming, versus parked. |
| +3. **The expected bound** — the route's replication class: |
| + `""`/`stream` ≈ 1s, `batched` 5m, `daily` 24h |
| + (`activator.go:205-230`, `lease.go:245-252`). |
| +4. **The drill verdict** — `dbverify`, per host, hibernation drill only. |
| + |
| +The first draft claimed "no explicit sync interval, so 1s default, |
| +therefore seconds behind live". That is wrong for the very instances |
| +this command targets, and wrong in the dangerous direction. An awake |
| +`daily` instance is **legitimately up to a day behind**, and reporting |
| +its age without its class makes correct platform behaviour look like |
| +current data. |
| + |
| +Retention differs between the two replicators too: the `SnapshotInterval` |
| +24h / `SnapshotRetention` 7d constants (`ops/litestream.go:64-65`) reach |
| +only `/etc/litestream.yml`. The activator's embedded store never sets |
| +`Retention`, so hibernating replicas run on litestream's |
| +`DefaultRetention` of **24h**. |
| + |
| +### A sleeping instance is consistent, not necessarily complete |
| + |
| +`stop()` logs and swallows a failed final checkpoint |
| +(`activator.go:876-881`) — the hole the instance-move spec had to add a |
| +re-checkpoint for. A parked replica is consistent at a *committed* |
| +boundary (LTX carries committed frames only) but may trail the sleep |
| +point. |
| + |
| +So: "consistent as of its last successful sync, which on a clean sleep |
| +is the sleep point" — never "complete as of when it went to sleep". |
| +`carlos restart` and Bounce go through the same `stop()`. |
| ## 9. Security posture |
| **The exposure is real and is the point of the gate.** This command |
| moves production data onto a laptop. Read-only does not reduce that; a |
| -copy leaks exactly as well as an original. Hence: owner-only, no app |
| -tokens, an audit row per open, `--out` required for anything kept, and |
| -mode-0600 with deletion on exit for anything temporary. |
| +copy leaks exactly as well as an original. Hence: members only, no |
| +service principals of any role, an audit row per open, `--out` required |
| +and refused if it exists, 0600 on anything kept, a 0700 private |
| +directory for anything temporary. |
| **Nothing new is reachable on the box.** No new door on the tenant |
| process, no new agent verb, no path that runs anything anywhere. The |
| -platform's sandbox (`DynamicUser`, `ProtectSystem=strict`, empty |
| +sandbox (`DynamicUser`, `ProtectSystem=strict`, empty |
| `CapabilityBoundingSet`) is untouched, and the same-uid sibling residual |
| -documented in `internal/activator/admintoken.go` is not widened, because |
| -no part of this design runs on the box at all. |
| +documented in `activator/admintoken.go:8-12` is not widened, because no |
| +part of this design runs on the box at all. |
| -**No new write authority anywhere.** The console serves bytes it already |
| -stores; the client has no write methods; the local copy is opened |
| -read-only. |
| +**One new read authority**, whose shape §6 leaves open: either the |
| +console gains data-bucket read (A), or tenant data transits the |
| +deployment bucket (B). Both are real, and neither should be adopted by |
| +default. |
| **The failure mode to design against is not corruption, it is |
| confidence.** Every serious way this command hurts someone routes |
| -through a caller believing stale data is live, or believing a write |
| -landed. §5 and the read-only enforcement in §4 exist for that, and are |
| -not decoration on top of the security story — they are most of it. |
| +through a caller believing old data is current, or believing a write |
| +landed. §5 and §4's read-only enforcement exist for that; they are not |
| +decoration on the security story, they are most of it. |
| ## 10. Failure matrix |
| @@ -303,64 +444,115 @@ Each row is a test. |
| | Condition | Result | |
| |---|---| |
| -| Not the account owner | Byte-identical 404, matching `instanceGate` | |
| -| App token presented | 403 — this door does not accept app tokens | |
| +| Not a member of the account | Byte-identical 404, matching `instanceGate` | |
| +| Any service principal (app token, or `publish`/`instance`/`operate`/`admin`) | Same byte-identical 404 — never a 403 (§6) | |
| | No such app | Byte-identical 404 | |
| | `replicate: "app"` | Refusal naming that cause specifically (§7) | |
| -| Several databases, no `--db` | Refusal listing the names | |
| | Several instances, no `--instance` | Refusal listing the hosts | |
| -| No snapshots yet (`ErrNoSnapshots`) | "Nothing replicated yet", distinct from a failure | |
| +| Instance homed in another region | Residency refusal naming the region — never a generic 502 | |
| +| Declared-but-unreplicated db named | Refusal naming it as never replicated (§7) | |
| +| Nothing replicated yet | "Nothing replicated yet" — match **both** `ErrNoSnapshots` and `ErrTxNotAvailable` (`lease.go:394`) | |
| +| Replica changed under the restore | One retry, then a clear failure (§12) | |
| | Restore fails integrity check | Refuse to hand over the file; surface the `dbverify` verdict | |
| -| Instance asleep | Succeeds, reports staleness and the reason (§8) | |
| -| Write attempted in `shell`/`query` | SQLite read-only error, surfaced plainly (R4) | |
| -| `pull` without `--out` | Usage error | |
| +| Instance asleep | Succeeds; reports age, class and parked state (§8) | |
| +| Instance awake on `daily` class | Succeeds; age reported beside its 24h bound (R2) | |
| +| Write attempted in `shell`/`query` | SQLite `attempt to write a readonly database`, surfaced plainly | |
| +| `--out` omitted, or already exists | Usage error | |
| +| Host failing `ValidHostShape` | Refused before it becomes a key segment | |
| | Client asked to write a replica | Unconditional error — no code path exists | |
| ## 11. Testing shape |
| Following `apischedule_test.go` and `internal/lease/check_test.go`: |
| -- Console door: the full gate matrix above, and a prefix-confinement |
| - test asserting a client naming a key outside its resolved prefix is |
| - refused. That test is the one that keeps this from becoming an object |
| +- **Door:** the full gate matrix above, plus a coordinate-confinement |
| + test asserting `lock.json` and any key outside the resolved prefix are |
| + unreachable. That test is what keeps this from becoming an object |
| proxy as the code ages. |
| -- Replica client: restore a known replica end to end through an |
| - `httptest` console; assert the three mutating methods error without |
| - touching the network. |
| -- Selection: `caps.Data` fixtures for one/many databases and the |
| +- **Replica client, both halves:** the three mutating methods error |
| + without touching the network **and** a call-counting fake asserts they |
| + stay at zero across a full restore. The second half is the one that |
| + catches a litestream upgrade changing the restore path. |
| +- **`ReplicaClientV3` negative assertion** — see §12, condition (i). |
| +- **Compile-time** `var _ litestream.ReplicaClient = (*client)(nil)`. |
| +- **Selection:** `caps.Data` fixtures for `DBs[1:]` unreplicated and the |
| `replicate: "app"` refusal. |
| -- Freshness: a table over (lag, hibernating, `dbverify` verdict) |
| - asserting the reported wording distinguishes asleep from failing. |
| -- Read-only: an `UPDATE` in `query` fails, and the file on disk is |
| - byte-identical afterwards. |
| - |
| -## 12. What this deliberately is not |
| - |
| -- **Not a live database connection.** It cannot be, and pretending |
| - otherwise in the copy is the main way this feature could do harm. |
| -- **Not a write path.** "Fix that stuck row" is a different feature with |
| - a different security design. If it is wanted, it should be an |
| - app-declared endpoint reached the way §2's rejected fork describes — |
| - the app choosing its surface — not a generic SQL door. |
| -- **Not a general object proxy.** §6's prefix confinement is the whole |
| - difference, and §11 tests it. |
| -- **Not `carlos console`.** That name already means the control plane in |
| - this codebase (`--console <origin>` is on nearly every verb), and the |
| - thing it would name — a REPL attached to a running Go app — does not |
| - exist for compiled Go. |
| +- **Freshness:** a table over (age, parked/awake, class, verdict) |
| + asserting the wording distinguishes asleep from failing, and always |
| + carries the class bound. |
| +- **Read-only:** an `UPDATE` in `query` fails with the pinned error |
| + string, and the file on disk is byte-identical afterwards. |
| +- **Permissions:** the temp directory is 0700 and `--out` lands 0600. |
| + |
| +Definition of done includes a `docs/site/cli.md` heading — the |
| +`internal/docsite` gate fails without one (AGENTS.md). |
| + |
| +## 12. Why the safety claim holds |
| + |
| +Verified against litestream v0.5.14 during review, because §6's central |
| +claim depends on it. |
| + |
| +`Replica.Restore` (`replica.go:544-720`) → `CalcRestorePlan` |
| +(`replica.go:1485ff`, `LTXFiles` per level) → `NewResumableReader`, |
| +whose client parameter is the narrow `LTXFileOpener` interface |
| +(`internal/resumable_reader.go:51,72`) — `OpenLTXFile` only. |
| +`WriteLTXFile`, `DeleteLTXFiles` and `DeleteAll` are reached only from |
| +`Sync`, `EnforceRetention`, and `db.go`'s snapshot/retention paths |
| +(`replica.go:192,312`; `db.go:2470,2526,2649`), none of which run |
| +without a `*DB`/`Store`. With a nil `*DB`, they are unreachable. |
| + |
| +Three conditions this depends on. Each is a test in §11: |
| + |
| +1. **The client must not implement `ReplicaClientV3`** (`v3.go:145`). |
| + `Restore` type-asserts for it (`replica.go:612`) and would take the |
| + v0.3 path instead. |
| +2. **Never set `RestoreOptions.Follow`.** It writes into the local |
| + output and a `-txid` sidecar, and keeps polling. |
| +3. **Run `lease.CheckDB` after the restore** rather than relying on |
| + `opt.IntegrityCheck`, matching `lease.Restore`'s existing pattern. |
| + |
| +### What the claim does *not* cover |
| + |
| +`check.go`'s reasoning is about not *shadowing the writer*. It does not |
| +promise the reader a stable listing. `CalcRestorePlan` lists every level |
| +once and opens lazily, so the writer's L0 retention |
| +(`DefaultL0Retention` 5m, checked hourly by the activator, |
| +`lease.go:78`) can delete a planned file before it is opened. |
| + |
| +The outcome is a **failed** restore, never a torn one: the output is |
| +`<out>.tmp`, renamed only on success (`replica.go:661-663`). One retry |
| +handles it, as `restore-verify` already does. |
| ## 13. Open questions |
| -1. **`shell`'s prompt.** An embedded REPL is a small amount of code but |
| - a real surface (history, output formatting, `.tables`-style dot |
| - commands). Ship `ls`/`pull`/`query` first and add `shell` once the |
| - door is proven, or ship all four together? |
| -2. **Does `pull` warn on a world-readable destination?** `--out` is the |
| - user's choice, but writing a production database to a mode-0644 path |
| - in a synced folder is a foreseeable mistake. |
| -3. **Console web view.** Out of scope here, but the door in §6 is what |
| - it would use. Worth confirming the response shape does not foreclose |
| - it. |
| -4. **Retention interaction.** `SnapshotRetention` is 7d. A restore to an |
| - arbitrary past point is not offered here — should it be, or is |
| - "current replica only" the right permanent scope? |
| +1. **§6's bucket decision — A or B.** Blocking; Paul's call. |
| +2. **`shell`'s prompt.** An embedded REPL is small code but a real |
| + surface (history, formatting, dot commands). Ship `ls`/`pull`/`query` |
| + first, or all four together? |
| +3. **Console web view.** Out of scope, but §6's door is what it would |
| + use; confirm the coordinate-addressed shape does not foreclose it. |
| +4. **Past points in time.** Not offered here. Given the activator's |
| + effective 24h `DefaultRetention` (§8), is "current replica only" the |
| + right permanent scope? |
| +5. **Rate limiting.** Pulls are large; neither the token-keyed instance |
| + burst nor the address-keyed write budget fits. Needs its own budget. |
| + |
| +## 14. What the review changed |
| + |
| +The first draft's safety claim survived verification (§12). Its |
| +*transport* did not, and several facts were wrong: |
| + |
| +- "The console serves objects it already stores" — false; the replicas |
| + are in data buckets the console has no client for, and cross-region |
| + reads are barred by IAM. Now §6's open decision. |
| +- "Selection among up to eight databases" — false; a route replicates |
| + exactly one. Now §7, with the unreplicated remainder surfaced instead. |
| +- "Seconds behind live" — wrong for the targeted instances, in the |
| + dangerous direction: `daily` class is legitimately 24h. Now §8. |
| +- "litestream is vendored" / "`FleetDBs` is the platform-replicated set" |
| + / "the console's registry snapshot" / "`dbverify` is per database" — |
| + all corrected in §3. |
| +- A 403 for service principals was an oracle; now a byte-identical 404 |
| + (§6). |
| +- The mode-0600 temp file was not constructible given how `Restore` |
| + creates its own output; now a 0700 private directory (§4). |