amadan
spec: carlos db — read an app's database from its replica
A design for the data-inspection gap: there is no carlos answer to "why does this row look wrong", so the answer is currently an SSM session and a sqlite3 prompt on the live file — which the no-infra-access rule forbids. The command hands over a restored copy of the litestream replica rather than connecting to the live database, and the query runs client-side so the console never executes tenant SQL. Both narrowings are deliberate: they make the command safe for an agent debugging production, which is the caller this is really for. Mostly assembly — litestream is vendored as a library, modernc.org/sqlite is pure Go, and lease.Restore plus the restore-verify drill already prove the replicas restore.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PEKBoE5EyWTUJPU15ZoenD
1 file changed,
+366
−0
docs/superpowers/specs/2026-09-02-carlos-db-design.md+366 −0
diff --git a/docs/superpowers/specs/2026-09-02-carlos-db-design.md b/docs/superpowers/specs/2026-09-02-carlos-db-design.md| new file mode 100644 |
| index 000000000..8d127b4bc |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-09-02-carlos-db-design.md |
| @@ -0,0 +1,366 @@ |
| +# `carlos db` — reading an app's database from its replica |
| + |
| +Design, 2026-09-02. |
| + |
| +## 1. What this is for |
| + |
| +There is no way to look at an app's data through the platform. The |
| +question "why does this row look wrong" currently has no `carlos` answer, |
| +which means it has an AWS answer — an SSM session onto the box and a |
| +`sqlite3` prompt against the live file. That violates the standing rule |
| +that every `carlos` command is designed for a user with no access to the |
| +underlying infra (AGENTS.md, Paul 2026-08-08), and it puts a hand-typed |
| +shell next to a production database for what is almost always a read. |
| + |
| +`carlos db` closes that gap by answering a deliberately narrower |
| +question. It does **not** connect you to the running database. It hands |
| +you a **restored copy of the replica** and lets you query that. |
| + |
| +The narrowing is the feature, not a compromise. It makes the command |
| +safe for a class of caller we specifically want to serve: an **agent** |
| +debugging a production problem. An agent that can read the data and |
| +cannot possibly change it is a useful colleague; an agent with a live |
| +`sqlite3` prompt on production is an incident waiting for a bad |
| +inference. Every design decision below is downstream of that. |
| + |
| +### Scope |
| + |
| +In: reading an app's platform-replicated SQLite, from the CLI, by an |
| +account owner, as a local file. |
| + |
| +Out: writing to tenant data by any path; querying the live database; |
| +non-SQLite stores; the console web UI (this is CLI-first — a web view |
| +can follow once the door exists, and is not designed here). |
| + |
| +## 2. The decision |
| + |
| +Two forks were considered and settled before this spec. |
| + |
| +**Replica, not live.** The alternative was a query door onto the running |
| +instance. That needs a synchronous console→instance path, which CARLOS |
| +does not have for ordinary boxes: the control plane is the bucket and |
| +boxes poll (`schedule.TickEvery` is 15s; `logship` drains every 10s). |
| +Building one would mean a new transport *and* a new door onto the tenant |
| +process, to deliver a capability whose main use is a read. The replica |
| +is already there, already off-box, and already proven restorable by the |
| +`carlos ops restore-verify` drill. |
| + |
| +**Client-side query, not console-side.** The console serves replica |
| +**bytes**; the member's own machine restores and queries. The console |
| +therefore never executes tenant SQL, never holds a restored tenant |
| +database, and needs no query timeouts, row caps, or result-set memory |
| +budget. It serves objects it already stores — which is a thing it |
| +already knows how to do safely. |
| + |
| +The client-side choice also buys the microcopy problem's best available |
| +answer, discussed in §5: the user is handed an actual file, so "this is |
| +a copy" is a true statement about an artifact in their hand rather than |
| +a warning against an illusion. |
| + |
| +Cost accepted: you transfer the whole database rather than one query's |
| +rows, and freshness is replica freshness rather than live. |
| + |
| +## 3. What already exists |
| + |
| +This design is mostly assembly. The parts: |
| + |
| +- `lease.Restore(ctx, dbPath, bucket, prefix, region)` restores a |
| + replica to a file. `lease.VerifyReplica` does it without taking the |
| + lease, and `internal/lease/check.go` documents at length why that is |
| + safe against a live replicator: litestream restores a committed |
| + snapshot, and the reader never writes. |
| +- litestream v0.5.14 is a **vendored Go library**, not a binary we |
| + shell out to. `litestream.NewReplicaWithClient(nil, client)` is its |
| + own URL-restore pattern, already used at `internal/lease/lease.go:390`. |
| +- `modernc.org/sqlite` v1.55.0 is already a direct dependency and is |
| + **pure Go**. The query engine embeds; `carlos db` needs no `sqlite3` |
| + on the user's machine and no cgo. |
| +- `activator.HibernatePrefix(prefix, host)` is the replica prefix layout |
| + for exec-backed instances (`<CARLOS_HIBERNATE_PREFIX>/<host>/`, prefix |
| + `instances` in production). |
| +- `ops.FleetDBs` is the box-wide replicator's set — the "glob proposes, |
| + registry disposes" rule for platform-replicated databases. |
| +- `internal/dbverify` already records, per database, whether its replica |
| + is `ok`, `failed`, or `unverified`, at `control/restore-verify.json`. |
| +- `internal/console` has the gate shape to copy: `instanceGate` |
| + (`apiinstances.go:70`) and `apiScheduleApp` (`apischedule.go:96`) — |
| + same principal proof, byte-identical 404, shared rate limiter — and |
| + `s.audit` (`server.go:2692`). |
| + |
| +Nothing here needs a new replication path, a new bucket, or a change to |
| +how any app runs. |
| + |
| +## 4. Command surface |
| + |
| +``` |
| +carlos db ls --app <a> [--account <a>] |
| +carlos db pull --app <a> [--instance <host>] [--db <name>] --out <path> |
| +carlos db shell --app <a> [--instance <host>] [--db <name>] |
| +carlos db query --app <a> [--instance <host>] [--db <name>] --sql <text> |
| +``` |
| + |
| +`ls` is the discovery verb and the one that makes the rest legible: for |
| +each database the platform replicates for this app, it prints the |
| +instance, the database name, how far behind the replica is, and its |
| +last restore-drill verdict from `dbverify`. A member who has never used |
| +this command should be able to type `ls` first and understand what the |
| +other three will hand them. |
| + |
| +`pull` writes the restored copy to a path you name and stops. `--out` is |
| +**required** — a command that leaves a production database somewhere you |
| +did not name is not a command we should ship. |
| + |
| +`shell` restores to a mode-0600 temp file, opens an interactive prompt |
| +against it with the embedded engine, and deletes the file on exit. This |
| +is the interactive verb, and it is the closest thing to `rails console` |
| +that a Go platform can honestly offer: a REPL over your *data*, which |
| +exists, rather than over your *app*, which cannot (Go is compiled ahead |
| +of time; there is no interpreter in the running binary and no way to |
| +evaluate an expression against its live heap). |
| + |
| +`query` is the non-interactive form — one statement, results to stdout, |
| +exit code reflects the query. This is the shape an agent uses, and it |
| +should be the shape that is easy to pipe. |
| + |
| +### Read-only is enforced, not assumed |
| + |
| +Every path opens the copy with SQLite in read-only mode |
| +(`query_only`, and the file opened `mode=ro`). This matters most for the |
| +agent case and it is worth being explicit about why: a copy is already |
| +harmless to write to, so enforcement is not about protecting production. |
| +It is about **not letting a caller believe it changed production**. An |
| +agent that runs an `UPDATE` against a writable copy sees `1 row |
| +affected` and concludes it has fixed the bug. The same agent against a |
| +read-only copy gets an error, and the error is the truth. Silent success |
| +on a discarded write is the single worst outcome this command could |
| +produce, and read-only mode is what forecloses it. |
| + |
| +## 5. The microcopy contract |
| + |
| +The safety property of this command is a *fact the user has to hold in |
| +their head* — that they are looking at a copy, seconds to minutes stale, |
| +and that nothing they do here reaches production. If they lose that |
| +fact, the command becomes a way to make confident wrong decisions. |
| +So the copy is load-bearing and is specified here as requirements. |
| + |
| +Exact strings are **not drafted in this spec**. They go through the |
| +`copy-review` gate at implementation time, before they are written into |
| +any source file. What follows is what each string has to accomplish. |
| + |
| +**R1 — Say it at the moment of handover, not only in help.** The line |
| +that reports a successful restore must state both that this is a copy |
| +and how stale it is. Help text is read once; this line is read every |
| +time. |
| + |
| +**R2 — Freshness is a number, never an adjective.** "Recent" and "up to |
| +date" are the failure mode. Report the replica's actual lag, and where |
| +it cannot be determined, say that it cannot be determined rather than |
| +defaulting to a reassuring word. This follows `dbverify`'s existing |
| +three-valued discipline: `ok`, `failed`, `unverified` — never a silent |
| +absence and never a `0`/`false` that reads as fine. |
| + |
| +**R3 — The interactive prompt carries it too.** A `shell` session can be |
| +open for an hour. The prompt itself, not just the banner, must be |
| +distinguishable from a live database prompt at a glance. |
| + |
| +**R4 — Refuse loudly, explain briefly.** The `replicate: "app"` refusal |
| +(§7) and the read-only write refusal both need to say what happened and |
| +what the caller can do, in one line, without a lecture. |
| + |
| +**R5 — The copy is written for two readers.** A human skims; an agent |
| +parses. The stale-copy fact must survive both — present in the human |
| +line, and present as a machine-readable field in `--json` output rather |
| +than only in prose an agent may not be reading. |
| + |
| +## 6. Transport: the replica door |
| + |
| +A new owner-gated console endpoint serves replica objects for one app. |
| + |
| +It is **not** a general object proxy. The handler resolves |
| +`(account, app, instance, db)` to exactly one replica prefix using the |
| +console's registry snapshot, and will serve keys under that prefix and |
| +no other. The client cannot name a bucket, a prefix, or a key: it names |
| +an app and a database, and the console decides what that means. This is |
| +the same posture as `caps` — a closed vocabulary, no passthrough field, |
| +no escape hatch. |
| + |
| +The gate reuses `instanceGate`'s shape: same principal proof, the same |
| +byte-identical 404 for "no such app" and "not yours", the same rate |
| +limiter. Owner-only. Unlike the logs door, an **app token may not read |
| +its own database** — logs are the app's own output, but the database |
| +may hold other tenants' or end users' data, and an app token is a |
| +weaker credential held in more places. |
| + |
| +Every successful open writes an audit row via `s.audit`, naming the |
| +account, app, database, and caller address. Pulling a production |
| +database is a meaningful act and the record should show it. |
| + |
| +### The read-only replica client |
| + |
| +On the client, litestream needs a `ReplicaClient` to restore from. The |
| +S3 one (`lease.newReplicaClient`) is unusable — the member has no AWS |
| +credentials, by design. So `carlos db` implements the interface over the |
| +console door, the way `internal/fleetapi` implements `release.Store` |
| +over the edge store door for exactly the same reason. |
| + |
| +The interface has eight methods. This implementation supplies the five a |
| +restore needs — `Type`, `Init`, `LTXFiles`, `OpenLTXFile`, `SetLogger` — |
| +and the three mutating ones (`WriteLTXFile`, `DeleteLTXFiles`, |
| +`DeleteAll`) return an error unconditionally. |
| + |
| +That is worth stating as a property rather than an implementation note: |
| +**the client cannot corrupt the replica, because it has no code that |
| +writes to one.** Not "is not supposed to" — has none. The two-replicators |
| +hazard that `internal/caps` and `internal/lease` both warn about at |
| +length cannot be reached from this path, and a future edit that tried to |
| +reach it would have to add the capability first. |
| + |
| +## 7. Which database |
| + |
| +`caps.Data` allows up to eight databases per app (`maxDBs = 8`), so |
| +selection is real, not hypothetical. |
| + |
| +- One database, no `--db`: use it. |
| +- Several, no `--db`: refuse and list the names. Guessing here means |
| + handing someone the wrong data while telling them it is the right |
| + data, which is worse than a round trip. |
| +- `--db <name>`: must match a declared name. |
| + |
| +`--instance` selects among an app's instances. Exec-backed instances |
| +each have their own replica under `activator.HibernatePrefix`; with one |
| +instance it is inferred, with several it is required, on the same |
| +reasoning as `--db`. |
| + |
| +### `replicate: "app"` has no replica |
| + |
| +An app declaring `replicate: "app"` streams its own WAL, and the |
| +platform replicates **nothing** for it (`caps.DataReplicateApp`; |
| +`caps.go:416` refuses `data.dbs` alongside it precisely to keep the |
| +two-replicators hazard unreachable). There is no platform replica to |
| +restore. |
| + |
| +This must fail with a specific message naming that cause. The dangerous |
| +failure is a generic "no replica found", which reads as a transient |
| +problem and invites a retry loop; the app's operator needs to know the |
| +platform was never going to have this data and that their own |
| +replication is where to look. |
| + |
| +## 8. Freshness, and the two clocks |
| + |
| +Litestream sets no explicit sync interval in this deployment, so WAL |
| +shipping runs at its 1s default; `SnapshotInterval` is 24h, but those |
| +are full snapshots with WAL segments covering the gap. A restore |
| +therefore typically lands within seconds of live. |
| + |
| +"Typically" is doing work in that sentence, and the command must not |
| +launder it. Two clocks can disagree: |
| + |
| +1. **Replica lag** — how far behind the replica's newest committed |
| + transaction is. Normally seconds. |
| +2. **Replicator health** — whether anything is streaming at all. A |
| + hibernating instance holds no lease and streams nothing while |
| + asleep. Its replica is *complete as of when it went to sleep*, which |
| + may be days ago, and is perfectly restorable the whole time. This is |
| + correct behaviour that looks identical to a broken replicator if you |
| + only report a timestamp. |
| + |
| +`ls` and the restore line report both: how stale, and why. "Asleep since |
| +Tuesday" and "replicator failing since Tuesday" are the same number and |
| +opposite situations, and R2 forbids collapsing them into one adjective. |
| + |
| +`dbverify`'s `ok`/`failed`/`unverified` verdict rides along as the third |
| +signal — a replica the drill has never managed to restore should say so |
| +before you build an argument on its contents. |
| + |
| +## 9. Security posture |
| + |
| +**The exposure is real and is the point of the gate.** This command |
| +moves production data onto a laptop. Read-only does not reduce that; a |
| +copy leaks exactly as well as an original. Hence: owner-only, no app |
| +tokens, an audit row per open, `--out` required for anything kept, and |
| +mode-0600 with deletion on exit for anything temporary. |
| + |
| +**Nothing new is reachable on the box.** No new door on the tenant |
| +process, no new agent verb, no path that runs anything anywhere. The |
| +platform's sandbox (`DynamicUser`, `ProtectSystem=strict`, empty |
| +`CapabilityBoundingSet`) is untouched, and the same-uid sibling residual |
| +documented in `internal/activator/admintoken.go` is not widened, because |
| +no part of this design runs on the box at all. |
| + |
| +**No new write authority anywhere.** The console serves bytes it already |
| +stores; the client has no write methods; the local copy is opened |
| +read-only. |
| + |
| +**The failure mode to design against is not corruption, it is |
| +confidence.** Every serious way this command hurts someone routes |
| +through a caller believing stale data is live, or believing a write |
| +landed. §5 and the read-only enforcement in §4 exist for that, and are |
| +not decoration on top of the security story — they are most of it. |
| + |
| +## 10. Failure matrix |
| + |
| +Each row is a test. |
| + |
| +| Condition | Result | |
| +|---|---| |
| +| Not the account owner | Byte-identical 404, matching `instanceGate` | |
| +| App token presented | 403 — this door does not accept app tokens | |
| +| No such app | Byte-identical 404 | |
| +| `replicate: "app"` | Refusal naming that cause specifically (§7) | |
| +| Several databases, no `--db` | Refusal listing the names | |
| +| Several instances, no `--instance` | Refusal listing the hosts | |
| +| No snapshots yet (`ErrNoSnapshots`) | "Nothing replicated yet", distinct from a failure | |
| +| Restore fails integrity check | Refuse to hand over the file; surface the `dbverify` verdict | |
| +| Instance asleep | Succeeds, reports staleness and the reason (§8) | |
| +| Write attempted in `shell`/`query` | SQLite read-only error, surfaced plainly (R4) | |
| +| `pull` without `--out` | Usage error | |
| +| Client asked to write a replica | Unconditional error — no code path exists | |
| + |
| +## 11. Testing shape |
| + |
| +Following `apischedule_test.go` and `internal/lease/check_test.go`: |
| + |
| +- Console door: the full gate matrix above, and a prefix-confinement |
| + test asserting a client naming a key outside its resolved prefix is |
| + refused. That test is the one that keeps this from becoming an object |
| + proxy as the code ages. |
| +- Replica client: restore a known replica end to end through an |
| + `httptest` console; assert the three mutating methods error without |
| + touching the network. |
| +- Selection: `caps.Data` fixtures for one/many databases and the |
| + `replicate: "app"` refusal. |
| +- Freshness: a table over (lag, hibernating, `dbverify` verdict) |
| + asserting the reported wording distinguishes asleep from failing. |
| +- Read-only: an `UPDATE` in `query` fails, and the file on disk is |
| + byte-identical afterwards. |
| + |
| +## 12. What this deliberately is not |
| + |
| +- **Not a live database connection.** It cannot be, and pretending |
| + otherwise in the copy is the main way this feature could do harm. |
| +- **Not a write path.** "Fix that stuck row" is a different feature with |
| + a different security design. If it is wanted, it should be an |
| + app-declared endpoint reached the way §2's rejected fork describes — |
| + the app choosing its surface — not a generic SQL door. |
| +- **Not a general object proxy.** §6's prefix confinement is the whole |
| + difference, and §11 tests it. |
| +- **Not `carlos console`.** That name already means the control plane in |
| + this codebase (`--console <origin>` is on nearly every verb), and the |
| + thing it would name — a REPL attached to a running Go app — does not |
| + exist for compiled Go. |
| + |
| +## 13. Open questions |
| + |
| +1. **`shell`'s prompt.** An embedded REPL is a small amount of code but |
| + a real surface (history, output formatting, `.tables`-style dot |
| + commands). Ship `ls`/`pull`/`query` first and add `shell` once the |
| + door is proven, or ship all four together? |
| +2. **Does `pull` warn on a world-readable destination?** `--out` is the |
| + user's choice, but writing a production database to a mode-0644 path |
| + in a synced folder is a foreseeable mistake. |
| +3. **Console web view.** Out of scope here, but the door in §6 is what |
| + it would use. Worth confirming the response shape does not foreclose |
| + it. |
| +4. **Retention interaction.** `SnapshotRetention` is 7d. A restore to an |
| + arbitrary past point is not offered here — should it be, or is |
| + "current replica only" the right permanent scope? |