amadan
spec: carlos db — settle on the console reading the region data bucket
Option A over B, on the merits rather than as a coin-flip: B needs a new box-side pass and a sweeper, which would have cost section 9 its "nothing runs on the box" sentence; its first-mirror latency is a CopyObject per LTX file; it lands tenant database bytes in the deployment bucket, which the moves design deliberately routed around; and it does not remove the data-bucket read anyway, since hibernate-verify still needs it. Its one advantage over A, cross-region reach, is a residency violation rather than a feature. Two further findings, both in the direction of overstating freshness: sync_class is unknowable off-box — Replication lives on registry.Route, not provision.Record — so the console knows it only for routes on its own box. R2 now requires an explicit "unknown" that must never render as `stream`, because a wrong bound turns "I don't know" into "one second old". Age must come from the LTX header timestamp rather than S3 LastModified, or a compacted L3 file reports its compaction as a fresh write. Also: the door adapts litestream's own lss3 read methods so reader and writer cannot drift on key layout, levels 0-9 answer empty rather than 404, lease_held becomes a derived ls field so "is it replicating" is answerable without exposing lock.json, and the restore retry wraps the whole Restore because ResumableReader treats ErrNotExist as fatal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PEKBoE5EyWTUJPU15ZoenD
1 file changed,
+341
−266
docs/superpowers/specs/2026-09-02-carlos-db-design.md+341 −266
diff --git a/docs/superpowers/specs/2026-09-02-carlos-db-design.md b/docs/superpowers/specs/2026-09-02-carlos-db-design.md| index 0a7d2f590..08856f2cf 100644 |
| --- a/docs/superpowers/specs/2026-09-02-carlos-db-design.md |
| +++ b/docs/superpowers/specs/2026-09-02-carlos-db-design.md |
| @@ -1,6 +1,6 @@ |
| # `carlos db` — reading an app's database from its replica |
| -Design, 2026-09-02. Revised after review, same day (§14 records what changed). |
| +Design, 2026-09-02. Revised twice after review, same day (§14). |
| ## 1. What this is for |
| @@ -26,97 +26,91 @@ inference. Every design decision below is downstream of that. |
| ### Scope |
| In: reading an app's platform-replicated SQLite, from the CLI, by an |
| -account member, as a local file. |
| +account member, as a local file, for **in-region, non-pooled, |
| +activator-replicated instances**. |
| Out: writing to tenant data by any path; querying the live database; |
| -non-SQLite stores; the console web UI (this is CLI-first — a web view |
| -can follow once the door exists, and is not designed here). |
| +fleet-homed and cross-region instances (§13 — a v2 with a different |
| +transport); box-wide replicas; non-SQLite stores; the console web UI. |
| -## 2. The decision |
| - |
| -Two forks were considered and settled before this spec. |
| +## 2. The two decisions behind the shape |
| **Replica, not live.** The alternative was a query door onto the running |
| instance. That needs a synchronous console→instance path, which CARLOS |
| does not have for ordinary boxes: the control plane is the bucket and |
| -boxes poll (`schedule.TickEvery` is 15s, `logship` drains every 10s, and |
| -the agent polls `control/serial` every 2s). Building one would mean a new |
| -transport *and* a new door onto the tenant process, to deliver a |
| -capability whose main use is a read. The replica is already there, |
| -already off-box, and already proven restorable by the restore drills. |
| +boxes poll (`schedule.TickEvery` 15s, `logship` 10s, `control/serial` |
| +2s). Building one would mean a new transport *and* a new door onto the |
| +tenant process, to deliver a capability whose main use is a read. The |
| +replica is already there, already off-box, and already proven restorable |
| +by the restore drills. |
| **Client-side query, not console-side.** The console serves replica |
| -**bytes**; the member's own machine restores and queries. The console |
| -therefore never executes tenant SQL, never holds a restored tenant |
| -database, and needs no query timeouts, row caps, or result-set memory |
| -budget. |
| - |
| -> **Note.** The first draft justified this as "the console serves objects |
| -> it already stores". That is false — see §6. The console holds the |
| -> deployment bucket; replicas live in customer data buckets. The |
| -> client-side argument survives on its own merits, but the transport it |
| -> rests on is a genuinely new door, and §6 is now the hardest part of |
| +**bytes**; the member's machine restores and queries. The console never |
| +executes tenant SQL, never holds a restored tenant database, and needs |
| +no query timeouts, row caps, or result-set memory budget. |
| + |
| +> The first draft justified this as "the console serves objects it |
| +> already stores". That is false — replicas live in customer data |
| +> buckets (§6). The client-side argument survives on its own merits, but |
| +> the transport is a genuinely new door, and §6 is the hardest part of |
| > this design rather than the easiest. |
| The client-side choice also buys the microcopy problem's best available |
| -answer, discussed in §5: the user is handed an actual file, so "this is |
| -a copy" is a true statement about an artifact in their hand rather than |
| -a warning against an illusion. |
| +answer (§5): the user is handed an actual file, so "this is a copy" is a |
| +true statement about an artifact in their hand rather than a warning |
| +against an illusion. |
| Cost accepted: you transfer the whole database rather than one query's |
| rows, and freshness is replica freshness rather than live. |
| ## 3. What already exists |
| -Accurate as of 2026-09-02. There are **two** replicators in this |
| -platform and they are easy to conflate: |
| +Accurate as of 2026-09-02. There are **two** replicators and they are |
| +easy to conflate: |
| - **The activator's embedded library.** litestream v0.5.14 is a `go.mod` |
| dependency (not vendored — there is no `vendor/`), linked into the |
| `carlos` binary. It replicates **hibernating exec-backed instances** |
| under the lease. `lease.Restore` (`lease/lease.go:378`) restores from |
| - it, using litestream's own nil-`*DB` URL-restore pattern |
| - (`lease.go:390`). This is the set `carlos db` primarily serves. |
| + it via litestream's nil-`*DB` URL-restore pattern (`lease.go:390`). |
| + This is the set `carlos db` serves. |
| - **The box-wide binary.** A separately pinned litestream **0.5.15** |
| - executable (`ops/litestream.go:43`), driven by a generated |
| - `/etc/litestream.yml`, replicating everything else. |
| + executable (`ops/litestream.go:43`) driven by `/etc/litestream.yml`. |
| `carlos ops restore-verify` shells out to it. |
| -`ops.FleetDBs` is the box-wide unit's set, and it **excludes** |
| -hibernating exec routes (`ops/hibernate_verify.go:5-8`) — it is the |
| -complement of the activator's set, not the whole. |
| +`ops.FleetDBs` is the box-wide unit's set and **excludes** hibernating |
| +exec routes (`ops/hibernate_verify.go:5-8`) — the complement of the |
| +activator's set, not the whole. Box-wide replica keys are absolute paths |
| +(`ops/litestream.go:170`) and **collide across boxes sharing a fleet |
| +prefix** (`/data/registry.db`), which is one reason they stay out of v1. |
| Other parts this design uses: |
| - `lease.VerifyReplica` restores without taking the lease; |
| - `internal/lease/check.go:66-79` documents why that cannot shadow the |
| - true holder. §12 records how far that reasoning actually reaches. |
| + `check.go:66-79` documents why that cannot shadow the true holder. |
| + §12 records how far that reasoning actually reaches. |
| - `activator.HibernatePrefix(prefix, host)` = `path.Join(prefix, host)` |
| - (`activator.go:113`), prefix `instances` in production. |
| + (`activator.go:113`), prefix `instances` in production. Hosts are |
| + unique FQDNs (`activator.go:118-121`), so prefixes cannot collide |
| + across tenants. |
| - `modernc.org/sqlite` v1.55.0 is a direct dependency and **pure Go** |
| - (`go.mod:23`). litestream itself imports it, so the CLI already links |
| - it; the no-cgo claim holds. The query engine embeds — `carlos db` |
| - needs no `sqlite3` on the user's machine. |
| -- `internal/dbverify` records restore-drill verdicts — but keyed by |
| - **host**, not by database (`dbverify.go:45`), at one deployment-wide |
| - key (`dbverify.go:28`), and only the hibernation drill publishes. |
| - `Merge` drops hosts outside the writing box's fleet |
| - (`dbverify.go:60-80`), so the last box to run overwrites the others. |
| - Treat a missing verdict as `unverified`, which is the record's own |
| - discipline. |
| -- `internal/console` gates: `instanceGate` (`apiinstances.go:70`), |
| - `apiScheduleApp` (`apischedule.go:96`), `s.audit` (`server.go:2692`). |
| - The gate is **membership** via `AccountsFor(addr)` (`apicli.go:208`) — |
| - people carry no owner/member role, so this spec says "member". |
| + (`go.mod:23`); litestream imports it, so the CLI already links it. The |
| + query engine embeds — no `sqlite3` on the user's machine, no cgo. |
| +- `internal/dbverify` records restore-drill verdicts keyed by **host** |
| + (`dbverify.go:45`), at one deployment-wide key (`dbverify.go:28`), |
| + published only by the hibernation drill. `Merge` drops hosts outside |
| + the writing box's fleet (`dbverify.go:60-80`), so the last box to run |
| + overwrites the others. A missing verdict renders `unverified`. |
| +- Console gates: `instanceGate` (`apiinstances.go:70`), `s.audit` |
| + (`server.go:2692`). The gate is **membership** via `AccountsFor(addr)` |
| + (`apicli.go:208`) — people carry no owner/member role. |
| **The console has no registry snapshot.** It opens the *local* |
| `registry.db` on its own box (`console/view.go:532-538`). The |
| `control/registry-snapshot/<box>.db` objects boxes write |
| (`host/snapshot_loop.go:34`) are read by nothing in `internal/console`. |
| -Fleet-wide instance knowledge lives in |
| -`apps/<sqid>/<app>/instances/<host>.json`. Any claim that the console |
| -"resolves from its registry snapshot" is wrong; §6 resolves from the |
| -instances records. |
| +Fleet-wide instance knowledge is `apps/<sqid>/<app>/instances/<host>.json`. |
| +§6 and §8 both depend on this distinction. |
| ## 4. Command surface |
| @@ -127,50 +121,49 @@ carlos db shell --app <a> [--instance <host>] |
| carlos db query --app <a> [--instance <host>] --sql <text> |
| ``` |
| -`ls` is the discovery verb and the one that makes the rest legible: for |
| -each instance, the database the platform replicates, how old the newest |
| -replicated write is, the instance's replication class, and its |
| -restore-drill verdict. It **also names databases the app declared in |
| -`data.dbs` that the platform does not replicate** (§7) — that fact is |
| -otherwise only visible in adopt logs a member cannot read. |
| +`ls` is the discovery verb and the one that makes the rest legible: per |
| +instance, the database the platform replicates, the age of the newest |
| +replicated write, whether the instance is replicating right now, its |
| +replication class *or* an honest "unknown" (§8), and its restore-drill |
| +verdict. It **also names databases declared in `data.dbs` that the |
| +platform does not replicate** (§7) — otherwise visible only in adopt |
| +logs a member cannot read. |
| `pull` writes the restored copy to a path you name and stops. `--out` is |
| -**required**, and must not already exist — a command that leaves a |
| -production database somewhere you did not name is not one we should |
| -ship. |
| +**required** and must not already exist. |
| `shell` restores into a private directory, opens an interactive prompt |
| against it with the embedded engine, and removes the directory on exit. |
| -This is the closest thing to `rails console` that a Go platform can |
| -honestly offer: a REPL over your *data*, which exists, rather than over |
| -your *app*, which cannot (Go is compiled ahead of time; there is no |
| -interpreter in the running binary and no way to evaluate an expression |
| -against its live heap). |
| +This is the closest thing to `rails console` a Go platform can honestly |
| +offer: a REPL over your *data*, which exists, rather than over your |
| +*app*, which cannot — Go is compiled ahead of time, and there is no |
| +interpreter in the running binary to evaluate against its live heap. |
| `query` is the non-interactive form — one statement, results to stdout, |
| exit code reflecting the query. This is the shape an agent uses and |
| -should be the shape that pipes cleanly. |
| +should pipe cleanly. |
| ### Where the copy lands |
| litestream's `Restore` refuses an existing output path |
| -(`replica.go:604`) and creates `<out>.tmp` itself with `os.Create`, |
| -so it lands at the process umask (typically 0644) and you **cannot** |
| -pre-create a mode-0600 file for it to fill. |
| +(`replica.go:604`) and creates `<out>.tmp` itself with `os.Create`, at |
| +the process umask (typically 0644). You **cannot** pre-create a 0600 |
| +file for it to fill. |
| So: always restore inside an `os.MkdirTemp` directory created 0700. That |
| -confines the `.tmp`, and also the `-wal`/`-shm` files a read-only open |
| -still needs (`check.go:96-98`). `shell` and `query` never leave that |
| -directory. `pull` restores there and then moves the result to `--out`, |
| -chmod 0600. Q2 in §13 is thereby answered: always 0600, and the user can |
| -loosen it themselves. |
| +confines the `.tmp` and the `-wal`/`-shm` files a read-only open still |
| +needs (`check.go:96-98`). `shell` and `query` never leave that |
| +directory. `pull` restores there, then **copies** to `--out` with fsync |
| +and 0600 — a copy, not `os.Rename`, because the temp directory and the |
| +user's destination are routinely on different filesystems. Open |
| +question 2 is thereby answered: always 0600, and the user can loosen it. |
| ### Read-only is enforced, not assumed |
| Open with `file:<path>?mode=ro` — the form `lease.CheckDB` already uses |
| (`check.go:36`) — plus `PRAGMA query_only=1`. |
| -This matters most for the agent case, for a non-obvious reason: a copy |
| +This matters most for the agent case, for a non-obvious reason. A copy |
| is already harmless to write to, so enforcement is not about protecting |
| production. It is about **not letting a caller believe it changed |
| production**. An agent that runs an `UPDATE` against a writable copy |
| @@ -183,13 +176,12 @@ single worst outcome this command could produce. |
| ## 5. The microcopy contract |
| -The safety property of this command is a *fact the user has to hold in |
| -their head* — that they are looking at a copy, of a stated age, and that |
| -nothing they do here reaches production. If they lose that fact, the |
| -command becomes a way to make confident wrong decisions. The copy is |
| -load-bearing and is specified here as requirements. |
| +The safety property is a *fact the user has to hold in their head* — |
| +that this is a copy, of a stated age, and that nothing they do here |
| +reaches production. If they lose that fact, the command becomes a way to |
| +make confident wrong decisions. The copy is load-bearing. |
| -Exact strings are **not drafted in this spec**. They go through the |
| +Exact strings are **not drafted here**. They go through the |
| `copy-review` gate at implementation time, before being written into any |
| source file. What follows is what each string must accomplish. |
| @@ -197,35 +189,43 @@ source file. What follows is what each string must accomplish. |
| reporting a successful restore states that this is a copy, and names |
| host, database, the newest replicated transaction id and its time. Two |
| pulls must be distinguishable, and an agent must be able to cite an |
| -"as of". Help text is read once; this line is read every time. |
| - |
| -**R2 — Age is a number, and carries its expected bound.** Report the |
| -**age of the newest replicated write**, never a "lag" (which implies a |
| -live comparison the client cannot make — §8), and never an adjective. |
| -Beside it, report the instance's replication class bound: an awake |
| -`daily` instance is legitimately up to 24h behind, and that is exactly |
| -the case where a member is most likely to mistake an old copy for a |
| -current one. Where age cannot be determined, say so rather than |
| -defaulting to a reassuring word — `dbverify`'s existing three-valued |
| -discipline (`ok`/`failed`/`unverified`, never a silent absence). |
| - |
| -**R3 — The interactive prompt carries it too.** A `shell` session can be |
| -open for an hour, long outliving its banner. The prompt itself carries |
| -both the word copy/replica and the as-of time. |
| - |
| -**R4 — Refuse loudly, explain briefly.** Three refusals need their own |
| -voice: `replicate: "app"` (§7), the read-only write refusal (§4), and |
| -the cross-region residency refusal (§6) — which must name residency, not |
| -read as a transient error. Per the repo's rule, store errors are logged |
| -rather than returned (`apischedule.go:145-147`): they carry bucket names |
| -and request ids. |
| - |
| -**R5 — Written for two readers.** A human skims; an agent parses. The |
| -stale-copy fact must survive both. No `carlos` verb has a `--json` flag |
| -today, so this **introduces** one, and the spec fixes its fields here so |
| -the human and machine forms cannot drift: `copy: true`, `as_of`, |
| -`replicated_txid`, `instance: awake|asleep|unknown`, `sync_class`, |
| -`verified: ok|failed|unverified`. |
| +"as of". |
| + |
| +**R2 — Age is a number, and carries its expected bound — or admits it |
| +cannot.** Report the **age of the newest replicated write**, never a |
| +"lag" (which implies a live comparison the client cannot make, §8) and |
| +never an adjective. Beside it, report the replication class bound: an |
| +awake `daily` instance is legitimately up to 24h behind, exactly the |
| +case where an old copy is most easily mistaken for a current one. |
| + |
| +Where the class is **unknown** — which is the common case, because the |
| +console only knows it for routes on its own box (§8) — say so. It must |
| +never render as `stream` by default. A wrong bound is worse than no |
| +bound: it converts "I don't know" into "one second old". |
| + |
| +**R3 — The interactive prompt carries it too.** A `shell` session can |
| +outlive its banner by an hour. The prompt itself carries both the word |
| +copy/replica and the as-of time. |
| + |
| +**R4 — Refuse loudly, explain briefly.** Four refusals need their own |
| +voice: `replicate: "app"` (§7); the read-only write refusal (§4); the |
| +residency refusal (§6), which must name residency rather than read as a |
| +transient error; and the pooled-instance refusal (§6). Per the repo's |
| +rule, store errors are logged rather than returned |
| +(`apischedule.go:145-147`) — they carry bucket names and request ids. |
| + |
| +**R5 — Written for two readers.** A human skims; an agent parses. No |
| +`carlos` verb has a `--json` flag today, so this **introduces** one, and |
| +the fields are fixed here so the human and machine forms cannot drift: |
| + |
| +``` |
| +copy: true |
| +as_of # time of the newest replicated write (§8) |
| +replicated_txid |
| +lease_held: bool # is the instance replicating right now (§6, §8) |
| +sync_class # stream|batched|daily|unknown — never defaulted |
| +verified # ok|failed|unverified |
| +``` |
| **R6 — Name what is not there.** A database declared in `data.dbs` but |
| not replicated (§7) is listed as such. "The platform has no copy of |
| @@ -233,77 +233,104 @@ not replicated (§7) is listed as such. "The platform has no copy of |
| ## 6. Transport: the replica door |
| -**This section contains the design's one unresolved decision.** |
| - |
| -### The problem |
| - |
| -The console cannot serve these bytes today, and the reason is a |
| -deliberate platform decision, not an oversight: |
| - |
| -- The console's only store is `CARLOS_DEPLOYMENT_BUCKET` |
| - (`cmd/console/main.go:395-405`). `internal/console` never reads |
| - `CARLOS_HIBERNATE_*`. |
| -- Hibernating instances' replicas live in `CARLOS_HIBERNATE_BUCKET` |
| - (`activator.go:73`) — the per-customer **data** bucket |
| - (`carlos fleets create --bucket`, edges-file `data_bucket`). |
| -- Box-wide replicas live in a third place, `s3://<bucket>/<prefix>/` |
| - keyed by absolute db path (`ops/litestream.go:170`), whose coordinates |
| - nothing publishes. |
| -- The instance-move design (2026-08-28, §149-157) makes cross-region |
| - reads of data buckets **impossible at IAM by design** — residency at |
| - rest. |
| - |
| -So the door needs a decision about buckets, credentials and residency |
| -before it can be built. |
| - |
| -### Option A — console reads the data bucket (recommended for v1) |
| - |
| -Scope v1 to activator-replicated exec instances. The console takes |
| -`CARLOS_HIBERNATE_BUCKET`/`_REGION`/`_PREFIX` (or reads `data_bucket` |
| -from edges files) and opens a second S3 client. An instance homed in |
| -another region gets a **specific residency refusal**, never a generic |
| -502. |
| - |
| -Smaller, and honest about what it can reach. Cost: the console gains |
| -read on a data bucket, which is a real widening of its blast radius, |
| -and it only works in-region. |
| - |
| -### Option B — mirror through the deployment bucket |
| - |
| -A request record; the owning box mirrors `instances/<host>/` into a |
| -staging prefix the console does serve, reusing the moves worker's |
| -CopyObject mirror (`host/movespass.go`), swept after N minutes. The |
| -console then genuinely serves what it stores, and residency stays a |
| -box-side decision. |
| - |
| -Cost: staging tenant data in the deployment bucket is *itself* a |
| -residency choice, and adds latency and a sweeper. |
| - |
| -**This is Paul's call, not mine.** It trades console blast radius |
| -against deployment-bucket residency, and both are decisions with |
| -precedent this week. |
| - |
| -### The door, once the bucket question is settled |
| - |
| -Not a general object proxy. It exposes LTX files by coordinate, not keys |
| -by name: |
| +**Decision: the console reads the region's data bucket directly.** |
| +Option B — having the owning box mirror replicas into a staging prefix |
| +in the deployment bucket — was considered and rejected. It needs a new |
| +box-side pass plus a sweeper, which contradicts §9's "no path that runs |
| +anything on the box"; its first-mirror latency is a CopyObject per LTX |
| +file; it lands tenant database bytes in the deployment bucket, which the |
| +moves design deliberately routed *around* (staging goes to a sibling |
| +data bucket, never the deployment bucket); and it does not even remove |
| +the data-bucket read, which `hibernate-verify` still needs. Its one |
| +advantage over A — cross-region reach — is a residency violation rather |
| +than a feature. |
| + |
| +### Configuration |
| + |
| +The console takes the same triple the agent reads — |
| +`CARLOS_HIBERNATE_BUCKET` / `_REGION` / `_PREFIX` — plus `CARLOS_REGION` |
| +for the placement compare, and opens one `*s3.Client` reusing |
| +`lease/endpoint.go`'s config path including `s3compat.Options()` and the |
| +endpoint override (`endpoint.go:110-131`). |
| + |
| +There is deliberately **no edges-file lookup** for another box's bucket. |
| +A bucket discovered that way is one IAM will refuse, so resolving it |
| +only converts a clear refusal into a confusing one. |
| + |
| +### Whether this needs an IAM change |
| + |
| +Probably not. The console opens S3 through the default credential chain |
| +(`cmd/console/main.go:394-399`) — the box's instance role, the same role |
| +the activator on that box uses to *write* `instances/<host>/`. If the |
| +console's box hosts any hibernating instance, the read authority is |
| +already there. If not, it is one read-only, in-region grant, smaller |
| +than the `iam:CreateUser`/`PutUserPolicy` the console already holds via |
| +`CARLOS_STORE_BACKEND=aws` (`objstore/aws.go:39-46`). |
| + |
| +### Resolution, and what is refused before S3 is touched |
| + |
| +`apps/<sqid>/<app>/instances/<host>.json` → `ValidHostShape` |
| +(`provision.go:250-262`, written as path-traversal defence — `path.Join` |
| +alone cleans `../x` to `x` rather than refusing it) → then three |
| +refusals, none of which make an S3 call: |
| + |
| +- `rec.Region != CARLOS_REGION` (both `""` in a single-region |
| + deployment) → residency refusal naming the region. |
| +- a `fleet/<name>` region (`console/fleets.go:535`) → residency refusal; |
| + the data is on the customer's own box. §13 records the v2. |
| +- `rec.Pool != ""` → refusal; a pooled record's data lives on the pool |
| + box, the same class `movespass.go:150-153` already refuses. |
| + |
| +An `AccessDenied` from S3 maps to the residency refusal as a backstop, |
| +never to a generic 502. |
| + |
| +### The door |
| + |
| +Two handlers, one gate, one rate budget: |
| - `list(app, instance, level)` → `[{level, min_txid, max_txid, size, created_at}]` |
| -- `get(app, instance, level, min_txid, max_txid)` with `Range` support — |
| - litestream's `ResumableReader` already resumes by offset. |
| +- `get(app, instance, level, min_txid, max_txid)` with `Range` support |
| + |
| +Implement them as an HTTP adapter over litestream's own |
| +`lss3.ReplicaClient` **read methods**, with `Path` set per request. That |
| +way the server derives the key layout |
| +(`<prefix>/%04x/%016x-%016x.ltx`, `s3/replica_client.go:655-656`) from |
| +the same code the writer uses, and reader and writer cannot drift. |
| Addressing by coordinate rather than key is what keeps `lock.json` |
| — which sits under the same prefix and is not a member's to fetch — |
| -unreachable. The host is resolved from |
| -`apps/<sqid>/<app>/instances/<host>.json` and **must** pass |
| -`provision.ValidHostShape` (`provision.go:250-262`, written as |
| -path-traversal defence) before becoming a key segment: `path.Join` alone |
| -cleans `../x` to `x` rather than refusing it. |
| - |
| -Hosts are unique FQDNs (`activator.go:118-121`), so hibernate prefixes |
| -cannot collide across tenants. Box-wide replica keys are absolute paths |
| -and **do** collide across boxes sharing a fleet prefix |
| -(`/data/registry.db`) — a further reason to keep them out of v1. |
| +unreachable. |
| + |
| +Following the edge store door's F8 precedent (`storedoor.go:55-59`): |
| +bytes **stream through the console**, never a presigned URL, and nothing |
| +the request supplies may influence which bucket or client is used. |
| + |
| +Six wire specifics the implementation must honour: |
| + |
| +1. `list` accepts **levels 0–9 inclusive**. `CalcRestorePlan` asks for |
| + `SnapshotLevel` (= 9, `compaction_level.go:9`) first, then 8 down to |
| + 0 (`replica.go:1452, 1476-1485`). The activator writes only 0–3 and |
| + 9; the door answers **empty** for the rest, never 404. |
| +2. Listings are ordered by `min_txid` **ascending** — |
| + `restoreLevelCursor` assumes filename order, which S3 gives because |
| + the names are zero-padded hex. |
| +3. `seek` and `useMetadata` are not on the wire. The client filters |
| + `seek` locally; nothing in a non-timestamp restore reads `CreatedAt` |
| + except debug logs. |
| +4. `size`, `min_txid` and `max_txid` are **mandatory**. `Restore` |
| + rejects `Size < ltx.HeaderSize` (`replica.go:630`) and |
| + `ResumableReader` uses size to detect a premature EOF |
| + (`resumable_reader.go:96`). Checksums are not needed. |
| +5. `get` Range semantics match the S3 client exactly: |
| + `bytes=<offset>-` when `size == 0`, else |
| + `bytes=<offset>-<offset+size-1>` (`s3/replica_client.go:647-652`). |
| + A missing object is a 404 the client maps to `os.ErrNotExist`. |
| +6. `ls` needs a third, **derived** field: `lease_held` (plus |
| + `expires_at`), computed console-side from `lock.json` |
| + (`leaser.go:31-35`; a released lease is deleted, `s3/leaser.go:180`). |
| + The lock's **body is never returned** — `Owner` is the box's |
| + identity. This is how §8's "is it replicating now" is answered |
| + without exposing an object coordinate addressing deliberately hides. |
| ### Who may open it |
| @@ -316,17 +343,17 @@ A 403 would be an oracle: it tells any token holder that a db door |
| exists for this app. The gate ethos here is one byte-identical 404. |
| No legitimate caller loses anything — an app reads its own database |
| -live, and CI has no business pulling production data. `admin` is |
| -refused **deliberately**, despite already reaching `store grant`. |
| +live, and CI has no business pulling production data. `admin` is refused |
| +**deliberately**, despite already reaching `store grant`. |
| Every successful open writes an `s.audit` row naming account, app, host, |
| database and the replicated TXID served. |
| ### The read-only replica client |
| -The client implements litestream's `ReplicaClient` over this door, the |
| -way `internal/fleetapi` implements `release.Store` over the edge store |
| -door for the same reason: the member has no AWS credentials, by design. |
| +The CLI implements litestream's `ReplicaClient` over this door, the way |
| +`internal/fleetapi` implements `release.Store` over the edge store door, |
| +and for the same reason: the member has no AWS credentials, by design. |
| The interface has eight methods (`replica_client.go:19-50`). This |
| implementation supplies the five a restore needs — `Type`, `Init`, |
| @@ -336,7 +363,7 @@ unconditionally. |
| **The client cannot corrupt the replica, because it has no code that |
| writes to one.** Not "is not supposed to" — has none. §12 records the |
| -verification of that claim and the three conditions it depends on. |
| +verification and the three conditions it depends on. |
| ## 7. Which database — one, not eight |
| @@ -346,12 +373,12 @@ only `DBs[0]` is stamped (`host/caps_pass.go:294`, whose own adopt |
| warning says so verbatim). The activator's `Replicate` takes one |
| `dbPath`. |
| -So there is no `--db` flag and no selection story. Instead: |
| +So there is no `--db` flag and no selection story: |
| - The replicated database is implied by the instance. |
| - `--instance` selects among an app's instances — inferred when there is |
| - one, required when there are several, because guessing means handing |
| - someone the wrong data while telling them it is the right data. |
| + one, required when there are several, because guessing hands someone |
| + the wrong data while telling them it is the right data. |
| - `ls` names the **declared-but-unreplicated** entries (`DBs[1:]`), |
| surfacing `dataAdvice`'s warning to a member who cannot read adopt |
| logs (R6). |
| @@ -370,32 +397,41 @@ operator needs to know the platform was never going to have this data. |
| ## 8. Age, and the three things that are not lag |
| The client **cannot compute lag**. `ops.freshness` gets the live change |
| -counter from the source file on the box (`freshness.go:96-115`); neither |
| -the console nor the CLI can reach it. Anything claiming a distance from |
| -live would be invented. |
| - |
| -What the replica alone supports, and what `ls` reports: |
| - |
| -1. **Age of the newest replicated write** — the newest LTX file's |
| - `MaxTXID` and timestamp. |
| -2. **Whether the instance is replicating now** — whether |
| - `<prefix>/lock.json` is currently held (`s3/leaser.go:23`): awake and |
| - streaming, versus parked. |
| -3. **The expected bound** — the route's replication class: |
| - `""`/`stream` ≈ 1s, `batched` 5m, `daily` 24h |
| - (`activator.go:205-230`, `lease.go:245-252`). |
| +counter from the source file on the box (`freshness.go:96-115`), which |
| +neither console nor CLI can reach. Any claimed distance from live would |
| +be invented. |
| + |
| +What the replica alone supports: |
| + |
| +1. **Age of the newest replicated write.** Take it from the **LTX |
| + header timestamp**, not S3 `LastModified` — a compacted file's |
| + `LastModified` is compaction time, and the `ltx` compactor carries |
| + `maxHdr.Timestamp` through (`compactor.go:118`). One ranged GET of |
| + `ltx.HeaderSize` bytes on the max-TXID file gives the true |
| + newest-write time at any level. Using `LastModified` would report an |
| + L3 compaction as a fresh write — a stale copy looking current, the |
| + exact failure this command exists to prevent. |
| +2. **Whether it is replicating now** — `lease_held`, derived from |
| + `lock.json` (§6). |
| +3. **The expected bound — when it is knowable.** `Replication` lives on |
| + `registry.Route` (`registry.go:126`) in the **box** registry; |
| + `provision.Record` has no such field. So the console knows the class |
| + only for routes on its own box. For an in-region sibling it is |
| + genuinely unknown, and R2 requires saying so rather than defaulting. |
| + §13 records the additive follow-up. |
| 4. **The drill verdict** — `dbverify`, per host, hibernation drill only. |
| The first draft claimed "no explicit sync interval, so 1s default, |
| -therefore seconds behind live". That is wrong for the very instances |
| -this command targets, and wrong in the dangerous direction. An awake |
| -`daily` instance is **legitimately up to a day behind**, and reporting |
| -its age without its class makes correct platform behaviour look like |
| -current data. |
| - |
| -Retention differs between the two replicators too: the `SnapshotInterval` |
| -24h / `SnapshotRetention` 7d constants (`ops/litestream.go:64-65`) reach |
| -only `/etc/litestream.yml`. The activator's embedded store never sets |
| +therefore seconds behind live". That is wrong for the instances this |
| +targets, and wrong in the dangerous direction: classes are `""`/`stream` |
| +≈ 1s, `batched` 5m, `daily` 24h (`activator.go:205-230`, |
| +`lease.go:245-252`). An awake `daily` instance is **legitimately up to a |
| +day behind**, and reporting its age without its bound makes correct |
| +platform behaviour look like current data. |
| + |
| +Retention differs between the two replicators too: `SnapshotInterval` |
| +24h / `SnapshotRetention` 7d (`ops/litestream.go:64-65`) reach only |
| +`/etc/litestream.yml`. The activator's embedded store never sets |
| `Retention`, so hibernating replicas run on litestream's |
| `DefaultRetention` of **24h**. |
| @@ -420,17 +456,18 @@ service principals of any role, an audit row per open, `--out` required |
| and refused if it exists, 0600 on anything kept, a 0700 private |
| directory for anything temporary. |
| -**Nothing new is reachable on the box.** No new door on the tenant |
| -process, no new agent verb, no path that runs anything anywhere. The |
| -sandbox (`DynamicUser`, `ProtectSystem=strict`, empty |
| +**Nothing new is reachable on the box, and nothing new runs there.** No |
| +door on the tenant process, no new agent verb, no pass in the converge |
| +loop. The sandbox (`DynamicUser`, `ProtectSystem=strict`, empty |
| `CapabilityBoundingSet`) is untouched, and the same-uid sibling residual |
| -documented in `activator/admintoken.go:8-12` is not widened, because no |
| -part of this design runs on the box at all. |
| +in `activator/admintoken.go:8-12` is not widened, because no part of |
| +this design runs on a box at all. Option B would have forfeited this |
| +sentence; A keeps it. |
| -**One new read authority**, whose shape §6 leaves open: either the |
| -console gains data-bucket read (A), or tenant data transits the |
| -deployment bucket (B). Both are real, and neither should be adopted by |
| -default. |
| +**One new read authority:** the console gains in-region, read-only |
| +access to the data bucket's `instances/*`. Bounded by residency — it |
| +cannot reach another region's data even if asked, and §6 refuses before |
| +trying rather than surfacing an `AccessDenied`. |
| **The failure mode to design against is not corruption, it is |
| confidence.** Every serious way this command hurts someone routes |
| @@ -445,17 +482,22 @@ Each row is a test. |
| | Condition | Result | |
| |---|---| |
| | Not a member of the account | Byte-identical 404, matching `instanceGate` | |
| -| Any service principal (app token, or `publish`/`instance`/`operate`/`admin`) | Same byte-identical 404 — never a 403 (§6) | |
| +| Any service principal (app token, or `publish`/`instance`/`operate`/`admin`) | Same byte-identical 404 — never a 403 | |
| | No such app | Byte-identical 404 | |
| | `replicate: "app"` | Refusal naming that cause specifically (§7) | |
| | Several instances, no `--instance` | Refusal listing the hosts | |
| -| Instance homed in another region | Residency refusal naming the region — never a generic 502 | |
| +| Instance in another region | Residency refusal naming the region — no S3 call | |
| +| Instance region `fleet/<name>` | Residency refusal — no S3 call (§13 records the v2) | |
| +| `rec.Pool != ""` | Refusal — data lives on the pool box | |
| +| S3 `AccessDenied` | Mapped to the residency refusal, never a generic 502 | |
| | Declared-but-unreplicated db named | Refusal naming it as never replicated (§7) | |
| | Nothing replicated yet | "Nothing replicated yet" — match **both** `ErrNoSnapshots` and `ErrTxNotAvailable` (`lease.go:394`) | |
| -| Replica changed under the restore | One retry, then a clear failure (§12) | |
| +| `list` for a level the activator never writes | Empty list, **not** 404 (§6.1) | |
| +| Replica changed under the restore | One retry around the whole `Restore` (§12) | |
| | Restore fails integrity check | Refuse to hand over the file; surface the `dbverify` verdict | |
| -| Instance asleep | Succeeds; reports age, class and parked state (§8) | |
| +| Instance asleep | Succeeds; reports age, `lease_held: false`, class or unknown | |
| | Instance awake on `daily` class | Succeeds; age reported beside its 24h bound (R2) | |
| +| Class not knowable (sibling box) | `sync_class: "unknown"`, never `stream` (R2) | |
| | Write attempted in `shell`/`query` | SQLite `attempt to write a readonly database`, surfaced plainly | |
| | `--out` omitted, or already exists | Usage error | |
| | Host failing `ValidHostShape` | Refused before it becomes a key segment | |
| @@ -465,24 +507,33 @@ Each row is a test. |
| Following `apischedule_test.go` and `internal/lease/check_test.go`: |
| -- **Door:** the full gate matrix above, plus a coordinate-confinement |
| - test asserting `lock.json` and any key outside the resolved prefix are |
| +- **Door:** the full gate matrix, plus a coordinate-confinement test |
| + asserting `lock.json` and any key outside the resolved prefix are |
| unreachable. That test is what keeps this from becoming an object |
| proxy as the code ages. |
| +- **Wire conformance:** levels 0–9 answer empty rather than 404; |
| + listings ascend by `min_txid`; `size`/`min_txid`/`max_txid` always |
| + present; Range semantics match `s3/replica_client.go:647-652` for both |
| + the `size == 0` and sized forms. |
| - **Replica client, both halves:** the three mutating methods error |
| without touching the network **and** a call-counting fake asserts they |
| - stay at zero across a full restore. The second half is the one that |
| - catches a litestream upgrade changing the restore path. |
| -- **`ReplicaClientV3` negative assertion** — see §12, condition (i). |
| + stay at zero across a full restore. The second half is what catches a |
| + litestream upgrade changing the restore path. |
| +- **`ReplicaClientV3` negative assertion** — §12 condition (i). |
| - **Compile-time** `var _ litestream.ReplicaClient = (*client)(nil)`. |
| +- **Freshness:** a table over (age, `lease_held`, class-or-unknown, |
| + verdict) asserting the wording distinguishes asleep from failing, |
| + always carries the bound or admits it is unknown, and **never** |
| + renders unknown as `stream`. |
| +- **Age source:** a compacted L3 file whose `LastModified` is far newer |
| + than its LTX header timestamp reports the header time (§8.1). |
| - **Selection:** `caps.Data` fixtures for `DBs[1:]` unreplicated and the |
| `replicate: "app"` refusal. |
| -- **Freshness:** a table over (age, parked/awake, class, verdict) |
| - asserting the wording distinguishes asleep from failing, and always |
| - carries the class bound. |
| - **Read-only:** an `UPDATE` in `query` fails with the pinned error |
| string, and the file on disk is byte-identical afterwards. |
| -- **Permissions:** the temp directory is 0700 and `--out` lands 0600. |
| +- **Permissions:** the temp directory is 0700, `--out` lands 0600, and |
| + `pull` succeeds when temp and destination are on different |
| + filesystems. |
| Definition of done includes a `docs/site/cli.md` heading — the |
| `internal/docsite` gate fails without one (AGENTS.md). |
| @@ -520,39 +571,63 @@ once and opens lazily, so the writer's L0 retention |
| `lease.go:78`) can delete a planned file before it is opened. |
| The outcome is a **failed** restore, never a torn one: the output is |
| -`<out>.tmp`, renamed only on success (`replica.go:661-663`). One retry |
| -handles it, as `restore-verify` already does. |
| - |
| -## 13. Open questions |
| - |
| -1. **§6's bucket decision — A or B.** Blocking; Paul's call. |
| -2. **`shell`'s prompt.** An embedded REPL is small code but a real |
| +`<out>.tmp`, renamed only on success (`replica.go:661-663`). |
| + |
| +The retry must wrap the **whole `Restore`**, not an individual read: |
| +`ResumableReader` treats `ErrNotExist` as fatal and does not retry |
| +(`resumable_reader.go:72-75`), so a vanished planned file needs a fresh |
| +plan, which only a new `Restore` produces. |
| + |
| +## 13. Open questions and follow-ups |
| + |
| +1. **Fleet-homed instances (v2).** §6 refuses them: their data is in the |
| + customer's bucket and IAM will not cross. The reverse tunnel |
| + (`edge/tunnel.go`) is a real synchronous edge→box path for exactly |
| + the hosts that dial in — so the v2 is the box serving its own replica |
| + coordinates over its tunnel channel, with the console never touching |
| + the customer's bucket. Different transport, same wire shape. |
| +2. **An additive `replication` field on the instance record.** Closes |
| + §8's `sync_class: "unknown"` for in-region siblings. Wire-additive |
| + and therefore allowed; worth doing regardless of this command. |
| +3. **`shell`'s prompt.** An embedded REPL is small code but a real |
| surface (history, formatting, dot commands). Ship `ls`/`pull`/`query` |
| - first, or all four together? |
| -3. **Console web view.** Out of scope, but §6's door is what it would |
| + first, then `shell`. |
| +4. **Console web view.** Out of scope, but §6's door is what it would |
| use; confirm the coordinate-addressed shape does not foreclose it. |
| -4. **Past points in time.** Not offered here. Given the activator's |
| - effective 24h `DefaultRetention` (§8), is "current replica only" the |
| - right permanent scope? |
| -5. **Rate limiting.** Pulls are large; neither the token-keyed instance |
| +5. **Past points in time.** Not offered. Given the activator's effective |
| + 24h `DefaultRetention` (§8), is "current replica only" the right |
| + permanent scope? |
| +6. **Rate limiting.** Pulls are large; neither the token-keyed instance |
| burst nor the address-keyed write budget fits. Needs its own budget. |
| -## 14. What the review changed |
| - |
| -The first draft's safety claim survived verification (§12). Its |
| -*transport* did not, and several facts were wrong: |
| - |
| -- "The console serves objects it already stores" — false; the replicas |
| - are in data buckets the console has no client for, and cross-region |
| - reads are barred by IAM. Now §6's open decision. |
| -- "Selection among up to eight databases" — false; a route replicates |
| - exactly one. Now §7, with the unreplicated remainder surfaced instead. |
| -- "Seconds behind live" — wrong for the targeted instances, in the |
| - dangerous direction: `daily` class is legitimately 24h. Now §8. |
| -- "litestream is vendored" / "`FleetDBs` is the platform-replicated set" |
| - / "the console's registry snapshot" / "`dbverify` is per database" — |
| - all corrected in §3. |
| -- A 403 for service principals was an oracle; now a byte-identical 404 |
| - (§6). |
| -- The mode-0600 temp file was not constructible given how `Restore` |
| - creates its own output; now a 0700 private directory (§4). |
| +### Decomposition |
| + |
| +Three units, in order: (1) door + read-only client + `pull` + `query`; |
| +(2) `ls` and its freshness fields; (3) `shell`. |
| + |
| +## 14. What review changed |
| + |
| +**Round one.** The safety claim survived verification (§12). The |
| +transport did not: "the console serves objects it already stores" was |
| +false. Also corrected — a route replicates exactly one database, not up |
| +to eight; `daily` class is legitimately 24h behind, not "seconds behind |
| +live"; a 403 for service principals was an oracle; litestream is a |
| +dependency, not vendored; `FleetDBs` is the complement of the |
| +activator's set; the console has no registry snapshot; `dbverify` is |
| +keyed by host; the mode-0600 temp file was not constructible. |
| + |
| +**Round two.** Option A chosen and B rejected on its merits (§6), so §9 |
| +regains "nothing runs on the box". The edges-file lookup was dropped as |
| +buying nothing IAM will allow. Six wire specifics added, including |
| +levels 0–9 answering empty rather than 404 and mandatory `size`. Two new |
| +findings, both in the direction of overstating freshness: |
| + |
| +- **`sync_class` is unknowable off-box** — `Replication` is on |
| + `registry.Route`, not `provision.Record`. R2 now requires an explicit |
| + "unknown" that must never render as `stream`. |
| +- **Age must come from the LTX header timestamp**, not S3 |
| + `LastModified`, or a compaction reads as a fresh write. |
| + |
| +Plus `lease_held` as a derived `ls` field (answering "is it replicating" |
| +without exposing `lock.json`), the retry wrapping the whole `Restore`, |
| +and `pull` copying rather than renaming across filesystems. |