carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1# `carlos db` for instances whose data lives in another region
2
3**Status:** rev 7, 2026-10-23. Rev 7 (plan 3, 2026-10-23): the pull's cap and chunk size, from a measurement on dev
4(§5.7); the request bound is unchanged.
5Rev 6, 2026-10-22, changes no mechanism: it states four
6things the plan review (`CODEX-FINDINGS-DB-REGION-FORWARD-PLANS.md`) found
7the plans had to decide, each marked **(plan review N)**: what a result
8names when a chunk cannot be confirmed (§5.4, 9), a box's `cancel` read
9before it elects (§5.5 step 4, §5.8, 11), a retry's reservation (§5.7, 10),
10and the order of the audit row, the creates and the serial bump, with
11nothing but `cancel` once the request has ended, checked before each write
12(§5.5 step 3, §9.3, 11 and 28). Round 2 of the plan review added DBF-32
13(§10) for SQL that JSON would rewrite, and the console's refusal of it
14(§5.5 step 3, 36).
15Rev 5 answers Astra's round 4
16(`CODEX-FINDINGS-DB-REGION-FORWARD-R4.md`), which found J1 to J5 fixed, with
17four direct fixes and no new mechanism: flushed progress lines and the
18backhaul header bound (codex 19), the result held until its chunks are out
19(codex 20), restart reconciliation by refusal and a 5 s worker exit (codex
208), and the retention size wording (codex 7). Design only, nothing built. Written for Paul,
21who approved the direction (§2). Astra accepted rev 5 in round 5, and Paul
22approved building it on 2026-10-22, with the rulings in §16 (ruling 1:
23ship through the deployment bucket, per-region bucket as plan 5). Review brief:
24`CODEX-REVIEW-BRIEF-DB-REGION-FORWARD.md`. Rev 1 was `b35b3ab1`, rev 2
25`2cf4a88f`, rev 3 `944a61fd`, rev 4 `c07629e4`. Rev 4 answered Astra's round 3
26(`CODEX-FINDINGS-DB-REGION-FORWARD-R3.md`) by making a forwarded read one
27synchronous HTTP request (§5.2): no follow, no cancel route, nothing the
28console writes after the request ends, and the store's lifecycle rule as
29the only deletion. Five invariants (§5.3) replace rev 3's seven, and every
30finding is mapped to the invariant that closes it or to the synchronous
31design that removes it. Each change is marked **(codex N)** where it lands,
32and §18 lists what was declined. The coordinator's provisional answers to
33the open questions are now Paul's rulings (§16, 2026-10-22).
34
35**Companion specs:** `2026-09-02-carlos-db-design.md` (the db door, cited
36below as "db spec"), `2026-10-08-carlos-run-design.md` (the run transport,
37"run spec"), `2026-09-11-isolated-exec-design.md` (the tenant unit sandbox).
38
39## 1. The problem
40
41`carlos db query`, `carlos db pull` and their `--at` forms restore a copy of
42an instance's database on the member's own machine. The CLI fetches the
43replica's LTX files by coordinate through the console's db door
44(`internal/console/apidb.go`). The console reads them from the data bucket
45of its own region.
46
47The console refuses every instance whose record `Region` differs from its
48own `CARLOS_REGION` (`resolveReplica`, `ErrResidency`,
49`dbResidencyRefusal`). That refusal is correct for the console: each
50region's box role is granted its own region's data bucket and no other
51(`infra/modules/region-host/main.tf`, `host_buckets`), and the console runs
52on the ie box.
53
54On 2026-10-21 all twelve instances of one customer account moved from ie to
55uk with the offline operator move. The customer can no longer run
56`carlos db query` against any of them. Every instance in a region other than
57the console's has the same gap, and self-serve moves and nearest-region
58placement will make that the common case.
59
60## 2. The direction, and what it changes
61
62Paul approved this direction: the console forwards a db read to a box in the
63region where the data lives. That box reads its own region's replica, runs
64the read-only operation there, and returns only the result. The console
65never reads another region's bucket. There is no console per region.
66
67Two earlier rulings bend under it, and this spec says so rather than
68working around them:
69
70- **db spec §9, "no path that runs anything on the box".** A forwarded query
71 runs member SQL on a box. §8 is how that stays safe.
72- **db spec §6 rejected "tenant database bytes in the deployment bucket".**
73 A forwarded read has to come back to the console through the deployment
74 bucket (§5.1). This spec puts only encrypted bytes there, sealed to a key
75 that exists only in the member's CLI (§5.6), until the store's lifecycle
76 rule expires them (§5.7).
77 In phase 1 that is only a
78 query's rows. Phase 2's pull would send the planned replica files, which
79 are a pull's result, under ruling 1 (§16, Paul, 2026-10-22). A per-region
80 bucket that keeps this ciphertext in the data's region is a follow-up
81 (plan 5, §14).
82
83One earlier ruling holds unchanged: the run spec's "the console still never
84connects to a box". The transport is the run's.
85
86## 3. What exists today (facts this design rests on)
87
881. **The CLI restores locally.** `db.go` restores the replica through
89 `dbdoor.RestoreAsOf` over a read-only client (`internal/dbdoor/client.go`)
90 into a private 0700 directory, checks it with `lease.CheckDB`, then
91 either copies it to `--out` or runs one query. The query runs with
92 `mode=ro`, `query_only`, `temp_store(2)`, `SQLITE_LIMIT_ATTACHED` 0 and
93 the one-statement and pragma guard `checkQuerySQL`. The console never
94 executes tenant SQL.
952. **`--at` depends on listing times.** litestream's timestamp plan keeps a
96 file only if its `CreatedAt` is before the target. The legacy door lists
97 S3 `LastModified`, which has second resolution, so the target is
98 truncated to its second. The claim-mode door lists no times, and `--at`
99 refuses there (`dbdoor.ErrNoTimes`).
1003. **`carlos run` already reaches any region's boxes.** Its transport is
101 objects in the deployment bucket under the app's subtree
102 (`internal/runs`): `request.json`, `out/<seq>` chunks of 256 KiB,
103 `result.json`, written create-only where it matters. The console bumps
104 `control/serial`; boxes poll it every 2 s. The console's follow door
105 pages chunks to the CLI, and its sweep (`internal/console/runsweep.go`)
106 expires, closes lost runs, and deletes old ones. A run is admitted with a
107 CAS on the instance record, because a run needs the instance to itself.
1084. **The deployment bucket is writable by every platform box**, so an
109 unsigned control object is a request any box could have written. The
110 host-retry grant (`internal/release/hostretry.go`) answers that with a
111 root-signed envelope: `release.SignRootEnvelope` on the console,
112 `VerifyRootEnvelope` against `CARLOS_RELEASE_PUBKEY` on the box. Scoped
113 release keys are refused.
1145. **Sealing to a public key exists.** `internal/appconfig/crypto.go`
115 seals with ECDH P-256, HKDF-SHA256 and AES-256-GCM, with bound associated
116 data (the v2 envelope), using only the standard library and
117 `golang.org/x/crypto/hkdf`, which `go.mod` already carries.
1186. **Boxes publish capabilities.** `control/edges/<box>.json` carries
119 `capabilities` (`region-ha-1`, `offline-move-1`, `offline-stage-1`).
120 `edgesfile.ReadLiveSnapshot` joins each file with the box's status
121 snapshot for liveness. The offline move's doors refuse a region holding
122 a box without the capability, and each box publishes the capability only
123 when its armed worker can act on it.
1247. **The box role can read its region's replicas.** In
125 carlosframework/carloku-infrastructure, `region-uk.tf` passes
126 `data_bucket_arn = aws_s3_bucket.data_uk.arn` to the `region-host`
127 module (ref v0.21.0; ie is at v0.28.8). At both refs `host_buckets`
128 grants `s3:GetObject` on `data_bucket_arn/*` and `s3:ListBucket` on the
129 bucket, plus read and write on the deployment bucket, which stays in
130 eu-west-1 and is shared by every region. Sibling regions get push-only
131 `moves/*`. So a uk box can read uk replicas, and nothing else can. No
132 IAM change is needed for this design.
1338. **Tenant processes already run sandboxed.** The claim-bound unit
134 template (`internal/host/claimunit_template.go`) and the `carlos-app@`
135 units use `ProtectSystem=strict`, `TemporaryFileSystem=/data:ro`,
136 `InaccessiblePaths=/etc/carlos`, `PrivateTmp`, `NoNewPrivileges`, an
137 empty capability set and an `IPAddressDeny` list that covers the
138 metadata service. `carlos-app@` sockets are bound by root systemd
139 (`SocketUser=carlos`), so the agent can start a sandboxed unit with no
140 privilege of its own by connecting to its socket.
1419. **The agent runs as `carlos`** (`carlos-edge.service`, `User=carlos`).
142 The console runs as `carlos-console`.
143
144## 4. Decisions
145
146| # | Decision | Section |
147|---|---|---|
148| D1 | Phase 1 forwards `query` and `query --at`. `pull` and `pull --at` forward in phase 2, under ruling 1 (§16). The claim-mode door and the seeds `--from` door stay refused. | §6 |
149| D2 | Any live box in the record's region that publishes `db-read-1` and is not draining may serve. The first to create the read's `picked` marker serves it. The instance's lease holder gets no preference. | §7 |
150| D3 | Transport is the run's objects in the deployment bucket and a serial bump, but the read is one synchronous HTTP request: the console waits on the box's objects inside it and streams them to the CLI. The request is a root-signed envelope. No record write. Every object is a create-only write under `db-reads/`, never replaced or deleted by the protocol; the store's lifecycle rule is the only deletion (§5.2, §5.3). | §5 |
151| D4 | Results are sealed to an ephemeral key the CLI makes for each read. The console and the deployment bucket only ever hold the result as ciphertext. The console does see the SQL, as the envelope's signer. | §5.6 |
152| D5 | The box splits the work. The agent (`carlos`) verifies, resolves, plans and streams objects. Everything that decodes tenant bytes (LTX, SQLite, the member's SQL) runs in a new sandboxed unit, `carlos-dbread@.service`, as a `DynamicUser` with no network. | §8 |
153| D6 | The worker gets the planned LTX files and nothing else. Its scratch is its own private `/var/tmp`, which systemd removes when the unit stops. | §8.3 |
154| D7 | Rollout: boxes publish `db-read-1`; the console forwards only with `CARLOS_DB_FORWARD=1` and a capable region. Dormant by default. | §11 |
155| D8 | Three plans here. Two follow-ups are listed, not written: plan 4 moves in-region reads to the box path after phase 3 (ruling 5); plan 5 moves this traffic to a per-region bucket (ruling 1). | §14 |
156
157## 5. Transport and authentication
158
159### 5.1 Why this transport
160
161Three candidates exist in the tree:
162
163- **Deployment-bucket objects** (`carlos run`, restart scope, the offline
164 move's stager and sealer reports). Established, provider-neutral, works
165 for a box with no inbound path, and already crosses regions.
166- **Edge backhaul** (`internal/edge/homemap.go`, `mesh.go`). A box-to-box
167 HTTPS hop for tenant traffic. Its HMAC key proves only "a box in this
168 deployment" and is deliberately allowed to buy nothing but
169 `X-Forwarded-For` belief. Using it for db reads would give that key a new
170 authority across the IAM region boundary, and the console would start
171 connecting to boxes, which the run spec ruled out.
172- **The fleet store door** (`internal/edge/storedoor.go`). Serves bucket
173 objects to fleet boxes on the tunnel hostname. Wrong direction, wrong
174 principal.
175
176Decision: deployment-bucket objects. The cost is latency (a pickup is
177seconds, not milliseconds) and a LIST per serial move on capable boxes
178(§5.8). Both are acceptable for a command a person runs by hand.
179
180### 5.2 One read, one request
181
182Rounds 1 to 3 kept finding defects in one place: rev 1 to rev 3 made a read
183a durable object with a life of its own. It could be followed, cancelled,
184expired or declared lost by the console after the fact, and its admission
185slots outlived the request that took them. Every rule added to manage that
186life made a new edge (codex 3, 4, 5, 7, 13, 14, 16, 17). Rev 4 removes the
187life: **a forwarded read is one synchronous HTTP request.** The CLI's
188request to the console stays open for the whole read, the console waits on
189the box's objects inside that request, and when the request ends the read
190is over for everyone. Nothing resumes it.
191
192**Why this is possible (the numbers).** Nothing between the CLI and the
193console cuts a long request:
194
195- **The edge.** The public listeners set `ReadHeaderTimeout` only, and say
196 why: "No ReadTimeout/WriteTimeout, those would cut long-lived streams"
197 (`internal/edge/edge.go:3354` to `3358`). The reverse proxies to a unix
198 or TCP upstream, which is how the console is reached, set no
199 `ResponseHeaderTimeout` and no overall timeout, only a 90 s idle bound on
200 pooled connections (`unixProxy`, `tcpProxy`, `edge.go:2377` to `2421`).
201- **The console.** `cmd/console/main.go:586` to `590` sets
202 `ReadHeaderTimeout` 10 s and `ReadTimeout` 5 minutes, and no
203 `WriteTimeout`. `ReadTimeout` bounds reading the request; the read door
204 reads a body of a few kilobytes and then, as the import door does
205 (`internal/console/apiinstanceimport.go:479` to `483`), sets its own read
206 deadline through `http.ResponseController` so the connection is held for
207 the read's whole bound and no longer.
208- **The CLI.** The ordinary console client has a 30 s timeout
209 (`auth.go:403`), which is why `carlos ship` already uses its own context
210 of 15 minutes (`apimode.go:4425`). The forwarded read does the same: its
211 own client with no overall timeout and a context of 23 minutes.
212- **Streaming precedent.** The db door already streams a long response with
213 a per-chunk write deadline and an idle watchdog (`handleAPIDBGet`,
214 `dbStream`, `internal/console/apidb.go`).
215
216The longest a read can hold the request is the pickup window (2 minutes
217with skew) plus the owner's deadline (20 minutes), so 22 minutes, and the
218CLI's 23-minute context is the outer bound. To keep an idle connection
219alive through home routers and NATs that drop silent TCP, the console
220writes and flushes a progress line at least every 15 s (§5.5, codex 19).
221
222### 5.3 The invariants
223
224| # | Invariant |
225|---|---|
226| **J1 One request** | A read exists for the console and the CLI only inside one open HTTP request. When the request ends, by a result, a timeout, a cancel or a dropped connection, the read is over: no route can follow, resume, cancel or end it later, and the console writes nothing about it afterwards except one best-effort `cancel`. |
227| **J2 Write once** | Every object is a conditional create (`If-None-Match: *`, `release.Store.Put` with an empty `ifMatch`) and is never replaced or deleted by the protocol. The console writes `request.json`, the queue entry and `cancel`; boxes write `picked`, chunks and `result.json`. |
228| **J3 One owner** | The first create of `picked` owns the read. A box whose create answers an error or a conflict reads `picked` back and owns the read only if it carries this box's own attempt nonce. Only the owner runs the read. |
229| **J4 The store's lifecycle rule is the only deletion** | A rule on the `db-reads/` prefix expires every object. It acts long after J1 has ended every reader, so it never races one. |
230| **J5 Admission is held by what does the work** | The console's per-person and per-console counts are held by the open request and released when its handler returns. A box's read slot and disk reservation are held until its worker unit has stopped. |
231
232What closes each open or earlier finding:
233
234| Finding | Closed by |
235|---|---|
236| codex 3 (progress ack read as the end) | removed by the synchronous design: there are no acks |
237| codex 4 (late writes after `lost`) | removed by the synchronous design and J1: the console never writes `lost`; a box's late chunks or result are objects nobody reads, and J4 removes them |
238| codex 5 (cleanup reopens the election) | J2, J4: `picked` is never deleted by the protocol, and the request is refused on age long before the lifecycle rule acts |
239| codex 7 (stored ciphertext) | J4, with the honest timing in §5.7 |
240| codex 8 (disk) | J5 |
241| codex 13 (completion deletes the evidence a read ended) | removed by the synchronous design: nothing needs that evidence after the request ends |
242| codex 14 (cleanup removes the identity to follow) | removed by the synchronous design: there is no follow |
243| codex 15 (the db gate on follow routes) | removed by the synchronous design: there are no follow routes. The one route runs the full gate once, at the start, and a `record-changed` result comes back in the same response |
244| codex 16 (lazy expiry stranded between two writes) | removed by the synchronous design (the console never writes `picked` or a result), and J3 for the box's own ambiguous `picked` |
245| codex 17 (abandoned reads hold admission) | J5: an abandoned read is a closed connection, and the handler's return releases its counts |
246| codex 18 (the forgery claim contradicts sealing) | withdrawn: §5.6 states the real trust boundary |
247
248### 5.4 Objects
249
250All objects live under `db-reads/`, so one lifecycle rule covers them (J4).
251`<id>` is `dbr_` and 26 lowercase base32 characters (128 random bits),
252minted the way `runs.MintID` mints run ids, with its own `ValidID`. Every
253object is create-only (J2).
254
255| Key | Writer | Purpose |
256|---|---|---|
257| `db-reads/<id>/request.json` | console | The request: a root envelope (§5.5). |
258| `db-reads/queue/<region>/<YYYYMMDDHH>/<unix>-<id>` | console | An empty pointer, written after the request. Boxes of `<region>` list the current and previous hour's prefix and skip any entry whose `<unix>` (the request's `issued_at`) is outside the pickup window without reading it. |
259| `db-reads/<id>/picked` | a box | The owner (J3): box name, boot id, a random attempt nonce and the owner's deadline. |
260| `db-reads/<id>/out/<seq>` | the owner | One sealed chunk of output (§5.6). |
261| `db-reads/<id>/result.json` | the owner | How the read ended. Written after every chunk is confirmed. |
262| `db-reads/<id>/cancel` | console | Best effort, after the request ended without a result, so the owner stops early. Never read by the console. |
263
264Rev 2 kept these under `apps/<acct>/<app>/` for app purge; one prefix is
265what lets one rule expire everything, and the rule is what removes them.
266
267There is no write to the instance record. A run takes a record CAS because
268it needs the instance to itself. A db read touches neither the instance nor
269its lease, needs no exclusivity, and is frequent.
270
271**Ambiguous creates on the box.** A box whose `picked` create answers an
272error or a conflict reads `picked` back: its own nonce means it owns the
273read, anything else means it steps away, and a read that still fails is
274treated as not owned (J3, codex 16). The box keeps each sealed chunk's
275bytes until its create is confirmed; a retry that answers "already exists"
276reads the object back and compares bytes, equal meaning its earlier attempt
277landed. It writes `result.json` only when every chunk is confirmed. A chunk
278it cannot confirm before its deadline is never named: the box stops the
279read and writes a `failed` result with `interrupted` whose `last_seq` is
280the last confirmed chunk, so the result names only objects that exist
281**(plan review 9)**.
282
283### 5.5 The request and its response
284
285The envelope is unchanged from rev 3:
286
287```json
288{
289 "purpose": "carlos-db-read-v1",
290 "v": 1,
291 "id": "dbr_...",
292 "account": "<sqid>",
293 "app": "shop",
294 "host": "shop.example.com",
295 "incarnation": "<hostclaim.Incarnation of the record>",
296 "region": "uk",
297 "op": "query",
298 "sql": "SELECT count(*) FROM orders",
299 "at": "2026-10-07T14:00:00Z",
300 "reply_key": "<base64 P-256 public key, uncompressed>",
301 "by": "member@example.com",
302 "issued_at": "2026-10-22T09:00:00Z"
303}
304```
305
306- Signed with `release.SignRootEnvelope` under the console's root key. A
307 console that holds only a scoped key cannot sign, and so cannot forward:
308 it answers with today's residency sentence.
309- `purpose` is a fixed string so the envelope cannot be confused with any
310 other root-signed payload. The box decodes it into the exact type with
311 `VerifyRootEnvelope`, which requires canonical JSON.
312- The box accepts an envelope only when `id` matches the key it was read
313 from and the queue entry that named it, `region` matches the queue key and
314 the box's own `CARLOS_REGION`, and `issued_at` is no more than 60 s old
315 and no more than 60 s in the future.
316- `at` is empty for the newest copy. `op` is `query` in phase 1; phase 2
317 adds `pull`.
318- The envelope carries no bucket, no prefix and no key. The box derives the
319 replica prefix itself (§8.2).
320
321**The sequence.**
322
3231. The CLI calls the existing list door, as today. A new console with
324 forwarding on answers a capable cross-region instance with 409 and an
325 additive body: `{"error": DBF-14, "forward": 1, "region": "uk",
326 "forward_ops": ["query"]}`. An old CLI prints the sentence. A new CLI
327 switches to the forwarded form.
3282. The CLI runs `checkQuerySQL` locally, makes an ephemeral P-256 key, and
329 sends one request, `POST /api/cli/apps/{account}/{app}/db/{host}/reads`,
330 with `op`, `sql`, `at` and `reply_key`.
3313. The console runs `dbGate`'s order (person only, membership with a
332 byte-identical 404, the db budget, configuration, the record, the class
333 rail, pooled, fleet), refuses a moving record, checks `checkQuerySQL`
334 again, checks capability (§7.1), and takes its admission counts (J5).
335 Any refusal so far is an ordinary status and JSON error, and so is SQL
336 holding U+FFFD, which is what JSON decoding makes of bytes that are not
337 UTF-8 (the CLI refuses such SQL first, with DBF-32) **(plan review 36)**.
338 Then it mints the id, writes the audit row (§9.3), creates the request
339 and the queue entry, bumps the serial, and starts a `200` response of
340 newline-delimited JSON. From here every outcome is a line in that body.
341 The console checks that the request is still open immediately before
342 each of those four writes, and the serial bump runs on the request's own
343 context. A request that ends before `request.json` leaves at most the
344 audit row; one that ends after it leaves `request.json`, perhaps the
345 queue entry, and `cancel`, and never a serial bump or a `200`
346 **(plan review 11, 28, 33)**.
3474. Each capable box lists its region's queue for this hour and the last
348 when the serial moves, and every 30 s if it does not. For an in-window
349 entry it has not decided, it reads and verifies the request, skips it if
350 `cancel` exists **(plan review 11)**, skips it while the box is draining
351 (§7.1), checks its own admission (§5.7), waits a random 0 to 500 ms,
352 checks draining again, and creates `picked`.
3535. The owner runs §8.2's identity bracket around planning and download,
354 runs the query in a worker, and writes sealed chunks as the output
355 arrives. It checks for `cancel` every 5 s. When the worker ends, it
356 confirms every chunk and creates `result.json`.
3576. The console, inside the same request, reads `picked` until it appears.
358 Then, about once a second, it reads `result.json` **first** and lists
359 `out/` **second**, the order the run's follow door uses for the same
360 reason (`internal/console/apirunsfollow.go:101` to `106`): a listing
361 taken before the result could miss a chunk the box confirmed just
362 before writing it **(codex 20)**. It writes to the response, in order:
363 `{"admitted": <id>}` at once, `{"picked": <box>}` once, `{"chunk":
364 <seq>, "data": <base64 sealed bytes>}` for each chunk in sequence order
365 with no gaps, `{"wait": <phase>}` whenever 15 s pass with nothing else
366 to send, and finally `{"result": <result.json>}`.
367 - **The result line comes last, and only when complete (codex 20).** It
368 is written only after every chunk from 1 to the result's open
369 `last_seq` has been written to the response. A result whose chunks are
370 not all listed yet is held, and the console keeps listing once a
371 second until they are, or until the request's own deadline ends it
372 with DBF-12. The console cannot open the sealed summary; the CLI still
373 checks the bytes against it (step 8).
374 - **Every line is flushed (codex 19).** Each line is written under a
375 fresh write deadline and then pushed out with
376 `http.ResponseController.Flush`, under the same deadline; a write or
377 flush error ends the request as a stalled client (step 7). Go buffers
378 small writes in the response writer, and `dbStream` flushes only at the
379 end, so without an explicit flush the wait lines would sit in the
380 console for minutes. The response's first line, `admitted`, is flushed
381 with the status and headers, so the headers leave at once.
382 - **Nothing downstream buffers it.** The response is
383 `Content-Type: application/x-ndjson` with `Cache-Control: no-store`
384 and no `Content-Length`. The edge's compressor does not gzip that
385 type (`compressibleType`, `internal/edge/compress.go:128`), and its
386 gzip writer passes `Flush` through when it does compress
387 (`compress.go:319`). The edge's reverse proxy flushes a response with
388 no `Content-Length` immediately (`httputil.ReverseProxy`), and
389 `no-store` keeps the cache recorder out.
390 - **Backhaul.** A request that reaches the console through another
391 region's edge travels over backhaul, whose transport waits at most 30 s
392 for response headers (`backhaulHeaderTimeout`,
393 `internal/edge/edge.go:2827`, used at `2958`). So the console bounds
394 everything before the `200` (the gate, the record read, the capability
395 read and the two creates) with a 20 s context, refusing with today's
396 "could not read the instance. Try again." on overrun, and flushes the
397 `200` and `admitted` the moment the creates land. After the headers,
398 backhaul sets no further timeout.
3997. The console ends the response **without** a result line, and the CLI
400 says DBF-3 or DBF-12, when:
401 - no `picked` has appeared 2 minutes after `issued_at` (DBF-3);
402 - the owner's deadline plus 1 minute passes with no result (DBF-12);
403 - the CLI's connection closes, or a write to it stalls for 30 s.
404 In each case the console then creates `cancel`, best effort, and
405 returns. Its admission counts are released by the return (J5).
4068. The CLI buffers every chunk (**codex 6**). On a `done` result it opens
407 the sealed summary, checks the buffered bytes against its count and
408 SHA-256, and only then writes the rows to stdout and today's status
409 line to stderr. Any other result maps to its sentence (§10). A response
410 that ends without a result prints nothing to stdout.
411
412**Ctrl-C** closes the connection. The console sees its context end, creates
413`cancel` and returns; the owner, if any, stops within 5 s. There is no
414cancel route.
415
416**A console restart** ends every read in flight: their connections close,
417and each CLI says DBF-12. Nothing on the console side needs recovering,
418because nothing outlived the request (J1). A box keeps running a read whose
419console went away until it sees `cancel` or its own deadline; its output is
420read by nobody and expires (J4).
421
422### 5.6 Sealing, the result and the trust boundary
423
424- One ephemeral P-256 key pair per read, made by the CLI and held only in
425 its memory. The public half travels in the request.
426- Each chunk is sealed with the appconfig v2 construction (ECDH P-256,
427 HKDF-SHA256, AES-256-GCM, a fresh ephemeral sender key per seal), with its
428 own HKDF info string, `carlos-db-read-v1`, and associated data binding the
429 read id, the chunk's sequence number, its stream and whether it is the
430 last chunk. A chunk moved to another read, reordered, or relabelled as
431 last fails its tag. The result's sealed summary carries the plaintext
432 byte count and SHA-256, so a dropped or added chunk fails the check.
433- **What this buys.** The deployment bucket and the console only ever hold
434 the rows as ciphertext. Nobody without the CLI's private key can read
435 them, including whoever holds the bucket. A replayed envelope produces
436 output only the original CLI could open, and that CLI is gone (J1).
437- **What it does not buy, plainly (codex 18).** Sealing proves who can
438 *read* an answer, not who *wrote* it: the recipient's key is public, and
439 each seal uses a fresh sender key, so no seal carries an author. Any
440 party that can write the deployment bucket, which is every platform box
441 (`infra/modules/region-host/main.tf`, `host_buckets`), could create
442 `picked`, chunks and a matching `result.json` for a read before the real
443 box does, and the CLI would accept that answer as consistent. The CLI's
444 checks prove the answer is complete and unaltered since it was sealed;
445 they do not prove which box sealed it. Rev 3's claim that a non-owner
446 "cannot change an answer the CLI shows" was wrong and is withdrawn. This
447 is the same trust every box already has over `carlos run` output and over
448 everything else in the deployment bucket, and the in-region door trusts
449 the bucket the same way. Authenticated authorship would need per-box
450 signing keys, which exist only in claim mode; §6 leaves claim mode alone.
451
452```json
453{
454 "state": "done",
455 "code": "",
456 "box": "carlos-flagship-uk",
457 "last_seq": 12,
458 "started": "...",
459 "ended": "...",
460 "sealed": "<sealed summary>"
461}
462```
463
464`state` is `done`, `failed`, `refused` or `cancelled`. `code` is a fixed
465vocabulary the console may read: `nothing-replicated`, `history-gone`,
466`check-failed`, `changed-twice`, `too-large`, `no-room`, `sql-error`,
467`timeout`, `output-cap`, `record-changed`, `moving`, `interrupted`. The
468sealed summary carries the transaction id and time of the copy, the oldest
469time for `history-gone`, the restored size and cap for `too-large`,
470SQLite's own error text for `sql-error`, and the byte count and SHA-256 of
471the output. The CLI maps codes to today's sentences where one exists
472(`dbRestoreError`), and to §10's rows otherwise.
473
474### 5.7 Limits and storage
475
476| Limit | Value | Enforced by |
477|---|---|---|
478| Pickup window | 60 s from `issued_at`, plus 60 s of skew | box (refuses older envelopes); console stops waiting after 2 minutes |
479| Planning and restore | 10 min | worker unit and agent context |
480| Query | 5 min wall clock | worker |
481| Owner deadline | 20 min from `picked`, stated in it | agent context, worker `RuntimeMaxSec`; console stops waiting 1 minute after it |
482| The request | at most 22 minutes; the CLI's own context is 23 | console, CLI |
483| Restored size | 2 GiB by default, `CARLOS_DB_READ_MAX_BYTES` on the box | agent, from the plan, before any byte moves |
484| Disk | a reservation per read (below) | agent |
485| Output | 64 MiB (`runs.MaxOutputBytes`, written "64 MB" in copy as the run's docs do); the read ends `output-cap` | agent; the CLI's buffer is bounded by the same number |
486| Chunk | 256 KiB or 1 s, whichever comes first (`runs.ChunkBytes`, `runs.ChunkEvery`) | agent |
487| Pull | at most 2 GiB of planned LTX files (measured on dev, 2026-10-23: relay rate 3.94 MB/s, the slowest of three runs; the rule gives 3.25 GiB, above the 2 GiB ceiling); `CARLOS_DB_PULL_MAX_BYTES` on a box can only lower it; the read ends `too-large` | agent, from the plan, before any byte moves; the CLI's spool is bounded by the same number |
488| Pull chunk | 1 MiB or 1 s, whichever comes first | agent |
489| Reads per box | 2, plus one connection kept for the readiness probe (`MaxConnections=3`) | agent, systemd |
490| Open reads per person | 2 | console, held by the open request (J5) |
491| Open reads per console | 16 | console, held by the open request, 429 with `Retry-After` |
492| Request budget | the db door's existing 600 per token per minute | console |
493
494**Admission (codex 17).** The console's counts are taken in step 3 and
495released by a `defer` in the same handler, so they are exactly the reads
496whose requests are open. An abandoned read is a closed connection; its
497handler returns within 30 s (the write stall bound) or at its next wait
498line, and the slot is free. No timer, sweep or restart is involved.
499
500**Stored ciphertext (codex 7).** The protocol deletes nothing. Every object
501of a read stays in the deployment bucket until the lifecycle rule removes
502it, and that timing is the store's, not ours. On S3 the rule (current
503versions expire 1 day after creation, noncurrent versions 1 day after they
504become noncurrent, and expired delete markers removed) makes an object
505eligible for expiry about a day after creation, with eligibility rounded to
506the next midnight UTC, and the noncurrent version eligible a day after
507that. S3 then removes eligible objects asynchronously, usually within a day
508or two, with no hard deadline. So the honest statement is: **a read's
509ciphertext lives until lifecycle expiry, typically two to four days after
510the read, with no guaranteed upper bound.** The size per read is the
511plaintext limit plus sealing overhead **(codex 7)**: the output cap limits
512the rows to 64 MiB of plaintext in phase 1; each chunk object stores its
513sealed envelope as raw bytes (not base64), which adds under 100 bytes per
514256 KiB chunk (a 65-byte sender key, a 12-byte nonce, a 16-byte tag and a
515short prefix), so a full answer stores about 64.03 MiB of ciphertext. The
516request, `picked`, `result.json`, the queue entry and `cancel` add a few
517kilobytes. Base64 appears only on the wire to the CLI. Plaintext rows are
518never in the bucket. A deployment that needs a hard deadline needs a different
519reclamation mechanism (version-aware deletes), which this design does not
520have.
521
522**Disk reservations (codex 8).** The agent is the only process that starts
523workers, so it keeps one reservation table under one mutex. Before `picked`
524it reserves the floor (5 GiB); after planning it grows that reservation to
525the plan's LTX bytes plus its restored size. Each step succeeds only if the
526filesystem holding `/var/tmp` keeps at least 5 GiB and 10% free after
527subtracting every other open reservation; otherwise the read ends `no-room`.
528A reservation is released only when its worker **unit** has stopped, since
529systemd removes the unit's private `/var/tmp` at stop. The agent learns
530that from `systemctl list-units 'carlos-dbread@*'` (readable without
531privilege). It admits a new read, or a probe, only when the number of
532active, activating or deactivating `carlos-dbread@` units is no more than
533the number of workers it is itself running. Any other unit is an orphan
534whose scratch is not in the table, so the agent refuses work until it is
535gone. There is no timeout escape: an orphan worker stops within 5 s of
536losing its connection (§8.3), and `RuntimeMaxSec` stops it in the end
537regardless.
538
539A read's one retry (a planned file that vanished, §8.2) is a second
540worker. Its first attempt's reservation stays in the table, under its own
541key, until that attempt's unit has stopped, and the retry dials only once
542the unit count says so; the retry takes a fresh reservation from the floor.
543Closing the first connection is not the unit's stop **(plan review 10)**.
544
545**Restart reconciliation (codex 8).** A restarted agent's table is empty,
546and it cannot know the sizes of workers it did not start. So it reconciles
547by refusal, against the units actually running: at start, and before every
548admission, it lists `carlos-dbread@` units, and while any is active,
549activating or deactivating that it is not itself running, it takes no read
550and no probe. It never guesses a size for an orphan and never admits beside
551one. Its first admission therefore happens only after every unit from
552before the restart has stopped and systemd has removed its scratch.
553
554### 5.8 Cost
555
556Each capable box does one LIST of its region's current and previous hour of
557queue entries per serial move and every 30 s, and two GETs per new
558in-window entry (the request, and `cancel` before electing, plan review 11).
559A read adds two console PUTs, one `picked`, one PUT per chunk and one
560result, and the console's own GETs and LISTs about once a second while the
561request is open. There are no deletes. The October bill was LIST and GET
562polling, so plan 1 measures the LIST rate on dev before the flagship roll
563and reports it.
564
565### 5.9 Replay and forgery
566
567| Attack | What stops it |
568|---|---|
569| A box writes its own request | No root signature: refused and logged once. |
570| A box writes its own queue entry for a real request | The box reads the request and requires the queue entry's region, hour and time to match the envelope. |
571| An envelope copied to another id or region | The id and region are inside the signature and must match the keys. |
572| The same envelope picked again | `picked` exists until lifecycle expiry, days after the 60 s window has closed (J2, J4); a second create fails on every box, before or after a restart **(codex 5)**. Past the window the envelope is refused on age. |
573| A box serves a read for another region | It refuses any envelope whose region is not its own `CARLOS_REGION`, and IAM refuses the bucket anyway. |
574| A request that names a bucket or prefix | There is no such field. The box derives the prefix (§8.2). |
575| A compromised box forges an answer | Not prevented. §5.6 states the boundary **(codex 18)**. |
576
577## 6. Which operations forward
578
579| Operation | Phase | Reason |
580|---|---|---|
581| `query` | 1 | This is the regression the customer hit. The result is rows, usually small, and only rows leave the region. |
582| `query --at` | 1 | Same path with a target time. The box lists the replica the way the console's door does today (`listReplicaLevel`, `LastModified` as `CreatedAt`), so the copy stops at the same transaction an in-region read would. Plan 1 tests that parity. |
583| `pull` | 2 | The result is the whole database. It rests on ruling 1 (§16, Paul, 2026-10-22: a whole copy's ciphertext rests in the deployment bucket's region until lifecycle expiry, typically days, until plan 5's per-region bucket), and a measured size cap. In phase 2 the box ships the planned LTX files, sealed, in the same single request, and the CLI restores and checks them locally with today's code. The box never restores for a pull. |
584| `pull --at` | 2 | As `pull`, with the target time. |
585| Claim-mode door (`apidbclaim.go`) | not planned | Claims are not active anywhere. The claim door reads exact object versions through `hostops`'s versioned boundary and signed file handles, and lists no times. A box-side version of that belongs to claim activation (#250), which must add times anyway. A host-claim console keeps today's residency refusal, as it refuses `carlos run`. |
586| Seeds `--from` (`resolveDBSource`) | not planned | It shares the source checks but restores into a new instance, a different flow. Cross-region seeding is its own design. |
587| Fleet and pooled instances | refused, as today | Their data is not in a platform region bucket. |
588
589Phase 1 reaches every cross-region instance a member can read today in its
590own region, for queries. Until phase 2, `pull` answers DBF-8 for a
591cross-region instance.
592
593## 7. Who serves
594
595### 7.1 Capability and selection
596
597A box publishes `db-read-1` in its edges file only when the agent's armed
598db-reads loop says so, the way `offline-move-1` comes from the armed moves
599worker (`armedOfflineView`). The loop answers yes only when all of these
600hold:
601
602- hibernation is configured (`CARLOS_HIBERNATE_BUCKET` set);
603- `CARLOS_REGION` is set;
604- `CARLOS_RELEASE_PUBKEY` is set, so it can verify envelopes;
605- `CARLOS_DB_READS` is not `0` (an operator off switch for one box);
606- `CARLOS_HOST_CLAIM` is not set;
607- the loop is running, and its last **readiness probe** passed **(codex 9)**.
608
609The readiness probe connects to the worker socket and runs the worker
610protocol's `hello` exchange, which starts a real `carlos-dbread@` instance
611and gets back the worker's protocol version and its view of its own
612sandbox (no network interface but loopback, `/data` empty, `/etc/carlos`
613unreadable). The loop probes at start and every 5 minutes, and after any
614read whose worker failed to start. A failed probe withdraws the capability
615on the next edges-file pass. A socket file whose service is missing,
616masked, broken or the wrong version therefore never advertises anything.
617
618**A probe never competes with reads (codex 9).** The socket allows three
619connections (`MaxConnections=3`): the agent runs at most two reads and at
620most one probe, so a probe always has a connection and a busy box is never
621mistaken for a broken one. The agent counts its own connections, so it
622never opens a fourth. A probe is skipped, not failed, while orphan workers
623block admission (§5.7); the last result stands until it can run. When a
624read's worker fails to start, the agent probes as soon as that connection
625is closed. Rev 2 shared two connections between reads and probes, so two
626reads at once made the probe fail and withdrew the capability.
627
628The console reads `edgesfile.ReadLiveSnapshot` before admitting a read and
629refuses up front only on complete evidence **(codex 10)**. That function
630returns a nil error when status snapshots cannot be loaded and reports the
631gap through `Complete` and `BoxesComplete` instead, so the rule reads those
632fields, never the error alone. Incomplete evidence comes first and wins
633over both refusals:
634
6351. `Complete` is false, or `BoxesComplete` is false (an unreadable edges
636 file, an unreadable or missing status snapshot, a fresh status snapshot
637 with no edges file): the request is written and the pickup window
638 decides. A person is waiting, and the console stops waiting after 2
639 minutes (DBF-3).
6402. Both are true, and no file in the record's region publishes `db-read-1`
641 on a non-draining box: DBF-1, no request written.
6423. Both are true, capable non-draining boxes exist, and none is `Fresh`:
643 DBF-2, no request written.
644
645Rev 2 checked `Complete` alone before DBF-1, so a fresh box whose edges
646file was missing could be refused as "no server can read databases".
647
648Any capable, live, non-draining box in the region may serve. There is no
649assignment by the console. Region HA (plan 2b, dormant) can put several
650boxes in a region, and they all read the same region bucket, so the
651create-only `picked` marker is the whole election. The 0 to 500 ms jitter
652spreads the race, and a box that fails admission (§5.7: no free read, the
653disk floor, or an orphan worker) does not enter it.
654
655### 7.2 Why not the lease holder
656
657A sleeping instance holds no lease, so there is no holder to ask. An awake
658instance's holder is busy serving it, and the read does not touch the
659instance, so there is nothing to gain by sending it there. Choosing by lease
660would also tie the read to the instance's lifecycle, which is exactly what
661"never the live database" rules out.
662
663### 7.3 When no box serves
664
665| Situation | Answer |
666|---|---|
667| No capable box in the region | DBF-1 at once |
668| Capable boxes, none live | DBF-2 at once |
669| Live but nobody picks in the window (all busy, all failing admission, agent stuck) | DBF-3, when the console stops waiting 2 minutes after the request |
670| The owning box dies or restarts mid-read | DBF-12, when the console stops waiting 1 minute after the owner's deadline, up to 22 minutes after the request (§5.5) |
671
672## 8. Read-only and isolation on the box
673
674### 8.1 The split
675
676| Half | Runs as | Does | Never does |
677|---|---|---|---|
678| Agent (`carlos agent`, `carlos-edge.service`) | `carlos` | lists requests; verifies envelopes; creates `picked`; reads the record; derives the prefix; lists the replica and plans the restore from the listing; reads the last planned file's 100-byte header for the size check; streams planned files from the region bucket into the worker; re-checks the record before the query runs; seals and writes output; writes the result; keeps the disk reservation table | opens SQLite, decodes LTX pages, runs SQL, writes files on disk, reads anything under `/data/instances`, takes or reads the lease |
679| Worker (`carlos system db-worker`, `carlos-dbread@.service`) | a `DynamicUser` | receives the plan and the files over its socket; writes them into its private `/var/tmp` as a litestream file replica; restores to the agent's target transaction; runs `lease.CheckDBContext`; runs the one statement with today's guards; streams rows back | anything with a network, the box's files, the bucket, other tenants' files |
680
681The rule is: everything that decodes page data or SQL runs in the worker.
682The member's SQL, the replica's pages and the restored database all do. The
683agent does object input and output and planning from listings, which it
684already does to wake instances. Its one decode is the last planned file's
685fixed 100-byte LTX header, for the size check, with the bounds
686`dbdoor.PlanRestore` already applies (a valid page size, a non-zero commit,
687TXIDs that match the listing).
688
689### 8.2 The identity bracket
690
691The replica prefix is keyed by host alone (`activator.HibernatePrefix`), so
692a host deleted and recreated, in the same account or another, reuses it.
693Checking the record once before the read would let a request admitted for
694the old instance read the new one's replica **(codex 1)**. So the agent
695brackets every byte it reads with two identical checks, and no output
696exists until the second one passes.
697
698**The check.** The agent reads the record fresh from the bucket and refuses
699(result `refused`) unless:
700
701- the record exists and its incarnation is the envelope's (`record-changed`);
702- the class rail passes (`dbClassRefusal`, moved where both console and box
703 can call it);
704- it is not pooled and not a fleet placement;
705- `rec.Region` equals the box's `CARLOS_REGION` (`record-changed`);
706- `rec.Move` is nil (`moving`);
707- `rec.Host` is the envelope's host and passes `provision.ValidHostShape`.
708
709**The order.**
710
7111. Check A, then plan from the listing.
7122. Stream every planned file into the worker. The worker restores and runs
713 `lease.CheckDBContext`, then answers `ready` with the transaction it holds. It
714 runs no SQL yet.
7153. Check B, the same check again, after the last byte has been read from
716 the bucket.
7174. Only if check B passes does the agent tell the worker `run`. The first
718 output chunk can exist only after this point.
719
720A failed check B ends the read `record-changed`, stops the worker and writes
721no chunk. A retried plan (today's "changed while it was being read" retry)
722runs both checks again around its own download.
723
724**Why two checks are enough.** The incarnation is
725`hostclaim.Incarnation(account, app, host, CreatedAt)`. A deleted and
726recreated host, including one recreated in another account, gets a new
727incarnation, and an incarnation never comes back once its record is gone.
728So a record that shows the same incarnation at A and at B was the same
729instance for the whole download, and the files read in between were that
730instance's. Pinning object versions would add nothing that this does not
731already give in non-claim mode (§18).
732
733The replica prefix is `activator.HibernatePrefix(<the box's own
734CARLOS_HIBERNATE_PREFIX>, rec.Host)` in the box's own
735`CARLOS_HIBERNATE_BUCKET`, read through `lease.ReadReplicaClient` and held
736only as the narrow `replicaReader` interface (`LTXFiles`, `OpenLTXFile`), so
737no write method is reachable. Nothing in the request chooses a bucket, a
738prefix, a key or a level beyond what the plan picks.
739
740The agent lists only level directories. It never reads `lock.json`, never
741calls the leaser, and never wakes the instance.
742
743### 8.3 The worker unit
744
745`carlos-dbread.socket`, bound by root systemd:
746
747- `ListenStream=/run/carlos-dbread/worker.sock`, `SocketUser=carlos`,
748 `SocketMode=0600`, `Accept=yes`, `MaxConnections=3` (two reads and one
749 probe, §7.1, codex 9).
750
751`carlos-dbread@.service`, one instance per connection, starting from the
752claim-bound template's hardening (§3 fact 8) with these differences:
753
754- `StandardInput=socket` and `StandardOutput=socket`, so the accepted
755 connection is the worker's stdin and stdout, and `StandardError=journal`
756 for its own diagnostics **(codex 2)**. With `Accept=yes` systemd passes
757 the connection as a file descriptor and does not put it on stdin unless
758 told to; the claim-bound template's `StandardOutput=file:` line is not
759 copied. All protocol traffic, both ways, is on that one connection;
760
761- `DynamicUser=yes` instead of `User=carlos`, so the worker owns no file on
762 the box and can read nothing `carlos` owns;
763- `PrivateNetwork=yes` and `RestrictAddressFamilies=AF_UNIX`, so no route to
764 the metadata service, the bucket or anything else exists, rather than a
765 deny list;
766- `TemporaryFileSystem=/data:ro /var/lib:ro /opt/carlos:ro /run/carlos:ro`
767 with `BindReadOnlyPaths=/opt/carlos/carlos`, and
768 `InaccessiblePaths=/etc/carlos /run/carlos-control`;
769- `PrivateTmp=yes`: its `/tmp` and `/var/tmp` are its own. The worker writes
770 only under `/var/tmp`, which is on disk on Debian 13, where `/tmp` is a
771 tmpfs. systemd deletes both when the unit stops, however it stops;
772- `MemoryMax=512M`, `CPUQuota=100%`, `TasksMax=16`, `LimitFSIZE` set to
773 the restored-size cap (it bounds each file; the agent's reservation in
774 §5.7 bounds the total, codex 8), `RuntimeMaxSec=20min`;
775- `SystemCallFilter=@system-service`, `NoNewPrivileges=yes`, empty
776 `CapabilityBoundingSet`, `ProtectProc=invisible`.
777
778`carlos system db-worker` refuses to run unless its stdin is a connected
779unix stream socket (checked with `getsockopt(SO_TYPE)` and `SO_PEERCRED`,
780whose peer must be the `carlos` user), so it cannot be started by hand or
781pointed at a file. Plan 1's acceptance test drives it only through the
782installed units.
783
784The worker protocol is length-framed messages with a fixed maximum frame
785size: `hello` (the readiness probe, §7.1), a header naming the plan's files,
786their sizes, the target transaction and the byte limits, then the files,
787then `ready` from the worker, then `run` or `stop` from the agent, then
788framed output and a final status. The worker refuses a file larger than its
789header said and any write past the header's totals.
790
791**The worker stops when its agent goes (codex 8).** From the moment it
792starts, the worker keeps one goroutine reading its connection. EOF or an
793error there cancels the one context that restore, `lease.CheckDBContext`
794(`internal/lease/check.go:44`, never `CheckDB`, which supplies a background
795context) and the query all run under, and the worker exits. Not every
796step honours a context promptly: litestream's file replica client opens a
797plain file without checking it
798(`litestream@v0.5.14/file/replica_client.go:135`), and the LTX compactor's
799page reads and writes ignore it (`ltx@v0.5.1/compactor.go:177`, `199`). So
800the watcher does not rely on cancellation alone: after cancelling, it
801waits 5 s for the worker to finish, and if it has not, it ends the process
802itself with `os.Exit` **(codex 8)**. A process exit stops a compaction
803wherever it is. A worker blocked in a write to a dead connection gets an
804error at once. So an
805agent restart ends every worker it had within seconds, systemd stops each
806unit and removes its private `/var/tmp`, and only then does the restarted
807agent's orphan count (§5.7) reach zero and admit work. While the agent
808reads the output, it writes nothing to the worker, so the watcher sees
809only EOF or the agent's own `stop`.
810
811The socket and service files are written and enabled by
812`ensureSystemUnits`, the same root path that installs the other units, on
813the next adopt pass or `update-platform-binary.sh`.
814
815Within the worker, the query keeps every guard the CLI has: `mode=ro`,
816`query_only`, `temp_store(2)`, `SQLITE_LIMIT_ATTACHED` 0 and
817`checkQuerySQL`. The database is opened with `immutable=1` as well, since
818nothing else ever writes the file. The guards and the query function move
819from `db.go` into a shared package (`internal/dbquery`), so the CLI and the
820worker run the same code and the output format cannot drift.
821
822### 8.4 Why a tenant cannot reach another tenant's data
823
8241. The console admits only a member of the account, for an instance of that
825 account's app, and signs exactly that host into the envelope.
8262. The box derives the prefix from that host, the fresh record and its own
827 config, and nothing else.
8283. The worker receives only the files the agent sent. It has no network, so
829 no credentials from the metadata service and no bucket. Its filesystem
830 view hides `/data`, `/etc/carlos` and `/run/carlos`, and its user owns
831 nothing. ATTACH is off. If a SQLite flaw let the statement open a path,
832 the sandbox leaves it nothing to open but its own scratch.
8334. The output is sealed to the requesting CLI's key.
834
835### 8.5 Why the live database is never touched
836
837The agent reads bucket objects only. The worker cannot see `/data`. Nothing
838calls the activator, the leaser or `lock.json`. A read of a busy instance's
839replica can race litestream's compaction, which is today's "changed while it
840was being read, twice" case, and is retried the same way.
841
842## 9. Console and CLI behaviour
843
844### 9.1 The one new console route
845
846```
847POST /api/cli/apps/{account}/{app}/db/{host}/reads
848```
849
850It is `svcrole.Set{}` (people only) and runs the db door's full gate once,
851at the start (§5.5 step 3). A host-claim console answers it with today's
852residency sentence. Rev 3's follow and cancel routes are gone (J1), and so
853is rev 2's acknowledgement pair. Rev 3 needed a second, smaller gate for
854follow calls so that a `record-changed` result could still be fetched after
855the instance was deleted or reclassified (codex 15). With one request there
856is nothing to fetch later: the full gate ran when the read began, and the
857box's `record-changed` result arrives as the last line of the same
858response.
859
860### 9.2 Mixed versions
861
862| CLI | Console | What happens |
863|---|---|---|
864| old | old | today's residency sentence |
865| old | new, forwarding off or region not capable | today's sentence, or DBF-1 or DBF-2 |
866| old | new, forwarding on and region capable | DBF-14: update carlos |
867| new | old | today's sentence; no forward marker, so the CLI prints it as is |
868| new | new | the forwarded read |
869
870A console with `CARLOS_DB_FORWARD` unset answers byte-identically to main.
871
872### 9.3 Audit
873
874One `db-read` activity row at admission, in the run's manner, with the
875detail DBF-21 or DBF-22. It is written before the request and the queue
876entry, while the request is open, so a read a box can see always has its
877row; a row the console cannot write refuses the read. A request that ends
878after the row and before the `200` leaves the row of an admission, and at
879most the request, the queue entry and `cancel`; never a serial bump
880**(plan review 11)**. It records the SQL's length in bytes and the first
88116 hex digits of its SHA-256, not its text (ruling 3). A member
882who kept the statement can match it to the row; the row itself holds none
883of the data a query's literals can carry. That differs from `carlos run`,
884which records argv, and keeps the in-region rule that SQL never reaches the
885audit log. The SQL itself stays in the request object, in the deployment
886bucket, until the lifecycle rule expires it with the rest of the read, on
887the timing §5.7 states: typically two to four days, with no hard bound.
888
889## 10. Errors and copy
890
891Every new user-facing sentence is below. Error and status strings follow
892Paul's voice rule (ruling 4): lower case at the start and after
893a full stop, no em dashes. Docs prose keeps the CLI page's own sentence
894case. None of this is approved yet; the copy-review gate runs before any of
895it is written into code.
896
897| ID | Where | Text |
898|---|---|---|
899| DBF-1 | console 409, no capable box in the region | `%s keeps its data in region %s. no server in that region can read databases yet, so carlos db cannot read it for now.` |
900| DBF-2 | console 409, capable boxes but none live | `%s keeps its data in region %s. no server in that region is answering right now. try again in a few minutes.` |
901| DBF-3 | CLI, nobody picked the read in 60 s | `no server in region %s picked up the read within 60 seconds. try again in a few minutes.` |
902| DBF-4 | CLI, `too-large` | `%s's database is %s once restored. carlos db can query at most %s in region %s.` |
903| DBF-5 | CLI, `no-room` | `the server in region %s does not have room to restore %s's database right now. try again later.` |
904| DBF-6 | CLI stderr, `output-cap` | `the result was cut off at 64 MB. narrow the query with WHERE or LIMIT and run it again.` |
905| DBF-7 | CLI, `timeout` | `the query ran for 5 minutes and was stopped. narrow it and run it again.` |
906| DBF-8 | CLI, `pull` before phase 2 | `%s keeps its data in region %s. carlos db pull cannot fetch a copy from there yet. carlos db query can read it.` |
907| DBF-9 | console 429, open-read bounds | `too many database reads are open. wait a minute and try again.` |
908| DBF-10 | console 409, record moving | `%s is moving between regions. read its database once the move has finished.` |
909| DBF-11 | CLI stderr while waiting | `reading a copy of %s's database in region %s.` |
910| DBF-12 | CLI, the response ended without a result (the box stopped or never finished, the console restarted, the connection dropped), or `interrupted` | `the read of %s's database in region %s did not finish, so nothing was printed. try again.` |
911| DBF-13 | CLI, a chunk or summary fails to open or add up | `the answer from region %s did not check out, so carlos db showed none of it. try again.` |
912| DBF-14 | console 409 with the forward marker, read by an old CLI | `%s keeps its data in region %s. this version of carlos cannot read it from there. run carlos update, then try again.` |
913| DBF-15 | CLI, `record-changed` | `%s changed while the read was waiting. try again.` |
914| DBF-16 | CLI, `moving` from the box | as DBF-10 |
915| DBF-21 | audit detail, query | `forwarded a database query to region %s. the query was %d bytes, sha256 %s (cli)` |
916| DBF-22 | audit detail, query with `--at` | `forwarded a database query to region %s, as of %s. the query was %d bytes, sha256 %s (cli)` |
917| DBF-30 | `docs/site/cli.md`, `carlos db`, replacing the paragraph that starts "If the app has more than one instance" | see below |
918| DBF-32 | CLI, a query that is not UTF-8 text or holds U+FFFD, refused before it is sent: JSON would rewrite it, and the box must run the bytes typed or none **(plan review 36)** | `this query cannot be sent to region %s, because it holds bytes that are not UTF-8 text or the character U+FFFD. fix the query and run it again.` |
919| DBF-31 | `docs/site/cli.md`, `carlos system`, the worker | `carlos system db-worker runs on platform boxes only. systemd starts it for each forwarded database query. you never run it yourself.` |
920
921DBF-13 is true as written only because the CLI buffers the whole answer and
922checks it before writing anything (§5.5, codex 6).
923
924DBF-30:
925
926> If the app has more than one instance, name one with `--instance`. Only
927> instances that sleep have a replica. An instance on your own fleet or in a
928> pool cannot be read this way, and the command says which.
929>
930> When an instance keeps its data in another region than the console's,
931> `carlos db query` runs in that region. A server there restores the copy,
932> runs your query and sends back only the rows. The rows are encrypted for
933> your terminal on the way, so nothing in between can read them, and
934> nothing is printed until all of them have arrived and checked out. There,
935> the copy can be at most 2 GB once restored, the query runs for at most 5
936> minutes, and the rows can come to at most 64 MB. `carlos db pull` cannot
937> fetch a copy from another region yet.
938
939Unchanged and reused: today's status line (`dbQueryStatus`), the
940`dbRestoreError` sentences for `nothing-replicated`, `history-gone`,
941`check-failed` and `changed-twice`, SQLite's own error text for
942`sql-error`, `stopped` for Ctrl-C, the guard refusals of `checkQuerySQL`,
943and the residency sentence when forwarding is off.
944
945DBF-17 to DBF-20 and DBF-23 to DBF-29 are left free for phase 2's pull rows.
946
947## 11. Rollout
948
949### 11.1 Switches and gates
950
951- **Box:** `db-read-1` (§7.1). Published only while the armed loop's
952 readiness probe passes. `CARLOS_DB_READS=0` withdraws one box.
953- **Console:** `CARLOS_DB_FORWARD=1` turns forwarding on. Unset, every
954 answer is byte-identical to main. The console also needs the root key.
955- **Store:** the `db-reads/` lifecycle rule (J4) is in place on the
956 deployment bucket before any console turns forwarding on. On the flagship
957 that is a change to the `deployment-buckets` module, a release tag, a ref
958 bump in carloku-infrastructure and a targeted apply. That is production
959 infrastructure, so it needs Paul's scoped approval under AGENTS.md's rule
960 before it is applied; plan 2 prepares the plan for that approval. A
961 self-hosted deployment sets the same rule (MinIO calls it ILM); without
962 it, `db-reads/` objects are never removed.
963- **CLI:** none. The forward marker negotiates (§9.2).
964- **Version floor:** none beyond the capability. A box that predates it
965 publishes no capability and is never asked.
966
967### 11.2 Mixed fleet
968
969A region with one capable box and one old box works: only the capable box
970lists the queue. A region with no capable box gets DBF-1. A box that loses
971the capability mid-read finishes the read it holds, within its deadline,
972and takes no new one.
973
974### 11.3 Roll order
975
976Paul's order (dev boxes, dev console, check, flagship boxes, flagship
977console), with the lifecycle rule first on each deployment's bucket. Boxes
978publish an idle capability. The dev console turns `CARLOS_DB_FORWARD=1` and
979the check is a forwarded query from a dev console to a dev box in another
980region, plus the LIST-rate measurement (§5.8) and a note of real database
981sizes (ruling 2).
982
983### 11.4 Tests
984
985Acceptance is production-wired. Stand-ins are allowed in unit tests and
986labelled so; they are not acceptance.
987
988| # | Test | Wiring |
989|---|---|---|
990| T1 | A forwarded query end to end: console in region ie, agent in region uk, two MinIO buckets on one server with separate credentials. The console's credentials cannot read the uk bucket (a MinIO policy), so any console read of it fails the test. | real console server, real agent loop, real store; CI runs it with the worker reached through a test socket, labelled non-acceptance |
991| T2 | The same on the root runner with the real `carlos-dbread.socket` and `carlos-dbread@.service` installed by the real `ensureSystemUnits`, so the socket is the worker's stdin and stdout as §8.3 says (codex 2) | acceptance gate for plan 1 |
992| T3 | Sandbox witness on the root runner, from inside the worker: open `/data/registry.db`, `/etc/carlos/host.env` and another instance's directory; connect to `169.254.169.254:80` and to any TCP address; write outside `/var/tmp`; `ATTACH` and `VACUUM INTO` with the attach limit removed (a mutation); `load_extension`. Every one fails. The private tmp is gone after a normal end, a cancel and `kill -9` of the agent mid-read. | real units |
993| T4 | Forgery and replay: unsigned, wrong key, scoped key, envelope moved to another id and to another region, a forged queue entry, the same envelope offered again before and after the agent restarts (`picked` holds, codex 5), a chunk sealed to another key, a chunk reordered, the last chunk dropped. And the boundary §5.6 states, as a documented negative: a writer that creates `picked`, chunks and a matching result first is accepted by the CLI (codex 18). | real verify and seal code, real store |
994| T5 | Two capable boxes in one region: exactly one runs each read, across 1,000 reads with random jitter. A box whose `picked` create commits on the server but answers an error reads it back and runs the read; a box whose create answers an error when another box owns it steps away (codex 16). | real store conditional create, a store wrapper that loses answers |
995| T6 | Parity: the forwarded query and the in-region query print byte-identical stdout for a fixture database (NULL, tabs, newlines, blobs, times). `--at` stops at the same transaction as the in-region door for the same replica and time, including a write in the same second as the target. | real restore, both paths |
996| T7 | The long request through the real path: a read that holds the request for 21 minutes passes through the real edge and the real console server (with its 5-minute `ReadTimeout`) and returns its result; a CLI context of 23 minutes is the outer bound. During a quiet restore the real CLI receives the `admitted` line within a second and a wait line at least every 15 s, both through the local edge and through backhaul from another edge, whose 30 s header timeout the request survives (codex 19). | real edges, real console, real CLI binary |
997| T8 | Mixed versions: the five rows of §9.2, with the forwarding-off 409 compared byte for byte against main. | real console |
998| T9 | Record changes between request and pick, during planning, during download and between download and `run`: moved, recreated in the same account, recreated in another account, class changed. Each ends with its code, the code reaches the CLI in the same response (codex 15), and no chunk is ever written (codex 1). | real records, a store wrapper that pauses the download |
999| T10 | Approved copy digests for every row in §10, in the `approved_copy_test.go` pattern. | |
1000| T11 | CLI buffering: a dropped or bad last chunk, a summary hash mismatch, and a response that ends without a result each leave stdout empty (codex 6). | real CLI binary through `carlosTestEnv` |
1001| T12 | Disk: two concurrent reads on one box whose free space fits one of them: one runs, the other ends `no-room`. An agent killed mid-read: the restarted agent takes no read and no probe while the old worker unit is active, its worker stops on EOF, and the restarted agent admits work only once the unit and its scratch are gone, without guessing a size for it. A disconnect during local compaction, after the download, still ends the worker within 5 s (codex 8). | real agent, real units, root runner |
1002| T13 | Readiness: a present socket with the service missing, masked, failing to start and speaking another protocol version each withdraws `db-read-1` within one probe and edges pass. A box running two reads keeps the capability, and its probe succeeds on the third connection (codex 9). | real units, root runner |
1003| T14 | Console evidence: status snapshots unreadable, one snapshot missing, an edges file unreadable, an empty edges listing beside a fresh status snapshot. Each admits the read; only complete evidence (both flags) gives DBF-1 or DBF-2 (codex 10). | real `ReadLiveSnapshot` over a real store |
1004| T15 | Admission (codex 17): two reads whose CLIs disconnect before pickup, and two whose CLIs disconnect after it, each free their person's and the console's count when the handler returns, without a console restart; a third read from the same person is then admitted. A console restart mid-read leaves nothing to recover. | real console |
1005| T16 | Late writes are harmless (codex 4, 13, 14): a box that keeps writing chunks and its result after the console's request has ended (by timeout and by disconnect) changes nothing any reader sees, and no protocol path issues a delete or a replacing write (a store wrapper fails the test on any `Delete` or non-create `Put` under `db-reads/`). | real console, real agent, a versioned MinIO bucket |
1006| T17 | The lifecycle rule: the module's rule covers `db-reads/` with current and noncurrent expiry and expired delete markers (a module test). | module test |
1007| T18 | The result overtaking the last chunk: a store wrapper that lets the box commit its last chunk and `result.json` between the console's listing and its result read, in both orders. The console never writes the result line before every chunk through `last_seq`, and the read succeeds (codex 20). | real console, real store |
1008
1009Mutation checks that must turn a test red: drop envelope verification (T4);
1010take the host or prefix from anywhere but the record and config (T1 with a
1011second tenant); remove `PrivateNetwork` (T3); skip the summary hash check
1012(T11); skip check B (T9); add any protocol delete or replacing write (T16);
1013release the console's counts anywhere but the handler's return (T15); skip
1014the read-back of an ambiguous `picked` (T5); drop the orphan count (T12);
1015give probes no connection of their own (T13); remove the read-deadline
1016extension or any flush (T7); list chunks before reading the result, or
1017write the result line before its last chunk (T18); drop the worker's 5 s
1018exit (T12).
1019
1020## 12. Provider boundary
1021
1022- **Core:** the transport is `release.Store` objects, so it runs on MinIO and
1023 any S3-compatible store. Every protocol write is a conditional create,
1024 which `release.Store.Put` already supports with an empty `ifMatch`, on S3
1025 and on MinIO. The box reads each planned file with a GET and writes chunks
1026 with PUTs; it never uses a server-side copy. Signing is ed25519, sealing
1027 is the standard library plus `x/crypto/hkdf`. The sandbox is systemd,
1028 which every box already needs.
1029- **One store endpoint per box (codex 11).** Every S3 client in a box process
1030 takes its endpoint from the one `CARLOS_S3_ENDPOINT`
1031 (`internal/s3compat/s3compat.go`), the deployment store through
1032 `s3compat.Options` and the replica client through `lease`'s `s3Endpoint`.
1033 So a box reaches its region's data bucket and the deployment bucket on the
1034 same server, as it already must to wake and to move instances. This
1035 design supports exactly that shape: separate buckets per region, with
1036 separate credentials, on one endpoint (§18).
1037- **Expiry.** The lifecycle rule is S3 lifecycle configuration, which MinIO
1038 supports as ILM. It is a deployment requirement (§11.1), not something the
1039 code can detect, and there is no fallback deleter: the store's rule is the
1040 only deletion (J4).
1041- **AWS adapter:** IAM is what makes the console unable to read another
1042 region's bucket on the flagship. A self-hosted deployment gets the same
1043 split from per-bucket credentials on its store.
1044- **What a non-AWS deployment loses:** nothing it has today. The capability,
1045 the liveness join and the box's identity use no AWS API. Multi-box regions
1046 are AWS-only today (H26), but nothing here depends on them; one box per
1047 region is enough.
1048- **Build:** the worker and its socket code are Linux-only, with
1049 `//go:build linux` and a `!linux` stub, so the CLI still cross-compiles for
1050 darwin and windows.
1051
1052## 13. Census
1053
1054**Plan 14 root-read census (codex 12).** The scanner reads every production
1055package and matches `Open`, `QueryContext` and the rest whatever user runs
1056them (`internal/provision/plan14_root_reads_test.go:33`). Its only classes
1057are `authority`, `hint`, `diagnostic` and `legacy` (line 478), and `legacy`
1058needs `accepted-legacy;rev3=<ruling>` as its basis. "Runs as a
1059`DynamicUser`" is a fact for the ruling's footing, not a class. So each plan
1060proposes, for a ruling on #280:
1061
1062| Declaration | Callers and identity | Proposed class and basis |
1063|---|---|---|
1064| `db.go` `runQueryContext` (main: `legacy`, `rev3=J4`, `basis=root-input`, `plan14-root-reads.tsv:202`) | moved out of `db.go` | retired through the overlay's removal row, the form the H3 carlos-run removal used |
1065| `internal/dbquery` `RunContext` (the moved body, unchanged) | the CLI on a member's machine, as the member; the worker on a box, as its `DynamicUser`. Neither is root, and neither decides anything for root | `legacy`, `accepted-legacy;rev3=<ruling>;basis=root-input`, the J4 footing main gave the same body |
1066| agent db-reads loop: the queue listing, the request read, the `picked` read-back, the record reads of checks A and B, the replica listing and GETs | the agent, `carlos`, `carlos-edge.service` | `legacy` with J1 (record and route reads) and J4 (object input), as the offline move's agent units took |
1067| worker: the scratch writes and reads, the file replica restore, `CheckDBContext`, the query, the connection watcher | `carlos-dbread@`, `DynamicUser` | `legacy` with J4 |
1068| agent readiness probe and admission (`Statfs` on `/var/tmp`, `systemctl list-units`) | the agent, `carlos` | `diagnostic` for the probe; `legacy` J4 for `Statfs` and the unit listing |
1069
1070(The census's J1 and J4 are its own dispositions, not this spec's
1071invariants of the same names.)
1072
1073The overlay re-pins the source envelopes of every changed package (`.`,
1074`internal/dbquery`, `internal/dbdoor`, `internal/host`, `internal/dbreads`)
1075and updates the legacy seal under the same ruling. The census owner may rule
1076other classes; the spec's requirement is only that every new and moved
1077declaration has one of the four, with both callers of the shared runner
1078named.
1079
1080- **Root writes.** `ensureSystemUnits` writes two new unit files. That is a
1081 root-written artifact and joins the Plan 14 write census as an overlay, as
1082 every new unit has.
1083- **9B8 record-write census.** No record writer is added. The new `Put`
1084 sites (request, queue entry, `picked`, chunks, result, cancel) are
1085 `nonrecord`, all conditional creates. There is no `Delete` site.
1086 `registerCLIRoutes` changes as a `key-site` for the one route.
1087- **Claim routes.** The route joins
1088 `internal/console/testdata/claim-routes.txt`, refused on a host-claim
1089 console like the run's.
1090- **go.mod.** No new module, so no root-envelope overlay for it.
1091
1092## 14. Plans
1093
10941. **Box side and wire, dormant.** `internal/dbreads` (ids, keys, the
1095 envelope, sealing, the result, limits); `internal/dbquery` (the query
1096 runner and guards moved out of `db.go`, with the CLI switched to it and
1097 no behaviour change); `dbdoor` planning over any `replicaReader`; the
1098 agent's db-reads loop with the identity bracket, conditional creates and
1099 the `picked` read-back, admission (reservations and the orphan count) and
1100 the readiness probe; `carlos system db-worker`, its protocol and its
1101 connection watcher; the two units in `ensureSystemUnits`; the
1102 `db-read-1` capability. Gates: T2, T3, T4, T5, T9, T12, T13, the parity
1103 half of T6 that needs no console. Census overlay. Nothing reaches a box
1104 unless a console asks.
11052. **Console and CLI for `query` and `query --at`.** The one route and its
1106 streamed response, the read-deadline extension, the forward marker, the
1107 evidence rule for capability and liveness, admission held by the
1108 request, audit, the CLI's buffered forwarded path, DBF copy,
1109 `docs/site/cli.md`, and the `deployment-buckets` lifecycle rule with the
1110 approval plan for applying it. Gates: T1, T6, T7, T8, T10, T11, T14,
1111 T15, T16, T17, T18, copy review, census overlay. Roll per §11.3.
11123. **`pull` and `pull --at`.** On ruling 1 (§16), which Paul confirmed
1113 with the storage statement on 2026-10-22. Sealed LTX files in the same single
1114 request, local restore and `CheckDBContext` in the CLI, and the pull copy
1115 rows. A 2 GiB pull in one request is the case most likely to need a
1116 longer request bound; plan 3 measures it before choosing.
1117
1118Two follow-ups are listed here and not written:
1119
11204. **Plan 4 (follow-up): in-region reads onto the box path** (ruling 5).
1121 Written once phase 3 has run on the flagship. It is not designed here
1122 beyond one constraint this spec already keeps: the box path is
1123 region-agnostic, so the console can send an in-region read through it
1124 unchanged.
11255. **Plan 5 (follow-up): per-region db-read bucket** (ruling 1). After the
1126 feature works, `db-reads/` traffic moves from the shared deployment
1127 bucket to a bucket in each data region, so that ciphertext and request
1128 SQL never rest outside the data's region. It needs its own design for
1129 the console's write grant into each region's bucket, the boxes' grant on
1130 their own, the queue and serial signal, and the lifecycle rule per
1131 bucket. Plans 1 to 3 keep the protocol's keys relative to one store
1132 (`db-reads/...`) so that plan 5 changes where the store is, not what is
1133 in it.
1134
1135## 15. Review focus: the five likeliest failure modes
1136
11371. **The worker reaches more than its copy.** A unit option missed or
1138 overridden (network, `/data`, `/etc/carlos`, the metadata service, the
1139 socket on stdin and stdout), or a SQLite path (ATTACH, `VACUUM INTO`, an
1140 extension, a temp file) that opens something. Any one of these turns a
1141 member's SQL into the box role, which reads every tenant in the region.
11422. **The prefix or host comes from the request, or the identity bracket
1143 leaks.** Any path where an envelope field, rather than the fresh record
1144 and the box's config, picks the bucket, the prefix or the host is a
1145 cross-tenant read. So is any byte read outside checks A and B, or any
1146 output written before check B.
11473. **Forgery and replay.** An unsigned or wrongly scoped request accepted,
1148 domain separation from other root envelopes missing, the region not
1149 bound, or a queue entry trusted without its request.
11504. **Something outliving the request.** Any state, route, timer or write
1151 on the console side that survives the request's end, any protocol delete
1152 or replacing write, or an admission count released anywhere but the
1153 handler's return. Each one reopens a finding J1 to J5 closed.
11545. **The long request.** A timeout on the real path that the spec missed
1155 (a proxy, a client, a keep-alive), wait lines that stall behind a slow
1156 reader, or a 2 GiB pull that does not fit the request bound.
1157
1158## 16. Rulings
1159
1160The coordinator answered rev 1's five questions provisionally on
11612026-10-22. Paul ruled on 2026-10-22, after Astra accepted rev 5: ruling 1
1162as below, and rulings 2 to 5 confirmed as written (he raised no objection).
1163All five are now Paul's.
1164
11651. **Pull through the deployment bucket (Paul, 2026-10-22).** Ship the
1166 feature through the deployment bucket as designed, for `query`,
1167 `query --at`, `pull` and `pull --at`. The true storage statement stands
1168 and is what Paul ruled on: **a read's whole ciphertext, up to its size
1169 cap, lives in the deployment bucket (eu-west-1 on the flagship) until
1170 lifecycle expiry, typically two to four days, with no guaranteed upper
1171 bound** (§5.7). For a query that is at most 64 MiB of rows; for a pull
1172 it is the planned replica files, up to the pull's cap. The SQL rests in
1173 the request object on the same timing (§9.3). The earlier provisional
1174 bound ("at most 64 MiB at a time for at most 1 hour") does not hold and
1175 is withdrawn. A short or hard bound would need version-aware deletes,
1176 with `s3:DeleteObjectVersion` on the deployment bucket for the console's
1177 role, which this design does not add.
1178 **Follow-up, after the feature works: a per-region bucket for this
1179 traffic** (plan 5, §14), so that ciphertext and request SQL never rest
1180 outside the data's region. Plans 1 to 3 do not build it and do not wait
1181 for it; they keep every `db-reads/` object key relative to one store so
1182 that plan 5 can move the store without changing the protocol.
11832. **Limits stand.** 2 GiB restored, 5 minutes, 64 MB of rows, two reads per
1184 box. Real database sizes on the flagship are still unmeasured; plan 2's
1185 dev check reports them.
1186 The pull's cap, 2 GiB of planned LTX files (§5.7, rev 7, measured on dev
1187 2026-10-23), is put to Paul as a limit: **pending his confirmation.**
11883. **Audit.** The row records the SQL's length and a SHA-256 prefix, not its
1189 text (§9.3). The SQL itself rests in the request object until expiry.
11904. **Copy casing.** New strings use Paul's voice rule: lower case after a
1191 full stop (§10). The existing db door strings that capitalise are left
1192 as they are; changing them is a separate copy review.
11935. **One path.** In-region reads move to the box path after phase 3 (§14).
1194
1195## 17. What this does not do
1196
1197- No console per region and no console-to-box connection.
1198- No resumable reads: a read that loses its connection is run again.
1199- No change to in-region reads (until the fourth plan), the claim-mode
1200 door, seeds `--from`, fleet or pooled instances.
1201- No IAM change. No new read authority for any role. The one infrastructure
1202 change is a lifecycle rule on the deployment bucket.
1203- No per-store S3 endpoints.
1204- No authenticated authorship of answers (§5.6).
1205- No `carlos db shell` and no browser view.
1206
1207## 18. Declined review findings
1208
1209**Round 1** (`CODEX-FINDINGS-DB-REGION-FORWARD.md`). Every citation was
1210checked against `main` at `5f341f33` and holds. Findings 2 to 10 and 12 are
1211accepted in full. Two are accepted in part, and rounds 2 and 3 agreed with
1212both declines:
1213
1214- **Codex 1, the "stronger guarantee" (pin object versions to the checked
1215 incarnation): declined.** The identity bracket (§8.2) closes the finding.
1216 Pinning versions needs `s3:GetObjectVersion` and `s3:ListBucketVersions`
1217 on the box's data bucket, which the flagship's uk, us and au boxes do not
1218 have at module ref v0.21.0, and a versioned reader that today lives only
1219 behind the claim boundary. Claim activation can revisit it.
1220- **Codex 11, independently configured deployment and regional-data
1221 endpoints: declined.** No box path supports regional data on a separate
1222 store server today, because every S3 client in the process reads one
1223 `CARLOS_S3_ENDPOINT`. Per-store endpoints, if wanted, are a platform-wide
1224 change with its own design.
1225
1226**Round 2** (`CODEX-FINDINGS-DB-REGION-FORWARD-R2.md`). Nothing declined.
1227
1228**Round 3** (`CODEX-FINDINGS-DB-REGION-FORWARD-R3.md`). Every citation was
1229checked and holds, including that the worker must call
1230`lease.CheckDBContext` (`internal/lease/check.go:44`), not `CheckDB`; §8.3
1231and §13 now say so. Nothing is declined. Rev 4 does not patch findings 13,
123214, 16 and 17 or the partly-holding invariants one by one: it removes the
1233durable read they all came from (§5.2), and §5.3 maps every finding to the
1234invariant that closes it or to the synchronous design that removes it.
1235Finding 7's fix is the honest timing statement, not a new mechanism; a hard
1236bound would need version-aware deletes, which is noted for Paul in §16
1237rather than built. Finding 18 is fixed by withdrawing the claim and stating
1238the trust boundary (§5.6), not by adding per-box signing.
1239
1240**Round 4** (`CODEX-FINDINGS-DB-REGION-FORWARD-R4.md`). Every citation was
1241checked and holds, including litestream's file client and the LTX
1242compactor ignoring their contexts, and the backhaul transport's 30 s header
1243timeout. Nothing is declined. All four fixes reuse what the spec already
1244had: the run's result-before-list order, `http.ResponseController`, the
1245orphan refusal, and a process exit.
1246