amadan
db-region-forward spec rev 4: a forwarded read is one synchronous request
Astra round 3 left 7, 13 and 14 partly open and found 16 to 18; all of them came from the read being a durable, resumable object. Rev 4 removes that: the CLI's one request stays open for the whole read (the edge sets no read, write or response-header timeout on its listeners or proxies; the console lifts its 5-minute ReadTimeout per request as the import door does; the CLI uses its own 23-minute context), the console waits on the box's objects inside it and streams them back, and nothing outlives the request. No follow or cancel route, no console-written picked, lost or expired, admission held by the handler, the store's lifecycle rule the only deletion with an honest timing statement. Five invariants replace seven, each finding mapped to what closes or removes it. The non-owner forgery claim is withdrawn and the trust boundary stated. Ruling 1's bound is flagged as not holding.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+359
−349
docs/superpowers/specs/2026-10-22-db-region-forward-design.md+359 −349
diff --git a/docs/superpowers/specs/2026-10-22-db-region-forward-design.md b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md| index b4f8dd272..3ae989ec8 100644 |
| --- a/docs/superpowers/specs/2026-10-22-db-region-forward-design.md |
| +++ b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md |
| @@ -1,17 +1,18 @@ |
| # `carlos db` for instances whose data lives in another region |
| -**Status:** rev 3, 2026-10-22. Design only, nothing built. Written for Paul, |
| +**Status:** rev 4, 2026-10-22. Design only, nothing built. Written for Paul, |
| who approved the direction (§2). Review brief: |
| `CODEX-REVIEW-BRIEF-DB-REGION-FORWARD.md`. Rev 1 was `b35b3ab1`, rev 2 |
| -`2cf4a88f`. Rev 2 answered Astra's round 1 |
| -(`CODEX-FINDINGS-DB-REGION-FORWARD.md`). Rev 3 answers round 2 |
| -(`CODEX-FINDINGS-DB-REGION-FORWARD-R2.md`): the request, result and |
| -acknowledgement lifecycle is rewritten as seven invariants (§5.2) with no |
| -deletes in the protocol, which closes findings 4, 7, 13 and 14 together; |
| -findings 8, 9, 10 and 15 are fixed directly. Each change is marked |
| -**(codex N)** where it lands, and §18 lists what was declined. The spec |
| -carries the coordinator's provisional answers to the open questions, which |
| -await Paul (§16); rev 3 changes the premise of ruling 1. |
| +`2cf4a88f`, rev 3 `944a61fd`. Rev 4 answers Astra's round 3 |
| +(`CODEX-FINDINGS-DB-REGION-FORWARD-R3.md`) by making a forwarded read one |
| +synchronous HTTP request (§5.2): no follow, no cancel route, nothing the |
| +console writes after the request ends, and the store's lifecycle rule as |
| +the only deletion. Five invariants (§5.3) replace rev 3's seven, and every |
| +finding is mapped to the invariant that closes it or to the synchronous |
| +design that removes it. Each change is marked **(codex N)** where it lands, |
| +and §18 lists what was declined. The spec carries the coordinator's |
| +provisional answers to the open questions, which await Paul (§16); ruling |
| +1's bound does not hold and is flagged. |
| **Companion specs:** `2026-09-02-carlos-db-design.md` (the db door, cited |
| below as "db spec"), `2026-10-08-carlos-run-design.md` (the run transport, |
| @@ -53,7 +54,8 @@ working around them: |
| - **db spec §6 rejected "tenant database bytes in the deployment bucket".** |
| A forwarded read has to come back to the console through the deployment |
| bucket (§5.1). This spec puts only encrypted bytes there, sealed to a key |
| - that exists only in the member's CLI (§5.6), for at most two days (§5.7). |
| + that exists only in the member's CLI (§5.6), until the store's lifecycle |
| + rule expires them (§5.7). |
| In phase 1 that is only a |
| query's rows. Phase 2's pull would send the planned replica files, which |
| are a pull's result, under provisional ruling 1 (§16), which awaits Paul. |
| @@ -125,7 +127,7 @@ connects to a box". The transport is the run's. |
| |---|---|---| |
| | D1 | Phase 1 forwards `query` and `query --at`. `pull` and `pull --at` forward in phase 2, under provisional ruling 1. The claim-mode door and the seeds `--from` door stay refused. | §6 | |
| | D2 | Any live box in the record's region that publishes `db-read-1` and is not draining may serve. The first to create the read's `picked` marker serves it. The instance's lease holder gets no preference. | §7 | |
| -| D3 | Transport is the run's: objects in the deployment bucket, a serial bump, chunks paged by the console. The request is a root-signed envelope. No record write. Every object is a create-only write under `db-reads/`, never replaced or deleted by the protocol; a bucket lifecycle rule expires them (§5.2). | §5 | |
| +| D3 | Transport is the run's objects in the deployment bucket and a serial bump, but the read is one synchronous HTTP request: the console waits on the box's objects inside it and streams them to the CLI. The request is a root-signed envelope. No record write. Every object is a create-only write under `db-reads/`, never replaced or deleted by the protocol; the store's lifecycle rule is the only deletion (§5.2, §5.3). | §5 | |
| | D4 | Results are sealed to an ephemeral key the CLI makes for each read. The console and the deployment bucket only ever hold the result as ciphertext. The console does see the SQL, as the envelope's signer. | §5.6 | |
| | D5 | The box splits the work. The agent (`carlos`) verifies, resolves, plans and streams objects. Everything that decodes tenant bytes (LTX, SQLite, the member's SQL) runs in a new sandboxed unit, `carlos-dbread@.service`, as a `DynamicUser` with no network. | §8 | |
| | D6 | The worker gets the planned LTX files and nothing else. Its scratch is its own private `/var/tmp`, which systemd removes when the unit stops. | §8.3 | |
| @@ -155,73 +157,108 @@ Decision: deployment-bucket objects. The cost is latency (a pickup is |
| seconds, not milliseconds) and a LIST per serial move on capable boxes |
| (§5.8). Both are acceptable for a command a person runs by hand. |
| -### 5.2 The lifecycle invariants |
| - |
| -Rounds 1 and 2 found six defects (codex 3, 4, 5, 7, 13, 14) in one place: |
| -which object ends a read, and who deletes what when. Each fix in rev 2 |
| -added a rule about deletes, and each rule made a new edge. Rev 3 removes |
| -the deletes instead. The lifecycle is these seven invariants, and §5.3 to |
| -§5.9 are the simplest mechanism that keeps them. |
| +### 5.2 One read, one request |
| + |
| +Rounds 1 to 3 kept finding defects in one place: rev 1 to rev 3 made a read |
| +a durable object with a life of its own. It could be followed, cancelled, |
| +expired or declared lost by the console after the fact, and its admission |
| +slots outlived the request that took them. Every rule added to manage that |
| +life made a new edge (codex 3, 4, 5, 7, 13, 14, 16, 17). Rev 4 removes the |
| +life: **a forwarded read is one synchronous HTTP request.** The CLI's |
| +request to the console stays open for the whole read, the console waits on |
| +the box's objects inside that request, and when the request ends the read |
| +is over for everyone. Nothing resumes it. |
| + |
| +**Why this is possible (the numbers).** Nothing between the CLI and the |
| +console cuts a long request: |
| + |
| +- **The edge.** The public listeners set `ReadHeaderTimeout` only, and say |
| + why: "No ReadTimeout/WriteTimeout, those would cut long-lived streams" |
| + (`internal/edge/edge.go:3354` to `3358`). The reverse proxies to a unix |
| + or TCP upstream, which is how the console is reached, set no |
| + `ResponseHeaderTimeout` and no overall timeout, only a 90 s idle bound on |
| + pooled connections (`unixProxy`, `tcpProxy`, `edge.go:2377` to `2421`). |
| +- **The console.** `cmd/console/main.go:586` to `590` sets |
| + `ReadHeaderTimeout` 10 s and `ReadTimeout` 5 minutes, and no |
| + `WriteTimeout`. `ReadTimeout` bounds reading the request; the read door |
| + reads a body of a few kilobytes and then, as the import door does |
| + (`internal/console/apiinstanceimport.go:479` to `483`), sets its own read |
| + deadline through `http.ResponseController` so the connection is held for |
| + the read's whole bound and no longer. |
| +- **The CLI.** The ordinary console client has a 30 s timeout |
| + (`auth.go:403`), which is why `carlos ship` already uses its own context |
| + of 15 minutes (`apimode.go:4425`). The forwarded read does the same: its |
| + own client with no overall timeout and a context of 23 minutes. |
| +- **Streaming precedent.** The db door already streams a long response with |
| + a per-chunk write deadline and an idle watchdog (`handleAPIDBGet`, |
| + `dbStream`, `internal/console/apidb.go`). |
| + |
| +The longest a read can hold the request is the pickup window (2 minutes |
| +with skew) plus the owner's deadline (20 minutes), so 22 minutes, and the |
| +CLI's 23-minute context is the outer bound. To keep an idle connection |
| +alive through home routers and NATs that drop silent TCP, the console |
| +writes a progress line at least every 15 s (§5.5). |
| + |
| +### 5.3 The invariants |
| | # | Invariant | |
| |---|---| |
| -| **I1 Immutable** | Every protocol object is created once with a conditional create (`If-None-Match: *`, `release.Store.Put` with an empty `ifMatch`) and is never replaced or deleted by the protocol. There are no acknowledgement objects and no in-place updates. | |
| -| **I2 Expiry is the store's** | Objects leave only through a bucket lifecycle rule on the whole `db-reads/` prefix: current versions expire 1 day after creation, noncurrent versions 1 day after that. A read lives at most 22 minutes (§5.7), so expiry never meets a live read. | |
| -| **I3 One owner** | The first create of `picked` owns the read: a box, or the console expiring it. Only the owning box writes chunks, and only it writes a result other than `expired` or `lost`. | |
| -| **I4 One end** | `result.json` is the read's only terminal marker. Its first create decides the outcome: the owner's, or the console's `expired` or `lost`. Storage enforces it; nothing written after it changes the outcome. | |
| -| **I5 Acceptance** | The CLI accepts output only from a `done` result written by the owner `picked` names: chunks 1 to `last_seq`, each opened under the read's key, adding up to the sealed summary. Every other object, including a late chunk, is ignored. | |
| -| **I6 Identity** | Account, app, host and region are fixed by the signed request, which lives exactly as long as every other object of the read (I1, I2). Every follow route authorises against it, never against the current instance record. | |
| -| **I7 Time is decided lazily** | `expired` (no `picked` 2 minutes after `issued_at`) and `lost` (no result 2 minutes after the owner's deadline) are created by the console when a follow call observes them, by the same conditional create as everything else. No sweep is needed to end a read. | |
| - |
| -What each invariant closes: |
| - |
| -| Finding | Closed by | How | |
| -|---|---|---| |
| -| codex 3 (progress ack read as the end) | I1, I4 | There are no acks. Only `result.json` ends a read. | |
| -| codex 4 (late writes after `lost`) | I4, I5 | `lost` is a conditional create of `result.json`. A box result that commits later fails its precondition on the server, whatever the client saw; a box result that committed first makes the console's `lost` fail. A late chunk is an object no one accepts. The fence is the store's create-only check, not a clock. | |
| -| codex 5 (cleanup reopens the election) | I1, I2, I3 | `picked` is never deleted by the protocol and outlives the 60 s acceptance window by a day. | |
| -| codex 7 (deletes leave noncurrent versions) | I1, I2 | The protocol deletes nothing. The lifecycle rule expires current and noncurrent versions of `db-reads/`. The bound is restated honestly in §5.7. | |
| -| codex 13 (completion deletes the evidence that a read ended) | I1, I4 | Nothing is deleted on completion. A console restart sees the same `result.json`. | |
| -| codex 14 (cleanup removes the identity needed to follow) | I1, I2, I6 | The request is deleted only with everything else, a day later. | |
| -| codex 15 (the db gate on follow routes) | I6 | Follow routes check membership and the request, not the current record (§9.1). | |
| +| **J1 One request** | A read exists for the console and the CLI only inside one open HTTP request. When the request ends, by a result, a timeout, a cancel or a dropped connection, the read is over: no route can follow, resume, cancel or end it later, and the console writes nothing about it afterwards except one best-effort `cancel`. | |
| +| **J2 Write once** | Every object is a conditional create (`If-None-Match: *`, `release.Store.Put` with an empty `ifMatch`) and is never replaced or deleted by the protocol. The console writes `request.json`, the queue entry and `cancel`; boxes write `picked`, chunks and `result.json`. | |
| +| **J3 One owner** | The first create of `picked` owns the read. A box whose create answers an error or a conflict reads `picked` back and owns the read only if it carries this box's own attempt nonce. Only the owner runs the read. | |
| +| **J4 The store's lifecycle rule is the only deletion** | A rule on the `db-reads/` prefix expires every object. It acts long after J1 has ended every reader, so it never races one. | |
| +| **J5 Admission is held by what does the work** | The console's per-person and per-console counts are held by the open request and released when its handler returns. A box's read slot and disk reservation are held until its worker unit has stopped. | |
| -### 5.3 Objects |
| +What closes each open or earlier finding: |
| -All objects live under one prefix, `db-reads/`, so one lifecycle rule covers |
| -them (I2). `<id>` is `dbr_` and 26 lowercase base32 characters (128 random |
| -bits), minted the way `runs.MintID` mints run ids, with its own `ValidID`. |
| -Every object below is create-only (I1). |
| +| Finding | Closed by | |
| +|---|---| |
| +| codex 3 (progress ack read as the end) | removed by the synchronous design: there are no acks | |
| +| codex 4 (late writes after `lost`) | removed by the synchronous design and J1: the console never writes `lost`; a box's late chunks or result are objects nobody reads, and J4 removes them | |
| +| codex 5 (cleanup reopens the election) | J2, J4: `picked` is never deleted by the protocol, and the request is refused on age long before the lifecycle rule acts | |
| +| codex 7 (stored ciphertext) | J4, with the honest timing in §5.7 | |
| +| codex 8 (disk) | J5 | |
| +| codex 13 (completion deletes the evidence a read ended) | removed by the synchronous design: nothing needs that evidence after the request ends | |
| +| codex 14 (cleanup removes the identity to follow) | removed by the synchronous design: there is no follow | |
| +| codex 15 (the db gate on follow routes) | removed by the synchronous design: there are no follow routes. The one route runs the full gate once, at the start, and a `record-changed` result comes back in the same response | |
| +| codex 16 (lazy expiry stranded between two writes) | removed by the synchronous design (the console never writes `picked` or a result), and J3 for the box's own ambiguous `picked` | |
| +| codex 17 (abandoned reads hold admission) | J5: an abandoned read is a closed connection, and the handler's return releases its counts | |
| +| codex 18 (the forgery claim contradicts sealing) | withdrawn: §5.6 states the real trust boundary | |
| + |
| +### 5.4 Objects |
| + |
| +All objects live under `db-reads/`, so one lifecycle rule covers them (J4). |
| +`<id>` is `dbr_` and 26 lowercase base32 characters (128 random bits), |
| +minted the way `runs.MintID` mints run ids, with its own `ValidID`. Every |
| +object is create-only (J2). |
| | Key | Writer | Purpose | |
| |---|---|---| |
| -| `db-reads/<id>/request.json` | console | The request: a root envelope (§5.4). The read's identity for its whole life (I6). | |
| -| `db-reads/queue/<region>/<YYYYMMDDHH>/<unix>-<id>` | console | An empty pointer, written after the request. Boxes of `<region>` list the current and previous hour's prefix to find reads, and skip any entry whose `<unix>` (the request's `issued_at`) is outside the pickup window without reading it. | |
| -| `db-reads/<id>/picked` | a box, or the console when expiring | The owner (I3): box name, boot id and the owner's deadline. The console's form says `expired`. | |
| -| `db-reads/<id>/out/<seq>` | the owning box | One sealed chunk of output (§5.6). | |
| -| `db-reads/<id>/cancel` | console | The CLI asked to stop. | |
| -| `db-reads/<id>/result.json` | the owning box, or the console for `expired` and `lost` | The end (I4). The box writes it after every chunk write has been confirmed. | |
| - |
| -The objects moved out of the app's subtree (rev 2 had them under |
| -`apps/<acct>/<app>/`). The run keeps its objects there so app purge sweeps |
| -them; here the lifecycle rule does that within a day, and keeping one |
| -prefix is what lets one rule cover every object. Nothing under `db-reads/` |
| -is readable without the CLI's key except the request (which holds the SQL; |
| -see §9.3) and the open half of the result. |
| +| `db-reads/<id>/request.json` | console | The request: a root envelope (§5.5). | |
| +| `db-reads/queue/<region>/<YYYYMMDDHH>/<unix>-<id>` | console | An empty pointer, written after the request. Boxes of `<region>` list the current and previous hour's prefix and skip any entry whose `<unix>` (the request's `issued_at`) is outside the pickup window without reading it. | |
| +| `db-reads/<id>/picked` | a box | The owner (J3): box name, boot id, a random attempt nonce and the owner's deadline. | |
| +| `db-reads/<id>/out/<seq>` | the owner | One sealed chunk of output (§5.6). | |
| +| `db-reads/<id>/result.json` | the owner | How the read ended. Written after every chunk is confirmed. | |
| +| `db-reads/<id>/cancel` | console | Best effort, after the request ended without a result, so the owner stops early. Never read by the console. | |
| + |
| +Rev 2 kept these under `apps/<acct>/<app>/` for app purge; one prefix is |
| +what lets one rule expire everything, and the rule is what removes them. |
| There is no write to the instance record. A run takes a record CAS because |
| it needs the instance to itself. A db read touches neither the instance nor |
| -its lease, needs no exclusivity, and is frequent. A record write per read |
| -would bump `record_seq`, race with moves and reconcile, and add a record |
| -writer to the 9B8 census for nothing. |
| +its lease, needs no exclusivity, and is frequent. |
| + |
| +**Ambiguous creates on the box.** A box whose `picked` create answers an |
| +error or a conflict reads `picked` back: its own nonce means it owns the |
| +read, anything else means it steps away, and a read that still fails is |
| +treated as not owned (J3, codex 16). The box keeps each sealed chunk's |
| +bytes until its create is confirmed; a retry that answers "already exists" |
| +reads the object back and compares bytes, equal meaning its earlier attempt |
| +landed. It writes `result.json` only when every chunk is confirmed. |
| -**A box's chunk write whose answer is lost.** The box keeps each sealed |
| -chunk's bytes until its create is confirmed. On a retry that answers |
| -"already exists", it reads the object back and compares bytes: equal means |
| -its earlier attempt landed; different ends the read `failed`, since only |
| -the owner writes there. It writes `result.json` only when every chunk is |
| -confirmed, so a `done` result never names a chunk that is not there. |
| +### 5.5 The request and its response |
| -### 5.4 The request envelope |
| +The envelope is unchanged from rev 3: |
| ```json |
| { |
| @@ -246,8 +283,8 @@ confirmed, so a `done` result never names a chunk that is not there. |
| console that holds only a scoped key cannot sign, and so cannot forward: |
| it answers with today's residency sentence. |
| - `purpose` is a fixed string so the envelope cannot be confused with any |
| - other root-signed payload. The box and the console decode it into the |
| - exact type with `VerifyRootEnvelope`, which requires canonical JSON. |
| + other root-signed payload. The box decodes it into the exact type with |
| + `VerifyRootEnvelope`, which requires canonical JSON. |
| - The box accepts an envelope only when `id` matches the key it was read |
| from and the queue entry that named it, `region` matches the queue key and |
| the box's own `CARLOS_REGION`, and `issued_at` is no more than 60 s old |
| @@ -257,56 +294,63 @@ confirmed, so a `done` result never names a chunk that is not there. |
| - The envelope carries no bucket, no prefix and no key. The box derives the |
| replica prefix itself (§8.2). |
| -### 5.5 Sequence of a forwarded query |
| +**The sequence.** |
| 1. The CLI calls the existing list door, as today. A new console with |
| forwarding on answers a capable cross-region instance with 409 and an |
| additive body: `{"error": DBF-14, "forward": 1, "region": "uk", |
| "forward_ops": ["query"]}`. An old CLI prints the sentence. A new CLI |
| switches to the forwarded form. |
| -2. The CLI runs `checkQuerySQL` locally (the same refusals as today), makes |
| - an ephemeral P-256 key, and sends |
| - `POST /api/cli/apps/{account}/{app}/db/{host}/reads` with `op`, `sql`, |
| - `at` and `reply_key`. |
| -3. Admission is the only call that reads the current instance. The console |
| - runs `dbGate`'s order (person only, membership with a byte-identical 404, |
| - the db budget, configuration, the record, the class rail, pooled, |
| - fleet), then refuses a moving record, checks `checkQuerySQL` again, |
| - checks capability (§7.1) and its concurrency bounds (§5.7), mints the id, |
| - creates the request, then the queue entry, bumps the serial, writes the |
| - audit row (§9.3) and answers 202 `{id, region}`. |
| -4. Each capable box lists its region's queue prefixes for this hour and the |
| - last when the serial moves, and every 30 s if it does not. For each |
| - entry inside the pickup window that it has not already decided, it |
| - reads and verifies the request, checks admission (§5.7), waits a random |
| - 0 to 500 ms, and creates `picked`. A box that loses the create steps |
| - away. A box remembers decided ids in memory; after a restart it reads |
| - each in-window entry once more and finds `picked` there. |
| +2. The CLI runs `checkQuerySQL` locally, makes an ephemeral P-256 key, and |
| + sends one request, `POST /api/cli/apps/{account}/{app}/db/{host}/reads`, |
| + with `op`, `sql`, `at` and `reply_key`. |
| +3. The console runs `dbGate`'s order (person only, membership with a |
| + byte-identical 404, the db budget, configuration, the record, the class |
| + rail, pooled, fleet), refuses a moving record, checks `checkQuerySQL` |
| + again, checks capability (§7.1), and takes its admission counts (J5). |
| + Any refusal so far is an ordinary status and JSON error. Then it mints |
| + the id, creates the request and the queue entry, bumps the serial, |
| + writes the audit row (§9.3), and starts a `200` response of |
| + newline-delimited JSON. From here every outcome is a line in that body. |
| +4. Each capable box lists its region's queue for this hour and the last |
| + when the serial moves, and every 30 s if it does not. For an in-window |
| + entry it has not decided, it reads and verifies the request, checks its |
| + own admission (§5.7), waits a random 0 to 500 ms and creates `picked`. |
| 5. The owner runs §8.2's identity bracket around planning and download, |
| - then runs the query in a worker and writes sealed chunks as the output |
| - arrives. It reads `cancel` every 5 s. When the worker ends, it confirms |
| - every chunk and creates `result.json`. |
| -6. The CLI polls `GET .../db/{host}/reads/{id}?after=<seq>` about once a |
| - second. The page is the run's `Page` shape: the state, chunks after |
| - `<seq>` (base64 of the sealed bytes) and the result when present, under |
| - 512 KiB. The page's chunks come only from the owner `picked` names, by |
| - key. The CLI opens each chunk into a local buffer and writes nothing to |
| - stdout yet **(codex 6)**. |
| -7. When the result is `done`, the CLI opens the sealed summary, checks the |
| - buffered bytes against its count and SHA-256, and only then writes the |
| - rows to stdout and today's status line to stderr. Any other result maps |
| - to its sentence (§10). Then it exits. It sends nothing to say it is |
| - finished; nothing needs to know (I1). |
| - |
| -The buffer is bounded by the output cap (64 MiB), which a terminal-run |
| -command can hold in memory. If the summary does not check out, the CLI has |
| -written nothing, so DBF-13's "showed none of it" is true. The status line |
| -(`dbQueryStatus`) needs the copy's transaction and time from the summary, |
| -so the CLI prints DBF-11 while it waits and today's status line with the |
| -rows. stdout carries the same rows in the same format as an in-region |
| -query. |
| - |
| -### 5.6 Sealing and the result |
| + runs the query in a worker, and writes sealed chunks as the output |
| + arrives. It checks for `cancel` every 5 s. When the worker ends, it |
| + confirms every chunk and creates `result.json`. |
| +6. The console, inside the same request, reads `picked` until it appears, |
| + then lists `out/` and reads `result.json` about once a second. It writes |
| + to the response, in order: `{"picked": <box>}` once, `{"chunk": <seq>, |
| + "data": <base64 sealed bytes>}` for each new chunk, `{"wait": <phase>}` |
| + whenever 15 s pass with nothing else to send, and finally |
| + `{"result": <result.json>}`. Each line is written under a fresh write |
| + deadline, as `dbStream` does. |
| +7. The console ends the response **without** a result line, and the CLI |
| + says DBF-3 or DBF-12, when: |
| + - no `picked` has appeared 2 minutes after `issued_at` (DBF-3); |
| + - the owner's deadline plus 1 minute passes with no result (DBF-12); |
| + - the CLI's connection closes, or a write to it stalls for 30 s. |
| + In each case the console then creates `cancel`, best effort, and |
| + returns. Its admission counts are released by the return (J5). |
| +8. The CLI buffers every chunk (**codex 6**). On a `done` result it opens |
| + the sealed summary, checks the buffered bytes against its count and |
| + SHA-256, and only then writes the rows to stdout and today's status |
| + line to stderr. Any other result maps to its sentence (§10). A response |
| + that ends without a result prints nothing to stdout. |
| + |
| +**Ctrl-C** closes the connection. The console sees its context end, creates |
| +`cancel` and returns; the owner, if any, stops within 5 s. There is no |
| +cancel route. |
| + |
| +**A console restart** ends every read in flight: their connections close, |
| +and each CLI says DBF-12. Nothing on the console side needs recovering, |
| +because nothing outlived the request (J1). A box keeps running a read whose |
| +console went away until it sees `cancel` or its own deadline; its output is |
| +read by nobody and expires (J4). |
| + |
| +### 5.6 Sealing, the result and the trust boundary |
| - One ephemeral P-256 key pair per read, made by the CLI and held only in |
| its memory. The public half travels in the request. |
| @@ -315,16 +359,26 @@ query. |
| own HKDF info string, `carlos-db-read-v1`, and associated data binding the |
| read id, the chunk's sequence number, its stream and whether it is the |
| last chunk. A chunk moved to another read, reordered, or relabelled as |
| - last fails its tag. |
| -- What this buys: the deployment bucket, which sits in the console's region, |
| - never holds readable rows. The console never sees rows. A replayed |
| - envelope (§5.9) produces output only the original CLI can open. |
| -- What it does not buy: authenticity of the sender. Any box can seal to a |
| - public key. A compromised box could write a false answer. That is the |
| - same trust every box already has over run output and over the |
| - deployment bucket, and the in-region door trusts the bucket in the same |
| - way. Per-box signing keys exist only in claim mode, which this design |
| - leaves alone (§6). |
| + last fails its tag. The result's sealed summary carries the plaintext |
| + byte count and SHA-256, so a dropped or added chunk fails the check. |
| +- **What this buys.** The deployment bucket and the console only ever hold |
| + the rows as ciphertext. Nobody without the CLI's private key can read |
| + them, including whoever holds the bucket. A replayed envelope produces |
| + output only the original CLI could open, and that CLI is gone (J1). |
| +- **What it does not buy, plainly (codex 18).** Sealing proves who can |
| + *read* an answer, not who *wrote* it: the recipient's key is public, and |
| + each seal uses a fresh sender key, so no seal carries an author. Any |
| + party that can write the deployment bucket, which is every platform box |
| + (`infra/modules/region-host/main.tf`, `host_buckets`), could create |
| + `picked`, chunks and a matching `result.json` for a read before the real |
| + box does, and the CLI would accept that answer as consistent. The CLI's |
| + checks prove the answer is complete and unaltered since it was sealed; |
| + they do not prove which box sealed it. Rev 3's claim that a non-owner |
| + "cannot change an answer the CLI shows" was wrong and is withdrawn. This |
| + is the same trust every box already has over `carlos run` output and over |
| + everything else in the deployment bucket, and the in-region door trusts |
| + the bucket the same way. Authenticated authorship would need per-box |
| + signing keys, which exist only in claim mode; §6 leaves claim mode alone. |
| ```json |
| { |
| @@ -338,49 +392,55 @@ query. |
| } |
| ``` |
| -`state` is `done`, `failed`, `refused`, `cancelled` (the owner's), or |
| -`expired` or `lost` (the console's). `code` is a fixed vocabulary the |
| -console may read: `nothing-replicated`, `history-gone`, `check-failed`, |
| -`changed-twice`, `too-large`, `no-room`, `sql-error`, `timeout`, |
| -`output-cap`, `record-changed`, `moving`, `interrupted`. The sealed summary |
| -carries the transaction id and time of the copy, the oldest time for |
| -`history-gone`, the restored size and cap for `too-large`, SQLite's own |
| -error text for `sql-error`, and the byte count and SHA-256 of the output. |
| -SQLite's error text is sealed because it can name schema. The CLI maps |
| -codes to today's sentences where one exists (`dbRestoreError`), and to |
| -§10's rows otherwise. |
| +`state` is `done`, `failed`, `refused` or `cancelled`. `code` is a fixed |
| +vocabulary the console may read: `nothing-replicated`, `history-gone`, |
| +`check-failed`, `changed-twice`, `too-large`, `no-room`, `sql-error`, |
| +`timeout`, `output-cap`, `record-changed`, `moving`, `interrupted`. The |
| +sealed summary carries the transaction id and time of the copy, the oldest |
| +time for `history-gone`, the restored size and cap for `too-large`, |
| +SQLite's own error text for `sql-error`, and the byte count and SHA-256 of |
| +the output. The CLI maps codes to today's sentences where one exists |
| +(`dbRestoreError`), and to §10's rows otherwise. |
| ### 5.7 Limits and storage |
| | Limit | Value | Enforced by | |
| |---|---|---| |
| -| Pickup window | 60 s from `issued_at`, plus 60 s of skew | box (refuses older envelopes); console creates `expired` after 2 minutes (I7) | |
| +| Pickup window | 60 s from `issued_at`, plus 60 s of skew | box (refuses older envelopes); console stops waiting after 2 minutes | |
| | Planning and restore | 10 min | worker unit and agent context | |
| | Query | 5 min wall clock | worker | |
| -| Owner deadline | 20 min from `picked`, stated in it | agent context, worker `RuntimeMaxSec`; console creates `lost` 2 minutes after it (I7) | |
| +| Owner deadline | 20 min from `picked`, stated in it | agent context, worker `RuntimeMaxSec`; console stops waiting 1 minute after it | |
| +| The request | at most 22 minutes; the CLI's own context is 23 | console, CLI | |
| | Restored size | 2 GiB by default, `CARLOS_DB_READ_MAX_BYTES` on the box | agent, from the plan, before any byte moves | |
| | Disk | a reservation per read (below) | agent | |
| -| Output | 64 MiB (`runs.MaxOutputBytes`, written "64 MB" in copy as the run's docs do); the read ends `output-cap` and the CLI says so | agent; the CLI's buffer is bounded by the same number | |
| +| Output | 64 MiB (`runs.MaxOutputBytes`, written "64 MB" in copy as the run's docs do); the read ends `output-cap` | agent; the CLI's buffer is bounded by the same number | |
| | Chunk | 256 KiB or 1 s, whichever comes first (`runs.ChunkBytes`, `runs.ChunkEvery`) | agent | |
| | Reads per box | 2, plus one connection kept for the readiness probe (`MaxConnections=3`) | agent, systemd | |
| -| Open reads per person | 2 | console, in memory | |
| -| Open reads per console | 16 | console, in memory, 429 with `Retry-After` | |
| +| Open reads per person | 2 | console, held by the open request (J5) | |
| +| Open reads per console | 16 | console, held by the open request, 429 with `Retry-After` | |
| | Request budget | the db door's existing 600 per token per minute | console | |
| -The console's open-read counts are in-memory ceilings, like the db door's |
| -token budget. After a console restart they start at zero. The real bound on |
| -work is the boxes' own two reads each, which a console restart does not |
| -change. |
| - |
| -**Stored ciphertext (codex 7).** Rev 2 promised at most 4 MiB at a time by |
| -deleting acknowledged chunks. That was not true: the deployment bucket is |
| -versioned (`infra/modules/deployment-buckets/main.tf`), `S3Store.Delete` |
| -sends no version id, and a delete only hid the chunk behind a delete |
| -marker. Rev 3 deletes nothing and states the real bound: one read's |
| -ciphertext is at most its output cap (64 MiB in phase 1), and every object |
| -of every read is gone two days after creation at most, through the |
| -lifecycle rule's current and noncurrent expiry. Plaintext rows are never in |
| -the bucket. |
| +**Admission (codex 17).** The console's counts are taken in step 3 and |
| +released by a `defer` in the same handler, so they are exactly the reads |
| +whose requests are open. An abandoned read is a closed connection; its |
| +handler returns within 30 s (the write stall bound) or at its next wait |
| +line, and the slot is free. No timer, sweep or restart is involved. |
| + |
| +**Stored ciphertext (codex 7).** The protocol deletes nothing. Every object |
| +of a read stays in the deployment bucket until the lifecycle rule removes |
| +it, and that timing is the store's, not ours. On S3 the rule (current |
| +versions expire 1 day after creation, noncurrent versions 1 day after they |
| +become noncurrent, and expired delete markers removed) makes an object |
| +eligible for expiry about a day after creation, with eligibility rounded to |
| +the next midnight UTC, and the noncurrent version eligible a day after |
| +that. S3 then removes eligible objects asynchronously, usually within a day |
| +or two, with no hard deadline. So the honest statement is: **a read's |
| +ciphertext lives until lifecycle expiry, typically two to four days after |
| +the read, with no guaranteed upper bound.** Per read it is at most the |
| +output cap (64 MiB of rows in phase 1). Plaintext rows are never in the |
| +bucket. A deployment that needs a hard deadline needs a different |
| +reclamation mechanism (version-aware deletes), which this design does not |
| +have. |
| **Disk reservations (codex 8).** The agent is the only process that starts |
| workers, so it keeps one reservation table under one mutex. Before `picked` |
| @@ -388,63 +448,26 @@ it reserves the floor (5 GiB); after planning it grows that reservation to |
| the plan's LTX bytes plus its restored size. Each step succeeds only if the |
| filesystem holding `/var/tmp` keeps at least 5 GiB and 10% free after |
| subtracting every other open reservation; otherwise the read ends `no-room`. |
| -A reservation is released only when its worker **unit** has stopped, not |
| -when its connection closes: systemd removes the unit's private `/var/tmp` |
| -at stop, so space is free only then. The agent learns that from |
| -`systemctl list-units 'carlos-dbread@*'` (readable without privilege). |
| - |
| -The agent admits a new read, or a probe, only when the number of active, |
| -activating or deactivating `carlos-dbread@` units is no more than the |
| -number of workers it is itself running. Any other unit is an orphan (from a |
| -previous agent, or one still stopping), and its scratch is not in the |
| -table, so the agent refuses work until it is gone. There is no timeout |
| -escape: an orphan worker stops when its connection is gone (§8.3), and |
| -`RuntimeMaxSec` stops it in the end regardless. A restarted agent therefore |
| -takes no read until every old worker has stopped. |
| - |
| -### 5.8 Ending a read, and cost |
| - |
| -Every end is one conditional create of `result.json` (I4): |
| - |
| -| End | Who creates the result | When | |
| -|---|---|---| |
| -| done, failed, refused | the owner | when the worker ends, after every chunk is confirmed | |
| -| cancelled | the owner | after it sees `cancel`, stops the worker and confirms its chunks | |
| -| expired | the console | on a follow call more than 2 minutes after `issued_at` with no `picked`: it creates `picked` (`expired`) first. If that create fails, a box owns the read and the console does nothing. If it succeeds, it creates the result. | |
| -| lost | the console | on a follow call more than 2 minutes after the owner's deadline with no result | |
| - |
| -If the owner's own result commits first, the console's `lost` create fails |
| -and the follow call returns the owner's result. If the console's commits |
| -first, the owner's fails, whenever its request reaches the store; the box |
| -logs it and stops. The 2 minutes after the deadline is not what makes this |
| -safe; it only keeps a slow box from being called lost while it is still |
| -finishing. |
| - |
| -A person whose box died waits up to 22 minutes for DBF-12, and can stop |
| -waiting with Ctrl-C at any time. Ctrl-C creates `cancel` and exits; the |
| -read ends whenever the owner sees it, or as `lost`. |
| - |
| -**Expiry and housekeeping.** The lifecycle rule (I2) is the only cleanup in |
| -the protocol. It is one rule in the `deployment-buckets` module, scoped to |
| -the `db-reads/` prefix: expire current versions after 1 day, noncurrent |
| -versions 1 day after that, and expired delete markers. A store without a |
| -lifecycle rule (a MinIO deployment that has not set one) gets the console's |
| -housekeeping pass instead: every hour it deletes `db-reads/` objects whose |
| -read's request is more than 2 days old. That pass is housekeeping, not |
| -protocol: it touches only reads that ended at least a day earlier, which |
| -nothing reads, and on an unversioned store it reclaims the space. On a |
| -versioned store only the lifecycle rule reclaims space, and §11 lists the |
| -rule as a rollout step. |
| - |
| -**On the box,** the agent holds no scratch at all. The worker's scratch is |
| -removed by systemd when its unit stops, whatever stopped it (§8.3). |
| - |
| -**Cost.** Each capable box does one LIST of its region's current and |
| -previous hour of queue entries per serial move and every 30 s, and one GET |
| -per new in-window entry. A read adds two console PUTs, one `picked`, one |
| -PUT per chunk, one result, and the CLI's follow GETs. There are no deletes. |
| -The October bill was LIST and GET polling, so plan 1 measures the LIST rate |
| -on dev before the flagship roll and reports it. |
| +A reservation is released only when its worker **unit** has stopped, since |
| +systemd removes the unit's private `/var/tmp` at stop. The agent learns |
| +that from `systemctl list-units 'carlos-dbread@*'` (readable without |
| +privilege). It admits a new read, or a probe, only when the number of |
| +active, activating or deactivating `carlos-dbread@` units is no more than |
| +the number of workers it is itself running. Any other unit is an orphan |
| +whose scratch is not in the table, so the agent refuses work until it is |
| +gone. There is no timeout escape: an orphan worker stops when its |
| +connection is gone (§8.3), and `RuntimeMaxSec` stops it in the end |
| +regardless. |
| + |
| +### 5.8 Cost |
| + |
| +Each capable box does one LIST of its region's current and previous hour of |
| +queue entries per serial move and every 30 s, and one GET per new in-window |
| +entry. A read adds two console PUTs, one `picked`, one PUT per chunk and one |
| +result, and the console's own GETs and LISTs about once a second while the |
| +request is open. There are no deletes. The October bill was LIST and GET |
| +polling, so plan 1 measures the LIST rate on dev before the flagship roll |
| +and reports it. |
| ### 5.9 Replay and forgery |
| @@ -453,10 +476,10 @@ on dev before the flagship roll and reports it. |
| | A box writes its own request | No root signature: refused and logged once. | |
| | A box writes its own queue entry for a real request | The box reads the request and requires the queue entry's region, hour and time to match the envelope. | |
| | An envelope copied to another id or region | The id and region are inside the signature and must match the keys. | |
| -| The same envelope picked again | `picked` exists for a day (I1, I2); a second create fails on every box, before or after a restart **(codex 5)**. Past the window the envelope is refused on age. Even if one ran, its output is sealed to the original CLI's key. | |
| -| A box writes chunks for a read it does not own | If it writes first, the owner's create of that chunk fails, the read-back differs, and the owner ends the read `failed`. If it writes later, its create fails. Either way no foreign bytes are accepted: they cannot match the owner's sealed summary (I5). A compromised box can spoil a read; it cannot change an answer the CLI shows. | |
| +| The same envelope picked again | `picked` exists until lifecycle expiry, days after the 60 s window has closed (J2, J4); a second create fails on every box, before or after a restart **(codex 5)**. Past the window the envelope is refused on age. | |
| | A box serves a read for another region | It refuses any envelope whose region is not its own `CARLOS_REGION`, and IAM refuses the bucket anyway. | |
| | A request that names a bucket or prefix | There is no such field. The box derives the prefix (§8.2). | |
| +| A compromised box forges an answer | Not prevented. §5.6 states the boundary **(codex 18)**. | |
| ## 6. Which operations forward |
| @@ -464,7 +487,7 @@ on dev before the flagship roll and reports it. |
| |---|---|---| |
| | `query` | 1 | This is the regression the customer hit. The result is rows, usually small, and only rows leave the region. | |
| | `query --at` | 1 | Same path with a target time. The box lists the replica the way the console's door does today (`listReplicaLevel`, `LastModified` as `CreatedAt`), so the copy stops at the same transaction an in-region read would. Plan 1 tests that parity. | |
| -| `pull` | 2 | The result is the whole database. It rests on provisional ruling 1, whose bound rev 3 corrects (§16: a whole copy's ciphertext rests in the deployment bucket's region for up to two days), and a measured size cap. In phase 2 the box ships the planned LTX files, sealed, under the same invariants, and the CLI restores and checks them locally with today's code. The box never restores for a pull. | |
| +| `pull` | 2 | The result is the whole database. It rests on provisional ruling 1, whose bound does not hold (§16: a whole copy's ciphertext rests in the deployment bucket's region until lifecycle expiry, typically days), and a measured size cap. In phase 2 the box ships the planned LTX files, sealed, in the same single request, and the CLI restores and checks them locally with today's code. The box never restores for a pull. | |
| | `pull --at` | 2 | As `pull`, with the target time. | |
| | Claim-mode door (`apidbclaim.go`) | not planned | Claims are not active anywhere. The claim door reads exact object versions through `hostops`'s versioned boundary and signed file handles, and lists no times. A box-side version of that belongs to claim activation (#250), which must add times anyway. A host-claim console keeps today's residency refusal, as it refuses `carlos run`. | |
| | Seeds `--from` (`resolveDBSource`) | not planned | It shares the source checks but restores into a new instance, a different flow. Cross-region seeding is its own design. | |
| @@ -519,8 +542,8 @@ over both refusals: |
| 1. `Complete` is false, or `BoxesComplete` is false (an unreadable edges |
| file, an unreadable or missing status snapshot, a fresh status snapshot |
| with no edges file): the request is written and the pickup window |
| - decides. A person is waiting, and expiry answers within about 2 |
| - minutes. |
| + decides. A person is waiting, and the console stops waiting after 2 |
| + minutes (DBF-3). |
| 2. Both are true, and no file in the record's region publishes `db-read-1` |
| on a non-draining box: DBF-1, no request written. |
| 3. Both are true, capable non-draining boxes exist, and none is `Fresh`: |
| @@ -550,8 +573,8 @@ would also tie the read to the instance's lifecycle, which is exactly what |
| |---|---| |
| | No capable box in the region | DBF-1 at once | |
| | Capable boxes, none live | DBF-2 at once | |
| -| Live but nobody picks in the window (all busy, all failing admission, agent stuck) | DBF-3, when a follow call creates `expired` about 2 minutes after the request | |
| -| The owning box dies or restarts mid-read | DBF-12, when a follow call creates `lost` 2 minutes after the owner's deadline, up to 22 minutes after the pick (§5.8) | |
| +| Live but nobody picks in the window (all busy, all failing admission, agent stuck) | DBF-3, when the console stops waiting 2 minutes after the request | |
| +| The owning box dies or restarts mid-read | DBF-12, when the console stops waiting 1 minute after the owner's deadline, up to 22 minutes after the request (§5.5) | |
| ## 8. Read-only and isolation on the box |
| @@ -560,7 +583,7 @@ would also tie the read to the instance's lifecycle, which is exactly what |
| | Half | Runs as | Does | Never does | |
| |---|---|---|---| |
| | Agent (`carlos agent`, `carlos-edge.service`) | `carlos` | lists requests; verifies envelopes; creates `picked`; reads the record; derives the prefix; lists the replica and plans the restore from the listing; reads the last planned file's 100-byte header for the size check; streams planned files from the region bucket into the worker; re-checks the record before the query runs; seals and writes output; writes the result; keeps the disk reservation table | opens SQLite, decodes LTX pages, runs SQL, writes files on disk, reads anything under `/data/instances`, takes or reads the lease | |
| -| Worker (`carlos system db-worker`, `carlos-dbread@.service`) | a `DynamicUser` | receives the plan and the files over its socket; writes them into its private `/var/tmp` as a litestream file replica; restores to the agent's target transaction; runs `lease.CheckDB`; runs the one statement with today's guards; streams rows back | anything with a network, the box's files, the bucket, other tenants' files | |
| +| Worker (`carlos system db-worker`, `carlos-dbread@.service`) | a `DynamicUser` | receives the plan and the files over its socket; writes them into its private `/var/tmp` as a litestream file replica; restores to the agent's target transaction; runs `lease.CheckDBContext`; runs the one statement with today's guards; streams rows back | anything with a network, the box's files, the bucket, other tenants' files | |
| The rule is: everything that decodes page data or SQL runs in the worker. |
| The member's SQL, the replica's pages and the restored database all do. The |
| @@ -594,7 +617,7 @@ exists until the second one passes. |
| 1. Check A, then plan from the listing. |
| 2. Stream every planned file into the worker. The worker restores and runs |
| - `lease.CheckDB`, then answers `ready` with the transaction it holds. It |
| + `lease.CheckDBContext`, then answers `ready` with the transaction it holds. It |
| runs no SQL yet. |
| 3. Check B, the same check again, after the last byte has been read from |
| the bucket. |
| @@ -674,9 +697,10 @@ header said and any write past the header's totals. |
| **The worker stops when its agent goes (codex 8).** From the moment it |
| starts, the worker keeps one goroutine reading its connection. EOF or an |
| -error there cancels the one context that restore, `lease.CheckDB` and the |
| -query all run under (each already stops on its context: `db.go`'s |
| -`runQueryContext`, `internal/lease/check.go`), and the worker exits. A |
| +error there cancels the one context that restore, `lease.CheckDBContext` |
| +(`internal/lease/check.go:44`, never `CheckDB`, which supplies a background |
| +context) and the query all run under (each stops on its context: `db.go`'s |
| +`runQueryContext`), and the worker exits. A |
| worker blocked in a write to a dead connection gets an error at once. So an |
| agent restart ends every worker it had within seconds, systemd stops each |
| unit and removes its private `/var/tmp`, and only then does the restarted |
| @@ -717,37 +741,21 @@ was being read, twice" case, and is retried the same way. |
| ## 9. Console and CLI behaviour |
| -### 9.1 New console routes |
| +### 9.1 The one new console route |
| ``` |
| POST /api/cli/apps/{account}/{app}/db/{host}/reads |
| -GET /api/cli/apps/{account}/{app}/db/{host}/reads/{id} |
| -POST /api/cli/apps/{account}/{app}/db/{host}/reads/{id}/cancel |
| ``` |
| -Rev 2's `ack` and `done` routes are gone with the deletes they drove (I1). |
| -All three are `svcrole.Set{}` (people only). A host-claim console answers |
| -all three with today's residency sentence. |
| - |
| -**Two gates, not one (codex 15).** Only the create route runs the db door's |
| -full gate, which reads the current instance record (class, pooled, fleet, |
| -moving). The follow and cancel routes run a smaller gate, in the repo's |
| -order: |
| - |
| -1. a person (the zero service-role set, as the db door); |
| -2. membership of `{account}`, with the byte-identical 404; |
| -3. the db token budget (429); |
| -4. the request: read `db-reads/<id>/request.json`, verify its root |
| - signature, and require its account, app and host to be the path's. Any |
| - mismatch, a missing request or a bad signature is the same |
| - byte-identical 404. |
| - |
| -They never read the instance record. An instance deleted, recreated or |
| -reclassified after admission is exactly the case where the box writes |
| -`record-changed`, and the CLI must still be able to fetch that result and |
| -to cancel. Rev 2 put all routes behind the full gate, which answered those |
| -calls 404 or 409 instead. This is the run's own split |
| -(`internal/console/apirunsfollow.go`, `requestOnHost`). |
| +It is `svcrole.Set{}` (people only) and runs the db door's full gate once, |
| +at the start (§5.5 step 3). A host-claim console answers it with today's |
| +residency sentence. Rev 3's follow and cancel routes are gone (J1), and so |
| +is rev 2's acknowledgement pair. Rev 3 needed a second, smaller gate for |
| +follow calls so that a `record-changed` result could still be fetched after |
| +the instance was deleted or reclassified (codex 15). With one request there |
| +is nothing to fetch later: the full gate ran when the read began, and the |
| +box's `record-changed` result arrives as the last line of the same |
| +response. |
| ### 9.2 Mixed versions |
| @@ -770,8 +778,8 @@ who kept the statement can match it to the row; the row itself holds none |
| of the data a query's literals can carry. That differs from `carlos run`, |
| which records argv, and keeps the in-region rule that SQL never reaches the |
| audit log. The SQL itself stays in the request object, in the deployment |
| -bucket, until the lifecycle rule expires it with the rest of the read |
| -(§5.8), at most two days. |
| +bucket, until the lifecycle rule expires it with the rest of the read, on |
| +the timing §5.7 states: typically two to four days, with no hard bound. |
| ## 10. Errors and copy |
| @@ -794,7 +802,7 @@ it is written into code. |
| | DBF-9 | console 429, open-read bounds | `too many database reads are open. wait a minute and try again.` | |
| | DBF-10 | console 409, record moving | `%s is moving between regions. read its database once the move has finished.` | |
| | DBF-11 | CLI stderr while waiting | `reading a copy of %s's database in region %s.` | |
| -| DBF-12 | CLI, `lost` or `interrupted` | `the server reading %s's database in region %s stopped before it finished. try again.` | |
| +| DBF-12 | CLI, the response ended without a result (the box stopped or never finished, the console restarted, the connection dropped), or `interrupted` | `the read of %s's database in region %s did not finish, so nothing was printed. try again.` | |
| | DBF-13 | CLI, a chunk or summary fails to open or add up | `the answer from region %s did not check out, so carlos db showed none of it. try again.` | |
| | DBF-14 | console 409 with the forward marker, read by an old CLI | `%s keeps its data in region %s. this version of carlos cannot read it from there. run carlos update, then try again.` | |
| | DBF-15 | CLI, `record-changed` | `%s changed while the read was waiting. try again.` | |
| @@ -838,12 +846,14 @@ DBF-17 to DBF-20 and DBF-23 to DBF-29 are left free for phase 2's pull rows. |
| readiness probe passes. `CARLOS_DB_READS=0` withdraws one box. |
| - **Console:** `CARLOS_DB_FORWARD=1` turns forwarding on. Unset, every |
| answer is byte-identical to main. The console also needs the root key. |
| -- **Store:** the `db-reads/` lifecycle rule (I2) is in place on the |
| +- **Store:** the `db-reads/` lifecycle rule (J4) is in place on the |
| deployment bucket before any console turns forwarding on. On the flagship |
| that is a change to the `deployment-buckets` module, a release tag, a ref |
| bump in carloku-infrastructure and a targeted apply. That is production |
| infrastructure, so it needs Paul's scoped approval under AGENTS.md's rule |
| - before it is applied; plan 2 prepares the plan for that approval. |
| + before it is applied; plan 2 prepares the plan for that approval. A |
| + self-hosted deployment sets the same rule (MinIO calls it ILM); without |
| + it, `db-reads/` objects are never removed. |
| - **CLI:** none. The forward marker negotiates (§9.2). |
| - **Version floor:** none beyond the capability. A box that predates it |
| publishes no capability and is never asked. |
| @@ -862,7 +872,7 @@ console), with the lifecycle rule first on each deployment's bucket. Boxes |
| publish an idle capability. The dev console turns `CARLOS_DB_FORWARD=1` and |
| the check is a forwarded query from a dev console to a dev box in another |
| region, plus the LIST-rate measurement (§5.8) and a note of real database |
| -sizes (ruling 2). Then the flagship boxes, then the flagship console. |
| +sizes (ruling 2). |
| ### 11.4 Tests |
| @@ -874,37 +884,39 @@ labelled so; they are not acceptance. |
| | T1 | A forwarded query end to end: console in region ie, agent in region uk, two MinIO buckets on one server with separate credentials. The console's credentials cannot read the uk bucket (a MinIO policy), so any console read of it fails the test. | real console server, real agent loop, real store; CI runs it with the worker reached through a test socket, labelled non-acceptance | |
| | T2 | The same on the root runner with the real `carlos-dbread.socket` and `carlos-dbread@.service` installed by the real `ensureSystemUnits`, so the socket is the worker's stdin and stdout as §8.3 says (codex 2) | acceptance gate for plan 1 | |
| | T3 | Sandbox witness on the root runner, from inside the worker: open `/data/registry.db`, `/etc/carlos/host.env` and another instance's directory; connect to `169.254.169.254:80` and to any TCP address; write outside `/var/tmp`; `ATTACH` and `VACUUM INTO` with the attach limit removed (a mutation); `load_extension`. Every one fails. The private tmp is gone after a normal end, a cancel and `kill -9` of the agent mid-read. | real units | |
| -| T4 | Forgery and replay: unsigned, wrong key, scoped key, envelope moved to another id and to another region, a forged queue entry, the same envelope offered again before and after the agent restarts (`picked` holds, codex 5), a non-owner's chunk written before and after the owner's, a chunk sealed to another key, a chunk reordered, the last chunk dropped. | real verify and seal code, real store | |
| -| T5 | Two capable boxes in one region: exactly one runs each read, across 1,000 reads with random jitter. Expiry against a late pick: whichever `picked` create wins decides, and the result agrees with it. | real store CAS | |
| +| T4 | Forgery and replay: unsigned, wrong key, scoped key, envelope moved to another id and to another region, a forged queue entry, the same envelope offered again before and after the agent restarts (`picked` holds, codex 5), a chunk sealed to another key, a chunk reordered, the last chunk dropped. And the boundary §5.6 states, as a documented negative: a writer that creates `picked`, chunks and a matching result first is accepted by the CLI (codex 18). | real verify and seal code, real store | |
| +| T5 | Two capable boxes in one region: exactly one runs each read, across 1,000 reads with random jitter. A box whose `picked` create commits on the server but answers an error reads it back and runs the read; a box whose create answers an error when another box owns it steps away (codex 16). | real store conditional create, a store wrapper that loses answers | |
| | T6 | Parity: the forwarded query and the in-region query print byte-identical stdout for a fixture database (NULL, tabs, newlines, blobs, times). `--at` stops at the same transaction as the in-region door for the same replica and time, including a write in the same second as the target. | real restore, both paths | |
| -| T7 | The invariants, through the real console and agent against a **versioned** MinIO bucket: no protocol path issues a delete or a replacing write (a store wrapper fails the test on any `Delete` or non-create `Put` under `db-reads/` outside housekeeping) (I1, codex 3, 7, 13, 14); a console restart after a finished read returns the same result (codex 13); a result fetched after the read is long finished still passes the request check (codex 14). | real console, real agent, real store | |
| +| T7 | The long request through the real path: a read that holds the request for 21 minutes passes through the real edge and the real console server (with its 5-minute `ReadTimeout`) and returns its result; the wait lines keep it alive; a CLI context of 23 minutes is the outer bound. | real edge, real console, real CLI binary | |
| | T8 | Mixed versions: the five rows of §9.2, with the forwarding-off 409 compared byte for byte against main. | real console | |
| -| T9 | Record changes between request and pick, during planning, during download and between download and `run`: moved, recreated in the same account, recreated in another account, class changed. Each ends with its code and no chunk is ever written (codex 1). | real records, a store wrapper that pauses the download | |
| +| T9 | Record changes between request and pick, during planning, during download and between download and `run`: moved, recreated in the same account, recreated in another account, class changed. Each ends with its code, the code reaches the CLI in the same response (codex 15), and no chunk is ever written (codex 1). | real records, a store wrapper that pauses the download | |
| | T10 | Approved copy digests for every row in §10, in the `approved_copy_test.go` pattern. | | |
| -| T11 | CLI buffering: a dropped or bad last chunk and a summary hash mismatch each leave stdout empty (codex 6). | real CLI binary through `carlosTestEnv` | |
| +| T11 | CLI buffering: a dropped or bad last chunk, a summary hash mismatch, and a response that ends without a result each leave stdout empty (codex 6). | real CLI binary through `carlosTestEnv` | |
| | T12 | Disk: two concurrent reads on one box whose free space fits one of them: one runs, the other ends `no-room`. An agent killed mid-read: the restarted agent takes no read and no probe while the old worker unit is active, its worker stops on EOF, and the restarted agent admits work only once the unit and its scratch are gone (codex 8). | real agent, real units, root runner | |
| | T13 | Readiness: a present socket with the service missing, masked, failing to start and speaking another protocol version each withdraws `db-read-1` within one probe and edges pass. A box running two reads keeps the capability, and its probe succeeds on the third connection (codex 9). | real units, root runner | |
| -| T14 | Console evidence: status snapshots unreadable, one snapshot missing, an edges file unreadable, an empty edges listing beside a fresh status snapshot. Each admits the read and lets expiry answer; only complete evidence (both flags) gives DBF-1 or DBF-2 (codex 10). | real `ReadLiveSnapshot` over a real store | |
| -| T15 | Follow after the instance changes: deleted, recreated and reclassified after admission. The CLI still fetches `record-changed` and can cancel; a follow call for another host, or with a forged request, is the byte-identical 404 (codex 15). | real console, real records | |
| -| T16 | A late result: a store wrapper that commits the owner's `result.json` on the server after its client has timed out, both before and after the console's `lost` create. Exactly one result exists, the follow page shows it, and the CLI acts on it (codex 4). | real store conditional create | |
| -| T17 | Lifecycle and housekeeping: the module's lifecycle rule covers `db-reads/` with current and noncurrent expiry (a module test); the housekeeping pass on an unversioned MinIO removes reads whose request is past 2 days and nothing younger. | module test, real store | |
| +| T14 | Console evidence: status snapshots unreadable, one snapshot missing, an edges file unreadable, an empty edges listing beside a fresh status snapshot. Each admits the read; only complete evidence (both flags) gives DBF-1 or DBF-2 (codex 10). | real `ReadLiveSnapshot` over a real store | |
| +| T15 | Admission (codex 17): two reads whose CLIs disconnect before pickup, and two whose CLIs disconnect after it, each free their person's and the console's count when the handler returns, without a console restart; a third read from the same person is then admitted. A console restart mid-read leaves nothing to recover. | real console | |
| +| T16 | Late writes are harmless (codex 4, 13, 14): a box that keeps writing chunks and its result after the console's request has ended (by timeout and by disconnect) changes nothing any reader sees, and no protocol path issues a delete or a replacing write (a store wrapper fails the test on any `Delete` or non-create `Put` under `db-reads/`). | real console, real agent, a versioned MinIO bucket | |
| +| T17 | The lifecycle rule: the module's rule covers `db-reads/` with current and noncurrent expiry and expired delete markers (a module test). | module test | |
| Mutation checks that must turn a test red: drop envelope verification (T4); |
| take the host or prefix from anywhere but the record and config (T1 with a |
| second tenant); remove `PrivateNetwork` (T3); skip the summary hash check |
| -(T11); skip check B (T9); make the result an ordinary `Put` (T16); delete |
| -any protocol object (T7); run the full db gate on follow (T15); drop the |
| -orphan count (T12); give probes no connection of their own (T13). |
| +(T11); skip check B (T9); add any protocol delete or replacing write (T16); |
| +release the console's counts anywhere but the handler's return (T15); skip |
| +the read-back of an ambiguous `picked` (T5); drop the orphan count (T12); |
| +give probes no connection of their own (T13); remove the read-deadline |
| +extension (T7). |
| ## 12. Provider boundary |
| - **Core:** the transport is `release.Store` objects, so it runs on MinIO and |
| any S3-compatible store. Every protocol write is a conditional create, |
| - which `release.Store.Put` already supports with an empty `ifMatch`. The |
| - box reads each planned file with a GET and writes chunks with PUTs; it |
| - never uses a server-side copy. Signing is ed25519, sealing is the standard |
| - library plus `x/crypto/hkdf`. The sandbox is systemd, which every box |
| - already needs. |
| + which `release.Store.Put` already supports with an empty `ifMatch`, on S3 |
| + and on MinIO. The box reads each planned file with a GET and writes chunks |
| + with PUTs; it never uses a server-side copy. Signing is ed25519, sealing |
| + is the standard library plus `x/crypto/hkdf`. The sandbox is systemd, |
| + which every box already needs. |
| - **One store endpoint per box (codex 11).** Every S3 client in a box process |
| takes its endpoint from the one `CARLOS_S3_ENDPOINT` |
| (`internal/s3compat/s3compat.go`), the deployment store through |
| @@ -912,15 +924,11 @@ orphan count (T12); give probes no connection of their own (T13). |
| So a box reaches its region's data bucket and the deployment bucket on the |
| same server, as it already must to wake and to move instances. This |
| design supports exactly that shape: separate buckets per region, with |
| - separate credentials, on one endpoint. Regional data on separate store |
| - servers is not something any box path supports today, and adding |
| - per-store endpoints is not this design's job (§18). |
| + separate credentials, on one endpoint (§18). |
| - **Expiry.** The lifecycle rule is S3 lifecycle configuration, which MinIO |
| - supports as ILM. A store with no lifecycle rule falls back to the |
| - console's housekeeping pass (§5.8), which reclaims space on an unversioned |
| - store. On a versioned store only a lifecycle rule reclaims space; that is |
| - a deployment requirement, stated in §11.1, not something the code can |
| - detect. |
| + supports as ILM. It is a deployment requirement (§11.1), not something the |
| + code can detect, and there is no fallback deleter: the store's rule is the |
| + only deletion (J4). |
| - **AWS adapter:** IAM is what makes the console unable to read another |
| region's bucket on the flagship. A self-hosted deployment gets the same |
| split from per-bucket credentials on its store. |
| @@ -946,10 +954,13 @@ proposes, for a ruling on #280: |
| |---|---|---| |
| | `db.go` `runQueryContext` (main: `legacy`, `rev3=J4`, `basis=root-input`, `plan14-root-reads.tsv:202`) | moved out of `db.go` | retired through the overlay's removal row, the form the H3 carlos-run removal used | |
| | `internal/dbquery` `RunContext` (the moved body, unchanged) | the CLI on a member's machine, as the member; the worker on a box, as its `DynamicUser`. Neither is root, and neither decides anything for root | `legacy`, `accepted-legacy;rev3=<ruling>;basis=root-input`, the J4 footing main gave the same body | |
| -| agent db-reads loop: the queue listing, the request read, the record reads of checks A and B, the replica listing and GETs | the agent, `carlos`, `carlos-edge.service` | `legacy` with J1 (record and route reads) and J4 (object input), as the offline move's agent units took | |
| -| worker: the scratch writes and reads, the file replica restore, `CheckDB`, the query, the connection watcher | `carlos-dbread@`, `DynamicUser` | `legacy` with J4 | |
| +| agent db-reads loop: the queue listing, the request read, the `picked` read-back, the record reads of checks A and B, the replica listing and GETs | the agent, `carlos`, `carlos-edge.service` | `legacy` with J1 (record and route reads) and J4 (object input), as the offline move's agent units took | |
| +| worker: the scratch writes and reads, the file replica restore, `CheckDBContext`, the query, the connection watcher | `carlos-dbread@`, `DynamicUser` | `legacy` with J4 | |
| | agent readiness probe and admission (`Statfs` on `/var/tmp`, `systemctl list-units`) | the agent, `carlos` | `diagnostic` for the probe; `legacy` J4 for `Statfs` and the unit listing | |
| +(The census's J1 and J4 are its own dispositions, not this spec's |
| +invariants of the same names.) |
| + |
| The overlay re-pins the source envelopes of every changed package (`.`, |
| `internal/dbquery`, `internal/dbdoor`, `internal/host`, `internal/dbreads`) |
| and updates the legacy seal under the same ruling. The census owner may rule |
| @@ -961,11 +972,10 @@ named. |
| root-written artifact and joins the Plan 14 write census as an overlay, as |
| every new unit has. |
| - **9B8 record-write census.** No record writer is added. The new `Put` |
| - sites (request, queue entry, `picked`, chunks, cancel, result) are |
| - `nonrecord`, all conditional creates. The housekeeping pass's `Delete` is |
| - `nonrecord`. `registerCLIRoutes` changes as a `key-site` for the three |
| - routes. |
| -- **Claim routes.** The three routes join |
| + sites (request, queue entry, `picked`, chunks, result, cancel) are |
| + `nonrecord`, all conditional creates. There is no `Delete` site. |
| + `registerCLIRoutes` changes as a `key-site` for the one route. |
| +- **Claim routes.** The route joins |
| `internal/console/testdata/claim-routes.txt`, refused on a host-claim |
| console like the run's. |
| - **go.mod.** No new module, so no root-envelope overlay for it. |
| @@ -976,23 +986,25 @@ named. |
| envelope, sealing, the result, limits); `internal/dbquery` (the query |
| runner and guards moved out of `db.go`, with the CLI switched to it and |
| no behaviour change); `dbdoor` planning over any `replicaReader`; the |
| - agent's db-reads loop with the identity bracket, conditional creates, |
| - admission (reservations and the orphan count) and the readiness probe; |
| - `carlos system db-worker`, its protocol and its connection watcher; the |
| - two units in `ensureSystemUnits`; the `db-read-1` capability. Gates: T2, |
| - T3, T4, T5, T9, T12, T13, T16, the parity half of T6 that needs no |
| - console. Census overlay. Nothing reaches a box unless a console asks. |
| -2. **Console and CLI for `query` and `query --at`.** The three routes and |
| - their two gates, the forward marker, the evidence rule for capability |
| - and liveness, the bounds, lazy expiry and lost, the housekeeping pass, |
| - audit, the CLI's buffered forwarded path, DBF copy, `docs/site/cli.md`, |
| - and the `deployment-buckets` lifecycle rule with the approval plan for |
| - applying it. Gates: T1, T6, T7, T8, T10, T11, T14, T15, T17, copy review, |
| - census overlay. Roll per §11.3. |
| + agent's db-reads loop with the identity bracket, conditional creates and |
| + the `picked` read-back, admission (reservations and the orphan count) and |
| + the readiness probe; `carlos system db-worker`, its protocol and its |
| + connection watcher; the two units in `ensureSystemUnits`; the |
| + `db-read-1` capability. Gates: T2, T3, T4, T5, T9, T12, T13, the parity |
| + half of T6 that needs no console. Census overlay. Nothing reaches a box |
| + unless a console asks. |
| +2. **Console and CLI for `query` and `query --at`.** The one route and its |
| + streamed response, the read-deadline extension, the forward marker, the |
| + evidence rule for capability and liveness, admission held by the |
| + request, audit, the CLI's buffered forwarded path, DBF copy, |
| + `docs/site/cli.md`, and the `deployment-buckets` lifecycle rule with the |
| + approval plan for applying it. Gates: T1, T6, T7, T8, T10, T11, T14, |
| + T15, T16, T17, copy review, census overlay. Roll per §11.3. |
| 3. **`pull` and `pull --at`.** On provisional ruling 1, re-confirmed with |
| - rev 3's storage bound (§16). Sealed LTX files under the same invariants, |
| - a raw chunk route for 4 MiB chunks, local restore and `CheckDB` in the |
| - CLI, and the pull copy rows. |
| + the storage statement in §16. Sealed LTX files in the same single |
| + request, local restore and `CheckDBContext` in the CLI, and the pull copy |
| + rows. A 2 GiB pull in one request is the case most likely to need a |
| + longer request bound; plan 3 measures it before choosing. |
| After phase 3, in-region reads move to the box path too (provisional ruling |
| 5). That is a fourth plan, written once phase 3 has run on the flagship. It |
| @@ -1015,15 +1027,13 @@ through it unchanged. |
| 3. **Forgery and replay.** An unsigned or wrongly scoped request accepted, |
| domain separation from other root envelopes missing, the region not |
| bound, or a queue entry trusted without its request. |
| -4. **An invariant broken by a convenience.** Any protocol write that is not |
| - a conditional create, any protocol delete, a result accepted from |
| - someone other than the owner, or a follow route that reads the current |
| - record. Each one reopens a finding that I1 to I7 closed. |
| -5. **Drift between the two paths, or admission that lies.** The forwarded |
| - `--at` stopping at a different transaction than the in-region door, |
| - output formatting that differs between CLI and box versions, a |
| - capability published without a passing probe, or reservations that |
| - count a worker as gone before its unit has stopped. |
| +4. **Something outliving the request.** Any state, route, timer or write |
| + on the console side that survives the request's end, any protocol delete |
| + or replacing write, or an admission count released anywhere but the |
| + handler's return. Each one reopens a finding J1 to J5 closed. |
| +5. **The long request.** A timeout on the real path that the spec missed |
| + (a proxy, a client, a keep-alive), wait lines that stall behind a slow |
| + reader, or a 2 GiB pull that does not fit the request bound. |
| ## 16. Open questions: provisional rulings awaiting Paul |
| @@ -1033,19 +1043,19 @@ confirms or changes it. |
| 1. **Pull through the deployment bucket.** Phase 2's pull may pass |
| ciphertext through the deployment bucket (eu-west-1 on the flagship). |
| - **The bound in that ruling no longer holds and must be re-confirmed |
| - before plan 3.** The ruling said "at most 64 MiB at a time for at most 1 |
| - hour". Round 2 (codex 7) showed that bound was never real: the bucket is |
| - versioned, so deleting chunks left them as noncurrent versions. Rev 3 |
| - deletes nothing, and the true bound is: a pull's whole ciphertext (up to |
| - its size cap) rests in the bucket until the lifecycle rule removes it, |
| - at most two days. The alternative that keeps a short window is |
| - version-aware deletes, which need `s3:DeleteObjectVersion` on the |
| - deployment bucket for the console's role, a new IAM grant and the |
| - deletes rev 3 removed. Phase 1 is unaffected: a query's ciphertext is at |
| - most 64 MiB of rows. **Paul's call** in either case: if UK data |
| - residency is a legal requirement rather than a technical one, whether |
| - encrypted data resting in an Irish bucket is acceptable is for him. |
| + **The bound in that ruling does not hold and must be re-confirmed before |
| + plan 3.** The ruling said "at most 64 MiB at a time for at most 1 hour". |
| + The deployment bucket is versioned and the protocol deletes nothing, so |
| + the true statement is: **a pull's whole ciphertext, up to its size cap, |
| + lives in the deployment bucket until lifecycle expiry, typically two to |
| + four days, with no guaranteed upper bound** (§5.7). A short or hard bound |
| + would need version-aware deletes, with `s3:DeleteObjectVersion` on the |
| + deployment bucket for the console's role: a new IAM grant and a deleter |
| + this design removed on purpose. Phase 1 is the same statement at a |
| + smaller size: a query's ciphertext is at most 64 MiB of rows. **Paul's |
| + call** in either case: if UK data residency is a legal requirement |
| + rather than a technical one, whether encrypted UK data resting for days |
| + in an Irish bucket is acceptable is for him. |
| 2. **Limits stand.** 2 GiB restored, 5 minutes, 64 MB of rows, two reads per |
| box. Real database sizes on the flagship are still unmeasured; plan 2's |
| dev check reports them. |
| @@ -1059,44 +1069,44 @@ confirms or changes it. |
| ## 17. What this does not do |
| - No console per region and no console-to-box connection. |
| +- No resumable reads: a read that loses its connection is run again. |
| - No change to in-region reads (until the fourth plan), the claim-mode |
| door, seeds `--from`, fleet or pooled instances. |
| - No IAM change. No new read authority for any role. The one infrastructure |
| change is a lifecycle rule on the deployment bucket. |
| - No per-store S3 endpoints. |
| +- No authenticated authorship of answers (§5.6). |
| - No `carlos db shell` and no browser view. |
| ## 18. Declined review findings |
| **Round 1** (`CODEX-FINDINGS-DB-REGION-FORWARD.md`). Every citation was |
| checked against `main` at `5f341f33` and holds. Findings 2 to 10 and 12 are |
| -accepted in full. Two are accepted in part, and round 2 agreed with both |
| -declines: |
| +accepted in full. Two are accepted in part, and rounds 2 and 3 agreed with |
| +both declines: |
| - **Codex 1, the "stronger guarantee" (pin object versions to the checked |
| - incarnation): declined.** The identity bracket (§8.2) is accepted and |
| - closes the finding: no byte is read outside two matching checks, and no |
| - output exists before the second. Pinning versions needs |
| - `s3:GetObjectVersion` and `s3:ListBucketVersions` on the box's data |
| - bucket, which the flagship's uk, us and au boxes do not have at module |
| - ref v0.21.0, and a versioned reader that today lives only behind the |
| - claim boundary. That is a new mechanism for a case the bracket already |
| - removes. Claim activation, which brings versioned reads, can revisit it. |
| + incarnation): declined.** The identity bracket (§8.2) closes the finding. |
| + Pinning versions needs `s3:GetObjectVersion` and `s3:ListBucketVersions` |
| + on the box's data bucket, which the flagship's uk, us and au boxes do not |
| + have at module ref v0.21.0, and a versioned reader that today lives only |
| + behind the claim boundary. Claim activation can revisit it. |
| - **Codex 11, independently configured deployment and regional-data |
| - endpoints: declined.** No box path (wake, moves, this one) supports |
| - regional data on a separate store server today, because every S3 client |
| - in the process reads one `CARLOS_S3_ENDPOINT`. The spec states the |
| - supported shape (one endpoint, separate buckets and credentials) and |
| - tests it (T1). Per-store endpoints, if wanted, are a platform-wide change |
| - with its own design. |
| - |
| -**Round 2** (`CODEX-FINDINGS-DB-REGION-FORWARD-R2.md`). Every citation was |
| -checked and holds. Nothing is declined. Findings 4, 7, 13 and 14 are closed |
| -by the lifecycle invariants (§5.2), not by patching each case, and the |
| -round-2 fix for finding 3 (separate `ack` and `done`) is superseded: with |
| -no deletes there is nothing for an acknowledgement to trigger, so both |
| -routes are gone. Findings 8, 9, 10 and 15 are fixed directly (§5.7, §8.3, |
| -§7.1, §9.1). |
| - |
| -One consequence is not a decline but needs a decision: closing finding 7 |
| -honestly invalidates the bound in provisional ruling 1 (§16). |
| + endpoints: declined.** No box path supports regional data on a separate |
| + store server today, because every S3 client in the process reads one |
| + `CARLOS_S3_ENDPOINT`. Per-store endpoints, if wanted, are a platform-wide |
| + change with its own design. |
| + |
| +**Round 2** (`CODEX-FINDINGS-DB-REGION-FORWARD-R2.md`). Nothing declined. |
| + |
| +**Round 3** (`CODEX-FINDINGS-DB-REGION-FORWARD-R3.md`). Every citation was |
| +checked and holds, including that the worker must call |
| +`lease.CheckDBContext` (`internal/lease/check.go:44`), not `CheckDB`; §8.3 |
| +and §13 now say so. Nothing is declined. Rev 4 does not patch findings 13, |
| +14, 16 and 17 or the partly-holding invariants one by one: it removes the |
| +durable read they all came from (§5.2), and §5.3 maps every finding to the |
| +invariant that closes it or to the synchronous design that removes it. |
| +Finding 7's fix is the honest timing statement, not a new mechanism; a hard |
| +bound would need version-aware deletes, which is noted for Paul in §16 |
| +rather than built. Finding 18 is fixed by withdrawing the claim and stating |
| +the trust boundary (§5.6), not by adding per-box signing. |