amadan
db-region-forward spec rev 2: answer Astra round 1
Ten findings accepted in full, two in part (version pinning and per-store endpoints declined in §18). The box now brackets every replica byte with two record-identity checks and writes no output before the second; the heartbeat-based lost rule is replaced by a box deadline and a write fence; picked is kept as a tombstone past the acceptance window; progress acks and completion are separate calls; acknowledged chunks are deleted; the CLI buffers and verifies before printing; disk is reserved in one table; the capability rests on a readiness probe; the console refuses up front only on complete evidence; census rows use the scanner's four classes. Folds in the coordinator's provisional answers to the open questions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+375
−176
docs/superpowers/specs/2026-10-22-db-region-forward-design.md+375 −176
diff --git a/docs/superpowers/specs/2026-10-22-db-region-forward-design.md b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md| index e38461f31..f6f957109 100644 |
| --- a/docs/superpowers/specs/2026-10-22-db-region-forward-design.md |
| +++ b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md |
| @@ -1,8 +1,13 @@ |
| # `carlos db` for instances whose data lives in another region |
| -**Status:** rev 1, 2026-10-22. Design only, nothing built. Written for Paul, |
| +**Status:** rev 2, 2026-10-22. Design only, nothing built. Written for Paul, |
| who approved the direction (§2). Review brief: |
| -`CODEX-REVIEW-BRIEF-DB-REGION-FORWARD.md`. |
| +`CODEX-REVIEW-BRIEF-DB-REGION-FORWARD.md`. Rev 1 was `b35b3ab1`. Rev 2 |
| +answers Astra's round 1 (`CODEX-FINDINGS-DB-REGION-FORWARD.md`, 12 |
| +findings: 10 accepted in full, 2 accepted in part); each change is marked |
| +**(codex N)** where it lands, and §18 lists what was declined. Rev 2 also |
| +folds in the coordinator's provisional answers to the open questions, which |
| +await Paul (§16). |
| **Companion specs:** `2026-09-02-carlos-db-design.md` (the db door, cited |
| below as "db spec"), `2026-10-08-carlos-run-design.md` (the run transport, |
| @@ -46,7 +51,7 @@ working around them: |
| bucket (§5.1). This spec puts only encrypted bytes there, sealed to a key |
| that exists only in the member's CLI (§5.5). In phase 1 that is only a |
| query's rows. Phase 2's pull would send the planned replica files, which |
| - are a pull's result, and waits on Paul's ruling (Q1). |
| + are a pull's result, under provisional ruling 1 (§16), which awaits Paul. |
| One earlier ruling holds unchanged: the run spec's "the console still never |
| connects to a box". The transport is the run's. |
| @@ -113,14 +118,14 @@ connects to a box". The transport is the run's. |
| | # | Decision | Section | |
| |---|---|---| |
| -| D1 | Phase 1 forwards `query` and `query --at`. `pull` and `pull --at` forward in phase 2, after Paul rules on Q1. The claim-mode door and the seeds `--from` door stay refused. | §6 | |
| +| D1 | Phase 1 forwards `query` and `query --at`. `pull` and `pull --at` forward in phase 2, under provisional ruling 1. The claim-mode door and the seeds `--from` door stay refused. | §6 | |
| | D2 | Any live box in the record's region that publishes `db-read-1` and is not draining may serve. The first to create the read's `picked` marker serves it. The instance's lease holder gets no preference. | §7 | |
| | D3 | Transport is the run's: objects in the deployment bucket, a serial bump, chunks paged by the console. The request is a root-signed envelope. No record write. | §5 | |
| | D4 | Results are sealed to an ephemeral key the CLI makes for each read. The console and the deployment bucket only ever hold the result as ciphertext. The console does see the SQL, as the envelope's signer. | §5.5 | |
| | D5 | The box splits the work. The agent (`carlos`) verifies, resolves, plans and streams objects. Everything that decodes tenant bytes (LTX, SQLite, the member's SQL) runs in a new sandboxed unit, `carlos-dbread@.service`, as a `DynamicUser` with no network. | §8 | |
| | D6 | The worker gets the planned LTX files and nothing else. Its scratch is its own private `/var/tmp`, which systemd removes when the unit stops. | §8.3 | |
| | D7 | Rollout: boxes publish `db-read-1`; the console forwards only with `CARLOS_DB_FORWARD=1` and a capable region. Dormant by default. | §11 | |
| -| D8 | Three plans. | §14 | |
| +| D8 | Three plans here. A fourth, after phase 3, moves in-region reads to the box path (provisional ruling 5). | §14 | |
| ## 5. Transport and authentication |
| @@ -153,12 +158,17 @@ minted the way `runs.MintID` mints run ids, with its own `ValidID`. |
| | Key | Writer | Write | Purpose | |
| |---|---|---|---| |
| | `control/db-reads/<region>/<id>.json` | console | create-only | The request: a root envelope (§5.3). Boxes of `<region>` discover reads by listing this prefix. | |
| -| `apps/<acct>/<app>/db-reads/<id>/picked` | box, or the console's expiry | create-only | Who took the read. Exactly one writer wins. | |
| -| `apps/<acct>/<app>/db-reads/<id>/beat` | the picking box | replace | Heartbeat every 15 s: phase (`planning`, `restoring`, `querying`, `sending`) and bytes so far. | |
| -| `apps/<acct>/<app>/db-reads/<id>/out/<seq>` | the picking box | create-only | One sealed chunk of output (§5.5). | |
| -| `apps/<acct>/<app>/db-reads/<id>/ack` | console | replace | The last sequence number the CLI has read. | |
| +| `apps/<acct>/<app>/db-reads/<id>/picked` | box, or the console's expiry | create-only | Who took the read, and the box's write deadline. Exactly one writer wins. It is also the read's tombstone and outlives the acceptance window (§5.9). | |
| +| `apps/<acct>/<app>/db-reads/<id>/out/<seq>` | the picking box | create-only | One sealed chunk of output (§5.5). Deleted once the CLI has acknowledged it. | |
| +| `apps/<acct>/<app>/db-reads/<id>/ack` | console | replace | The CLI's progress: the highest sequence number it holds. Never moves backwards. | |
| | `apps/<acct>/<app>/db-reads/<id>/cancel` | console | create-only | The CLI asked to stop. | |
| -| `apps/<acct>/<app>/db-reads/<id>/result.json` | the picking box, or the console for expired and lost reads | create-only | How the read ended (§5.6). Written after the last chunk. | |
| +| `apps/<acct>/<app>/db-reads/<id>/result.json` | the picking box, or the console for expired and lost reads | create-only | How the read ended (§5.6). The box writes it after its last chunk write has returned. | |
| + |
| +There is no heartbeat object. Rev 1 had one and declared a read lost after |
| +90 s without it. That let a slow box keep writing after its read was |
| +declared lost (codex 4). Rev 2 replaces it with one rule: a box's writes |
| +stop at a deadline it states in `picked`, and nothing is declared lost |
| +until that deadline has passed (§5.8). |
| The job objects live under the app's subtree for the run's reason: app trash |
| and purge sweep them, and the owner gate maps onto them. The request lives |
| @@ -226,28 +236,37 @@ writer to the 9B8 census for nothing. |
| bounds (§5.7), mints the id, writes the envelope create-only, bumps the |
| serial, writes the audit row (§9.3) and answers 202 `{id, region}`. |
| 4. Each capable box in the region lists `control/db-reads/<region>/` when the |
| - serial moves, and every 30 s if it does not. For each envelope it has |
| - not seen, it verifies, checks it has a free worker slot and is above the disk floor (§5.7), |
| - waits a random 0 to 500 ms, and creates `picked`. A box that loses the |
| - create steps away. |
| -5. The winner reads the record fresh and runs §8.2's checks, then plans, |
| - streams the planned files into a worker, and seals the worker's output |
| - into chunks. It writes `beat` every 15 s, reads `cancel` and `ack` at |
| - each beat and whenever its unacked window is full, and writes |
| - `result.json` after the last chunk. |
| + serial moves, and every 30 s if it does not. For each envelope it |
| + verifies, checks that it has a free worker slot and can reserve the disk |
| + floor (§5.7), waits a random 0 to 500 ms, and creates `picked` with its |
| + box name, boot id and deadline. A box that loses the create steps away. |
| +5. The winner runs §8.2's identity bracket around planning and download, |
| + then runs the query in a worker and seals the output into chunks. It |
| + reads `cancel` and `ack` every 5 s and whenever its unacked window is |
| + full, and writes `result.json` after its last chunk write has returned. |
| 6. The CLI polls `GET .../db/{host}/reads/{id}?after=<seq>` about once a |
| second. The page is the run's `Page` shape: state, chunks (base64 of the |
| sealed bytes) and the result when present, under 512 KiB. The CLI opens |
| - each chunk, writes rows to stdout, and posts |
| - `POST .../reads/{id}/ack` with its last sequence number. |
| -7. On the result, the CLI checks the sealed summary (§5.6), prints today's |
| - status line to stderr, posts its final ack, and exits. The console |
| - deletes the job objects and the request when it records that ack. |
| - |
| -The status line (`dbQueryStatus`) needs the copy's transaction and time, |
| -which arrive in the sealed summary after the rows. So the forwarded CLI |
| -prints DBF-11 while it waits and today's status line at the end, both on |
| -stderr. stdout carries the same rows in the same format as an in-region |
| + each chunk into a local buffer and posts a progress ack, |
| + `POST .../reads/{id}/ack` with `{last_seq}`. It writes nothing to stdout |
| + yet **(codex 6)**. |
| +7. On the result, the CLI opens the sealed summary, checks the buffered |
| + bytes against its count and SHA-256, and only then writes the rows to |
| + stdout and today's status line to stderr. It then posts a completion, |
| + `POST .../reads/{id}/done` with `{result_sha256}`, the SHA-256 of the |
| + `result.json` bytes it verified **(codex 3)**, and exits. |
| + |
| +A progress ack and a completion are different calls with different bodies, |
| +so a late progress ack can never be read as the end. The console accepts a |
| +completion only when `result.json` exists and hashes to `result_sha256`. |
| + |
| +The buffer is bounded by the output cap (64 MiB), which a terminal-run |
| +command can hold in memory. Progress acks keep the box's window moving |
| +while the CLI buffers. If the summary does not check out, the CLI has |
| +written nothing, so DBF-13's "showed none of it" is true. The status line |
| +(`dbQueryStatus`) needs the copy's transaction and time from the summary, |
| +so the CLI prints DBF-11 while it waits and today's status line with the |
| +rows. stdout carries the same rows in the same format as an in-region |
| query. |
| ### 5.5 Sealing |
| @@ -306,13 +325,13 @@ The CLI maps codes to today's sentences where one exists |
| | Limit | Value | Enforced by | |
| |---|---|---| |
| | Pickup window | 60 s from `issued_at` | box (refuses older envelopes), console expiry | |
| -| Heartbeat | every 15 s; lost after 90 s without one | box, console sweep | |
| | Planning and restore | 10 min | worker unit and agent context | |
| | Query | 5 min wall clock | worker | |
| -| Whole read | 20 min | worker `RuntimeMaxSec`, agent context | |
| +| Box deadline | 20 min from `picked`, stated in it | agent context on every read and write, worker `RuntimeMaxSec` | |
| +| One write | 60 s timeout on every PUT the box makes | agent | |
| | Restored size | 2 GiB by default, `CARLOS_DB_READ_MAX_BYTES` on the box | agent, from the plan, before any byte moves | |
| -| Disk floor | before `picked`: at least 5 GiB and 10% of the filesystem holding `/var/tmp` free. After planning: the floor must still hold once the plan's LTX bytes and restored size are counted, or the read ends `no-room` | agent | |
| -| Output | 64 MiB (`runs.MaxOutputBytes`, written "64 MB" in copy as the run's docs do); the read ends `output-cap` and the CLI says so | agent | |
| +| Disk | a reservation per read (below) | agent | |
| +| Output | 64 MiB (`runs.MaxOutputBytes`, written "64 MB" in copy as the run's docs do); the read ends `output-cap` and the CLI says so | agent; the CLI's buffer is bounded by the same number | |
| | Chunk | 256 KiB or 1 s, whichever comes first (`runs.ChunkBytes`, `runs.ChunkEvery`) | agent | |
| | Unacked window | 16 chunks (4 MiB); the box pauses until the CLI's ack moves | agent | |
| | Worker slots per box | 2 (`MaxConnections=2` on the socket) | systemd, agent | |
| @@ -320,32 +339,72 @@ The CLI maps codes to today's sentences where one exists |
| | Open reads per console | 16 | console, 429 with `Retry-After` | |
| | Request budget | the db door's existing 600 per token per minute | console | |
| -The unacked window bounds both the box's memory and how much ciphertext a |
| -read can leave in the deployment bucket at once. |
| - |
| -### 5.8 Cleanup and cost |
| - |
| -- **On the CLI's final ack, or its cancel once the box's result has |
| - landed,** the console deletes `apps/<acct>/<app>/db-reads/<id>/` and the |
| - request. A CLI that went away leaves them to the sweep below. |
| -- **The console sweep** joins `runsweep.go`'s loop (30 s): an unpicked read |
| - past its window gets the console's own `picked` (create-only, so it races |
| - safely with a late box) and an `expired` result; a picked read with no |
| - beat for 90 s gets a `lost` result. Reads whose result is older than 1 h, |
| - and request objects older than 1 h, are deleted every 10 minutes. |
| - `KeepFor` is 1 h, not the run's 24 h, because these objects are copies of |
| - data, even if encrypted. |
| +**Disk reservations (codex 8).** The agent is the only process that starts |
| +workers, so it keeps one reservation table under one mutex. Before `picked` |
| +it reserves the floor (5 GiB); after planning it grows that reservation to |
| +the plan's LTX bytes plus its restored size. Each step succeeds only if the |
| +filesystem holding `/var/tmp` keeps at least 5 GiB and 10% free after |
| +subtracting every other open reservation; otherwise the read ends `no-room` |
| +and the reservation is released. Reservations are released on every exit |
| +of the read's goroutine. The worker refuses to write past the byte counts |
| +its header names, so a reservation is an upper bound, not an estimate. A |
| +restarted agent starts with an empty table and takes no read until no |
| +`carlos-dbread@` unit is active (`systemctl list-units`, readable without |
| +privilege) or 2 minutes have passed, so the old workers' scratch is gone |
| +before anything is counted again. |
| + |
| +**Stored ciphertext (codex 7).** On each progress ack the console deletes the |
| +chunks at or below `last_seq` (deletes are idempotent, and an ack lower than |
| +the stored one is ignored). With the box paused at 16 unacknowledged |
| +chunks, a read never holds more than 4 MiB of ciphertext in the deployment |
| +bucket, plus one chunk in flight. Phase 2's pull uses the same rule with |
| +4 MiB chunks, so 64 MiB. |
| + |
| +### 5.8 Ending, cleanup and cost |
| + |
| +**The write fence (codex 4).** `picked` names the box's deadline (20 minutes |
| +after the pick, by the box's clock). Every read and write the box makes for |
| +that read runs on a context that ends at the deadline, and every PUT has a |
| +60 s timeout, so no box write can land after the deadline plus 60 s. The |
| +console treats deadline plus 2 minutes (one minute more for clock skew) as |
| +the **fence**: after it, the box can write nothing more. |
| + |
| +- **Done.** The box writes `result.json` only after its last chunk write has |
| + returned. On the CLI's completion (§5.4 step 7) the console deletes the |
| + remaining chunks, `ack`, `cancel` and `result.json`. |
| +- **Cancelled.** The console writes `cancel`. The box stops the worker and |
| + writes a `cancelled` result, which ends the read like any other result. |
| +- **Expired.** An unpicked read past its window gets the console's own |
| + `picked` (create-only, so it races safely with a late box; whoever creates |
| + it owns the read) and an `expired` result. No box ever owns it, so no box |
| + writes. |
| +- **Lost.** A picked read with no result at the fence gets a `lost` result |
| + from the console. The box can no longer write, so nothing lands after it. |
| + A person whose box died waits up to 22 minutes for DBF-12. That is the |
| + price of never answering before the box is fenced. The CLI says DBF-11 |
| + while it waits and can be stopped with Ctrl-C. |
| +- **The sweep** joins `runsweep.go`'s loop (30 s) for expiry and lost. Every |
| + 10 minutes it deletes job objects whose result is older than 1 hour, or |
| + whose fence passed more than 1 hour ago. `KeepFor` is 1 hour, not the |
| + run's 24, because these objects are copies of data, even if encrypted. |
| +- **The tombstone (codex 5).** `picked` and the request object are never |
| + deleted with the rest. The sweep deletes them only once both hold: the |
| + read has ended (a result exists, or its fence has passed), and |
| + `issued_at` is more than 5 minutes old, past the 60 s acceptance window |
| + and its 60 s of skew, with margin. Until then, `picked` exists, so no box, including the |
| + same box after a restart, can pick that envelope again. |
| - **On the box,** the agent holds no scratch at all. The worker's scratch is |
| removed by systemd when its unit stops, whatever stopped it (§8.3). An |
| agent that restarts mid-read loses the connection, the worker exits, and |
| - the console's lost rule ends the read. |
| + the console's lost rule ends the read at the fence. |
| - **A bucket lifecycle rule** on `apps/*/db-reads/` and `control/db-reads/` |
| (1 day) is an optional backstop for the AWS adapter. The sweep is the |
| real cleanup, so a store without lifecycle rules loses nothing. |
| - **Cost.** Each capable box does one LIST of an almost always empty prefix |
| - per serial move and every 30 s. A read adds a few PUTs, one GET per beat |
| - and one per chunk. The October bill was LIST and GET polling, so plan 1 |
| - measures the LIST rate on dev before the flagship roll and reports it. |
| + per serial move and every 30 s. A read adds a few PUTs, a GET of `cancel` |
| + and `ack` every 5 s, and one GET and one DELETE per chunk. The October |
| + bill was LIST and GET polling, so plan 1 measures the LIST rate on dev |
| + before the flagship roll and reports it. |
| ### 5.9 Replay and forgery |
| @@ -353,7 +412,7 @@ read can leave in the deployment bucket at once. |
| |---|---| |
| | A box writes its own request | No root signature: refused and logged once. | |
| | An envelope copied to another id or region key | The id and region are inside the signature and must match the key. | |
| -| The same envelope rewritten after cleanup | `issued_at` older than 60 s is refused. Inside the window, the box keeps a seen set of ids for 5 minutes and refuses a second pick. Even if one ran, its output is sealed to the original CLI's key. | |
| +| The same envelope picked again | `picked` is a tombstone that outlives the 60 s window plus skew (§5.8), so a second create fails on every box, before or after a restart **(codex 5)**. Past the window the envelope is refused on age. Rev 1's in-memory seen set is gone. Even if one ran, its output is sealed to the original CLI's key. | |
| | A box serves a read for another region | It refuses any envelope whose region is not its own `CARLOS_REGION`, and IAM refuses the bucket anyway. | |
| | A request that names a bucket or prefix | There is no such field. The box derives the prefix (§8.2). | |
| @@ -363,7 +422,7 @@ read can leave in the deployment bucket at once. |
| |---|---|---| |
| | `query` | 1 | This is the regression the customer hit. The result is rows, usually small, and only rows leave the region. | |
| | `query --at` | 1 | Same path with a target time. The box lists the replica the way the console's door does today (`listReplicaLevel`, `LastModified` as `CreatedAt`), so the copy stops at the same transaction an in-region read would. Plan 1 tests that parity. | |
| -| `pull` | 2 | The result is the whole database. It needs Paul's ruling on Q1 (ciphertext of a whole copy passing through the deployment bucket's region), and a measured size cap. In phase 2 the box ships the planned LTX files, sealed and windowed, and the CLI restores and checks them locally with today's code. The box never restores for a pull. | |
| +| `pull` | 2 | The result is the whole database. It rests on provisional ruling 1 (ciphertext of a whole copy passing through the deployment bucket's region, at most 64 MiB at a time), and a measured size cap. In phase 2 the box ships the planned LTX files, sealed and windowed, and the CLI restores and checks them locally with today's code. The box never restores for a pull. | |
| | `pull --at` | 2 | As `pull`, with the target time. | |
| | Claim-mode door (`apidbclaim.go`) | not planned | Claims are not active anywhere. The claim door reads exact object versions through `hostops`'s versioned boundary and signed file handles, and lists no times. A box-side version of that belongs to claim activation (#250), which must add times anyway. A host-claim console keeps today's residency refusal, as it refuses `carlos run`. | |
| | Seeds `--from` (`resolveDBSource`) | not planned | It shares the source checks but restores into a new instance, a different flow. Cross-region seeding is its own design. | |
| @@ -377,24 +436,41 @@ cross-region instance. |
| ### 7.1 Capability and selection |
| -A box publishes `db-read-1` in its edges file only when all of these hold, |
| -as its own armed worker's answer, the way `offline-move-1` is published: |
| +A box publishes `db-read-1` in its edges file only when the agent's armed |
| +db-reads loop says so, the way `offline-move-1` comes from the armed moves |
| +worker (`armedOfflineView`). The loop answers yes only when all of these |
| +hold: |
| - hibernation is configured (`CARLOS_HIBERNATE_BUCKET` set); |
| - `CARLOS_REGION` is set; |
| - `CARLOS_RELEASE_PUBKEY` is set, so it can verify envelopes; |
| -- the worker socket `/run/carlos-dbread/worker.sock` exists and is a socket; |
| - `CARLOS_DB_READS` is not `0` (an operator off switch for one box); |
| -- `CARLOS_HOST_CLAIM` is not set. |
| - |
| -The console reads `edgesfile.ReadLiveSnapshot` before admitting a read: |
| - |
| -- No box in the record's region publishes `db-read-1`, or every one that |
| - does is draining: DBF-1, no request written. |
| -- Some do, but none has a fresh status snapshot: DBF-2, no request written. |
| -- Snapshots cannot be loaded: the request is written anyway and the |
| - pickup window decides. This follows the explicit placement door's |
| - generous read; a person is waiting and the expiry answers in 60 s. |
| +- `CARLOS_HOST_CLAIM` is not set; |
| +- the loop is running, and its last **readiness probe** passed **(codex 9)**. |
| + |
| +The readiness probe connects to the worker socket and runs the worker |
| +protocol's `hello` exchange, which starts a real `carlos-dbread@` instance |
| +and gets back the worker's protocol version and its view of its own |
| +sandbox (no network interface but loopback, `/data` empty, `/etc/carlos` |
| +unreadable). The loop probes at start and every 5 minutes, and after any |
| +read whose worker failed to start. A failed probe withdraws the capability |
| +on the next edges-file pass. A socket file whose service is missing, |
| +masked, broken or the wrong version therefore never advertises anything. |
| + |
| +The console reads `edgesfile.ReadLiveSnapshot` before admitting a read and |
| +refuses up front only on complete evidence **(codex 10)**. That function |
| +returns a nil error when status snapshots cannot be loaded and reports the |
| +gap through `Complete` and `BoxesComplete` instead, so the rule reads those |
| +fields, never the error alone: |
| + |
| +1. `Complete` is true and no file in the record's region publishes |
| + `db-read-1` on a non-draining box: DBF-1, no request written. |
| +2. `Complete` and `BoxesComplete` are both true, capable non-draining boxes |
| + exist, and none is `Fresh`: DBF-2, no request written. |
| +3. Anything else (an unreadable edges file, an unreadable or missing status |
| + snapshot, a fresh snapshot without an edges file): the request is |
| + written and the pickup window decides. A person is waiting, and the |
| + expiry answers within 60 s. |
| Any capable, live, non-draining box in the region may serve. There is no |
| assignment by the console. Region HA (plan 2b, dormant) can put several |
| @@ -418,7 +494,7 @@ would also tie the read to the instance's lifecycle, which is exactly what |
| | No capable box in the region | DBF-1 at once | |
| | Capable boxes, none live | DBF-2 at once | |
| | Live but nobody picks in 60 s (all busy, all below the disk floor, agent stuck) | DBF-3 after the window | |
| -| The picking box dies or restarts mid-read | DBF-12 after 90 s without a beat | |
| +| The picking box dies or restarts mid-read | DBF-12 at the fence, up to 22 minutes after the pick (§5.8) | |
| ## 8. Read-only and isolation on the box |
| @@ -426,7 +502,7 @@ would also tie the read to the instance's lifecycle, which is exactly what |
| | Half | Runs as | Does | Never does | |
| |---|---|---|---| |
| -| Agent (`carlos agent`, `carlos-edge.service`) | `carlos` | lists requests; verifies envelopes; creates `picked`; reads the record; derives the prefix; lists the replica and plans the restore from the listing; reads the last planned file's 100-byte header for the size check; streams planned files from the region bucket into the worker; seals and writes output; writes beat and result | opens SQLite, decodes LTX pages, runs SQL, writes files on disk, reads anything under `/data/instances`, takes or reads the lease | |
| +| Agent (`carlos agent`, `carlos-edge.service`) | `carlos` | lists requests; verifies envelopes; creates `picked`; reads the record; derives the prefix; lists the replica and plans the restore from the listing; reads the last planned file's 100-byte header for the size check; streams planned files from the region bucket into the worker; re-checks the record before the query runs; seals and writes output; writes the result | opens SQLite, decodes LTX pages, runs SQL, writes files on disk, reads anything under `/data/instances`, takes or reads the lease | |
| | Worker (`carlos system db-worker`, `carlos-dbread@.service`) | a `DynamicUser` | receives the plan and the files over its socket; writes them into its private `/var/tmp` as a litestream file replica; restores to the agent's target transaction; runs `lease.CheckDB`; runs the one statement with today's guards; streams rows back | anything with a network, the box's files, the bucket, other tenants' files | |
| The rule is: everything that decodes page data or SQL runs in the worker. |
| @@ -437,10 +513,17 @@ fixed 100-byte LTX header, for the size check, with the bounds |
| `dbdoor.PlanRestore` already applies (a valid page size, a non-zero commit, |
| TXIDs that match the listing). |
| -### 8.2 What the agent checks before the worker starts |
| +### 8.2 The identity bracket |
| + |
| +The replica prefix is keyed by host alone (`activator.HibernatePrefix`), so |
| +a host deleted and recreated, in the same account or another, reuses it. |
| +Checking the record once before the read would let a request admitted for |
| +the old instance read the new one's replica **(codex 1)**. So the agent |
| +brackets every byte it reads with two identical checks, and no output |
| +exists until the second one passes. |
| -After winning `picked`, the agent reads the record fresh from the bucket and |
| -refuses (result `refused`) unless: |
| +**The check.** The agent reads the record fresh from the bucket and refuses |
| +(result `refused`) unless: |
| - the record exists and its incarnation is the envelope's (`record-changed`); |
| - the class rail passes (`dbClassRefusal`, moved where both console and box |
| @@ -450,6 +533,30 @@ refuses (result `refused`) unless: |
| - `rec.Move` is nil (`moving`); |
| - `rec.Host` is the envelope's host and passes `provision.ValidHostShape`. |
| +**The order.** |
| + |
| +1. Check A, then plan from the listing. |
| +2. Stream every planned file into the worker. The worker restores and runs |
| + `lease.CheckDB`, then answers `ready` with the transaction it holds. It |
| + runs no SQL yet. |
| +3. Check B, the same check again, after the last byte has been read from |
| + the bucket. |
| +4. Only if check B passes does the agent tell the worker `run`. The first |
| + output chunk can exist only after this point. |
| + |
| +A failed check B ends the read `record-changed`, stops the worker and writes |
| +no chunk. A retried plan (today's "changed while it was being read" retry) |
| +runs both checks again around its own download. |
| + |
| +**Why two checks are enough.** The incarnation is |
| +`hostclaim.Incarnation(account, app, host, CreatedAt)`. A deleted and |
| +recreated host, including one recreated in another account, gets a new |
| +incarnation, and an incarnation never comes back once its record is gone. |
| +So a record that shows the same incarnation at A and at B was the same |
| +instance for the whole download, and the files read in between were that |
| +instance's. Pinning object versions would add nothing that this does not |
| +already give in non-claim mode (§18). |
| + |
| The replica prefix is `activator.HibernatePrefix(<the box's own |
| CARLOS_HIBERNATE_PREFIX>, rec.Host)` in the box's own |
| `CARLOS_HIBERNATE_BUCKET`, read through `lease.ReadReplicaClient` and held |
| @@ -470,6 +577,13 @@ calls the leaser, and never wakes the instance. |
| `carlos-dbread@.service`, one instance per connection, starting from the |
| claim-bound template's hardening (§3 fact 8) with these differences: |
| +- `StandardInput=socket` and `StandardOutput=socket`, so the accepted |
| + connection is the worker's stdin and stdout, and `StandardError=journal` |
| + for its own diagnostics **(codex 2)**. With `Accept=yes` systemd passes |
| + the connection as a file descriptor and does not put it on stdin unless |
| + told to; the claim-bound template's `StandardOutput=file:` line is not |
| + copied. All protocol traffic, both ways, is on that one connection; |
| + |
| - `DynamicUser=yes` instead of `User=carlos`, so the worker owns no file on |
| the box and can read nothing `carlos` owns; |
| - `PrivateNetwork=yes` and `RestrictAddressFamilies=AF_UNIX`, so no route to |
| @@ -482,14 +596,23 @@ claim-bound template's hardening (§3 fact 8) with these differences: |
| only under `/var/tmp`, which is on disk on Debian 13, where `/tmp` is a |
| tmpfs. systemd deletes both when the unit stops, however it stops; |
| - `MemoryMax=512M`, `CPUQuota=100%`, `TasksMax=16`, `LimitFSIZE` set to |
| - the restored-size cap (it bounds each file; the agent's disk check in |
| - §5.7 bounds the total), `RuntimeMaxSec=20min`; |
| + the restored-size cap (it bounds each file; the agent's reservation in |
| + §5.7 bounds the total, codex 8), `RuntimeMaxSec=20min`; |
| - `SystemCallFilter=@system-service`, `NoNewPrivileges=yes`, empty |
| `CapabilityBoundingSet`, `ProtectProc=invisible`. |
| -`carlos system db-worker` refuses to run unless systemd handed it a |
| -connected socket on stdin (`Accept=yes`), so it cannot be started by hand or |
| -pointed at a file. |
| +`carlos system db-worker` refuses to run unless its stdin is a connected |
| +unix stream socket (checked with `getsockopt(SO_TYPE)` and `SO_PEERCRED`, |
| +whose peer must be the `carlos` user), so it cannot be started by hand or |
| +pointed at a file. Plan 1's acceptance test drives it only through the |
| +installed units. |
| + |
| +The worker protocol is length-framed messages with a fixed maximum frame |
| +size: `hello` (the readiness probe, §7.1), a header naming the plan's files, |
| +their sizes, the target transaction and the byte limits, then the files, |
| +then `ready` from the worker, then `run` or `stop` from the agent, then |
| +framed output and a final status. The worker refuses a file larger than its |
| +header said and any write past the header's totals. |
| The socket and service files are written and enabled by |
| `ensureSystemUnits`, the same root path that installs the other units, on |
| @@ -530,13 +653,15 @@ was being read, twice" case, and is retried the same way. |
| POST /api/cli/apps/{account}/{app}/db/{host}/reads |
| GET /api/cli/apps/{account}/{app}/db/{host}/reads/{id} |
| POST /api/cli/apps/{account}/{app}/db/{host}/reads/{id}/ack |
| +POST /api/cli/apps/{account}/{app}/db/{host}/reads/{id}/done |
| POST /api/cli/apps/{account}/{app}/db/{host}/reads/{id}/cancel |
| ``` |
| -All four are `svcrole.Set{}` (people only), behind the same gate order as |
| +All five are `svcrole.Set{}` (people only), behind the same gate order as |
| the db door. A read id is checked against its account, app and host on every |
| call; a mismatch is the byte-identical 404. A host-claim console answers all |
| -four with today's residency sentence. |
| +five with today's residency sentence. `ack` is progress only and never ends |
| +a read; `done` ends it and requires the result's hash (§5.4, codex 3). |
| ### 9.2 Mixed versions |
| @@ -553,19 +678,21 @@ A console with `CARLOS_DB_FORWARD` unset answers byte-identically to main. |
| ### 9.3 Audit |
| One `db-read` activity row at admission, in the run's manner, with the |
| -detail DBF-21 or DBF-22. The SQL text is not recorded. That differs from |
| -`carlos run`, which records argv, and matches the in-region door, which |
| -never sees SQL. A query's literals are data, and the audit log is not the |
| -place for them. The console keeps the SQL only inside the envelope, which is |
| -deleted with the read. |
| +detail DBF-21 or DBF-22. It records the SQL's length in bytes and the first |
| +16 hex digits of its SHA-256, not its text (provisional ruling 3). A member |
| +who kept the statement can match it to the row; the row itself holds none |
| +of the data a query's literals can carry. That differs from `carlos run`, |
| +which records argv, and keeps the in-region rule that SQL never reaches the |
| +audit log. The console keeps the SQL only inside the envelope, which the |
| +sweep deletes with the tombstone (§5.8). |
| ## 10. Errors and copy |
| Every new user-facing sentence is below. Error and status strings follow |
| -the run door's casing: lower case at the start and after a full stop, no em |
| -dashes. Docs prose keeps the CLI page's own sentence case. None of this is |
| -approved yet; the copy-review gate runs before any of it is written into |
| -code. |
| +Paul's voice rule (provisional ruling 4): lower case at the start and after |
| +a full stop, no em dashes. Docs prose keeps the CLI page's own sentence |
| +case. None of this is approved yet; the copy-review gate runs before any of |
| +it is written into code. |
| | ID | Where | Text | |
| |---|---|---| |
| @@ -585,11 +712,14 @@ code. |
| | DBF-14 | console 409 with the forward marker, read by an old CLI | `%s keeps its data in region %s. this version of carlos cannot read it from there. run carlos update, then try again.` | |
| | DBF-15 | CLI, `record-changed` | `%s changed while the read was waiting. try again.` | |
| | DBF-16 | CLI, `moving` from the box | as DBF-10 | |
| -| DBF-21 | audit detail, query | `forwarded a database query to region %s (cli)` | |
| -| DBF-22 | audit detail, query with `--at` | `forwarded a database query to region %s, as of %s (cli)` | |
| +| DBF-21 | audit detail, query | `forwarded a database query to region %s. the query was %d bytes, sha256 %s (cli)` | |
| +| DBF-22 | audit detail, query with `--at` | `forwarded a database query to region %s, as of %s. the query was %d bytes, sha256 %s (cli)` | |
| | DBF-30 | `docs/site/cli.md`, `carlos db`, replacing the paragraph that starts "If the app has more than one instance" | see below | |
| | DBF-31 | `docs/site/cli.md`, `carlos system`, the worker | `carlos system db-worker runs on platform boxes only. systemd starts it for each forwarded database query. you never run it yourself.` | |
| +DBF-13 is true as written only because the CLI buffers the whole answer and |
| +checks it before writing anything (§5.4, codex 6). |
| + |
| DBF-30: |
| > If the app has more than one instance, name one with `--instance`. Only |
| @@ -599,8 +729,9 @@ DBF-30: |
| > When an instance keeps its data in another region than the console's, |
| > `carlos db query` runs in that region. A server there restores the copy, |
| > runs your query and sends back only the rows. The rows are encrypted for |
| -> your terminal on the way, so nothing in between can read them. There, the |
| -> copy can be at most 2 GB once restored, the query runs for at most 5 |
| +> your terminal on the way, so nothing in between can read them, and |
| +> nothing is printed until all of them have arrived and checked out. There, |
| +> the copy can be at most 2 GB once restored, the query runs for at most 5 |
| > minutes, and the rows can come to at most 64 MB. `carlos db pull` cannot |
| > fetch a copy from another region yet. |
| @@ -616,8 +747,8 @@ DBF-17 to DBF-20 and DBF-23 to DBF-29 are left free for phase 2's pull rows. |
| ### 11.1 Switches and gates |
| -- **Box:** `db-read-1` (§7.1). Published once the binary, the units and the |
| - agent's worker loop are in place. `CARLOS_DB_READS=0` withdraws one box. |
| +- **Box:** `db-read-1` (§7.1). Published only while the armed loop's |
| + readiness probe passes. `CARLOS_DB_READS=0` withdraws one box. |
| - **Console:** `CARLOS_DB_FORWARD=1` turns forwarding on. Unset, every |
| answer is byte-identical to main. The console also needs the root key. |
| - **CLI:** none. The forward marker negotiates (§9.2). |
| @@ -628,8 +759,8 @@ DBF-17 to DBF-20 and DBF-23 to DBF-29 are left free for phase 2's pull rows. |
| A region with one capable box and one old box works: only the capable box |
| lists the request prefix. A region with no capable box gets DBF-1. A box |
| -that loses the capability mid-read finishes the read it holds and takes no |
| -new one. |
| +that loses the capability mid-read finishes the read it holds, within its |
| +deadline, and takes no new one. |
| ### 11.3 Roll order |
| @@ -646,59 +777,89 @@ labelled so; they are not acceptance. |
| | # | Test | Wiring | |
| |---|---|---| |
| -| T1 | A forwarded query end to end: console in region ie, agent in region uk, two MinIO buckets. The console's credentials cannot read the uk bucket (a MinIO policy), so any console read of it fails the test. | real console server, real agent loop, real stores; CI runs it with the worker reached through a test socket, labelled non-acceptance | |
| -| T2 | The same on the root runner with the real `carlos-dbread.socket` and `carlos-dbread@.service` installed by the real `ensureSystemUnits` | acceptance gate for plan 1 | |
| +| T1 | A forwarded query end to end: console in region ie, agent in region uk, two MinIO buckets on one server with separate credentials. The console's credentials cannot read the uk bucket (a MinIO policy), so any console read of it fails the test. | real console server, real agent loop, real store; CI runs it with the worker reached through a test socket, labelled non-acceptance | |
| +| T2 | The same on the root runner with the real `carlos-dbread.socket` and `carlos-dbread@.service` installed by the real `ensureSystemUnits`, so the socket is the worker's stdin and stdout as §8.3 says (codex 2) | acceptance gate for plan 1 | |
| | T3 | Sandbox witness on the root runner, from inside the worker: open `/data/registry.db`, `/etc/carlos/host.env` and another instance's directory; connect to `169.254.169.254:80` and to any TCP address; write outside `/var/tmp`; `ATTACH` and `VACUUM INTO` with the attach limit removed (a mutation); `load_extension`. Every one fails. The private tmp is gone after a normal end, a cancel and `kill -9` of the agent mid-read. | real units | |
| -| T4 | Forgery and replay: unsigned, wrong key, scoped key, envelope moved to another id and to another region, replay after cleanup, a chunk sealed to another key, a chunk reordered, the last chunk dropped. | real verify and seal code | |
| +| T4 | Forgery and replay: unsigned, wrong key, scoped key, envelope moved to another id and to another region, the same envelope offered again before and after the agent restarts and after the job objects are cleaned (the tombstone holds, codex 5), a chunk sealed to another key, a chunk reordered, the last chunk dropped. | real verify and seal code, real store | |
| | T5 | Two capable boxes in one region: exactly one runs each read, across 1,000 reads with random jitter. Expiry against a late pick: one `picked` exists, and the result matches it. | real store CAS | |
| | T6 | Parity: the forwarded query and the in-region query print byte-identical stdout for a fixture database (NULL, tabs, newlines, blobs, times). `--at` stops at the same transaction as the in-region door for the same replica and time, including a write in the same second as the target. | real restore, both paths | |
| -| T7 | Cleanup: after ack, cancel, expiry, lost and `KeepFor`, nothing remains under either prefix. | real sweep | |
| +| T7 | Ending and cleanup: after done, cancel, expiry, lost and `KeepFor`, nothing remains but the tombstone, and the tombstone goes after 5 minutes. A progress ack that arrives after the result never ends the read, and `done` with the wrong hash is refused (codex 3). Acknowledged chunks are gone while the read is still running, and stored ciphertext never exceeds the window (codex 7). A box whose PUTs stall past the old 90 s mark still owns the read, and nothing it writes lands after the fence (codex 4). | real sweep, real store, a store wrapper that delays PUTs | |
| | T8 | Mixed versions: the five rows of §9.2, with the forwarding-off 409 compared byte for byte against main. | real console | |
| -| T9 | Record changes between request and pick: moved, recreated (new incarnation), class changed. Each is refused with its code. | real records | |
| +| T9 | Record changes between request and pick, during planning, during download and between download and `run`: moved, recreated in the same account, recreated in another account, class changed. Each ends with its code and no chunk is ever written (codex 1). | real records, a store wrapper that pauses the download | |
| | T10 | Approved copy digests for every row in §10, in the `approved_copy_test.go` pattern. | | |
| +| T11 | CLI buffering: a dropped or bad last chunk and a summary hash mismatch each leave stdout empty (codex 6). | real CLI binary through `carlosTestEnv` | |
| +| T12 | Two concurrent reads on one box whose free space fits one of them: one runs, the other ends `no-room`; an agent restart mid-read takes no new read until the old worker is gone (codex 8). | real agent, root runner | |
| +| T13 | Readiness: a present socket with the service missing, masked, failing to start and speaking another protocol version; each withdraws `db-read-1` within one probe and edges pass (codex 9). | real units, root runner | |
| +| T14 | Console evidence: status snapshots unreadable, one snapshot missing, an edges file unreadable. Each admits the read and lets expiry answer; only complete evidence gives DBF-1 or DBF-2 (codex 10). | real `ReadLiveSnapshot` over a real store | |
| Mutation checks that must turn a test red: drop envelope verification (T4); |
| take the host or prefix from anywhere but the record and config (T1 with a |
| second tenant); remove `PrivateNetwork` (T3); skip the summary hash check |
| -(T4). |
| +(T11); skip check B (T9); delete `picked` with the rest (T4); let `ack` end |
| +a read (T7); drop the reservation table (T12). |
| ## 12. Provider boundary |
| - **Core:** the transport is `release.Store` objects, so it runs on MinIO and |
| - any S3-compatible store. The box streams each planned file with a GET from |
| - its region store and never uses a server-side copy, since a self-hosted |
| - deployment's region stores may be different servers. Signing is ed25519, |
| - sealing is the standard library plus `x/crypto/hkdf`. The sandbox is |
| - systemd, which every box already needs. |
| + any S3-compatible store. The box reads each planned file with a GET and |
| + writes chunks with PUTs; it never uses a server-side copy. Signing is |
| + ed25519, sealing is the standard library plus `x/crypto/hkdf`. The sandbox |
| + is systemd, which every box already needs. |
| +- **One store endpoint per box (codex 11).** Every S3 client in a box process |
| + takes its endpoint from the one `CARLOS_S3_ENDPOINT` |
| + (`internal/s3compat/s3compat.go`), the deployment store through |
| + `s3compat.Options` and the replica client through `lease`'s `s3Endpoint`. |
| + So a box reaches its region's data bucket and the deployment bucket on the |
| + same server, as it already must to wake and to move instances. This |
| + design supports exactly that shape: separate buckets per region, with |
| + separate credentials, on one endpoint. Regional data on separate store |
| + servers is not something any box path supports today, and adding |
| + per-store endpoints is not this design's job (§18). Rev 1's claim that |
| + region stores "may be different servers" was wrong and is withdrawn. |
| - **AWS adapter:** IAM is what makes the console unable to read another |
| region's bucket on the flagship, and the optional lifecycle rule (§5.8). |
| - Neither is required for the design to work. A self-hosted deployment with |
| - separate stores per region gets the same split from its own credentials. |
| -- **What a non-AWS deployment loses:** nothing. The capability, the |
| - liveness join and the box's identity use no AWS API. Multi-box regions are |
| - AWS-only today (H26), but nothing here depends on them; one box per region |
| - is enough. |
| + Neither is required for the design to work. A self-hosted deployment gets |
| + the same split from per-bucket credentials on its store. |
| +- **What a non-AWS deployment loses:** nothing it has today. The capability, |
| + the liveness join and the box's identity use no AWS API. Multi-box regions |
| + are AWS-only today (H26), but nothing here depends on them; one box per |
| + region is enough. |
| - **Build:** the worker and its socket code are Linux-only, with |
| `//go:build linux` and a `!linux` stub, so the CLI still cross-compiles for |
| darwin and windows. |
| ## 13. Census |
| -- **Plan 14 root-read census.** No root unit gains a read. The agent's new |
| - sites (the request listing and the record read) run as `carlos` in |
| - `carlos-edge.service` and take the J1 and J4 dispositions their siblings |
| - in the offline move took. The worker's sites (`Open`, `QueryContext`, the |
| - scratch writes) run as a `DynamicUser` and are classified as non-root. |
| - Each plan proposes an overlay in `docs/superpowers/notes/` for a ruling on |
| - #280 and adds its `plan14-root-reads-*.tsv`. |
| +**Plan 14 root-read census (codex 12).** The scanner reads every production |
| +package and matches `Open`, `QueryContext` and the rest whatever user runs |
| +them (`internal/provision/plan14_root_reads_test.go:33`). Its only classes |
| +are `authority`, `hint`, `diagnostic` and `legacy` (line 478), and `legacy` |
| +needs `accepted-legacy;rev3=<ruling>` as its basis. "Runs as a |
| +`DynamicUser`" is a fact for the ruling's footing, not a class. So each plan |
| +proposes, for a ruling on #280: |
| + |
| +| Declaration | Callers and identity | Proposed class and basis | |
| +|---|---|---| |
| +| `db.go` `runQueryContext` (main: `legacy`, `rev3=J4`, `basis=root-input`, `plan14-root-reads.tsv:202`) | moved out of `db.go` | retired through the overlay's removal row, the form the H3 carlos-run removal used | |
| +| `internal/dbquery` `RunContext` (the moved body, unchanged) | the CLI on a member's machine, as the member; the worker on a box, as its `DynamicUser`. Neither is root, and neither decides anything for root | `legacy`, `accepted-legacy;rev3=<ruling>;basis=root-input`, the J4 footing main gave the same body | |
| +| agent db-reads loop: the request listing, the record reads of checks A and B, the replica listing and GETs | the agent, `carlos`, `carlos-edge.service` | `legacy` with J1 (record and route reads) and J4 (object input), as the offline move's agent units took | |
| +| worker: the scratch writes and reads, the file replica restore, `CheckDB`, the query | `carlos-dbread@`, `DynamicUser` | `legacy` with J4 | |
| +| agent readiness probe and reservation table (`Statfs` on `/var/tmp`, `systemctl list-units`) | the agent, `carlos` | `diagnostic` for the probe; `legacy` J4 for `Statfs` | |
| + |
| +The overlay re-pins the source envelopes of every changed package (`.`, |
| +`internal/dbquery`, `internal/dbdoor`, `internal/host`, `internal/dbreads`) |
| +and updates the legacy seal under the same ruling. The census owner may rule |
| +other classes; the spec's requirement is only that every new and moved |
| +declaration has one of the four, with both callers of the shared runner |
| +named. |
| + |
| - **Root writes.** `ensureSystemUnits` writes two new unit files. That is a |
| root-written artifact and joins the Plan 14 write census as an overlay, as |
| every new unit has. |
| - **9B8 record-write census.** No record writer is added. The new `Put` |
| - sites (request, `picked`, beat, chunks, ack, cancel, result) are |
| - `nonrecord`. `registerCLIRoutes` changes as a `key-site` for the four |
| - routes. |
| -- **Claim routes.** The four routes join |
| + sites (request, `picked`, chunks, ack, cancel, result) are `nonrecord`. |
| + `registerCLIRoutes` changes as a `key-site` for the five routes. |
| +- **Claim routes.** The five routes join |
| `internal/console/testdata/claim-routes.txt`, refused on a host-claim |
| console like the run's. |
| - **go.mod.** No new module, so no root-envelope overlay for it. |
| @@ -706,70 +867,108 @@ second tenant); remove `PrivateNetwork` (T3); skip the summary hash check |
| ## 14. Plans |
| 1. **Box side and wire, dormant.** `internal/dbreads` (ids, keys, the |
| - envelope, sealing, the result, limits); `internal/dbquery` (the query |
| - runner and guards moved out of `db.go`, with the CLI switched to it and |
| - no behaviour change); `dbdoor` planning over any `replicaReader`; the |
| - agent's db-reads loop; `carlos system db-worker`; the two units in |
| - `ensureSystemUnits`; the `db-read-1` capability. Gates: T2, T3, T4, T5, |
| - the parity half of T6 that needs no console. Census overlay. Nothing |
| - reaches a box unless a console asks. |
| -2. **Console and CLI for `query` and `query --at`.** The four routes, the |
| - forward marker, capability and liveness checks, the bounds, expiry and |
| - lost handling in the run sweep, audit, the CLI's forwarded path, DBF |
| - copy, `docs/site/cli.md`. Gates: T1, T6, T7, T8, T9, T10, copy review, |
| + envelope, sealing, the result, limits, the fence); `internal/dbquery` |
| + (the query runner and guards moved out of `db.go`, with the CLI switched |
| + to it and no behaviour change); `dbdoor` planning over any |
| + `replicaReader`; the agent's db-reads loop with the identity bracket, the |
| + reservation table and the readiness probe; `carlos system db-worker` and |
| + its protocol; the two units in `ensureSystemUnits`; the `db-read-1` |
| + capability. Gates: T2, T3, T4, T5, T9, T12, T13, the parity half of T6 |
| + that needs no console. Census overlay. Nothing reaches a box unless a |
| + console asks. |
| +2. **Console and CLI for `query` and `query --at`.** The five routes, the |
| + forward marker, the evidence rule for capability and liveness, the |
| + bounds, chunk deletion on ack, expiry, lost and the tombstone in the run |
| + sweep, audit, the CLI's buffered forwarded path, DBF copy, |
| + `docs/site/cli.md`. Gates: T1, T6, T7, T8, T10, T11, T14, copy review, |
| census overlay. Roll per §11.3. |
| -3. **`pull` and `pull --at`.** After Paul's Q1 ruling. Sealed LTX files with |
| - the same window, a raw chunk route for 4 MiB chunks, local restore and |
| - `CheckDB` in the CLI, and the pull copy rows. |
| +3. **`pull` and `pull --at`.** On provisional ruling 1. Sealed LTX files |
| + with the same window and chunk deletion, a raw chunk route for 4 MiB |
| + chunks, local restore and `CheckDB` in the CLI, and the pull copy rows. |
| + |
| +After phase 3, in-region reads move to the box path too (provisional ruling |
| +5). That is a fourth plan, written once phase 3 has run on the flagship. It |
| +is not designed here beyond one constraint this spec already keeps: the |
| +box path is region-agnostic, so the console can send an in-region read |
| +through it unchanged. |
| ## 15. Review focus: the five likeliest failure modes |
| 1. **The worker reaches more than its copy.** A unit option missed or |
| - overridden (network, `/data`, `/etc/carlos`, the metadata service), or a |
| - SQLite path (ATTACH, `VACUUM INTO`, an extension, a temp file) that opens |
| - something. Any one of these turns a member's SQL into the box role, which |
| - reads every tenant in the region. |
| -2. **The prefix or host comes from the request.** Any path where an |
| - envelope field, rather than the fresh record and the box's config, |
| - picks the bucket, the prefix or the host is a cross-tenant read. So is |
| - a gap between the incarnation check and the read. |
| + overridden (network, `/data`, `/etc/carlos`, the metadata service, the |
| + socket on stdin and stdout), or a SQLite path (ATTACH, `VACUUM INTO`, an |
| + extension, a temp file) that opens something. Any one of these turns a |
| + member's SQL into the box role, which reads every tenant in the region. |
| +2. **The prefix or host comes from the request, or the identity bracket |
| + leaks.** Any path where an envelope field, rather than the fresh record |
| + and the box's config, picks the bucket, the prefix or the host is a |
| + cross-tenant read. So is any byte read outside checks A and B, or any |
| + output written before check B. |
| 3. **Forgery and replay.** An unsigned or wrongly scoped request accepted, |
| domain separation from other root envelopes missing, the region not |
| - bound, or a replay inside the window that runs twice. |
| -4. **Races and leaks.** Two boxes running one read, expiry and a late pick |
| - both "winning", a result written before its last chunk, ciphertext or |
| - worker scratch left behind after a crash, a full disk on a 60 GiB box, |
| - or an unacked window that never drains. |
| + bound, or the tombstone removed before the acceptance window closes. |
| +4. **Ending.** A write after the fence, a progress ack taken for the end, a |
| + result before its last chunk, two owners of one read, ciphertext kept |
| + past the window, reservations that leak or double count. |
| 5. **Drift between the two paths.** The forwarded `--at` stopping at a |
| different transaction than the in-region door, output formatting that |
| - differs between CLI and box versions, or a capability published by a box |
| - whose worker socket is not actually armed. |
| - |
| -## 16. Open questions for Paul |
| - |
| -1. **Q1, pull through the deployment bucket.** In phase 2 a pull's planned |
| - LTX files pass through the deployment bucket (eu-west-1 on the flagship) |
| - as ciphertext only the member's CLI can open, at most 64 MiB at a time |
| - (16 unacked chunks of 4 MiB) and for at most an hour. The db spec rejected plaintext there. Is sealed and |
| - windowed acceptable, or should cross-region pull stay refused? |
| -2. **Q2, limits.** 2 GiB restored, 5 minutes, 64 MB of rows, two reads per |
| - box at once. Right for launch? Real database sizes on the flagship are |
| - unmeasured. |
| -3. **Q3, audit.** The audit row records that a query ran, not its SQL. That |
| - differs from `carlos run`, which records argv. |
| -4. **Q4, copy casing.** The new strings start lower case after a full stop, |
| - as the run door's do. The existing db door strings capitalise after a |
| - full stop ("too many database requests. Wait a minute..."). Keep the new |
| - ones as drafted, or match the db door? |
| -5. **Q5, one path or two.** In-region reads keep today's path (the CLI |
| - restores locally, the console streams LTX). Forwarded queries run on a |
| - box with limits. Should in-region queries move to the box path later, so |
| - there is one behaviour, or should the two stay apart? |
| + differs between CLI and box versions, or a capability published without |
| + a passing readiness probe. |
| + |
| +## 16. Open questions: provisional rulings awaiting Paul |
| + |
| +The coordinator answered rev 1's five questions provisionally on |
| +2026-10-22. The spec is written to these answers. Each stands until Paul |
| +confirms or changes it. |
| + |
| +1. **Pull through the deployment bucket.** Phase 2's pull may pass |
| + ciphertext through the deployment bucket (eu-west-1 on the flagship), at |
| + most 64 MiB at a time (16 unacknowledged chunks of 4 MiB, deleted as they |
| + are acknowledged) and for at most 1 hour. **Paul's call:** if UK data |
| + residency is a legal requirement rather than a technical one, whether |
| + encrypted data in transit through an Irish bucket is acceptable is a |
| + question for him, not for engineering. |
| +2. **Limits stand.** 2 GiB restored, 5 minutes, 64 MB of rows, two reads per |
| + box. Real database sizes on the flagship are still unmeasured; plan 2's |
| + dev check reports them. |
| +3. **Audit.** The row records the SQL's length and a SHA-256 prefix, not its |
| + text (§9.3). |
| +4. **Copy casing.** New strings use Paul's voice rule: lower case after a |
| + full stop (§10). The existing db door strings that capitalise are left |
| + as they are; changing them is a separate copy review. |
| +5. **One path.** In-region reads move to the box path after phase 3 (§14). |
| ## 17. What this does not do |
| - No console per region and no console-to-box connection. |
| -- No change to in-region reads, the claim-mode door, seeds `--from`, fleet |
| - or pooled instances. |
| +- No change to in-region reads (until the fourth plan), the claim-mode |
| + door, seeds `--from`, fleet or pooled instances. |
| - No IAM change. No new read authority for any role. |
| +- No per-store S3 endpoints. |
| - No `carlos db shell` and no browser view. |
| + |
| +## 18. Declined review findings |
| + |
| +Round 1 (`CODEX-FINDINGS-DB-REGION-FORWARD.md`). Every citation was checked |
| +against `main` at `5f341f33` and holds. Findings 2 to 10 and 12 are accepted |
| +in full. Two are accepted in part: |
| + |
| +- **Codex 1, the "stronger guarantee" (pin object versions to the checked |
| + incarnation): declined.** The identity bracket (§8.2) is accepted and |
| + closes the finding: no byte is read outside two matching checks, and no |
| + output exists before the second. Pinning versions needs |
| + `s3:GetObjectVersion` and `s3:ListBucketVersions` on the box's data |
| + bucket, which the flagship's uk, us and au boxes do not have at module |
| + ref v0.21.0, and a versioned reader that today lives only behind the |
| + claim boundary. That is a new mechanism for a case the bracket already |
| + removes. Claim activation, which brings versioned reads, can revisit it. |
| +- **Codex 11, independently configured deployment and regional-data |
| + endpoints: declined.** The finding is right that rev 1 overclaimed, and |
| + §12 now says so. But no box path (wake, moves, this one) supports |
| + regional data on a separate store server today, because every S3 client |
| + in the process reads one `CARLOS_S3_ENDPOINT`. Adding per-store endpoints |
| + for db reads alone would make this the only path that works in a shape |
| + the rest of the box does not. The simpler rule is to state the supported |
| + shape (one endpoint, separate buckets and credentials) and test it (T1). |
| + Per-store endpoints, if wanted, are a platform-wide change with its own |
| + design. |