carlos / platform Public

db-region-forward spec rev 3: the read lifecycle as seven invariants

Astra round 2 found the remaining defects in one place: which object ends
a read and who deletes what when. Rev 3 states that lifecycle as seven
invariants and drops every delete from the protocol: all objects are
create-only under db-reads/, one create-only result.json ends a read,
expired and lost are created lazily on follow calls, and a bucket
lifecycle rule (current and noncurrent) expires everything. That closes
findings 4, 7, 13 and 14 together and supersedes the ack/done routes.
Findings 8 (release reservations only when the worker unit stops, no
orphan escape, worker cancels on EOF), 9 (a connection kept for the
probe), 10 (incomplete evidence first) and 15 (follow routes gate on the
signed request, not the current record) are fixed directly. The storage
bound in provisional ruling 1 is restated and flagged for re-confirmation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 2 days ago 944a61fdb68928668c8bdf61930398b1232e57f9 parent c5b023c

1 file changed, +392 −264

docs/superpowers/specs/2026-10-22-db-region-forward-design.md +392 −264

diff --git a/docs/superpowers/specs/2026-10-22-db-region-forward-design.md b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md

index f6f957109..b4f8dd272 100644

--- a/docs/superpowers/specs/2026-10-22-db-region-forward-design.md

+++ b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md

@@ -1,13 +1,17 @@

# `carlos db` for instances whose data lives in another region

-**Status:** rev 2, 2026-10-22. Design only, nothing built. Written for Paul,

+**Status:** rev 3, 2026-10-22. Design only, nothing built. Written for Paul,

who approved the direction (§2). Review brief:

-`CODEX-REVIEW-BRIEF-DB-REGION-FORWARD.md`. Rev 1 was `b35b3ab1`. Rev 2

-answers Astra's round 1 (`CODEX-FINDINGS-DB-REGION-FORWARD.md`, 12

-findings: 10 accepted in full, 2 accepted in part); each change is marked

-**(codex N)** where it lands, and §18 lists what was declined. Rev 2 also

-folds in the coordinator's provisional answers to the open questions, which

-await Paul (§16).

+`CODEX-REVIEW-BRIEF-DB-REGION-FORWARD.md`. Rev 1 was `b35b3ab1`, rev 2

+`2cf4a88f`. Rev 2 answered Astra's round 1

+(`CODEX-FINDINGS-DB-REGION-FORWARD.md`). Rev 3 answers round 2

+(`CODEX-FINDINGS-DB-REGION-FORWARD-R2.md`): the request, result and

+acknowledgement lifecycle is rewritten as seven invariants (§5.2) with no

+deletes in the protocol, which closes findings 4, 7, 13 and 14 together;

+findings 8, 9, 10 and 15 are fixed directly. Each change is marked

+**(codex N)** where it lands, and §18 lists what was declined. The spec

+carries the coordinator's provisional answers to the open questions, which

+await Paul (§16); rev 3 changes the premise of ruling 1.

**Companion specs:** `2026-09-02-carlos-db-design.md` (the db door, cited

below as "db spec"), `2026-10-08-carlos-run-design.md` (the run transport,

@@ -49,7 +53,8 @@ working around them:

- **db spec §6 rejected "tenant database bytes in the deployment bucket".**

A forwarded read has to come back to the console through the deployment

bucket (§5.1). This spec puts only encrypted bytes there, sealed to a key

- that exists only in the member's CLI (§5.5). In phase 1 that is only a

+ that exists only in the member's CLI (§5.6), for at most two days (§5.7).

+ In phase 1 that is only a

query's rows. Phase 2's pull would send the planned replica files, which

are a pull's result, under provisional ruling 1 (§16), which awaits Paul.

@@ -120,8 +125,8 @@ connects to a box". The transport is the run's.

|---|---|---|

| D1 | Phase 1 forwards `query` and `query --at`. `pull` and `pull --at` forward in phase 2, under provisional ruling 1. The claim-mode door and the seeds `--from` door stay refused. | §6 |

| D2 | Any live box in the record's region that publishes `db-read-1` and is not draining may serve. The first to create the read's `picked` marker serves it. The instance's lease holder gets no preference. | §7 |

-| D3 | Transport is the run's: objects in the deployment bucket, a serial bump, chunks paged by the console. The request is a root-signed envelope. No record write. | §5 |

-| D4 | Results are sealed to an ephemeral key the CLI makes for each read. The console and the deployment bucket only ever hold the result as ciphertext. The console does see the SQL, as the envelope's signer. | §5.5 |

+| D3 | Transport is the run's: objects in the deployment bucket, a serial bump, chunks paged by the console. The request is a root-signed envelope. No record write. Every object is a create-only write under `db-reads/`, never replaced or deleted by the protocol; a bucket lifecycle rule expires them (§5.2). | §5 |

+| D4 | Results are sealed to an ephemeral key the CLI makes for each read. The console and the deployment bucket only ever hold the result as ciphertext. The console does see the SQL, as the envelope's signer. | §5.6 |

| D5 | The box splits the work. The agent (`carlos`) verifies, resolves, plans and streams objects. Everything that decodes tenant bytes (LTX, SQLite, the member's SQL) runs in a new sandboxed unit, `carlos-dbread@.service`, as a `DynamicUser` with no network. | §8 |

| D6 | The worker gets the planned LTX files and nothing else. Its scratch is its own private `/var/tmp`, which systemd removes when the unit stops. | §8.3 |

| D7 | Rollout: boxes publish `db-read-1`; the console forwards only with `CARLOS_DB_FORWARD=1` and a capable region. Dormant by default. | §11 |

@@ -150,31 +155,58 @@ Decision: deployment-bucket objects. The cost is latency (a pickup is

seconds, not milliseconds) and a LIST per serial move on capable boxes

(§5.8). Both are acceptable for a command a person runs by hand.

-### 5.2 Objects

+### 5.2 The lifecycle invariants

-`<id>` is `dbr_` and 26 lowercase base32 characters (128 random bits),

-minted the way `runs.MintID` mints run ids, with its own `ValidID`.

+Rounds 1 and 2 found six defects (codex 3, 4, 5, 7, 13, 14) in one place:

+which object ends a read, and who deletes what when. Each fix in rev 2

+added a rule about deletes, and each rule made a new edge. Rev 3 removes

+the deletes instead. The lifecycle is these seven invariants, and §5.3 to

+§5.9 are the simplest mechanism that keeps them.

-|---|---|---|---|

-| `apps/<acct>/<app>/db-reads/<id>/picked` | box, or the console's expiry | create-only | Who took the read, and the box's write deadline. Exactly one writer wins. It is also the read's tombstone and outlives the acceptance window (§5.9). |

-There is no heartbeat object. Rev 1 had one and declared a read lost after

-90 s without it. That let a slow box keep writing after its read was

-declared lost (codex 4). Rev 2 replaces it with one rule: a box's writes

-stop at a deadline it states in `picked`, and nothing is declared lost

-until that deadline has passed (§5.8).

-The job objects live under the app's subtree for the run's reason: app trash

-and purge sweep them, and the owner gate maps onto them. The request lives

-under `control/` so a box lists one small prefix for its own region and

-never walks every app. `<region>` is the record's `Region`, which must pass

-the region-code shape check; a `fleet/<name>` region never gets here (§6).

+| # | Invariant |

+|---|---|

+| **I1 Immutable** | Every protocol object is created once with a conditional create (`If-None-Match: *`, `release.Store.Put` with an empty `ifMatch`) and is never replaced or deleted by the protocol. There are no acknowledgement objects and no in-place updates. |

+| **I2 Expiry is the store's** | Objects leave only through a bucket lifecycle rule on the whole `db-reads/` prefix: current versions expire 1 day after creation, noncurrent versions 1 day after that. A read lives at most 22 minutes (§5.7), so expiry never meets a live read. |

+| **I3 One owner** | The first create of `picked` owns the read: a box, or the console expiring it. Only the owning box writes chunks, and only it writes a result other than `expired` or `lost`. |

+| **I4 One end** | `result.json` is the read's only terminal marker. Its first create decides the outcome: the owner's, or the console's `expired` or `lost`. Storage enforces it; nothing written after it changes the outcome. |

+| **I5 Acceptance** | The CLI accepts output only from a `done` result written by the owner `picked` names: chunks 1 to `last_seq`, each opened under the read's key, adding up to the sealed summary. Every other object, including a late chunk, is ignored. |

+| **I6 Identity** | Account, app, host and region are fixed by the signed request, which lives exactly as long as every other object of the read (I1, I2). Every follow route authorises against it, never against the current instance record. |

+| **I7 Time is decided lazily** | `expired` (no `picked` 2 minutes after `issued_at`) and `lost` (no result 2 minutes after the owner's deadline) are created by the console when a follow call observes them, by the same conditional create as everything else. No sweep is needed to end a read. |

+What each invariant closes:

+| Finding | Closed by | How |

+|---|---|---|

+| codex 3 (progress ack read as the end) | I1, I4 | There are no acks. Only `result.json` ends a read. |

+| codex 4 (late writes after `lost`) | I4, I5 | `lost` is a conditional create of `result.json`. A box result that commits later fails its precondition on the server, whatever the client saw; a box result that committed first makes the console's `lost` fail. A late chunk is an object no one accepts. The fence is the store's create-only check, not a clock. |

+| codex 5 (cleanup reopens the election) | I1, I2, I3 | `picked` is never deleted by the protocol and outlives the 60 s acceptance window by a day. |

+| codex 7 (deletes leave noncurrent versions) | I1, I2 | The protocol deletes nothing. The lifecycle rule expires current and noncurrent versions of `db-reads/`. The bound is restated honestly in §5.7. |

+| codex 13 (completion deletes the evidence that a read ended) | I1, I4 | Nothing is deleted on completion. A console restart sees the same `result.json`. |

+| codex 14 (cleanup removes the identity needed to follow) | I1, I2, I6 | The request is deleted only with everything else, a day later. |

+| codex 15 (the db gate on follow routes) | I6 | Follow routes check membership and the request, not the current record (§9.1). |

+### 5.3 Objects

+All objects live under one prefix, `db-reads/`, so one lifecycle rule covers

+them (I2). `<id>` is `dbr_` and 26 lowercase base32 characters (128 random

+bits), minted the way `runs.MintID` mints run ids, with its own `ValidID`.

+Every object below is create-only (I1).

+| Key | Writer | Purpose |

+|---|---|---|

+| `db-reads/<id>/request.json` | console | The request: a root envelope (§5.4). The read's identity for its whole life (I6). |

+| `db-reads/queue/<region>/<YYYYMMDDHH>/<unix>-<id>` | console | An empty pointer, written after the request. Boxes of `<region>` list the current and previous hour's prefix to find reads, and skip any entry whose `<unix>` (the request's `issued_at`) is outside the pickup window without reading it. |

+| `db-reads/<id>/picked` | a box, or the console when expiring | The owner (I3): box name, boot id and the owner's deadline. The console's form says `expired`. |

+| `db-reads/<id>/out/<seq>` | the owning box | One sealed chunk of output (§5.6). |

+| `db-reads/<id>/cancel` | console | The CLI asked to stop. |

+| `db-reads/<id>/result.json` | the owning box, or the console for `expired` and `lost` | The end (I4). The box writes it after every chunk write has been confirmed. |

+The objects moved out of the app's subtree (rev 2 had them under

+`apps/<acct>/<app>/`). The run keeps its objects there so app purge sweeps

+them; here the lifecycle rule does that within a day, and keeping one

+prefix is what lets one rule cover every object. Nothing under `db-reads/`

+is readable without the CLI's key except the request (which holds the SQL;

+see §9.3) and the open half of the result.

There is no write to the instance record. A run takes a record CAS because

it needs the instance to itself. A db read touches neither the instance nor

@@ -182,7 +214,14 @@ its lease, needs no exclusivity, and is frequent. A record write per read

would bump `record_seq`, race with moves and reconcile, and add a record

writer to the 9B8 census for nothing.

-### 5.3 The request envelope

+**A box's chunk write whose answer is lost.** The box keeps each sealed

+chunk's bytes until its create is confirmed. On a retry that answers

+"already exists", it reads the object back and compares bytes: equal means

+its earlier attempt landed; different ends the read `failed`, since only

+the owner writes there. It writes `result.json` only when every chunk is

+confirmed, so a `done` result never names a chunk that is not there.

+### 5.4 The request envelope

```json

{

@@ -207,18 +246,18 @@ writer to the 9B8 census for nothing.

console that holds only a scoped key cannot sign, and so cannot forward:

it answers with today's residency sentence.

- `purpose` is a fixed string so the envelope cannot be confused with any

- other root-signed payload. The box decodes into the exact type with

- `VerifyRootEnvelope`, which requires canonical JSON.

-- The box accepts an envelope only when `id` matches the key it was listed

- under, `region` matches both the key and the box's own `CARLOS_REGION`,

- and `issued_at` is no more than 60 s old (the pickup window) and no more

- than 60 s in the future (skew).

+ other root-signed payload. The box and the console decode it into the

+ exact type with `VerifyRootEnvelope`, which requires canonical JSON.

+- The box accepts an envelope only when `id` matches the key it was read

+ from and the queue entry that named it, `region` matches the queue key and

+ the box's own `CARLOS_REGION`, and `issued_at` is no more than 60 s old

+ and no more than 60 s in the future.

- `at` is empty for the newest copy. `op` is `query` in phase 1; phase 2

adds `pull`.

- The envelope carries no bucket, no prefix and no key. The box derives the

replica prefix itself (§8.2).

-### 5.4 Sequence of a forwarded query

+### 5.5 Sequence of a forwarded query

1. The CLI calls the existing list door, as today. A new console with

forwarding on answers a capable cross-region instance with 409 and an

@@ -229,62 +268,56 @@ writer to the 9B8 census for nothing.

an ephemeral P-256 key, and sends

`POST /api/cli/apps/{account}/{app}/db/{host}/reads` with `op`, `sql`,

`at` and `reply_key`.

-3. The console runs `dbGate`'s order (person only, membership with a

- byte-identical 404, the db budget, configuration, the record, the class

- rail, pooled, fleet), then refuses a moving record, checks

- `checkQuerySQL` again, checks capability (§7.1) and its concurrency

- bounds (§5.7), mints the id, writes the envelope create-only, bumps the

- serial, writes the audit row (§9.3) and answers 202 `{id, region}`.

-4. Each capable box in the region lists `control/db-reads/<region>/` when the

- serial moves, and every 30 s if it does not. For each envelope it

- verifies, checks that it has a free worker slot and can reserve the disk

- floor (§5.7), waits a random 0 to 500 ms, and creates `picked` with its

- box name, boot id and deadline. A box that loses the create steps away.

-5. The winner runs §8.2's identity bracket around planning and download,

- then runs the query in a worker and seals the output into chunks. It

- reads `cancel` and `ack` every 5 s and whenever its unacked window is

- full, and writes `result.json` after its last chunk write has returned.

+3. Admission is the only call that reads the current instance. The console

+ runs `dbGate`'s order (person only, membership with a byte-identical 404,

+ the db budget, configuration, the record, the class rail, pooled,

+ fleet), then refuses a moving record, checks `checkQuerySQL` again,

+ checks capability (§7.1) and its concurrency bounds (§5.7), mints the id,

+ creates the request, then the queue entry, bumps the serial, writes the

+ audit row (§9.3) and answers 202 `{id, region}`.

+4. Each capable box lists its region's queue prefixes for this hour and the

+ last when the serial moves, and every 30 s if it does not. For each

+ entry inside the pickup window that it has not already decided, it

+ reads and verifies the request, checks admission (§5.7), waits a random

+ 0 to 500 ms, and creates `picked`. A box that loses the create steps

+ away. A box remembers decided ids in memory; after a restart it reads

+ each in-window entry once more and finds `picked` there.

+5. The owner runs §8.2's identity bracket around planning and download,

+ then runs the query in a worker and writes sealed chunks as the output

+ arrives. It reads `cancel` every 5 s. When the worker ends, it confirms

+ every chunk and creates `result.json`.

6. The CLI polls `GET .../db/{host}/reads/{id}?after=<seq>` about once a

- second. The page is the run's `Page` shape: state, chunks (base64 of the

- sealed bytes) and the result when present, under 512 KiB. The CLI opens

- each chunk into a local buffer and posts a progress ack,

- `POST .../reads/{id}/ack` with `{last_seq}`. It writes nothing to stdout

- yet **(codex 6)**.

-7. On the result, the CLI opens the sealed summary, checks the buffered

- bytes against its count and SHA-256, and only then writes the rows to

- stdout and today's status line to stderr. It then posts a completion,

- `POST .../reads/{id}/done` with `{result_sha256}`, the SHA-256 of the

- `result.json` bytes it verified **(codex 3)**, and exits.

-A progress ack and a completion are different calls with different bodies,

-so a late progress ack can never be read as the end. The console accepts a

-completion only when `result.json` exists and hashes to `result_sha256`.

+ second. The page is the run's `Page` shape: the state, chunks after

+ `<seq>` (base64 of the sealed bytes) and the result when present, under

+ 512 KiB. The page's chunks come only from the owner `picked` names, by

+ key. The CLI opens each chunk into a local buffer and writes nothing to

+ stdout yet **(codex 6)**.

+7. When the result is `done`, the CLI opens the sealed summary, checks the

+ buffered bytes against its count and SHA-256, and only then writes the

+ rows to stdout and today's status line to stderr. Any other result maps

+ to its sentence (§10). Then it exits. It sends nothing to say it is

+ finished; nothing needs to know (I1).

The buffer is bounded by the output cap (64 MiB), which a terminal-run

-command can hold in memory. Progress acks keep the box's window moving

-while the CLI buffers. If the summary does not check out, the CLI has

+command can hold in memory. If the summary does not check out, the CLI has

written nothing, so DBF-13's "showed none of it" is true. The status line

(`dbQueryStatus`) needs the copy's transaction and time from the summary,

so the CLI prints DBF-11 while it waits and today's status line with the

rows. stdout carries the same rows in the same format as an in-region

query.

-### 5.5 Sealing

+### 5.6 Sealing and the result

- One ephemeral P-256 key pair per read, made by the CLI and held only in

- its memory. The public half travels in the request and the envelope.

+ its memory. The public half travels in the request.

- Each chunk is sealed with the appconfig v2 construction (ECDH P-256,

HKDF-SHA256, AES-256-GCM, a fresh ephemeral sender key per seal), with its

own HKDF info string, `carlos-db-read-v1`, and associated data binding the

read id, the chunk's sequence number, its stream and whether it is the

last chunk. A chunk moved to another read, reordered, or relabelled as

last fails its tag.

-- `result.json` has an open part for the console and a sealed part for the

- CLI (§5.6). The sealed part carries the plaintext byte count and its

- SHA-256, and the CLI refuses a result whose chunks do not add up to it

- (DBF-13). That closes truncation.

- What this buys: the deployment bucket, which sits in the console's region,

- never holds readable tenant data. The console never sees rows. A replayed

+ never holds readable rows. The console never sees rows. A replayed

envelope (§5.9) produces output only the original CLI can open.

- What it does not buy: authenticity of the sender. Any box can seal to a

public key. A compromised box could write a false answer. That is the

@@ -293,8 +326,6 @@ query.

way. Per-box signing keys exist only in claim mode, which this design

leaves alone (§6).

-### 5.6 The result

```json

{

"state": "done",

@@ -307,112 +338,123 @@ query.

}

```

-`state` is `done`, `failed`, `refused`, `expired`, `lost` or `cancelled`.

-`code` is a fixed vocabulary the console may read: `nothing-replicated`,

-`history-gone`, `check-failed`, `changed-twice`, `too-large`, `no-room`,

-`sql-error`, `timeout`, `output-cap`, `record-changed`, `moving`,

-`interrupted`. The sealed summary carries the transaction id and time of the

-copy, the oldest time for `history-gone`, the restored size and cap for

-`too-large`, SQLite's own error text for `sql-error`, and the byte count and

-SHA-256 of the output. SQLite's error text is sealed because it can name

-schema.

-The CLI maps codes to today's sentences where one exists

-(`dbRestoreError`), and to §10's rows otherwise.

+`state` is `done`, `failed`, `refused`, `cancelled` (the owner's), or

+`expired` or `lost` (the console's). `code` is a fixed vocabulary the

+console may read: `nothing-replicated`, `history-gone`, `check-failed`,

+`changed-twice`, `too-large`, `no-room`, `sql-error`, `timeout`,

+`output-cap`, `record-changed`, `moving`, `interrupted`. The sealed summary

+carries the transaction id and time of the copy, the oldest time for

+`history-gone`, the restored size and cap for `too-large`, SQLite's own

+error text for `sql-error`, and the byte count and SHA-256 of the output.

+SQLite's error text is sealed because it can name schema. The CLI maps

+codes to today's sentences where one exists (`dbRestoreError`), and to

+§10's rows otherwise.

-### 5.7 Limits

+### 5.7 Limits and storage

| Limit | Value | Enforced by |

|---|---|---|

-| Pickup window | 60 s from `issued_at` | box (refuses older envelopes), console expiry |

+| Pickup window | 60 s from `issued_at`, plus 60 s of skew | box (refuses older envelopes); console creates `expired` after 2 minutes (I7) |

| Planning and restore | 10 min | worker unit and agent context |

| Query | 5 min wall clock | worker |

-| Box deadline | 20 min from `picked`, stated in it | agent context on every read and write, worker `RuntimeMaxSec` |

-| One write | 60 s timeout on every PUT the box makes | agent |

+| Owner deadline | 20 min from `picked`, stated in it | agent context, worker `RuntimeMaxSec`; console creates `lost` 2 minutes after it (I7) |

| Restored size | 2 GiB by default, `CARLOS_DB_READ_MAX_BYTES` on the box | agent, from the plan, before any byte moves |

| Disk | a reservation per read (below) | agent |

| Output | 64 MiB (`runs.MaxOutputBytes`, written "64 MB" in copy as the run's docs do); the read ends `output-cap` and the CLI says so | agent; the CLI's buffer is bounded by the same number |

| Chunk | 256 KiB or 1 s, whichever comes first (`runs.ChunkBytes`, `runs.ChunkEvery`) | agent |

-| Unacked window | 16 chunks (4 MiB); the box pauses until the CLI's ack moves | agent |

-| Worker slots per box | 2 (`MaxConnections=2` on the socket) | systemd, agent |

-| Open reads per person | 2 | console |

-| Open reads per console | 16 | console, 429 with `Retry-After` |

+| Reads per box | 2, plus one connection kept for the readiness probe (`MaxConnections=3`) | agent, systemd |

+| Open reads per person | 2 | console, in memory |

+| Open reads per console | 16 | console, in memory, 429 with `Retry-After` |

| Request budget | the db door's existing 600 per token per minute | console |

+The console's open-read counts are in-memory ceilings, like the db door's

+token budget. After a console restart they start at zero. The real bound on

+work is the boxes' own two reads each, which a console restart does not

+change.

+**Stored ciphertext (codex 7).** Rev 2 promised at most 4 MiB at a time by

+deleting acknowledged chunks. That was not true: the deployment bucket is

+versioned (`infra/modules/deployment-buckets/main.tf`), `S3Store.Delete`

+sends no version id, and a delete only hid the chunk behind a delete

+marker. Rev 3 deletes nothing and states the real bound: one read's

+ciphertext is at most its output cap (64 MiB in phase 1), and every object

+of every read is gone two days after creation at most, through the

+lifecycle rule's current and noncurrent expiry. Plaintext rows are never in

+the bucket.

**Disk reservations (codex 8).** The agent is the only process that starts

workers, so it keeps one reservation table under one mutex. Before `picked`

it reserves the floor (5 GiB); after planning it grows that reservation to

the plan's LTX bytes plus its restored size. Each step succeeds only if the

filesystem holding `/var/tmp` keeps at least 5 GiB and 10% free after

-subtracting every other open reservation; otherwise the read ends `no-room`

-and the reservation is released. Reservations are released on every exit

-of the read's goroutine. The worker refuses to write past the byte counts

-its header names, so a reservation is an upper bound, not an estimate. A

-restarted agent starts with an empty table and takes no read until no

-`carlos-dbread@` unit is active (`systemctl list-units`, readable without

-privilege) or 2 minutes have passed, so the old workers' scratch is gone

-before anything is counted again.

-**Stored ciphertext (codex 7).** On each progress ack the console deletes the

-chunks at or below `last_seq` (deletes are idempotent, and an ack lower than

-the stored one is ignored). With the box paused at 16 unacknowledged

-chunks, a read never holds more than 4 MiB of ciphertext in the deployment

-bucket, plus one chunk in flight. Phase 2's pull uses the same rule with

-4 MiB chunks, so 64 MiB.

-### 5.8 Ending, cleanup and cost

-**The write fence (codex 4).** `picked` names the box's deadline (20 minutes

-after the pick, by the box's clock). Every read and write the box makes for

-that read runs on a context that ends at the deadline, and every PUT has a

-60 s timeout, so no box write can land after the deadline plus 60 s. The

-console treats deadline plus 2 minutes (one minute more for clock skew) as

-the **fence**: after it, the box can write nothing more.

-- **Done.** The box writes `result.json` only after its last chunk write has

- returned. On the CLI's completion (§5.4 step 7) the console deletes the

- remaining chunks, `ack`, `cancel` and `result.json`.

-- **Cancelled.** The console writes `cancel`. The box stops the worker and

- writes a `cancelled` result, which ends the read like any other result.

-- **Expired.** An unpicked read past its window gets the console's own

- `picked` (create-only, so it races safely with a late box; whoever creates

- it owns the read) and an `expired` result. No box ever owns it, so no box

- writes.

-- **Lost.** A picked read with no result at the fence gets a `lost` result

- from the console. The box can no longer write, so nothing lands after it.

- A person whose box died waits up to 22 minutes for DBF-12. That is the

- price of never answering before the box is fenced. The CLI says DBF-11

- while it waits and can be stopped with Ctrl-C.

-- **The sweep** joins `runsweep.go`'s loop (30 s) for expiry and lost. Every

- 10 minutes it deletes job objects whose result is older than 1 hour, or

- whose fence passed more than 1 hour ago. `KeepFor` is 1 hour, not the

- run's 24, because these objects are copies of data, even if encrypted.

-- **The tombstone (codex 5).** `picked` and the request object are never

- deleted with the rest. The sweep deletes them only once both hold: the

- read has ended (a result exists, or its fence has passed), and

- `issued_at` is more than 5 minutes old, past the 60 s acceptance window

- and its 60 s of skew, with margin. Until then, `picked` exists, so no box, including the

- same box after a restart, can pick that envelope again.

-- **On the box,** the agent holds no scratch at all. The worker's scratch is

- removed by systemd when its unit stops, whatever stopped it (§8.3). An

- agent that restarts mid-read loses the connection, the worker exits, and

- the console's lost rule ends the read at the fence.

-- **A bucket lifecycle rule** on `apps/*/db-reads/` and `control/db-reads/`

- (1 day) is an optional backstop for the AWS adapter. The sweep is the

- real cleanup, so a store without lifecycle rules loses nothing.

-- **Cost.** Each capable box does one LIST of an almost always empty prefix

- per serial move and every 30 s. A read adds a few PUTs, a GET of `cancel`

- and `ack` every 5 s, and one GET and one DELETE per chunk. The October

- bill was LIST and GET polling, so plan 1 measures the LIST rate on dev

- before the flagship roll and reports it.

+subtracting every other open reservation; otherwise the read ends `no-room`.

+A reservation is released only when its worker **unit** has stopped, not

+when its connection closes: systemd removes the unit's private `/var/tmp`

+at stop, so space is free only then. The agent learns that from

+`systemctl list-units 'carlos-dbread@*'` (readable without privilege).

+The agent admits a new read, or a probe, only when the number of active,

+activating or deactivating `carlos-dbread@` units is no more than the

+number of workers it is itself running. Any other unit is an orphan (from a

+previous agent, or one still stopping), and its scratch is not in the

+table, so the agent refuses work until it is gone. There is no timeout

+escape: an orphan worker stops when its connection is gone (§8.3), and

+`RuntimeMaxSec` stops it in the end regardless. A restarted agent therefore

+takes no read until every old worker has stopped.

+### 5.8 Ending a read, and cost

+Every end is one conditional create of `result.json` (I4):

+| End | Who creates the result | When |

+|---|---|---|

+| done, failed, refused | the owner | when the worker ends, after every chunk is confirmed |

+| cancelled | the owner | after it sees `cancel`, stops the worker and confirms its chunks |

+| expired | the console | on a follow call more than 2 minutes after `issued_at` with no `picked`: it creates `picked` (`expired`) first. If that create fails, a box owns the read and the console does nothing. If it succeeds, it creates the result. |

+| lost | the console | on a follow call more than 2 minutes after the owner's deadline with no result |

+If the owner's own result commits first, the console's `lost` create fails

+and the follow call returns the owner's result. If the console's commits

+first, the owner's fails, whenever its request reaches the store; the box

+logs it and stops. The 2 minutes after the deadline is not what makes this

+safe; it only keeps a slow box from being called lost while it is still

+finishing.

+A person whose box died waits up to 22 minutes for DBF-12, and can stop

+waiting with Ctrl-C at any time. Ctrl-C creates `cancel` and exits; the

+read ends whenever the owner sees it, or as `lost`.

+**Expiry and housekeeping.** The lifecycle rule (I2) is the only cleanup in

+the protocol. It is one rule in the `deployment-buckets` module, scoped to

+the `db-reads/` prefix: expire current versions after 1 day, noncurrent

+versions 1 day after that, and expired delete markers. A store without a

+lifecycle rule (a MinIO deployment that has not set one) gets the console's

+housekeeping pass instead: every hour it deletes `db-reads/` objects whose

+read's request is more than 2 days old. That pass is housekeeping, not

+protocol: it touches only reads that ended at least a day earlier, which

+nothing reads, and on an unversioned store it reclaims the space. On a

+versioned store only the lifecycle rule reclaims space, and §11 lists the

+rule as a rollout step.

+**On the box,** the agent holds no scratch at all. The worker's scratch is

+removed by systemd when its unit stops, whatever stopped it (§8.3).

+**Cost.** Each capable box does one LIST of its region's current and

+previous hour of queue entries per serial move and every 30 s, and one GET

+per new in-window entry. A read adds two console PUTs, one `picked`, one

+PUT per chunk, one result, and the CLI's follow GETs. There are no deletes.

+The October bill was LIST and GET polling, so plan 1 measures the LIST rate

+on dev before the flagship roll and reports it.

### 5.9 Replay and forgery

| Attack | What stops it |

|---|---|

| A box writes its own request | No root signature: refused and logged once. |

-| An envelope copied to another id or region key | The id and region are inside the signature and must match the key. |

-| The same envelope picked again | `picked` is a tombstone that outlives the 60 s window plus skew (§5.8), so a second create fails on every box, before or after a restart **(codex 5)**. Past the window the envelope is refused on age. Rev 1's in-memory seen set is gone. Even if one ran, its output is sealed to the original CLI's key. |

+| A box writes its own queue entry for a real request | The box reads the request and requires the queue entry's region, hour and time to match the envelope. |

+| An envelope copied to another id or region | The id and region are inside the signature and must match the keys. |

+| The same envelope picked again | `picked` exists for a day (I1, I2); a second create fails on every box, before or after a restart **(codex 5)**. Past the window the envelope is refused on age. Even if one ran, its output is sealed to the original CLI's key. |

+| A box writes chunks for a read it does not own | If it writes first, the owner's create of that chunk fails, the read-back differs, and the owner ends the read `failed`. If it writes later, its create fails. Either way no foreign bytes are accepted: they cannot match the owner's sealed summary (I5). A compromised box can spoil a read; it cannot change an answer the CLI shows. |

| A box serves a read for another region | It refuses any envelope whose region is not its own `CARLOS_REGION`, and IAM refuses the bucket anyway. |

| A request that names a bucket or prefix | There is no such field. The box derives the prefix (§8.2). |

@@ -422,7 +464,7 @@ the **fence**: after it, the box can write nothing more.

|---|---|---|

| `query` | 1 | This is the regression the customer hit. The result is rows, usually small, and only rows leave the region. |

| `query --at` | 1 | Same path with a target time. The box lists the replica the way the console's door does today (`listReplicaLevel`, `LastModified` as `CreatedAt`), so the copy stops at the same transaction an in-region read would. Plan 1 tests that parity. |

-| `pull` | 2 | The result is the whole database. It rests on provisional ruling 1 (ciphertext of a whole copy passing through the deployment bucket's region, at most 64 MiB at a time), and a measured size cap. In phase 2 the box ships the planned LTX files, sealed and windowed, and the CLI restores and checks them locally with today's code. The box never restores for a pull. |

+| `pull` | 2 | The result is the whole database. It rests on provisional ruling 1, whose bound rev 3 corrects (§16: a whole copy's ciphertext rests in the deployment bucket's region for up to two days), and a measured size cap. In phase 2 the box ships the planned LTX files, sealed, under the same invariants, and the CLI restores and checks them locally with today's code. The box never restores for a pull. |

| `pull --at` | 2 | As `pull`, with the target time. |

| Claim-mode door (`apidbclaim.go`) | not planned | Claims are not active anywhere. The claim door reads exact object versions through `hostops`'s versioned boundary and signed file handles, and lists no times. A box-side version of that belongs to claim activation (#250), which must add times anyway. A host-claim console keeps today's residency refusal, as it refuses `carlos run`. |

| Seeds `--from` (`resolveDBSource`) | not planned | It shares the source checks but restores into a new instance, a different flow. Cross-region seeding is its own design. |

@@ -457,27 +499,42 @@ read whose worker failed to start. A failed probe withdraws the capability

on the next edges-file pass. A socket file whose service is missing,

masked, broken or the wrong version therefore never advertises anything.

+**A probe never competes with reads (codex 9).** The socket allows three

+connections (`MaxConnections=3`): the agent runs at most two reads and at

+most one probe, so a probe always has a connection and a busy box is never

+mistaken for a broken one. The agent counts its own connections, so it

+never opens a fourth. A probe is skipped, not failed, while orphan workers

+block admission (§5.7); the last result stands until it can run. When a

+read's worker fails to start, the agent probes as soon as that connection

+is closed. Rev 2 shared two connections between reads and probes, so two

+reads at once made the probe fail and withdrew the capability.

The console reads `edgesfile.ReadLiveSnapshot` before admitting a read and

refuses up front only on complete evidence **(codex 10)**. That function

returns a nil error when status snapshots cannot be loaded and reports the

gap through `Complete` and `BoxesComplete` instead, so the rule reads those

-fields, never the error alone:

-1. `Complete` is true and no file in the record's region publishes

- `db-read-1` on a non-draining box: DBF-1, no request written.

-2. `Complete` and `BoxesComplete` are both true, capable non-draining boxes

- exist, and none is `Fresh`: DBF-2, no request written.

-3. Anything else (an unreadable edges file, an unreadable or missing status

- snapshot, a fresh snapshot without an edges file): the request is

- written and the pickup window decides. A person is waiting, and the

- expiry answers within 60 s.

+fields, never the error alone. Incomplete evidence comes first and wins

+over both refusals:

+1. `Complete` is false, or `BoxesComplete` is false (an unreadable edges

+ file, an unreadable or missing status snapshot, a fresh status snapshot

+ with no edges file): the request is written and the pickup window

+ decides. A person is waiting, and expiry answers within about 2

+ minutes.

+2. Both are true, and no file in the record's region publishes `db-read-1`

+ on a non-draining box: DBF-1, no request written.

+3. Both are true, capable non-draining boxes exist, and none is `Fresh`:

+ DBF-2, no request written.

+Rev 2 checked `Complete` alone before DBF-1, so a fresh box whose edges

+file was missing could be refused as "no server can read databases".

Any capable, live, non-draining box in the region may serve. There is no

assignment by the console. Region HA (plan 2b, dormant) can put several

boxes in a region, and they all read the same region bucket, so the

create-only `picked` marker is the whole election. The 0 to 500 ms jitter

-spreads the race, and a box with no free slot or below the disk floor does not

-enter it.

+spreads the race, and a box that fails admission (§5.7: no free read, the

+disk floor, or an orphan worker) does not enter it.

### 7.2 Why not the lease holder

@@ -493,8 +550,8 @@ would also tie the read to the instance's lifecycle, which is exactly what

|---|---|

| No capable box in the region | DBF-1 at once |

| Capable boxes, none live | DBF-2 at once |

-| Live but nobody picks in 60 s (all busy, all below the disk floor, agent stuck) | DBF-3 after the window |

-| The picking box dies or restarts mid-read | DBF-12 at the fence, up to 22 minutes after the pick (§5.8) |

+| Live but nobody picks in the window (all busy, all failing admission, agent stuck) | DBF-3, when a follow call creates `expired` about 2 minutes after the request |

+| The owning box dies or restarts mid-read | DBF-12, when a follow call creates `lost` 2 minutes after the owner's deadline, up to 22 minutes after the pick (§5.8) |

## 8. Read-only and isolation on the box

@@ -502,7 +559,7 @@ would also tie the read to the instance's lifecycle, which is exactly what

|---|---|---|---|

-| Agent (`carlos agent`, `carlos-edge.service`) | `carlos` | lists requests; verifies envelopes; creates `picked`; reads the record; derives the prefix; lists the replica and plans the restore from the listing; reads the last planned file's 100-byte header for the size check; streams planned files from the region bucket into the worker; re-checks the record before the query runs; seals and writes output; writes the result | opens SQLite, decodes LTX pages, runs SQL, writes files on disk, reads anything under `/data/instances`, takes or reads the lease |

+| Agent (`carlos agent`, `carlos-edge.service`) | `carlos` | lists requests; verifies envelopes; creates `picked`; reads the record; derives the prefix; lists the replica and plans the restore from the listing; reads the last planned file's 100-byte header for the size check; streams planned files from the region bucket into the worker; re-checks the record before the query runs; seals and writes output; writes the result; keeps the disk reservation table | opens SQLite, decodes LTX pages, runs SQL, writes files on disk, reads anything under `/data/instances`, takes or reads the lease |

| Worker (`carlos system db-worker`, `carlos-dbread@.service`) | a `DynamicUser` | receives the plan and the files over its socket; writes them into its private `/var/tmp` as a litestream file replica; restores to the agent's target transaction; runs `lease.CheckDB`; runs the one statement with today's guards; streams rows back | anything with a network, the box's files, the bucket, other tenants' files |

The rule is: everything that decodes page data or SQL runs in the worker.

@@ -572,7 +629,8 @@ calls the leaser, and never wakes the instance.

`carlos-dbread.socket`, bound by root systemd:

- `ListenStream=/run/carlos-dbread/worker.sock`, `SocketUser=carlos`,

- `SocketMode=0600`, `Accept=yes`, `MaxConnections=2`.

+ `SocketMode=0600`, `Accept=yes`, `MaxConnections=3` (two reads and one

+ probe, §7.1, codex 9).

`carlos-dbread@.service`, one instance per connection, starting from the

claim-bound template's hardening (§3 fact 8) with these differences:

@@ -614,6 +672,18 @@ then `ready` from the worker, then `run` or `stop` from the agent, then

framed output and a final status. The worker refuses a file larger than its

header said and any write past the header's totals.

+**The worker stops when its agent goes (codex 8).** From the moment it

+starts, the worker keeps one goroutine reading its connection. EOF or an

+error there cancels the one context that restore, `lease.CheckDB` and the

+query all run under (each already stops on its context: `db.go`'s

+`runQueryContext`, `internal/lease/check.go`), and the worker exits. A

+worker blocked in a write to a dead connection gets an error at once. So an

+agent restart ends every worker it had within seconds, systemd stops each

+unit and removes its private `/var/tmp`, and only then does the restarted

+agent's orphan count (§5.7) reach zero and admit work. While the agent

+reads the output, it writes nothing to the worker, so the watcher sees

+only EOF or the agent's own `stop`.

The socket and service files are written and enabled by

`ensureSystemUnits`, the same root path that installs the other units, on

the next adopt pass or `update-platform-binary.sh`.

@@ -652,16 +722,32 @@ was being read, twice" case, and is retried the same way.

```

POST /api/cli/apps/{account}/{app}/db/{host}/reads

GET /api/cli/apps/{account}/{app}/db/{host}/reads/{id}

-POST /api/cli/apps/{account}/{app}/db/{host}/reads/{id}/ack

-POST /api/cli/apps/{account}/{app}/db/{host}/reads/{id}/done

POST /api/cli/apps/{account}/{app}/db/{host}/reads/{id}/cancel

```

-All five are `svcrole.Set{}` (people only), behind the same gate order as

-the db door. A read id is checked against its account, app and host on every

-call; a mismatch is the byte-identical 404. A host-claim console answers all

-five with today's residency sentence. `ack` is progress only and never ends

-a read; `done` ends it and requires the result's hash (§5.4, codex 3).

+Rev 2's `ack` and `done` routes are gone with the deletes they drove (I1).

+All three are `svcrole.Set{}` (people only). A host-claim console answers

+all three with today's residency sentence.

+**Two gates, not one (codex 15).** Only the create route runs the db door's

+full gate, which reads the current instance record (class, pooled, fleet,

+moving). The follow and cancel routes run a smaller gate, in the repo's

+order:

+1. a person (the zero service-role set, as the db door);

+2. membership of `{account}`, with the byte-identical 404;

+3. the db token budget (429);

+4. the request: read `db-reads/<id>/request.json`, verify its root

+ signature, and require its account, app and host to be the path's. Any

+ mismatch, a missing request or a bad signature is the same

+ byte-identical 404.

+They never read the instance record. An instance deleted, recreated or

+reclassified after admission is exactly the case where the box writes

+`record-changed`, and the CLI must still be able to fetch that result and

+to cancel. Rev 2 put all routes behind the full gate, which answered those

+calls 404 or 409 instead. This is the run's own split

+(`internal/console/apirunsfollow.go`, `requestOnHost`).

### 9.2 Mixed versions

@@ -683,8 +769,9 @@ detail DBF-21 or DBF-22. It records the SQL's length in bytes and the first

who kept the statement can match it to the row; the row itself holds none

of the data a query's literals can carry. That differs from `carlos run`,

which records argv, and keeps the in-region rule that SQL never reaches the

-audit log. The console keeps the SQL only inside the envelope, which the

-sweep deletes with the tombstone (§5.8).

+audit log. The SQL itself stays in the request object, in the deployment

+bucket, until the lifecycle rule expires it with the rest of the read

+(§5.8), at most two days.

## 10. Errors and copy

@@ -718,7 +805,7 @@ it is written into code.

| DBF-31 | `docs/site/cli.md`, `carlos system`, the worker | `carlos system db-worker runs on platform boxes only. systemd starts it for each forwarded database query. you never run it yourself.` |

DBF-13 is true as written only because the CLI buffers the whole answer and

-checks it before writing anything (§5.4, codex 6).

+checks it before writing anything (§5.5, codex 6).

DBF-30:

@@ -751,6 +838,12 @@ DBF-17 to DBF-20 and DBF-23 to DBF-29 are left free for phase 2's pull rows.

readiness probe passes. `CARLOS_DB_READS=0` withdraws one box.

- **Console:** `CARLOS_DB_FORWARD=1` turns forwarding on. Unset, every

answer is byte-identical to main. The console also needs the root key.

+- **Store:** the `db-reads/` lifecycle rule (I2) is in place on the

+ deployment bucket before any console turns forwarding on. On the flagship

+ that is a change to the `deployment-buckets` module, a release tag, a ref

+ bump in carloku-infrastructure and a targeted apply. That is production

+ infrastructure, so it needs Paul's scoped approval under AGENTS.md's rule

+ before it is applied; plan 2 prepares the plan for that approval.

- **CLI:** none. The forward marker negotiates (§9.2).

- **Version floor:** none beyond the capability. A box that predates it

publishes no capability and is never asked.

@@ -758,17 +851,18 @@ DBF-17 to DBF-20 and DBF-23 to DBF-29 are left free for phase 2's pull rows.

### 11.2 Mixed fleet

A region with one capable box and one old box works: only the capable box

-lists the request prefix. A region with no capable box gets DBF-1. A box

-that loses the capability mid-read finishes the read it holds, within its

-deadline, and takes no new one.

+lists the queue. A region with no capable box gets DBF-1. A box that loses

+the capability mid-read finishes the read it holds, within its deadline,

+and takes no new one.

### 11.3 Roll order

Paul's order (dev boxes, dev console, check, flagship boxes, flagship

-console). Boxes first publish an idle capability. The dev console turns

-`CARLOS_DB_FORWARD=1` and the check is a forwarded query from a dev console

-to a dev box in another region, plus the LIST-rate measurement (§5.8). Then

-the flagship boxes, then the flagship console.

+console), with the lifecycle rule first on each deployment's bucket. Boxes

+publish an idle capability. The dev console turns `CARLOS_DB_FORWARD=1` and

+the check is a forwarded query from a dev console to a dev box in another

+region, plus the LIST-rate measurement (§5.8) and a note of real database

+sizes (ruling 2). Then the flagship boxes, then the flagship console.

### 11.4 Tests

@@ -780,31 +874,37 @@ labelled so; they are not acceptance.

| T1 | A forwarded query end to end: console in region ie, agent in region uk, two MinIO buckets on one server with separate credentials. The console's credentials cannot read the uk bucket (a MinIO policy), so any console read of it fails the test. | real console server, real agent loop, real store; CI runs it with the worker reached through a test socket, labelled non-acceptance |

| T2 | The same on the root runner with the real `carlos-dbread.socket` and `carlos-dbread@.service` installed by the real `ensureSystemUnits`, so the socket is the worker's stdin and stdout as §8.3 says (codex 2) | acceptance gate for plan 1 |

| T3 | Sandbox witness on the root runner, from inside the worker: open `/data/registry.db`, `/etc/carlos/host.env` and another instance's directory; connect to `169.254.169.254:80` and to any TCP address; write outside `/var/tmp`; `ATTACH` and `VACUUM INTO` with the attach limit removed (a mutation); `load_extension`. Every one fails. The private tmp is gone after a normal end, a cancel and `kill -9` of the agent mid-read. | real units |

-| T4 | Forgery and replay: unsigned, wrong key, scoped key, envelope moved to another id and to another region, the same envelope offered again before and after the agent restarts and after the job objects are cleaned (the tombstone holds, codex 5), a chunk sealed to another key, a chunk reordered, the last chunk dropped. | real verify and seal code, real store |

-| T5 | Two capable boxes in one region: exactly one runs each read, across 1,000 reads with random jitter. Expiry against a late pick: one `picked` exists, and the result matches it. | real store CAS |

+| T4 | Forgery and replay: unsigned, wrong key, scoped key, envelope moved to another id and to another region, a forged queue entry, the same envelope offered again before and after the agent restarts (`picked` holds, codex 5), a non-owner's chunk written before and after the owner's, a chunk sealed to another key, a chunk reordered, the last chunk dropped. | real verify and seal code, real store |

+| T5 | Two capable boxes in one region: exactly one runs each read, across 1,000 reads with random jitter. Expiry against a late pick: whichever `picked` create wins decides, and the result agrees with it. | real store CAS |

| T6 | Parity: the forwarded query and the in-region query print byte-identical stdout for a fixture database (NULL, tabs, newlines, blobs, times). `--at` stops at the same transaction as the in-region door for the same replica and time, including a write in the same second as the target. | real restore, both paths |

-| T7 | Ending and cleanup: after done, cancel, expiry, lost and `KeepFor`, nothing remains but the tombstone, and the tombstone goes after 5 minutes. A progress ack that arrives after the result never ends the read, and `done` with the wrong hash is refused (codex 3). Acknowledged chunks are gone while the read is still running, and stored ciphertext never exceeds the window (codex 7). A box whose PUTs stall past the old 90 s mark still owns the read, and nothing it writes lands after the fence (codex 4). | real sweep, real store, a store wrapper that delays PUTs |

+| T7 | The invariants, through the real console and agent against a **versioned** MinIO bucket: no protocol path issues a delete or a replacing write (a store wrapper fails the test on any `Delete` or non-create `Put` under `db-reads/` outside housekeeping) (I1, codex 3, 7, 13, 14); a console restart after a finished read returns the same result (codex 13); a result fetched after the read is long finished still passes the request check (codex 14). | real console, real agent, real store |

| T8 | Mixed versions: the five rows of §9.2, with the forwarding-off 409 compared byte for byte against main. | real console |

| T9 | Record changes between request and pick, during planning, during download and between download and `run`: moved, recreated in the same account, recreated in another account, class changed. Each ends with its code and no chunk is ever written (codex 1). | real records, a store wrapper that pauses the download |

| T10 | Approved copy digests for every row in §10, in the `approved_copy_test.go` pattern. | |

| T11 | CLI buffering: a dropped or bad last chunk and a summary hash mismatch each leave stdout empty (codex 6). | real CLI binary through `carlosTestEnv` |

-| T12 | Two concurrent reads on one box whose free space fits one of them: one runs, the other ends `no-room`; an agent restart mid-read takes no new read until the old worker is gone (codex 8). | real agent, root runner |

-| T13 | Readiness: a present socket with the service missing, masked, failing to start and speaking another protocol version; each withdraws `db-read-1` within one probe and edges pass (codex 9). | real units, root runner |

-| T14 | Console evidence: status snapshots unreadable, one snapshot missing, an edges file unreadable. Each admits the read and lets expiry answer; only complete evidence gives DBF-1 or DBF-2 (codex 10). | real `ReadLiveSnapshot` over a real store |

+| T12 | Disk: two concurrent reads on one box whose free space fits one of them: one runs, the other ends `no-room`. An agent killed mid-read: the restarted agent takes no read and no probe while the old worker unit is active, its worker stops on EOF, and the restarted agent admits work only once the unit and its scratch are gone (codex 8). | real agent, real units, root runner |

+| T13 | Readiness: a present socket with the service missing, masked, failing to start and speaking another protocol version each withdraws `db-read-1` within one probe and edges pass. A box running two reads keeps the capability, and its probe succeeds on the third connection (codex 9). | real units, root runner |

+| T14 | Console evidence: status snapshots unreadable, one snapshot missing, an edges file unreadable, an empty edges listing beside a fresh status snapshot. Each admits the read and lets expiry answer; only complete evidence (both flags) gives DBF-1 or DBF-2 (codex 10). | real `ReadLiveSnapshot` over a real store |

+| T15 | Follow after the instance changes: deleted, recreated and reclassified after admission. The CLI still fetches `record-changed` and can cancel; a follow call for another host, or with a forged request, is the byte-identical 404 (codex 15). | real console, real records |

+| T16 | A late result: a store wrapper that commits the owner's `result.json` on the server after its client has timed out, both before and after the console's `lost` create. Exactly one result exists, the follow page shows it, and the CLI acts on it (codex 4). | real store conditional create |

+| T17 | Lifecycle and housekeeping: the module's lifecycle rule covers `db-reads/` with current and noncurrent expiry (a module test); the housekeeping pass on an unversioned MinIO removes reads whose request is past 2 days and nothing younger. | module test, real store |

Mutation checks that must turn a test red: drop envelope verification (T4);

take the host or prefix from anywhere but the record and config (T1 with a

second tenant); remove `PrivateNetwork` (T3); skip the summary hash check

-(T11); skip check B (T9); delete `picked` with the rest (T4); let `ack` end

-a read (T7); drop the reservation table (T12).

+(T11); skip check B (T9); make the result an ordinary `Put` (T16); delete

+any protocol object (T7); run the full db gate on follow (T15); drop the

+orphan count (T12); give probes no connection of their own (T13).

## 12. Provider boundary

- **Core:** the transport is `release.Store` objects, so it runs on MinIO and

- any S3-compatible store. The box reads each planned file with a GET and

- writes chunks with PUTs; it never uses a server-side copy. Signing is

- ed25519, sealing is the standard library plus `x/crypto/hkdf`. The sandbox

- is systemd, which every box already needs.

+ any S3-compatible store. Every protocol write is a conditional create,

+ which `release.Store.Put` already supports with an empty `ifMatch`. The

+ box reads each planned file with a GET and writes chunks with PUTs; it

+ never uses a server-side copy. Signing is ed25519, sealing is the standard

+ library plus `x/crypto/hkdf`. The sandbox is systemd, which every box

+ already needs.

- **One store endpoint per box (codex 11).** Every S3 client in a box process

takes its endpoint from the one `CARLOS_S3_ENDPOINT`

(`internal/s3compat/s3compat.go`), the deployment store through

@@ -814,12 +914,16 @@ a read (T7); drop the reservation table (T12).

design supports exactly that shape: separate buckets per region, with

separate credentials, on one endpoint. Regional data on separate store

servers is not something any box path supports today, and adding

- per-store endpoints is not this design's job (§18). Rev 1's claim that

- region stores "may be different servers" was wrong and is withdrawn.

+ per-store endpoints is not this design's job (§18).

+- **Expiry.** The lifecycle rule is S3 lifecycle configuration, which MinIO

+ supports as ILM. A store with no lifecycle rule falls back to the

+ console's housekeeping pass (§5.8), which reclaims space on an unversioned

+ store. On a versioned store only a lifecycle rule reclaims space; that is

+ a deployment requirement, stated in §11.1, not something the code can

+ detect.

- **AWS adapter:** IAM is what makes the console unable to read another

- region's bucket on the flagship, and the optional lifecycle rule (§5.8).

- Neither is required for the design to work. A self-hosted deployment gets

- the same split from per-bucket credentials on its store.

+ region's bucket on the flagship. A self-hosted deployment gets the same

+ split from per-bucket credentials on its store.

- **What a non-AWS deployment loses:** nothing it has today. The capability,

the liveness join and the box's identity use no AWS API. Multi-box regions

are AWS-only today (H26), but nothing here depends on them; one box per

@@ -842,9 +946,9 @@ proposes, for a ruling on #280:

|---|---|---|

| `db.go` `runQueryContext` (main: `legacy`, `rev3=J4`, `basis=root-input`, `plan14-root-reads.tsv:202`) | moved out of `db.go` | retired through the overlay's removal row, the form the H3 carlos-run removal used |

| `internal/dbquery` `RunContext` (the moved body, unchanged) | the CLI on a member's machine, as the member; the worker on a box, as its `DynamicUser`. Neither is root, and neither decides anything for root | `legacy`, `accepted-legacy;rev3=<ruling>;basis=root-input`, the J4 footing main gave the same body |

-| agent db-reads loop: the request listing, the record reads of checks A and B, the replica listing and GETs | the agent, `carlos`, `carlos-edge.service` | `legacy` with J1 (record and route reads) and J4 (object input), as the offline move's agent units took |

-| worker: the scratch writes and reads, the file replica restore, `CheckDB`, the query | `carlos-dbread@`, `DynamicUser` | `legacy` with J4 |

-| agent readiness probe and reservation table (`Statfs` on `/var/tmp`, `systemctl list-units`) | the agent, `carlos` | `diagnostic` for the probe; `legacy` J4 for `Statfs` |

+| agent db-reads loop: the queue listing, the request read, the record reads of checks A and B, the replica listing and GETs | the agent, `carlos`, `carlos-edge.service` | `legacy` with J1 (record and route reads) and J4 (object input), as the offline move's agent units took |

+| worker: the scratch writes and reads, the file replica restore, `CheckDB`, the query, the connection watcher | `carlos-dbread@`, `DynamicUser` | `legacy` with J4 |

+| agent readiness probe and admission (`Statfs` on `/var/tmp`, `systemctl list-units`) | the agent, `carlos` | `diagnostic` for the probe; `legacy` J4 for `Statfs` and the unit listing |

The overlay re-pins the source envelopes of every changed package (`.`,

`internal/dbquery`, `internal/dbdoor`, `internal/host`, `internal/dbreads`)

@@ -857,9 +961,11 @@ named.

root-written artifact and joins the Plan 14 write census as an overlay, as

every new unit has.

- **9B8 record-write census.** No record writer is added. The new `Put`

- sites (request, `picked`, chunks, ack, cancel, result) are `nonrecord`.

- `registerCLIRoutes` changes as a `key-site` for the five routes.

-- **Claim routes.** The five routes join

+ sites (request, queue entry, `picked`, chunks, cancel, result) are

+ `nonrecord`, all conditional creates. The housekeeping pass's `Delete` is

+ `nonrecord`. `registerCLIRoutes` changes as a `key-site` for the three

+ routes.

+- **Claim routes.** The three routes join

`internal/console/testdata/claim-routes.txt`, refused on a host-claim

console like the run's.

- **go.mod.** No new module, so no root-envelope overlay for it.

@@ -867,24 +973,26 @@ named.

## 14. Plans

1. **Box side and wire, dormant.** `internal/dbreads` (ids, keys, the

- envelope, sealing, the result, limits, the fence); `internal/dbquery`

- (the query runner and guards moved out of `db.go`, with the CLI switched

- to it and no behaviour change); `dbdoor` planning over any

- `replicaReader`; the agent's db-reads loop with the identity bracket, the

- reservation table and the readiness probe; `carlos system db-worker` and

- its protocol; the two units in `ensureSystemUnits`; the `db-read-1`

- capability. Gates: T2, T3, T4, T5, T9, T12, T13, the parity half of T6

- that needs no console. Census overlay. Nothing reaches a box unless a

- console asks.

-2. **Console and CLI for `query` and `query --at`.** The five routes, the

- forward marker, the evidence rule for capability and liveness, the

- bounds, chunk deletion on ack, expiry, lost and the tombstone in the run

- sweep, audit, the CLI's buffered forwarded path, DBF copy,

- `docs/site/cli.md`. Gates: T1, T6, T7, T8, T10, T11, T14, copy review,

+ envelope, sealing, the result, limits); `internal/dbquery` (the query

+ runner and guards moved out of `db.go`, with the CLI switched to it and

+ no behaviour change); `dbdoor` planning over any `replicaReader`; the

+ agent's db-reads loop with the identity bracket, conditional creates,

+ admission (reservations and the orphan count) and the readiness probe;

+ `carlos system db-worker`, its protocol and its connection watcher; the

+ two units in `ensureSystemUnits`; the `db-read-1` capability. Gates: T2,

+ T3, T4, T5, T9, T12, T13, T16, the parity half of T6 that needs no

+ console. Census overlay. Nothing reaches a box unless a console asks.

+2. **Console and CLI for `query` and `query --at`.** The three routes and

+ their two gates, the forward marker, the evidence rule for capability

+ and liveness, the bounds, lazy expiry and lost, the housekeeping pass,

+ audit, the CLI's buffered forwarded path, DBF copy, `docs/site/cli.md`,

+ and the `deployment-buckets` lifecycle rule with the approval plan for

+ applying it. Gates: T1, T6, T7, T8, T10, T11, T14, T15, T17, copy review,

census overlay. Roll per §11.3.

-3. **`pull` and `pull --at`.** On provisional ruling 1. Sealed LTX files

- with the same window and chunk deletion, a raw chunk route for 4 MiB

- chunks, local restore and `CheckDB` in the CLI, and the pull copy rows.

+3. **`pull` and `pull --at`.** On provisional ruling 1, re-confirmed with

+ rev 3's storage bound (§16). Sealed LTX files under the same invariants,

+ a raw chunk route for 4 MiB chunks, local restore and `CheckDB` in the

+ CLI, and the pull copy rows.

After phase 3, in-region reads move to the box path too (provisional ruling

5). That is a fourth plan, written once phase 3 has run on the flagship. It

@@ -906,14 +1014,16 @@ through it unchanged.

output written before check B.

3. **Forgery and replay.** An unsigned or wrongly scoped request accepted,

domain separation from other root envelopes missing, the region not

- bound, or the tombstone removed before the acceptance window closes.

-4. **Ending.** A write after the fence, a progress ack taken for the end, a

- result before its last chunk, two owners of one read, ciphertext kept

- past the window, reservations that leak or double count.

-5. **Drift between the two paths.** The forwarded `--at` stopping at a

- different transaction than the in-region door, output formatting that

- differs between CLI and box versions, or a capability published without

- a passing readiness probe.

+ bound, or a queue entry trusted without its request.

+4. **An invariant broken by a convenience.** Any protocol write that is not

+ a conditional create, any protocol delete, a result accepted from

+ someone other than the owner, or a follow route that reads the current

+ record. Each one reopens a finding that I1 to I7 closed.

+5. **Drift between the two paths, or admission that lies.** The forwarded

+ `--at` stopping at a different transaction than the in-region door,

+ output formatting that differs between CLI and box versions, a

+ capability published without a passing probe, or reservations that

+ count a worker as gone before its unit has stopped.

## 16. Open questions: provisional rulings awaiting Paul

@@ -922,17 +1032,25 @@ The coordinator answered rev 1's five questions provisionally on

confirms or changes it.

1. **Pull through the deployment bucket.** Phase 2's pull may pass

- ciphertext through the deployment bucket (eu-west-1 on the flagship), at

- most 64 MiB at a time (16 unacknowledged chunks of 4 MiB, deleted as they

- are acknowledged) and for at most 1 hour. **Paul's call:** if UK data

+ ciphertext through the deployment bucket (eu-west-1 on the flagship).

+ **The bound in that ruling no longer holds and must be re-confirmed

+ before plan 3.** The ruling said "at most 64 MiB at a time for at most 1

+ hour". Round 2 (codex 7) showed that bound was never real: the bucket is

+ versioned, so deleting chunks left them as noncurrent versions. Rev 3

+ deletes nothing, and the true bound is: a pull's whole ciphertext (up to

+ its size cap) rests in the bucket until the lifecycle rule removes it,

+ at most two days. The alternative that keeps a short window is

+ version-aware deletes, which need `s3:DeleteObjectVersion` on the

+ deployment bucket for the console's role, a new IAM grant and the

+ deletes rev 3 removed. Phase 1 is unaffected: a query's ciphertext is at

+ most 64 MiB of rows. **Paul's call** in either case: if UK data

residency is a legal requirement rather than a technical one, whether

- encrypted data in transit through an Irish bucket is acceptable is a

- question for him, not for engineering.

+ encrypted data resting in an Irish bucket is acceptable is for him.

2. **Limits stand.** 2 GiB restored, 5 minutes, 64 MB of rows, two reads per

box. Real database sizes on the flagship are still unmeasured; plan 2's

dev check reports them.

3. **Audit.** The row records the SQL's length and a SHA-256 prefix, not its

- text (§9.3).

+ text (§9.3). The SQL itself rests in the request object until expiry.

4. **Copy casing.** New strings use Paul's voice rule: lower case after a

full stop (§10). The existing db door strings that capitalise are left

as they are; changing them is a separate copy review.

@@ -943,15 +1061,17 @@ confirms or changes it.

- No console per region and no console-to-box connection.

- No change to in-region reads (until the fourth plan), the claim-mode

door, seeds `--from`, fleet or pooled instances.

-- No IAM change. No new read authority for any role.

+- No IAM change. No new read authority for any role. The one infrastructure

+ change is a lifecycle rule on the deployment bucket.

- No per-store S3 endpoints.

- No `carlos db shell` and no browser view.

## 18. Declined review findings

-Round 1 (`CODEX-FINDINGS-DB-REGION-FORWARD.md`). Every citation was checked

-against `main` at `5f341f33` and holds. Findings 2 to 10 and 12 are accepted

-in full. Two are accepted in part:

+**Round 1** (`CODEX-FINDINGS-DB-REGION-FORWARD.md`). Every citation was

+checked against `main` at `5f341f33` and holds. Findings 2 to 10 and 12 are

+accepted in full. Two are accepted in part, and round 2 agreed with both

+declines:

- **Codex 1, the "stronger guarantee" (pin object versions to the checked

incarnation): declined.** The identity bracket (§8.2) is accepted and

@@ -963,12 +1083,20 @@ in full. Two are accepted in part:

claim boundary. That is a new mechanism for a case the bracket already

removes. Claim activation, which brings versioned reads, can revisit it.

- **Codex 11, independently configured deployment and regional-data

- endpoints: declined.** The finding is right that rev 1 overclaimed, and

- §12 now says so. But no box path (wake, moves, this one) supports

+ endpoints: declined.** No box path (wake, moves, this one) supports

regional data on a separate store server today, because every S3 client

- in the process reads one `CARLOS_S3_ENDPOINT`. Adding per-store endpoints

- for db reads alone would make this the only path that works in a shape

- the rest of the box does not. The simpler rule is to state the supported

- shape (one endpoint, separate buckets and credentials) and test it (T1).

- Per-store endpoints, if wanted, are a platform-wide change with its own

- design.

+ in the process reads one `CARLOS_S3_ENDPOINT`. The spec states the

+ supported shape (one endpoint, separate buckets and credentials) and

+ tests it (T1). Per-store endpoints, if wanted, are a platform-wide change

+ with its own design.

+**Round 2** (`CODEX-FINDINGS-DB-REGION-FORWARD-R2.md`). Every citation was

+checked and holds. Nothing is declined. Findings 4, 7, 13 and 14 are closed

+by the lifecycle invariants (§5.2), not by patching each case, and the

+round-2 fix for finding 3 (separate `ack` and `done`) is superseded: with

+no deletes there is nothing for an acknowledgement to trigger, so both

+routes are gone. Findings 8, 9, 10 and 15 are fixed directly (§5.7, §8.3,

+§7.1, §9.1).

+One consequence is not a decline but needs a decision: closing finding 7

+honestly invalidates the bound in provisional ruling 1 (§16).