carlos db for instances whose data lives in another region

Status: rev 7, 2026-10-23. Rev 7 (plan 3, 2026-10-23): the pull's cap and chunk size, from a measurement on dev (§5.7); the request bound is unchanged. Rev 6, 2026-10-22, changes no mechanism: it states four things the plan review (CODEX-FINDINGS-DB-REGION-FORWARD-PLANS.md) found the plans had to decide, each marked (plan review N): what a result names when a chunk cannot be confirmed (§5.4, 9), a box's cancel read before it elects (§5.5 step 4, §5.8, 11), a retry's reservation (§5.7, 10), and the order of the audit row, the creates and the serial bump, with nothing but cancel once the request has ended, checked before each write (§5.5 step 3, §9.3, 11 and 28). Round 2 of the plan review added DBF-32 (§10) for SQL that JSON would rewrite, and the console's refusal of it (§5.5 step 3, 36). Rev 5 answers Astra's round 4 (CODEX-FINDINGS-DB-REGION-FORWARD-R4.md), which found J1 to J5 fixed, with four direct fixes and no new mechanism: flushed progress lines and the backhaul header bound (codex 19), the result held until its chunks are out (codex 20), restart reconciliation by refusal and a 5 s worker exit (codex 8), and the retention size wording (codex 7). Design only, nothing built. Written for Paul, who approved the direction (§2). Astra accepted rev 5 in round 5, and Paul approved building it on 2026-10-22, with the rulings in §16 (ruling 1: ship through the deployment bucket, per-region bucket as plan 5). Review brief: CODEX-REVIEW-BRIEF-DB-REGION-FORWARD.md. Rev 1 was b35b3ab1, rev 2 2cf4a88f, rev 3 944a61fd, rev 4 c07629e4. Rev 4 answered Astra's round 3 (CODEX-FINDINGS-DB-REGION-FORWARD-R3.md) by making a forwarded read one synchronous HTTP request (§5.2): no follow, no cancel route, nothing the console writes after the request ends, and the store's lifecycle rule as the only deletion. Five invariants (§5.3) replace rev 3's seven, and every finding is mapped to the invariant that closes it or to the synchronous design that removes it. Each change is marked (codex N) where it lands, and §18 lists what was declined. The coordinator's provisional answers to the open questions are now Paul's rulings (§16, 2026-10-22).

Companion specs: 2026-09-02-carlos-db-design.md (the db door, cited below as "db spec"), 2026-10-08-carlos-run-design.md (the run transport, "run spec"), 2026-09-11-isolated-exec-design.md (the tenant unit sandbox).

1. The problem

carlos db query, carlos db pull and their --at forms restore a copy of an instance's database on the member's own machine. The CLI fetches the replica's LTX files by coordinate through the console's db door (internal/console/apidb.go). The console reads them from the data bucket of its own region.

The console refuses every instance whose record Region differs from its own CARLOS_REGION (resolveReplica, ErrResidency, dbResidencyRefusal). That refusal is correct for the console: each region's box role is granted its own region's data bucket and no other (infra/modules/region-host/main.tf, host_buckets), and the console runs on the ie box.

On 2026-10-21 all twelve instances of one customer account moved from ie to uk with the offline operator move. The customer can no longer run carlos db query against any of them. Every instance in a region other than the console's has the same gap, and self-serve moves and nearest-region placement will make that the common case.

2. The direction, and what it changes

Paul approved this direction: the console forwards a db read to a box in the region where the data lives. That box reads its own region's replica, runs the read-only operation there, and returns only the result. The console never reads another region's bucket. There is no console per region.

Two earlier rulings bend under it, and this spec says so rather than working around them:

• db spec §9, "no path that runs anything on the box". A forwarded query runs member SQL on a box. §8 is how that stays safe.
• db spec §6 rejected "tenant database bytes in the deployment bucket". A forwarded read has to come back to the console through the deployment bucket (§5.1). This spec puts only encrypted bytes there, sealed to a key that exists only in the member's CLI (§5.6), until the store's lifecycle rule expires them (§5.7). In phase 1 that is only a query's rows. Phase 2's pull would send the planned replica files, which are a pull's result, under ruling 1 (§16, Paul, 2026-10-22). A per-region bucket that keeps this ciphertext in the data's region is a follow-up (plan 5, §14).

One earlier ruling holds unchanged: the run spec's "the console still never connects to a box". The transport is the run's.

3. What exists today (facts this design rests on)

1. The CLI restores locally. db.go restores the replica through dbdoor.RestoreAsOf over a read-only client (internal/dbdoor/client.go) into a private 0700 directory, checks it with lease.CheckDB, then either copies it to --out or runs one query. The query runs with mode=ro, query_only, temp_store(2), SQLITE_LIMIT_ATTACHED 0 and the one-statement and pragma guard checkQuerySQL. The console never executes tenant SQL.
2. --at depends on listing times. litestream's timestamp plan keeps a file only if its CreatedAt is before the target. The legacy door lists S3 LastModified, which has second resolution, so the target is truncated to its second. The claim-mode door lists no times, and --at refuses there (dbdoor.ErrNoTimes).
3. carlos run already reaches any region's boxes. Its transport is objects in the deployment bucket under the app's subtree (internal/runs): request.json, out/<seq> chunks of 256 KiB, result.json, written create-only where it matters. The console bumps control/serial; boxes poll it every 2 s. The console's follow door pages chunks to the CLI, and its sweep (internal/console/runsweep.go) expires, closes lost runs, and deletes old ones. A run is admitted with a CAS on the instance record, because a run needs the instance to itself.
4. The deployment bucket is writable by every platform box, so an unsigned control object is a request any box could have written. The host-retry grant (internal/release/hostretry.go) answers that with a root-signed envelope: release.SignRootEnvelope on the console, VerifyRootEnvelope against CARLOS_RELEASE_PUBKEY on the box. Scoped release keys are refused.
5. Sealing to a public key exists. internal/appconfig/crypto.go seals with ECDH P-256, HKDF-SHA256 and AES-256-GCM, with bound associated data (the v2 envelope), using only the standard library and golang.org/x/crypto/hkdf, which go.mod already carries.
6. Boxes publish capabilities. control/edges/<box>.json carries capabilities (region-ha-1, offline-move-1, offline-stage-1). edgesfile.ReadLiveSnapshot joins each file with the box's status snapshot for liveness. The offline move's doors refuse a region holding a box without the capability, and each box publishes the capability only when its armed worker can act on it.
7. The box role can read its region's replicas. In carlosframework/carloku-infrastructure, region-uk.tf passes data_bucket_arn = aws_s3_bucket.data_uk.arn to the region-host module (ref v0.21.0; ie is at v0.28.8). At both refs host_buckets grants s3:GetObject on data_bucket_arn/* and s3:ListBucket on the bucket, plus read and write on the deployment bucket, which stays in eu-west-1 and is shared by every region. Sibling regions get push-only moves/*. So a uk box can read uk replicas, and nothing else can. No IAM change is needed for this design.
8. Tenant processes already run sandboxed. The claim-bound unit template (internal/host/claimunit_template.go) and the carlos-app@ units use ProtectSystem=strict, TemporaryFileSystem=/data:ro, InaccessiblePaths=/etc/carlos, PrivateTmp, NoNewPrivileges, an empty capability set and an IPAddressDeny list that covers the metadata service. carlos-app@ sockets are bound by root systemd (SocketUser=carlos), so the agent can start a sandboxed unit with no privilege of its own by connecting to its socket.
9. The agent runs as carlos (carlos-edge.service, User=carlos). The console runs as carlos-console.

4. Decisions

| # | Decision | Section | |---|---|---| | D1 | Phase 1 forwards query and query --at. pull and pull --at forward in phase 2, under ruling 1 (§16). The claim-mode door and the seeds --from door stay refused. | §6 | | D2 | Any live box in the record's region that publishes db-read-1 and is not draining may serve. The first to create the read's picked marker serves it. The instance's lease holder gets no preference. | §7 | | D3 | Transport is the run's objects in the deployment bucket and a serial bump, but the read is one synchronous HTTP request: the console waits on the box's objects inside it and streams them to the CLI. The request is a root-signed envelope. No record write. Every object is a create-only write under db-reads/, never replaced or deleted by the protocol; the store's lifecycle rule is the only deletion (§5.2, §5.3). | §5 | | D4 | Results are sealed to an ephemeral key the CLI makes for each read. The console and the deployment bucket only ever hold the result as ciphertext. The console does see the SQL, as the envelope's signer. | §5.6 | | D5 | The box splits the work. The agent (carlos) verifies, resolves, plans and streams objects. Everything that decodes tenant bytes (LTX, SQLite, the member's SQL) runs in a new sandboxed unit, carlos-dbread@.service, as a DynamicUser with no network. | §8 | | D6 | The worker gets the planned LTX files and nothing else. Its scratch is its own private /var/tmp, which systemd removes when the unit stops. | §8.3 | | D7 | Rollout: boxes publish db-read-1; the console forwards only with CARLOS_DB_FORWARD=1 and a capable region. Dormant by default. | §11 | | D8 | Three plans here. Two follow-ups are listed, not written: plan 4 moves in-region reads to the box path after phase 3 (ruling 5); plan 5 moves this traffic to a per-region bucket (ruling 1). | §14 |

5. Transport and authentication

5.1 Why this transport

Three candidates exist in the tree:

• Deployment-bucket objects (carlos run, restart scope, the offline move's stager and sealer reports). Established, provider-neutral, works for a box with no inbound path, and already crosses regions.
• Edge backhaul (internal/edge/homemap.go, mesh.go). A box-to-box HTTPS hop for tenant traffic. Its HMAC key proves only "a box in this deployment" and is deliberately allowed to buy nothing but X-Forwarded-For belief. Using it for db reads would give that key a new authority across the IAM region boundary, and the console would start connecting to boxes, which the run spec ruled out.
• The fleet store door (internal/edge/storedoor.go). Serves bucket objects to fleet boxes on the tunnel hostname. Wrong direction, wrong principal.

Decision: deployment-bucket objects. The cost is latency (a pickup is seconds, not milliseconds) and a LIST per serial move on capable boxes (§5.8). Both are acceptable for a command a person runs by hand.

5.2 One read, one request

Rounds 1 to 3 kept finding defects in one place: rev 1 to rev 3 made a read a durable object with a life of its own. It could be followed, cancelled, expired or declared lost by the console after the fact, and its admission slots outlived the request that took them. Every rule added to manage that life made a new edge (codex 3, 4, 5, 7, 13, 14, 16, 17). Rev 4 removes the life: a forwarded read is one synchronous HTTP request. The CLI's request to the console stays open for the whole read, the console waits on the box's objects inside that request, and when the request ends the read is over for everyone. Nothing resumes it.

Why this is possible (the numbers). Nothing between the CLI and the console cuts a long request:

• The edge. The public listeners set ReadHeaderTimeout only, and say why: "No ReadTimeout/WriteTimeout, those would cut long-lived streams" (internal/edge/edge.go:3354 to 3358). The reverse proxies to a unix or TCP upstream, which is how the console is reached, set no ResponseHeaderTimeout and no overall timeout, only a 90 s idle bound on pooled connections (unixProxy, tcpProxy, edge.go:2377 to 2421).
• The console. cmd/console/main.go:586 to 590 sets ReadHeaderTimeout 10 s and ReadTimeout 5 minutes, and no WriteTimeout. ReadTimeout bounds reading the request; the read door reads a body of a few kilobytes and then, as the import door does (internal/console/apiinstanceimport.go:479 to 483), sets its own read deadline through http.ResponseController so the connection is held for the read's whole bound and no longer.
• The CLI. The ordinary console client has a 30 s timeout (auth.go:403), which is why carlos ship already uses its own context of 15 minutes (apimode.go:4425). The forwarded read does the same: its own client with no overall timeout and a context of 23 minutes.
• Streaming precedent. The db door already streams a long response with a per-chunk write deadline and an idle watchdog (handleAPIDBGet, dbStream, internal/console/apidb.go).

The longest a read can hold the request is the pickup window (2 minutes with skew) plus the owner's deadline (20 minutes), so 22 minutes, and the CLI's 23-minute context is the outer bound. To keep an idle connection alive through home routers and NATs that drop silent TCP, the console writes and flushes a progress line at least every 15 s (§5.5, codex 19).

5.3 The invariants

| # | Invariant | |---|---| | J1 One request | A read exists for the console and the CLI only inside one open HTTP request. When the request ends, by a result, a timeout, a cancel or a dropped connection, the read is over: no route can follow, resume, cancel or end it later, and the console writes nothing about it afterwards except one best-effort cancel. | | J2 Write once | Every object is a conditional create (If-None-Match: *, release.Store.Put with an empty ifMatch) and is never replaced or deleted by the protocol. The console writes request.json, the queue entry and cancel; boxes write picked, chunks and result.json. | | J3 One owner | The first create of picked owns the read. A box whose create answers an error or a conflict reads picked back and owns the read only if it carries this box's own attempt nonce. Only the owner runs the read. | | J4 The store's lifecycle rule is the only deletion | A rule on the db-reads/ prefix expires every object. It acts long after J1 has ended every reader, so it never races one. | | J5 Admission is held by what does the work | The console's per-person and per-console counts are held by the open request and released when its handler returns. A box's read slot and disk reservation are held until its worker unit has stopped. |

What closes each open or earlier finding:

| Finding | Closed by | |---|---| | codex 3 (progress ack read as the end) | removed by the synchronous design: there are no acks | | codex 4 (late writes after lost) | removed by the synchronous design and J1: the console never writes lost; a box's late chunks or result are objects nobody reads, and J4 removes them | | codex 5 (cleanup reopens the election) | J2, J4: picked is never deleted by the protocol, and the request is refused on age long before the lifecycle rule acts | | codex 7 (stored ciphertext) | J4, with the honest timing in §5.7 | | codex 8 (disk) | J5 | | codex 13 (completion deletes the evidence a read ended) | removed by the synchronous design: nothing needs that evidence after the request ends | | codex 14 (cleanup removes the identity to follow) | removed by the synchronous design: there is no follow | | codex 15 (the db gate on follow routes) | removed by the synchronous design: there are no follow routes. The one route runs the full gate once, at the start, and a record-changed result comes back in the same response | | codex 16 (lazy expiry stranded between two writes) | removed by the synchronous design (the console never writes picked or a result), and J3 for the box's own ambiguous picked | | codex 17 (abandoned reads hold admission) | J5: an abandoned read is a closed connection, and the handler's return releases its counts | | codex 18 (the forgery claim contradicts sealing) | withdrawn: §5.6 states the real trust boundary |

5.4 Objects

All objects live under db-reads/, so one lifecycle rule covers them (J4). <id> is dbr_ and 26 lowercase base32 characters (128 random bits), minted the way runs.MintID mints run ids, with its own ValidID. Every object is create-only (J2).

| Key | Writer | Purpose | |---|---|---| | db-reads/<id>/request.json | console | The request: a root envelope (§5.5). | | db-reads/queue/<region>/<YYYYMMDDHH>/<unix>-<id> | console | An empty pointer, written after the request. Boxes of <region> list the current and previous hour's prefix and skip any entry whose <unix> (the request's issued_at) is outside the pickup window without reading it. | | db-reads/<id>/picked | a box | The owner (J3): box name, boot id, a random attempt nonce and the owner's deadline. | | db-reads/<id>/out/<seq> | the owner | One sealed chunk of output (§5.6). | | db-reads/<id>/result.json | the owner | How the read ended. Written after every chunk is confirmed. | | db-reads/<id>/cancel | console | Best effort, after the request ended without a result, so the owner stops early. Never read by the console. |

Rev 2 kept these under apps/<acct>/<app>/ for app purge; one prefix is what lets one rule expire everything, and the rule is what removes them.

There is no write to the instance record. A run takes a record CAS because it needs the instance to itself. A db read touches neither the instance nor its lease, needs no exclusivity, and is frequent.

Ambiguous creates on the box. A box whose picked create answers an error or a conflict reads picked back: its own nonce means it owns the read, anything else means it steps away, and a read that still fails is treated as not owned (J3, codex 16). The box keeps each sealed chunk's bytes until its create is confirmed; a retry that answers "already exists" reads the object back and compares bytes, equal meaning its earlier attempt landed. It writes result.json only when every chunk is confirmed. A chunk it cannot confirm before its deadline is never named: the box stops the read and writes a failed result with interrupted whose last_seq is the last confirmed chunk, so the result names only objects that exist (plan review 9).

5.5 The request and its response

The envelope is unchanged from rev 3:

{
  "purpose": "carlos-db-read-v1",
  "v": 1,
  "id": "dbr_...",
  "account": "<sqid>",
  "app": "shop",
  "host": "shop.example.com",
  "incarnation": "<hostclaim.Incarnation of the record>",
  "region": "uk",
  "op": "query",
  "sql": "SELECT count(*) FROM orders",
  "at": "2026-10-07T14:00:00Z",
  "reply_key": "<base64 P-256 public key, uncompressed>",
  "by": "member@example.com",
  "issued_at": "2026-10-22T09:00:00Z"
}

• Signed with release.SignRootEnvelope under the console's root key. A console that holds only a scoped key cannot sign, and so cannot forward: it answers with today's residency sentence.
• purpose is a fixed string so the envelope cannot be confused with any other root-signed payload. The box decodes it into the exact type with VerifyRootEnvelope, which requires canonical JSON.
• The box accepts an envelope only when id matches the key it was read from and the queue entry that named it, region matches the queue key and the box's own CARLOS_REGION, and issued_at is no more than 60 s old and no more than 60 s in the future.
• at is empty for the newest copy. op is query in phase 1; phase 2 adds pull.
• The envelope carries no bucket, no prefix and no key. The box derives the replica prefix itself (§8.2).

The sequence.

1. The CLI calls the existing list door, as today. A new console with forwarding on answers a capable cross-region instance with 409 and an additive body: {"error": DBF-14, "forward": 1, "region": "uk", "forward_ops": ["query"]}. An old CLI prints the sentence. A new CLI switches to the forwarded form.
2. The CLI runs checkQuerySQL locally, makes an ephemeral P-256 key, and sends one request, POST /api/cli/apps/{account}/{app}/db/{host}/reads, with op, sql, at and reply_key.
3. The console runs dbGate's order (person only, membership with a byte-identical 404, the db budget, configuration, the record, the class rail, pooled, fleet), refuses a moving record, checks checkQuerySQL again, checks capability (§7.1), and takes its admission counts (J5). Any refusal so far is an ordinary status and JSON error, and so is SQL holding U+FFFD, which is what JSON decoding makes of bytes that are not UTF-8 (the CLI refuses such SQL first, with DBF-32) (plan review 36). Then it mints the id, writes the audit row (§9.3), creates the request and the queue entry, bumps the serial, and starts a 200 response of newline-delimited JSON. From here every outcome is a line in that body. The console checks that the request is still open immediately before each of those four writes, and the serial bump runs on the request's own context. A request that ends before request.json leaves at most the audit row; one that ends after it leaves request.json, perhaps the queue entry, and cancel, and never a serial bump or a 200 (plan review 11, 28, 33).
4. Each capable box lists its region's queue for this hour and the last when the serial moves, and every 30 s if it does not. For an in-window entry it has not decided, it reads and verifies the request, skips it if cancel exists (plan review 11), skips it while the box is draining (§7.1), checks its own admission (§5.7), waits a random 0 to 500 ms, checks draining again, and creates picked.
5. The owner runs §8.2's identity bracket around planning and download, runs the query in a worker, and writes sealed chunks as the output arrives. It checks for cancel every 5 s. When the worker ends, it confirms every chunk and creates result.json.
6. The console, inside the same request, reads picked until it appears. Then, about once a second, it reads result.json first and lists out/ second, the order the run's follow door uses for the same reason (internal/console/apirunsfollow.go:101 to 106): a listing taken before the result could miss a chunk the box confirmed just before writing it (codex 20). It writes to the response, in order: {"admitted": <id>} at once, {"picked": <box>} once, {"chunk": <seq>, "data": <base64 sealed bytes>} for each chunk in sequence order with no gaps, {"wait": <phase>} whenever 15 s pass with nothing else to send, and finally {"result": <result.json>}. - The result line comes last, and only when complete (codex 20). It is written only after every chunk from 1 to the result's open last_seq has been written to the response. A result whose chunks are not all listed yet is held, and the console keeps listing once a second until they are, or until the request's own deadline ends it with DBF-12. The console cannot open the sealed summary; the CLI still checks the bytes against it (step 8). - Every line is flushed (codex 19). Each line is written under a fresh write deadline and then pushed out with http.ResponseController.Flush, under the same deadline; a write or flush error ends the request as a stalled client (step 7). Go buffers small writes in the response writer, and dbStream flushes only at the end, so without an explicit flush the wait lines would sit in the console for minutes. The response's first line, admitted, is flushed with the status and headers, so the headers leave at once. - Nothing downstream buffers it. The response is Content-Type: application/x-ndjson with Cache-Control: no-store and no Content-Length. The edge's compressor does not gzip that type (compressibleType, internal/edge/compress.go:128), and its gzip writer passes Flush through when it does compress (compress.go:319). The edge's reverse proxy flushes a response with no Content-Length immediately (httputil.ReverseProxy), and no-store keeps the cache recorder out. - Backhaul. A request that reaches the console through another region's edge travels over backhaul, whose transport waits at most 30 s for response headers (backhaulHeaderTimeout, internal/edge/edge.go:2827, used at 2958). So the console bounds everything before the 200 (the gate, the record read, the capability read and the two creates) with a 20 s context, refusing with today's "could not read the instance. Try again." on overrun, and flushes the 200 and admitted the moment the creates land. After the headers, backhaul sets no further timeout.
7. The console ends the response without a result line, and the CLI says DBF-3 or DBF-12, when: - no picked has appeared 2 minutes after issued_at (DBF-3); - the owner's deadline plus 1 minute passes with no result (DBF-12); - the CLI's connection closes, or a write to it stalls for 30 s. In each case the console then creates cancel, best effort, and returns. Its admission counts are released by the return (J5).
8. The CLI buffers every chunk (codex 6). On a done result it opens the sealed summary, checks the buffered bytes against its count and SHA-256, and only then writes the rows to stdout and today's status line to stderr. Any other result maps to its sentence (§10). A response that ends without a result prints nothing to stdout.

Ctrl-C closes the connection. The console sees its context end, creates cancel and returns; the owner, if any, stops within 5 s. There is no cancel route.

A console restart ends every read in flight: their connections close, and each CLI says DBF-12. Nothing on the console side needs recovering, because nothing outlived the request (J1). A box keeps running a read whose console went away until it sees cancel or its own deadline; its output is read by nobody and expires (J4).

5.6 Sealing, the result and the trust boundary

• One ephemeral P-256 key pair per read, made by the CLI and held only in its memory. The public half travels in the request.
• Each chunk is sealed with the appconfig v2 construction (ECDH P-256, HKDF-SHA256, AES-256-GCM, a fresh ephemeral sender key per seal), with its own HKDF info string, carlos-db-read-v1, and associated data binding the read id, the chunk's sequence number, its stream and whether it is the last chunk. A chunk moved to another read, reordered, or relabelled as last fails its tag. The result's sealed summary carries the plaintext byte count and SHA-256, so a dropped or added chunk fails the check.
• What this buys. The deployment bucket and the console only ever hold the rows as ciphertext. Nobody without the CLI's private key can read them, including whoever holds the bucket. A replayed envelope produces output only the original CLI could open, and that CLI is gone (J1).
• What it does not buy, plainly (codex 18). Sealing proves who can read an answer, not who wrote it: the recipient's key is public, and each seal uses a fresh sender key, so no seal carries an author. Any party that can write the deployment bucket, which is every platform box (infra/modules/region-host/main.tf, host_buckets), could create picked, chunks and a matching result.json for a read before the real box does, and the CLI would accept that answer as consistent. The CLI's checks prove the answer is complete and unaltered since it was sealed; they do not prove which box sealed it. Rev 3's claim that a non-owner "cannot change an answer the CLI shows" was wrong and is withdrawn. This is the same trust every box already has over carlos run output and over everything else in the deployment bucket, and the in-region door trusts the bucket the same way. Authenticated authorship would need per-box signing keys, which exist only in claim mode; §6 leaves claim mode alone.

{
  "state": "done",
  "code": "",
  "box": "carlos-flagship-uk",
  "last_seq": 12,
  "started": "...",
  "ended": "...",
  "sealed": "<sealed summary>"
}

state is done, failed, refused or cancelled. code is a fixed vocabulary the console may read: nothing-replicated, history-gone, check-failed, changed-twice, too-large, no-room, sql-error, timeout, output-cap, record-changed, moving, interrupted. The sealed summary carries the transaction id and time of the copy, the oldest time for history-gone, the restored size and cap for too-large, SQLite's own error text for sql-error, and the byte count and SHA-256 of the output. The CLI maps codes to today's sentences where one exists (dbRestoreError), and to §10's rows otherwise.

5.7 Limits and storage

| Limit | Value | Enforced by | |---|---|---| | Pickup window | 60 s from issued_at, plus 60 s of skew | box (refuses older envelopes); console stops waiting after 2 minutes | | Planning and restore | 10 min | worker unit and agent context | | Query | 5 min wall clock | worker | | Owner deadline | 20 min from picked, stated in it | agent context, worker RuntimeMaxSec; console stops waiting 1 minute after it | | The request | at most 22 minutes; the CLI's own context is 23 | console, CLI | | Restored size | 2 GiB by default, CARLOS_DB_READ_MAX_BYTES on the box | agent, from the plan, before any byte moves | | Disk | a reservation per read (below) | agent | | Output | 64 MiB (runs.MaxOutputBytes, written "64 MB" in copy as the run's docs do); the read ends output-cap | agent; the CLI's buffer is bounded by the same number | | Chunk | 256 KiB or 1 s, whichever comes first (runs.ChunkBytes, runs.ChunkEvery) | agent | | Pull | at most 2 GiB of planned LTX files (measured on dev, 2026-10-23: relay rate 3.94 MB/s, the slowest of three runs; the rule gives 3.25 GiB, above the 2 GiB ceiling); CARLOS_DB_PULL_MAX_BYTES on a box can only lower it; the read ends too-large | agent, from the plan, before any byte moves; the CLI's spool is bounded by the same number | | Pull chunk | 1 MiB or 1 s, whichever comes first | agent | | Reads per box | 2, plus one connection kept for the readiness probe (MaxConnections=3) | agent, systemd | | Open reads per person | 2 | console, held by the open request (J5) | | Open reads per console | 16 | console, held by the open request, 429 with Retry-After | | Request budget | the db door's existing 600 per token per minute | console |

Admission (codex 17). The console's counts are taken in step 3 and released by a defer in the same handler, so they are exactly the reads whose requests are open. An abandoned read is a closed connection; its handler returns within 30 s (the write stall bound) or at its next wait line, and the slot is free. No timer, sweep or restart is involved.

Stored ciphertext (codex 7). The protocol deletes nothing. Every object of a read stays in the deployment bucket until the lifecycle rule removes it, and that timing is the store's, not ours. On S3 the rule (current versions expire 1 day after creation, noncurrent versions 1 day after they become noncurrent, and expired delete markers removed) makes an object eligible for expiry about a day after creation, with eligibility rounded to the next midnight UTC, and the noncurrent version eligible a day after that. S3 then removes eligible objects asynchronously, usually within a day or two, with no hard deadline. So the honest statement is: a read's ciphertext lives until lifecycle expiry, typically two to four days after the read, with no guaranteed upper bound. The size per read is the plaintext limit plus sealing overhead (codex 7): the output cap limits the rows to 64 MiB of plaintext in phase 1; each chunk object stores its sealed envelope as raw bytes (not base64), which adds under 100 bytes per 256 KiB chunk (a 65-byte sender key, a 12-byte nonce, a 16-byte tag and a short prefix), so a full answer stores about 64.03 MiB of ciphertext. The request, picked, result.json, the queue entry and cancel add a few kilobytes. Base64 appears only on the wire to the CLI. Plaintext rows are never in the bucket. A deployment that needs a hard deadline needs a different reclamation mechanism (version-aware deletes), which this design does not have.

Disk reservations (codex 8). The agent is the only process that starts workers, so it keeps one reservation table under one mutex. Before picked it reserves the floor (5 GiB); after planning it grows that reservation to the plan's LTX bytes plus its restored size. Each step succeeds only if the filesystem holding /var/tmp keeps at least 5 GiB and 10% free after subtracting every other open reservation; otherwise the read ends no-room. A reservation is released only when its worker unit has stopped, since systemd removes the unit's private /var/tmp at stop. The agent learns that from systemctl list-units 'carlos-dbread@*' (readable without privilege). It admits a new read, or a probe, only when the number of active, activating or deactivating carlos-dbread@ units is no more than the number of workers it is itself running. Any other unit is an orphan whose scratch is not in the table, so the agent refuses work until it is gone. There is no timeout escape: an orphan worker stops within 5 s of losing its connection (§8.3), and RuntimeMaxSec stops it in the end regardless.

A read's one retry (a planned file that vanished, §8.2) is a second worker. Its first attempt's reservation stays in the table, under its own key, until that attempt's unit has stopped, and the retry dials only once the unit count says so; the retry takes a fresh reservation from the floor. Closing the first connection is not the unit's stop (plan review 10).

Restart reconciliation (codex 8). A restarted agent's table is empty, and it cannot know the sizes of workers it did not start. So it reconciles by refusal, against the units actually running: at start, and before every admission, it lists carlos-dbread@ units, and while any is active, activating or deactivating that it is not itself running, it takes no read and no probe. It never guesses a size for an orphan and never admits beside one. Its first admission therefore happens only after every unit from before the restart has stopped and systemd has removed its scratch.

5.8 Cost

Each capable box does one LIST of its region's current and previous hour of queue entries per serial move and every 30 s, and two GETs per new in-window entry (the request, and cancel before electing, plan review 11). A read adds two console PUTs, one picked, one PUT per chunk and one result, and the console's own GETs and LISTs about once a second while the request is open. There are no deletes. The October bill was LIST and GET polling, so plan 1 measures the LIST rate on dev before the flagship roll and reports it.

5.9 Replay and forgery

| Attack | What stops it | |---|---| | A box writes its own request | No root signature: refused and logged once. | | A box writes its own queue entry for a real request | The box reads the request and requires the queue entry's region, hour and time to match the envelope. | | An envelope copied to another id or region | The id and region are inside the signature and must match the keys. | | The same envelope picked again | picked exists until lifecycle expiry, days after the 60 s window has closed (J2, J4); a second create fails on every box, before or after a restart (codex 5). Past the window the envelope is refused on age. | | A box serves a read for another region | It refuses any envelope whose region is not its own CARLOS_REGION, and IAM refuses the bucket anyway. | | A request that names a bucket or prefix | There is no such field. The box derives the prefix (§8.2). | | A compromised box forges an answer | Not prevented. §5.6 states the boundary (codex 18). |

6. Which operations forward

| Operation | Phase | Reason | |---|---|---| | query | 1 | This is the regression the customer hit. The result is rows, usually small, and only rows leave the region. | | query --at | 1 | Same path with a target time. The box lists the replica the way the console's door does today (listReplicaLevel, LastModified as CreatedAt), so the copy stops at the same transaction an in-region read would. Plan 1 tests that parity. | | pull | 2 | The result is the whole database. It rests on ruling 1 (§16, Paul, 2026-10-22: a whole copy's ciphertext rests in the deployment bucket's region until lifecycle expiry, typically days, until plan 5's per-region bucket), and a measured size cap. In phase 2 the box ships the planned LTX files, sealed, in the same single request, and the CLI restores and checks them locally with today's code. The box never restores for a pull. | | pull --at | 2 | As pull, with the target time. | | Claim-mode door (apidbclaim.go) | not planned | Claims are not active anywhere. The claim door reads exact object versions through hostops's versioned boundary and signed file handles, and lists no times. A box-side version of that belongs to claim activation (#250), which must add times anyway. A host-claim console keeps today's residency refusal, as it refuses carlos run. | | Seeds --from (resolveDBSource) | not planned | It shares the source checks but restores into a new instance, a different flow. Cross-region seeding is its own design. | | Fleet and pooled instances | refused, as today | Their data is not in a platform region bucket. |

Phase 1 reaches every cross-region instance a member can read today in its own region, for queries. Until phase 2, pull answers DBF-8 for a cross-region instance.

7. Who serves

7.1 Capability and selection

A box publishes db-read-1 in its edges file only when the agent's armed db-reads loop says so, the way offline-move-1 comes from the armed moves worker (armedOfflineView). The loop answers yes only when all of these hold:

• hibernation is configured (CARLOS_HIBERNATE_BUCKET set);
• CARLOS_REGION is set;
• CARLOS_RELEASE_PUBKEY is set, so it can verify envelopes;
• CARLOS_DB_READS is not 0 (an operator off switch for one box);
• CARLOS_HOST_CLAIM is not set;
• the loop is running, and its last readiness probe passed (codex 9).

The readiness probe connects to the worker socket and runs the worker protocol's hello exchange, which starts a real carlos-dbread@ instance and gets back the worker's protocol version and its view of its own sandbox (no network interface but loopback, /data empty, /etc/carlos unreadable). The loop probes at start and every 5 minutes, and after any read whose worker failed to start. A failed probe withdraws the capability on the next edges-file pass. A socket file whose service is missing, masked, broken or the wrong version therefore never advertises anything.

A probe never competes with reads (codex 9). The socket allows three connections (MaxConnections=3): the agent runs at most two reads and at most one probe, so a probe always has a connection and a busy box is never mistaken for a broken one. The agent counts its own connections, so it never opens a fourth. A probe is skipped, not failed, while orphan workers block admission (§5.7); the last result stands until it can run. When a read's worker fails to start, the agent probes as soon as that connection is closed. Rev 2 shared two connections between reads and probes, so two reads at once made the probe fail and withdrew the capability.

The console reads edgesfile.ReadLiveSnapshot before admitting a read and refuses up front only on complete evidence (codex 10). That function returns a nil error when status snapshots cannot be loaded and reports the gap through Complete and BoxesComplete instead, so the rule reads those fields, never the error alone. Incomplete evidence comes first and wins over both refusals:

1. Complete is false, or BoxesComplete is false (an unreadable edges file, an unreadable or missing status snapshot, a fresh status snapshot with no edges file): the request is written and the pickup window decides. A person is waiting, and the console stops waiting after 2 minutes (DBF-3).
2. Both are true, and no file in the record's region publishes db-read-1 on a non-draining box: DBF-1, no request written.
3. Both are true, capable non-draining boxes exist, and none is Fresh: DBF-2, no request written.

Rev 2 checked Complete alone before DBF-1, so a fresh box whose edges file was missing could be refused as "no server can read databases".

Any capable, live, non-draining box in the region may serve. There is no assignment by the console. Region HA (plan 2b, dormant) can put several boxes in a region, and they all read the same region bucket, so the create-only picked marker is the whole election. The 0 to 500 ms jitter spreads the race, and a box that fails admission (§5.7: no free read, the disk floor, or an orphan worker) does not enter it.

7.2 Why not the lease holder

A sleeping instance holds no lease, so there is no holder to ask. An awake instance's holder is busy serving it, and the read does not touch the instance, so there is nothing to gain by sending it there. Choosing by lease would also tie the read to the instance's lifecycle, which is exactly what "never the live database" rules out.

7.3 When no box serves

| Situation | Answer | |---|---| | No capable box in the region | DBF-1 at once | | Capable boxes, none live | DBF-2 at once | | Live but nobody picks in the window (all busy, all failing admission, agent stuck) | DBF-3, when the console stops waiting 2 minutes after the request | | The owning box dies or restarts mid-read | DBF-12, when the console stops waiting 1 minute after the owner's deadline, up to 22 minutes after the request (§5.5) |

8. Read-only and isolation on the box

8.1 The split

| Half | Runs as | Does | Never does | |---|---|---|---| | Agent (carlos agent, carlos-edge.service) | carlos | lists requests; verifies envelopes; creates picked; reads the record; derives the prefix; lists the replica and plans the restore from the listing; reads the last planned file's 100-byte header for the size check; streams planned files from the region bucket into the worker; re-checks the record before the query runs; seals and writes output; writes the result; keeps the disk reservation table | opens SQLite, decodes LTX pages, runs SQL, writes files on disk, reads anything under /data/instances, takes or reads the lease | | Worker (carlos system db-worker, carlos-dbread@.service) | a DynamicUser | receives the plan and the files over its socket; writes them into its private /var/tmp as a litestream file replica; restores to the agent's target transaction; runs lease.CheckDBContext; runs the one statement with today's guards; streams rows back | anything with a network, the box's files, the bucket, other tenants' files |

The rule is: everything that decodes page data or SQL runs in the worker. The member's SQL, the replica's pages and the restored database all do. The agent does object input and output and planning from listings, which it already does to wake instances. Its one decode is the last planned file's fixed 100-byte LTX header, for the size check, with the bounds dbdoor.PlanRestore already applies (a valid page size, a non-zero commit, TXIDs that match the listing).

8.2 The identity bracket

The replica prefix is keyed by host alone (activator.HibernatePrefix), so a host deleted and recreated, in the same account or another, reuses it. Checking the record once before the read would let a request admitted for the old instance read the new one's replica (codex 1). So the agent brackets every byte it reads with two identical checks, and no output exists until the second one passes.

The check. The agent reads the record fresh from the bucket and refuses (result refused) unless:

• the record exists and its incarnation is the envelope's (record-changed);
• the class rail passes (dbClassRefusal, moved where both console and box can call it);
• it is not pooled and not a fleet placement;
• rec.Region equals the box's CARLOS_REGION (record-changed);
• rec.Move is nil (moving);
• rec.Host is the envelope's host and passes provision.ValidHostShape.

The order.

1. Check A, then plan from the listing.
2. Stream every planned file into the worker. The worker restores and runs lease.CheckDBContext, then answers ready with the transaction it holds. It runs no SQL yet.
3. Check B, the same check again, after the last byte has been read from the bucket.
4. Only if check B passes does the agent tell the worker run. The first output chunk can exist only after this point.

A failed check B ends the read record-changed, stops the worker and writes no chunk. A retried plan (today's "changed while it was being read" retry) runs both checks again around its own download.

Why two checks are enough. The incarnation is hostclaim.Incarnation(account, app, host, CreatedAt). A deleted and recreated host, including one recreated in another account, gets a new incarnation, and an incarnation never comes back once its record is gone. So a record that shows the same incarnation at A and at B was the same instance for the whole download, and the files read in between were that instance's. Pinning object versions would add nothing that this does not already give in non-claim mode (§18).

The replica prefix is activator.HibernatePrefix(<the box's own CARLOS_HIBERNATE_PREFIX>, rec.Host) in the box's own CARLOS_HIBERNATE_BUCKET, read through lease.ReadReplicaClient and held only as the narrow replicaReader interface (LTXFiles, OpenLTXFile), so no write method is reachable. Nothing in the request chooses a bucket, a prefix, a key or a level beyond what the plan picks.

The agent lists only level directories. It never reads lock.json, never calls the leaser, and never wakes the instance.

8.3 The worker unit

carlos-dbread.socket, bound by root systemd:

• ListenStream=/run/carlos-dbread/worker.sock, SocketUser=carlos, SocketMode=0600, Accept=yes, MaxConnections=3 (two reads and one probe, §7.1, codex 9).

carlos-dbread@.service, one instance per connection, starting from the claim-bound template's hardening (§3 fact 8) with these differences:

• StandardInput=socket and StandardOutput=socket, so the accepted connection is the worker's stdin and stdout, and StandardError=journal for its own diagnostics (codex 2). With Accept=yes systemd passes the connection as a file descriptor and does not put it on stdin unless told to; the claim-bound template's StandardOutput=file: line is not copied. All protocol traffic, both ways, is on that one connection;

• DynamicUser=yes instead of User=carlos, so the worker owns no file on the box and can read nothing carlos owns;
• PrivateNetwork=yes and RestrictAddressFamilies=AF_UNIX, so no route to the metadata service, the bucket or anything else exists, rather than a deny list;
• TemporaryFileSystem=/data:ro /var/lib:ro /opt/carlos:ro /run/carlos:ro with BindReadOnlyPaths=/opt/carlos/carlos, and InaccessiblePaths=/etc/carlos /run/carlos-control;
• PrivateTmp=yes: its /tmp and /var/tmp are its own. The worker writes only under /var/tmp, which is on disk on Debian 13, where /tmp is a tmpfs. systemd deletes both when the unit stops, however it stops;
• MemoryMax=512M, CPUQuota=100%, TasksMax=16, LimitFSIZE set to the restored-size cap (it bounds each file; the agent's reservation in §5.7 bounds the total, codex 8), RuntimeMaxSec=20min;
• SystemCallFilter=@system-service, NoNewPrivileges=yes, empty CapabilityBoundingSet, ProtectProc=invisible.

carlos system db-worker refuses to run unless its stdin is a connected unix stream socket (checked with getsockopt(SO_TYPE) and SO_PEERCRED, whose peer must be the carlos user), so it cannot be started by hand or pointed at a file. Plan 1's acceptance test drives it only through the installed units.

The worker protocol is length-framed messages with a fixed maximum frame size: hello (the readiness probe, §7.1), a header naming the plan's files, their sizes, the target transaction and the byte limits, then the files, then ready from the worker, then run or stop from the agent, then framed output and a final status. The worker refuses a file larger than its header said and any write past the header's totals.

The worker stops when its agent goes (codex 8). From the moment it starts, the worker keeps one goroutine reading its connection. EOF or an error there cancels the one context that restore, lease.CheckDBContext (internal/lease/check.go:44, never CheckDB, which supplies a background context) and the query all run under, and the worker exits. Not every step honours a context promptly: litestream's file replica client opens a plain file without checking it (litestream@v0.5.14/file/replica_client.go:135), and the LTX compactor's page reads and writes ignore it (ltx@v0.5.1/compactor.go:177, 199). So the watcher does not rely on cancellation alone: after cancelling, it waits 5 s for the worker to finish, and if it has not, it ends the process itself with os.Exit (codex 8). A process exit stops a compaction wherever it is. A worker blocked in a write to a dead connection gets an error at once. So an agent restart ends every worker it had within seconds, systemd stops each unit and removes its private /var/tmp, and only then does the restarted agent's orphan count (§5.7) reach zero and admit work. While the agent reads the output, it writes nothing to the worker, so the watcher sees only EOF or the agent's own stop.

The socket and service files are written and enabled by ensureSystemUnits, the same root path that installs the other units, on the next adopt pass or update-platform-binary.sh.

Within the worker, the query keeps every guard the CLI has: mode=ro, query_only, temp_store(2), SQLITE_LIMIT_ATTACHED 0 and checkQuerySQL. The database is opened with immutable=1 as well, since nothing else ever writes the file. The guards and the query function move from db.go into a shared package (internal/dbquery), so the CLI and the worker run the same code and the output format cannot drift.

8.4 Why a tenant cannot reach another tenant's data

1. The console admits only a member of the account, for an instance of that account's app, and signs exactly that host into the envelope.
2. The box derives the prefix from that host, the fresh record and its own config, and nothing else.
3. The worker receives only the files the agent sent. It has no network, so no credentials from the metadata service and no bucket. Its filesystem view hides /data, /etc/carlos and /run/carlos, and its user owns nothing. ATTACH is off. If a SQLite flaw let the statement open a path, the sandbox leaves it nothing to open but its own scratch.
4. The output is sealed to the requesting CLI's key.

8.5 Why the live database is never touched

The agent reads bucket objects only. The worker cannot see /data. Nothing calls the activator, the leaser or lock.json. A read of a busy instance's replica can race litestream's compaction, which is today's "changed while it was being read, twice" case, and is retried the same way.

9. Console and CLI behaviour

9.1 The one new console route

POST /api/cli/apps/{account}/{app}/db/{host}/reads

It is svcrole.Set{} (people only) and runs the db door's full gate once, at the start (§5.5 step 3). A host-claim console answers it with today's residency sentence. Rev 3's follow and cancel routes are gone (J1), and so is rev 2's acknowledgement pair. Rev 3 needed a second, smaller gate for follow calls so that a record-changed result could still be fetched after the instance was deleted or reclassified (codex 15). With one request there is nothing to fetch later: the full gate ran when the read began, and the box's record-changed result arrives as the last line of the same response.

9.2 Mixed versions

| CLI | Console | What happens | |---|---|---| | old | old | today's residency sentence | | old | new, forwarding off or region not capable | today's sentence, or DBF-1 or DBF-2 | | old | new, forwarding on and region capable | DBF-14: update carlos | | new | old | today's sentence; no forward marker, so the CLI prints it as is | | new | new | the forwarded read |

A console with CARLOS_DB_FORWARD unset answers byte-identically to main.

9.3 Audit

One db-read activity row at admission, in the run's manner, with the detail DBF-21 or DBF-22. It is written before the request and the queue entry, while the request is open, so a read a box can see always has its row; a row the console cannot write refuses the read. A request that ends after the row and before the 200 leaves the row of an admission, and at most the request, the queue entry and cancel; never a serial bump (plan review 11). It records the SQL's length in bytes and the first 16 hex digits of its SHA-256, not its text (ruling 3). A member who kept the statement can match it to the row; the row itself holds none of the data a query's literals can carry. That differs from carlos run, which records argv, and keeps the in-region rule that SQL never reaches the audit log. The SQL itself stays in the request object, in the deployment bucket, until the lifecycle rule expires it with the rest of the read, on the timing §5.7 states: typically two to four days, with no hard bound.

10. Errors and copy

Every new user-facing sentence is below. Error and status strings follow Paul's voice rule (ruling 4): lower case at the start and after a full stop, no em dashes. Docs prose keeps the CLI page's own sentence case. None of this is approved yet; the copy-review gate runs before any of it is written into code.

| ID | Where | Text | |---|---|---| | DBF-1 | console 409, no capable box in the region | %s keeps its data in region %s. no server in that region can read databases yet, so carlos db cannot read it for now. | | DBF-2 | console 409, capable boxes but none live | %s keeps its data in region %s. no server in that region is answering right now. try again in a few minutes. | | DBF-3 | CLI, nobody picked the read in 60 s | no server in region %s picked up the read within 60 seconds. try again in a few minutes. | | DBF-4 | CLI, too-large | %s's database is %s once restored. carlos db can query at most %s in region %s. | | DBF-5 | CLI, no-room | the server in region %s does not have room to restore %s's database right now. try again later. | | DBF-6 | CLI stderr, output-cap | the result was cut off at 64 MB. narrow the query with WHERE or LIMIT and run it again. | | DBF-7 | CLI, timeout | the query ran for 5 minutes and was stopped. narrow it and run it again. | | DBF-8 | CLI, pull before phase 2 | %s keeps its data in region %s. carlos db pull cannot fetch a copy from there yet. carlos db query can read it. | | DBF-9 | console 429, open-read bounds | too many database reads are open. wait a minute and try again. | | DBF-10 | console 409, record moving | %s is moving between regions. read its database once the move has finished. | | DBF-11 | CLI stderr while waiting | reading a copy of %s's database in region %s. | | DBF-12 | CLI, the response ended without a result (the box stopped or never finished, the console restarted, the connection dropped), or interrupted | the read of %s's database in region %s did not finish, so nothing was printed. try again. | | DBF-13 | CLI, a chunk or summary fails to open or add up | the answer from region %s did not check out, so carlos db showed none of it. try again. | | DBF-14 | console 409 with the forward marker, read by an old CLI | %s keeps its data in region %s. this version of carlos cannot read it from there. run carlos update, then try again. | | DBF-15 | CLI, record-changed | %s changed while the read was waiting. try again. | | DBF-16 | CLI, moving from the box | as DBF-10 | | DBF-21 | audit detail, query | forwarded a database query to region %s. the query was %d bytes, sha256 %s (cli) | | DBF-22 | audit detail, query with --at | forwarded a database query to region %s, as of %s. the query was %d bytes, sha256 %s (cli) | | DBF-30 | docs/site/cli.md, carlos db, replacing the paragraph that starts "If the app has more than one instance" | see below | | DBF-32 | CLI, a query that is not UTF-8 text or holds U+FFFD, refused before it is sent: JSON would rewrite it, and the box must run the bytes typed or none (plan review 36) | this query cannot be sent to region %s, because it holds bytes that are not UTF-8 text or the character U+FFFD. fix the query and run it again. | | DBF-31 | docs/site/cli.md, carlos system, the worker | carlos system db-worker runs on platform boxes only. systemd starts it for each forwarded database query. you never run it yourself. |

DBF-13 is true as written only because the CLI buffers the whole answer and checks it before writing anything (§5.5, codex 6).

DBF-30:

> If the app has more than one instance, name one with --instance. Only > instances that sleep have a replica. An instance on your own fleet or in a > pool cannot be read this way, and the command says which. > > When an instance keeps its data in another region than the console's, > carlos db query runs in that region. A server there restores the copy, > runs your query and sends back only the rows. The rows are encrypted for > your terminal on the way, so nothing in between can read them, and > nothing is printed until all of them have arrived and checked out. There, > the copy can be at most 2 GB once restored, the query runs for at most 5 > minutes, and the rows can come to at most 64 MB. carlos db pull cannot > fetch a copy from another region yet.

Unchanged and reused: today's status line (dbQueryStatus), the dbRestoreError sentences for nothing-replicated, history-gone, check-failed and changed-twice, SQLite's own error text for sql-error, stopped for Ctrl-C, the guard refusals of checkQuerySQL, and the residency sentence when forwarding is off.

DBF-17 to DBF-20 and DBF-23 to DBF-29 are left free for phase 2's pull rows.

11. Rollout

11.1 Switches and gates

• Box: db-read-1 (§7.1). Published only while the armed loop's readiness probe passes. CARLOS_DB_READS=0 withdraws one box.
• Console: CARLOS_DB_FORWARD=1 turns forwarding on. Unset, every answer is byte-identical to main. The console also needs the root key.
• Store: the db-reads/ lifecycle rule (J4) is in place on the deployment bucket before any console turns forwarding on. On the flagship that is a change to the deployment-buckets module, a release tag, a ref bump in carloku-infrastructure and a targeted apply. That is production infrastructure, so it needs Paul's scoped approval under AGENTS.md's rule before it is applied; plan 2 prepares the plan for that approval. A self-hosted deployment sets the same rule (MinIO calls it ILM); without it, db-reads/ objects are never removed.
• CLI: none. The forward marker negotiates (§9.2).
• Version floor: none beyond the capability. A box that predates it publishes no capability and is never asked.

11.2 Mixed fleet

A region with one capable box and one old box works: only the capable box lists the queue. A region with no capable box gets DBF-1. A box that loses the capability mid-read finishes the read it holds, within its deadline, and takes no new one.

11.3 Roll order

Paul's order (dev boxes, dev console, check, flagship boxes, flagship console), with the lifecycle rule first on each deployment's bucket. Boxes publish an idle capability. The dev console turns CARLOS_DB_FORWARD=1 and the check is a forwarded query from a dev console to a dev box in another region, plus the LIST-rate measurement (§5.8) and a note of real database sizes (ruling 2).

11.4 Tests

Acceptance is production-wired. Stand-ins are allowed in unit tests and labelled so; they are not acceptance.

| # | Test | Wiring | |---|---|---| | T1 | A forwarded query end to end: console in region ie, agent in region uk, two MinIO buckets on one server with separate credentials. The console's credentials cannot read the uk bucket (a MinIO policy), so any console read of it fails the test. | real console server, real agent loop, real store; CI runs it with the worker reached through a test socket, labelled non-acceptance | | T2 | The same on the root runner with the real carlos-dbread.socket and carlos-dbread@.service installed by the real ensureSystemUnits, so the socket is the worker's stdin and stdout as §8.3 says (codex 2) | acceptance gate for plan 1 | | T3 | Sandbox witness on the root runner, from inside the worker: open /data/registry.db, /etc/carlos/host.env and another instance's directory; connect to 169.254.169.254:80 and to any TCP address; write outside /var/tmp; ATTACH and VACUUM INTO with the attach limit removed (a mutation); load_extension. Every one fails. The private tmp is gone after a normal end, a cancel and kill -9 of the agent mid-read. | real units | | T4 | Forgery and replay: unsigned, wrong key, scoped key, envelope moved to another id and to another region, a forged queue entry, the same envelope offered again before and after the agent restarts (picked holds, codex 5), a chunk sealed to another key, a chunk reordered, the last chunk dropped. And the boundary §5.6 states, as a documented negative: a writer that creates picked, chunks and a matching result first is accepted by the CLI (codex 18). | real verify and seal code, real store | | T5 | Two capable boxes in one region: exactly one runs each read, across 1,000 reads with random jitter. A box whose picked create commits on the server but answers an error reads it back and runs the read; a box whose create answers an error when another box owns it steps away (codex 16). | real store conditional create, a store wrapper that loses answers | | T6 | Parity: the forwarded query and the in-region query print byte-identical stdout for a fixture database (NULL, tabs, newlines, blobs, times). --at stops at the same transaction as the in-region door for the same replica and time, including a write in the same second as the target. | real restore, both paths | | T7 | The long request through the real path: a read that holds the request for 21 minutes passes through the real edge and the real console server (with its 5-minute ReadTimeout) and returns its result; a CLI context of 23 minutes is the outer bound. During a quiet restore the real CLI receives the admitted line within a second and a wait line at least every 15 s, both through the local edge and through backhaul from another edge, whose 30 s header timeout the request survives (codex 19). | real edges, real console, real CLI binary | | T8 | Mixed versions: the five rows of §9.2, with the forwarding-off 409 compared byte for byte against main. | real console | | T9 | Record changes between request and pick, during planning, during download and between download and run: moved, recreated in the same account, recreated in another account, class changed. Each ends with its code, the code reaches the CLI in the same response (codex 15), and no chunk is ever written (codex 1). | real records, a store wrapper that pauses the download | | T10 | Approved copy digests for every row in §10, in the approved_copy_test.go pattern. | | | T11 | CLI buffering: a dropped or bad last chunk, a summary hash mismatch, and a response that ends without a result each leave stdout empty (codex 6). | real CLI binary through carlosTestEnv | | T12 | Disk: two concurrent reads on one box whose free space fits one of them: one runs, the other ends no-room. An agent killed mid-read: the restarted agent takes no read and no probe while the old worker unit is active, its worker stops on EOF, and the restarted agent admits work only once the unit and its scratch are gone, without guessing a size for it. A disconnect during local compaction, after the download, still ends the worker within 5 s (codex 8). | real agent, real units, root runner | | T13 | Readiness: a present socket with the service missing, masked, failing to start and speaking another protocol version each withdraws db-read-1 within one probe and edges pass. A box running two reads keeps the capability, and its probe succeeds on the third connection (codex 9). | real units, root runner | | T14 | Console evidence: status snapshots unreadable, one snapshot missing, an edges file unreadable, an empty edges listing beside a fresh status snapshot. Each admits the read; only complete evidence (both flags) gives DBF-1 or DBF-2 (codex 10). | real ReadLiveSnapshot over a real store | | T15 | Admission (codex 17): two reads whose CLIs disconnect before pickup, and two whose CLIs disconnect after it, each free their person's and the console's count when the handler returns, without a console restart; a third read from the same person is then admitted. A console restart mid-read leaves nothing to recover. | real console | | T16 | Late writes are harmless (codex 4, 13, 14): a box that keeps writing chunks and its result after the console's request has ended (by timeout and by disconnect) changes nothing any reader sees, and no protocol path issues a delete or a replacing write (a store wrapper fails the test on any Delete or non-create Put under db-reads/). | real console, real agent, a versioned MinIO bucket | | T17 | The lifecycle rule: the module's rule covers db-reads/ with current and noncurrent expiry and expired delete markers (a module test). | module test | | T18 | The result overtaking the last chunk: a store wrapper that lets the box commit its last chunk and result.json between the console's listing and its result read, in both orders. The console never writes the result line before every chunk through last_seq, and the read succeeds (codex 20). | real console, real store |

Mutation checks that must turn a test red: drop envelope verification (T4); take the host or prefix from anywhere but the record and config (T1 with a second tenant); remove PrivateNetwork (T3); skip the summary hash check (T11); skip check B (T9); add any protocol delete or replacing write (T16); release the console's counts anywhere but the handler's return (T15); skip the read-back of an ambiguous picked (T5); drop the orphan count (T12); give probes no connection of their own (T13); remove the read-deadline extension or any flush (T7); list chunks before reading the result, or write the result line before its last chunk (T18); drop the worker's 5 s exit (T12).

12. Provider boundary

• Core: the transport is release.Store objects, so it runs on MinIO and any S3-compatible store. Every protocol write is a conditional create, which release.Store.Put already supports with an empty ifMatch, on S3 and on MinIO. The box reads each planned file with a GET and writes chunks with PUTs; it never uses a server-side copy. Signing is ed25519, sealing is the standard library plus x/crypto/hkdf. The sandbox is systemd, which every box already needs.
• One store endpoint per box (codex 11). Every S3 client in a box process takes its endpoint from the one CARLOS_S3_ENDPOINT (internal/s3compat/s3compat.go), the deployment store through s3compat.Options and the replica client through lease's s3Endpoint. So a box reaches its region's data bucket and the deployment bucket on the same server, as it already must to wake and to move instances. This design supports exactly that shape: separate buckets per region, with separate credentials, on one endpoint (§18).
• Expiry. The lifecycle rule is S3 lifecycle configuration, which MinIO supports as ILM. It is a deployment requirement (§11.1), not something the code can detect, and there is no fallback deleter: the store's rule is the only deletion (J4).
• AWS adapter: IAM is what makes the console unable to read another region's bucket on the flagship. A self-hosted deployment gets the same split from per-bucket credentials on its store.
• What a non-AWS deployment loses: nothing it has today. The capability, the liveness join and the box's identity use no AWS API. Multi-box regions are AWS-only today (H26), but nothing here depends on them; one box per region is enough.
• Build: the worker and its socket code are Linux-only, with //go:build linux and a !linux stub, so the CLI still cross-compiles for darwin and windows.

13. Census

Plan 14 root-read census (codex 12). The scanner reads every production package and matches Open, QueryContext and the rest whatever user runs them (internal/provision/plan14_root_reads_test.go:33). Its only classes are authority, hint, diagnostic and legacy (line 478), and legacy needs accepted-legacy;rev3=<ruling> as its basis. "Runs as a DynamicUser" is a fact for the ruling's footing, not a class. So each plan proposes, for a ruling on #280:

| Declaration | Callers and identity | Proposed class and basis | |---|---|---| | db.go runQueryContext (main: legacy, rev3=J4, basis=root-input, plan14-root-reads.tsv:202) | moved out of db.go | retired through the overlay's removal row, the form the H3 carlos-run removal used | | internal/dbquery RunContext (the moved body, unchanged) | the CLI on a member's machine, as the member; the worker on a box, as its DynamicUser. Neither is root, and neither decides anything for root | legacy, accepted-legacy;rev3=<ruling>;basis=root-input, the J4 footing main gave the same body | | agent db-reads loop: the queue listing, the request read, the picked read-back, the record reads of checks A and B, the replica listing and GETs | the agent, carlos, carlos-edge.service | legacy with J1 (record and route reads) and J4 (object input), as the offline move's agent units took | | worker: the scratch writes and reads, the file replica restore, CheckDBContext, the query, the connection watcher | carlos-dbread@, DynamicUser | legacy with J4 | | agent readiness probe and admission (Statfs on /var/tmp, systemctl list-units) | the agent, carlos | diagnostic for the probe; legacy J4 for Statfs and the unit listing |

(The census's J1 and J4 are its own dispositions, not this spec's invariants of the same names.)

The overlay re-pins the source envelopes of every changed package (., internal/dbquery, internal/dbdoor, internal/host, internal/dbreads) and updates the legacy seal under the same ruling. The census owner may rule other classes; the spec's requirement is only that every new and moved declaration has one of the four, with both callers of the shared runner named.

• Root writes. ensureSystemUnits writes two new unit files. That is a root-written artifact and joins the Plan 14 write census as an overlay, as every new unit has.
• 9B8 record-write census. No record writer is added. The new Put sites (request, queue entry, picked, chunks, result, cancel) are nonrecord, all conditional creates. There is no Delete site. registerCLIRoutes changes as a key-site for the one route.
• Claim routes. The route joins internal/console/testdata/claim-routes.txt, refused on a host-claim console like the run's.
• go.mod. No new module, so no root-envelope overlay for it.

14. Plans

1. Box side and wire, dormant. internal/dbreads (ids, keys, the envelope, sealing, the result, limits); internal/dbquery (the query runner and guards moved out of db.go, with the CLI switched to it and no behaviour change); dbdoor planning over any replicaReader; the agent's db-reads loop with the identity bracket, conditional creates and the picked read-back, admission (reservations and the orphan count) and the readiness probe; carlos system db-worker, its protocol and its connection watcher; the two units in ensureSystemUnits; the db-read-1 capability. Gates: T2, T3, T4, T5, T9, T12, T13, the parity half of T6 that needs no console. Census overlay. Nothing reaches a box unless a console asks.
2. Console and CLI for query and query --at. The one route and its streamed response, the read-deadline extension, the forward marker, the evidence rule for capability and liveness, admission held by the request, audit, the CLI's buffered forwarded path, DBF copy, docs/site/cli.md, and the deployment-buckets lifecycle rule with the approval plan for applying it. Gates: T1, T6, T7, T8, T10, T11, T14, T15, T16, T17, T18, copy review, census overlay. Roll per §11.3.
3. pull and pull --at. On ruling 1 (§16), which Paul confirmed with the storage statement on 2026-10-22. Sealed LTX files in the same single request, local restore and CheckDBContext in the CLI, and the pull copy rows. A 2 GiB pull in one request is the case most likely to need a longer request bound; plan 3 measures it before choosing.

Two follow-ups are listed here and not written:

1. Plan 4 (follow-up): in-region reads onto the box path (ruling 5). Written once phase 3 has run on the flagship. It is not designed here beyond one constraint this spec already keeps: the box path is region-agnostic, so the console can send an in-region read through it unchanged.
2. Plan 5 (follow-up): per-region db-read bucket (ruling 1). After the feature works, db-reads/ traffic moves from the shared deployment bucket to a bucket in each data region, so that ciphertext and request SQL never rest outside the data's region. It needs its own design for the console's write grant into each region's bucket, the boxes' grant on their own, the queue and serial signal, and the lifecycle rule per bucket. Plans 1 to 3 keep the protocol's keys relative to one store (db-reads/...) so that plan 5 changes where the store is, not what is in it.

15. Review focus: the five likeliest failure modes

1. The worker reaches more than its copy. A unit option missed or overridden (network, /data, /etc/carlos, the metadata service, the socket on stdin and stdout), or a SQLite path (ATTACH, VACUUM INTO, an extension, a temp file) that opens something. Any one of these turns a member's SQL into the box role, which reads every tenant in the region.
2. The prefix or host comes from the request, or the identity bracket leaks. Any path where an envelope field, rather than the fresh record and the box's config, picks the bucket, the prefix or the host is a cross-tenant read. So is any byte read outside checks A and B, or any output written before check B.
3. Forgery and replay. An unsigned or wrongly scoped request accepted, domain separation from other root envelopes missing, the region not bound, or a queue entry trusted without its request.
4. Something outliving the request. Any state, route, timer or write on the console side that survives the request's end, any protocol delete or replacing write, or an admission count released anywhere but the handler's return. Each one reopens a finding J1 to J5 closed.
5. The long request. A timeout on the real path that the spec missed (a proxy, a client, a keep-alive), wait lines that stall behind a slow reader, or a 2 GiB pull that does not fit the request bound.

16. Rulings

The coordinator answered rev 1's five questions provisionally on 2026-10-22. Paul ruled on 2026-10-22, after Astra accepted rev 5: ruling 1 as below, and rulings 2 to 5 confirmed as written (he raised no objection). All five are now Paul's.

1. Pull through the deployment bucket (Paul, 2026-10-22). Ship the feature through the deployment bucket as designed, for query, query --at, pull and pull --at. The true storage statement stands and is what Paul ruled on: a read's whole ciphertext, up to its size cap, lives in the deployment bucket (eu-west-1 on the flagship) until lifecycle expiry, typically two to four days, with no guaranteed upper bound (§5.7). For a query that is at most 64 MiB of rows; for a pull it is the planned replica files, up to the pull's cap. The SQL rests in the request object on the same timing (§9.3). The earlier provisional bound ("at most 64 MiB at a time for at most 1 hour") does not hold and is withdrawn. A short or hard bound would need version-aware deletes, with s3:DeleteObjectVersion on the deployment bucket for the console's role, which this design does not add. Follow-up, after the feature works: a per-region bucket for this traffic (plan 5, §14), so that ciphertext and request SQL never rest outside the data's region. Plans 1 to 3 do not build it and do not wait for it; they keep every db-reads/ object key relative to one store so that plan 5 can move the store without changing the protocol.
2. Limits stand. 2 GiB restored, 5 minutes, 64 MB of rows, two reads per box. Real database sizes on the flagship are still unmeasured; plan 2's dev check reports them. The pull's cap, 2 GiB of planned LTX files (§5.7, rev 7, measured on dev 2026-10-23), is put to Paul as a limit: pending his confirmation.
3. Audit. The row records the SQL's length and a SHA-256 prefix, not its text (§9.3). The SQL itself rests in the request object until expiry.
4. Copy casing. New strings use Paul's voice rule: lower case after a full stop (§10). The existing db door strings that capitalise are left as they are; changing them is a separate copy review.
5. One path. In-region reads move to the box path after phase 3 (§14).

17. What this does not do

• No console per region and no console-to-box connection.
• No resumable reads: a read that loses its connection is run again.
• No change to in-region reads (until the fourth plan), the claim-mode door, seeds --from, fleet or pooled instances.
• No IAM change. No new read authority for any role. The one infrastructure change is a lifecycle rule on the deployment bucket.
• No per-store S3 endpoints.
• No authenticated authorship of answers (§5.6).
• No carlos db shell and no browser view.

18. Declined review findings

Round 1 (CODEX-FINDINGS-DB-REGION-FORWARD.md). Every citation was checked against main at 5f341f33 and holds. Findings 2 to 10 and 12 are accepted in full. Two are accepted in part, and rounds 2 and 3 agreed with both declines:

• Codex 1, the "stronger guarantee" (pin object versions to the checked incarnation): declined. The identity bracket (§8.2) closes the finding. Pinning versions needs s3:GetObjectVersion and s3:ListBucketVersions on the box's data bucket, which the flagship's uk, us and au boxes do not have at module ref v0.21.0, and a versioned reader that today lives only behind the claim boundary. Claim activation can revisit it.
• Codex 11, independently configured deployment and regional-data endpoints: declined. No box path supports regional data on a separate store server today, because every S3 client in the process reads one CARLOS_S3_ENDPOINT. Per-store endpoints, if wanted, are a platform-wide change with its own design.

Round 2 (CODEX-FINDINGS-DB-REGION-FORWARD-R2.md). Nothing declined.

Round 3 (CODEX-FINDINGS-DB-REGION-FORWARD-R3.md). Every citation was checked and holds, including that the worker must call lease.CheckDBContext (internal/lease/check.go:44), not CheckDB; §8.3 and §13 now say so. Nothing is declined. Rev 4 does not patch findings 13, 14, 16 and 17 or the partly-holding invariants one by one: it removes the durable read they all came from (§5.2), and §5.3 maps every finding to the invariant that closes it or to the synchronous design that removes it. Finding 7's fix is the honest timing statement, not a new mechanism; a hard bound would need version-aware deletes, which is noted for Paul in §16 rather than built. Finding 18 is fixed by withdrawing the claim and stating the trust boundary (§5.6), not by adding per-box signing.

Round 4 (CODEX-FINDINGS-DB-REGION-FORWARD-R4.md). Every citation was checked and holds, including litestream's file client and the LTX compactor ignoring their contexts, and the backhaul transport's 30 s header timeout. Nothing is declined. All four fixes reuse what the spec already had: the run's result-before-list order, http.ResponseController, the orphan refusal, and a process exit.