amadan
db-region-forward spec rev 1: carlos db reads for instances in another region
The console forwards a db read to a box in the region where the data lives, over the run's transport (deployment-bucket objects, serial bump, paged chunks). The request is a root-signed envelope; results are sealed to an ephemeral key the CLI makes, so the console and the deployment bucket hold only ciphertext. Any live box in the region publishing db-read-1 may serve; a create-only picked marker elects one. The agent plans and streams objects; LTX decode, restore and the member's SQL run in a new socket-activated DynamicUser unit with no network. Phase 1 forwards query and query --at; pull waits on a ruling. Three plans.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+775
−0
docs/superpowers/specs/2026-10-22-db-region-forward-design.md+775 −0
diff --git a/docs/superpowers/specs/2026-10-22-db-region-forward-design.md b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md| new file mode 100644 |
| index 000000000..e38461f31 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md |
| @@ -0,0 +1,775 @@ |
| +# `carlos db` for instances whose data lives in another region |
| + |
| +**Status:** rev 1, 2026-10-22. Design only, nothing built. Written for Paul, |
| +who approved the direction (§2). Review brief: |
| +`CODEX-REVIEW-BRIEF-DB-REGION-FORWARD.md`. |
| + |
| +**Companion specs:** `2026-09-02-carlos-db-design.md` (the db door, cited |
| +below as "db spec"), `2026-10-08-carlos-run-design.md` (the run transport, |
| +"run spec"), `2026-09-11-isolated-exec-design.md` (the tenant unit sandbox). |
| + |
| +## 1. The problem |
| + |
| +`carlos db query`, `carlos db pull` and their `--at` forms restore a copy of |
| +an instance's database on the member's own machine. The CLI fetches the |
| +replica's LTX files by coordinate through the console's db door |
| +(`internal/console/apidb.go`). The console reads them from the data bucket |
| +of its own region. |
| + |
| +The console refuses every instance whose record `Region` differs from its |
| +own `CARLOS_REGION` (`resolveReplica`, `ErrResidency`, |
| +`dbResidencyRefusal`). That refusal is correct for the console: each |
| +region's box role is granted its own region's data bucket and no other |
| +(`infra/modules/region-host/main.tf`, `host_buckets`), and the console runs |
| +on the ie box. |
| + |
| +On 2026-10-21 all twelve instances of one customer account moved from ie to |
| +uk with the offline operator move. The customer can no longer run |
| +`carlos db query` against any of them. Every instance in a region other than |
| +the console's has the same gap, and self-serve moves and nearest-region |
| +placement will make that the common case. |
| + |
| +## 2. The direction, and what it changes |
| + |
| +Paul approved this direction: the console forwards a db read to a box in the |
| +region where the data lives. That box reads its own region's replica, runs |
| +the read-only operation there, and returns only the result. The console |
| +never reads another region's bucket. There is no console per region. |
| + |
| +Two earlier rulings bend under it, and this spec says so rather than |
| +working around them: |
| + |
| +- **db spec §9, "no path that runs anything on the box".** A forwarded query |
| + runs member SQL on a box. §8 is how that stays safe. |
| +- **db spec §6 rejected "tenant database bytes in the deployment bucket".** |
| + A forwarded read has to come back to the console through the deployment |
| + bucket (§5.1). This spec puts only encrypted bytes there, sealed to a key |
| + that exists only in the member's CLI (§5.5). In phase 1 that is only a |
| + query's rows. Phase 2's pull would send the planned replica files, which |
| + are a pull's result, and waits on Paul's ruling (Q1). |
| + |
| +One earlier ruling holds unchanged: the run spec's "the console still never |
| +connects to a box". The transport is the run's. |
| + |
| +## 3. What exists today (facts this design rests on) |
| + |
| +1. **The CLI restores locally.** `db.go` restores the replica through |
| + `dbdoor.RestoreAsOf` over a read-only client (`internal/dbdoor/client.go`) |
| + into a private 0700 directory, checks it with `lease.CheckDB`, then |
| + either copies it to `--out` or runs one query. The query runs with |
| + `mode=ro`, `query_only`, `temp_store(2)`, `SQLITE_LIMIT_ATTACHED` 0 and |
| + the one-statement and pragma guard `checkQuerySQL`. The console never |
| + executes tenant SQL. |
| +2. **`--at` depends on listing times.** litestream's timestamp plan keeps a |
| + file only if its `CreatedAt` is before the target. The legacy door lists |
| + S3 `LastModified`, which has second resolution, so the target is |
| + truncated to its second. The claim-mode door lists no times, and `--at` |
| + refuses there (`dbdoor.ErrNoTimes`). |
| +3. **`carlos run` already reaches any region's boxes.** Its transport is |
| + objects in the deployment bucket under the app's subtree |
| + (`internal/runs`): `request.json`, `out/<seq>` chunks of 256 KiB, |
| + `result.json`, written create-only where it matters. The console bumps |
| + `control/serial`; boxes poll it every 2 s. The console's follow door |
| + pages chunks to the CLI, and its sweep (`internal/console/runsweep.go`) |
| + expires, closes lost runs, and deletes old ones. A run is admitted with a |
| + CAS on the instance record, because a run needs the instance to itself. |
| +4. **The deployment bucket is writable by every platform box**, so an |
| + unsigned control object is a request any box could have written. The |
| + host-retry grant (`internal/release/hostretry.go`) answers that with a |
| + root-signed envelope: `release.SignRootEnvelope` on the console, |
| + `VerifyRootEnvelope` against `CARLOS_RELEASE_PUBKEY` on the box. Scoped |
| + release keys are refused. |
| +5. **Sealing to a public key exists.** `internal/appconfig/crypto.go` |
| + seals with ECDH P-256, HKDF-SHA256 and AES-256-GCM, with bound associated |
| + data (the v2 envelope), using only the standard library and |
| + `golang.org/x/crypto/hkdf`, which `go.mod` already carries. |
| +6. **Boxes publish capabilities.** `control/edges/<box>.json` carries |
| + `capabilities` (`region-ha-1`, `offline-move-1`, `offline-stage-1`). |
| + `edgesfile.ReadLiveSnapshot` joins each file with the box's status |
| + snapshot for liveness. The offline move's doors refuse a region holding |
| + a box without the capability, and each box publishes the capability only |
| + when its armed worker can act on it. |
| +7. **The box role can read its region's replicas.** In |
| + carlosframework/carloku-infrastructure, `region-uk.tf` passes |
| + `data_bucket_arn = aws_s3_bucket.data_uk.arn` to the `region-host` |
| + module (ref v0.21.0; ie is at v0.28.8). At both refs `host_buckets` |
| + grants `s3:GetObject` on `data_bucket_arn/*` and `s3:ListBucket` on the |
| + bucket, plus read and write on the deployment bucket, which stays in |
| + eu-west-1 and is shared by every region. Sibling regions get push-only |
| + `moves/*`. So a uk box can read uk replicas, and nothing else can. No |
| + IAM change is needed for this design. |
| +8. **Tenant processes already run sandboxed.** The claim-bound unit |
| + template (`internal/host/claimunit_template.go`) and the `carlos-app@` |
| + units use `ProtectSystem=strict`, `TemporaryFileSystem=/data:ro`, |
| + `InaccessiblePaths=/etc/carlos`, `PrivateTmp`, `NoNewPrivileges`, an |
| + empty capability set and an `IPAddressDeny` list that covers the |
| + metadata service. `carlos-app@` sockets are bound by root systemd |
| + (`SocketUser=carlos`), so the agent can start a sandboxed unit with no |
| + privilege of its own by connecting to its socket. |
| +9. **The agent runs as `carlos`** (`carlos-edge.service`, `User=carlos`). |
| + The console runs as `carlos-console`. |
| + |
| +## 4. Decisions |
| + |
| +| # | Decision | Section | |
| +|---|---|---| |
| +| D1 | Phase 1 forwards `query` and `query --at`. `pull` and `pull --at` forward in phase 2, after Paul rules on Q1. The claim-mode door and the seeds `--from` door stay refused. | §6 | |
| +| D2 | Any live box in the record's region that publishes `db-read-1` and is not draining may serve. The first to create the read's `picked` marker serves it. The instance's lease holder gets no preference. | §7 | |
| +| D3 | Transport is the run's: objects in the deployment bucket, a serial bump, chunks paged by the console. The request is a root-signed envelope. No record write. | §5 | |
| +| D4 | Results are sealed to an ephemeral key the CLI makes for each read. The console and the deployment bucket only ever hold the result as ciphertext. The console does see the SQL, as the envelope's signer. | §5.5 | |
| +| D5 | The box splits the work. The agent (`carlos`) verifies, resolves, plans and streams objects. Everything that decodes tenant bytes (LTX, SQLite, the member's SQL) runs in a new sandboxed unit, `carlos-dbread@.service`, as a `DynamicUser` with no network. | §8 | |
| +| D6 | The worker gets the planned LTX files and nothing else. Its scratch is its own private `/var/tmp`, which systemd removes when the unit stops. | §8.3 | |
| +| D7 | Rollout: boxes publish `db-read-1`; the console forwards only with `CARLOS_DB_FORWARD=1` and a capable region. Dormant by default. | §11 | |
| +| D8 | Three plans. | §14 | |
| + |
| +## 5. Transport and authentication |
| + |
| +### 5.1 Why this transport |
| + |
| +Three candidates exist in the tree: |
| + |
| +- **Deployment-bucket objects** (`carlos run`, restart scope, the offline |
| + move's stager and sealer reports). Established, provider-neutral, works |
| + for a box with no inbound path, and already crosses regions. |
| +- **Edge backhaul** (`internal/edge/homemap.go`, `mesh.go`). A box-to-box |
| + HTTPS hop for tenant traffic. Its HMAC key proves only "a box in this |
| + deployment" and is deliberately allowed to buy nothing but |
| + `X-Forwarded-For` belief. Using it for db reads would give that key a new |
| + authority across the IAM region boundary, and the console would start |
| + connecting to boxes, which the run spec ruled out. |
| +- **The fleet store door** (`internal/edge/storedoor.go`). Serves bucket |
| + objects to fleet boxes on the tunnel hostname. Wrong direction, wrong |
| + principal. |
| + |
| +Decision: deployment-bucket objects. The cost is latency (a pickup is |
| +seconds, not milliseconds) and a LIST per serial move on capable boxes |
| +(§5.8). Both are acceptable for a command a person runs by hand. |
| + |
| +### 5.2 Objects |
| + |
| +`<id>` is `dbr_` and 26 lowercase base32 characters (128 random bits), |
| +minted the way `runs.MintID` mints run ids, with its own `ValidID`. |
| + |
| +| Key | Writer | Write | Purpose | |
| +|---|---|---|---| |
| +| `control/db-reads/<region>/<id>.json` | console | create-only | The request: a root envelope (§5.3). Boxes of `<region>` discover reads by listing this prefix. | |
| +| `apps/<acct>/<app>/db-reads/<id>/picked` | box, or the console's expiry | create-only | Who took the read. Exactly one writer wins. | |
| +| `apps/<acct>/<app>/db-reads/<id>/beat` | the picking box | replace | Heartbeat every 15 s: phase (`planning`, `restoring`, `querying`, `sending`) and bytes so far. | |
| +| `apps/<acct>/<app>/db-reads/<id>/out/<seq>` | the picking box | create-only | One sealed chunk of output (§5.5). | |
| +| `apps/<acct>/<app>/db-reads/<id>/ack` | console | replace | The last sequence number the CLI has read. | |
| +| `apps/<acct>/<app>/db-reads/<id>/cancel` | console | create-only | The CLI asked to stop. | |
| +| `apps/<acct>/<app>/db-reads/<id>/result.json` | the picking box, or the console for expired and lost reads | create-only | How the read ended (§5.6). Written after the last chunk. | |
| + |
| +The job objects live under the app's subtree for the run's reason: app trash |
| +and purge sweep them, and the owner gate maps onto them. The request lives |
| +under `control/` so a box lists one small prefix for its own region and |
| +never walks every app. `<region>` is the record's `Region`, which must pass |
| +the region-code shape check; a `fleet/<name>` region never gets here (§6). |
| + |
| +There is no write to the instance record. A run takes a record CAS because |
| +it needs the instance to itself. A db read touches neither the instance nor |
| +its lease, needs no exclusivity, and is frequent. A record write per read |
| +would bump `record_seq`, race with moves and reconcile, and add a record |
| +writer to the 9B8 census for nothing. |
| + |
| +### 5.3 The request envelope |
| + |
| +```json |
| +{ |
| + "purpose": "carlos-db-read-v1", |
| + "v": 1, |
| + "id": "dbr_...", |
| + "account": "<sqid>", |
| + "app": "shop", |
| + "host": "shop.example.com", |
| + "incarnation": "<hostclaim.Incarnation of the record>", |
| + "region": "uk", |
| + "op": "query", |
| + "sql": "SELECT count(*) FROM orders", |
| + "at": "2026-10-07T14:00:00Z", |
| + "reply_key": "<base64 P-256 public key, uncompressed>", |
| + "by": "member@example.com", |
| + "issued_at": "2026-10-22T09:00:00Z" |
| +} |
| +``` |
| + |
| +- Signed with `release.SignRootEnvelope` under the console's root key. A |
| + console that holds only a scoped key cannot sign, and so cannot forward: |
| + it answers with today's residency sentence. |
| +- `purpose` is a fixed string so the envelope cannot be confused with any |
| + other root-signed payload. The box decodes into the exact type with |
| + `VerifyRootEnvelope`, which requires canonical JSON. |
| +- The box accepts an envelope only when `id` matches the key it was listed |
| + under, `region` matches both the key and the box's own `CARLOS_REGION`, |
| + and `issued_at` is no more than 60 s old (the pickup window) and no more |
| + than 60 s in the future (skew). |
| +- `at` is empty for the newest copy. `op` is `query` in phase 1; phase 2 |
| + adds `pull`. |
| +- The envelope carries no bucket, no prefix and no key. The box derives the |
| + replica prefix itself (§8.2). |
| + |
| +### 5.4 Sequence of a forwarded query |
| + |
| +1. The CLI calls the existing list door, as today. A new console with |
| + forwarding on answers a capable cross-region instance with 409 and an |
| + additive body: `{"error": DBF-14, "forward": 1, "region": "uk", |
| + "forward_ops": ["query"]}`. An old CLI prints the sentence. A new CLI |
| + switches to the forwarded form. |
| +2. The CLI runs `checkQuerySQL` locally (the same refusals as today), makes |
| + an ephemeral P-256 key, and sends |
| + `POST /api/cli/apps/{account}/{app}/db/{host}/reads` with `op`, `sql`, |
| + `at` and `reply_key`. |
| +3. The console runs `dbGate`'s order (person only, membership with a |
| + byte-identical 404, the db budget, configuration, the record, the class |
| + rail, pooled, fleet), then refuses a moving record, checks |
| + `checkQuerySQL` again, checks capability (§7.1) and its concurrency |
| + bounds (§5.7), mints the id, writes the envelope create-only, bumps the |
| + serial, writes the audit row (§9.3) and answers 202 `{id, region}`. |
| +4. Each capable box in the region lists `control/db-reads/<region>/` when the |
| + serial moves, and every 30 s if it does not. For each envelope it has |
| + not seen, it verifies, checks it has a free worker slot and is above the disk floor (§5.7), |
| + waits a random 0 to 500 ms, and creates `picked`. A box that loses the |
| + create steps away. |
| +5. The winner reads the record fresh and runs §8.2's checks, then plans, |
| + streams the planned files into a worker, and seals the worker's output |
| + into chunks. It writes `beat` every 15 s, reads `cancel` and `ack` at |
| + each beat and whenever its unacked window is full, and writes |
| + `result.json` after the last chunk. |
| +6. The CLI polls `GET .../db/{host}/reads/{id}?after=<seq>` about once a |
| + second. The page is the run's `Page` shape: state, chunks (base64 of the |
| + sealed bytes) and the result when present, under 512 KiB. The CLI opens |
| + each chunk, writes rows to stdout, and posts |
| + `POST .../reads/{id}/ack` with its last sequence number. |
| +7. On the result, the CLI checks the sealed summary (§5.6), prints today's |
| + status line to stderr, posts its final ack, and exits. The console |
| + deletes the job objects and the request when it records that ack. |
| + |
| +The status line (`dbQueryStatus`) needs the copy's transaction and time, |
| +which arrive in the sealed summary after the rows. So the forwarded CLI |
| +prints DBF-11 while it waits and today's status line at the end, both on |
| +stderr. stdout carries the same rows in the same format as an in-region |
| +query. |
| + |
| +### 5.5 Sealing |
| + |
| +- One ephemeral P-256 key pair per read, made by the CLI and held only in |
| + its memory. The public half travels in the request and the envelope. |
| +- Each chunk is sealed with the appconfig v2 construction (ECDH P-256, |
| + HKDF-SHA256, AES-256-GCM, a fresh ephemeral sender key per seal), with its |
| + own HKDF info string, `carlos-db-read-v1`, and associated data binding the |
| + read id, the chunk's sequence number, its stream and whether it is the |
| + last chunk. A chunk moved to another read, reordered, or relabelled as |
| + last fails its tag. |
| +- `result.json` has an open part for the console and a sealed part for the |
| + CLI (§5.6). The sealed part carries the plaintext byte count and its |
| + SHA-256, and the CLI refuses a result whose chunks do not add up to it |
| + (DBF-13). That closes truncation. |
| +- What this buys: the deployment bucket, which sits in the console's region, |
| + never holds readable tenant data. The console never sees rows. A replayed |
| + envelope (§5.9) produces output only the original CLI can open. |
| +- What it does not buy: authenticity of the sender. Any box can seal to a |
| + public key. A compromised box could write a false answer. That is the |
| + same trust every box already has over run output and over the |
| + deployment bucket, and the in-region door trusts the bucket in the same |
| + way. Per-box signing keys exist only in claim mode, which this design |
| + leaves alone (§6). |
| + |
| +### 5.6 The result |
| + |
| +```json |
| +{ |
| + "state": "done", |
| + "code": "", |
| + "box": "carlos-flagship-uk", |
| + "last_seq": 12, |
| + "started": "...", |
| + "ended": "...", |
| + "sealed": "<sealed summary>" |
| +} |
| +``` |
| + |
| +`state` is `done`, `failed`, `refused`, `expired`, `lost` or `cancelled`. |
| +`code` is a fixed vocabulary the console may read: `nothing-replicated`, |
| +`history-gone`, `check-failed`, `changed-twice`, `too-large`, `no-room`, |
| +`sql-error`, `timeout`, `output-cap`, `record-changed`, `moving`, |
| +`interrupted`. The sealed summary carries the transaction id and time of the |
| +copy, the oldest time for `history-gone`, the restored size and cap for |
| +`too-large`, SQLite's own error text for `sql-error`, and the byte count and |
| +SHA-256 of the output. SQLite's error text is sealed because it can name |
| +schema. |
| + |
| +The CLI maps codes to today's sentences where one exists |
| +(`dbRestoreError`), and to §10's rows otherwise. |
| + |
| +### 5.7 Limits |
| + |
| +| Limit | Value | Enforced by | |
| +|---|---|---| |
| +| Pickup window | 60 s from `issued_at` | box (refuses older envelopes), console expiry | |
| +| Heartbeat | every 15 s; lost after 90 s without one | box, console sweep | |
| +| Planning and restore | 10 min | worker unit and agent context | |
| +| Query | 5 min wall clock | worker | |
| +| Whole read | 20 min | worker `RuntimeMaxSec`, agent context | |
| +| Restored size | 2 GiB by default, `CARLOS_DB_READ_MAX_BYTES` on the box | agent, from the plan, before any byte moves | |
| +| Disk floor | before `picked`: at least 5 GiB and 10% of the filesystem holding `/var/tmp` free. After planning: the floor must still hold once the plan's LTX bytes and restored size are counted, or the read ends `no-room` | agent | |
| +| Output | 64 MiB (`runs.MaxOutputBytes`, written "64 MB" in copy as the run's docs do); the read ends `output-cap` and the CLI says so | agent | |
| +| Chunk | 256 KiB or 1 s, whichever comes first (`runs.ChunkBytes`, `runs.ChunkEvery`) | agent | |
| +| Unacked window | 16 chunks (4 MiB); the box pauses until the CLI's ack moves | agent | |
| +| Worker slots per box | 2 (`MaxConnections=2` on the socket) | systemd, agent | |
| +| Open reads per person | 2 | console | |
| +| Open reads per console | 16 | console, 429 with `Retry-After` | |
| +| Request budget | the db door's existing 600 per token per minute | console | |
| + |
| +The unacked window bounds both the box's memory and how much ciphertext a |
| +read can leave in the deployment bucket at once. |
| + |
| +### 5.8 Cleanup and cost |
| + |
| +- **On the CLI's final ack, or its cancel once the box's result has |
| + landed,** the console deletes `apps/<acct>/<app>/db-reads/<id>/` and the |
| + request. A CLI that went away leaves them to the sweep below. |
| +- **The console sweep** joins `runsweep.go`'s loop (30 s): an unpicked read |
| + past its window gets the console's own `picked` (create-only, so it races |
| + safely with a late box) and an `expired` result; a picked read with no |
| + beat for 90 s gets a `lost` result. Reads whose result is older than 1 h, |
| + and request objects older than 1 h, are deleted every 10 minutes. |
| + `KeepFor` is 1 h, not the run's 24 h, because these objects are copies of |
| + data, even if encrypted. |
| +- **On the box,** the agent holds no scratch at all. The worker's scratch is |
| + removed by systemd when its unit stops, whatever stopped it (§8.3). An |
| + agent that restarts mid-read loses the connection, the worker exits, and |
| + the console's lost rule ends the read. |
| +- **A bucket lifecycle rule** on `apps/*/db-reads/` and `control/db-reads/` |
| + (1 day) is an optional backstop for the AWS adapter. The sweep is the |
| + real cleanup, so a store without lifecycle rules loses nothing. |
| +- **Cost.** Each capable box does one LIST of an almost always empty prefix |
| + per serial move and every 30 s. A read adds a few PUTs, one GET per beat |
| + and one per chunk. The October bill was LIST and GET polling, so plan 1 |
| + measures the LIST rate on dev before the flagship roll and reports it. |
| + |
| +### 5.9 Replay and forgery |
| + |
| +| Attack | What stops it | |
| +|---|---| |
| +| A box writes its own request | No root signature: refused and logged once. | |
| +| An envelope copied to another id or region key | The id and region are inside the signature and must match the key. | |
| +| The same envelope rewritten after cleanup | `issued_at` older than 60 s is refused. Inside the window, the box keeps a seen set of ids for 5 minutes and refuses a second pick. Even if one ran, its output is sealed to the original CLI's key. | |
| +| A box serves a read for another region | It refuses any envelope whose region is not its own `CARLOS_REGION`, and IAM refuses the bucket anyway. | |
| +| A request that names a bucket or prefix | There is no such field. The box derives the prefix (§8.2). | |
| + |
| +## 6. Which operations forward |
| + |
| +| Operation | Phase | Reason | |
| +|---|---|---| |
| +| `query` | 1 | This is the regression the customer hit. The result is rows, usually small, and only rows leave the region. | |
| +| `query --at` | 1 | Same path with a target time. The box lists the replica the way the console's door does today (`listReplicaLevel`, `LastModified` as `CreatedAt`), so the copy stops at the same transaction an in-region read would. Plan 1 tests that parity. | |
| +| `pull` | 2 | The result is the whole database. It needs Paul's ruling on Q1 (ciphertext of a whole copy passing through the deployment bucket's region), and a measured size cap. In phase 2 the box ships the planned LTX files, sealed and windowed, and the CLI restores and checks them locally with today's code. The box never restores for a pull. | |
| +| `pull --at` | 2 | As `pull`, with the target time. | |
| +| Claim-mode door (`apidbclaim.go`) | not planned | Claims are not active anywhere. The claim door reads exact object versions through `hostops`'s versioned boundary and signed file handles, and lists no times. A box-side version of that belongs to claim activation (#250), which must add times anyway. A host-claim console keeps today's residency refusal, as it refuses `carlos run`. | |
| +| Seeds `--from` (`resolveDBSource`) | not planned | It shares the source checks but restores into a new instance, a different flow. Cross-region seeding is its own design. | |
| +| Fleet and pooled instances | refused, as today | Their data is not in a platform region bucket. | |
| + |
| +Phase 1 reaches every cross-region instance a member can read today in its |
| +own region, for queries. Until phase 2, `pull` answers DBF-8 for a |
| +cross-region instance. |
| + |
| +## 7. Who serves |
| + |
| +### 7.1 Capability and selection |
| + |
| +A box publishes `db-read-1` in its edges file only when all of these hold, |
| +as its own armed worker's answer, the way `offline-move-1` is published: |
| + |
| +- hibernation is configured (`CARLOS_HIBERNATE_BUCKET` set); |
| +- `CARLOS_REGION` is set; |
| +- `CARLOS_RELEASE_PUBKEY` is set, so it can verify envelopes; |
| +- the worker socket `/run/carlos-dbread/worker.sock` exists and is a socket; |
| +- `CARLOS_DB_READS` is not `0` (an operator off switch for one box); |
| +- `CARLOS_HOST_CLAIM` is not set. |
| + |
| +The console reads `edgesfile.ReadLiveSnapshot` before admitting a read: |
| + |
| +- No box in the record's region publishes `db-read-1`, or every one that |
| + does is draining: DBF-1, no request written. |
| +- Some do, but none has a fresh status snapshot: DBF-2, no request written. |
| +- Snapshots cannot be loaded: the request is written anyway and the |
| + pickup window decides. This follows the explicit placement door's |
| + generous read; a person is waiting and the expiry answers in 60 s. |
| + |
| +Any capable, live, non-draining box in the region may serve. There is no |
| +assignment by the console. Region HA (plan 2b, dormant) can put several |
| +boxes in a region, and they all read the same region bucket, so the |
| +create-only `picked` marker is the whole election. The 0 to 500 ms jitter |
| +spreads the race, and a box with no free slot or below the disk floor does not |
| +enter it. |
| + |
| +### 7.2 Why not the lease holder |
| + |
| +A sleeping instance holds no lease, so there is no holder to ask. An awake |
| +instance's holder is busy serving it, and the read does not touch the |
| +instance, so there is nothing to gain by sending it there. Choosing by lease |
| +would also tie the read to the instance's lifecycle, which is exactly what |
| +"never the live database" rules out. |
| + |
| +### 7.3 When no box serves |
| + |
| +| Situation | Answer | |
| +|---|---| |
| +| No capable box in the region | DBF-1 at once | |
| +| Capable boxes, none live | DBF-2 at once | |
| +| Live but nobody picks in 60 s (all busy, all below the disk floor, agent stuck) | DBF-3 after the window | |
| +| The picking box dies or restarts mid-read | DBF-12 after 90 s without a beat | |
| + |
| +## 8. Read-only and isolation on the box |
| + |
| +### 8.1 The split |
| + |
| +| Half | Runs as | Does | Never does | |
| +|---|---|---|---| |
| +| Agent (`carlos agent`, `carlos-edge.service`) | `carlos` | lists requests; verifies envelopes; creates `picked`; reads the record; derives the prefix; lists the replica and plans the restore from the listing; reads the last planned file's 100-byte header for the size check; streams planned files from the region bucket into the worker; seals and writes output; writes beat and result | opens SQLite, decodes LTX pages, runs SQL, writes files on disk, reads anything under `/data/instances`, takes or reads the lease | |
| +| Worker (`carlos system db-worker`, `carlos-dbread@.service`) | a `DynamicUser` | receives the plan and the files over its socket; writes them into its private `/var/tmp` as a litestream file replica; restores to the agent's target transaction; runs `lease.CheckDB`; runs the one statement with today's guards; streams rows back | anything with a network, the box's files, the bucket, other tenants' files | |
| + |
| +The rule is: everything that decodes page data or SQL runs in the worker. |
| +The member's SQL, the replica's pages and the restored database all do. The |
| +agent does object input and output and planning from listings, which it |
| +already does to wake instances. Its one decode is the last planned file's |
| +fixed 100-byte LTX header, for the size check, with the bounds |
| +`dbdoor.PlanRestore` already applies (a valid page size, a non-zero commit, |
| +TXIDs that match the listing). |
| + |
| +### 8.2 What the agent checks before the worker starts |
| + |
| +After winning `picked`, the agent reads the record fresh from the bucket and |
| +refuses (result `refused`) unless: |
| + |
| +- the record exists and its incarnation is the envelope's (`record-changed`); |
| +- the class rail passes (`dbClassRefusal`, moved where both console and box |
| + can call it); |
| +- it is not pooled and not a fleet placement; |
| +- `rec.Region` equals the box's `CARLOS_REGION` (`record-changed`); |
| +- `rec.Move` is nil (`moving`); |
| +- `rec.Host` is the envelope's host and passes `provision.ValidHostShape`. |
| + |
| +The replica prefix is `activator.HibernatePrefix(<the box's own |
| +CARLOS_HIBERNATE_PREFIX>, rec.Host)` in the box's own |
| +`CARLOS_HIBERNATE_BUCKET`, read through `lease.ReadReplicaClient` and held |
| +only as the narrow `replicaReader` interface (`LTXFiles`, `OpenLTXFile`), so |
| +no write method is reachable. Nothing in the request chooses a bucket, a |
| +prefix, a key or a level beyond what the plan picks. |
| + |
| +The agent lists only level directories. It never reads `lock.json`, never |
| +calls the leaser, and never wakes the instance. |
| + |
| +### 8.3 The worker unit |
| + |
| +`carlos-dbread.socket`, bound by root systemd: |
| + |
| +- `ListenStream=/run/carlos-dbread/worker.sock`, `SocketUser=carlos`, |
| + `SocketMode=0600`, `Accept=yes`, `MaxConnections=2`. |
| + |
| +`carlos-dbread@.service`, one instance per connection, starting from the |
| +claim-bound template's hardening (§3 fact 8) with these differences: |
| + |
| +- `DynamicUser=yes` instead of `User=carlos`, so the worker owns no file on |
| + the box and can read nothing `carlos` owns; |
| +- `PrivateNetwork=yes` and `RestrictAddressFamilies=AF_UNIX`, so no route to |
| + the metadata service, the bucket or anything else exists, rather than a |
| + deny list; |
| +- `TemporaryFileSystem=/data:ro /var/lib:ro /opt/carlos:ro /run/carlos:ro` |
| + with `BindReadOnlyPaths=/opt/carlos/carlos`, and |
| + `InaccessiblePaths=/etc/carlos /run/carlos-control`; |
| +- `PrivateTmp=yes`: its `/tmp` and `/var/tmp` are its own. The worker writes |
| + only under `/var/tmp`, which is on disk on Debian 13, where `/tmp` is a |
| + tmpfs. systemd deletes both when the unit stops, however it stops; |
| +- `MemoryMax=512M`, `CPUQuota=100%`, `TasksMax=16`, `LimitFSIZE` set to |
| + the restored-size cap (it bounds each file; the agent's disk check in |
| + §5.7 bounds the total), `RuntimeMaxSec=20min`; |
| +- `SystemCallFilter=@system-service`, `NoNewPrivileges=yes`, empty |
| + `CapabilityBoundingSet`, `ProtectProc=invisible`. |
| + |
| +`carlos system db-worker` refuses to run unless systemd handed it a |
| +connected socket on stdin (`Accept=yes`), so it cannot be started by hand or |
| +pointed at a file. |
| + |
| +The socket and service files are written and enabled by |
| +`ensureSystemUnits`, the same root path that installs the other units, on |
| +the next adopt pass or `update-platform-binary.sh`. |
| + |
| +Within the worker, the query keeps every guard the CLI has: `mode=ro`, |
| +`query_only`, `temp_store(2)`, `SQLITE_LIMIT_ATTACHED` 0 and |
| +`checkQuerySQL`. The database is opened with `immutable=1` as well, since |
| +nothing else ever writes the file. The guards and the query function move |
| +from `db.go` into a shared package (`internal/dbquery`), so the CLI and the |
| +worker run the same code and the output format cannot drift. |
| + |
| +### 8.4 Why a tenant cannot reach another tenant's data |
| + |
| +1. The console admits only a member of the account, for an instance of that |
| + account's app, and signs exactly that host into the envelope. |
| +2. The box derives the prefix from that host, the fresh record and its own |
| + config, and nothing else. |
| +3. The worker receives only the files the agent sent. It has no network, so |
| + no credentials from the metadata service and no bucket. Its filesystem |
| + view hides `/data`, `/etc/carlos` and `/run/carlos`, and its user owns |
| + nothing. ATTACH is off. If a SQLite flaw let the statement open a path, |
| + the sandbox leaves it nothing to open but its own scratch. |
| +4. The output is sealed to the requesting CLI's key. |
| + |
| +### 8.5 Why the live database is never touched |
| + |
| +The agent reads bucket objects only. The worker cannot see `/data`. Nothing |
| +calls the activator, the leaser or `lock.json`. A read of a busy instance's |
| +replica can race litestream's compaction, which is today's "changed while it |
| +was being read, twice" case, and is retried the same way. |
| + |
| +## 9. Console and CLI behaviour |
| + |
| +### 9.1 New console routes |
| + |
| +``` |
| +POST /api/cli/apps/{account}/{app}/db/{host}/reads |
| +GET /api/cli/apps/{account}/{app}/db/{host}/reads/{id} |
| +POST /api/cli/apps/{account}/{app}/db/{host}/reads/{id}/ack |
| +POST /api/cli/apps/{account}/{app}/db/{host}/reads/{id}/cancel |
| +``` |
| + |
| +All four are `svcrole.Set{}` (people only), behind the same gate order as |
| +the db door. A read id is checked against its account, app and host on every |
| +call; a mismatch is the byte-identical 404. A host-claim console answers all |
| +four with today's residency sentence. |
| + |
| +### 9.2 Mixed versions |
| + |
| +| CLI | Console | What happens | |
| +|---|---|---| |
| +| old | old | today's residency sentence | |
| +| old | new, forwarding off or region not capable | today's sentence, or DBF-1 or DBF-2 | |
| +| old | new, forwarding on and region capable | DBF-14: update carlos | |
| +| new | old | today's sentence; no forward marker, so the CLI prints it as is | |
| +| new | new | the forwarded read | |
| + |
| +A console with `CARLOS_DB_FORWARD` unset answers byte-identically to main. |
| + |
| +### 9.3 Audit |
| + |
| +One `db-read` activity row at admission, in the run's manner, with the |
| +detail DBF-21 or DBF-22. The SQL text is not recorded. That differs from |
| +`carlos run`, which records argv, and matches the in-region door, which |
| +never sees SQL. A query's literals are data, and the audit log is not the |
| +place for them. The console keeps the SQL only inside the envelope, which is |
| +deleted with the read. |
| + |
| +## 10. Errors and copy |
| + |
| +Every new user-facing sentence is below. Error and status strings follow |
| +the run door's casing: lower case at the start and after a full stop, no em |
| +dashes. Docs prose keeps the CLI page's own sentence case. None of this is |
| +approved yet; the copy-review gate runs before any of it is written into |
| +code. |
| + |
| +| ID | Where | Text | |
| +|---|---|---| |
| +| DBF-1 | console 409, no capable box in the region | `%s keeps its data in region %s. no server in that region can read databases yet, so carlos db cannot read it for now.` | |
| +| DBF-2 | console 409, capable boxes but none live | `%s keeps its data in region %s. no server in that region is answering right now. try again in a few minutes.` | |
| +| DBF-3 | CLI, nobody picked the read in 60 s | `no server in region %s picked up the read within 60 seconds. try again in a few minutes.` | |
| +| DBF-4 | CLI, `too-large` | `%s's database is %s once restored. carlos db can query at most %s in region %s.` | |
| +| DBF-5 | CLI, `no-room` | `the server in region %s does not have room to restore %s's database right now. try again later.` | |
| +| DBF-6 | CLI stderr, `output-cap` | `the result was cut off at 64 MB. narrow the query with WHERE or LIMIT and run it again.` | |
| +| DBF-7 | CLI, `timeout` | `the query ran for 5 minutes and was stopped. narrow it and run it again.` | |
| +| DBF-8 | CLI, `pull` before phase 2 | `%s keeps its data in region %s. carlos db pull cannot fetch a copy from there yet. carlos db query can read it.` | |
| +| DBF-9 | console 429, open-read bounds | `too many database reads are open. wait a minute and try again.` | |
| +| DBF-10 | console 409, record moving | `%s is moving between regions. read its database once the move has finished.` | |
| +| DBF-11 | CLI stderr while waiting | `reading a copy of %s's database in region %s.` | |
| +| DBF-12 | CLI, `lost` or `interrupted` | `the server reading %s's database in region %s stopped before it finished. try again.` | |
| +| DBF-13 | CLI, a chunk or summary fails to open or add up | `the answer from region %s did not check out, so carlos db showed none of it. try again.` | |
| +| DBF-14 | console 409 with the forward marker, read by an old CLI | `%s keeps its data in region %s. this version of carlos cannot read it from there. run carlos update, then try again.` | |
| +| DBF-15 | CLI, `record-changed` | `%s changed while the read was waiting. try again.` | |
| +| DBF-16 | CLI, `moving` from the box | as DBF-10 | |
| +| DBF-21 | audit detail, query | `forwarded a database query to region %s (cli)` | |
| +| DBF-22 | audit detail, query with `--at` | `forwarded a database query to region %s, as of %s (cli)` | |
| +| DBF-30 | `docs/site/cli.md`, `carlos db`, replacing the paragraph that starts "If the app has more than one instance" | see below | |
| +| DBF-31 | `docs/site/cli.md`, `carlos system`, the worker | `carlos system db-worker runs on platform boxes only. systemd starts it for each forwarded database query. you never run it yourself.` | |
| + |
| +DBF-30: |
| + |
| +> If the app has more than one instance, name one with `--instance`. Only |
| +> instances that sleep have a replica. An instance on your own fleet or in a |
| +> pool cannot be read this way, and the command says which. |
| +> |
| +> When an instance keeps its data in another region than the console's, |
| +> `carlos db query` runs in that region. A server there restores the copy, |
| +> runs your query and sends back only the rows. The rows are encrypted for |
| +> your terminal on the way, so nothing in between can read them. There, the |
| +> copy can be at most 2 GB once restored, the query runs for at most 5 |
| +> minutes, and the rows can come to at most 64 MB. `carlos db pull` cannot |
| +> fetch a copy from another region yet. |
| + |
| +Unchanged and reused: today's status line (`dbQueryStatus`), the |
| +`dbRestoreError` sentences for `nothing-replicated`, `history-gone`, |
| +`check-failed` and `changed-twice`, SQLite's own error text for |
| +`sql-error`, `stopped` for Ctrl-C, the guard refusals of `checkQuerySQL`, |
| +and the residency sentence when forwarding is off. |
| + |
| +DBF-17 to DBF-20 and DBF-23 to DBF-29 are left free for phase 2's pull rows. |
| + |
| +## 11. Rollout |
| + |
| +### 11.1 Switches and gates |
| + |
| +- **Box:** `db-read-1` (§7.1). Published once the binary, the units and the |
| + agent's worker loop are in place. `CARLOS_DB_READS=0` withdraws one box. |
| +- **Console:** `CARLOS_DB_FORWARD=1` turns forwarding on. Unset, every |
| + answer is byte-identical to main. The console also needs the root key. |
| +- **CLI:** none. The forward marker negotiates (§9.2). |
| +- **Version floor:** none beyond the capability. A box that predates it |
| + publishes no capability and is never asked. |
| + |
| +### 11.2 Mixed fleet |
| + |
| +A region with one capable box and one old box works: only the capable box |
| +lists the request prefix. A region with no capable box gets DBF-1. A box |
| +that loses the capability mid-read finishes the read it holds and takes no |
| +new one. |
| + |
| +### 11.3 Roll order |
| + |
| +Paul's order (dev boxes, dev console, check, flagship boxes, flagship |
| +console). Boxes first publish an idle capability. The dev console turns |
| +`CARLOS_DB_FORWARD=1` and the check is a forwarded query from a dev console |
| +to a dev box in another region, plus the LIST-rate measurement (§5.8). Then |
| +the flagship boxes, then the flagship console. |
| + |
| +### 11.4 Tests |
| + |
| +Acceptance is production-wired. Stand-ins are allowed in unit tests and |
| +labelled so; they are not acceptance. |
| + |
| +| # | Test | Wiring | |
| +|---|---|---| |
| +| T1 | A forwarded query end to end: console in region ie, agent in region uk, two MinIO buckets. The console's credentials cannot read the uk bucket (a MinIO policy), so any console read of it fails the test. | real console server, real agent loop, real stores; CI runs it with the worker reached through a test socket, labelled non-acceptance | |
| +| T2 | The same on the root runner with the real `carlos-dbread.socket` and `carlos-dbread@.service` installed by the real `ensureSystemUnits` | acceptance gate for plan 1 | |
| +| T3 | Sandbox witness on the root runner, from inside the worker: open `/data/registry.db`, `/etc/carlos/host.env` and another instance's directory; connect to `169.254.169.254:80` and to any TCP address; write outside `/var/tmp`; `ATTACH` and `VACUUM INTO` with the attach limit removed (a mutation); `load_extension`. Every one fails. The private tmp is gone after a normal end, a cancel and `kill -9` of the agent mid-read. | real units | |
| +| T4 | Forgery and replay: unsigned, wrong key, scoped key, envelope moved to another id and to another region, replay after cleanup, a chunk sealed to another key, a chunk reordered, the last chunk dropped. | real verify and seal code | |
| +| T5 | Two capable boxes in one region: exactly one runs each read, across 1,000 reads with random jitter. Expiry against a late pick: one `picked` exists, and the result matches it. | real store CAS | |
| +| T6 | Parity: the forwarded query and the in-region query print byte-identical stdout for a fixture database (NULL, tabs, newlines, blobs, times). `--at` stops at the same transaction as the in-region door for the same replica and time, including a write in the same second as the target. | real restore, both paths | |
| +| T7 | Cleanup: after ack, cancel, expiry, lost and `KeepFor`, nothing remains under either prefix. | real sweep | |
| +| T8 | Mixed versions: the five rows of §9.2, with the forwarding-off 409 compared byte for byte against main. | real console | |
| +| T9 | Record changes between request and pick: moved, recreated (new incarnation), class changed. Each is refused with its code. | real records | |
| +| T10 | Approved copy digests for every row in §10, in the `approved_copy_test.go` pattern. | | |
| + |
| +Mutation checks that must turn a test red: drop envelope verification (T4); |
| +take the host or prefix from anywhere but the record and config (T1 with a |
| +second tenant); remove `PrivateNetwork` (T3); skip the summary hash check |
| +(T4). |
| + |
| +## 12. Provider boundary |
| + |
| +- **Core:** the transport is `release.Store` objects, so it runs on MinIO and |
| + any S3-compatible store. The box streams each planned file with a GET from |
| + its region store and never uses a server-side copy, since a self-hosted |
| + deployment's region stores may be different servers. Signing is ed25519, |
| + sealing is the standard library plus `x/crypto/hkdf`. The sandbox is |
| + systemd, which every box already needs. |
| +- **AWS adapter:** IAM is what makes the console unable to read another |
| + region's bucket on the flagship, and the optional lifecycle rule (§5.8). |
| + Neither is required for the design to work. A self-hosted deployment with |
| + separate stores per region gets the same split from its own credentials. |
| +- **What a non-AWS deployment loses:** nothing. The capability, the |
| + liveness join and the box's identity use no AWS API. Multi-box regions are |
| + AWS-only today (H26), but nothing here depends on them; one box per region |
| + is enough. |
| +- **Build:** the worker and its socket code are Linux-only, with |
| + `//go:build linux` and a `!linux` stub, so the CLI still cross-compiles for |
| + darwin and windows. |
| + |
| +## 13. Census |
| + |
| +- **Plan 14 root-read census.** No root unit gains a read. The agent's new |
| + sites (the request listing and the record read) run as `carlos` in |
| + `carlos-edge.service` and take the J1 and J4 dispositions their siblings |
| + in the offline move took. The worker's sites (`Open`, `QueryContext`, the |
| + scratch writes) run as a `DynamicUser` and are classified as non-root. |
| + Each plan proposes an overlay in `docs/superpowers/notes/` for a ruling on |
| + #280 and adds its `plan14-root-reads-*.tsv`. |
| +- **Root writes.** `ensureSystemUnits` writes two new unit files. That is a |
| + root-written artifact and joins the Plan 14 write census as an overlay, as |
| + every new unit has. |
| +- **9B8 record-write census.** No record writer is added. The new `Put` |
| + sites (request, `picked`, beat, chunks, ack, cancel, result) are |
| + `nonrecord`. `registerCLIRoutes` changes as a `key-site` for the four |
| + routes. |
| +- **Claim routes.** The four routes join |
| + `internal/console/testdata/claim-routes.txt`, refused on a host-claim |
| + console like the run's. |
| +- **go.mod.** No new module, so no root-envelope overlay for it. |
| + |
| +## 14. Plans |
| + |
| +1. **Box side and wire, dormant.** `internal/dbreads` (ids, keys, the |
| + envelope, sealing, the result, limits); `internal/dbquery` (the query |
| + runner and guards moved out of `db.go`, with the CLI switched to it and |
| + no behaviour change); `dbdoor` planning over any `replicaReader`; the |
| + agent's db-reads loop; `carlos system db-worker`; the two units in |
| + `ensureSystemUnits`; the `db-read-1` capability. Gates: T2, T3, T4, T5, |
| + the parity half of T6 that needs no console. Census overlay. Nothing |
| + reaches a box unless a console asks. |
| +2. **Console and CLI for `query` and `query --at`.** The four routes, the |
| + forward marker, capability and liveness checks, the bounds, expiry and |
| + lost handling in the run sweep, audit, the CLI's forwarded path, DBF |
| + copy, `docs/site/cli.md`. Gates: T1, T6, T7, T8, T9, T10, copy review, |
| + census overlay. Roll per §11.3. |
| +3. **`pull` and `pull --at`.** After Paul's Q1 ruling. Sealed LTX files with |
| + the same window, a raw chunk route for 4 MiB chunks, local restore and |
| + `CheckDB` in the CLI, and the pull copy rows. |
| + |
| +## 15. Review focus: the five likeliest failure modes |
| + |
| +1. **The worker reaches more than its copy.** A unit option missed or |
| + overridden (network, `/data`, `/etc/carlos`, the metadata service), or a |
| + SQLite path (ATTACH, `VACUUM INTO`, an extension, a temp file) that opens |
| + something. Any one of these turns a member's SQL into the box role, which |
| + reads every tenant in the region. |
| +2. **The prefix or host comes from the request.** Any path where an |
| + envelope field, rather than the fresh record and the box's config, |
| + picks the bucket, the prefix or the host is a cross-tenant read. So is |
| + a gap between the incarnation check and the read. |
| +3. **Forgery and replay.** An unsigned or wrongly scoped request accepted, |
| + domain separation from other root envelopes missing, the region not |
| + bound, or a replay inside the window that runs twice. |
| +4. **Races and leaks.** Two boxes running one read, expiry and a late pick |
| + both "winning", a result written before its last chunk, ciphertext or |
| + worker scratch left behind after a crash, a full disk on a 60 GiB box, |
| + or an unacked window that never drains. |
| +5. **Drift between the two paths.** The forwarded `--at` stopping at a |
| + different transaction than the in-region door, output formatting that |
| + differs between CLI and box versions, or a capability published by a box |
| + whose worker socket is not actually armed. |
| + |
| +## 16. Open questions for Paul |
| + |
| +1. **Q1, pull through the deployment bucket.** In phase 2 a pull's planned |
| + LTX files pass through the deployment bucket (eu-west-1 on the flagship) |
| + as ciphertext only the member's CLI can open, at most 64 MiB at a time |
| + (16 unacked chunks of 4 MiB) and for at most an hour. The db spec rejected plaintext there. Is sealed and |
| + windowed acceptable, or should cross-region pull stay refused? |
| +2. **Q2, limits.** 2 GiB restored, 5 minutes, 64 MB of rows, two reads per |
| + box at once. Right for launch? Real database sizes on the flagship are |
| + unmeasured. |
| +3. **Q3, audit.** The audit row records that a query ran, not its SQL. That |
| + differs from `carlos run`, which records argv. |
| +4. **Q4, copy casing.** The new strings start lower case after a full stop, |
| + as the run door's do. The existing db door strings capitalise after a |
| + full stop ("too many database requests. Wait a minute..."). Keep the new |
| + ones as drafted, or match the db door? |
| +5. **Q5, one path or two.** In-region reads keep today's path (the CLI |
| + restores locally, the console streams LTX). Forwarded queries run on a |
| + box with limits. Should in-region queries move to the box path later, so |
| + there is one behaviour, or should the two stay apart? |
| + |
| +## 17. What this does not do |
| + |
| +- No console per region and no console-to-box connection. |
| +- No change to in-region reads, the claim-mode door, seeds `--from`, fleet |
| + or pooled instances. |
| +- No IAM change. No new read authority for any role. |
| +- No `carlos db shell` and no browser view. |