amadan

carlos / platform Public

db-region-forward-3: carlos db pull forwarded to the data's region; LTX header checks for query and pull

Squashed from db-region-forward-3 into main.
claude 64a002c8a6d28bd766ef9171c68800a359e2c049 parent 0d9e81d
34 files changed, +1956 −94
  • .amadan/test-weights.tsv +4 −0
  • approved_db_forward_copy_test.go +13 −7
  • db.go +7 −2
  • dbforward.go +315 −26
  • dbforward_e2e_test.go +184 −0
  • dbforward_test.go +394 −5
  • docs/site/cli.md +2 −2
  • docs/superpowers/notes/census-proposal-db-region-forward-3.md +76 −0
  • docs/superpowers/plans/2026-10-22-db-region-forward-copy.md +16 −2
  • docs/superpowers/specs/2026-10-22-db-region-forward-design.md +7 −1
  • internal/console/apidbforward.go +36 −12
  • internal/console/apidbforward_test.go +59 −5
  • internal/console/dbforward_minio_test.go +33 −0
  • internal/dbdoor/plan_from.go +34 −1
  • internal/dbdoor/plan_from_test.go +66 −0
  • internal/dbreads/agent.go +15 −3
  • internal/dbreads/chunks.go +7 −6
  • internal/dbreads/envelope.go +12 −4
  • internal/dbreads/envelope_test.go +19 −0
  • internal/dbreads/fixture_test.go +26 −2
  • internal/dbreads/limits.go +7 −0
  • internal/dbreads/lines.go +3 −2
  • internal/dbreads/lines_test.go +10 −0
  • internal/dbreads/pull_test.go +325 −0
  • internal/dbreads/read.go +36 −8
  • internal/dbreads/read_test.go +1 −1
  • internal/dbreads/worker.go +86 −1
  • internal/host/dbreads.go +15 −3
  • internal/host/dbreads_rootrunner_linux_test.go +48 −0
  • internal/host/dbreads_test.go +30 −0
  • internal/provision/db_region_forward_3_census_test.go +28 −0
  • internal/provision/plan14_root_reads_test.go +18 −0
  • internal/provision/plan9_b8_census_test.go +1 −1
  • internal/provision/testdata/plan14-root-reads-db-region-forward-3.tsv +23 −0

CI passed — run details

CI log 
=== gate ===
.amadan/test-gate.sh
=== launchctx (1 s)
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
=== vet (15 s)
=== gofmt (1 s)
gofmt -l .: clean
=== rest (167 s)
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.549s
ok  	github.com/carlosframework/platform/cmd/moveprobe	11.663s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.730s
ok  	github.com/carlosframework/platform/internal/activator	58.488s
ok  	github.com/carlosframework/platform/internal/agentsock	0.024s
ok  	github.com/carlosframework/platform/internal/alerting	0.005s
ok  	github.com/carlosframework/platform/internal/analytics	0.020s
ok  	github.com/carlosframework/platform/internal/appconfig	2.796s
ok  	github.com/carlosframework/platform/internal/apperr	0.105s
ok  	github.com/carlosframework/platform/internal/applog	0.165s
ok  	github.com/carlosframework/platform/internal/awsinv	0.005s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.008s
ok  	github.com/carlosframework/platform/internal/boxid	0.709s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.012s
ok  	github.com/carlosframework/platform/internal/caps	0.019s
ok  	github.com/carlosframework/platform/internal/certwarm	0.007s
ok  	github.com/carlosframework/platform/internal/cgroup	0.718s
ok  	github.com/carlosframework/platform/internal/changelog	0.401s
ok  	github.com/carlosframework/platform/internal/claimactivation	0.972s
ok  	github.com/carlosframework/platform/internal/clamd	0.056s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	8.232s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.015s
ok  	github.com/carlosframework/platform/internal/dbquery	0.006s
ok  	github.com/carlosframework/platform/internal/dbreads	2.117s
?   	github.com/carlosframework/platform/internal/dbreads/dbreadstest	[no test files]
ok  	github.com/carlosframework/platform/internal/dbverify	0.007s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.003s
ok  	github.com/carlosframework/platform/internal/docsite	0.012s
ok  	github.com/carlosframework/platform/internal/domains	2.771s
ok  	github.com/carlosframework/platform/internal/doorkey	0.011s
ok  	github.com/carlosframework/platform/internal/economics	0.010s
ok  	github.com/carlosframework/platform/internal/edge	45.288s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.019s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.192s
ok  	github.com/carlosframework/platform/internal/gate	1.750s
ok  	github.com/carlosframework/platform/internal/genselect	0.032s
ok  	github.com/carlosframework/platform/internal/geo	0.011s
ok  	github.com/carlosframework/platform/internal/geoip	0.046s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.228s
ok  	github.com/carlosframework/platform/internal/host	142.056s
ok  	github.com/carlosframework/platform/internal/hostactors	6.487s
ok  	github.com/carlosframework/platform/internal/hostactors/awsmembership	0.012s
ok  	github.com/carlosframework/platform/internal/hostbox	2.984s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.726s
ok  	github.com/carlosframework/platform/internal/hostenv	0.005s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.013s
ok  	github.com/carlosframework/platform/internal/hostexec	5.345s
ok  	github.com/carlosframework/platform/internal/hostfence	0.029s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.377s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.021s
ok  	github.com/carlosframework/platform/internal/hostguard	1.026s
ok  	github.com/carlosframework/platform/internal/hostops	1.939s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.012s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.012s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.612s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.259s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.012s
ok  	github.com/carlosframework/platform/internal/infraview	0.015s
ok  	github.com/carlosframework/platform/internal/instancectl	5.238s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.863s
ok  	github.com/carlosframework/platform/internal/instancefs	0.006s
ok  	github.com/carlosframework/platform/internal/instancekey	0.016s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.022s
ok  	github.com/carlosframework/platform/internal/layout	1.759s
ok  	github.com/carlosframework/platform/internal/lbhealth	0.015s
ok  	github.com/carlosframework/platform/internal/lease	10.137s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.553s
ok  	github.com/carlosframework/platform/internal/ledger	0.027s
ok  	github.com/carlosframework/platform/internal/mail	0.187s
ok  	github.com/carlosframework/platform/internal/maintenance	0.090s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.524s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.096s
ok  	github.com/carlosframework/platform/internal/metrics	0.011s
ok  	github.com/carlosframework/platform/internal/moves	21.800s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.930s
ok  	github.com/carlosframework/platform/internal/objstore	0.243s
ok  	github.com/carlosframework/platform/internal/ops	0.214s
ok  	github.com/carlosframework/platform/internal/park	0.343s
ok  	github.com/carlosframework/platform/internal/plans	0.006s
ok  	github.com/carlosframework/platform/internal/privatefile	0.004s
ok  	github.com/carlosframework/platform/internal/provision	20.150s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.014s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.540s
ok  	github.com/carlosframework/platform/internal/relay	7.245s
ok  	github.com/carlosframework/platform/internal/release	25.372s
ok  	github.com/carlosframework/platform/internal/rollout	3.832s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.004s
ok  	github.com/carlosframework/platform/internal/rootaws	0.007s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.061s
ok  	github.com/carlosframework/platform/internal/rootstate	0.028s
ok  	github.com/carlosframework/platform/internal/runs	0.607s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.004s
ok  	github.com/carlosframework/platform/internal/schedule	2.664s
ok  	github.com/carlosframework/platform/internal/seeds	0.015s
ok  	github.com/carlosframework/platform/internal/sim/engines	2.627s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	10.092s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	4.151s
ok  	github.com/carlosframework/platform/internal/status	0.074s
ok  	github.com/carlosframework/platform/internal/steering	0.009s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.269s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.010s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.006s
ok  	github.com/carlosframework/platform/internal/unixdial	0.003s
ok  	github.com/carlosframework/platform/internal/updatewords	0.011s
ok  	github.com/carlosframework/platform/internal/usage	0.008s
ok  	github.com/carlosframework/platform/internal/version	0.003s
ok  	github.com/carlosframework/platform/internal/vet	1.646s
=== internal_console@0 (82 s)
ok  	github.com/carlosframework/platform/internal/console	79.906s
=== internal_console@1 (44 s)
ok  	github.com/carlosframework/platform/internal/console	41.702s
=== internal_console@2 (43 s)
ok  	github.com/carlosframework/platform/internal/console	40.987s
=== internal_console@3 (41 s)
ok  	github.com/carlosframework/platform/internal/console	38.973s
=== internal_console@4 (43 s)
ok  	github.com/carlosframework/platform/internal/console	41.046s
=== internal_console@5 (42 s)
ok  	github.com/carlosframework/platform/internal/console	39.772s
=== internal_console@6 (37 s)
ok  	github.com/carlosframework/platform/internal/console	34.995s
=== internal_console@7 (46 s)
ok  	github.com/carlosframework/platform/internal/console	43.815s
=== root@0 (110 s)
ok  	github.com/carlosframework/platform	107.833s
=== root@1 (83 s)
ok  	github.com/carlosframework/platform	81.242s
=== root@2 (95 s)
ok  	github.com/carlosframework/platform	92.840s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50528418 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: cached, digest verified
  mc: cached, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.022s
=== RUN   TestT4StorageProbeMinio
2026/10/23 20:54:40 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E143FCEB5B6C1B, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.021s
=== RUN   TestS3StoreMinioContract
=== RUN   TestS3StoreMinioContract/conditional_get_and_etag_listing
=== RUN   TestS3StoreMinioContract/bump_through_every_mutation_path
--- PASS: TestS3StoreMinioContract (0.02s)
    --- PASS: TestS3StoreMinioContract/conditional_get_and_etag_listing (0.01s)
    --- PASS: TestS3StoreMinioContract/bump_through_every_mutation_path (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/release	0.031s
Bucket created successfully `local/dbf-deploy`.
Bucket created successfully `local/dbf-data-ie`.
Bucket created successfully `local/dbf-data-uk`.
local/dbf-deploy versioning is enabled
Created policy `dbf-console` successfully.
Created policy `dbf-ukbox` successfully.
Added user `dbfconsole` successfully.
Added user `dbfukbox` successfully.
Attached Policies: [dbf-console]
To User: dbfconsole
Attached Policies: [dbf-ukbox]
To User: dbfukbox
=== RUN   TestDBForwardMinio
2026/10/23 20:54:42 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/23 20:54:42 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/23 20:54:42 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/23 20:54:42 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/23 20:54:42 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/23 20:54:42 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/23 20:54:42 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/23 20:54:42 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/23 20:54:42 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/23 20:54:42 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/23 20:54:42 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/23 20:54:42 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/23 20:54:42 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/23 20:54:42 console: could not notify member@example.com of a CLI token issue: keymail: lookup member@example.com: keymail: GET /api/federation/lookup?addr=member%40example.com&from=console%40127.0.0.1%3A34111: could not determine target mailbox
    agent.go:135: dbreads: dbr_3otkjmg66y7uccam5p6svzew5m: planned 3 files (1544 bytes) and restored to 3 in 9ms
    agent.go:135: dbreads: dbr_3otkjmg66y7uccam5p6svzew5m: output 5 bytes in 1 confirmed chunks in 1ms
2026/10/23 20:54:43 console: db read dbr_3otkjmg66y7uccam5p6svzew5m: relayed 1 chunks (102 bytes) in 1.004s, ended done
    agent.go:135: dbreads: dbr_jj5cuqicirliy645avvxpkye6a: planned 1 files (720 bytes) and restored to 1 in 8ms
    agent.go:135: dbreads: dbr_jj5cuqicirliy645avvxpkye6a: output 5 bytes in 1 confirmed chunks in 1ms
2026/10/23 20:54:44 console: db read dbr_jj5cuqicirliy645avvxpkye6a: relayed 1 chunks (102 bytes) in 1.004s, ended done
    agent.go:135: dbreads: dbr_z4ntllfb7su3epbglhkcfzh5zu: planned 3 files (1544 bytes) and held through 3 in 8ms
    agent.go:135: dbreads: dbr_z4ntllfb7su3epbglhkcfzh5zu: output 1544 bytes in 1 confirmed chunks in 1ms
2026/10/23 20:54:45 console: db read dbr_z4ntllfb7su3epbglhkcfzh5zu: relayed 1 chunks (1641 bytes) in 1.004s, ended done
=== RUN   TestDBForwardMinio/by_disconnect
=== NAME  TestDBForwardMinio
    agent.go:135: dbreads: dbr_3nnkmfs2lzf7ydarjew452ax3q: planned 3 files (1544 bytes) and restored to 3 in 9ms
2026/10/23 20:54:45 console: db read dbr_3nnkmfs2lzf7ydarjew452ax3q: relayed 0 chunks (0 bytes) in 62ms, ended client gone
    agent.go:135: dbreads: dbr_3nnkmfs2lzf7ydarjew452ax3q: output 5 bytes in 1 confirmed chunks in 104ms
=== RUN   TestDBForwardMinio/by_timeout
=== NAME  TestDBForwardMinio
    agent.go:135: dbreads: dbr_rk5tp3g4zgqmlk5vtzmi7aa7gi: planned 3 files (1544 bytes) and restored to 3 in 9ms
2026/10/23 20:55:05 console: db read dbr_rk5tp3g4zgqmlk5vtzmi7aa7gi: relayed 0 chunks (0 bytes) in 19.107s, ended no-result
    agent.go:135: dbreads: dbr_rk5tp3g4zgqmlk5vtzmi7aa7gi: output 5 bytes in 1 confirmed chunks in 19.083s
--- PASS: TestDBForwardMinio (22.56s)
    --- PASS: TestDBForwardMinio/by_disconnect (0.24s)
    --- PASS: TestDBForwardMinio/by_timeout (19.22s)
PASS
ok  	github.com/carlosframework/platform/internal/console	22.573s
minio contract: PASS
diff --git a/.amadan/test-weights.tsv b/.amadan/test-weights.tsv
index 5a4a098fc..6241b5499 100644
--- a/.amadan/test-weights.tsv
+++ b/.amadan/test-weights.tsv
@@ -10,6 +10,7 @@ github.com/carlosframework/platform/internal/console TestClaimDeleteRetryWhilePa
github.com/carlosframework/platform/internal/console TestClaimProbeRunnerDetectsAHollowRequest 3.2
github.com/carlosframework/platform/internal/console TestDBReadEachTenantGetsItsOwnRows 3.2
github.com/carlosframework/platform/internal/console TestDBReadSlowRelayEndsAtTheBound 3.0
+github.com/carlosframework/platform/internal/console TestDBReadPullStreamsTheFiles 2.8
github.com/carlosframework/platform/internal/console TestClaimPurgeMailRecordRefusesBeforeEffects 2.5
github.com/carlosframework/platform/internal/console TestDBReadStalledStoreKeepsTheStreamAndItsBound 2.5
github.com/carlosframework/platform/internal/console TestDBReadEndedDuringAdmissionWritesOnlyTheCancel 2.4
@@ -39,6 +40,9 @@ github.com/carlosframework/platform/internal/console TestDBReadRequestBoundClamp
github.com/carlosframework/platform TestDBForwardPrintsNothingForABadAnswer 67.0
github.com/carlosframework/platform TestDBForwardLongRequestThroughTheEdge 44.5
github.com/carlosframework/platform TestDBForwardThroughTheBackhaulHop 44.1
+github.com/carlosframework/platform TestDBForwardPullParityThroughTheCLI 4.8
github.com/carlosframework/platform TestDBForwardParityThroughTheCLI 4.6
+github.com/carlosframework/platform TestDBForwardPullInterruptedLeavesNothing 2.1
github.com/carlosframework/platform TestDBForwardMixedVersions 1.9
+github.com/carlosframework/platform TestDBForwardPullSavesNothingForABadAnswer 1.8
github.com/carlosframework/platform TestDBForwardRefusesSQLThatIsNotUTF8 1.0
diff --git a/approved_db_forward_copy_test.go b/approved_db_forward_copy_test.go
index fc172bf0e..9eebffc01 100644
--- a/approved_db_forward_copy_test.go
+++ b/approved_db_forward_copy_test.go
@@ -24,11 +24,15 @@ const dbForwardCopyRecordPath = "docs/superpowers/plans/2026-10-22-db-region-for
var dbForwardCopyIDPattern = regexp.MustCompile(`^DBF-[0-9]+$`)
-// dbForwardCopyIDs is the record's inventory, pinned: DBF-1 to DBF-16,
-// DBF-21, DBF-22, DBF-30, DBF-31 and DBF-32. Plan 3 adds its pull rows here.
+// dbForwardCopyIDs is the record's inventory, pinned: DBF-1 to DBF-20,
+// DBF-23 and DBF-24 (plan 3's pull rows among them), then DBF-21, DBF-22,
+// DBF-30, DBF-31 and DBF-32.
var dbForwardCopyIDs = func() []string {
- ids := make([]string, 0, 21)
- for i := 1; i <= 16; i++ {
+ ids := make([]string, 0, 27)
+ for i := 1; i <= 24; i++ {
+ if i == 21 || i == 22 {
+ continue // appended below, in the record's order
+ }
ids = append(ids, "DBF-"+strconv.Itoa(i))
}
return append(ids, "DBF-21", "DBF-22", "DBF-30", "DBF-31", "DBF-32")
@@ -40,7 +44,9 @@ var dbForwardCopyIDs = func() []string {
var dbForwardCopyBuilt = []string{
"DBF-1", "DBF-2", "DBF-14", "DBF-9", "DBF-10", "DBF-21", "DBF-22", // the console
"DBF-3", "DBF-4", "DBF-5", "DBF-6", "DBF-7", "DBF-8", "DBF-11", "DBF-12", "DBF-13", "DBF-15", "DBF-16", "DBF-32", // the CLI
- "DBF-30", "DBF-31", // docs/site/cli.md
+ "DBF-19", "DBF-20", // the console's pull audit details (plan 3)
+ "DBF-17", "DBF-18", "DBF-24", // the CLI's pull sentences (plan 3)
+ "DBF-23", "DBF-31", // docs/site/cli.md (DBF-23 replaced DBF-30 there)
}
// dbForwardCopyWired is every id whose PENDING text the code already
@@ -53,8 +59,8 @@ var dbForwardCopyBuilt = []string{
// that has not turned forwarding on. The copy review moves each id from here
// to dbForwardCopyBuilt, applying Paul's edits in the same commit. Docs rows
// are public the moment the site syncs, so they are never wired. Luna
-// approved every row on 2026-10-23, so the list is empty and every row is
-// built.
+// approved every plan 2 row and every plan 3 pull row on 2026-10-23, so the
+// list is empty and every row is built.
var dbForwardCopyWired = []string{}
func mustParseDBForwardRecord(t *testing.T) []copyRow {
diff --git a/db.go b/db.go
index 0757bdde4..bf697ae3d 100644
--- a/db.go
+++ b/db.go
@@ -284,8 +284,13 @@ func dbPull(ctx context.Context, t dbTarget, out string) error {
r, err := dbRestore(ctx, t)
var fwd *errDBForward
if errors.As(err, &fwd) {
- // Phase 1 forwards queries only (db region forward spec §6): DBF-8.
- return fmt.Errorf(dbfNoPull, t.host, fwd.door.Forward.Region)
+ // The copy's files come from the region that holds them (db region
+ // forward spec §6), when the console forwards pulls there.
+ if !slices.Contains(fwd.door.Forward.Ops, dbreads.OpPull) {
+ // A console that forwards queries and not pulls (plan 2's): DBF-8.
+ return fmt.Errorf(dbfNoPull, t.host, fwd.door.Forward.Region)
+ }
+ r, err = dbPullForward(ctx, t, *fwd.door.Forward)
}
if err != nil {
return err
diff --git a/dbforward.go b/dbforward.go
index 491df7887..5bf8af68a 100644
--- a/dbforward.go
+++ b/dbforward.go
@@ -1,9 +1,11 @@
-// dbforward.go is the CLI's half of a forwarded `carlos db query` (spec
-// docs/superpowers/specs/2026-10-22-db-region-forward-design.md §5.5): when
-// the db door answers with the forward marker, the CLI makes an ephemeral
-// key, holds one request to the console's reads route open, buffers every
-// sealed chunk, and prints nothing until a done result's summary checks out
-// against the bytes (codex 6).
+// dbforward.go is the CLI's half of a forwarded `carlos db query` and
+// `carlos db pull` (spec
+// docs/superpowers/specs/2026-10-22-db-region-forward-design.md §5.5, §6):
+// when the db door answers with the forward marker, the CLI makes an
+// ephemeral key, holds one request to the console's reads route open, keeps
+// every sealed chunk (in memory for a query, spooled to a private directory
+// for a pull), and prints or saves nothing until a done result's summary
+// checks out against the bytes (codex 6).
package main
import (
@@ -20,10 +22,16 @@ import (
"net/http"
"net/url"
"os"
+ "path/filepath"
+ "strconv"
"strings"
"time"
"unicode/utf8"
+ "github.com/benbjohnson/litestream"
+ lsfile "github.com/benbjohnson/litestream/file"
+ "github.com/superfly/ltx"
+
"github.com/carlosframework/platform/internal/dbdoor"
"github.com/carlosframework/platform/internal/dbreads"
)
@@ -45,6 +53,14 @@ const (
dbfNotUTF8 = "this query cannot be sent to region %s, because it contains invalid UTF-8 or the replacement character U+FFFD. fix the query and run it again." // DBF-32
)
+// The forwarded pull's CLI sentences (copy record DBF-17, DBF-18, DBF-24),
+// approved by Luna on 2026-10-23.
+const (
+ dbfPullTooLarge = "the database for %s has %s of files to fetch. carlos db pull can fetch at most %s from region %s." // DBF-17
+ dbfFetching = "fetching a copy of %s's database from region %s." // DBF-18
+ dbfPullNoRoom = "the server in region %s does not have room to prepare a copy of %s's database right now. try again later." // DBF-24
+)
+
// errDBForward is the db door's forward marker, handed back by dbRestore so
// a verb can switch to the forwarded read.
type errDBForward struct {
@@ -98,7 +114,12 @@ func dbQueryForward(ctx context.Context, t dbTarget, sqlText string, fwd dbdoor.
raw, _ := io.ReadAll(io.LimitReader(res.Body, 64<<10))
return apiErrorFromBody(res.StatusCode, raw)
}
- ans, err := readDBForward(ctx, res.Body, key, t.host, fwd.Region, os.Stderr)
+ sink := &memSink{}
+ id, result, err := readDBForward(ctx, res.Body, key, t.host, fwd.Region, dbfReading, os.Stderr, sink, newDBFChunkBound(dbreads.MaxOutputBytes, dbreads.ChunkBytes))
+ if err != nil {
+ return err
+ }
+ ans, err := openDBForward(id, key, sink, *result, t.host, fwd.Region)
if err != nil {
return err
}
@@ -141,55 +162,98 @@ func (b *dbfChunkBound) admit(sealed []byte) bool {
return true
}
+// dbChunkSink holds a read's sealed chunks until its result arrives: in
+// memory for a query (at most 64 MiB), in the pull's private directory for a
+// pull (up to its cap).
+type dbChunkSink interface {
+ add(sealed []byte) error
+ count() int
+ chunk(i int) ([]byte, error) // 0-based
+}
+
+type memSink struct{ chunks [][]byte }
+
+func (m *memSink) add(b []byte) error { m.chunks = append(m.chunks, b); return nil }
+func (m *memSink) count() int { return len(m.chunks) }
+func (m *memSink) chunk(i int) ([]byte, error) { return m.chunks[i], nil }
+
+// fileSink spools sealed chunks to dir, one file each, 0600 in a 0700
+// directory the caller removes.
+type fileSink struct {
+ dir string
+ n int
+}
+
+func (f *fileSink) add(b []byte) error {
+ if f.n == 0 {
+ if err := os.MkdirAll(f.dir, 0o700); err != nil {
+ return err
+ }
+ }
+ f.n++
+ return dbPullSpoolWrite(filepath.Join(f.dir, strconv.Itoa(f.n)), b, 0o600)
+}
+
+func (f *fileSink) count() int { return f.n }
+
+func (f *fileSink) chunk(i int) ([]byte, error) {
+ return os.ReadFile(filepath.Join(f.dir, strconv.Itoa(i+1)))
+}
+
+// drop removes chunk i once its bytes are in the files, so the spool and
+// the files are never both whole on the member's disk.
+func (f *fileSink) drop(i int) { os.Remove(filepath.Join(f.dir, strconv.Itoa(i+1))) }
+
// readDBForward reads the reads route's lines (spec §5.5 steps 6 to 8). It
-// says DBF-11 once admitted, buffers every chunk in sequence, and opens and
-// checks the answer only when the result arrives.
-func readDBForward(ctx context.Context, body io.Reader, key *ecdh.PrivateKey, host, region string, status io.Writer) (dbForwardAnswer, error) {
+// says the reading sentence once admitted, keeps every chunk in sequence in
+// sink (each one checked against bound before the sink keeps anything, plan
+// review finding 37), and returns the read's id and its result unopened.
+func readDBForward(ctx context.Context, body io.Reader, key *ecdh.PrivateKey, host, region, reading string, status io.Writer, sink dbChunkSink, bound *dbfChunkBound) (string, *dbreads.Result, error) {
unchecked := fmt.Errorf(dbfUnchecked, region)
sc := bufio.NewScanner(body)
sc.Buffer(make([]byte, 64<<10), dbreads.MaxLineBytes)
var id, ended string
- var sealed [][]byte
- bound := newDBFChunkBound(dbreads.MaxOutputBytes, dbreads.ChunkBytes)
for sc.Scan() {
var l dbreads.Line
if err := json.Unmarshal(sc.Bytes(), &l); err != nil {
- return dbForwardAnswer{}, unchecked
+ return "", nil, unchecked
}
switch {
case l.Admitted != "":
if id != "" || !dbreads.ValidID(l.Admitted) {
- return dbForwardAnswer{}, unchecked
+ return "", nil, unchecked
}
id = l.Admitted
- fmt.Fprintln(status, fmt.Sprintf(dbfReading, host, region))
+ fmt.Fprintln(status, fmt.Sprintf(reading, host, region))
case l.Chunk != 0:
- if id == "" || l.Chunk != int64(len(sealed))+1 || !bound.admit(l.Data) {
- return dbForwardAnswer{}, unchecked
+ if id == "" || l.Chunk != int64(sink.count())+1 || !bound.admit(l.Data) {
+ return "", nil, unchecked
+ }
+ if err := sink.add(l.Data); err != nil {
+ return "", nil, err
}
- sealed = append(sealed, l.Data)
case l.Result != nil:
if id == "" {
- return dbForwardAnswer{}, unchecked
+ return "", nil, unchecked
}
- return openDBForward(id, key, sealed, *l.Result, host, region)
+ return id, l.Result, nil
case l.Ended != "":
ended = l.Ended
}
}
if errors.Is(ctx.Err(), context.Canceled) {
- return dbForwardAnswer{}, errors.New("stopped")
+ return "", nil, errors.New("stopped")
}
if ended == dbreads.EndedNotPicked {
- return dbForwardAnswer{}, fmt.Errorf(dbfNotPicked, region)
+ return "", nil, fmt.Errorf(dbfNotPicked, region)
}
- return dbForwardAnswer{}, fmt.Errorf(dbfUnfinished, host, region)
+ return "", nil, fmt.Errorf(dbfUnfinished, host, region)
}
// openDBForward opens a result: a done one's every chunk and its summary,
// checked against each other before anything is returned; any other state
// mapped to its sentence (spec §5.6, §10).
-func openDBForward(id string, key *ecdh.PrivateKey, sealed [][]byte, res dbreads.Result, host, region string) (dbForwardAnswer, error) {
+func openDBForward(id string, key *ecdh.PrivateKey, sink dbChunkSink, res dbreads.Result, host, region string) (dbForwardAnswer, error) {
unchecked := fmt.Errorf(dbfUnchecked, region)
if res.Valid() != nil {
return dbForwardAnswer{}, unchecked
@@ -200,11 +264,15 @@ func openDBForward(id string, key *ecdh.PrivateKey, sealed [][]byte, res dbreads
}
switch res.State {
case dbreads.StateDone:
- if res.LastSeq != int64(len(sealed)) {
+ if res.LastSeq != int64(sink.count()) {
return dbForwardAnswer{}, unchecked
}
var rows bytes.Buffer
- for i, c := range sealed {
+ for i := 0; i < sink.count(); i++ {
+ c, err := sink.chunk(i)
+ if err != nil {
+ return dbForwardAnswer{}, unchecked
+ }
seq := int64(i + 1)
plain, err := dbreads.OpenChunk(key, id, seq, dbreads.StreamRows, seq == res.LastSeq, c)
if err != nil {
@@ -246,3 +314,224 @@ func openDBForward(id string, key *ecdh.PrivateKey, sealed [][]byte, res dbreads
}
return dbForwardAnswer{}, fmt.Errorf(dbfUnfinished, host, region)
}
+
+// openDBPull opens a pull's result (spec §6, phase 2): a done one's chunks,
+// opened in order, are the planned files concatenated; they are checked
+// against the summary's count and hash and split by its file list into a
+// litestream file replica under dir, which today's restore and check turn
+// into the copy. Any other state maps to its sentence.
+func openDBPull(ctx context.Context, id string, key *ecdh.PrivateKey, sink dbChunkSink, res dbreads.Result, host, region, dir string) (*dbdoor.Restored, error) {
+ unchecked := fmt.Errorf(dbfUnchecked, region)
+ if res.Valid() != nil {
+ return nil, unchecked
+ }
+ s, err := dbreads.OpenSummary(key, id, res.Sealed)
+ if err != nil {
+ return nil, unchecked
+ }
+ if res.State != dbreads.StateDone {
+ switch res.Code {
+ case dbreads.CodeTooLarge:
+ return nil, fmt.Errorf(dbfPullTooLarge, host, humanBytes(&s.Restored), humanBytes(&s.Cap), region)
+ case dbreads.CodeNoRoom:
+ return nil, fmt.Errorf(dbfPullNoRoom, region, host)
+ }
+ _, err := openDBForward(id, key, sink, res, host, region)
+ return nil, err
+ }
+ // A done pull is the whole answer: no output cap applies to one.
+ if res.Code != "" || res.LastSeq != int64(sink.count()) || checkPullManifest(s) != nil {
+ return nil, unchecked
+ }
+ root := filepath.Join(dir, "replica")
+ h := sha256.New()
+ fi, written := 0, int64(0)
+ var head []byte // the current file's first ltx.HeaderSize bytes
+ var out *os.File
+ defer func() {
+ if out != nil {
+ out.Close()
+ }
+ }()
+ drop, _ := sink.(interface{ drop(int) })
+ for i := 0; i < sink.count(); i++ {
+ // Ctrl-C and the read's deadline stop the rebuild too (Astra round
+ // 1 on plan 3, finding 2); RestoreLocal's reads check ctx after it.
+ if err := ctx.Err(); err != nil {
+ return nil, dbPullStopped(err, host, region)
+ }
+ sealed, err := sink.chunk(i)
+ if err != nil {
+ return nil, unchecked
+ }
+ seq := int64(i + 1)
+ plain, err := dbreads.OpenChunk(key, id, seq, dbreads.StreamLTX, seq == res.LastSeq, sealed)
+ if err != nil {
+ return nil, unchecked
+ }
+ h.Write(plain)
+ for len(plain) > 0 {
+ if fi >= len(s.Files) {
+ return nil, unchecked
+ }
+ f := s.Files[fi]
+ if out == nil {
+ p := litestream.LTXFilePath(root, f.Level, ltx.TXID(f.MinTXID), ltx.TXID(f.MaxTXID))
+ if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {
+ return nil, err
+ }
+ if out, err = os.OpenFile(p, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600); err != nil {
+ return nil, err
+ }
+ }
+ n := min(int64(len(plain)), f.Size-written)
+ if _, err := out.Write(plain[:n]); err != nil {
+ return nil, err
+ }
+ if len(head) < ltx.HeaderSize {
+ head = append(head, plain[:min(n, int64(ltx.HeaderSize-len(head)))]...)
+ }
+ plain, written = plain[n:], written+n
+ if written == f.Size {
+ if err := out.Close(); err != nil {
+ out = nil
+ return nil, err
+ }
+ // litestream's restore picks files by name and compacts them
+ // by header: a file whose header is not its manifest entry
+ // would restore as a transaction it does not hold (Astra
+ // round 1 on plan 3, finding 1).
+ var lh ltx.Header
+ if lh.UnmarshalBinary(head) != nil || uint64(lh.MinTXID) != f.MinTXID || uint64(lh.MaxTXID) != f.MaxTXID {
+ out = nil
+ return nil, unchecked
+ }
+ out, fi, written, head = nil, fi+1, 0, nil
+ }
+ }
+ if drop != nil {
+ drop.drop(i)
+ }
+ }
+ if fi != len(s.Files) || hex.EncodeToString(h.Sum(nil)) != s.SHA256 {
+ return nil, unchecked
+ }
+ db := filepath.Join(dir, "replica.db")
+ if err := dbdoor.RestoreLocal(ctx, lsfile.NewReplicaClient(root), db, s.TXID); err != nil {
+ if cerr := ctx.Err(); cerr != nil {
+ return nil, dbPullStopped(cerr, host, region)
+ }
+ return nil, dbRestoreError(host, err)
+ }
+ os.RemoveAll(root)
+ return &dbdoor.Restored{Path: db, Dir: dir, TXID: s.TXID, At: s.At}, nil
+}
+
+// dbPullStopped is a pull that ended on its context: "stopped" for Ctrl-C,
+// DBF-12 for the read's own deadline.
+func dbPullStopped(err error, host, region string) error {
+ if errors.Is(err, context.Canceled) {
+ return errors.New("stopped")
+ }
+ return fmt.Errorf(dbfUnfinished, host, region)
+}
+
+// checkPullManifest is the summary's file list checked whole before any
+// file is created (plan review finding 22): every size a possible LTX file
+// and within the cap, every level one litestream writes, every range a
+// range, no file twice, sizes that sum without overflow to the summary's
+// byte count, and a last file that ends at its transaction.
+func checkPullManifest(s dbreads.Summary) error {
+ bad := errors.New("the file list does not add up")
+ if len(s.Files) == 0 || s.Bytes <= 0 || s.Bytes > dbreads.MaxPullBytes {
+ return bad
+ }
+ type coord struct {
+ level int
+ min, max uint64
+ }
+ seen := make(map[coord]bool, len(s.Files))
+ var sum int64
+ for _, f := range s.Files {
+ c := coord{f.Level, f.MinTXID, f.MaxTXID}
+ switch {
+ case f.Size < ltx.HeaderSize+ltx.TrailerSize, f.Size > dbreads.MaxPullBytes,
+ f.Level < 0, f.Level > litestream.SnapshotLevel,
+ f.MinTXID < 1, f.MaxTXID < f.MinTXID, seen[c],
+ sum > dbreads.MaxPullBytes-f.Size: // sum + f.Size would pass the cap
+ return bad
+ }
+ seen[c] = true
+ sum += f.Size
+ }
+ if sum != s.Bytes || s.Files[len(s.Files)-1].MaxTXID != s.TXID {
+ return bad
+ }
+ return nil
+}
+
+// dbPullSpoolWrite writes one spooled chunk: os.WriteFile, a seam so a test
+// can fail the spool part way (plan review finding 23).
+var dbPullSpoolWrite = os.WriteFile
+
+// dbPullForward is a pull through the forwarded read (spec §6). The spool,
+// the files and the copy live in one private 0700 directory under the system
+// temp dir, removed on every path but success, where the caller removes it
+// after copyOut, as it does today's restore directory.
+func dbPullForward(ctx context.Context, t dbTarget, fwd dbdoor.Forward) (*dbdoor.Restored, error) {
+ ctx, cancel := context.WithTimeout(ctx, dbreads.CLIBound)
+ defer cancel()
+ key, err := dbreads.NewReplyKey()
+ if err != nil {
+ return nil, err
+ }
+ at := ""
+ if !t.at.IsZero() {
+ at = t.at.UTC().Format(time.RFC3339Nano)
+ }
+ body, err := json.Marshal(dbreads.ReadBody{Op: dbreads.OpPull, At: at, ReplyKey: dbreads.EncodeReplyKey(key.PublicKey())})
+ if err != nil {
+ return nil, err
+ }
+ res, err := t.sess.c.doStream(ctx, http.MethodPost, apiPath(t.acct, t.app, "/db/"+url.PathEscape(t.host)+"/reads"),
+ bytes.NewReader(body), int64(len(body)), http.Header{"Content-Type": {"application/json"}})
+ if err != nil {
+ if errors.Is(ctx.Err(), context.Canceled) {
+ return nil, errors.New("stopped")
+ }
+ return nil, fmt.Errorf(dbfUnfinished, t.host, fwd.Region)
+ }
+ defer res.Body.Close()
+ if res.StatusCode != http.StatusOK {
+ raw, _ := io.ReadAll(io.LimitReader(res.Body, 64<<10))
+ return nil, apiErrorFromBody(res.StatusCode, raw)
+ }
+ dir, err := os.MkdirTemp("", "carlos-db-")
+ if err != nil {
+ return nil, err
+ }
+ // MkdirTemp already creates 0700; say it anyway, as the in-region
+ // restore does, because the copy's privacy rests on it.
+ if err := os.Chmod(dir, 0o700); err != nil {
+ os.RemoveAll(dir)
+ return nil, err
+ }
+ done := false
+ defer func() {
+ if !done {
+ os.RemoveAll(dir)
+ }
+ }()
+ sink := &fileSink{dir: filepath.Join(dir, "sealed")}
+ id, result, err := readDBForward(ctx, res.Body, key, t.host, fwd.Region, dbfFetching, os.Stderr, sink, newDBFChunkBound(dbreads.MaxPullBytes, dbreads.PullChunkBytes))
+ if err != nil {
+ return nil, err
+ }
+ r, err := openDBPull(ctx, id, key, sink, *result, t.host, fwd.Region, dir)
+ if err != nil {
+ return nil, err
+ }
+ os.RemoveAll(sink.dir)
+ done = true
+ return r, nil
+}
diff --git a/dbforward_e2e_test.go b/dbforward_e2e_test.go
index e94c1ac38..f509cbddd 100644
--- a/dbforward_e2e_test.go
+++ b/dbforward_e2e_test.go
@@ -33,6 +33,8 @@ import (
"time"
"github.com/benbjohnson/litestream"
+ lsfile "github.com/benbjohnson/litestream/file"
+ "github.com/superfly/ltx"
"github.com/carlosframework/platform/internal/console"
"github.com/carlosframework/platform/internal/dbdoor"
@@ -783,3 +785,185 @@ func TestDBForwardThroughTheBackhaulHop(t *testing.T) {
e := newDBFE2E(t, dbfE2EOptions{forward: true, serve: throughBackhaul(2*time.Second, &tap), worker: slowRestore(40 * time.Second)})
checkProgressAtTheCLI(t, e, tap, 40*time.Second)
}
+
+func sha256File(t *testing.T, path string) string {
+ t.Helper()
+ b, err := os.ReadFile(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ sum := sha256.Sum256(b)
+ return hex.EncodeToString(sum[:])
+}
+
+// The pull's parity (plan 3): a forwarded pull and an in-region pull of the
+// same replica save byte-identical copies, for the newest copy and as of a
+// time, and print the same lines but for the host and the path.
+func TestDBForwardPullParityThroughTheCLI(t *testing.T) {
+ e := newDBFE2E(t, dbfE2EOptions{forward: true})
+ for _, at := range []string{"", e.betweenTheSyncs(t)} {
+ dir := t.TempDir()
+ args := []string{"db", "pull", "--account", "ops", "--app", "shop"}
+ if at != "" {
+ args = append(args, "--at", at)
+ }
+ hereOut, farOut := filepath.Join(dir, "here.db"), filepath.Join(dir, "far.db")
+ here, hereErr, err := e.run(t, append(args, "--instance", dbfE2EHere, "--out", hereOut)...)
+ if err != nil {
+ t.Fatalf("in region: %v\n%s", err, hereErr)
+ }
+ far, farErr, err := e.run(t, append(args, "--instance", dbfE2EFar, "--out", farOut)...)
+ if err != nil {
+ t.Fatalf("forwarded: %v\n%s", err, farErr)
+ }
+ if sha256File(t, hereOut) != sha256File(t, farOut) {
+ t.Fatalf("at %q the copies differ", at)
+ }
+ if here == "" || strings.ReplaceAll(strings.ReplaceAll(far, dbfE2EFar, dbfE2EHere), farOut, hereOut) != here {
+ t.Fatalf("at %q: in region %q, forwarded %q", at, here, far)
+ }
+ if !strings.Contains(farErr, fmt.Sprintf(dbfFetching, dbfE2EFar, "uk")) {
+ t.Fatalf("no DBF-18: %s", farErr)
+ }
+ db, _ := sql.Open("sqlite", "file:"+farOut+"?mode=ro")
+ var later int
+ err = db.QueryRow(`SELECT count(*) FROM orders WHERE note = 'later'`).Scan(&later)
+ db.Close()
+ if err != nil || (at == "") != (later == 1) {
+ t.Fatalf("at %q the copy holds %d of the second sync's rows (%v)", at, later, err)
+ }
+ }
+}
+
+// T11 for a pull: a hash mismatch saves nothing at --out. Everything else
+// is a pull the CLI would accept: far's real planned files, whole and
+// valid, a manifest that passes checkPullManifest, and a summary whose only
+// fault is its SHA-256 (plan review finding 32). Mutation: openDBPull skips
+// the hash comparison (red: the files restore and a copy is saved).
+func TestDBForwardPullSavesNothingForABadAnswer(t *testing.T) {
+ e := newDBFE2E(t, dbfE2EOptions{forward: true, noBox: true})
+ stop := e.scriptBox(t, func(b *e2eScript) {
+ ctx := context.Background()
+ rd := lsfile.NewReplicaClient(filepath.Join(e.root, "instances", dbfE2EFar))
+ plan, err := dbdoor.PlanFrom(ctx, rd, time.Time{})
+ if err != nil {
+ t.Errorf("plan far's replica: %v", err)
+ return
+ }
+ var all bytes.Buffer
+ for _, f := range plan.Planned {
+ rc, err := rd.OpenLTXFile(ctx, f.Level, ltx.TXID(f.MinTXID), ltx.TXID(f.MaxTXID), 0, 0)
+ if err != nil {
+ t.Errorf("open %+v: %v", f, err)
+ return
+ }
+ io.Copy(&all, rc)
+ rc.Close()
+ }
+ b.picked(time.Now().Add(time.Minute))
+ sealed, _ := dbreads.SealChunk(b.pub, b.id, 1, dbreads.StreamLTX, true, all.Bytes())
+ b.store.Put(ctx, dbreads.ChunkKey(b.id, 1), sealed, "")
+ wrong := sha256.Sum256([]byte("not these bytes"))
+ summary, _ := dbreads.SealSummary(b.pub, b.id, dbreads.Summary{TXID: plan.TXID, At: plan.At, Files: plan.Planned,
+ Bytes: int64(all.Len()), SHA256: hex.EncodeToString(wrong[:])})
+ r, _ := json.Marshal(dbreads.Result{State: dbreads.StateDone, LastSeq: 1, Box: "uk-1", Sealed: summary})
+ b.store.Put(ctx, dbreads.ResultKey(b.id), r, "")
+ })
+ defer stop()
+ out := filepath.Join(t.TempDir(), "far.db")
+ _, stderr, err := e.run(t, "db", "pull", "--account", "ops", "--app", "shop", "--instance", dbfE2EFar, "--out", out)
+ if err == nil || !strings.Contains(stderr, fmt.Sprintf(dbfUnchecked, "uk")) {
+ t.Fatalf("%q %v", stderr, err)
+ }
+ if _, err := os.Stat(out); !os.IsNotExist(err) {
+ t.Fatalf("something was saved at --out: %v", err)
+ }
+}
+
+// Review Focus 4 and plan review finding 23, through the binary: SIGINT
+// while the CLI is spooling a pull prints "stopped", saves nothing at --out
+// and removes the pull's directory. The scripted box (a stand-in) writes two
+// chunks and never a result, so the CLI is mid-pull when the signal lands.
+// Mutation: drop dbPullForward's deferred RemoveAll (red: carlos-db-* is
+// left in the CLI's TMPDIR).
+func TestDBForwardPullInterruptedLeavesNothing(t *testing.T) {
+ e := newDBFE2E(t, dbfE2EOptions{forward: true, noBox: true})
+ stop := e.scriptBox(t, func(b *e2eScript) {
+ b.picked(time.Now().Add(time.Minute))
+ for seq := int64(1); seq <= 2; seq++ {
+ sealed, _ := dbreads.SealChunk(b.pub, b.id, seq, dbreads.StreamLTX, false, bytes.Repeat([]byte{'x'}, 4096))
+ b.store.Put(context.Background(), dbreads.ChunkKey(b.id, seq), sealed, "")
+ }
+ })
+ defer stop()
+ spool := t.TempDir()
+ out := filepath.Join(t.TempDir(), "far.db")
+ cmd := exec.Command(e.bin, "db", "pull", "--account", "ops", "--app", "shop", "--instance", dbfE2EFar, "--out", out)
+ // The last TMPDIR wins (os/exec keeps the last of a duplicated key).
+ cmd.Env = append(carlosTestEnv(t, append([]string{"CARLOS_CREDENTIALS=" + e.creds}, e.env...)...), "TMPDIR="+spool)
+ cmd.Dir = t.TempDir()
+ var stderr bytes.Buffer
+ cmd.Stderr = &stderr
+ if err := cmd.Start(); err != nil {
+ t.Fatal(err)
+ }
+ deadline := time.Now().Add(30 * time.Second)
+ for {
+ if got, _ := filepath.Glob(filepath.Join(spool, "carlos-db-*", "sealed", "2")); len(got) == 1 {
+ break
+ }
+ if time.Now().After(deadline) {
+ cmd.Process.Kill()
+ cmd.Wait()
+ t.Fatalf("the CLI never spooled two chunks: %s", stderr.String())
+ }
+ time.Sleep(50 * time.Millisecond)
+ }
+ if err := cmd.Process.Signal(os.Interrupt); err != nil {
+ t.Fatal(err)
+ }
+ if err := cmd.Wait(); err == nil || !strings.Contains(stderr.String(), "stopped") {
+ t.Fatalf("%v: %s", err, stderr.String())
+ }
+ if _, err := os.Stat(out); !os.IsNotExist(err) {
+ t.Fatalf("something was saved at --out: %v", err)
+ }
+ if left, _ := filepath.Glob(filepath.Join(spool, "carlos-db-*")); len(left) != 0 {
+ t.Fatalf("the pull's directory is still there: %v", left)
+ }
+}
+
+// A console from plan 2 forwards queries and not pulls. Its marker is
+// played here by a wrapper that drops "pull" from forward_ops on the way
+// out: the new CLI says DBF-8 and saves nothing. The wrapper buffers each
+// response, so it serves only where no response streams (the list door's
+// 409 is all this test reads).
+func TestDBForwardPullOnAQueryOnlyConsole(t *testing.T) {
+ queryOnly := func(t *testing.T, h http.Handler) (string, []string) {
+ ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ rec := httptest.NewRecorder()
+ h.ServeHTTP(rec, r)
+ body := rec.Body.Bytes()
+ if rec.Code == http.StatusConflict {
+ body = bytes.Replace(body, []byte(`"forward_ops":["query","pull"]`), []byte(`"forward_ops":["query"]`), 1)
+ }
+ for k, vs := range rec.Header() {
+ w.Header()[k] = vs
+ }
+ w.Header().Del("Content-Length")
+ w.WriteHeader(rec.Code)
+ w.Write(body)
+ }))
+ t.Cleanup(ts.Close)
+ return ts.URL, nil
+ }
+ e := newDBFE2E(t, dbfE2EOptions{forward: true, noBox: true, serve: queryOnly})
+ out := filepath.Join(t.TempDir(), "far.db")
+ _, stderr, err := e.run(t, "db", "pull", "--account", "ops", "--app", "shop", "--instance", dbfE2EFar, "--out", out)
+ if err == nil || !strings.Contains(stderr, fmt.Sprintf(dbfNoPull, dbfE2EFar, "uk")) {
+ t.Fatalf("%q %v", stderr, err)
+ }
+ if _, err := os.Stat(out); !os.IsNotExist(err) {
+ t.Fatalf("something was saved at --out: %v", err)
+ }
+}
diff --git a/dbforward_test.go b/dbforward_test.go
index e6710057c..74dd9c2b6 100644
--- a/dbforward_test.go
+++ b/dbforward_test.go
@@ -5,18 +5,26 @@ import (
"context"
"crypto/ecdh"
"crypto/sha256"
+ "database/sql"
"encoding/hex"
"encoding/json"
+ "errors"
"fmt"
+ "io"
"net/http"
"net/http/httptest"
"os"
+ "path/filepath"
"strings"
"sync/atomic"
"testing"
"time"
+ lsfile "github.com/benbjohnson/litestream/file"
+ "github.com/superfly/ltx"
+
"github.com/carlosframework/platform/internal/dbdoor"
+ "github.com/carlosframework/platform/internal/dbdoor/dbdoortest"
"github.com/carlosframework/platform/internal/dbreads"
)
@@ -65,7 +73,12 @@ func summaryOf(rows string) dbreads.Summary {
func readFwd(t *testing.T, s *stream, key *ecdh.PrivateKey) (dbForwardAnswer, string, error) {
t.Helper()
var status bytes.Buffer
- ans, err := readDBForward(context.Background(), &s.buf, key, "shop.example.com", "uk", &status)
+ sink := &memSink{}
+ id, res, err := readDBForward(context.Background(), &s.buf, key, "shop.example.com", "uk", dbfReading, &status, sink, newDBFChunkBound(dbreads.MaxOutputBytes, dbreads.ChunkBytes))
+ if err != nil {
+ return dbForwardAnswer{}, status.String(), err
+ }
+ ans, err := openDBForward(id, key, sink, *res, "shop.example.com", "uk")
return ans, status.String(), err
}
@@ -206,9 +219,9 @@ func TestForwardedQueryCtrlCPrintsNothing(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
var status bytes.Buffer
- ans, err := readDBForward(ctx, &cutReader{r: &s.buf}, key, "shop.example.com", "uk", &status)
- if err == nil || err.Error() != "stopped" || ans.rows != nil {
- t.Fatalf("%q %v", ans.rows, err)
+ _, _, err := readDBForward(ctx, &cutReader{r: &s.buf}, key, "shop.example.com", "uk", dbfReading, &status, &memSink{}, newDBFChunkBound(dbreads.MaxOutputBytes, dbreads.ChunkBytes))
+ if err == nil || err.Error() != "stopped" {
+ t.Fatalf("%v", err)
}
}
@@ -243,7 +256,12 @@ func TestForwardedQueryBoundsChunkCountAndSize(t *testing.T) {
s := newStream(t, key)
write(s)
var status bytes.Buffer
- if _, err := readDBForward(context.Background(), &s.buf, key, "shop.example.com", "uk", &status); err == nil || err.Error() != unchecked {
+ sink := &memSink{}
+ id, res, err := readDBForward(context.Background(), &s.buf, key, "shop.example.com", "uk", dbfReading, &status, sink, newDBFChunkBound(dbreads.MaxOutputBytes, dbreads.ChunkBytes))
+ if err == nil {
+ _, err = openDBForward(id, key, sink, *res, "shop.example.com", "uk")
+ }
+ if err == nil || err.Error() != unchecked {
t.Errorf("%s: %v", name, err)
}
}
@@ -286,3 +304,374 @@ func (c *cutReader) Read(p []byte) (int, error) {
func dbdoorErrNothing() error { return dbdoor.ErrNothingReplicated }
func errVanished() error { return os.ErrNotExist }
+
+// pullStream is a pull's body for the files of dir's replica, in one chunk.
+func pullStream(t *testing.T, key *ecdh.PrivateKey, dir string, mutate func(s *dbreads.Summary, files [][]byte)) *stream {
+ t.Helper()
+ return pullStreamN(t, key, dir, 1, mutate)
+}
+
+// pullStreamN is pullStream with the files' bytes split across n chunks.
+func pullStreamN(t *testing.T, key *ecdh.PrivateKey, dir string, n int, mutate func(s *dbreads.Summary, files [][]byte)) *stream {
+ t.Helper()
+ rd := lsfile.NewReplicaClient(dir)
+ plan, err := dbdoor.PlanFrom(context.Background(), rd, time.Time{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ var files [][]byte
+ var all bytes.Buffer
+ for _, f := range plan.Planned {
+ rc, err := rd.OpenLTXFile(context.Background(), f.Level, ltx.TXID(f.MinTXID), ltx.TXID(f.MaxTXID), 0, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ b, _ := io.ReadAll(rc)
+ rc.Close()
+ files = append(files, b)
+ all.Write(b)
+ }
+ sum := sha256.Sum256(all.Bytes())
+ s := dbreads.Summary{TXID: plan.TXID, At: plan.At, Files: plan.Planned, Bytes: int64(all.Len()), SHA256: hex.EncodeToString(sum[:])}
+ if mutate != nil {
+ mutate(&s, files)
+ }
+ st := newStream(t, key)
+ b := all.Bytes()
+ per := (len(b) + n - 1) / n
+ for seq := 1; seq <= n; seq++ {
+ part := b[min((seq-1)*per, len(b)):min(seq*per, len(b))]
+ sealed, err := dbreads.SealChunk(key.PublicKey(), fwdID, int64(seq), dbreads.StreamLTX, seq == n, part)
+ if err != nil {
+ t.Fatal(err)
+ }
+ st.line(dbreads.Line{Chunk: int64(seq), Data: sealed})
+ }
+ st.result(dbreads.StateDone, "", int64(n), s)
+ return st
+}
+
+func readPull(t *testing.T, st *stream, key *ecdh.PrivateKey) (*dbdoor.Restored, error) {
+ t.Helper()
+ return readPullIn(t, st, key, t.TempDir())
+}
+
+func readPullIn(t *testing.T, st *stream, key *ecdh.PrivateKey, dir string) (*dbdoor.Restored, error) {
+ t.Helper()
+ sink := &fileSink{dir: filepath.Join(dir, "sealed")}
+ id, res, err := readDBForward(context.Background(), &st.buf, key, "shop.example.com", "uk", dbfFetching, io.Discard, sink, newDBFChunkBound(dbreads.MaxPullBytes, dbreads.PullChunkBytes))
+ if err != nil {
+ return nil, err
+ }
+ return openDBPull(context.Background(), id, key, sink, *res, "shop.example.com", "uk", dir)
+}
+
+func TestForwardedPullRestoresTheFiles(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ for _, n := range []int{1, 4} {
+ key, _ := dbreads.NewReplyKey()
+ out := t.TempDir()
+ r, err := readPullIn(t, pullStreamN(t, key, dir, n, nil), key, out)
+ if err != nil {
+ t.Fatalf("%d chunks: %v", n, err)
+ }
+ db, _ := sql.Open("sqlite", "file:"+r.Path+"?mode=ro")
+ var rows int
+ err = db.QueryRow("SELECT count(*) FROM t").Scan(&rows)
+ db.Close()
+ if err != nil || rows != 30 {
+ t.Fatalf("%d chunks: %d rows, %v", n, rows, err)
+ }
+ // Only the copy is left: each spooled chunk went once its bytes were
+ // in the files (so the two are never both whole on the disk), and
+ // the files went once the copy was restored. dbPullForward removes
+ // the empty spool directory.
+ if left, _ := os.ReadDir(filepath.Join(out, "sealed")); len(left) != 0 {
+ t.Errorf("%d chunks: %d spooled chunks are still there", n, len(left))
+ }
+ if _, err := os.Stat(filepath.Join(out, "replica")); !os.IsNotExist(err) {
+ t.Errorf("%d chunks: the files are still there (%v)", n, err)
+ }
+ }
+}
+
+// Review Focus 3 and plan review findings 22 and 24: a file list that does
+// not add up is DBF-13, never a panic, and no file is created before the
+// list is checked whole. Mutations: drop checkPullManifest's per-file size
+// bounds (red: "sizes -1 and N+1" panics or creates files); drop its sum
+// (red: "sizes that do not sum" creates the first file before failing);
+// drop the duplicate check (red: "a file twice" creates a file); drop the
+// level bound (red: "level 12").
+func TestForwardedPullRefusesFilesThatDoNotAddUp(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ for name, mutate := range map[string]func(s *dbreads.Summary, files [][]byte){
+ "sizes that do not sum": func(s *dbreads.Summary, _ [][]byte) { s.Files[0].Size++ },
+ "a list that ends elsewhere": func(s *dbreads.Summary, _ [][]byte) { s.TXID++ },
+ "no files": func(s *dbreads.Summary, _ [][]byte) { s.Files = nil },
+ "sizes -1 and N+1": func(s *dbreads.Summary, _ [][]byte) {
+ s.Files[1].Size += s.Files[0].Size + 1
+ s.Files[0].Size = -1
+ },
+ "a file twice": func(s *dbreads.Summary, _ [][]byte) {
+ s.Files[1] = s.Files[0]
+ s.Bytes = 0
+ for _, f := range s.Files {
+ s.Bytes += f.Size
+ }
+ },
+ "level 12": func(s *dbreads.Summary, _ [][]byte) { s.Files[0].Level = 12 },
+ "a range that runs back": func(s *dbreads.Summary, _ [][]byte) { s.Files[0].MinTXID = s.Files[0].MaxTXID + 1 },
+ "a size past the pull's cap": func(s *dbreads.Summary, _ [][]byte) { s.Files[0].Size = dbreads.MaxPullBytes + 1 },
+ } {
+ key, _ := dbreads.NewReplyKey()
+ out := t.TempDir()
+ func() {
+ defer func() {
+ if p := recover(); p != nil {
+ t.Errorf("%s: panic %v", name, p)
+ }
+ }()
+ if _, err := readPullIn(t, pullStream(t, key, dir, mutate), key, out); err == nil || err.Error() != fmt.Sprintf(dbfUnchecked, "uk") {
+ t.Errorf("%s: %v", name, err)
+ }
+ }()
+ if _, err := os.Stat(filepath.Join(out, "replica")); !os.IsNotExist(err) {
+ t.Errorf("%s: a file was created before the list was checked (%v)", name, err)
+ }
+ }
+}
+
+// The other endings of a pull: too-large is DBF-17 with the planned bytes
+// and the cap, no-room DBF-24, and the rest the query's mapping.
+func TestForwardedPullEndings(t *testing.T) {
+ t.Parallel()
+ for name, c := range map[string]struct {
+ code string
+ sum dbreads.Summary
+ want string
+ }{
+ "too large": {dbreads.CodeTooLarge, dbreads.Summary{Restored: 3 << 30, Cap: 2 << 30},
+ fmt.Sprintf(dbfPullTooLarge, "shop.example.com", humanBytes(ptr(int64(3<<30))), humanBytes(ptr(int64(2<<30))), "uk")},
+ "no room": {dbreads.CodeNoRoom, dbreads.Summary{}, fmt.Sprintf(dbfPullNoRoom, "uk", "shop.example.com")},
+ "nothing replicated": {dbreads.CodeNothingReplicated, dbreads.Summary{}, dbRestoreError("shop.example.com", dbdoor.ErrNothingReplicated).Error()},
+ } {
+ key, _ := dbreads.NewReplyKey()
+ st := newStream(t, key)
+ st.result(dbreads.StateFailed, c.code, 0, c.sum)
+ if _, err := readPull(t, st, key); err == nil || err.Error() != c.want {
+ t.Errorf("%s: %v, want %q", name, err, c.want)
+ }
+ }
+}
+
+func ptr[T any](v T) *T { return &v }
+
+// Plan review finding 37, at the spool: a chunk shorter than a sealed
+// empty one creates no spool file, and a stream of tiny chunks stops at the
+// bound's count, so the spool never holds more files than a read can make.
+// (TestForwardedQueryBoundsChunkCountAndSize pins the count's derivation; a
+// bound of 10 keeps this test small.) Mutation: add to the sink before
+// admit (red: a file for the short chunk, eleven files for the tiny ones).
+func TestForwardedPullSpoolsNoMalformedOrExcessChunk(t *testing.T) {
+ t.Parallel()
+ key, _ := dbreads.NewReplyKey()
+ for name, write := range map[string]func(s *stream) int{
+ "a chunk shorter than its seal": func(s *stream) int {
+ s.line(dbreads.Line{Chunk: 1, Data: []byte("short")})
+ return 0
+ },
+ "tiny chunks past the count": func(s *stream) int {
+ for seq := int64(1); seq <= 11; seq++ {
+ sealed, _ := dbreads.SealChunk(key.PublicKey(), fwdID, seq, dbreads.StreamLTX, false, []byte("x"))
+ s.line(dbreads.Line{Chunk: seq, Data: sealed})
+ }
+ return 10
+ },
+ } {
+ s := newStream(t, key)
+ want := write(s)
+ dir := filepath.Join(t.TempDir(), "sealed")
+ sink := &fileSink{dir: dir}
+ _, _, err := readDBForward(context.Background(), &s.buf, key, "shop.example.com", "uk", dbfFetching, io.Discard, sink,
+ &dbfChunkBound{maxBytes: dbreads.MaxPullBytes, maxChunks: 10})
+ files, _ := os.ReadDir(dir)
+ if err == nil || err.Error() != fmt.Sprintf(dbfUnchecked, "uk") || len(files) != want {
+ t.Errorf("%s: %v, %d spool files, want %d", name, err, len(files), want)
+ }
+ }
+}
+
+// standInConsole answers the list door with the forward marker offering a
+// pull, and the reads route with body (then holds the request open until
+// the client goes when hold is set). It is a stand-in for the console, so
+// that these tests run dbPull itself.
+func standInConsole(t *testing.T, body []byte, hold bool) dbTarget {
+ t.Helper()
+ ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if !strings.HasSuffix(r.URL.Path, "/reads") {
+ w.Header().Set("Content-Type", "application/json")
+ w.WriteHeader(http.StatusConflict)
+ json.NewEncoder(w).Encode(dbdoor.ForwardBody{Error: "shop.example.com keeps its data in region uk.",
+ Forward: 1, Region: "uk", ForwardOps: []string{dbreads.OpQuery, dbreads.OpPull}})
+ return
+ }
+ io.Copy(io.Discard, r.Body)
+ w.Header().Set("Content-Type", "application/x-ndjson")
+ w.WriteHeader(http.StatusOK)
+ w.Write(body)
+ w.(http.Flusher).Flush()
+ if hold {
+ <-r.Context().Done()
+ }
+ }))
+ t.Cleanup(ts.Close)
+ return dbTarget{sess: &apiSession{c: newConsoleClient(ts.URL, "tok"), console: ts.URL}, acct: apiAcct{Seg: "acme"},
+ app: "shop", host: "shop.example.com"}
+}
+
+// leftBehind is what a pull left in its temp dir and at --out.
+func leftBehind(t *testing.T, tmp, out string) {
+ t.Helper()
+ if _, err := os.Stat(out); !os.IsNotExist(err) {
+ t.Errorf("something is at --out: %v", err)
+ }
+ if left, _ := filepath.Glob(filepath.Join(tmp, "carlos-db-*")); len(left) != 0 {
+ t.Errorf("the pull's directory is still there: %v", left)
+ }
+}
+
+// Review Focus 1 and plan review finding 23: a spool that fails part way,
+// through dbPull itself, leaves nothing at --out and removes its directory.
+// Not parallel: it sets TMPDIR and dbPullSpoolWrite. Mutation: drop
+// dbPullForward's deferred RemoveAll (red: carlos-db-* is left).
+func TestForwardedPullLeavesNothingWhenTheSpoolFails(t *testing.T) {
+ tmp := t.TempDir()
+ t.Setenv("TMPDIR", tmp)
+ writes := 0
+ dbPullSpoolWrite = func(name string, b []byte, mode os.FileMode) error {
+ if writes++; writes == 3 {
+ return errors.New("no space left on device")
+ }
+ return os.WriteFile(name, b, mode)
+ }
+ t.Cleanup(func() { dbPullSpoolWrite = os.WriteFile })
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ // The stand-in's chunks are sealed to a key the CLI does not hold; the
+ // pull fails at the spool, before anything is opened.
+ key, _ := dbreads.NewReplyKey()
+ st := pullStreamN(t, key, dir, 4, nil)
+ out := filepath.Join(t.TempDir(), "copy.db")
+ if err := dbPull(context.Background(), standInConsole(t, st.buf.Bytes(), false), out); err == nil {
+ t.Fatal("a spool that failed was accepted")
+ }
+ if writes != 3 {
+ t.Fatalf("%d spool writes: the failure was not part way", writes)
+ }
+ leftBehind(t, tmp, out)
+}
+
+// Review Focus 4 and plan review finding 23: Ctrl-C while chunks are
+// spooling, through dbPull itself, is "stopped", with nothing at --out and
+// the directory removed. Not parallel: it sets TMPDIR and dbPullSpoolWrite.
+// Mutation: drop dbPullForward's deferred RemoveAll (red). The binary's
+// SIGINT is Task 9's TestDBForwardPullInterruptedLeavesNothing.
+func TestForwardedPullCtrlCLeavesNothing(t *testing.T) {
+ tmp := t.TempDir()
+ t.Setenv("TMPDIR", tmp)
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ dbPullSpoolWrite = func(name string, b []byte, mode os.FileMode) error {
+ err := os.WriteFile(name, b, mode)
+ cancel() // Ctrl-C once the first chunk is on disk
+ return err
+ }
+ t.Cleanup(func() { dbPullSpoolWrite = os.WriteFile })
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ key, _ := dbreads.NewReplyKey()
+ st := pullStreamN(t, key, dir, 4, nil)
+ // Admitted, picked and the first chunk only (newStream writes admitted
+ // and picked first), then the stand-in holds the request.
+ lines := bytes.SplitAfter(st.buf.Bytes(), []byte("\n"))
+ out := filepath.Join(t.TempDir(), "copy.db")
+ err := dbPull(ctx, standInConsole(t, bytes.Join(lines[:3], nil), true), out)
+ if err == nil || err.Error() != "stopped" {
+ t.Fatalf("%v", err)
+ }
+ leftBehind(t, tmp, out)
+}
+
+// pullStreamOf is a pull's body for the given bytes and summary, one chunk.
+func pullStreamOf(t *testing.T, key *ecdh.PrivateKey, b []byte, s dbreads.Summary) *stream {
+ t.Helper()
+ st := newStream(t, key)
+ sealed, err := dbreads.SealChunk(key.PublicKey(), fwdID, 1, dbreads.StreamLTX, true, b)
+ if err != nil {
+ t.Fatal(err)
+ }
+ st.line(dbreads.Line{Chunk: 1, Data: sealed})
+ st.result(dbreads.StateDone, "", 1, s)
+ return st
+}
+
+// Astra round 1 on plan 3, finding 1, at the CLI: a valid snapshot of
+// transaction 1, listed as [1,2] with a summary at 2 that adds up in every
+// other way, is DBF-13 and restores nothing: the CLI checks each rebuilt
+// file's LTX header against its manifest entry. Mutation: drop the header
+// comparison (red: it restores and reports transaction 2).
+func TestForwardedPullRefusesARelabelledSnapshot(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 2)
+ rc, err := lsfile.NewReplicaClient(dir).OpenLTXFile(context.Background(), 0, 1, 1, 0, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ b, _ := io.ReadAll(rc)
+ rc.Close()
+ sum := sha256.Sum256(b)
+ s := dbreads.Summary{TXID: 2, Files: []dbdoor.LTXFile{{Level: 0, MinTXID: 1, MaxTXID: 2, Size: int64(len(b))}},
+ Bytes: int64(len(b)), SHA256: hex.EncodeToString(sum[:])}
+ key, _ := dbreads.NewReplyKey()
+ out := t.TempDir()
+ r, err := readPullIn(t, pullStreamOf(t, key, b, s), key, out)
+ if err == nil || err.Error() != fmt.Sprintf(dbfUnchecked, "uk") {
+ t.Fatalf("a relabelled snapshot gave %+v, %v", r, err)
+ }
+ if _, err := os.Stat(filepath.Join(out, "replica.db")); !os.IsNotExist(err) {
+ t.Fatalf("a copy was restored: %v", err)
+ }
+}
+
+// Astra round 1 on plan 3, finding 2, after the download: a pull cancelled
+// once every chunk has arrived is "stopped" and rebuilds no file. Mutation:
+// drop the rebuild loop's context check (red: the files are written and
+// restored before the cancel is seen).
+func TestForwardedPullStopsWhenCancelledAfterTheDownload(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ key, _ := dbreads.NewReplyKey()
+ st := pullStreamN(t, key, dir, 4, nil)
+ out := t.TempDir()
+ sink := &fileSink{dir: filepath.Join(out, "sealed")}
+ id, res, err := readDBForward(context.Background(), &st.buf, key, "shop.example.com", "uk", dbfFetching, io.Discard, sink, newDBFChunkBound(dbreads.MaxPullBytes, dbreads.PullChunkBytes))
+ if err != nil {
+ t.Fatal(err)
+ }
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+ if _, err := openDBPull(ctx, id, key, sink, *res, "shop.example.com", "uk", out); err == nil || err.Error() != "stopped" {
+ t.Fatalf("a cancelled pull answered %v", err)
+ }
+ for _, gone := range []string{"replica", "replica.db"} {
+ if _, err := os.Stat(filepath.Join(out, gone)); !os.IsNotExist(err) {
+ t.Errorf("%s was written after the cancel (%v)", gone, err)
+ }
+ }
+}
diff --git a/docs/site/cli.md b/docs/site/cli.md
index a7b3f8e27..0a8e384f2 100644
--- a/docs/site/cli.md
+++ b/docs/site/cli.md
@@ -574,9 +574,9 @@ own error, `attempt to write a readonly database`. That is on purpose.
Changing a copy would change nothing on the instance, and a command that
seemed to succeed would say otherwise.
-If the app has more than one instance, name one with `--instance`. Only sleeping instances have a database copy. An instance on your own fleet or in a pool cannot be read this way, and the command says which.
+If the app has more than one instance, name one with `--instance`. Only instances that sleep have a database copy. An instance on your own fleet or in a pool cannot be read this way, and the command says which.
-When an instance keeps its data in a different region from the console's, `carlos db query` runs in that region. A server there restores the copy, runs your query and sends back only the rows. The rows are encrypted for your terminal on the way, so nothing in between can read them, and nothing is printed until all of them have arrived and been checked. In that region, the copy can be at most 2 GB once restored, the query runs for at most 5 minutes, and the rows can total at most 64 MB. `carlos db pull` cannot fetch a copy from another region yet.
+When an instance keeps its data in a different region from the console's, `carlos db query` runs in that region. A server there restores the copy, runs your query and sends back only the rows. `carlos db pull` fetches the copy's files from there and restores them on your machine. Both are encrypted for your terminal on the way, so nothing in between can read them, and nothing is printed or saved until all of it has arrived and been checked. In that region, a query's copy can be at most 2 GB once restored, the query runs for at most 5 minutes, and the rows can total at most 64 MB. A pull can fetch at most 2 GB.
### carlos run
diff --git a/docs/superpowers/notes/census-proposal-db-region-forward-3.md b/docs/superpowers/notes/census-proposal-db-region-forward-3.md
new file mode 100644
index 000000000..4dcc5447a
--- /dev/null
+++ b/docs/superpowers/notes/census-proposal-db-region-forward-3.md
@@ -0,0 +1,76 @@
+# Census: db region forward, plan 3 (`carlos db pull` and `pull --at` for an instance in another region)
+
+Status: ACCEPTED in full by the census owner on #280, 2026-10-23, as
+`census-ruling-db-region-forward-3`, for every row below and the round 1 fix
+rows. Every row now carries `rev3=census-ruling-db-region-forward-3`; the
+`@legacy` fingerprints were recomputed and the seal moved to
+`79cb56b4c7d3a8e155d1eff1902715d818fcf4c09b1d8a82921c413b116938a9`.
+
+The census owner's rationale: the box-side reads stay the agent's or the
+sandboxed worker's; the CLI reads only its own spool; the `admitDBRead`
+re-pin is the same write class; Astra matched every digest.
+
+The branch is `db-region-forward-3`, cut from amadan `main` at 0d9e81dc. It
+builds plan 3 of `docs/superpowers/specs/2026-10-22-db-region-forward-design.md`
+(rev 7): the box's worker holds a pull's planned LTX files, verifies their
+checksums and sends them back after the agent's check B; the agent seals them
+as the `ltx` stream in 1 MiB chunks, with the plan in the summary; the console
+forwards and audits the new op; the CLI spools the sealed chunks to a private
+directory, checks the file list, the count and the SHA-256, restores locally
+with `dbdoor.RestoreLocal` and saves the copy with today's `copyOut`. The
+box never restores for a pull.
+
+It adds one root-read overlay,
+`internal/provision/testdata/plan14-root-reads-db-region-forward-3.tsv`,
+loaded last in `TestPlan14RootReadCensus`, after `static-redirects` (the
+overlay main loads last at 0d9e81dc). Its record-write entry is layered on
+`plans3aWriteCensusInventory` in
+`internal/provision/db_region_forward_3_census_test.go`.
+
+## Root reads
+
+Executing identities, named once:
+
+- **the worker**: `carlos system db-worker` under `carlos-dbread@.service`, a
+ DynamicUser with no network, its own PrivateTmp, `/data` and `/etc/carlos`
+ hidden.
+- **the CLI**: `carlos db pull` on a person's machine, as the person.
+
+The scanner names no agent, console or host unit: the agent's pull branch in
+`attempt`, `NewAgent`'s `MaxPull`, the console's op check and audit detail,
+and `CARLOS_DB_PULL_MAX_BYTES` in `dbReadsSettings` make no read it counts
+(plan 1's R8 again).
+
+| unit | change | executing identity | proposed disposition |
+|---|---|---|---|
+| `internal/dbreads/worker.go` `holdAndSend` | new: opens each planned LTX file it received and verified, in its own private `/var/tmp`, after check B, and copies it back over its socket | the worker | legacy, accepted-legacy, J4 (`root-input`, as `receiveFile`) |
+| `internal/dbreads/worker.go` `verifyLTX` | new: opens one received LTX file and decodes it whole to check its checksums, before ready | the worker | legacy, accepted-legacy, J4 |
+| `internal/dbreads/worker.go` `ServeWorker` | changed: accepts a pull header and hands it to `holdAndSend` after the same bounded receive; the `Stat` is unchanged | the worker | re-frozen at `census-ruling-db-region-forward-1` with this change appended |
+| `dbforward.go` `dbPullForward` | new: the same bounded 64 KiB read of a refused reads-route answer as `dbQueryForward` (`io.ReadAll` over the response body, not a file); creates and removes the pull's private 0700 directory | the CLI | legacy, accepted-legacy, J4 (main's footing for `db.go`'s units) |
+| `dbforward.go` `fileSink.chunk` | new: reads back one sealed chunk it spooled itself, 0600 in that directory, admitted by the chunk bound before it was written | the CLI | legacy, accepted-legacy, J4 |
+| `dbforward.go` `openDBPull` | new: creates each planned LTX file exclusively (`O_EXCL`, 0600) in that directory, only after the whole file list is checked; then restores, checks and removes them | the CLI | legacy, accepted-legacy, J4 |
+| `internal/dbdoor/plan_from.go` `ctxReadCloser.Read` | new (Astra round 1, finding 2): `RestoreLocal` wraps its local replica's reads so each checks the restore's context first; litestream's compaction checks none of its own | the CLI and the worker | legacy, accepted-legacy, J4 (as main's `downloaded.Read`) |
+| `dbforward.go` `dbQueryForward` | changed: keeps its chunks in a `memSink` and opens them after `readDBForward` returns the result; the same bounded refusal read | the CLI | re-frozen at `census-ruling-db-region-forward-2` with this change appended |
+
+Round 1 (Astra on cc77ca95): `ServeWorker` now checks every received file's
+LTX header against its plan entry through `verifyLTX`, for a query as well as a
+pull, and the plan's end against its target; `holdAndSend` only sends the checked
+files back; `openDBPull` compares each rebuilt file's header with its manifest
+entry and checks its context per chunk. Their rows carry a round 1 clause; no
+read is added to them.
+
+Envelopes re-pinned: `.`, `internal/console`, `internal/dbdoor`,
+`internal/dbreads`, `internal/host`. Frozen declarations are added, so the seal moves:
+static-redirects' `65857c29…` is checked first, the proposed seal was
+`36ecd2eca043f1d0056f086be47ba1fb887dcdc2eda67427e698c2b6d1b9031b` (79cb56b4… once the markers flipped).
+
+## Record writes (9B8)
+
+`internal/console/apidbforward.go` `Server.admitDBRead` is re-digested and
+stays `nonrecord`: it admits a pull beside a query and audits it with its own
+detail; its writes are the same conditional creates under `db-reads/` in the
+deployment bucket. No write site is added.
+
+## Unchanged
+
+Claim routes (no new route: a pull uses plan 2's reads route), `go.mod`.
diff --git a/docs/superpowers/plans/2026-10-22-db-region-forward-copy.md b/docs/superpowers/plans/2026-10-22-db-region-forward-copy.md
index 8c7a8e5d1..a4a1db96c 100644
--- a/docs/superpowers/plans/2026-10-22-db-region-forward-copy.md
+++ b/docs/superpowers/plans/2026-10-22-db-region-forward-copy.md
@@ -24,8 +24,8 @@ The rules:
`\n\n`. In `docs/site/cli.md` each paragraph of a row is one line.
- No em dashes. A full stop and a new sentence instead, continuing in lower
case. Docs prose keeps the CLI page's sentence case.
-- Ids are not renumbered. DBF-17 to DBF-20 and DBF-23 to DBF-29 are kept
- for plan 3's pull rows.
+- Ids are not renumbered. DBF-17 to DBF-20, DBF-23 and DBF-24 are plan 3's
+ pull rows; DBF-25 to DBF-29 stay free.
Placeholders: `<host>` the instance's hostname; `<region>` a placement code
such as uk; `<size>` and `<cap>` byte sizes as the CLI prints them; `<n>` a
@@ -61,6 +61,20 @@ RFC 3339 time.
| DBF-16 | dbforward.go | <host> is moving between regions. read its database once the move has finished. | approved |
| DBF-32 | dbforward.go | this query cannot be sent to region <region>, because it contains invalid UTF-8 or the replacement character U+FFFD. fix the query and run it again. | approved |
+## Pull (plan 3)
+
+Approved by Luna on 2026-10-23: DBF-18, DBF-19, DBF-20 and DBF-24 as drafted, DBF-17 and DBF-23 rewritten.
+DBF-23 replaces DBF-30 on the CLI page; DBF-30 stays here, approved, and leaves the built list.
+
+| id | file | text | status |
+|---|---|---|---|
+| DBF-17 | dbforward.go | the database for <host> has <size> of files to fetch. carlos db pull can fetch at most <cap> from region <region>. | approved |
+| DBF-18 | dbforward.go | fetching a copy of <host>'s database from region <region>. | approved |
+| DBF-19 | internal/console/apidbforward.go | forwarded a database pull to region <region> (cli) | approved |
+| DBF-20 | internal/console/apidbforward.go | forwarded a database pull to region <region>, as of <time> (cli) | approved |
+| DBF-23 | docs/site/cli.md | If the app has more than one instance, name one with `--instance`. Only instances that sleep have a database copy. An instance on your own fleet or in a pool cannot be read this way, and the command says which.\n\nWhen an instance keeps its data in a different region from the console's, `carlos db query` runs in that region. A server there restores the copy, runs your query and sends back only the rows. `carlos db pull` fetches the copy's files from there and restores them on your machine. Both are encrypted for your terminal on the way, so nothing in between can read them, and nothing is printed or saved until all of it has arrived and been checked. In that region, a query's copy can be at most 2 GB once restored, the query runs for at most 5 minutes, and the rows can total at most 64 MB. A pull can fetch at most 2 GB. | approved |
+| DBF-24 | dbforward.go | the server in region <region> does not have room to prepare a copy of <host>'s database right now. try again later. | approved |
+
## Docs
| id | file | text | status |
diff --git a/docs/superpowers/specs/2026-10-22-db-region-forward-design.md b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md
index 23500f4fe..238a09614 100644
--- a/docs/superpowers/specs/2026-10-22-db-region-forward-design.md
+++ b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md
@@ -1,6 +1,8 @@
# `carlos db` for instances whose data lives in another region
-**Status:** rev 6, 2026-10-22. Rev 6 changes no mechanism: it states four
+**Status:** rev 7, 2026-10-23. Rev 7 (plan 3, 2026-10-23): the pull's cap and chunk size, from a measurement on dev
+(§5.7); the request bound is unchanged.
+Rev 6, 2026-10-22, changes no mechanism: it states four
things the plan review (`CODEX-FINDINGS-DB-REGION-FORWARD-PLANS.md`) found
the plans had to decide, each marked **(plan review N)**: what a result
names when a chunk cannot be confirmed (§5.4, 9), a box's `cancel` read
@@ -482,6 +484,8 @@ the output. The CLI maps codes to today's sentences where one exists
| Disk | a reservation per read (below) | agent |
| Output | 64 MiB (`runs.MaxOutputBytes`, written "64 MB" in copy as the run's docs do); the read ends `output-cap` | agent; the CLI's buffer is bounded by the same number |
| Chunk | 256 KiB or 1 s, whichever comes first (`runs.ChunkBytes`, `runs.ChunkEvery`) | agent |
+| Pull | at most 2 GiB of planned LTX files (measured on dev, 2026-10-23: relay rate 3.94 MB/s, the slowest of three runs; the rule gives 3.25 GiB, above the 2 GiB ceiling); `CARLOS_DB_PULL_MAX_BYTES` on a box can only lower it; the read ends `too-large` | agent, from the plan, before any byte moves; the CLI's spool is bounded by the same number |
+| Pull chunk | 1 MiB or 1 s, whichever comes first | agent |
| Reads per box | 2, plus one connection kept for the readiness probe (`MaxConnections=3`) | agent, systemd |
| Open reads per person | 2 | console, held by the open request (J5) |
| Open reads per console | 16 | console, held by the open request, 429 with `Retry-After` |
@@ -1179,6 +1183,8 @@ All five are now Paul's.
2. **Limits stand.** 2 GiB restored, 5 minutes, 64 MB of rows, two reads per
box. Real database sizes on the flagship are still unmeasured; plan 2's
dev check reports them.
+ The pull's cap, 2 GiB of planned LTX files (§5.7, rev 7, measured on dev
+ 2026-10-23), is put to Paul as a limit: **pending his confirmation.**
3. **Audit.** The row records the SQL's length and a SHA-256 prefix, not its
text (§9.3). The SQL itself rests in the request object until expiry.
4. **Copy casing.** New strings use Paul's voice rule: lower case after a
diff --git a/internal/console/apidbforward.go b/internal/console/apidbforward.go
index 6a2181415..3b6a66777 100644
--- a/internal/console/apidbforward.go
+++ b/internal/console/apidbforward.go
@@ -144,7 +144,7 @@ func (s *Server) dbResidency(w http.ResponseWriter, r *http.Request, rec provisi
}
writeJSON(w, http.StatusConflict, dbdoor.ForwardBody{
Error: fmt.Sprintf(dbfUpdateCarlos, rec.Host, rec.Region), Forward: 1,
- Region: rec.Region, ForwardOps: []string{dbreads.OpQuery},
+ Region: rec.Region, ForwardOps: []string{dbreads.OpQuery, dbreads.OpPull},
})
}
@@ -157,11 +157,25 @@ const (
dbfAuditAt = "forwarded a database query to region %s, as of %s. the query was %d bytes, sha256 %s (cli)"
)
+// The pull's audit details (copy record DBF-19, DBF-20), approved by Luna
+// on 2026-10-23.
+const (
+ dbfAuditPull = "forwarded a database pull to region %s (cli)"
+ dbfAuditPullAt = "forwarded a database pull to region %s, as of %s (cli)"
+)
+
const dbReadRetryAfter = "30"
-// dbForwardAuditDetail is the db-read row's detail (spec §9.3): the SQL's
-// length and the first 16 hex digits of its SHA-256, never its text.
-func dbForwardAuditDetail(region, sqlText string, at time.Time) string {
+// dbForwardAuditDetail is the db-read row's detail (spec §9.3): for a query
+// the SQL's length and the first 16 hex digits of its SHA-256, never its
+// text; a pull carries no SQL.
+func dbForwardAuditDetail(op, region, sqlText string, at time.Time) string {
+ if op == dbreads.OpPull {
+ if at.IsZero() {
+ return fmt.Sprintf(dbfAuditPull, region)
+ }
+ return fmt.Sprintf(dbfAuditPullAt, region, at.UTC().Format(time.RFC3339))
+ }
sum := sha256.Sum256([]byte(sqlText))
sha := hex.EncodeToString(sum[:])[:16]
if at.IsZero() {
@@ -326,7 +340,15 @@ func (s *Server) admitDBRead(w http.ResponseWriter, r *http.Request) (dbReadAdmi
jsonError(w, http.StatusConflict, fmt.Sprintf(dbfMoving, host))
return none, "", false
}
- if body.Op != dbreads.OpQuery {
+ switch body.Op {
+ case dbreads.OpQuery:
+ case dbreads.OpPull:
+ // A pull runs nothing: SQL in one is a request no CLI builds.
+ if body.SQL != "" {
+ http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
+ return none, "", false
+ }
+ default:
http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
return none, "", false
}
@@ -340,13 +362,15 @@ func (s *Server) admitDBRead(w http.ResponseWriter, r *http.Request) (dbReadAdmi
http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
return none, "", false
}
- if err := dbquery.CheckSQL(body.SQL); err != nil || body.SQL == "" {
- msg := http.StatusText(http.StatusBadRequest)
- if err != nil {
- msg = err.Error()
+ if body.Op == dbreads.OpQuery {
+ if err := dbquery.CheckSQL(body.SQL); err != nil || body.SQL == "" {
+ msg := http.StatusText(http.StatusBadRequest)
+ if err != nil {
+ msg = err.Error()
+ }
+ jsonError(w, http.StatusBadRequest, msg)
+ return none, "", false
}
- jsonError(w, http.StatusBadRequest, msg)
- return none, "", false
}
var at time.Time
if body.At != "" {
@@ -430,7 +454,7 @@ func (s *Server) admitDBRead(w http.ResponseWriter, r *http.Request) (dbReadAdmi
return none, "", false
}
if err := s.store.RecordActivityContext(ctx, acct.ID, app, actor, Activity{Action: dbAuditAction,
- TargetType: "instance", Target: host, Detail: dbForwardAuditDetail(rec.Region, body.SQL, at)}, time.Now()); err != nil {
+ TargetType: "instance", Target: host, Detail: dbForwardAuditDetail(body.Op, rec.Region, body.SQL, at)}, time.Now()); err != nil {
return fail("audit", err)
}
if ended("request", false) {
diff --git a/internal/console/apidbforward_test.go b/internal/console/apidbforward_test.go
index 07c395eb5..cf69f06f2 100644
--- a/internal/console/apidbforward_test.go
+++ b/internal/console/apidbforward_test.go
@@ -206,7 +206,7 @@ func TestListDoorMarksACapableRegion(t *testing.T) {
var got dbdoor.ForwardBody
json.Unmarshal([]byte(body), &got)
if status != http.StatusConflict || got.Forward != 1 || got.Region != "uk" ||
- !slices.Equal(got.ForwardOps, []string{dbreads.OpQuery}) ||
+ !slices.Equal(got.ForwardOps, []string{dbreads.OpQuery, dbreads.OpPull}) ||
got.Error != fmt.Sprintf(dbfUpdateCarlos, dbfHost, "uk") {
t.Fatalf("%d %s", status, body)
}
@@ -297,8 +297,10 @@ func TestDBReadRouteRefusesBeforeAdmitting(t *testing.T) {
Move: &provision.MoveState{ID: "mv_x", To: "ie", State: "requested"}})
two := good
two.SQL = "SELECT 1; SELECT 2"
- pull := good
- pull.Op, pull.SQL = dbreads.OpPull, ""
+ shell := good
+ shell.Op = "shell"
+ pullSQL := good
+ pullSQL.Op = dbreads.OpPull
badKey := good
badKey.ReplyKey = "AAAA"
future := good
@@ -313,7 +315,8 @@ func TestDBReadRouteRefusesBeforeAdmitting(t *testing.T) {
"no such instance": {"nope.example.com", good, http.StatusNotFound, ""},
"moving": {"moving.example.com", good, http.StatusConflict, fmt.Sprintf(dbfMoving, "moving.example.com")},
"two statements": {dbfHost, two, http.StatusBadRequest, dbquery.ErrOneStatement.Error()},
- "an op this console does not forward": {dbfHost, pull, http.StatusBadRequest, ""},
+ "an op this console does not forward": {dbfHost, shell, http.StatusBadRequest, ""},
+ "a pull carrying SQL": {dbfHost, pullSQL, http.StatusBadRequest, ""},
"a reply key that is not a key": {dbfHost, badKey, http.StatusBadRequest, ""},
"a time in the future": {dbfHost, future, http.StatusBadRequest, ""},
} {
@@ -394,7 +397,7 @@ func TestDBReadQueuesASignedRequestAndAudits(t *testing.T) {
t.Fatalf("the serial was not bumped: %v", err)
}
evs, _ := fx.st.Events(OpsAccountID, "shop", 10)
- want := dbForwardAuditDetail("uk", body.SQL, time.Time{})
+ want := dbForwardAuditDetail(dbreads.OpQuery, "uk", body.SQL, time.Time{})
found := false
for _, e := range evs {
found = found || (e.Action == "db-read" && e.Target == dbfHost && e.Detail == want)
@@ -649,6 +652,57 @@ func openAnswer(t *testing.T, key *ecdh.PrivateKey, lines []dbreads.Line) ([]byt
return rows.Bytes(), s
}
+// A pull through the console: the route admits it, relays the box's ltx
+// stream and its summary's file list, and audits it with DBF-19; with --at,
+// DBF-20.
+func TestDBReadPullStreamsTheFiles(t *testing.T) {
+ fx := newDBFFixture(t, true)
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ fx.startBox(t, dir)
+ key, _ := dbreads.NewReplyKey()
+ res := fx.openRead(t, dbfHost, fx.member, dbreads.ReadBody{Op: dbreads.OpPull, ReplyKey: dbreads.EncodeReplyKey(key.PublicKey())})
+ lines := readLines(t, res.Body)
+ res.Body.Close()
+ last := lines[len(lines)-1].Result
+ if last == nil || last.State != dbreads.StateDone {
+ t.Fatalf("lines %+v", lines)
+ }
+ s, err := dbreads.OpenSummary(key, lines[0].Admitted, last.Sealed)
+ if err != nil || len(s.Files) == 0 || s.TXID != s.Files[len(s.Files)-1].MaxTXID {
+ t.Fatalf("summary %+v %v", s, err)
+ }
+ h := sha256.New()
+ for _, l := range lines {
+ if l.Chunk == 0 {
+ continue
+ }
+ plain, err := dbreads.OpenChunk(key, lines[0].Admitted, l.Chunk, dbreads.StreamLTX, l.Chunk == last.LastSeq, l.Data)
+ if err != nil {
+ t.Fatalf("chunk %d: %v", l.Chunk, err)
+ }
+ h.Write(plain)
+ }
+ if hex.EncodeToString(h.Sum(nil)) != s.SHA256 {
+ t.Fatal("the pulled bytes do not add up to the summary")
+ }
+ at := time.Now().Add(-time.Second).UTC().Truncate(time.Second)
+ res = fx.openRead(t, dbfHost, fx.member, dbreads.ReadBody{Op: dbreads.OpPull, At: at.Format(time.RFC3339),
+ ReplyKey: dbreads.EncodeReplyKey(key.PublicKey())})
+ readLines(t, res.Body)
+ res.Body.Close()
+ evs, _ := fx.st.Events(OpsAccountID, "shop", 10)
+ for _, want := range []string{fmt.Sprintf(dbfAuditPull, "uk"), fmt.Sprintf(dbfAuditPullAt, "uk", at.Format(time.RFC3339))} {
+ found := false
+ for _, e := range evs {
+ found = found || (e.Action == "db-read" && e.Target == dbfHost && e.Detail == want)
+ }
+ if !found {
+ t.Errorf("no %q audit row in %+v", want, evs)
+ }
+ }
+}
+
func TestDBReadStreamsAQueryEndToEnd(t *testing.T) {
fx := newDBFFixture(t, true)
dir := t.TempDir()
diff --git a/internal/console/dbforward_minio_test.go b/internal/console/dbforward_minio_test.go
index ce20a55dd..dc8d20633 100644
--- a/internal/console/dbforward_minio_test.go
+++ b/internal/console/dbforward_minio_test.go
@@ -3,6 +3,8 @@ package console
import (
"bufio"
"context"
+ "crypto/sha256"
+ "encoding/hex"
"encoding/json"
"net/http"
"os"
@@ -204,6 +206,37 @@ func TestDBForwardMinio(t *testing.T) {
}
}
+ // A pull through MinIO (plan 3): the planned files, sealed as the ltx
+ // stream, adding up to the summary, whose file list ends at its copy.
+ pkey, _ := dbreads.NewReplyKey()
+ pres := dfx.openRead(t, shop, member, dbreads.ReadBody{Op: dbreads.OpPull, ReplyKey: dbreads.EncodeReplyKey(pkey.PublicKey())})
+ plines := readLines(t, pres.Body)
+ pres.Body.Close()
+ presult := plines[len(plines)-1].Result
+ if presult == nil || presult.State != dbreads.StateDone {
+ t.Fatalf("pull: %+v", plines[len(plines)-1])
+ }
+ psum, err := dbreads.OpenSummary(pkey, plines[0].Admitted, presult.Sealed)
+ if err != nil || len(psum.Files) == 0 || psum.TXID != psum.Files[len(psum.Files)-1].MaxTXID {
+ t.Fatalf("pull summary %+v %v", psum, err)
+ }
+ ph := sha256.New()
+ var pn int64
+ for _, l := range plines {
+ if l.Chunk == 0 {
+ continue
+ }
+ plain, err := dbreads.OpenChunk(pkey, plines[0].Admitted, l.Chunk, dbreads.StreamLTX, l.Chunk == presult.LastSeq, l.Data)
+ if err != nil {
+ t.Fatalf("pull chunk %d: %v", l.Chunk, err)
+ }
+ ph.Write(plain)
+ pn += int64(len(plain))
+ }
+ if hex.EncodeToString(ph.Sum(nil)) != psum.SHA256 || pn != psum.Bytes {
+ t.Fatal("the pulled bytes do not add up to the summary")
+ }
+
// T16 (plan review finding 19): the real box reaches publication and is
// held there; the console's request ends, by disconnect and then by its
// bound; only then does the box publish. Its late chunks and result land
diff --git a/internal/dbdoor/plan_from.go b/internal/dbdoor/plan_from.go
index 23b378e61..a0764c837 100644
--- a/internal/dbdoor/plan_from.go
+++ b/internal/dbdoor/plan_from.go
@@ -144,7 +144,10 @@ func PlanFrom(ctx context.Context, rd ReplicaReader, at time.Time) (RestorePlan,
// pull), exactly to target, at out, and proves it with lease.CheckDBContext.
// On failure nothing is left at out.
func RestoreLocal(ctx context.Context, c litestream.ReplicaClient, out string, target uint64) error {
- if err := restoreOnce(ctx, c, out, target, time.Time{}); err != nil {
+ // litestream's compaction checks no context, so every read of the local
+ // files checks it here, as the door path's downloaded.Read does (Astra
+ // round 1 on plan 3, finding 2).
+ if err := restoreOnce(ctx, ctxReplica{ReplicaClient: c, ctx: ctx}, out, target, time.Time{}); err != nil {
removeCopy(out)
if cerr := ctx.Err(); cerr != nil {
return cerr
@@ -164,3 +167,33 @@ func RestoreLocal(ctx context.Context, c litestream.ReplicaClient, out string, t
}
return nil
}
+
+// ctxReplica is a local replica client whose file reads stop once ctx is
+// done.
+type ctxReplica struct {
+ litestream.ReplicaClient
+ ctx context.Context
+}
+
+func (c ctxReplica) OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error) {
+ if err := c.ctx.Err(); err != nil {
+ return nil, err
+ }
+ rc, err := c.ReplicaClient.OpenLTXFile(ctx, level, minTXID, maxTXID, offset, size)
+ if err != nil {
+ return nil, err
+ }
+ return ctxReadCloser{ReadCloser: rc, ctx: c.ctx}, nil
+}
+
+type ctxReadCloser struct {
+ io.ReadCloser
+ ctx context.Context
+}
+
+func (r ctxReadCloser) Read(p []byte) (int, error) {
+ if err := r.ctx.Err(); err != nil {
+ return 0, err
+ }
+ return r.ReadCloser.Read(p)
+}
diff --git a/internal/dbdoor/plan_from_test.go b/internal/dbdoor/plan_from_test.go
index e0503147c..95b907511 100644
--- a/internal/dbdoor/plan_from_test.go
+++ b/internal/dbdoor/plan_from_test.go
@@ -6,6 +6,7 @@ import (
"io"
"os"
"path/filepath"
+ "sync"
"testing"
"time"
@@ -152,3 +153,68 @@ func TestRestoreLocalRestoresThePlan(t *testing.T) {
t.Fatalf("restored %d bytes, planned %d", st.Size(), plan.RestoredBytes)
}
}
+
+// countingReads is a litestream.ReplicaClient over a file replica that
+// cancels a context on the first byte read and counts every read made after
+// it, so a test can see whether a restore keeps reading once cancelled.
+type countingReads struct {
+ litestream.ReplicaClient
+ cancel context.CancelFunc
+ mu sync.Mutex
+ reads int
+ after int
+}
+
+func (c *countingReads) OpenLTXFile(ctx context.Context, level int, lo, hi ltx.TXID, off, size int64) (io.ReadCloser, error) {
+ rc, err := c.ReplicaClient.OpenLTXFile(ctx, level, lo, hi, off, size)
+ if err != nil {
+ return nil, err
+ }
+ return &countedReader{ReadCloser: rc, c: c}, nil
+}
+
+type countedReader struct {
+ io.ReadCloser
+ c *countingReads
+}
+
+func (r *countedReader) Read(p []byte) (int, error) {
+ r.c.mu.Lock()
+ r.c.reads++
+ if r.c.reads == 1 {
+ r.c.cancel()
+ } else {
+ r.c.after++
+ }
+ r.c.mu.Unlock()
+ return r.ReadCloser.Read(p)
+}
+
+// Astra round 1 on plan 3, finding 2: litestream's local compaction checks
+// no context, so RestoreLocal's replica reads must: once the restore is
+// cancelled (Ctrl-C, or the read's deadline), no further byte is read and
+// nothing is left at out. Mutation: RestoreLocal reads the client without
+// its context wrapper (red: reads continue after the cancel).
+func TestRestoreLocalStopsReadingOnceCancelled(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 4)
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ c := &countingReads{ReplicaClient: lsfile.NewReplicaClient(dir), cancel: cancel}
+ plan, err := PlanFrom(context.Background(), lsfile.NewReplicaClient(dir), time.Time{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ db := filepath.Join(t.TempDir(), "copy.db")
+ err = RestoreLocal(ctx, c, db, plan.TXID)
+ if !errors.Is(err, context.Canceled) {
+ t.Fatalf("a cancelled restore answered %v", err)
+ }
+ if c.after != 0 {
+ t.Fatalf("%d replica reads after the cancel", c.after)
+ }
+ if _, err := os.Stat(db); !os.IsNotExist(err) {
+ t.Fatalf("something is at out: %v", err)
+ }
+}
diff --git a/internal/dbreads/agent.go b/internal/dbreads/agent.go
index 41e6ddb79..78ac62fee 100644
--- a/internal/dbreads/agent.go
+++ b/internal/dbreads/agent.go
@@ -43,9 +43,13 @@ type AgentConfig struct {
// Free reads the filesystem holding /var/tmp.
Free func() (avail, total uint64, err error)
MaxRestored int64 // DefaultMaxRestored when zero
- Now func() time.Time
- Jitter func() time.Duration // 0 to MaxJitter
- Logf func(format string, args ...any)
+ // MaxPull is a pull's cap on its planned LTX files: MaxPullBytes when
+ // zero or larger, so no box ships more than a CLI accepts, and never
+ // more than the worker holds (PlanBytesFactor times MaxRestored).
+ MaxPull int64
+ Now func() time.Time
+ Jitter func() time.Duration // 0 to MaxJitter
+ Logf func(format string, args ...any)
// Draining is this box's own published draining flag (its edges file's
// draining field); a draining box takes no new read. Never draining
// when nil.
@@ -80,6 +84,14 @@ func NewAgent(cfg AgentConfig) *Agent {
if cfg.MaxRestored <= 0 {
cfg.MaxRestored = DefaultMaxRestored
}
+ if cfg.MaxPull <= 0 || cfg.MaxPull > MaxPullBytes {
+ cfg.MaxPull = MaxPullBytes
+ }
+ // The worker refuses a plan whose files total more than this (its
+ // planWithinBounds), so a pull over it ends too-large here instead.
+ if held := PlanBytesFactor * cfg.MaxRestored; cfg.MaxPull > held {
+ cfg.MaxPull = held
+ }
if cfg.Now == nil {
cfg.Now = time.Now
}
diff --git a/internal/dbreads/chunks.go b/internal/dbreads/chunks.go
index d160a1557..f5df0cb94 100644
--- a/internal/dbreads/chunks.go
+++ b/internal/dbreads/chunks.go
@@ -45,7 +45,7 @@ func putCreateOnly(ctx context.Context, s release.Store, key string, body []byte
}
// chunkWriter seals a read's output into out/<seq> objects: one every
-// ChunkBytes, and whatever is pending every `every`; close seals what is
+// size bytes (ChunkBytes for a query, PullChunkBytes for a pull), and whatever is pending every `every`; close seals what is
// left, possibly nothing, as the last chunk. Past max nothing more is kept
// and Write answers errCapped. Each put is create-only and confirmed before
// the next is sealed, under the lock, so seqs are in order (the run
@@ -57,6 +57,7 @@ type chunkWriter struct {
pub *ecdh.PublicKey
stream string
max int64
+ size int
mu sync.Mutex
pending []byte
@@ -70,8 +71,8 @@ type chunkWriter struct {
done chan struct{}
}
-func newChunkWriter(ctx context.Context, s release.Store, id string, pub *ecdh.PublicKey, stream string, every time.Duration, max int64) *chunkWriter {
- c := &chunkWriter{ctx: ctx, store: s, id: id, pub: pub, stream: stream, max: max, sum: sha256.New(),
+func newChunkWriter(ctx context.Context, s release.Store, id string, pub *ecdh.PublicKey, stream string, every time.Duration, max int64, size int) *chunkWriter {
+ c := &chunkWriter{ctx: ctx, store: s, id: id, pub: pub, stream: stream, max: max, size: size, sum: sha256.New(),
stop: make(chan struct{}), done: make(chan struct{})}
go c.tick(every)
return c
@@ -112,12 +113,12 @@ func (c *chunkWriter) Write(p []byte) (int, error) {
c.sum.Write(p)
c.total += int64(len(p))
c.pending = append(c.pending, p...)
- for len(c.pending) >= ChunkBytes {
- if err := c.flushLocked(c.pending[:ChunkBytes], false); err != nil {
+ for len(c.pending) >= c.size {
+ if err := c.flushLocked(c.pending[:c.size], false); err != nil {
c.err = err
return 0, err
}
- c.pending = append([]byte(nil), c.pending[ChunkBytes:]...)
+ c.pending = append([]byte(nil), c.pending[c.size:]...)
}
if capped {
c.isCapped = true
diff --git a/internal/dbreads/envelope.go b/internal/dbreads/envelope.go
index 3fc536284..f44a583a0 100644
--- a/internal/dbreads/envelope.go
+++ b/internal/dbreads/envelope.go
@@ -45,8 +45,8 @@ func (r Request) AtTime() (time.Time, error) {
return time.Parse(time.RFC3339, r.At)
}
-// ops are the operations this build serves. Plan 3 adds OpPull.
-var ops = map[string]bool{OpQuery: true}
+// ops are the operations this build serves.
+var ops = map[string]bool{OpQuery: true, OpPull: true}
func (r Request) valid() bool {
if r.Purpose != Purpose || r.V != 1 || !ValidID(r.ID) || !ValidRegion(r.Region) ||
@@ -54,8 +54,16 @@ func (r Request) valid() bool {
r.Incarnation == "" || !ops[r.Op] || r.By == "" {
return false
}
- if r.Op == OpQuery && (r.SQL == "" || len(r.SQL) > MaxSQLBytes || !utf8.ValidString(r.SQL)) {
- return false
+ switch r.Op {
+ case OpQuery:
+ if r.SQL == "" || len(r.SQL) > MaxSQLBytes || !utf8.ValidString(r.SQL) {
+ return false
+ }
+ case OpPull:
+ // A pull runs nothing: SQL in one is a request nobody built.
+ if r.SQL != "" {
+ return false
+ }
}
if _, err := r.AtTime(); err != nil {
return false
diff --git a/internal/dbreads/envelope_test.go b/internal/dbreads/envelope_test.go
index 379f1323d..aa26621dc 100644
--- a/internal/dbreads/envelope_test.go
+++ b/internal/dbreads/envelope_test.go
@@ -175,3 +175,22 @@ func TestQueueMatchesNeedsEveryPart(t *testing.T) {
}
}
}
+
+// A pull is an op this build signs and serves (plan 3), and carries no SQL.
+func TestEnvelopeCarriesAPull(t *testing.T) {
+ t.Parallel()
+ pub, priv := rootKeys(t)
+ req := testRequest(t)
+ req.Op, req.SQL = OpPull, ""
+ wire, err := Sign(release.RootSigner(priv), req)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got, err := Verify(pub, wire, req.ID, "uk", testNow); err != nil || got.Op != OpPull {
+ t.Fatalf("%+v %v", got, err)
+ }
+ req.SQL = "SELECT 1"
+ if _, err := Sign(release.RootSigner(priv), req); !errors.Is(err, ErrEnvelope) {
+ t.Fatalf("a pull carrying SQL signed: %v", err)
+ }
+}
diff --git a/internal/dbreads/fixture_test.go b/internal/dbreads/fixture_test.go
index bcf0e7582..c4a481a30 100644
--- a/internal/dbreads/fixture_test.go
+++ b/internal/dbreads/fixture_test.go
@@ -46,6 +46,30 @@ type inProcWorkers struct {
// failStart makes the worker end at once without a word, as a unit
// whose service fails to start after the socket accepted does.
failStart atomic.Bool
+ // sent is the type of every frame the agent wrote, in order.
+ sent []byte
+}
+
+// frameTap records the first byte of each Write the agent makes: one frame
+// type per frame (writeFrame writes a frame in one Write).
+type frameTap struct {
+ net.Conn
+ w *inProcWorkers
+}
+
+func (f frameTap) Write(p []byte) (int, error) {
+ if len(p) > 0 {
+ f.w.mu.Lock()
+ f.w.sent = append(f.w.sent, p[0])
+ f.w.mu.Unlock()
+ }
+ return f.Conn.Write(p)
+}
+
+func (w *inProcWorkers) sentFrames() []byte {
+ w.mu.Lock()
+ defer w.mu.Unlock()
+ return append([]byte(nil), w.sent...)
}
func (w *inProcWorkers) dial(context.Context) (net.Conn, error) {
@@ -64,7 +88,7 @@ func (w *inProcWorkers) dial(context.Context) (net.Conn, error) {
time.Sleep(w.linger)
w.live.Add(-1)
}()
- return agentEnd, nil
+ return frameTap{Conn: agentEnd, w: w}, nil
}
opt := w.opt
if opt.Scratch == "" {
@@ -82,7 +106,7 @@ func (w *inProcWorkers) dial(context.Context) (net.Conn, error) {
time.Sleep(w.linger)
w.live.Add(-1)
}()
- return agentEnd, nil
+ return frameTap{Conn: agentEnd, w: w}, nil
}
func (w *inProcWorkers) units(context.Context) (int, error) { return int(w.live.Load()), nil }
diff --git a/internal/dbreads/limits.go b/internal/dbreads/limits.go
index 09a085f6a..613224574 100644
--- a/internal/dbreads/limits.go
+++ b/internal/dbreads/limits.go
@@ -51,6 +51,13 @@ const (
MaxOutputBytes = runs.MaxOutputBytes
ChunkBytes = runs.ChunkBytes
ChunkEvery = runs.ChunkEvery
+ // MaxPullBytes is a pull's cap on its planned LTX files (spec §5.7, rev
+ // 7: measured on dev, 2026-10-23). CARLOS_DB_PULL_MAX_BYTES on a box can
+ // only lower it, so no box ships more than a CLI accepts.
+ MaxPullBytes int64 = 2 << 30
+ // PullChunkBytes is a pull's chunk: larger than a query's, so a full
+ // pull is about two thousand objects, not eight.
+ PullChunkBytes = 1 << 20
ReadsPerBox = 2
MaxConnections = 3 // two reads and one probe (carlos-dbread.socket)
diff --git a/internal/dbreads/lines.go b/internal/dbreads/lines.go
index 20e1e4112..2b5a81c96 100644
--- a/internal/dbreads/lines.go
+++ b/internal/dbreads/lines.go
@@ -2,8 +2,9 @@ package dbreads
// The reads route's 200 body is newline-delimited JSON, one Line per line
// (spec §5.5 step 6). MaxLineBytes bounds one line: a ChunkBytes chunk,
-// sealed and base64-encoded, is about 350 KiB.
-const MaxLineBytes = 1 << 20
+// sealed and base64-encoded, is about 350 KiB, and a pull's PullChunkBytes
+// chunk about 1.4 MB.
+const MaxLineBytes = 2 << 20
// Wait lines name what the console is waiting on; ended lines say why the
// response ends without a result.
diff --git a/internal/dbreads/lines_test.go b/internal/dbreads/lines_test.go
index f693558d5..bb1265cba 100644
--- a/internal/dbreads/lines_test.go
+++ b/internal/dbreads/lines_test.go
@@ -39,3 +39,13 @@ func TestLinesAreOneOutcomeEach(t *testing.T) {
t.Fatalf("a full chunk line is %d bytes, over MaxLineBytes", len(full))
}
}
+
+// A full pull chunk, sealed and base64-encoded in its line, fits
+// MaxLineBytes (plan 3: a 1 MiB chunk's line is about 1.4 MB).
+func TestAFullPullChunkLineFits(t *testing.T) {
+ t.Parallel()
+ full, _ := json.Marshal(Line{Chunk: 1, Data: bytes.Repeat([]byte{0xff}, PullChunkBytes+SealOverhead)})
+ if len(full)+1 > MaxLineBytes {
+ t.Fatalf("a full pull chunk line is %d bytes", len(full))
+ }
+}
diff --git a/internal/dbreads/pull_test.go b/internal/dbreads/pull_test.go
new file mode 100644
index 000000000..421d05d1a
--- /dev/null
+++ b/internal/dbreads/pull_test.go
@@ -0,0 +1,325 @@
+package dbreads
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "io"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ lsfile "github.com/benbjohnson/litestream/file"
+ "github.com/superfly/ltx"
+
+ "github.com/carlosframework/platform/internal/dbdoor"
+ "github.com/carlosframework/platform/internal/dbdoor/dbdoortest"
+)
+
+// sendPullPlan sends a pull header and the planned files; it returns the
+// files' bytes concatenated, which is what the worker must send back.
+// net.Pipe to the real worker code: not acceptance (Task 5's root-runner
+// test is).
+func sendPullPlan(t *testing.T, w io.Writer, dir string, corrupt bool) (dbdoor.RestorePlan, []byte) {
+ t.Helper()
+ rd := lsfile.NewReplicaClient(dir)
+ plan, err := dbdoor.PlanFrom(context.Background(), rd, time.Time{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := writeJSONFrame(w, frameHeader, planHeader{V: ProtocolVersion, Op: OpPull, Files: plan.Planned, Target: plan.TXID, MaxRestored: DefaultMaxRestored}); err != nil {
+ t.Fatal(err)
+ }
+ var all bytes.Buffer
+ for i, f := range plan.Planned {
+ rc, err := rd.OpenLTXFile(context.Background(), f.Level, ltx.TXID(f.MinTXID), ltx.TXID(f.MaxTXID), 0, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ b, _ := io.ReadAll(rc)
+ rc.Close()
+ if corrupt && i == 0 {
+ b[len(b)/2] ^= 0xff
+ }
+ all.Write(b)
+ if err := writeFrame(w, frameFile, b); err != nil {
+ t.Fatal(err)
+ }
+ }
+ return plan, all.Bytes()
+}
+
+func TestWorkerHoldsThenSendsThePlannedFiles(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ conn, done := workerPipe(t, WorkerOptions{})
+ plan, want := sendPullPlan(t, conn, dir, false)
+ var ready readyMsg
+ readTyped(t, conn, frameReady, &ready)
+ if ready.TXID != plan.TXID {
+ t.Fatalf("ready at %d", ready.TXID)
+ }
+ if err := writeJSONFrame(conn, frameRun, runMsg{}); err != nil {
+ t.Fatal(err)
+ }
+ var got bytes.Buffer
+ for {
+ typ, p, err := readFrame(conn)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if typ == frameOutput {
+ got.Write(p)
+ continue
+ }
+ var end endMsg
+ if typ != frameEnd || json.Unmarshal(p, &end) != nil || end.Code != "" {
+ t.Fatalf("end %q %s", typ, p)
+ }
+ break
+ }
+ if !bytes.Equal(got.Bytes(), want) {
+ t.Fatalf("sent back %d bytes, held %d", got.Len(), len(want))
+ }
+ if err := <-done; err != nil {
+ t.Fatal(err)
+ }
+}
+
+// Review Focus 2: a corrupt file is refused before ready, never shipped.
+// Mutation: holdAndSend skips verifyLTX (red: ready, not check-failed).
+func TestWorkerHoldRefusesACorruptFile(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 2)
+ conn, done := workerPipe(t, WorkerOptions{})
+ sendPullPlan(t, conn, dir, true)
+ var end endMsg
+ readTyped(t, conn, frameEnd, &end)
+ if end.Code != CodeCheckFailed {
+ t.Fatalf("end = %+v", end)
+ }
+ <-done
+}
+
+func TestWorkerHoldStopsWithoutSending(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 1)
+ conn, done := workerPipe(t, WorkerOptions{})
+ sendPullPlan(t, conn, dir, false)
+ readTyped(t, conn, frameReady, nil)
+ if err := writeFrame(conn, frameStop, nil); err != nil {
+ t.Fatal(err)
+ }
+ if err := <-done; err != nil {
+ t.Fatal(err)
+ }
+}
+
+func (fx *fixture) pullRequest(at string) Request {
+ req := fx.request("", at)
+ req.Op = OpPull
+ return req
+}
+
+// answerPull opens a pull's chunks as the CLI does: the ltx stream.
+func (fx *fixture) answerPull(id string) ([]byte, Summary, Result) {
+ fx.t.Helper()
+ raw, _, err := fx.store.Get(context.Background(), ResultKey(id))
+ if err != nil {
+ fx.t.Fatal(err)
+ }
+ var r Result
+ if err := json.Unmarshal(raw, &r); err != nil {
+ fx.t.Fatal(err)
+ }
+ s, err := OpenSummary(fx.reply, id, r.Sealed)
+ if err != nil {
+ fx.t.Fatal(err)
+ }
+ var out bytes.Buffer
+ for seq := int64(1); seq <= r.LastSeq; seq++ {
+ sealed, _, _ := fx.store.Get(context.Background(), ChunkKey(id, seq))
+ plain, err := OpenChunk(fx.reply, id, seq, StreamLTX, seq == r.LastSeq, sealed)
+ if err != nil {
+ fx.t.Fatalf("chunk %d: %v", seq, err)
+ }
+ out.Write(plain)
+ }
+ return out.Bytes(), s, r
+}
+
+// plannedBytes is dir's planned files, concatenated in plan order.
+func plannedBytes(t *testing.T, dir string) (dbdoor.RestorePlan, []byte) {
+ t.Helper()
+ rd := lsfile.NewReplicaClient(dir)
+ plan, err := dbdoor.PlanFrom(context.Background(), rd, time.Time{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ var want bytes.Buffer
+ for _, f := range plan.Planned {
+ rc, err := rd.OpenLTXFile(context.Background(), f.Level, ltx.TXID(f.MinTXID), ltx.TXID(f.MaxTXID), 0, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ io.Copy(&want, rc)
+ rc.Close()
+ }
+ return plan, want.Bytes()
+}
+
+// The agent's pull ships the planned files whole, in plan order, as the
+// ltx stream in chunks of at most PullChunkBytes, with the file list in the
+// summary, and logs plan 1's two timing lines (plan review finding 26).
+// Mutation: drop the timing lines (red).
+func TestPullShipsThePlannedFiles(t *testing.T) {
+ t.Parallel()
+ fx := newFixture(t, 4)
+ var logMu sync.Mutex
+ var logs []string
+ a, _ := fx.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) {
+ c.Logf = func(format string, args ...any) {
+ logMu.Lock()
+ defer logMu.Unlock()
+ logs = append(logs, fmt.Sprintf(format, args...))
+ }
+ })
+ req := fx.pullRequest("")
+ fx.submit(req)
+ a.Pass(context.Background())
+ a.Wait()
+ got, s, r := fx.answerPull(req.ID)
+ plan, want := plannedBytes(t, fx.replica)
+ if r.State != StateDone || !bytes.Equal(got, want) || s.TXID != plan.TXID || len(s.Files) != len(plan.Planned) || s.Bytes != plan.LTXBytes {
+ t.Fatalf("result %+v summary %+v: %d bytes, want %d", r, s, len(got), len(want))
+ }
+ for i, f := range s.Files {
+ if f.Level != plan.Planned[i].Level || f.MinTXID != plan.Planned[i].MinTXID || f.MaxTXID != plan.Planned[i].MaxTXID || f.Size != plan.Planned[i].Size {
+ t.Fatalf("file %d is %+v, planned %+v", i, f, plan.Planned[i])
+ }
+ }
+ for seq := int64(1); seq < r.LastSeq; seq++ {
+ sealed, _, _ := fx.store.Get(context.Background(), ChunkKey(req.ID, seq))
+ if len(sealed)-SealOverhead > PullChunkBytes {
+ t.Fatalf("chunk %d is %d bytes", seq, len(sealed))
+ }
+ }
+ // Task 1's measurement reads these two lines (plan review finding 26).
+ logMu.Lock()
+ all := strings.Join(logs, "\n")
+ logMu.Unlock()
+ if !strings.Contains(all, "and held through") || !strings.Contains(all, "confirmed chunks in") {
+ t.Fatalf("the pull's timing lines are missing:\n%s", all)
+ }
+}
+
+// A pull over the agent's cap ends too-large with the planned bytes and the
+// cap, and a record change after the last byte left the bucket (T9 for a
+// pull) writes no chunk and never sends the worker run. Mutation: the
+// agent's pull sends run before check B (red: the recorded frames hold a
+// run).
+func TestPullTooLargeAndNoChunkBeforeCheckB(t *testing.T) {
+ t.Parallel()
+ fx := newFixture(t, 2)
+ a, _ := fx.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) { c.MaxPull = 100 })
+ req := fx.pullRequest("")
+ fx.submit(req)
+ a.Pass(context.Background())
+ a.Wait()
+ _, s, r := fx.answerPull(req.ID)
+ if r.Code != CodeTooLarge || s.Cap != 100 || s.Restored <= 100 {
+ t.Fatalf("%+v %+v", r, s)
+ }
+ fx2 := newFixture(t, 2)
+ plan, _ := plannedBytes(t, fx2.replica)
+ last := plan.Planned[len(plan.Planned)-1]
+ hook := &stageHook{ReplicaReader: lsfile.NewReplicaClient(fx2.replica), stage: "after the last byte",
+ lastSpan: dbdoor.Span(last.MinTXID, last.MaxTXID), change: func() { r := fx2.record(); r.Region = "ie"; fx2.putRecord(r) }}
+ b, w := fx2.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) {
+ c.OpenReplica = func(context.Context, string) (dbdoor.ReplicaReader, error) { return hook, nil }
+ })
+ req2 := fx2.pullRequest("")
+ fx2.submit(req2)
+ b.Pass(context.Background())
+ b.Wait()
+ if _, _, r := fx2.answerPull(req2.ID); r.Code != CodeRecordChanged || len(fx2.chunkKeys(req2.ID)) != 0 {
+ t.Fatalf("%+v, chunks %v", r, fx2.chunkKeys(req2.ID))
+ }
+ // The worker never heard run: check B failed, so the agent said stop
+ // (plan review finding 24). Without this, an agent that sent run first
+ // and refused the files afterwards would pass the two checks above.
+ sent := w.sentFrames()
+ if !bytes.Contains(sent, []byte{frameStop}) {
+ t.Fatalf("the agent never said stop: %q", sent)
+ }
+ if bytes.Contains(sent, []byte{frameRun}) {
+ t.Fatalf("the agent sent run before check B passed: %q", sent)
+ }
+}
+
+// Plan review finding 24: NewAgent itself clamps the pull's cap, whatever a
+// box's settings hand it. Mutation: NewAgent keeps a MaxPull above
+// MaxPullBytes (red).
+func TestNewAgentClampsMaxPull(t *testing.T) {
+ t.Parallel()
+ fx := newFixture(t, 1)
+ for give, want := range map[int64]int64{0: MaxPullBytes, -1: MaxPullBytes, 1 << 20: 1 << 20, MaxPullBytes + 1: MaxPullBytes} {
+ a, _ := fx.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) { c.MaxPull = give })
+ if a.cfg.MaxPull != want {
+ t.Errorf("MaxPull %d became %d, want %d", give, a.cfg.MaxPull, want)
+ }
+ }
+ // Never more than the worker holds: PlanBytesFactor times MaxRestored.
+ a, _ := fx.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) { c.MaxRestored = 256 << 20 })
+ if a.cfg.MaxPull != PlanBytesFactor*(256<<20) {
+ t.Errorf("MaxPull %d with a 256 MiB restored cap", a.cfg.MaxPull)
+ }
+}
+
+// relabelled is the bytes of dir's first planned file, a valid snapshot at
+// transaction 1, under the coordinates of a file that ends at 2.
+func relabelled(t *testing.T, dir string) (dbdoor.LTXFile, []byte) {
+ t.Helper()
+ rd := lsfile.NewReplicaClient(dir)
+ rc, err := rd.OpenLTXFile(context.Background(), 0, 1, 1, 0, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ b, _ := io.ReadAll(rc)
+ rc.Close()
+ return dbdoor.LTXFile{Level: 0, MinTXID: 1, MaxTXID: 2, Size: int64(len(b))}, b
+}
+
+// Astra round 1 on plan 3, finding 1: a valid snapshot under another file's
+// coordinates is refused before ready, for a pull and for a query alike: the
+// worker checks each file's LTX header against its plan entry, so a copy
+// can never be reported at a transaction it does not hold. Mutation: drop
+// the header comparison (red: the query restores and answers ready at 2;
+// the pull verifies and answers ready).
+func TestWorkerRefusesARelabelledSnapshot(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 2)
+ for _, op := range []string{OpQuery, OpPull} {
+ lf, b := relabelled(t, dir)
+ conn, done := workerPipe(t, WorkerOptions{})
+ if err := writeJSONFrame(conn, frameHeader, planHeader{V: ProtocolVersion, Op: op, Files: []dbdoor.LTXFile{lf}, Target: 2, MaxRestored: DefaultMaxRestored}); err != nil {
+ t.Fatal(err)
+ }
+ if err := writeFrame(conn, frameFile, b); err != nil {
+ t.Fatal(err)
+ }
+ typ, p, err := readFrame(conn)
+ var end endMsg
+ if err != nil || typ != frameEnd || json.Unmarshal(p, &end) != nil || end.Code != CodeCheckFailed {
+ t.Errorf("%s: a relabelled snapshot answered %q %s %v", op, typ, p, err)
+ writeFrame(conn, frameStop, nil)
+ }
+ <-done
+ }
+}
diff --git a/internal/dbreads/read.go b/internal/dbreads/read.go
index 8ba085390..b3aafa8ce 100644
--- a/internal/dbreads/read.go
+++ b/internal/dbreads/read.go
@@ -183,10 +183,21 @@ func (a *Agent) attempt(ctx context.Context, req Request, pub *ecdh.PublicKey, a
a.logf("dbreads: %s: plan: %v", req.ID, err)
return fail(CodeInterrupted, Summary{})
}
- if plan.RestoredBytes > a.cfg.MaxRestored {
+ // A pull's cap is on the files it ships, a query's on the copy it
+ // restores (spec §5.7). Both endings are the replica's word, so both wait
+ // for check B (Astra round 1, finding 2).
+ pull := req.Op == OpPull
+ switch {
+ case pull && plan.LTXBytes > a.cfg.MaxPull:
+ return failB(CodeTooLarge, Summary{Restored: plan.LTXBytes, Cap: a.cfg.MaxPull})
+ case !pull && plan.RestoredBytes > a.cfg.MaxRestored:
return failB(CodeTooLarge, Summary{Restored: plan.RestoredBytes, Cap: a.cfg.MaxRestored})
}
- if err := a.adm.grow(req.ID, plan.LTXBytes+plan.RestoredBytes); err != nil {
+ need := plan.LTXBytes + plan.RestoredBytes
+ if pull {
+ need = plan.LTXBytes // the worker holds the files and restores nothing
+ }
+ if err := a.adm.grow(req.ID, need); err != nil {
return failB(CodeNoRoom, Summary{})
}
conn, err := a.dial(pctx)
@@ -242,20 +253,37 @@ func (a *Agent) attempt(ctx context.Context, req Request, pub *ecdh.PublicKey, a
}
return refuse(code)
}
- stopPlanBound() // the query is bounded by the worker and the owner's deadline
- // Phase timings, for operators and for plan 3's throughput measurement:
- // planning and the worker's restore, then the output's sealing and
- // confirmed creates.
- a.logf("dbreads: %s: planned %d files (%d bytes) and restored to %d in %s", req.ID, plan.Files, plan.LTXBytes, plan.TXID, time.Since(attemptStart).Round(time.Millisecond))
+ stopPlanBound() // the run is bounded by the worker and the owner's deadline
+ stream, size, max := StreamRows, ChunkBytes, a.cfg.MaxOutput
+ if pull {
+ stream, size, max = StreamLTX, PullChunkBytes, plan.LTXBytes
+ }
+ // Phase timings, for operators and for plan 3's throughput measurement
+ // (plan review finding 26): a query's first phase plans and restores, a
+ // pull's plans and has the worker verify and hold the files; then the
+ // output's sealing and confirmed creates.
+ held := "restored to"
+ if pull {
+ held = "held through"
+ }
+ a.logf("dbreads: %s: planned %d files (%d bytes) and %s %d in %s", req.ID, plan.Files, plan.LTXBytes, held, plan.TXID, time.Since(attemptStart).Round(time.Millisecond))
runStart := time.Now()
if err := writeJSONFrame(conn, frameRun, runMsg{SQL: req.SQL}); err != nil {
return fail(CodeInterrupted, Summary{})
}
- cw := newChunkWriter(ctx, a.cfg.Store, req.ID, pub, StreamRows, a.cfg.ChunkEvery, a.cfg.MaxOutput)
+ cw := newChunkWriter(ctx, a.cfg.Store, req.ID, pub, stream, a.cfg.ChunkEvery, max, size)
endCode, sqlErr, rerr := relay(conn, cw)
last, n, sum, werr := cw.close()
a.logf("dbreads: %s: output %d bytes in %d confirmed chunks in %s", req.ID, n, last, time.Since(runStart).Round(time.Millisecond))
s := Summary{TXID: plan.TXID, At: plan.At, Bytes: n, SHA256: sum, Error: sqlErr}
+ if pull {
+ s.Files = plan.Planned
+ // A worker that sends more or less than it held is a check
+ // failure, never an output cap.
+ if werr == nil && rerr == nil && endCode == "" && (cw.capped() || n != plan.LTXBytes) {
+ return a.ended(pub, req.ID, StateFailed, CodeCheckFailed, Summary{}, last), false
+ }
+ }
switch {
case werr != nil:
a.logf("dbreads: %s: write output: %v", req.ID, werr)
diff --git a/internal/dbreads/read_test.go b/internal/dbreads/read_test.go
index b754e860a..ad0bd5d50 100644
--- a/internal/dbreads/read_test.go
+++ b/internal/dbreads/read_test.go
@@ -265,7 +265,7 @@ func TestChunkWriterCapIsExact(t *testing.T) {
for _, extra := range []int{0, 1} {
fx := newFixture(t, 0)
max := int64(3*ChunkBytes + 7)
- cw := newChunkWriter(context.Background(), fx.store, "dbr_aaaaaaaaaaaaaaaaaaaaaaaaaa", fx.reply.PublicKey(), StreamRows, time.Hour, max)
+ cw := newChunkWriter(context.Background(), fx.store, "dbr_aaaaaaaaaaaaaaaaaaaaaaaaaa", fx.reply.PublicKey(), StreamRows, time.Hour, max, ChunkBytes)
big := bytes.Repeat([]byte("x"), 2*ChunkBytes+5)
rest := bytes.Repeat([]byte("y"), int(max)-len(big)+extra)
cw.Write(big)
diff --git a/internal/dbreads/worker.go b/internal/dbreads/worker.go
index f8b7d88cf..20b2cbf00 100644
--- a/internal/dbreads/worker.go
+++ b/internal/dbreads/worker.go
@@ -133,7 +133,7 @@ func ServeWorker(conn io.ReadWriter, opt WorkerOptions) error {
return errProtocol
}
var h planHeader
- if err := json.Unmarshal(f.p, &h); err != nil || h.V != ProtocolVersion || h.Op != OpQuery {
+ if err := json.Unmarshal(f.p, &h); err != nil || h.V != ProtocolVersion || (h.Op != OpQuery && h.Op != OpPull) {
return errProtocol
}
// Every byte this worker writes is bounded before the first is
@@ -157,6 +157,20 @@ func ServeWorker(conn io.ReadWriter, opt WorkerOptions) error {
}
return end(CodeCheckFailed, err)
}
+ // litestream's restore picks files by their names and compacts them
+ // by their headers, so a file whose header is not its plan entry
+ // would be reported at a transaction it does not hold (Astra round 1
+ // on plan 3, finding 1). A pull's files go back unrestored, so each is
+ // decoded whole as well.
+ if err := verifyLTX(path, lf, h.Op == OpPull); err != nil {
+ return end(CodeCheckFailed, err)
+ }
+ }
+ if len(h.Files) == 0 || h.Files[len(h.Files)-1].MaxTXID != h.Target {
+ return end(CodeCheckFailed, errors.New("dbreads: the plan does not end at its target"))
+ }
+ if h.Op == OpPull {
+ return holdAndSend(ctx, conn, next, end, root, h)
}
db := filepath.Join(dir, "copy.db")
if err := opt.Restore(ctx, lsfile.NewReplicaClient(root), db, h.Target); err != nil {
@@ -204,6 +218,77 @@ func ServeWorker(conn io.ReadWriter, opt WorkerOptions) error {
return end(CodeSQLError, qerr)
}
+// holdAndSend is a pull's whole worker job after the files are received and
+// checked (spec §6: the box never restores for a pull): the files are held
+// in this unit's private scratch, and after the agent's check B they go back
+// in the header's order, whole, as output.
+func holdAndSend(ctx context.Context, conn io.Writer, next func() (frame, error), end func(string, error) error, root string, h planHeader) error {
+ // ServeWorker has checked every file whole against its plan entry, and
+ // the plan's end against its target.
+ paths := make([]string, len(h.Files))
+ for i, lf := range h.Files {
+ paths[i] = litestream.LTXFilePath(root, lf.Level, ltx.TXID(lf.MinTXID), ltx.TXID(lf.MaxTXID))
+ }
+ if err := writeJSONFrame(conn, frameReady, readyMsg{TXID: h.Target}); err != nil {
+ return err
+ }
+ f, err := next()
+ if err != nil {
+ return err
+ }
+ switch f.typ {
+ case frameStop:
+ return nil
+ case frameRun:
+ default:
+ return errProtocol
+ }
+ out := &frameWriter{w: conn}
+ for _, p := range paths {
+ if ctx.Err() != nil {
+ return errAgentGone
+ }
+ in, err := os.Open(p)
+ if err != nil {
+ return end(CodeCheckFailed, err)
+ }
+ _, err = io.Copy(out, in)
+ in.Close()
+ if err != nil {
+ return err
+ }
+ }
+ if err := out.flush(); err != nil {
+ return err
+ }
+ return end("", nil)
+}
+
+// verifyLTX checks one received file against its plan entry: its LTX
+// header must name exactly the entry's transactions. With whole, every page
+// and the checksums are decoded too (the decode is the worker's, as every
+// decode of tenant bytes is).
+func verifyLTX(path string, lf dbdoor.LTXFile, whole bool) error {
+ f, err := os.Open(path)
+ if err != nil {
+ return err
+ }
+ defer f.Close()
+ dec := ltx.NewDecoder(f)
+ if whole {
+ err = dec.Verify()
+ } else {
+ err = dec.DecodeHeader()
+ }
+ if err != nil {
+ return err
+ }
+ if h := dec.Header(); uint64(h.MinTXID) != lf.MinTXID || uint64(h.MaxTXID) != lf.MaxTXID {
+ return fmt.Errorf("dbreads: %s holds transactions %s", filepath.Base(path), dbdoor.Span(uint64(h.MinTXID), uint64(h.MaxTXID)))
+ }
+ return nil
+}
+
// receiveFile writes exactly size bytes of file frames to path, refusing a
// file larger than its header said (spec §8.3).
func receiveFile(next func() (frame, error), path string, size int64) error {
diff --git a/internal/host/dbreads.go b/internal/host/dbreads.go
index a836646ff..777dc8a60 100644
--- a/internal/host/dbreads.go
+++ b/internal/host/dbreads.go
@@ -26,6 +26,8 @@ const (
// (carlos-dbread.socket says why).
dbReadSocketPath = "/run/carlos/dbread/worker.sock"
dbReadMaxBytesEnv = "CARLOS_DB_READ_MAX_BYTES"
+ // dbPullMaxBytesEnv can only lower a pull's cap (spec §5.7, rev 7).
+ dbPullMaxBytesEnv = "CARLOS_DB_PULL_MAX_BYTES"
dbReadsSwitchEnv = "CARLOS_DB_READS"
)
@@ -33,6 +35,7 @@ type dbReadsEnv struct {
region string
pub ed25519.PublicKey
maxRestored int64
+ maxPull int64
}
// dbReadsSettings is the environment half of spec §7.1's conditions:
@@ -57,7 +60,7 @@ func dbReadsSettings(getenv func(string) string) (dbReadsEnv, string) {
if err != nil {
return dbReadsEnv{}, err.Error()
}
- env := dbReadsEnv{region: region, pub: pub, maxRestored: dbreads.DefaultMaxRestored}
+ env := dbReadsEnv{region: region, pub: pub, maxRestored: dbreads.DefaultMaxRestored, maxPull: dbreads.MaxPullBytes}
if raw := strings.TrimSpace(getenv(dbReadMaxBytesEnv)); raw != "" {
n, err := strconv.ParseInt(raw, 10, 64)
if err != nil || n <= 0 {
@@ -65,6 +68,15 @@ func dbReadsSettings(getenv func(string) string) (dbReadsEnv, string) {
}
env.maxRestored = n
}
+ if raw := strings.TrimSpace(getenv(dbPullMaxBytesEnv)); raw != "" {
+ n, err := strconv.ParseInt(raw, 10, 64)
+ if err != nil || n <= 0 {
+ return dbReadsEnv{}, dbPullMaxBytesEnv + " is not a whole number of bytes"
+ }
+ // Lowering only: no box ships more than a CLI accepts (Review
+ // Focus 5). NewAgent clamps again.
+ env.maxPull = min(n, dbreads.MaxPullBytes)
+ }
return env, ""
}
@@ -143,7 +155,7 @@ func armDBReads(hib activator.Config) {
return lease.ReadReplicaClient(ctx, hib.Bucket, prefix, hib.Region)
},
DialWorker: dialDBReadWorker, Units: dbReadUnits, Free: varTmpFree,
- MaxRestored: env.maxRestored, Logf: log.Printf, Draining: dbReadsDraining.Load,
+ MaxRestored: env.maxRestored, MaxPull: env.maxPull, Logf: log.Printf, Draining: dbReadsDraining.Load,
})
armedDBReads.Store(a)
go a.Run(ctx, dbReadsNudge)
@@ -151,7 +163,7 @@ func armDBReads(hib activator.Config) {
}
// dbReadsHostEnvKeys are the host.env settings dbReadsSettings reads.
-var dbReadsHostEnvKeys = []string{"CARLOS_HIBERNATE_BUCKET", "CARLOS_REGION", "CARLOS_RELEASE_PUBKEY", dbReadsSwitchEnv, "CARLOS_HOST_CLAIM", dbReadMaxBytesEnv}
+var dbReadsHostEnvKeys = []string{"CARLOS_HIBERNATE_BUCKET", "CARLOS_REGION", "CARLOS_RELEASE_PUBKEY", dbReadsSwitchEnv, "CARLOS_HOST_CLAIM", dbReadMaxBytesEnv, dbPullMaxBytesEnv}
// dbReadsSocketWanted is root's half of the dark gate (Astra round 1,
// finding 1): the adopt pass arms carlos-dbread.socket only when the box's
diff --git a/internal/host/dbreads_rootrunner_linux_test.go b/internal/host/dbreads_rootrunner_linux_test.go
index 16d5b61d6..f6dd18cc5 100644
--- a/internal/host/dbreads_rootrunner_linux_test.go
+++ b/internal/host/dbreads_rootrunner_linux_test.go
@@ -9,8 +9,10 @@ import (
"crypto/ecdh"
"crypto/ed25519"
"crypto/rand"
+ "crypto/sha256"
"database/sql"
"encoding/base64"
+ "encoding/hex"
"encoding/json"
"fmt"
"io"
@@ -25,6 +27,7 @@ import (
"time"
lsfile "github.com/benbjohnson/litestream/file"
+ "github.com/superfly/ltx"
"github.com/carlosframework/platform/internal/dbdoor"
"github.com/carlosframework/platform/internal/dbdoor/dbdoortest"
@@ -342,6 +345,51 @@ func TestRootDBReadServesThroughTheInstalledUnits(t *testing.T) {
privateTmpGone(t, 10*time.Second)
}
+// A forwarded pull through the real socket and the real service (plan 3
+// Task 5): the installed worker verifies and holds the planned files and
+// sends them back after check B; the answer, opened as the ltx stream, is
+// the planned files byte for byte in plan order, and adds up to the sealed
+// summary. The worker's private tmp is gone afterwards.
+func TestRootDBReadPullThroughTheInstalledUnits(t *testing.T) {
+ r := newDBReadRunner(t, "40g")
+ id, reply := r.submit(dbreads.OpPull, "")
+ rep, _ := r.payload("serve", true, id)
+ if rep["capable"] != true {
+ t.Fatalf("the installed worker failed its probe: %v", rep)
+ }
+ got, res := r.answer(id, reply, dbreads.StreamLTX)
+ if res.State != dbreads.StateDone || res.Code != "" {
+ t.Fatalf("result %+v", res)
+ }
+ sum, err := dbreads.OpenSummary(reply, id, res.Sealed)
+ if err != nil {
+ t.Fatal(err)
+ }
+ h := sha256.Sum256(got)
+ if hex.EncodeToString(h[:]) != sum.SHA256 || int64(len(got)) != sum.Bytes {
+ t.Fatalf("%d bytes do not add up to the summary %+v", len(got), sum)
+ }
+ rd := lsfile.NewReplicaClient(filepath.Join(r.dir, "replica", "instances", dbreadRunnerHost))
+ plan, err := dbdoor.PlanFrom(context.Background(), rd, time.Time{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ var want bytes.Buffer
+ for _, f := range plan.Planned {
+ rc, err := rd.OpenLTXFile(context.Background(), f.Level, ltx.TXID(f.MinTXID), ltx.TXID(f.MaxTXID), 0, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ io.Copy(&want, rc)
+ rc.Close()
+ }
+ if !bytes.Equal(got, want.Bytes()) || sum.TXID != plan.TXID || len(sum.Files) != len(plan.Planned) {
+ t.Fatalf("pulled %d bytes at %d in %d files; planned %d bytes at %d in %d files",
+ len(got), sum.TXID, len(sum.Files), want.Len(), plan.TXID, len(plan.Planned))
+ }
+ privateTmpGone(t, 10*time.Second)
+}
+
// T3: from inside the worker, nothing outside its scratch is reachable,
// including any network interface but loopback; the private tmp is gone
// after a normal end, a cancel and kill -9 of the agent. Mutation: remove
diff --git a/internal/host/dbreads_test.go b/internal/host/dbreads_test.go
index 620804c23..542464139 100644
--- a/internal/host/dbreads_test.go
+++ b/internal/host/dbreads_test.go
@@ -252,3 +252,33 @@ func TestEnsureSystemUnitsKeepsTheDBReadSocketDarkWithoutTheGate(t *testing.T) {
}
}
}
+
+// Review Focus 5: the pull's cap can only be lowered on a box. Mutation:
+// dbReadsSettings keeps a CARLOS_DB_PULL_MAX_BYTES above the cap (red).
+func TestPullCapOnlyLowers(t *testing.T) {
+ t.Parallel()
+ pub, _, _ := ed25519.GenerateKey(rand.Reader)
+ base := map[string]string{"CARLOS_HIBERNATE_BUCKET": "d", "CARLOS_REGION": "uk", "CARLOS_RELEASE_PUBKEY": base64.StdEncoding.EncodeToString(pub)}
+ get := func(v string) func(string) string {
+ return func(k string) string {
+ if k == "CARLOS_DB_PULL_MAX_BYTES" {
+ return v
+ }
+ return base[k]
+ }
+ }
+ if env, why := dbReadsSettings(get("")); why != "" || env.maxPull != dbreads.MaxPullBytes {
+ t.Fatalf("default %d %q", env.maxPull, why)
+ }
+ if env, why := dbReadsSettings(get("1048576")); why != "" || env.maxPull != 1<<20 {
+ t.Fatalf("lowered %d %q", env.maxPull, why)
+ }
+ if env, why := dbReadsSettings(get("99999999999999")); why != "" || env.maxPull != dbreads.MaxPullBytes {
+ t.Fatalf("raised past the built-in cap: %d %q", env.maxPull, why)
+ }
+ for _, bad := range []string{"0", "-5", "2GB"} {
+ if _, why := dbReadsSettings(get(bad)); why == "" {
+ t.Fatalf("CARLOS_DB_PULL_MAX_BYTES=%s armed the loop", bad)
+ }
+ }
+}
diff --git a/internal/provision/db_region_forward_3_census_test.go b/internal/provision/db_region_forward_3_census_test.go
new file mode 100644
index 000000000..4cbd070d2
--- /dev/null
+++ b/internal/provision/db_region_forward_3_census_test.go
@@ -0,0 +1,28 @@
+package provision
+
+// Record-write census entries for db region forward plan 3 (carlos db
+// pull), layered on plans3aWriteCensusInventory (main's last at 0d9e81dc).
+// census-ruling-db-region-forward-3, accepted in full by the census owner on
+// #280, 2026-10-23 (docs/superpowers/notes/census-proposal-db-region-forward-3.md).
+// admitDBRead now admits a pull beside a query and audits it with its own
+// detail; its writes are the same conditional creates under db-reads/ in
+// the deployment bucket (the request, the queue entry), never an instance
+// or sidecar record: still nonrecord. No write site is added.
+func dbRegionForward3WriteCensusInventory() []censusEntry {
+ entries := append([]censusEntry(nil), plans3aWriteCensusInventory()...)
+ replacements := []censusEntry{
+ {"internal/console/apidbforward.go", "Server.admitDBRead", "nonrecord", "sel:Put", "3f657a9ad3b11512944ad557573bd59adf257e3fd3fe9cad9fc10a974658328b"},
+ }
+ for _, r := range replacements {
+ found := false
+ for i, e := range entries {
+ if e.File == r.File && e.Unit == r.Unit {
+ entries[i], found = r, true
+ }
+ }
+ if !found {
+ entries = append(entries, r)
+ }
+ }
+ return entries
+}
diff --git a/internal/provision/plan14_root_reads_test.go b/internal/provision/plan14_root_reads_test.go
index a38913fe7..f74e754c9 100644
--- a/internal/provision/plan14_root_reads_test.go
+++ b/internal/provision/plan14_root_reads_test.go
@@ -1423,6 +1423,24 @@ func TestPlan14RootReadCensus(t *testing.T) {
t.Fatal(err)
}
seal = "65857c29501dc7676f52921bea79d08217542ecb1d775c67d20a3460c851d831"
+ // Census overlay db-region-forward-3 (db region forward plan 3: carlos
+ // db pull and pull --at for an instance in another region), loaded last,
+ // after static-redirects. census-ruling-db-region-forward-3, accepted in
+ // full by the census owner on #280, 2026-10-23
+ // (docs/superpowers/notes/census-proposal-db-region-forward-3.md). The
+ // worker's pull units (holdAndSend, verifyLTX) are accepted legacy at J4
+ // inside its sandbox, the CLI's (dbPullForward, fileSink.chunk,
+ // openDBPull) at J4 as the person, RestoreLocal's context-checked reads
+ // (ctxReadCloser.Read, round 1) at J4 as downloaded.Read is;
+ // ServeWorker and dbQueryForward are re-frozen with the pull appended. Frozen declarations are added, so
+ // the seal moves; static-redirects' is checked first.
+ if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 {
+ t.Fatal(errs)
+ }
+ if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-db-region-forward-3.tsv"), &inv, true); err != nil {
+ t.Fatal(err)
+ }
+ seal = "79cb56b4c7d3a8e155d1eff1902715d818fcf4c09b1d8a82921c413b116938a9"
errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...)
errs = append(errs, checkRootReadDeclarations(got, inv)...)
if len(errs) > 0 {
diff --git a/internal/provision/plan9_b8_census_test.go b/internal/provision/plan9_b8_census_test.go
index 389ad2e5c..525edfb2c 100644
--- a/internal/provision/plan9_b8_census_test.go
+++ b/internal/provision/plan9_b8_census_test.go
@@ -366,7 +366,7 @@ func TestPlan9B8RecordWriteCensus(t *testing.T) {
if len(got) < 100 {
t.Fatalf("census found only %d units under %s; the walk is broken", len(got), root)
}
- inventory := plans3aWriteCensusInventory()
+ inventory := dbRegionForward3WriteCensusInventory()
problems := append(censusDiff(inventory, got), censusPolicy(inventory)...)
if len(problems) > 0 {
t.Fatalf("record-write census rejected %d site(s); update the explicit inventory only after reviewing the source:\n %s", len(problems), strings.Join(problems, "\n "))
diff --git a/internal/provision/testdata/plan14-root-reads-db-region-forward-3.tsv b/internal/provision/testdata/plan14-root-reads-db-region-forward-3.tsv
new file mode 100644
index 000000000..806049068
--- /dev/null
+++ b/internal/provision/testdata/plan14-root-reads-db-region-forward-3.tsv
@@ -0,0 +1,23 @@
+# Census overlay db-region-forward-3 (db region forward plan 3: carlos db pull and pull --at for an instance in another region; spec docs/superpowers/specs/2026-10-22-db-region-forward-design.md rev 7, §6, §13), loaded LAST in TestPlan14RootReadCensus, after static-redirects, the overlay main loads last at 0d9e81dc. Every row carries rev3=census-ruling-db-region-forward-3, accepted in full by the census owner on #280, 2026-10-23 (docs/superpowers/notes/census-proposal-db-region-forward-3.md).
+# The scanner names two new worker units (holdAndSend, verifyLTX: the pull's hold mode, inside the db-read worker's sandbox, over the files the agent sent), three new CLI units (dbPullForward, fileSink.chunk, openDBPull: the pull's spool, its file replica, as the person), one new dbdoor unit (ctxReadCloser.Read, round 1: RestoreLocal's context-checked reads, for the CLI and the worker), and two changed legacy units (ServeWorker, dbQueryForward), re-frozen at their accepted dispositions with plan 3's change appended. No agent, console or host unit is named: the agent's pull, the console's forward of it and CARLOS_DB_PULL_MAX_BYTES add no read the scanner sees. Frozen declarations are added, so the seal moves; static-redirects' is checked first. Envelopes . (the CLI), internal/console, internal/dbdoor, internal/dbreads and internal/host are re-pinned.
+@legacy 7b7f70fe5b2c4b215433a00e1742b99d4e95df82d429b7845f9a59f493c0ce76 internal/dbreads/worker.go ServeWorker legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-1;basis=root-input (J4); the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network, PrivateTmp, /data and /etc/carlos hidden): new, stats the copy it restored in its own private /var/tmp for the restored-size cap; reads only the files the agent sent and its own scratch; round 1: it refuses, before writing anything, a plan whose restored-size cap exceeds its own LimitFSIZE or whose files total more than twice that cap;rev3=census-ruling-db-region-forward-3;basis=accepts a pull header beside a query's and hands a pull, after the same bounded receive, to holdAndSend before any restore; the Stat is unchanged, now reached by a query only; no read added Round 1 (Astra on plan 3, finding 1): after each receive it calls verifyLTX, for a query as for a pull, and refuses a plan that does not end at its target; no read added here. file:Stat@3347 37763b2fe7774cb2ad602559c5c55ac7b3021b3abad89d3c3c6cbb8e59faa6c9
+internal/dbreads/worker.go ServeWorker legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-1;basis=root-input (J4); the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network, PrivateTmp, /data and /etc/carlos hidden): new, stats the copy it restored in its own private /var/tmp for the restored-size cap; reads only the files the agent sent and its own scratch; round 1: it refuses, before writing anything, a plan whose restored-size cap exceeds its own LimitFSIZE or whose files total more than twice that cap;rev3=census-ruling-db-region-forward-3;basis=accepts a pull header beside a query's and hands a pull, after the same bounded receive, to holdAndSend before any restore; the Stat is unchanged, now reached by a query only; no read added Round 1 (Astra on plan 3, finding 1): after each receive it calls verifyLTX, for a query as for a pull, and refuses a plan that does not end at its target; no read added here. file:Stat@3347 37763b2fe7774cb2ad602559c5c55ac7b3021b3abad89d3c3c6cbb8e59faa6c9
+@legacy 9df9b8719b2f4669d689736ab4f50b32f7eed042e8756f3d7b96448f7377316c internal/dbreads/worker.go holdAndSend legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4); the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network, PrivateTmp, /data and /etc/carlos hidden): new (plan 3, a pull), opens each planned LTX file it received and verified, in its own private /var/tmp, once check B has passed, and copies it back to the agent over its socket; reads only the files the agent sent and its own scratch; decides nothing for root Round 1: the per-file verification moved to ServeWorker; this unit only opens the checked files to send them back. file:Open@796 a084883e6c399444fd88f81b22b739819eb305f85975a280210ba749f595d84a
+internal/dbreads/worker.go holdAndSend legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4); the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network, PrivateTmp, /data and /etc/carlos hidden): new (plan 3, a pull), opens each planned LTX file it received and verified, in its own private /var/tmp, once check B has passed, and copies it back to the agent over its socket; reads only the files the agent sent and its own scratch; decides nothing for root Round 1: the per-file verification moved to ServeWorker; this unit only opens the checked files to send them back. file:Open@796 a084883e6c399444fd88f81b22b739819eb305f85975a280210ba749f595d84a
+@legacy 2ec89dd852f01aaac42d7ea8142198d065a0aa399cd69423136e4f9b7c048538 internal/dbreads/worker.go verifyLTX legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4); the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network, PrivateTmp, /data and /etc/carlos hidden): new (plan 3, a pull), opens one planned LTX file it received, in its own private /var/tmp, and decodes it whole to check its LTX checksums before ready (the worker's decode of tenant bytes, as its restore is); reads only the files the agent sent and its own scratch Round 1: also compares the decoded header's transaction range with the file's plan entry, for a query (header only) as for a pull (decoded whole), so a file is never restored or shipped as a transaction it does not hold. file:Open@78 76da616e0866788c581e9eb70c648595ffc63010501efb4a6f9ee8cdf4ab1442
+internal/dbreads/worker.go verifyLTX legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4); the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network, PrivateTmp, /data and /etc/carlos hidden): new (plan 3, a pull), opens one planned LTX file it received, in its own private /var/tmp, and decodes it whole to check its LTX checksums before ready (the worker's decode of tenant bytes, as its restore is); reads only the files the agent sent and its own scratch Round 1: also compares the decoded header's transaction range with the file's plan entry, for a query (header only) as for a pull (decoded whole), so a file is never restored or shipped as a transaction it does not hold. file:Open@78 76da616e0866788c581e9eb70c648595ffc63010501efb4a6f9ee8cdf4ab1442
+@legacy 8b7f718c59b7d328dbc383b0cfdb2e54540b695cc66890d7e8cf7fe4f5f1d912 dbforward.go dbQueryForward legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-2;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new, reads at most 64 KiB of a refused reads-route answer from the console (io.ReadAll over the HTTP response body, not a file) to print the console's own sentence; decides nothing for root;rev3=census-ruling-db-region-forward-3;basis=keeps its chunks in a memSink and opens them with openDBForward after readDBForward returns the result, the same bytes and checks as before; the same bounded refusal read; no read added file:ReadAll@1371 757573f0b60a4b2e4eafdef63b4d1157363768dcbc9354e1317f52e886394dc8
+dbforward.go dbQueryForward legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-2;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new, reads at most 64 KiB of a refused reads-route answer from the console (io.ReadAll over the HTTP response body, not a file) to print the console's own sentence; decides nothing for root;rev3=census-ruling-db-region-forward-3;basis=keeps its chunks in a memSink and opens them with openDBForward after readDBForward returns the result, the same bytes and checks as before; the same bounded refusal read; no read added file:ReadAll@1371 757573f0b60a4b2e4eafdef63b4d1157363768dcbc9354e1317f52e886394dc8
+@legacy a076452b9b26a17382535b2f5145f84742c8324457827f20b107dccbef494555 dbforward.go dbPullForward legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new (plan 3), reads at most 64 KiB of a refused reads-route answer from the console (io.ReadAll over the HTTP response body, not a file) to print the console's own sentence, as dbQueryForward does; creates the pull's one private 0700 directory under the system temp dir and removes it on every path but success, where db.go's dbPull removes it after copyOut; decides nothing for root file:ReadAll@939 9f9d794b2d34a5613098f39edc8aa2e1d572a95ef41baa6c621173b3911833d2
+dbforward.go dbPullForward legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new (plan 3), reads at most 64 KiB of a refused reads-route answer from the console (io.ReadAll over the HTTP response body, not a file) to print the console's own sentence, as dbQueryForward does; creates the pull's one private 0700 directory under the system temp dir and removes it on every path but success, where db.go's dbPull removes it after copyOut; decides nothing for root file:ReadAll@939 9f9d794b2d34a5613098f39edc8aa2e1d572a95ef41baa6c621173b3911833d2
+@legacy 2425eaa4ccaa924c1796aa2c4489494da992fffb0c3c0a3a4ceaa883fe9f4067 dbforward.go fileSink.chunk legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new (plan 3), reads back one sealed chunk it spooled itself, 0600 in the pull's private 0700 directory under the system temp dir, each admitted by the chunk bound before it was written; the bytes open only with the CLI's own ephemeral key; decides nothing for root file:ReadFile@58 6a1f96c3ac25abc41412246cb50d6412d61fd91fcd1e7ba9d8ef6be6c97d6e8b
+dbforward.go fileSink.chunk legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new (plan 3), reads back one sealed chunk it spooled itself, 0600 in the pull's private 0700 directory under the system temp dir, each admitted by the chunk bound before it was written; the bytes open only with the CLI's own ephemeral key; decides nothing for root file:ReadFile@58 6a1f96c3ac25abc41412246cb50d6412d61fd91fcd1e7ba9d8ef6be6c97d6e8b
+@legacy 9a11fd8342a7c74d49979995cd603bc2d62329005f580eb0dc5ce3bda9ddfac0 dbforward.go openDBPull legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new (plan 3), creates each planned LTX file exclusively (O_EXCL, 0600) as a litestream file replica in the pull's private 0700 directory, only after the summary's whole file list has been checked, from chunks opened with the CLI's own key; then restores and checks them with dbdoor.RestoreLocal and removes them; the count and SHA-256 are checked before the restore; decides nothing for root Round 1 (Astra on plan 3, findings 1 and 2): compares each rebuilt file's LTX header, from the bytes it wrote, with its manifest entry, and checks the context before each chunk; no read added. file:OpenFile@2009 51fc426dbd5d8730bbd15c92fb023fa9af85c999eee769decfd63bb97635f66a
+dbforward.go openDBPull legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new (plan 3), creates each planned LTX file exclusively (O_EXCL, 0600) as a litestream file replica in the pull's private 0700 directory, only after the summary's whole file list has been checked, from chunks opened with the CLI's own key; then restores and checks them with dbdoor.RestoreLocal and removes them; the count and SHA-256 are checked before the restore; decides nothing for root Round 1 (Astra on plan 3, findings 1 and 2): compares each rebuilt file's LTX header, from the bytes it wrote, with its manifest entry, and checks the context before each chunk; no read added. file:OpenFile@2009 51fc426dbd5d8730bbd15c92fb023fa9af85c999eee769decfd63bb97635f66a
+@legacy f89c10f1e2034212c6b36f0ea099bd5933f0a5e24621c3374479a9829c568bbb internal/dbdoor/plan_from.go ctxReadCloser.Read legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input (J4, as main's downloaded.Read);rev3=census-ruling-db-region-forward-3;identity=the carlos CLI on a person's machine, as the person, and the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network): new (Astra round 1 on plan 3, finding 2), RestoreLocal's reads of the local replica files it was handed, each checking the restore's context first so litestream's compaction stops on Ctrl-C or the deadline; reads only those files; decides nothing for root file:Read@117,file:Read@23 e6bf708f2f7a6643b55b6afc64bd94f65cd60e4235f99a5a91ced9d353a6e67b
+internal/dbdoor/plan_from.go ctxReadCloser.Read legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input (J4, as main's downloaded.Read);rev3=census-ruling-db-region-forward-3;identity=the carlos CLI on a person's machine, as the person, and the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network): new (Astra round 1 on plan 3, finding 2), RestoreLocal's reads of the local replica files it was handed, each checking the restore's context first so litestream's compaction stops on Ctrl-C or the deadline; reads only those files; decides nothing for root file:Read@117,file:Read@23 e6bf708f2f7a6643b55b6afc64bd94f65cd60e4235f99a5a91ced9d353a6e67b
+@coverage . cc17fa0a3414d5c8811d5728f339a9a54b0e9fe0aee0a547f9bc677f1ddd1ab7
+@coverage internal/console a09cd3a5db072ff86040580fb038d723c9a114e72a15a39db688bd2523f6bcb4
+@coverage internal/dbreads 64d1eb057fc028865c5b99b2b9bef2714a95eac75ae711f84018980f077d9403
+@coverage internal/host 218f8d6a50eb852a22424236fa89427b4b5509e08fd956ce778d95bc5a63c9a0
+@coverage internal/dbdoor e705cafd886bc300e3bacdf65bcfcf09e0e824c30967fd8d82c36238a2ad8573