amadan

carlos / platform Public

db-region-forward plans: answer Astra's plan review round 1; spec rev 6

All 24 findings fixed, 11 in part: the audit insert's own cancellation is
declined (shared claim SQL; the row now comes first, before anything a box
can see). P0s checked against origin/main; tests moved so each task builds
on the ones before (plan 2 T15 to Task 7, plan 3 root-runner pull to Task 5).
Mutation and test-strength findings each name the mutation that turns red.

Spec rev 6 states four things the plans had to decide, no new mechanism:
the result names only confirmed chunks (5.4), the audit row first and only
cancel once a request has ended (5.5, 9.3), a box reads cancel before
electing (5.5, 5.8), and a retry keeps its first reservation until that
unit stops (5.7).

The brief is round 2: a disposition table, then new P0-P2 only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude 3bdd1b142c30d7cc315cedf817bc93f80ab13169 parent 9371156
5 files changed, +1948 −499
  • docs/superpowers/plans/2026-10-22-db-region-forward-plan-1-box.md +652 −102
  • docs/superpowers/plans/2026-10-22-db-region-forward-plan-2-query.md +829 −178
  • docs/superpowers/plans/2026-10-22-db-region-forward-plan-3-pull.md +342 −67
  • docs/superpowers/specs/2026-10-22-db-region-forward-design.md +37 −9
  • docs/superpowers/specs/CODEX-REVIEW-BRIEF-DB-REGION-FORWARD-PLANS.md +88 −143

CI passed — run details

CI log 
=== gate ===
.amadan/test-gate.sh
=== launchctx (5 s)
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
=== vet (9 s)
=== gofmt (1 s)
gofmt -l .: clean
=== rest (156 s)
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	5.315s
ok  	github.com/carlosframework/platform/cmd/moveprobe	11.610s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.716s
ok  	github.com/carlosframework/platform/internal/activator	58.711s
ok  	github.com/carlosframework/platform/internal/agentsock	0.017s
ok  	github.com/carlosframework/platform/internal/alerting	0.004s
ok  	github.com/carlosframework/platform/internal/analytics	0.024s
ok  	github.com/carlosframework/platform/internal/appconfig	3.450s
ok  	github.com/carlosframework/platform/internal/apperr	0.080s
ok  	github.com/carlosframework/platform/internal/applog	0.141s
ok  	github.com/carlosframework/platform/internal/awsinv	0.005s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.006s
ok  	github.com/carlosframework/platform/internal/boxid	0.784s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.012s
ok  	github.com/carlosframework/platform/internal/caps	0.023s
ok  	github.com/carlosframework/platform/internal/certwarm	0.002s
ok  	github.com/carlosframework/platform/internal/cgroup	0.803s
ok  	github.com/carlosframework/platform/internal/changelog	0.382s
ok  	github.com/carlosframework/platform/internal/claimactivation	0.986s
ok  	github.com/carlosframework/platform/internal/clamd	0.057s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.067s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.010s
ok  	github.com/carlosframework/platform/internal/dbverify	0.008s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.002s
ok  	github.com/carlosframework/platform/internal/docsite	0.010s
ok  	github.com/carlosframework/platform/internal/domains	3.551s
ok  	github.com/carlosframework/platform/internal/doorkey	0.008s
ok  	github.com/carlosframework/platform/internal/economics	0.012s
ok  	github.com/carlosframework/platform/internal/edge	46.629s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.008s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.133s
ok  	github.com/carlosframework/platform/internal/gate	1.866s
ok  	github.com/carlosframework/platform/internal/genselect	0.016s
ok  	github.com/carlosframework/platform/internal/geo	0.016s
ok  	github.com/carlosframework/platform/internal/geoip	0.031s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.170s
ok  	github.com/carlosframework/platform/internal/host	142.307s
ok  	github.com/carlosframework/platform/internal/hostactors	8.799s
ok  	github.com/carlosframework/platform/internal/hostactors/awsmembership	0.016s
ok  	github.com/carlosframework/platform/internal/hostbox	4.806s
ok  	github.com/carlosframework/platform/internal/hostclaim	1.249s
ok  	github.com/carlosframework/platform/internal/hostenv	0.014s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.016s
ok  	github.com/carlosframework/platform/internal/hostexec	8.539s
ok  	github.com/carlosframework/platform/internal/hostfence	0.090s
ok  	github.com/carlosframework/platform/internal/hostfiles	12.299s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.071s
ok  	github.com/carlosframework/platform/internal/hostguard	1.165s
ok  	github.com/carlosframework/platform/internal/hostops	3.046s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.015s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.013s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.939s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.294s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.014s
ok  	github.com/carlosframework/platform/internal/infraview	0.011s
ok  	github.com/carlosframework/platform/internal/instancectl	5.365s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.913s
ok  	github.com/carlosframework/platform/internal/instancefs	0.011s
ok  	github.com/carlosframework/platform/internal/instancekey	0.040s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.021s
ok  	github.com/carlosframework/platform/internal/layout	2.610s
ok  	github.com/carlosframework/platform/internal/lbhealth	0.065s
ok  	github.com/carlosframework/platform/internal/lease	10.226s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.803s
ok  	github.com/carlosframework/platform/internal/ledger	0.027s
ok  	github.com/carlosframework/platform/internal/mail	0.265s
ok  	github.com/carlosframework/platform/internal/maintenance	0.123s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	6.003s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.136s
ok  	github.com/carlosframework/platform/internal/metrics	0.008s
ok  	github.com/carlosframework/platform/internal/moves	24.245s
ok  	github.com/carlosframework/platform/internal/moves/archive	1.315s
ok  	github.com/carlosframework/platform/internal/objstore	0.429s
ok  	github.com/carlosframework/platform/internal/ops	0.207s
ok  	github.com/carlosframework/platform/internal/park	0.424s
ok  	github.com/carlosframework/platform/internal/plans	0.006s
ok  	github.com/carlosframework/platform/internal/privatefile	0.006s
ok  	github.com/carlosframework/platform/internal/provision	19.180s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.008s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	7.044s
ok  	github.com/carlosframework/platform/internal/relay	7.348s
ok  	github.com/carlosframework/platform/internal/release	28.105s
ok  	github.com/carlosframework/platform/internal/rollout	5.925s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.009s
ok  	github.com/carlosframework/platform/internal/rootaws	0.008s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.063s
ok  	github.com/carlosframework/platform/internal/rootstate	0.026s
ok  	github.com/carlosframework/platform/internal/runs	0.547s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.005s
ok  	github.com/carlosframework/platform/internal/schedule	3.676s
ok  	github.com/carlosframework/platform/internal/seeds	0.015s
ok  	github.com/carlosframework/platform/internal/sim/engines	5.864s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	16.696s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.935s
ok  	github.com/carlosframework/platform/internal/status	0.077s
ok  	github.com/carlosframework/platform/internal/steering	0.007s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.299s
ok  	github.com/carlosframework/platform/internal/svcrole	0.003s
ok  	github.com/carlosframework/platform/internal/telemetry	0.006s
ok  	github.com/carlosframework/platform/internal/termfence	0.005s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.008s
ok  	github.com/carlosframework/platform/internal/unixdial	0.002s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.006s
ok  	github.com/carlosframework/platform/internal/version	0.003s
ok  	github.com/carlosframework/platform/internal/vet	1.854s
=== internal_console@0 (84 s)
ok  	github.com/carlosframework/platform/internal/console	81.481s
=== internal_console@1 (37 s)
ok  	github.com/carlosframework/platform/internal/console	34.299s
=== internal_console@2 (43 s)
ok  	github.com/carlosframework/platform/internal/console	40.480s
=== internal_console@3 (39 s)
ok  	github.com/carlosframework/platform/internal/console	36.781s
=== internal_console@4 (41 s)
ok  	github.com/carlosframework/platform/internal/console	38.277s
=== internal_console@5 (40 s)
ok  	github.com/carlosframework/platform/internal/console	38.074s
=== internal_console@6 (44 s)
ok  	github.com/carlosframework/platform/internal/console	41.517s
=== internal_console@7 (45 s)
ok  	github.com/carlosframework/platform/internal/console	42.118s
=== root@0 (36 s)
ok  	github.com/carlosframework/platform	32.870s
=== root@1 (46 s)
ok  	github.com/carlosframework/platform	42.648s
=== root@2 (43 s)
ok  	github.com/carlosframework/platform	39.962s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50069666 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: cached, digest verified
  mc: cached, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.03s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.030s
=== RUN   TestT4StorageProbeMinio
2026/10/22 16:08:48 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E0E5CF000703C9, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.027s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-22-db-region-forward-plan-1-box.md b/docs/superpowers/plans/2026-10-22-db-region-forward-plan-1-box.md
index ee4c6e679..4b03b38f3 100644
--- a/docs/superpowers/plans/2026-10-22-db-region-forward-plan-1-box.md
+++ b/docs/superpowers/plans/2026-10-22-db-region-forward-plan-1-box.md
@@ -8,7 +8,7 @@
**Tech Stack:** Go 1.26, litestream v0.5.14 (`CalcRestorePlan`, the file replica client), superfly/ltx v0.5.1, modernc.org/sqlite v1.55.0, `crypto/ecdh` + `golang.org/x/crypto/hkdf` + AES-256-GCM, ed25519 root envelopes (`internal/release`), systemd socket activation, `golang.org/x/sys/unix`.
-**Spec:** `docs/superpowers/specs/2026-10-22-db-region-forward-design.md` (rev 5, accepted by Astra in round 5; Paul approved building it on 2026-10-22 with the rulings in §16). Read §3, §5, §7, §8, §11.4, §12, §13 and §15 before starting. Every task argues from the spec; where this plan and the spec disagree, the spec wins and the plan is the bug.
+**Spec:** `docs/superpowers/specs/2026-10-22-db-region-forward-design.md` (rev 6: rev 5, accepted by Astra in round 5 and approved by Paul for building on 2026-10-22 with the rulings in §16, plus the four clarifications the plan review forced, §5.4, §5.5, §5.7 and §9.3). Read §3, §5, §7, §8, §11.4, §12, §13 and §15 before starting. Every task argues from the spec; where this plan and the spec disagree, the spec wins and the plan is the bug.
**Branch:** `db-region-forward-1-box`, cut from amadan `main`. The worktree is made with the superpowers:using-git-worktrees skill. Plans 2 and 3 branch from `main` after this plan has merged.
@@ -41,6 +41,26 @@ The five input classes the spec implies but no gate test names, most likely firs
4. **Output at the cap.** Exactly 64 MiB, one byte over, and one row bigger than a chunk. A person expects the first 64 MiB, `output-cap`, and a summary whose count and hash cover exactly the kept bytes. Test: Task 11, `TestChunkWriterCapIsExact`.
5. **`systemctl` missing or failing.** The unit listing cannot be read. A person expects the box to take no read and skip (not fail) its probe, so a transient listing failure neither admits beside an orphan nor withdraws the capability. Test: Task 9, `TestAdmissionFailsClosedWhenUnitsCannotBeListed`, and Task 12, `TestProbeIsSkippedWhileAdmissionIsBlocked`.
+## Plan review, round 1
+
+Astra's findings (`docs/superpowers/specs/CODEX-FINDINGS-DB-REGION-FORWARD-PLANS.md`) for this plan, and where each landed:
+
+- **1** (the installed worker could not pass readiness): `SelfView` reads `/proc/self/net/dev` (`net.Interfaces` needs `AF_NETLINK`, which `RestrictAddressFamilies=AF_UNIX` refuses), reports every observation it could not make in `HelloReply.Errors`, and the probe fails on any. Tests: `TestParseNetDev`, the probe's "an observation that failed" case.
+- **2** (`waitFor` collides with `internal/host/housekeeper_test.go:139`): `dbreadWaitFor`.
+- **3** (the payload's report parsed with `PASS` after it): the report is one `DBREAD-REPORT ` line the parent scans for.
+- **4** (the readiness tests removed the wrong file): `dbreadServiceFile` is `/etc/systemd/system/carlos-dbread@.service`, restored by `mustSystemctl` in cleanup.
+- **5** (the census bootstrap could not run): `TestZZDBForwardFingerprints` computes each `@legacy` line from its row; plans 2 and 3 use it.
+- **9** (a result could name an unconfirmed chunk): `flushLocked` advances `seq` only once the create is confirmed; spec §5.4 says what the result then names. Test: `TestResultNamesOnlyConfirmedChunks`.
+- **10** (a retry reused the reservation): `admission.retry` keeps the first attempt's reservation under its own key and `settle` waits for its unit to stop before the retry dials; spec §5.7 says so. Tests: `TestRetryWaitsForTheFirstWorkerToStop`, `TestAdmissionRetryHoldsTheOldReservation`.
+- **11, the box's half**: a box reads `cancel` before it elects; spec §5.5 step 4 and §5.8. Test: `TestPassSkipsACancelledRequest`.
+- **14** (draining boxes could win): `AgentConfig.Draining`, from the box's own edges record, checked before admission and again after the jitter. Test: `TestADrainingBoxDoesNotPick`.
+- **15** (some start failures asked for no probe): every pre-ready transport failure asks for one. Test: `TestAWorkerThatNeverStartsAsksForAProbe`, and T13's fail-start case through the installed unit.
+- **16** (the disk test assumed queue order): the second read is submitted only once the first's unit is running.
+- **17** (T3 could not see `PrivateNetwork` go): the witness reports the interfaces the unit sees, and fails on any but loopback; ATTACH targets a real SQLite file.
+- **21, the box's half**: the agent logs its planning-and-restore time and its confirmed output, for plan 3's measurement.
+
+None declined.
+
## Census impact
- **Plan 14 root-read census:** one overlay, `plan14-root-reads-db-region-forward-1.tsv`, `rev3=PROPOSED-db-region-forward-1`:
@@ -2069,6 +2089,7 @@ import (
"encoding/json"
"io"
"net"
+ "strings"
"sync/atomic"
"testing"
"time"
@@ -2159,6 +2180,24 @@ func TestWorkerAnswersHelloWithItsSelfView(t *testing.T) {
}
}
+// The default self-view reads /proc/self/net/dev, which needs no socket
+// (plan review finding 1). Mutation: SelfView back on net.Interfaces makes
+// TestRootDBReadReadiness's baseline red inside the installed unit.
+func TestParseNetDev(t *testing.T) {
+ t.Parallel()
+ only := "Inter-| Receive\n face |bytes\n lo: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
+ if got := parseNetDev(only); len(got) != 1 || got[0] != "lo" {
+ t.Fatalf("%q", got)
+ }
+ two := only + " eth0: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16\n"
+ if got := parseNetDev(two); len(got) != 2 || got[1] != "eth0" {
+ t.Fatalf("%q", got)
+ }
+ if got := SelfView(); len(got.Errors) != 0 && !strings.Contains(strings.Join(got.Errors, " "), "/proc") {
+ t.Fatalf("SelfView on this machine: %+v", got)
+ }
+}
+
func TestWorkerRestoresThenRunsOnlyWhenTold(t *testing.T) {
t.Parallel()
dir := t.TempDir()
@@ -2377,6 +2416,9 @@ type HelloReply struct {
Interfaces []string `json:"interfaces"`
DataEntries int `json:"data_entries"`
EtcReadable bool `json:"etc_readable"`
+ // Errors names every observation the worker could not make. A probe
+ // fails on any: an unobserved sandbox is not a sandbox.
+ Errors []string `json:"errors,omitempty"`
}
type planHeader struct {
@@ -2448,9 +2490,9 @@ import (
"errors"
"fmt"
"io"
- "net"
"os"
"path/filepath"
+ "strings"
"time"
"github.com/benbjohnson/litestream"
@@ -2690,15 +2732,24 @@ func (f *frameWriter) flush() error {
// SelfView is what the worker can see of the box: in the unit, the loopback
// interface alone, an empty /data and no readable /etc/carlos.
+//
+// Interfaces come from /proc/self/net/dev, this process's own network
+// namespace. net.Interfaces cannot be used: on Linux it opens an AF_NETLINK
+// socket, which the unit's RestrictAddressFamilies=AF_UNIX refuses, and an
+// empty list would read as "no network" (plan review finding 1). Every
+// observation that fails is reported in Errors, and a probe fails on any.
func SelfView() HelloReply {
h := HelloReply{V: ProtocolVersion}
- if ifs, err := net.Interfaces(); err == nil {
- for _, i := range ifs {
- h.Interfaces = append(h.Interfaces, i.Name)
- }
+ if raw, err := os.ReadFile("/proc/self/net/dev"); err != nil {
+ h.Errors = append(h.Errors, "interfaces: "+err.Error())
+ } else {
+ h.Interfaces = parseNetDev(string(raw))
}
- if ents, err := os.ReadDir("/data"); err == nil {
+ switch ents, err := os.ReadDir("/data"); {
+ case err == nil:
h.DataEntries = len(ents)
+ case !errors.Is(err, os.ErrNotExist):
+ h.Errors = append(h.Errors, "/data: "+err.Error())
}
if f, err := os.Open("/etc/carlos/host.env"); err == nil {
f.Close()
@@ -2706,6 +2757,22 @@ func SelfView() HelloReply {
}
return h
}
+
+// parseNetDev reads /proc/net/dev's interface names: two header lines, then
+// one "<name>: <counters>" line per interface.
+func parseNetDev(s string) []string {
+ var out []string
+ for i, line := range strings.Split(s, "\n") {
+ if i < 2 {
+ continue
+ }
+ name, _, ok := strings.Cut(line, ":")
+ if name = strings.TrimSpace(name); ok && name != "" {
+ out = append(out, name)
+ }
+ }
+ return out
+}
```
- [ ] **Step 5: Run the tests**
@@ -2972,6 +3039,7 @@ import (
"context"
"errors"
"testing"
+ "time"
)
type unitsSeam struct {
@@ -3108,6 +3176,40 @@ func TestAdmissionAbandonFreesAtOnce(t *testing.T) {
}
}
+// Plan review finding 10: a retried read keeps its first attempt's
+// reservation while that worker's unit is still stopping, and settle waits
+// for the unit. Mutation: retry leaves held[id] in place instead of moving
+// it to a retired key; the grow below then fits and this test is red.
+func TestAdmissionRetryHoldsTheOldReservation(t *testing.T) {
+ t.Parallel()
+ u := &unitsSeam{}
+ a := newAdmission(u.count, func() (uint64, uint64, error) { return gib(12), gib(20), nil })
+ ctx := context.Background()
+ if err := a.admitRead(ctx, "r1"); err != nil {
+ t.Fatal(err)
+ }
+ if err := a.grow("r1", int64(gib(6))); err != nil {
+ t.Fatal(err)
+ }
+ u.n = 1 // the first worker's connection is closed; its unit is stopping
+ a.retry("r1", 1)
+ if err := a.grow("r1", int64(gib(5))); !errors.Is(err, errNoRoom) {
+ t.Fatalf("the stopping worker's scratch was not held: %v", err)
+ }
+ short, cancel := context.WithTimeout(ctx, 100*time.Millisecond)
+ defer cancel()
+ if err := a.settle(short, 10*time.Millisecond); !errors.Is(err, context.DeadlineExceeded) {
+ t.Fatalf("settle returned beside a live unit: %v", err)
+ }
+ u.n = 0
+ if err := a.settle(ctx, 10*time.Millisecond); err != nil {
+ t.Fatal(err)
+ }
+ if err := a.grow("r1", int64(gib(5))); err != nil {
+ t.Fatalf("after the unit stopped: %v", err)
+ }
+}
+
func TestAdmissionFailsClosedWhenTheDiskCannotBeRead(t *testing.T) {
t.Parallel()
a := newAdmission((&unitsSeam{}).count, func() (uint64, uint64, error) { return 0, 0, errors.New("statfs") })
@@ -3132,6 +3234,7 @@ import (
"errors"
"fmt"
"sync"
+ "time"
)
var (
@@ -3248,6 +3351,39 @@ func (a *admission) abandon(id string) {
delete(a.held, id)
}
+// retry is a read starting a fresh plan after a planned file vanished (spec
+// §8.2). Its previous worker may still hold scratch until its unit stops, so
+// that reservation moves to a retired key, held until no unit is left (J5),
+// and the read starts again from the floor (plan review finding 10).
+func (a *admission) retry(id string, attempt int) {
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ if v, ok := a.held[id]; ok {
+ a.held[fmt.Sprintf("%s#%d", id, attempt)] = v
+ }
+ a.held[id] = DiskFloor
+}
+
+// settle waits until no carlos-dbread@ unit is running beyond this agent's
+// open connections, so a retry opens its next worker only once the last one
+// is gone, and never into the probe's connection. It releases retired
+// reservations once no unit is left at all.
+func (a *admission) settle(ctx context.Context, poll time.Duration) error {
+ for {
+ a.mu.Lock()
+ err := a.reconcileLocked(ctx)
+ a.mu.Unlock()
+ if !errors.Is(err, errOrphans) {
+ return err
+ }
+ select {
+ case <-ctx.Done():
+ return ctx.Err()
+ case <-time.After(poll):
+ }
+ }
+}
+
// admitProbe admits the readiness probe: no orphan and no other probe. The
// probe's connection is the third (opened checks the count).
func (a *admission) admitProbe(ctx context.Context) bool {
@@ -3331,6 +3467,7 @@ import (
"net"
"os"
"sort"
+ "sync"
"sync/atomic"
"testing"
"time"
@@ -3354,12 +3491,35 @@ type inProcWorkers struct {
opt WorkerOptions
live atomic.Int64
dials atomic.Int64
+ // linger keeps a finished worker counted as a live unit for a while,
+ // as systemd's deactivation does; liveAtDial records the live count
+ // each dial saw.
+ linger time.Duration
+ mu sync.Mutex
+ liveAtDial []int64
+ // failStart makes the worker end at once without a word, as a unit
+ // whose service fails to start after the socket accepted does.
+ failStart bool
}
func (w *inProcWorkers) dial(context.Context) (net.Conn, error) {
agentEnd, workerEnd := net.Pipe()
+ w.mu.Lock()
+ w.liveAtDial = append(w.liveAtDial, w.live.Load())
+ w.mu.Unlock()
w.dials.Add(1)
w.live.Add(1)
+ if w.failStart {
+ // The socket accepted and the service died: the plan's header is
+ // read by nobody useful, and the connection drops.
+ go func() {
+ readFrame(workerEnd)
+ workerEnd.Close()
+ time.Sleep(w.linger)
+ w.live.Add(-1)
+ }()
+ return agentEnd, nil
+ }
opt := w.opt
if opt.Scratch == "" {
opt.Scratch = w.t.TempDir()
@@ -3371,9 +3531,10 @@ func (w *inProcWorkers) dial(context.Context) (net.Conn, error) {
opt.Self = func() HelloReply { return HelloReply{V: ProtocolVersion, Interfaces: []string{"lo"}} }
}
go func() {
- defer w.live.Add(-1)
ServeWorker(workerEnd, opt)
workerEnd.Close()
+ time.Sleep(w.linger)
+ w.live.Add(-1)
}()
return agentEnd, nil
}
@@ -3803,6 +3964,10 @@ type AgentConfig struct {
Now func() time.Time
Jitter func() time.Duration // 0 to MaxJitter
Logf func(format string, args ...any)
+ // Draining is this box's own published draining flag (its edges file's
+ // draining field); a draining box takes no new read. Never draining
+ // when nil.
+ Draining func() bool
// Test seams; zero is the spec's value.
OwnerDeadline time.Duration
@@ -3842,6 +4007,9 @@ func NewAgent(cfg AgentConfig) *Agent {
if cfg.Logf == nil {
cfg.Logf = func(string, ...any) {}
}
+ if cfg.Draining == nil {
+ cfg.Draining = func() bool { return false }
+ }
if cfg.OwnerDeadline == 0 {
cfg.OwnerDeadline = OwnerDeadline
}
@@ -4009,6 +4177,16 @@ func (a *Agent) consider(ctx context.Context, e QueueEntry) {
a.logf("dbreads: %s: refused the request: %v", e.ID, err)
return
}
+ // A request whose console has already given up (its request ended
+ // before it was announced) is never picked (spec §5.5 step 4).
+ if _, _, err := a.cfg.Store.Get(ctx, CancelKey(e.ID)); err == nil {
+ a.decide(e.ID)
+ return
+ }
+ // A draining box does not enter the election (spec §7.1, D2).
+ if a.cfg.Draining() {
+ return
+ }
if err := a.adm.admitRead(ctx, e.ID); err != nil {
return
}
@@ -4020,6 +4198,11 @@ func (a *Agent) consider(ctx context.Context, e QueueEntry) {
case <-time.After(d):
}
}
+ // It may have started draining during the jitter.
+ if a.cfg.Draining() {
+ a.adm.abandon(e.ID)
+ return
+ }
p, owned := a.elect(ctx, e.ID)
a.decide(e.ID)
if !owned {
@@ -4079,6 +4262,8 @@ import (
"context"
"crypto/sha256"
"encoding/hex"
+ "encoding/json"
+ "errors"
"io"
"os"
"strings"
@@ -4166,6 +4351,46 @@ func TestReadRePlansOnceThenSaysChangedTwice(t *testing.T) {
}
}
+// Plan review finding 10: after a vanished file, the retry waits for the
+// first worker's unit to stop before it dials again, and keeps that unit's
+// reservation until then. 12 GiB of 20 fits one floor at a time.
+// Mutations: drop the settle call in read (the second dial sees a live unit,
+// and the retired floor makes the new one no-room); or drop retry's move of
+// the old reservation (TestAdmissionRetryHoldsTheOldReservation goes red).
+func TestRetryWaitsForTheFirstWorkerToStop(t *testing.T) {
+ t.Parallel()
+ fx := newFixture(t, 2)
+ v := &vanishing{ReplicaReader: lsfile.NewReplicaClient(fx.replica), times: 1}
+ a, w := fx.agent("uk-1", func(c *AgentConfig, w *inProcWorkers) {
+ c.OpenReplica = func(context.Context, string) (dbdoor.ReplicaReader, error) { return v, nil }
+ c.Free = func() (uint64, uint64, error) { return 12 << 30, 20 << 30, nil }
+ w.linger = 300 * time.Millisecond
+ })
+ _, _, r := serveOne(t, a, fx, fx.request("SELECT count(*) FROM t", ""))
+ if r.State != StateDone {
+ t.Fatalf("result %+v", r)
+ }
+ w.mu.Lock()
+ defer w.mu.Unlock()
+ if len(w.liveAtDial) != 2 || w.liveAtDial[1] != 0 {
+ t.Fatalf("live units at each dial: %v; the retry dialed beside the first worker", w.liveAtDial)
+ }
+}
+
+// Plan review finding 15: a worker that ends before ready (its service
+// failed after the socket accepted) asks for a probe at once. Mutation:
+// drop the askProbe after a pre-ready sendFile or readFrame failure;
+// probeAfter stays empty and this test is red.
+func TestAWorkerThatNeverStartsAsksForAProbe(t *testing.T) {
+ t.Parallel()
+ fx := newFixture(t, 1)
+ a, _ := fx.agent("uk-1", func(_ *AgentConfig, w *inProcWorkers) { w.failStart = true })
+ _, _, r := serveOne(t, a, fx, fx.request("SELECT 1", ""))
+ if r.Code != CodeInterrupted || len(a.probeAfter) != 1 {
+ t.Fatalf("result %+v, probe asked %d", r, len(a.probeAfter))
+ }
+}
+
// stageHook changes the record at one stage of the read: planning (the
// first listing), download (the first full-file open), or after the last
// byte has been read from the bucket (EOF of the last planned file).
@@ -4361,6 +4586,50 @@ func TestCancelStopsTheOwner(t *testing.T) {
}
}
+// refusedChunk never lets one chunk's create land: every attempt answers an
+// error and the object stays absent.
+type refusedChunk struct {
+ release.Store
+ key string
+}
+
+func (s refusedChunk) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
+ if key == s.key {
+ return "", errors.New("connection reset")
+ }
+ return s.Store.Put(ctx, key, body, ifMatch)
+}
+
+// Plan review finding 9: a chunk that is never confirmed is never named by
+// the result. The read ends interrupted, and every chunk through its
+// last_seq exists. Mutation: flushLocked increments seq before the create is
+// confirmed; the result then names chunk 2 and this test is red.
+func TestResultNamesOnlyConfirmedChunks(t *testing.T) {
+ t.Parallel()
+ fx := newFixture(t, 1)
+ req := fx.request("SELECT hex(randomblob(300000))", "") // about 600 KB: three chunks
+ a, _ := fx.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) {
+ c.Store = refusedChunk{Store: fx.store, key: ChunkKey(req.ID, 2)}
+ c.OwnerDeadline = 2 * time.Second
+ })
+ a.consider(context.Background(), fx.submit(req))
+ a.Wait()
+ raw, _, err := fx.store.Get(context.Background(), ResultKey(req.ID))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var r Result
+ json.Unmarshal(raw, &r)
+ if r.State != StateFailed || r.Code != CodeInterrupted || r.LastSeq != 1 {
+ t.Fatalf("result %+v", r)
+ }
+ for seq := int64(1); seq <= r.LastSeq; seq++ {
+ if _, _, err := fx.store.Get(context.Background(), ChunkKey(req.ID, seq)); err != nil {
+ t.Fatalf("the result names chunk %d, which does not exist", seq)
+ }
+ }
+}
+
// strictStore fails the test on any protocol delete or replacing write
// under db-reads/ (T16, the box's half).
type strictStore struct {
@@ -4546,13 +4815,22 @@ func (c *chunkWriter) Write(p []byte) (int, error) {
return len(p), nil
}
+// flushLocked seals and creates the next chunk. seq moves only once the
+// create is confirmed, so close never reports, and no result ever names, a
+// chunk that was not confirmed (spec §5.4; plan review finding 9). A chunk
+// whose create stayed ambiguous until the context ended may land later at
+// seq+1; no result names it, so no reader relays it.
func (c *chunkWriter) flushLocked(b []byte, last bool) error {
- c.seq++
- sealed, err := SealChunk(c.pub, c.id, c.seq, c.stream, last, b)
+ next := c.seq + 1
+ sealed, err := SealChunk(c.pub, c.id, next, c.stream, last, b)
if err != nil {
return err
}
- return putCreateOnly(c.ctx, c.store, ChunkKey(c.id, c.seq), sealed)
+ if err := putCreateOnly(c.ctx, c.store, ChunkKey(c.id, next), sealed); err != nil {
+ return err
+ }
+ c.seq = next
+ return nil
}
func (c *chunkWriter) capped() bool {
@@ -4662,6 +4940,14 @@ func (a *Agent) read(ctx context.Context, req Request) Result {
}
at, _ := req.AtTime()
for attempt := 0; attempt < 2; attempt++ {
+ if attempt > 0 {
+ // The first attempt's worker may still be stopping: keep its
+ // reservation, and wait for its unit before the next one (J5).
+ a.adm.retry(req.ID, attempt)
+ if err := a.adm.settle(ctx, settlePoll); err != nil {
+ return a.ended(pub, req.ID, StateFailed, CodeInterrupted, Summary{}, 0)
+ }
+ }
if res, retry := a.attempt(ctx, req, pub, at); !retry {
return res
}
@@ -4669,6 +4955,9 @@ func (a *Agent) read(ctx context.Context, req Request) Result {
return a.ended(pub, req.ID, StateFailed, CodeChangedTwice, Summary{}, 0)
}
+// settlePoll is how often a retry rereads the unit listing while it waits.
+var settlePoll = 200 * time.Millisecond
+
// attempt is the identity bracket (spec §8.2): check A, plan, stream every
// planned file into a worker that restores and checks it, check B after the
// last byte has left the bucket, and only then run. No output exists before
@@ -4680,6 +4969,7 @@ func (a *Agent) attempt(ctx context.Context, req Request, pub *ecdh.PublicKey, a
refuse := func(code string) (Result, bool) {
return a.ended(pub, req.ID, StateRefused, code, Summary{}, 0), false
}
+ attemptStart := time.Now()
pctx, pcancel := context.WithTimeout(ctx, a.cfg.PlanBudget)
defer pcancel()
@@ -4739,12 +5029,17 @@ func (a *Agent) attempt(ctx context.Context, req Request, pub *ecdh.PublicKey, a
return Result{}, true
}
if err != nil {
+ // The worker went before ready: its unit may have failed to
+ // start after the socket accepted (spec §7.1; plan review
+ // finding 15). Probe once this connection is closed.
+ a.askProbe()
return fail(CodeInterrupted, Summary{})
}
}
typ, p, err := readFrame(conn)
switch {
case err != nil:
+ a.askProbe()
return fail(CodeInterrupted, Summary{})
case typ == frameEnd:
var e endMsg
@@ -4766,12 +5061,18 @@ func (a *Agent) attempt(ctx context.Context, req Request, pub *ecdh.PublicKey, a
return refuse(code)
}
stopPlanBound() // the query is bounded by the worker and the owner's deadline
+ // Phase timings, for operators and for plan 3's throughput measurement:
+ // planning and the worker's restore, then the output's sealing and
+ // confirmed creates.
+ a.logf("dbreads: %s: planned %d files (%d bytes) and restored to %d in %s", req.ID, plan.Files, plan.LTXBytes, plan.TXID, time.Since(attemptStart).Round(time.Millisecond))
+ runStart := time.Now()
if err := writeJSONFrame(conn, frameRun, runMsg{SQL: req.SQL}); err != nil {
return fail(CodeInterrupted, Summary{})
}
cw := newChunkWriter(ctx, a.cfg.Store, req.ID, pub, StreamRows, a.cfg.ChunkEvery, a.cfg.MaxOutput)
endCode, sqlErr, rerr := relay(conn, cw)
last, n, sum, werr := cw.close()
+ a.logf("dbreads: %s: output %d bytes in %d confirmed chunks in %s", req.ID, n, last, time.Since(runStart).Round(time.Millisecond))
s := Summary{TXID: plan.TXID, At: plan.At, Bytes: n, SHA256: sum, Error: sqlErr}
switch {
case werr != nil:
@@ -5035,6 +5336,53 @@ func TestTwoAgentsServeEachReadOnce(t *testing.T) {
}
}
+// Plan review finding 14 (spec §7.1, D2): a draining box beside a healthy
+// one never wins a read, including one that starts draining during its
+// jitter. Mutation: drop the Draining check after the jitter; the draining
+// agent picks every read and this test is red. (The check before admission
+// only spares a reservation; the one after the jitter is the gate.)
+func TestADrainingBoxDoesNotPick(t *testing.T) {
+ t.Parallel()
+ fx := newFixture(t, 1)
+ var flip atomic.Bool
+ drain, dw := fx.agent("uk-drain", func(c *AgentConfig, _ *inProcWorkers) {
+ c.Draining = flip.Load
+ c.Jitter = func() time.Duration { flip.Store(true); return time.Millisecond }
+ })
+ healthy, _ := fx.agent("uk-1")
+ for i := 0; i < 5; i++ {
+ req := fx.request("SELECT 1", "")
+ fx.submit(req)
+ drain.Pass(context.Background()) // first: it passes the first check, then drains in its jitter
+ drain.Wait()
+ healthy.Pass(context.Background())
+ healthy.Wait()
+ if _, _, r := fx.answer(req.ID); r.Box != "uk-1" {
+ t.Fatalf("read %d served by %s", i, r.Box)
+ }
+ flip.Store(false)
+ }
+ if dw.dials.Load() != 0 {
+ t.Fatal("the draining box started a worker")
+ }
+}
+
+// Finding 11's box half: a request already cancelled is never picked.
+// Mutation: drop the cancel read in consider; the agent picks it.
+func TestPassSkipsACancelledRequest(t *testing.T) {
+ t.Parallel()
+ fx := newFixture(t, 1)
+ a, w := fx.agent("uk-1")
+ req := fx.request("SELECT 1", "")
+ fx.submit(req)
+ fx.store.Put(context.Background(), CancelKey(req.ID), nil, "")
+ a.Pass(context.Background())
+ a.Wait()
+ if _, _, err := fx.store.Get(context.Background(), PickedKey(req.ID)); err == nil || w.dials.Load() != 0 {
+ t.Fatal("a cancelled request was picked")
+ }
+}
+
func TestPassNeedsTheCapability(t *testing.T) {
t.Parallel()
fx := newFixture(t, 0)
@@ -5059,6 +5407,9 @@ func TestProbe(t *testing.T) {
"a network": {func() HelloReply { return HelloReply{V: 1, Interfaces: []string{"lo", "eth0"}} }, nil, false},
"a visible /data": {func() HelloReply { return HelloReply{V: 1, Interfaces: []string{"lo"}, DataEntries: 3} }, nil, false},
"a readable host.env": {func() HelloReply { return HelloReply{V: 1, Interfaces: []string{"lo"}, EtcReadable: true} }, nil, false},
+ "an observation that failed": {func() HelloReply {
+ return HelloReply{V: 1, Errors: []string{"interfaces: permission denied"}}
+ }, nil, false},
"no service": {nil, errors.New("connection refused"), false},
} {
t.Run(name, func(t *testing.T) {
@@ -5245,7 +5596,7 @@ func (a *Agent) probe(ctx context.Context) error {
if h.V != ProtocolVersion {
return fmt.Errorf("dbreads: the worker speaks protocol %d, this agent %d", h.V, ProtocolVersion)
}
- if len(h.Interfaces) != 1 || h.Interfaces[0] != "lo" || h.DataEntries != 0 || h.EtcReadable {
+ if len(h.Errors) != 0 || len(h.Interfaces) != 1 || h.Interfaces[0] != "lo" || h.DataEntries != 0 || h.EtcReadable {
return fmt.Errorf("%w: %+v", errSandbox, h)
}
return nil
@@ -5645,6 +5996,13 @@ var armedDBReads atomic.Pointer[dbreads.Agent]
// dbReadsNudge is the serial pass's word to the loop: list the queue now.
var dbReadsNudge = make(chan struct{}, 1)
+// dbReadsDraining is the draining flag this box last published in its edges
+// file, which the edges loop stores on every pass. The agent reads the same
+// value the console's evidence rule reads, so the two never disagree (spec
+// §7.1). Nothing on main sets an edges file's Draining yet (region HA's
+// placer will); until then it is false.
+var dbReadsDraining atomic.Bool
+
func nudgeDBReads() {
select {
case dbReadsNudge <- struct{}{}:
@@ -5707,7 +6065,7 @@ func armDBReads(hib activator.Config) {
return lease.ReadReplicaClient(ctx, hib.Bucket, prefix, hib.Region)
},
DialWorker: dialDBReadWorker, Units: dbReadUnits, Free: varTmpFree,
- MaxRestored: env.maxRestored, Logf: log.Printf,
+ MaxRestored: env.maxRestored, Logf: log.Printf, Draining: dbReadsDraining.Load,
})
armedDBReads.Store(a)
go a.Run(ctx, dbReadsNudge)
@@ -5783,9 +6141,10 @@ In `internal/host/edgesfile.go`, line 890:
```go
ident.Capabilities = withDBReadCapability(withOfflineCapability(baseCaps, armedOfflineCapable()), dbReadsCapable())
+ dbReadsDraining.Store(ident.Draining)
```
-and extend the comment above `baseCaps` to say db-read-1 joins per pass too.
+and extend the comment above `baseCaps` to say db-read-1 joins per pass too, and that the agent's draining check reads the flag this file publishes.
In `internal/host/agent.go`, right after `armRunsPass(svc, reg, dd, rolloutSink)` (line 612):
@@ -6148,9 +6507,11 @@ import (
"context"
"database/sql"
"encoding/json"
+ "errors"
"net"
"os"
"path/filepath"
+ "strings"
"time"
"github.com/benbjohnson/litestream"
@@ -6234,12 +6595,31 @@ func dbreadsWitness() {
try("open another instance", open("/data/instances/other.example.com/other.db"))
try("connect the metadata service", dial("169.254.169.254:80"))
try("connect a TCP address", dial("1.1.1.1:443"))
+ // Network-namespace evidence that needs no socket (plan review finding
+ // 17): with RestrictAddressFamilies=AF_UNIX the TCP dials above fail
+ // even without PrivateNetwork, so they cannot witness it. This can: it
+ // "succeeds" (and so fails the test) when any interface but loopback is
+ // visible, which is the case the moment PrivateNetwork is removed.
+ try("see an interface other than loopback", func() error {
+ raw, err := os.ReadFile("/proc/self/net/dev")
+ if err != nil {
+ return nil // unobservable counts as reachable: the test fails
+ }
+ ifs := strings.Join(dbreadsParseNetDev(string(raw)), ",")
+ if ifs == "lo" {
+ return errors.New("only lo")
+ }
+ return nil
+ })
try("write /opt/carlos", write("/opt/carlos/witness"))
try("write /var/lib", write("/var/lib/witness"))
try("write /etc", write("/etc/witness"))
try("write /var/tmp", write("/var/tmp/witness")) // the positive control
// SQLite with the attach limit removed: the mutation. Only the sandbox
- // stands between these statements and the box.
+ // stands between these statements and the box. The attached targets are
+ // real SQLite databases the runner wrote on the host, so a statement
+ // fails only because the file cannot be reached, never because it is
+ // not a database.
sqlTry := func(name, q string) {
try(name, func() error {
db, err := sql.Open("sqlite", "file:/var/tmp/witness.db")
@@ -6252,14 +6632,29 @@ func dbreadsWitness() {
})
}
sqlTry("attach /data/registry.db", "ATTACH DATABASE '/data/registry.db' AS r; SELECT count(*) FROM r.sqlite_master")
- sqlTry("attach /etc/carlos/host.env", "ATTACH DATABASE '/etc/carlos/host.env' AS e; SELECT count(*) FROM e.sqlite_master")
+ sqlTry("attach /etc/carlos/witness.db", "ATTACH DATABASE '/etc/carlos/witness.db' AS e; SELECT count(*) FROM e.sqlite_master")
sqlTry("vacuum into /opt/carlos", "VACUUM INTO '/opt/carlos/copy.db'")
sqlTry("load_extension", "SELECT load_extension('/lib/x86_64-linux-gnu/libc.so.6')")
json.NewEncoder(conn).Encode(report)
}
+
+// dbreadsParseNetDev is dbreads.parseNetDev, restated: that one is
+// unexported in another package.
+func dbreadsParseNetDev(s string) []string {
+ var out []string
+ for i, line := range strings.Split(s, "\n") {
+ if i < 2 {
+ continue
+ }
+ if name, _, ok := strings.Cut(line, ":"); ok && strings.TrimSpace(name) != "" {
+ out = append(out, strings.TrimSpace(name))
+ }
+ }
+ return out
+}
```
-Import `_ "modernc.org/sqlite"` in this file if no other file of the host test binary registers the driver (check with `grep -rln 'modernc.org/sqlite' internal/host/*.go`; the registry package registers it for production code, so it is likely already linked).
+The SQLite driver is already linked into the host test binary (`internal/registry` imports `modernc.org/sqlite`).
In `TestMain` (`internal/host/plan14_generation_registry_root_linux_test.go:18`), add at the very top:
@@ -6289,11 +6684,13 @@ Create `internal/host/dbreads_rootrunner_linux_test.go`:
package host
import (
+ "bufio"
"bytes"
"context"
"crypto/ecdh"
"crypto/ed25519"
"crypto/rand"
+ "database/sql"
"encoding/base64"
"encoding/json"
"fmt"
@@ -6318,7 +6715,16 @@ import (
"github.com/carlosframework/platform/internal/release"
)
-const dbreadRunnerHost = "shop.example.com"
+const (
+ dbreadRunnerHost = "shop.example.com"
+ // dbreadReportPrefix marks the payload's one report line. m.Run prints
+ // PASS after the payload test returns, so the report is never the last
+ // line (plan review finding 3).
+ dbreadReportPrefix = "DBREAD-REPORT "
+ // dbreadServiceFile is where ensureSystemUnits("/etc", ...) installs the
+ // worker's service (plan review finding 4).
+ dbreadServiceFile = "/etc/systemd/system/carlos-dbread@.service"
+)
// dbreadRunner is the production wiring on a throwaway root runner: the real
// units installed by the real ensureSystemUnits into /etc, as the agent's
@@ -6341,6 +6747,27 @@ func runSystemctl(args ...string) (string, error) {
return string(out), err
}
+func mustSystemctl(t *testing.T, args ...string) {
+ t.Helper()
+ if out, err := runSystemctl(args...); err != nil {
+ t.Fatalf("systemctl %s: %v: %s", strings.Join(args, " "), err, out)
+ }
+}
+
+// sqliteFixture writes a real SQLite database at path, so an ATTACH of it
+// fails only when the file cannot be reached (plan review finding 17).
+func sqliteFixture(t *testing.T, path string) {
+ t.Helper()
+ db, err := sql.Open("sqlite", path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer db.Close()
+ if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS secret (v TEXT); INSERT INTO secret VALUES ('not for tenants')`); err != nil {
+ t.Fatal(err)
+ }
+}
+
func newDBReadRunner(t *testing.T, tmpSize string) *dbreadRunner {
t.Helper()
if os.Geteuid() != 0 {
@@ -6375,17 +6802,22 @@ func newDBReadRunner(t *testing.T, tmpSize string) *dbreadRunner {
t.Fatal(err)
}
}
- // Things the witness must not reach, present on the host.
- os.WriteFile("/data/registry.db", []byte("registry"), 0o644)
- os.WriteFile("/etc/carlos/host.env", []byte("CARLOS_REGION=uk\n"), 0o640)
- os.WriteFile("/data/instances/other.example.com/other.db", []byte("other"), 0o644)
+ // Things the witness must not reach, present on the host: real SQLite
+ // databases where the witness ATTACHes, a host.env it opens.
+ sqliteFixture(t, "/data/registry.db")
+ sqliteFixture(t, "/etc/carlos/witness.db")
+ sqliteFixture(t, "/data/instances/other.example.com/other.db")
+ if err := os.WriteFile("/etc/carlos/host.env", []byte("CARLOS_REGION=uk\n"), 0o640); err != nil {
+ t.Fatal(err)
+ }
if err := os.WriteFile("/opt/carlos/carlos", self, 0o755); err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
- for _, p := range []string{"/opt/carlos/carlos", "/data/registry.db", "/etc/carlos/host.env"} {
+ for _, p := range []string{"/opt/carlos/carlos", "/data/registry.db", "/etc/carlos/witness.db", "/etc/carlos/host.env"} {
os.Remove(p)
}
+ os.RemoveAll("/data/instances/other.example.com")
os.RemoveAll(dbreadTestMarkers)
})
ensureSystemUnits("/etc", filepath.Join(t.TempDir(), "lock"), runSystemctl, systemctlProbe, t.Logf)
@@ -6397,6 +6829,9 @@ func newDBReadRunner(t *testing.T, tmpSize string) *dbreadRunner {
}
runSystemctl("daemon-reload")
})
+ if _, err := os.Stat(dbreadServiceFile); err != nil {
+ t.Fatalf("ensureSystemUnits installed no %s: %v", dbreadServiceFile, err)
+ }
if out, _ := systemctlProbe("is-active", dbReadSocketUnit); strings.TrimSpace(out) != "active" {
t.Fatalf("carlos-dbread.socket is %q after ensureSystemUnits", out)
}
@@ -6426,37 +6861,48 @@ func (r *dbreadRunner) chown() {
}
func (r *dbreadRunner) marker(name string, on bool) {
+ r.t.Helper()
p := filepath.Join(dbreadTestMarkers, name)
+ var err error
if on {
- os.WriteFile(p, nil, 0o644)
- } else {
- os.Remove(p)
+ err = os.WriteFile(p, nil, 0o644)
+ } else if err = os.Remove(p); os.IsNotExist(err) {
+ err = nil
+ }
+ if err != nil {
+ r.t.Fatal(err)
}
}
-// submit is the console's two creates; it returns the reply key.
-func (r *dbreadRunner) submit(sqlText string) (string, *ecdh.PrivateKey) {
+// submit is the console's two creates for a read of op; it returns the
+// read's id and reply key.
+func (r *dbreadRunner) submit(op, sqlText string) (string, *ecdh.PrivateKey) {
r.t.Helper()
reply, _ := dbreads.NewReplyKey()
id, _ := dbreads.MintID()
inc, _ := hostclaim.Incarnation("bab", "shop", dbreadRunnerHost, time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC))
req := dbreads.Request{ID: id, Account: "bab", App: "shop", Host: dbreadRunnerHost, Incarnation: inc, Region: "uk",
- Op: dbreads.OpQuery, SQL: sqlText, ReplyKey: dbreads.EncodeReplyKey(reply.PublicKey()),
+ Op: op, SQL: sqlText, ReplyKey: dbreads.EncodeReplyKey(reply.PublicKey()),
By: "member@example.com", IssuedAt: time.Now().UTC().Format(time.RFC3339)}
wire, err := dbreads.Sign(release.RootSigner(r.rootPriv), req)
if err != nil {
r.t.Fatal(err)
}
s := r.store()
- s.Put(context.Background(), dbreads.RequestKey(id), wire, "")
+ if _, err := s.Put(context.Background(), dbreads.RequestKey(id), wire, ""); err != nil {
+ r.t.Fatal(err)
+ }
issued, _ := req.Issued()
- s.Put(context.Background(), dbreads.QueueKey("uk", issued, id), nil, "")
+ if _, err := s.Put(context.Background(), dbreads.QueueKey("uk", issued, id), nil, ""); err != nil {
+ r.t.Fatal(err)
+ }
r.chown()
return id, reply
}
-// payload runs the agent half as carlos and returns its JSON report. id, when
-// set, is the read the scenario waits on or reports about.
+// payload runs the agent half as carlos. With wait it returns the payload's
+// report, the line marked dbreadReportPrefix; without, the running command.
+// id, when set, is the read the scenario waits on or reports about.
func (r *dbreadRunner) payload(scenario string, wait bool, id string) (map[string]any, *exec.Cmd) {
r.t.Helper()
cmd := exec.Command(os.Args[0])
@@ -6474,19 +6920,21 @@ func (r *dbreadRunner) payload(scenario string, wait bool, id string) (map[strin
if err := cmd.Run(); err != nil {
r.t.Fatalf("payload %s: %v: %s", scenario, err, out.String())
}
- var rep map[string]any
- if err := json.Unmarshal(lastLine(out.Bytes()), &rep); err != nil {
- r.t.Fatalf("payload %s said %q", scenario, out.String())
+ sc := bufio.NewScanner(&out)
+ for sc.Scan() {
+ if line, ok := strings.CutPrefix(sc.Text(), dbreadReportPrefix); ok {
+ var rep map[string]any
+ if err := json.Unmarshal([]byte(line), &rep); err != nil {
+ r.t.Fatalf("payload %s: unreadable report %q", scenario, line)
+ }
+ return rep, cmd
+ }
}
- return rep, cmd
+ r.t.Fatalf("payload %s gave no report: %s", scenario, out.String())
+ return nil, cmd
}
-func lastLine(b []byte) []byte {
- lines := bytes.Split(bytes.TrimSpace(b), []byte("\n"))
- return lines[len(lines)-1]
-}
-
-func (r *dbreadRunner) answer(id string, reply *ecdh.PrivateKey) ([]byte, dbreads.Result) {
+func (r *dbreadRunner) answer(id string, reply *ecdh.PrivateKey, stream string) ([]byte, dbreads.Result) {
r.t.Helper()
s := r.store()
raw, _, err := s.Get(context.Background(), dbreads.ResultKey(id))
@@ -6498,7 +6946,7 @@ func (r *dbreadRunner) answer(id string, reply *ecdh.PrivateKey) ([]byte, dbread
var out bytes.Buffer
for seq := int64(1); seq <= res.LastSeq; seq++ {
sealed, _, _ := s.Get(context.Background(), dbreads.ChunkKey(id, seq))
- plain, err := dbreads.OpenChunk(reply, id, seq, dbreads.StreamRows, seq == res.LastSeq, sealed)
+ plain, err := dbreads.OpenChunk(reply, id, seq, stream, seq == res.LastSeq, sealed)
if err != nil {
r.t.Fatal(err)
}
@@ -6532,33 +6980,50 @@ func unitsActive(t *testing.T) int {
return n
}
+// dbreadWaitFor polls ok until it holds or limit passes. (waitFor is
+// housekeeper_test.go's, with another signature: plan review finding 2.)
+func dbreadWaitFor(t *testing.T, limit time.Duration, ok func() bool) {
+ t.Helper()
+ deadline := time.Now().Add(limit)
+ for !ok() {
+ if time.Now().After(deadline) {
+ t.Fatal("timed out")
+ }
+ time.Sleep(200 * time.Millisecond)
+ }
+}
+
// T2: a forwarded query through the real socket and the real service,
// installed by the real ensureSystemUnits; the socket is the worker's stdin
-// and stdout (codex 2).
+// and stdout (codex 2). The probe that makes the box capable runs the
+// default SelfView inside the unit (plan review finding 1).
func TestRootDBReadServesThroughTheInstalledUnits(t *testing.T) {
r := newDBReadRunner(t, "40g")
- id, reply := r.submit("SELECT count(*) AS n FROM t")
+ id, reply := r.submit(dbreads.OpQuery, "SELECT count(*) AS n FROM t")
rep, _ := r.payload("serve", true, id)
if rep["capable"] != true {
t.Fatalf("the installed worker failed its probe: %v", rep)
}
- rows, res := r.answer(id, reply)
+ rows, res := r.answer(id, reply, dbreads.StreamRows)
if res.State != dbreads.StateDone || string(rows) != "n\n30\n" {
t.Fatalf("result %+v rows %q", res, rows)
}
privateTmpGone(t, 10*time.Second)
}
-// T3: from inside the worker, nothing outside its scratch is reachable; the
-// private tmp is gone after a normal end, a cancel and kill -9 of the agent.
+// T3: from inside the worker, nothing outside its scratch is reachable,
+// including any network interface but loopback; the private tmp is gone
+// after a normal end, a cancel and kill -9 of the agent. Mutation: remove
+// PrivateNetwork=yes from the installed unit; "see an interface other than
+// loopback" succeeds and this test is red (plan review finding 17).
func TestRootDBReadWorkerSandbox(t *testing.T) {
r := newDBReadRunner(t, "40g")
r.marker("witness", true)
rep, _ := r.payload("witness", true, "")
r.marker("witness", false)
report, _ := rep["report"].(map[string]any)
- if len(report) == 0 {
- t.Fatalf("no witness report: %v", rep)
+ if len(report) != 14 {
+ t.Fatalf("an incomplete witness report: %v", rep)
}
for name, v := range report {
failed := v.(string) != ""
@@ -6575,53 +7040,47 @@ func TestRootDBReadWorkerSandbox(t *testing.T) {
// A cancel, then kill -9 of the agent, mid-read.
r.marker("hang-restore", true)
defer r.marker("hang-restore", false)
- id, _ := r.submit("SELECT 1")
+ id, _ := r.submit(dbreads.OpQuery, "SELECT 1")
_, cmd := r.payload("serve-and-hang", false, "")
- waitFor(t, 30*time.Second, func() bool { return unitsActive(t) >= 1 })
- r.store().Put(context.Background(), dbreads.CancelKey(id), nil, "")
- waitFor(t, 20*time.Second, func() bool { return unitsActive(t) == 0 })
+ dbreadWaitFor(t, 30*time.Second, func() bool { return unitsActive(t) >= 1 })
+ if _, err := r.store().Put(context.Background(), dbreads.CancelKey(id), nil, ""); err != nil {
+ t.Fatal(err)
+ }
+ dbreadWaitFor(t, 20*time.Second, func() bool { return unitsActive(t) == 0 })
privateTmpGone(t, 10*time.Second)
cmd.Process.Kill()
cmd.Wait()
- r.submit("SELECT 1")
+ r.submit(dbreads.OpQuery, "SELECT 1")
_, cmd = r.payload("serve-and-hang", false, "")
- waitFor(t, 30*time.Second, func() bool { return unitsActive(t) >= 1 })
+ dbreadWaitFor(t, 30*time.Second, func() bool { return unitsActive(t) >= 1 })
cmd.Process.Signal(syscall.SIGKILL)
cmd.Wait()
- waitFor(t, 20*time.Second, func() bool { return unitsActive(t) == 0 })
+ dbreadWaitFor(t, 20*time.Second, func() bool { return unitsActive(t) == 0 })
privateTmpGone(t, 10*time.Second)
}
-func waitFor(t *testing.T, limit time.Duration, ok func() bool) {
- t.Helper()
- deadline := time.Now().Add(limit)
- for !ok() {
- if time.Now().After(deadline) {
- t.Fatal("timed out")
- }
- time.Sleep(200 * time.Millisecond)
- }
-}
-
// T12 with real units. The disk floor: /var/tmp on a 12 GiB tmpfs fits one
-// read's floor reservation (5 GiB and 10%) and not two, so the second read
-// waits unpicked until the first worker unit and its scratch are gone. A
-// fixture database small enough for the runner cannot make the plan's
-// growth bind, so the no-room result itself is pinned by
-// TestReadTooLargeAndNoRoom. Then an agent killed mid-read: the restarted
-// agent takes no read and no probe while the old unit is active, its worker
-// ends within 5 s of losing the connection though its restore ignores the
-// context, and the restarted agent admits only once the unit is gone.
+// read's floor reservation (5 GiB and 10%) and not two. The first read is
+// running (its unit observed) before the second is submitted, so which read
+// waits never depends on queue order (plan review finding 16); the agent
+// passes every second and still never picks the second while the first
+// holds its floor. A fixture database small enough for the runner cannot
+// make the plan's growth bind, so the no-room result itself is pinned by
+// TestReadTooLargeAndNoRoom. Then the agent is killed mid-read: the
+// restarted agent takes no read and no probe while the old unit is active,
+// the old worker ends within 5 s of losing the connection though its
+// restore ignores the context, and the restarted agent admits the held-back
+// read once the unit and its scratch are gone. Mutations: drop the orphan
+// rule in reconcileLocked (the restarted agent picks beside the old unit),
+// or drop the worker's 5 s exit (the old unit outlives the 15 s bound).
func TestRootDBReadAdmissionWithRealUnits(t *testing.T) {
- // 12 GiB: one read's 5 GiB floor leaves 7 GiB (at least 5 GiB and 10%),
- // a second leaves 2 GiB, which is refused.
r := newDBReadRunner(t, "12g")
r.marker("hang-restore", true)
- r.submit("SELECT 1")
- second, reply := r.submit("SELECT count(*) AS n FROM t")
+ r.submit(dbreads.OpQuery, "SELECT 1")
_, hung := r.payload("serve-and-hang", false, "")
- waitFor(t, 30*time.Second, func() bool { return unitsActive(t) == 1 })
- time.Sleep(3 * time.Second)
+ dbreadWaitFor(t, 30*time.Second, func() bool { return unitsActive(t) == 1 })
+ second, reply := r.submit(dbreads.OpQuery, "SELECT count(*) AS n FROM t")
+ time.Sleep(5 * time.Second) // five passes
if _, _, err := r.store().Get(context.Background(), dbreads.PickedKey(second)); err == nil {
t.Fatal("a second read was picked with room for one")
}
@@ -6629,10 +7088,13 @@ func TestRootDBReadAdmissionWithRealUnits(t *testing.T) {
hung.Wait()
killedAt := time.Now()
rep, _ := r.payload("restarted", true, second)
- if rep["units_at_start"].(float64) > 0 && (rep["capable"] == true || rep["picked"] == true) {
+ if rep["units_at_start"].(float64) < 1 {
+ t.Fatalf("the old unit was gone before the restarted agent looked: %v", rep)
+ }
+ if rep["capable"] == true || rep["picked"] == true {
t.Fatalf("a restarted agent admitted beside the old unit: %v", rep)
}
- waitFor(t, 20*time.Second, func() bool { return unitsActive(t) == 0 })
+ dbreadWaitFor(t, 20*time.Second, func() bool { return unitsActive(t) == 0 })
if gone := time.Since(killedAt); gone > 15*time.Second {
t.Fatalf("the old worker took %s to stop; its 5 s exit did not hold", gone)
}
@@ -6642,14 +7104,16 @@ func TestRootDBReadAdmissionWithRealUnits(t *testing.T) {
if rep["capable"] != true {
t.Fatalf("the agent did not admit once the unit was gone: %v", rep)
}
- if rows, res := r.answer(second, reply); res.State != dbreads.StateDone || string(rows) != "n\n30\n" {
+ if rows, res := r.answer(second, reply, dbreads.StreamRows); res.State != dbreads.StateDone || string(rows) != "n\n30\n" {
t.Fatalf("the held-back read: %+v %q", res, rows)
}
}
// T13: a socket whose service is missing, masked, failing to start or
// speaking another protocol version withdraws db-read-1 within one probe and
-// edges pass; a box running two reads keeps it (codex 9).
+// edges pass; a read whose worker fails to start withdraws it at once
+// through the real loop (plan review finding 15); a box running two reads
+// keeps it (codex 9).
func TestRootDBReadReadiness(t *testing.T) {
r := newDBReadRunner(t, "40g")
capable := func() bool {
@@ -6659,20 +7123,34 @@ func TestRootDBReadReadiness(t *testing.T) {
if !capable() {
t.Fatal("baseline: not capable")
}
- svc := "/run/systemd/system/carlos-dbread@.service"
- body, _ := os.ReadFile(svc)
- os.Remove(svc)
- runSystemctl("daemon-reload")
+ body, err := os.ReadFile(dbreadServiceFile)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() {
+ os.WriteFile(dbreadServiceFile, body, 0o644)
+ runSystemctl("unmask", "--runtime", "carlos-dbread@.service")
+ runSystemctl("daemon-reload")
+ })
+ if err := os.Remove(dbreadServiceFile); err != nil {
+ t.Fatal(err)
+ }
+ mustSystemctl(t, "daemon-reload")
if capable() {
t.Error("capable with the service missing")
}
- os.WriteFile(svc, body, 0o644)
- runSystemctl("daemon-reload")
- runSystemctl("mask", "--runtime", "carlos-dbread@.service")
+ if err := os.WriteFile(dbreadServiceFile, body, 0o644); err != nil {
+ t.Fatal(err)
+ }
+ mustSystemctl(t, "daemon-reload")
+ if !capable() {
+ t.Fatal("not capable once the service file is back")
+ }
+ mustSystemctl(t, "mask", "--runtime", "carlos-dbread@.service")
if capable() {
t.Error("capable with the service masked")
}
- runSystemctl("unmask", "--runtime", "carlos-dbread@.service")
+ mustSystemctl(t, "unmask", "--runtime", "carlos-dbread@.service")
for _, m := range []string{"exit-1", "version-2"} {
r.marker(m, true)
if capable() {
@@ -6683,10 +7161,27 @@ func TestRootDBReadReadiness(t *testing.T) {
if !capable() {
t.Fatal("not capable again once restored")
}
+
+ // A read whose worker fails to start, through the real loop.
+ _, loop := r.payload("fail-start", false, "")
+ dbreadWaitFor(t, 30*time.Second, func() bool { _, err := os.Stat(filepath.Join(r.dir, "probed")); return err == nil })
+ r.marker("exit-1", true)
+ r.submit(dbreads.OpQuery, "SELECT 1")
+ if err := os.WriteFile(filepath.Join(r.dir, "go"), nil, 0o644); err != nil {
+ t.Fatal(err)
+ }
+ if err := loop.Wait(); err != nil {
+ t.Fatalf("fail-start payload: %v", err)
+ }
+ if _, err := os.Stat(filepath.Join(r.dir, "withdrawn")); err != nil {
+ t.Fatal("the loop kept db-read-1 after a worker failed to start")
+ }
+ r.marker("exit-1", false)
+
r.marker("hang-restore", true)
defer r.marker("hang-restore", false)
- r.submit("SELECT 1")
- r.submit("SELECT 1")
+ r.submit(dbreads.OpQuery, "SELECT 1")
+ r.submit(dbreads.OpQuery, "SELECT 1")
rep, _ := r.payload("busy-probe", true, "")
if rep["capable"] != true {
t.Fatalf("a box running two reads lost the capability: %v", rep)
@@ -6715,7 +7210,7 @@ func TestRootDBReadPayload(t *testing.T) {
report := map[string]any{}
say := func() {
b, _ := json.Marshal(report)
- fmt.Println(string(b))
+ fmt.Println(dbreadReportPrefix + string(b))
}
switch scenario {
case "probe":
@@ -6737,8 +7232,10 @@ func TestRootDBReadPayload(t *testing.T) {
}
case "serve-and-hang":
a.Probe(ctx)
- a.Pass(ctx)
- select {} // the parent kills this process
+ for { // the parent kills this process
+ a.Pass(ctx)
+ time.Sleep(time.Second)
+ }
case "restarted":
units, _ := dbReadUnits(ctx)
report["units_at_start"] = units
@@ -6747,6 +7244,31 @@ func TestRootDBReadPayload(t *testing.T) {
report["capable"] = a.Capable()
_, _, err := store.Get(ctx, dbreads.PickedKey(os.Getenv("DBREAD_RUNNER_ID")))
report["picked"] = err == nil
+ case "fail-start":
+ // The production loop: probe at start, a pass on a nudge, a probe
+ // as soon as a read's worker fails to start.
+ nudge := make(chan struct{}, 1)
+ rctx, cancel := context.WithCancel(ctx)
+ defer cancel()
+ go a.Run(rctx, nudge)
+ for !a.Capable() {
+ time.Sleep(100 * time.Millisecond)
+ }
+ os.WriteFile(filepath.Join(dir, "probed"), nil, 0o644)
+ for {
+ if _, err := os.Stat(filepath.Join(dir, "go")); err == nil {
+ break
+ }
+ time.Sleep(100 * time.Millisecond)
+ }
+ nudge <- struct{}{}
+ deadline := time.Now().Add(30 * time.Second)
+ for a.Capable() && time.Now().Before(deadline) {
+ time.Sleep(100 * time.Millisecond)
+ }
+ if !a.Capable() {
+ os.WriteFile(filepath.Join(dir, "withdrawn"), nil, 0o644)
+ }
case "busy-probe":
a.Probe(ctx)
a.Pass(ctx)
@@ -6934,7 +7456,28 @@ Create `internal/provision/testdata/plan14-root-reads-db-region-forward-1.tsv`.
```
- **Envelopes:** one `@coverage <dir> <hash>` line for every directory the census reports changed or new.
-Compute each `@legacy` fingerprint with `rootReadLegacyFingerprint` over exactly the row's seven fields: add a `t.Logf` to the dump test that loads the overlay file with `loadRootReadInventory` into a fresh inventory and prints `rootReadLegacyFingerprint(e)` for every legacy entry, then paste the values.
+Compute each `@legacy` fingerprint straight from the row's seven fields, never by loading the unfinished overlay: the loader refuses a `@legacy` line whose fingerprint is wrong (`plan14_root_reads_test.go:413-421`) and an `@remove` of an entry no earlier file declared (`:470`), so it cannot bootstrap its own fingerprints (plan review finding 5). Add to the temporary dump test:
+
+```go
+// TestZZDBForwardFingerprints prints the @legacy line for every plain legacy
+// row of the overlay, from the row's own seven fields.
+func TestZZDBForwardFingerprints(t *testing.T) {
+ raw, err := os.ReadFile(filepath.Join(censusRoot(t), "internal/provision/testdata/plan14-root-reads-db-region-forward-1.tsv"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, line := range strings.Split(string(raw), "\n") {
+ col := strings.Split(line, "\t")
+ if len(col) != 7 || strings.HasPrefix(line, "#") || strings.HasPrefix(line, "@") || col[2] != "legacy" {
+ continue
+ }
+ e := classifiedRootRead{rootReadEntry: rootReadEntry{File: col[0], Unit: col[1], Kinds: col[5], Digest: col[6]}, Class: col[2], Task: col[3], Basis: col[4]}
+ t.Logf("@legacy\t%s\t%s", rootReadLegacyFingerprint(e), line)
+ }
+}
+```
+
+(add `"os"` and `"path/filepath"` to the dump file's imports). Write the plain rows first, run this, paste each printed `@legacy` line above its row, and only then run `TestPlan14RootReadCensus`, which loads the finished overlay over main's layered inventory, so its `@remove` finds main's `db.go runQueryContext` entry. Plans 2 and 3 use this same procedure with their own overlay file names.
- [ ] **Step 4: Load the overlay last, with the seal**
@@ -7017,7 +7560,14 @@ Each mutation below must turn its test red. Make them in a copy of the tree, nev
|---|---|
| `Verify` returns the decoded request without calling `release.VerifyRootEnvelope` | `TestEnvelopeRefusesForgeryAndReplay`, `TestPassRefusesForgedEntriesAndUnsignedRequests` |
| `check` drops its `rec.Host != req.Host` and incarnation comparisons | `TestCheckIsTheRecordAsTheEnvelopeNamedIt`, `TestRecordChangeAtEveryStageWritesNoChunk` (the prefix-from-the-request mutation is plan 2's, through T1's second tenant) |
-| `PrivateNetwork=yes` removed from `carlos-dbread@.service` | `TestDBReadUnitsCarryTheSandbox`; on the root runner, `TestRootDBReadWorkerSandbox` |
+| `PrivateNetwork=yes` removed from `carlos-dbread@.service` | `TestDBReadUnitsCarryTheSandbox`; on the root runner, `TestRootDBReadWorkerSandbox` (its "see an interface other than loopback" witness, which needs no socket) |
+| `SelfView` back on `net.Interfaces` | on the root runner, `TestRootDBReadServesThroughTheInstalledUnits` and `TestRootDBReadReadiness` (the probe fails inside the AF_UNIX-only unit) |
+| `flushLocked` increments `seq` before the create is confirmed | `TestResultNamesOnlyConfirmedChunks` |
+| `read` drops `settle` before a retry | `TestRetryWaitsForTheFirstWorkerToStop` |
+| `retry` leaves the old reservation under the read's own key | `TestAdmissionRetryHoldsTheOldReservation` |
+| `consider` drops the Draining check after the jitter | `TestADrainingBoxDoesNotPick` |
+| `consider` drops the cancel read | `TestPassSkipsACancelledRequest` |
+| `attempt` drops `askProbe` after a pre-ready `sendFile` or `readFrame` failure | `TestAWorkerThatNeverStartsAsksForAProbe`; on the root runner, `TestRootDBReadReadiness`'s fail-start case |
| check B removed from `attempt` | `TestRecordChangeAtEveryStageWritesNoChunk` |
| `putCreateOnly` called with a non-empty `ifMatch`, or a `Delete` added after `result.json` | `TestReadNeverDeletesOrReplaces` |
| `elect` returns `false` without reading `picked` back | `TestElectionReadsBackALostAnswer` |
diff --git a/docs/superpowers/plans/2026-10-22-db-region-forward-plan-2-query.md b/docs/superpowers/plans/2026-10-22-db-region-forward-plan-2-query.md
index 7052ec29b..3c71fa9d1 100644
--- a/docs/superpowers/plans/2026-10-22-db-region-forward-plan-2-query.md
+++ b/docs/superpowers/plans/2026-10-22-db-region-forward-plan-2-query.md
@@ -4,11 +4,11 @@
**Goal:** Let a member run `carlos db query` and `carlos db query --at` against an instance whose data lives in another region: the console admits the read, signs and queues it for plan 1's boxes, holds one synchronous HTTP request open while it relays the box's sealed answer, and the CLI buffers, opens and checks the answer before printing it; plus the docs, the copy, and the deployment bucket's `db-reads/` lifecycle rule with the approval plan for applying it.
-**Architecture:** The console's db door keeps every in-region behaviour byte for byte. A cross-region record on a console with `CARLOS_DB_FORWARD=1` and a root key gets an additive 409 marker on the list door (DBF-14 to an old CLI). A new route, `POST /api/cli/apps/{account}/{app}/db/{host}/reads`, runs the db door's whole gate once, checks capability and liveness on complete evidence only, takes admission counts held by the request itself, writes the signed request and its queue entry, bumps the serial, and then streams newline-delimited JSON (admitted, picked, chunks, wait lines, result) inside the same request, reading the box's `result.json` before listing its chunks. Nothing about a read outlives its request except one best-effort `cancel`. The CLI switches to that route on the marker, makes an ephemeral key, and prints nothing until a done result's summary checks out.
+**Architecture:** The console's db door keeps every in-region behaviour byte for byte. A cross-region record on a console with `CARLOS_DB_FORWARD=1` and a root key gets an additive 409 marker on the list door (DBF-14 to an old CLI). A new route, `POST /api/cli/apps/{account}/{app}/db/{host}/reads`, runs the db door's whole gate once, checks capability and liveness on complete evidence only, takes admission counts held by the request itself, writes the audit row, then the signed request and its queue entry, bumps the serial on the request's own context (a request that has ended writes only `cancel`), and then streams newline-delimited JSON (admitted, picked, chunks, wait lines, result) inside the same request, reading the box's `result.json` before listing its chunks. Nothing about a read outlives its request except one best-effort `cancel`. The CLI switches to that route on the marker, makes an ephemeral key, and prints nothing until a done result's summary checks out.
**Tech Stack:** Go 1.26, `net/http` (`http.ResponseController` read and write deadlines and `Flush`), `bufio.Scanner` over NDJSON, plan 1's `internal/dbreads` and `internal/dbreads/dbreadstest`, MinIO in `.amadan/ci.d/30-minio-contract`, OpenTofu module tests.
-**Spec:** `docs/superpowers/specs/2026-10-22-db-region-forward-design.md` (rev 5, accepted by Astra in round 5; Paul approved building it on 2026-10-22, with the rulings in §16). Read §5.2 to §5.7, §7.1, §9, §10, §11 and §15 before starting. Plan 1 (`2026-10-22-db-region-forward-plan-1-box.md`) must have merged: this plan consumes its packages by the names its tasks list.
+**Spec:** `docs/superpowers/specs/2026-10-22-db-region-forward-design.md` (rev 6: rev 5, accepted by Astra in round 5 and approved by Paul for building on 2026-10-22 with the rulings in §16, plus the four clarifications the plan review forced, §5.4, §5.5, §5.7 and §9.3). Read §5.2 to §5.7, §7.1, §9, §10, §11 and §15 before starting. Plan 1 (`2026-10-22-db-region-forward-plan-1-box.md`) must have merged: this plan consumes its packages by the names its tasks list.
**Branch:** `db-region-forward-2-query`, cut from amadan `main` after plan 1 merged.
@@ -36,12 +36,31 @@ Every task's requirements include this section.
The five input classes the spec implies but no gate test names, most likely first. Each has its test in the owning task.
-1. **Ctrl-C mid-stream.** A person interrupts while chunks are arriving. They expect `stopped`, no rows on stdout, and the console to write `cancel` and free their slot. Test: Task 8, `TestForwardedQueryCtrlCPrintsNothing`, and Task 7, `TestDBReadDisconnectWritesCancelAndFreesTheSlot`.
+1. **Ctrl-C mid-stream.** A person interrupts while chunks are arriving. They expect `stopped`, no rows on stdout, and the console to write `cancel` and free their slot. Test: Task 8, `TestForwardedQueryCtrlCPrintsNothing`, and Task 7, `TestDBReadAdmissionBounds`.
2. **A full-size chunk line.** A 256 KiB chunk sealed and base64-encoded is a line of about 350 KiB, past `bufio.Scanner`'s 64 KiB default. A person expects a large result to arrive. Test: Task 8, `TestForwardedQueryReadsAFullSizeChunkLine`.
3. **A `picked` whose deadline is absurd.** Any box can write `picked` (spec §5.6), so its deadline may be far in the future. A person expects the console to give up 22 minutes after `issued_at` at the latest. Test: Task 7, `TestDBReadRequestBoundClampsAForgedDeadline`.
4. **A `result.json` that is not a result.** Garbage, or a state outside the vocabulary. A person expects the console to keep waiting until its deadline and the CLI to say DBF-12, never a panic or a relayed garbage line. Test: Task 7, `TestDBReadIgnoresAMalformedResult`.
5. **A home-region record seen from a regional console.** Its record region is empty. A person expects today's residency sentence, never a forward marker naming region "". Test: Task 5, `TestListDoorKeepsMainsSentenceForTheHomeRegion`.
+## Plan review, round 1
+
+Astra's findings (`docs/superpowers/specs/CODEX-FINDINGS-DB-REGION-FORWARD-PLANS.md`) for this plan, and where each landed:
+
+- **2** (`hookStore` collides with `internal/console/temphook_test.go:16`): renamed `dbReadHookStore`.
+- **6** (fixtures create over an existing object): `putFixtureObject` reads the etag and replaces with it.
+- **7** (Task 6's admission test needs Task 7's stream): the test is Task 7's, `TestDBReadAdmissionBounds`.
+- **11** (writes after the request ended): the audit row is written first, before any box can see the read; after the queue entry, a request that has ended writes `cancel` and returns, with no serial bump; the bump runs on the request's context (`announceDBRead`). Spec §9.3 says so in rev 6. Partly declined, below.
+- **12** (bounds do not hold blocking work): phase deadline contexts, every store call through `dbfCall` (5 s, wait lines while it blocks), the bound checked before the result line, write deadlines clamped to the bound plus 10 s. Tests: `TestDBReadStalledStoreKeepsTheStreamAndItsBound`, `TestDBReadSlowRelayEndsAtTheBound`.
+- **13** (a slow body outlives the pre-header bound): the read deadline is `preHeader` until admission, then the read's bound. Test: `TestDBReadStalledBodyIsCutAtThePreHeaderBound`.
+- **18** (T15): both of a person's slots freed before and after pickup, the seventeenth read across nine people, and a restart under open reads.
+- **19** (T16 passes without late output): a gate holds the real box at publication until the console's request has ended, by disconnect and by the bound.
+- **20** (result-before-list): its own operation-order check and its own mutation row.
+- **21** (the console half): the stream's last log line, `relayed N chunks (B bytes) in T, ended <why>`.
+
+### Declined plan findings
+
+- **11, in part: making the audit insert itself cancellation-aware.** `RecordActivityContext` takes a context, but its claim plumbing ends in `claimRawRecordActivity`'s context-free `Exec`, shared by every audited route (`internal/console/store.go:2588`, `2668`). Changing that is a change to the claim SQL layer and its census, for one local SQLite insert. The order makes it unnecessary: the row is written while the request is open (checked just before) and before any object a box can see, so a request that ends during the insert leaves the row of an admission and no read. The serial bump, the one bucket write, is request-bound.
+
## Census impact
- **Plan 14 root-read census:** one overlay, `plan14-root-reads-db-region-forward-2.tsv`, `rev3=PROPOSED-db-region-forward-2`:
@@ -357,7 +376,7 @@ git commit -m "db forward: the reads route's lines and the list door's additive
**Interfaces:**
- Consumes: `release.RootSigner`, `newDBStreamLimits` (`apidb.go`), `dbreads.OpenPerConsole`, `dbreads.OpenPerPerson`.
-- Produces: `console.NewHTTPServer(h http.Handler) *http.Server`; `Config.DBForward bool`; `(*Server).dbForwardSigner() (release.Signer, bool)`; `Server.dbReads *dbStreamLimits`; `Server.dbf dbForwardTimes` with fields `poll, wait, stall, preHeader, pickup, grace, bound time.Duration`.
+- Produces: `console.NewHTTPServer(h http.Handler) *http.Server`; `Config.DBForward bool`; `(*Server).dbForwardSigner() (release.Signer, bool)`; `Server.dbReads *dbStreamLimits`; `Server.dbf dbForwardTimes` with fields `poll, wait, stall, preHeader, pickup, grace, bound, call time.Duration`.
- [ ] **Step 1: Write the failing tests**
@@ -513,12 +532,14 @@ type dbForwardTimes struct {
pickup time.Duration // picked must appear this long after issued_at
grace time.Duration // after the owner's deadline
bound time.Duration // the request's own bound after issued_at
+ call time.Duration // one store call inside the stream (plan review finding 12)
}
func defaultDBForwardTimes() dbForwardTimes {
return dbForwardTimes{
poll: time.Second, wait: 15 * time.Second, stall: 30 * time.Second, preHeader: 20 * time.Second,
pickup: dbreads.ConsoleWaitsPicked, grace: dbreads.OwnerGrace, bound: dbreads.RequestBound,
+ call: 5 * time.Second,
}
}
@@ -568,23 +589,34 @@ git commit -m "console: CARLOS_DB_FORWARD, the root signer check, the read limit
Append to `internal/console/apidbforward_test.go`:
```go
+// putFixtureObject creates key, or replaces it under its current etag: a
+// fixture may republish a box's files, as a box does every pass. Protocol
+// writes stay create-only; this helper is for fixtures only (plan review
+// finding 6: DirStore.Put refuses a create over an existing key).
+func putFixtureObject(t *testing.T, s release.Store, key string, body []byte) {
+ t.Helper()
+ etag := ""
+ if _, cur, err := s.Get(context.Background(), key); err == nil {
+ etag = cur
+ }
+ if _, err := s.Put(context.Background(), key, body, etag); err != nil {
+ t.Fatal(err)
+ }
+}
+
// putEdgesFile writes control/edges/<box>.json as a box publishes it.
func putEdgesFile(t *testing.T, s release.Store, box, placement string, caps []string, draining bool) {
t.Helper()
body, _ := json.Marshal(map[string]any{"v": 1, "box": box, "eip": "203.0.113.10", "placement": placement,
"capabilities": caps, "draining": draining, "hosts": map[string]any{}})
- if _, err := s.Put(context.Background(), "control/edges/"+box+".json", body, ""); err != nil {
- t.Fatal(err)
- }
+ putFixtureObject(t, s, "control/edges/"+box+".json", body)
}
// putBoxStatus writes status/boxes/<box>.json generated at at.
func putBoxStatus(t *testing.T, s release.Store, box string, at time.Time) {
t.Helper()
body, _ := json.Marshal(map[string]any{"generated_at": at.UTC(), "label": box})
- if _, err := s.Put(context.Background(), "status/boxes/"+box+".json", body, ""); err != nil {
- t.Fatal(err)
- }
+ putFixtureObject(t, s, "status/boxes/"+box+".json", body)
}
// T14 (codex 10): only complete evidence, both flags, refuses up front.
@@ -899,8 +931,8 @@ Waits for Task 1 (DBF-9, DBF-10, DBF-21, DBF-22 approved).
- Modify: `approved_db_forward_copy_test.go` (`dbForwardCopyBuilt` gains DBF-9, DBF-10, DBF-21, DBF-22)
**Interfaces:**
-- Consumes: `dbForwardSigner`, `forwardable`, `dbForwardEvidence`, `s.dbReads`, `s.dbf` (Tasks 3 to 5); `resolveDBSource`, `dbClassRefusal`, `dbResidencyRefusal`, `apiAccountApp`, `principalFrom`, `s.cli.db.allow`, `hashToken`, `bearer`, `readJSONLimit`, `jsonError`, `s.bumpSerial`, `s.auditActivity`, `dbAuditAction` (existing console code); `dbreads.Sign`, `MintID`, `RequestKey`, `QueueKey`, `ParseReplyKey`, `ReadBody`; `dbquery.CheckSQL`; `hostclaim.Incarnation`.
-- Produces: `(*Server).handleAPIDBRead(w, r)`; `(*Server).admitDBRead(w, r) (dbReadAdmitted, string, bool)`; `type dbReadAdmitted struct{ id, region string; issued time.Time }`; `dbForwardAuditDetail(region, sqlText string, at time.Time) string`; `dbfTooMany`, `dbfMoving`, `dbfAudit`, `dbfAuditAt`; the stream half (`streamDBRead`) is Task 7's; test helpers `(*dbfFixture).readURL()`, `(*dbfFixture).openRead(t, body dbreads.ReadBody) *http.Response`, `readLines(t, body io.Reader) []dbreads.Line`.
+- Consumes: `dbForwardSigner`, `forwardable`, `dbForwardEvidence`, `s.dbReads`, `s.dbf` (Tasks 3 to 5); `resolveDBSource`, `dbClassRefusal`, `dbResidencyRefusal`, `apiAccountApp`, `principalFrom`, `s.cli.db.allow`, `hashToken`, `bearer`, `readJSONLimit`, `jsonError`, `s.store.RecordActivityContext`, `release.BumpSerial`, `release.SerialKey`, `dbAuditAction` (existing console code); `dbreads.Sign`, `MintID`, `RequestKey`, `QueueKey`, `ParseReplyKey`, `ReadBody`; `dbquery.CheckSQL`; `hostclaim.Incarnation`.
+- Produces: `(*Server).handleAPIDBRead(w, r)`; `(*Server).admitDBRead(w, r) (dbReadAdmitted, string, bool)`; `type dbReadAdmitted struct{ id, region string; issued time.Time }`; `dbForwardAuditDetail(region, sqlText string, at time.Time) string`; `(*Server).cancelDBRead(id)`; `(*Server).announceDBRead(ctx)`; `dbfTooMany`, `dbfMoving`, `dbfAudit`, `dbfAuditAt`; the stream half (`streamDBRead`) is Task 7's; test helpers `(*dbfFixture).readURL(host)`, `queryBody(t, sql)`, `(*dbfFixture).openRead(t, host, bearer, body) *http.Response`, `readLines(t, body io.Reader) []dbreads.Line`, `refusal(t, res)`, `dbfPutHook`, `dbReadAudited(t, st)`.
- [ ] **Step 1: Write the failing tests**
@@ -1054,35 +1086,132 @@ func TestDBReadQueuesASignedRequestAndAudits(t *testing.T) {
}
}
-// DBF-9: two open reads per person, sixteen per console, held by the open
-// requests (J5).
-func TestDBReadAdmissionBounds(t *testing.T) {
- fx := newDBFFixture(t, true)
- fx.srv.dbf.pickup = 5 * time.Second
- open := func() *http.Response {
+// dbfPutHook runs hook after each create of a key with suffix, once the
+// create has landed.
+type dbfPutHook struct {
+ release.Store
+ suffix string
+ hook func()
+}
+
+func (h *dbfPutHook) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
+ etag, err := h.Store.Put(ctx, key, body, ifMatch)
+ if strings.HasSuffix(key, h.suffix) || (h.suffix == "queue" && strings.HasPrefix(key, dbreads.Prefix+"queue/")) {
+ h.hook()
+ }
+ return etag, err
+}
+
+// dbReadAudited is safe from a handler's goroutine: it never calls t.Fatal.
+func dbReadAudited(t *testing.T, st *Store) bool {
+ t.Helper()
+ evs, err := st.Events(OpsAccountID, "shop", 10)
+ if err != nil {
+ t.Error(err)
+ return false
+ }
+ for _, e := range evs {
+ if e.Action == "db-read" && e.Target == dbfHost {
+ return true
+ }
+ }
+ return false
+}
+
+// Plan review finding 11 (J1, spec §9.3 rev 6): the audit row is written
+// before any box can see the read, and a request that ends once the queue
+// entry is down writes the cancel and nothing else: no serial bump.
+// Mutations: move the audit after the creates (red: no row when request.json
+// lands); call s.bumpSerial instead of announceDBRead, or drop the ctx check
+// after the queue create (red: the serial moves); drop the cancel on that
+// path (red: no cancel object).
+func TestDBReadEndedDuringAdmissionWritesOnlyTheCancel(t *testing.T) {
+ t.Run("the audit row is down before the request", func(t *testing.T) {
+ fx := newDBFFixture(t, true)
+ auditedFirst := false
+ fx.srv.cfg.Releases = &dbfPutHook{Store: fx.rel, suffix: "/request.json", hook: func() {
+ auditedFirst = dbReadAudited(t, fx.st)
+ }}
+ fx.srv.dbf.pickup = 100 * time.Millisecond
body, _ := queryBody(t, "SELECT 1")
res := fx.openRead(t, dbfHost, fx.member, body)
- if res.StatusCode == http.StatusOK {
- bufio.NewReader(res.Body).ReadBytes('\n') // admitted
+ io.Copy(io.Discard, res.Body)
+ res.Body.Close()
+ if !auditedFirst {
+ t.Fatal("request.json was written before the audit row")
}
- return res
+ })
+ t.Run("ended after the queue entry", func(t *testing.T) {
+ fx := newDBFFixture(t, true)
+ _, serialBefore, _ := fx.rel.Get(context.Background(), release.SerialKey)
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ fx.srv.cfg.Releases = &dbfPutHook{Store: fx.rel, suffix: "queue", hook: cancel}
+ body, _ := queryBody(t, "SELECT 2")
+ raw, _ := json.Marshal(body)
+ req := httptest.NewRequest(http.MethodPost, fx.readURL(dbfHost), bytes.NewReader(raw)).WithContext(ctx)
+ req.Header.Set("Content-Type", "application/json")
+ req.Header.Set("Authorization", "Bearer "+fx.member)
+ rec := httptest.NewRecorder()
+ fx.srv.ServeHTTP(rec, req)
+ // The ended path writes no response (nobody is there to read it), and
+ // a recorder that was never written reports 200: read the body.
+ if strings.Contains(rec.Body.String(), "admitted") {
+ t.Fatalf("an ended request was admitted: %s", rec.Body)
+ }
+ if _, serialAfter, _ := fx.rel.Get(context.Background(), release.SerialKey); serialAfter != serialBefore {
+ t.Fatal("the serial was bumped after the request ended")
+ }
+ keys, _ := fx.rel.List(context.Background(), dbreads.Prefix)
+ var queued, cancelled int
+ for _, k := range keys {
+ switch {
+ case strings.HasPrefix(k, dbreads.Prefix+"queue/"):
+ queued++
+ case strings.HasSuffix(k, "/cancel"):
+ cancelled++
+ }
+ }
+ if queued != 1 || cancelled != 1 || !dbReadAudited(t, fx.st) {
+ t.Fatalf("queue entries %d, cancels %d, audited %v in %v", queued, cancelled, dbReadAudited(t, fx.st), keys)
+ }
+ })
+}
+
+// Plan review finding 13: a body delivered slowly is cut at s.dbf.preHeader,
+// well inside backhaul's 30 s header wait, and admits nothing.
+// Mutation: drop the preHeader read deadline, keeping only the bound's (red:
+// the connection is still open at 3 s).
+func TestDBReadStalledBodyIsCutAtThePreHeaderBound(t *testing.T) {
+ fx := newDBFFixture(t, true)
+ fx.srv.dbf.preHeader = 300 * time.Millisecond
+ conn, err := net.Dial("tcp", strings.TrimPrefix(fx.ts.URL, "http://"))
+ if err != nil {
+ t.Fatal(err)
}
- a, b := open(), open()
- defer a.Body.Close()
- defer b.Body.Close()
- c := open()
- status, body := refusal(t, c)
- if status != http.StatusTooManyRequests || c.Header.Get("Retry-After") == "" || !strings.Contains(body, dbfTooMany) {
- t.Fatalf("a third read: %d %s", status, body)
+ defer conn.Close()
+ path := strings.TrimPrefix(fx.readURL(dbfHost), fx.ts.URL)
+ fmt.Fprintf(conn, "POST %s HTTP/1.1\r\nHost: console\r\nAuthorization: Bearer %s\r\n"+
+ "Content-Type: application/json\r\nContent-Length: 4096\r\n\r\n{\"op\":", path, fx.member)
+ conn.SetReadDeadline(time.Now().Add(3 * time.Second))
+ start := time.Now()
+ _, err = io.ReadAll(conn) // a refusal, then the close; or the close alone
+ if ne, ok := err.(net.Error); ok && ne.Timeout() {
+ t.Fatalf("the stalled body still held the request after %s", time.Since(start))
+ }
+ if keys, _ := fx.rel.List(context.Background(), dbreads.Prefix); len(keys) != 0 {
+ t.Fatalf("a stalled body wrote %v", keys)
}
}
```
-Add `"bufio"`, `"bytes"`, `"crypto/ecdh"`, `"io"`, `"net/http"` and the `dbquery` import.
+Add `"bufio"`, `"bytes"`, `"crypto/ecdh"`, `"io"`, `"net"`, `"net/http"`, `"net/http/httptest"` and the `dbquery` import.
+
+The admission bounds test (DBF-9, T15) is Task 7's: it needs the stream to hold its requests open (plan review finding 7).
- [ ] **Step 2: Run them to verify they fail**
-Run: `go test ./internal/console/ -run 'TestDBReadRoute|TestDBReadQueues|TestDBReadAdmission' -count=1`
+Run: `go test ./internal/console/ -run 'TestDBReadRoute|TestDBReadQueues|TestDBReadEndedDuringAdmission|TestDBReadStalledBody' -count=1`
Expected: FAIL: the route answers 404 for everything.
- [ ] **Step 3: Write the admission half**
@@ -1124,20 +1253,49 @@ type dbReadAdmitted struct {
// nowhere else (J5).
func (s *Server) handleAPIDBRead(w http.ResponseWriter, r *http.Request) {
rc := http.NewResponseController(w)
- // The server's ReadTimeout would end this connection under a read that
- // lasts longer; the read's own bound holds it instead, as the import
- // door's read deadline does (apiinstanceimport.go).
- if err := rc.SetReadDeadline(time.Now().Add(s.dbf.bound + time.Minute)); err != nil && !errors.Is(err, http.ErrNotSupported) {
- log.Printf("console: db read: extend the read deadline: %v", err)
+ // Everything before the 200, the body included, gets s.dbf.preHeader:
+ // a body delivered slowly must not hold the request past backhaul's
+ // 30 s header wait (plan review finding 13).
+ if err := rc.SetReadDeadline(time.Now().Add(s.dbf.preHeader)); err != nil && !errors.Is(err, http.ErrNotSupported) {
+ log.Printf("console: db read: set the admission read deadline: %v", err)
}
a, actor, ok := s.admitDBRead(w, r)
if !ok {
return
}
defer s.dbReads.release(actor)
+ // Then the server's ReadTimeout would end this connection under a read
+ // that lasts longer; the read's own bound holds it instead, as the
+ // import door's read deadline does (apiinstanceimport.go).
+ if err := rc.SetReadDeadline(time.Now().Add(s.dbf.bound + time.Minute)); err != nil && !errors.Is(err, http.ErrNotSupported) {
+ log.Printf("console: db read: extend the read deadline: %v", err)
+ }
s.streamDBRead(w, r, rc, a)
}
+// cancelDBRead is the one write J1 allows after a read's request ends
+// without a result: best effort, so the owner stops early and a box that
+// has not picked it skips it.
+func (s *Server) cancelDBRead(id string) {
+ ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+ defer cancel()
+ if _, err := s.cfg.Releases.Put(ctx, dbreads.CancelKey(id), nil, ""); err != nil && !errors.Is(err, release.ErrCASConflict) {
+ log.Printf("console: db read %s: cancel: %v", id, err)
+ }
+}
+
+// announceDBRead bumps the serial on the request's own context, so a
+// request that has already ended announces nothing (J1; plan review finding
+// 11). bumpSerial detaches on purpose, for callers whose writes outlive
+// them; this read does not outlive its request.
+func (s *Server) announceDBRead(ctx context.Context) {
+ ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
+ defer cancel()
+ if err := release.BumpSerial(ctx, s.cfg.Releases, time.Now()); err != nil && ctx.Err() == nil {
+ log.Printf("console: db read: serial bump: %v (boxes list every 30 s anyway)", err)
+ }
+}
+
// admitDBRead is spec §5.5 step 3, all of it bounded by s.dbf.preHeader so
// the 200 leaves well inside backhaul's 30 s header wait: dbGate's order
// (person, membership with the byte-identical 404, the db budget,
@@ -1277,15 +1435,35 @@ func (s *Server) admitDBRead(w http.ResponseWriter, r *http.Request) (dbReadAdmi
http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
return none, "", false
}
+ // The audit row comes first, while the request is open and before any box
+ // can see the read: a read a box runs always has its row, and a request
+ // that ends here leaves the row of an admission and nothing else (spec
+ // §9.3 rev 6; plan review finding 11). A row the store cannot write
+ // refuses the read rather than run it unaudited.
+ if err := ctx.Err(); err != nil {
+ return none, "", false
+ }
+ if err := s.store.RecordActivityContext(ctx, acct.ID, app, actor, Activity{Action: dbAuditAction,
+ TargetType: "instance", Target: host, Detail: dbForwardAuditDetail(rec.Region, body.SQL, at)}, time.Now()); err != nil {
+ return fail("audit", err)
+ }
if _, err := s.cfg.Releases.Put(ctx, dbreads.RequestKey(id), wire, ""); err != nil {
return fail("write the request", err)
}
if _, err := s.cfg.Releases.Put(ctx, dbreads.QueueKey(rec.Region, issued, id), nil, ""); err != nil {
+ // The queue entry may have landed on an ambiguous answer: cancel, so
+ // a box that lists it skips it (J1's one write).
+ s.cancelDBRead(id)
return fail("write the queue entry", err)
}
- s.bumpSerial(ctx)
- s.auditActivity(acct, app, actor, Activity{Action: dbAuditAction, TargetType: "instance", Target: host,
- Detail: dbForwardAuditDetail(rec.Region, body.SQL, at)})
+ // J1: once the request has ended, nothing about the read is written but
+ // the cancel. The serial bump runs on the request's context, not
+ // bumpSerial's detached one.
+ if err := ctx.Err(); err != nil {
+ s.cancelDBRead(id)
+ return none, "", false
+ }
+ s.announceDBRead(ctx)
admitted = true
return dbReadAdmitted{id: id, region: rec.Region, issued: issued}, actor, true
}
@@ -1316,7 +1494,7 @@ Add `"DBF-9", "DBF-10", "DBF-21", "DBF-22"` to `dbForwardCopyBuilt`.
- [ ] **Step 4: Run the tests**
-Run: `go test ./internal/console/ -run 'TestDBReadRoute|TestDBReadQueues|TestDBReadAdmission|TestClaim.*Routes' -count=1 && go test . -run 'TestDBForward' -count=1`
+Run: `go test ./internal/console/ -run 'TestDBReadRoute|TestDBReadQueues|TestDBReadEndedDuringAdmission|TestDBReadStalledBody|TestClaim.*Routes' -count=1 && go test . -run 'TestDBForward' -count=1`
Expected: PASS. (`TestClaim.*Routes` is the claim-routes census; list the exact name with `grep -n 'claim-routes.txt' internal/console/*_test.go` if the pattern matches nothing.)
- [ ] **Step 5: Commit**
@@ -1336,7 +1514,8 @@ git commit -m "console: the reads route's gate and admission (DBF-9, DBF-10, DBF
**Interfaces:**
- Consumes: `dbReadAdmitted`, `s.dbf` (Tasks 3, 6); `dbreads.Line`, `Picked`, `Result`, `PickedKey`, `ResultKey`, `OutPrefix`, `ParseChunkKey`, `ChunkKey`, `CancelKey`, the wait and ended words; `dbreadstest.Start`, `Box.Agent` (plan 1).
-- Produces: `(*Server).streamDBRead(w, r, rc, a)`; `type dbLineWriter`; `(*Server).readDBResult(ctx, id) *dbreads.Result`; `(*Server).cancelDBRead(id)`; test helper `runBox(t, box)` and `(*dbfFixture).startBox(t) *dbreadstest.Box`.
+- Consumes also: `(*Server).cancelDBRead` (Task 6), `s.dbf.call` (Task 3).
+- Produces: `(*Server).streamDBRead(w, r, rc, a)`; `type dbLineWriter` (with `until`), `(*dbLineWriter).tick(ctx, phase, poll, every) error`; `errDBClientGone`; `dbfEndSlack`; `type dbStream`; `dbfCall[T](st, ctx, phase, op) (T, error)`; `(*Server).readDBResult(ctx, id) *dbreads.Result`; test helpers `runBox(t, box)`, `(*dbfFixture).startBox(t, dir) *dbreadstest.Box`, `dbfOpen`, `(*dbfFixture).openHeld`, `(*dbfFixture).admitWithin`, `refusedTooMany`, `slowStore`, `scriptedBox`, `dbReadHookStore`, `strictStore`.
- [ ] **Step 1: Write the failing tests**
@@ -1460,45 +1639,297 @@ func TestDBReadLinesAreFlushedAsWritten(t *testing.T) {
}
}
-// T15 and Review Focus 1 (codex 17): reads whose CLIs disconnect, before
-// and after pickup, free their person's slots when their handlers return,
-// and each writes cancel.
-func TestDBReadDisconnectWritesCancelAndFreesTheSlot(t *testing.T) {
- fx := newDBFFixture(t, true)
- fx.srv.dbf.poll, fx.srv.dbf.wait, fx.srv.dbf.stall = 10*time.Millisecond, 50*time.Millisecond, 200*time.Millisecond
- fx.srv.dbf.pickup = time.Minute
- var ids []string
- for i := 0; i < 2; i++ {
- body, _ := queryBody(t, "SELECT 1")
- res := fx.openRead(t, dbfHost, fx.member, body)
- line, _ := bufio.NewReader(res.Body).ReadBytes('\n')
+// dbfOpen is one open read: its response, a reader over its lines, and its
+// id ("" when it was refused).
+type dbfOpen struct {
+ res *http.Response
+ br *bufio.Reader
+ id string
+}
+
+func (fx *dbfFixture) openHeld(t *testing.T, base, bearer string) dbfOpen {
+ t.Helper()
+ body, _ := queryBody(t, "SELECT 1")
+ raw, _ := json.Marshal(body)
+ req, _ := http.NewRequest(http.MethodPost, base+"/api/cli/apps/"+release.OpsAccount+"/shop/db/"+dbfHost+"/reads", bytes.NewReader(raw))
+ req.Header.Set("Content-Type", "application/json")
+ req.Header.Set("Authorization", "Bearer "+bearer)
+ res, err := http.DefaultClient.Do(req)
+ if err != nil {
+ t.Fatal(err)
+ }
+ o := dbfOpen{res: res, br: bufio.NewReader(res.Body)}
+ if res.StatusCode == http.StatusOK {
+ line, _ := o.br.ReadBytes('\n')
var l dbreads.Line
json.Unmarshal(line, &l)
- ids = append(ids, l.Admitted)
- if i == 1 { // the second is picked before its CLI goes
- p, _ := json.Marshal(dbreads.Picked{Box: "uk-1", Nonce: "n", Deadline: time.Now().Add(time.Minute)})
- fx.rel.Put(context.Background(), dbreads.PickedKey(l.Admitted), p, "")
+ o.id = l.Admitted
+ }
+ return o
+}
+
+// untilPicked reads lines until the console has written picked, so the
+// handler is in the owner's phase.
+func (o dbfOpen) untilPicked(t *testing.T) {
+ t.Helper()
+ for {
+ line, err := o.br.ReadBytes('\n')
+ if err != nil {
+ t.Fatalf("%s ended before picked: %v", o.id, err)
+ }
+ var l dbreads.Line
+ json.Unmarshal(line, &l)
+ if l.Picked != "" {
+ return
}
- res.Body.Close()
}
+}
+
+func refusedTooMany(t *testing.T, o dbfOpen, what string) {
+ t.Helper()
+ status, body := refusal(t, o.res)
+ if status != http.StatusTooManyRequests || o.res.Header.Get("Retry-After") == "" || !strings.Contains(body, dbfTooMany) {
+ t.Fatalf("%s: %d %s", what, status, body)
+ }
+}
+
+// admitWithin retries until a read is admitted: a handler frees its slot when
+// it notices its client went, which takes up to a poll.
+func (fx *dbfFixture) admitWithin(t *testing.T, base, bearer string) dbfOpen {
+ t.Helper()
deadline := time.Now().Add(5 * time.Second)
for {
- body, _ := queryBody(t, "SELECT 1")
- res := fx.openRead(t, dbfHost, fx.member, body)
- ok := res.StatusCode == http.StatusOK
- res.Body.Close()
- if ok {
- break
+ o := fx.openHeld(t, base, bearer)
+ if o.id != "" {
+ return o
}
+ o.res.Body.Close()
if time.Now().After(deadline) {
- t.Fatal("the disconnected reads still hold their slots")
+ t.Fatal("a disconnected read still holds its slot")
+ }
+ time.Sleep(20 * time.Millisecond)
+ }
+}
+
+// DBF-9 and T15 (J5; plan review findings 7 and 18): two open reads per
+// person and sixteen per console, held by the open requests. Both of a
+// person's slots come back when both CLIs leave before pickup, and again
+// when both leave after it; the seventeenth read across nine people is
+// refused; each disconnected read writes cancel; and a console restarted
+// under open reads recovers nothing and starts with empty counts.
+// Mutations: drop handleAPIDBRead's defer release (red: the second pair is
+// never admitted); release at the end of admitDBRead (red: the third read of
+// a pair is admitted); skip cancelDBRead on a client that went (red: no
+// cancel); a startup pass that writes under db-reads/ (red: the listing
+// changes across NewServer).
+func TestDBReadAdmissionBounds(t *testing.T) {
+ fx := newDBFFixture(t, true)
+ fx.srv.dbf.poll, fx.srv.dbf.wait, fx.srv.dbf.pickup = 10*time.Millisecond, 50*time.Millisecond, time.Minute
+ base := fx.ts.URL
+ var left []string
+
+ // Before pickup.
+ a, b := fx.openHeld(t, base, fx.member), fx.openHeld(t, base, fx.member)
+ if a.id == "" || b.id == "" {
+ t.Fatal("the first two reads were refused")
+ }
+ refusedTooMany(t, fx.openHeld(t, base, fx.member), "a third read")
+ a.res.Body.Close()
+ b.res.Body.Close()
+ left = append(left, a.id, b.id)
+
+ // Both slots come back; the pair is picked, then both CLIs leave.
+ c, d := fx.admitWithin(t, base, fx.member), fx.admitWithin(t, base, fx.member)
+ refusedTooMany(t, fx.openHeld(t, base, fx.member), "a third read after the first pair left")
+ for _, o := range []dbfOpen{c, d} {
+ scriptFor(t, fx.rel, o.id).picked(time.Now().Add(time.Minute))
+ o.untilPicked(t)
+ o.res.Body.Close()
+ left = append(left, o.id)
+ }
+ e, f := fx.admitWithin(t, base, fx.member), fx.admitWithin(t, base, fx.member)
+ refusedTooMany(t, fx.openHeld(t, base, fx.member), "a third read after the picked pair left")
+ e.res.Body.Close()
+ f.res.Body.Close()
+ left = append(left, e.id, f.id)
+ for _, id := range left {
+ deadline := time.Now().Add(5 * time.Second)
+ for {
+ if _, _, err := fx.rel.Get(context.Background(), dbreads.CancelKey(id)); err == nil {
+ break
+ }
+ if time.Now().After(deadline) {
+ t.Fatalf("%s: no cancel after its CLI left", id)
+ }
+ time.Sleep(20 * time.Millisecond)
}
- time.Sleep(50 * time.Millisecond)
}
- for _, id := range ids {
- if _, _, err := fx.rel.Get(context.Background(), dbreads.CancelKey(id)); err != nil {
- t.Errorf("%s: no cancel after its CLI left: %v", id, err)
+
+ // Sixteen per console, across people.
+ var held []dbfOpen
+ people := dbreads.OpenPerConsole/dbreads.OpenPerPerson + 1
+ for i := 0; i < people; i++ {
+ tok := fx.member
+ if i > 0 {
+ addr := fmt.Sprintf("member%d@example.com", i)
+ if err := fx.st.AddMember(OpsAccountID, addr, RoleMember, time.Now()); err != nil {
+ t.Fatal(err)
+ }
+ tok = cliToken(t, fx.consoleFixture, addr, fmt.Sprintf("code-dbf-%d", i))
}
+ for j := 0; j < dbreads.OpenPerPerson; j++ {
+ if len(held) == dbreads.OpenPerConsole {
+ refusedTooMany(t, fx.openHeld(t, base, tok), "the seventeenth read on the console")
+ continue
+ }
+ o := fx.admitWithin(t, base, tok)
+ held = append(held, o)
+ }
+ }
+ if len(held) != dbreads.OpenPerConsole {
+ t.Fatalf("held %d reads", len(held))
+ }
+
+ // A restart under open reads: nothing to recover, nothing written, and
+ // the new console's counts start empty.
+ before, _ := fx.rel.List(context.Background(), dbreads.Prefix)
+ srv2, err := NewServer(fx.srv.cfg, fx.st)
+ if err != nil {
+ t.Fatal(err)
+ }
+ srv2.dbf = fx.srv.dbf
+ ts2 := httptest.NewServer(srv2)
+ defer ts2.Close()
+ after, _ := fx.rel.List(context.Background(), dbreads.Prefix)
+ if !slices.Equal(before, after) {
+ t.Fatalf("starting a console wrote under db-reads/: %v", after)
+ }
+ g := fx.openHeld(t, ts2.URL, fx.member)
+ if g.id == "" {
+ status, body := refusal(t, g.res)
+ t.Fatalf("the restarted console refused a read: %d %s", status, body)
+ }
+ g.res.Body.Close()
+ for _, o := range held {
+ o.res.Body.Close()
+ }
+}
+
+// slowStore holds every call whose key contains match until free closes,
+// ignoring its context: the worst store, for plan review finding 12.
+type slowStore struct {
+ release.Store
+ match string
+ delay time.Duration // 0: until free closes
+ free chan struct{}
+}
+
+func (s *slowStore) hold(key string) {
+ if !strings.Contains(key, s.match) {
+ return
+ }
+ if s.delay > 0 {
+ time.Sleep(s.delay)
+ return
+ }
+ <-s.free
+}
+
+func (s *slowStore) Get(ctx context.Context, key string) ([]byte, string, error) {
+ s.hold(key)
+ return s.Store.Get(ctx, key)
+}
+
+func (s *slowStore) List(ctx context.Context, prefix string) ([]string, error) {
+ s.hold(prefix)
+ return s.Store.List(ctx, prefix)
+}
+
+// Plan review finding 12: a store that stops answering, in either phase,
+// neither silences the stream nor holds it past the phase's deadline.
+// Mutations: call the store on r.Context() without dbfCall (red: no wait
+// line and no end until free closes); drop the pickup deadline context (red:
+// same); drop the owner's deadline context (red: same in the owner case).
+func TestDBReadStalledStoreKeepsTheStreamAndItsBound(t *testing.T) {
+ for _, phase := range []string{"pickup", "owner"} {
+ t.Run(phase, func(t *testing.T) {
+ fx := newDBFFixture(t, true)
+ fx.srv.dbf.poll, fx.srv.dbf.wait, fx.srv.dbf.call = 10*time.Millisecond, 100*time.Millisecond, time.Minute
+ fx.srv.dbf.pickup, fx.srv.dbf.grace = 1500*time.Millisecond, 0
+ ss := &slowStore{Store: fx.rel, match: "/picked", free: make(chan struct{})}
+ want := dbreads.EndedNotPicked
+ if phase == "owner" {
+ ss.match, want = "/result.json", dbreads.EndedNoResult
+ }
+ defer close(ss.free)
+ fx.srv.cfg.Releases = ss
+ o := fx.openHeld(t, fx.ts.URL, fx.member)
+ defer o.res.Body.Close()
+ if phase == "owner" {
+ scriptFor(t, fx.rel, o.id).picked(time.Now().Add(time.Second))
+ }
+ start, prev := time.Now(), time.Now()
+ var last dbreads.Line
+ for {
+ line, err := o.br.ReadBytes('\n')
+ if err != nil {
+ break
+ }
+ if gap := time.Since(prev); gap > 500*time.Millisecond {
+ t.Fatalf("a %s gap with the store stalled", gap)
+ }
+ prev = time.Now()
+ json.Unmarshal(line, &last)
+ if time.Since(start) > 5*time.Second {
+ t.Fatal("the stalled store held the request past its deadline")
+ }
+ }
+ if last.Ended != want {
+ t.Fatalf("last line %+v, want %s", last, want)
+ }
+ })
+ }
+}
+
+// Plan review finding 12: a relay slower than the time left ends at the
+// request's bound, without the rest of the chunks and without the result,
+// though both are in the bucket. issued_at is truncated to the second, so
+// the bound is 2 to 3 s after the request; relaying all forty chunks takes 6.
+// Mutation: relay without the owner's deadline, the chunk Gets on
+// r.Context() and no ownCtx check after them (red: all forty are relayed
+// and the request lasts 6 s).
+// Not witnessed alone: the ownCtx check before the result line; dbfCall
+// already cuts every call after the deadline, so that check guards only a
+// line write that outlasts it, which no in-process client can force.
+func TestDBReadSlowRelayEndsAtTheBound(t *testing.T) {
+ fx := newDBFFixture(t, true)
+ fx.srv.dbf.poll, fx.srv.dbf.bound, fx.srv.dbf.grace = 10*time.Millisecond, 3*time.Second, 0
+ fx.srv.cfg.Releases = &slowStore{Store: fx.rel, match: "/out/", delay: 150 * time.Millisecond}
+ start := time.Now()
+ o := fx.openHeld(t, fx.ts.URL, fx.member)
+ defer o.res.Body.Close()
+ box := scriptFor(t, fx.rel, o.id)
+ const n = 40
+ for seq := int64(1); seq <= n; seq++ {
+ box.chunk(seq, seq == n, "row\n")
+ }
+ box.result(n, strings.Repeat("row\n", n))
+ box.picked(time.Now().Add(time.Minute))
+ lines := readLines(t, o.br)
+ chunks := 0
+ for _, l := range lines {
+ if l.Result != nil {
+ t.Fatalf("a result relayed past the bound: %+v", l.Result)
+ }
+ if l.Chunk != 0 {
+ chunks++
+ }
+ }
+ if took := time.Since(start); took > 4500*time.Millisecond || chunks >= n {
+ t.Fatalf("relayed %d chunks in %s", chunks, took)
+ }
+ if last := lines[len(lines)-1]; last.Ended != dbreads.EndedNoResult {
+ t.Fatalf("last line %+v", last)
}
}
@@ -1546,18 +1977,36 @@ func (b *scriptedBox) result(last int64, rows string) {
b.s.Put(context.Background(), dbreads.ResultKey(b.id), r, "")
}
-// hookStore runs once on the console's first listing of a read's chunks,
-// after the listing is taken: a box committing between the console's
-// listing and its next result read.
-type hookStore struct {
+// dbReadHookStore runs once on the console's first listing of a read's
+// chunks, after the listing is taken: a box committing between the console's
+// listing and its next result read. It also records, in order, each result
+// read and each chunk listing. (hookStore is temphook_test.go's: plan review
+// finding 2.)
+type dbReadHookStore struct {
release.Store
once sync.Once
hook func()
+ mu sync.Mutex
+ ops []string
}
-func (h *hookStore) List(ctx context.Context, prefix string) ([]string, error) {
+func (h *dbReadHookStore) record(op string) {
+ h.mu.Lock()
+ defer h.mu.Unlock()
+ h.ops = append(h.ops, op)
+}
+
+func (h *dbReadHookStore) Get(ctx context.Context, key string) ([]byte, string, error) {
+ if strings.HasSuffix(key, "/result.json") {
+ h.record("result")
+ }
+ return h.Store.Get(ctx, key)
+}
+
+func (h *dbReadHookStore) List(ctx context.Context, prefix string) ([]string, error) {
keys, err := h.Store.List(ctx, prefix)
if strings.HasSuffix(prefix, "/out/") {
+ h.record("list")
h.once.Do(h.hook)
}
return keys, err
@@ -1565,12 +2014,16 @@ func (h *hookStore) List(ctx context.Context, prefix string) ([]string, error) {
// T18 (codex 20): the box commits its last chunk and result.json between the
// console's listing and its result read, in both orders. The result line
-// never comes before every chunk through last_seq, and the read succeeds.
+// never comes before every chunk through last_seq, the read succeeds, and in
+// every round the console reads result.json before it lists the chunks.
+// Mutations, each red on its own (plan review finding 20): list out/ before
+// reading result.json (the operation-order check), or write the result line
+// without waiting for next > last_seq (the chunk-count check).
func TestDBReadResultWaitsForItsLastChunk(t *testing.T) {
for _, resultFirst := range []bool{false, true} {
fx := newDBFFixture(t, true)
fx.srv.dbf.poll = 10 * time.Millisecond
- hs := &hookStore{Store: fx.rel}
+ hs := &dbReadHookStore{Store: fx.rel}
fx.srv.cfg.Releases = hs
body, key := queryBody(t, "SELECT 1")
res := fx.openRead(t, dbfHost, fx.member, body)
@@ -1604,6 +2057,14 @@ func TestDBReadResultWaitsForItsLastChunk(t *testing.T) {
if rows, _ := openAnswer(t, key, lines); string(rows) != "a\nb\n" {
t.Fatalf("rows %q", rows)
}
+ hs.mu.Lock()
+ ops := append([]string(nil), hs.ops...)
+ hs.mu.Unlock()
+ for i, op := range ops {
+ if op == "list" && (i == 0 || ops[i-1] != "result") {
+ t.Fatalf("result-first %v: a chunk listing not preceded by its round's result read: %v", resultFirst, ops)
+ }
+ }
}
}
@@ -1709,11 +2170,11 @@ func TestDBReadLateWritesChangeNothing(t *testing.T) {
}
```
-Add imports `"crypto/sha256"`, `"encoding/base64"`, `"encoding/hex"`, `"sync"`, `lsfile "github.com/benbjohnson/litestream/file"`, `dbdoortest`, `dbreadstest`.
+Add imports `"crypto/sha256"`, `"encoding/base64"`, `"encoding/hex"`, `"slices"`, `"sync"`, `lsfile "github.com/benbjohnson/litestream/file"`, `dbdoortest`, `dbreadstest`.
- [ ] **Step 2: Run them to verify they fail**
-Run: `go test ./internal/console/ -run 'TestDBReadStreams|TestDBReadEnds|TestDBReadLines|TestDBReadDisconnect|TestDBReadResultWaits|TestDBReadRequestBound|TestDBReadIgnores|TestDBReadLateWrites' -count=1`
+Run: `go test ./internal/console/ -run 'TestDBReadStreams|TestDBReadEnds|TestDBReadLines|TestDBReadAdmissionBounds|TestDBReadStalledStore|TestDBReadSlowRelay|TestDBReadResultWaits|TestDBReadRequestBound|TestDBReadIgnores|TestDBReadLateWrites' -count=1`
Expected: FAIL: the stub writes only `admitted`.
- [ ] **Step 3: Write the stream**
@@ -1721,13 +2182,23 @@ Expected: FAIL: the stub writes only `admitted`.
Replace the stub in `internal/console/apidbforward.go` with:
```go
+// errDBClientGone is a line the client did not take within s.dbf.stall.
+var errDBClientGone = errors.New("the client went or stalled")
+
+// dbfEndSlack is how far past the request's bound the last line may still be
+// written: room for the ended line, and no more.
+const dbfEndSlack = 10 * time.Second
+
// dbLineWriter writes one line at a time, each under a fresh write deadline
-// and flushed under the same deadline (spec §5.5 step 6, codex 19). A write
-// or flush error ends the request as a stalled client.
+// and flushed under the same deadline (spec §5.5 step 6, codex 19). The
+// deadline never passes until, the request's bound plus dbfEndSlack (plan
+// review finding 12). A write or flush error ends the request as a stalled
+// client.
type dbLineWriter struct {
w http.ResponseWriter
rc *http.ResponseController
stall time.Duration
+ until time.Time
last time.Time
}
@@ -1736,7 +2207,11 @@ func (l *dbLineWriter) write(line dbreads.Line) error {
if err != nil {
return err
}
- if err := l.rc.SetWriteDeadline(time.Now().Add(l.stall)); err != nil && !errors.Is(err, http.ErrNotSupported) {
+ dl := time.Now().Add(l.stall)
+ if dl.After(l.until) {
+ dl = l.until
+ }
+ if err := l.rc.SetWriteDeadline(dl); err != nil && !errors.Is(err, http.ErrNotSupported) {
return err
}
if _, err := l.w.Write(append(b, '\n')); err != nil {
@@ -1750,17 +2225,60 @@ func (l *dbLineWriter) write(line dbreads.Line) error {
}
// tick waits one poll and writes a wait line when nothing has been written
-// for every; false ends the request.
-func (l *dbLineWriter) tick(ctx context.Context, phase string, poll, every time.Duration) bool {
+// for every. It answers ctx's error when ctx ends first, and errDBClientGone
+// when the line cannot be written.
+func (l *dbLineWriter) tick(ctx context.Context, phase string, poll, every time.Duration) error {
select {
case <-ctx.Done():
- return false
+ return ctx.Err()
case <-time.After(poll):
}
- if time.Since(l.last) >= every {
- return l.write(dbreads.Line{Wait: phase}) == nil
+ if time.Since(l.last) >= every && l.write(dbreads.Line{Wait: phase}) != nil {
+ return errDBClientGone
+ }
+ return nil
+}
+
+// dbStream is one admitted read's relay, with what its last log line counts.
+type dbStream struct {
+ s *Server
+ lw *dbLineWriter
+ chunks int
+ bytes int
+}
+
+// dbfCall runs one store call under s.dbf.call and the phase's context, and
+// keeps writing wait lines while it blocks, so a stalled store neither holds
+// the request past its phase nor silences the stream (plan review finding
+// 12). A store that ignores its context is left behind: its goroutine ends
+// when the call does, into a buffered channel nobody reads.
+func dbfCall[T any](st *dbStream, ctx context.Context, phase string, op func(context.Context) (T, error)) (T, error) {
+ var zero T
+ cctx, cancel := context.WithTimeout(ctx, st.s.dbf.call)
+ defer cancel()
+ type outcome struct {
+ v T
+ err error
+ }
+ done := make(chan outcome, 1)
+ go func() {
+ v, err := op(cctx)
+ done <- outcome{v, err}
+ }()
+ t := time.NewTicker(st.s.dbf.poll)
+ defer t.Stop()
+ for {
+ select {
+ case o := <-done:
+ return o.v, o.err
+ case <-cctx.Done():
+ return zero, cctx.Err()
+ case <-t.C:
+ if time.Since(st.lw.last) >= st.s.dbf.wait && st.lw.write(dbreads.Line{Wait: phase}) != nil {
+ return zero, errDBClientGone
+ }
+ }
}
- return true
}
// readDBResult is result.json, or nil while there is none, and nil for one
@@ -1777,69 +2295,98 @@ func (s *Server) readDBResult(ctx context.Context, id string) *dbreads.Result {
return &res
}
-// cancelDBRead is the one write J1 allows after a read's request ends
-// without a result: best effort, so the owner stops early.
-func (s *Server) cancelDBRead(id string) {
- ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
- defer cancel()
- if _, err := s.cfg.Releases.Put(ctx, dbreads.CancelKey(id), nil, ""); err != nil && !errors.Is(err, release.ErrCASConflict) {
- log.Printf("console: db read %s: cancel: %v", id, err)
- }
-}
-
// streamDBRead is spec §5.5 steps 6 and 7 inside the admitted request:
// admitted at once, picked once, then about once a second result.json first
// and the chunk listing second (codex 20; the run follow door's order), each
// chunk in sequence with no gaps, a wait line whenever s.dbf.wait passes
// with nothing else, and the result last, only once every chunk through its
-// last_seq has been written. It ends without a result, and writes cancel,
-// when no picked appears in s.dbf.pickup, when the owner's deadline plus
-// s.dbf.grace passes (never later than s.dbf.bound after issued_at), or when
-// the client goes or stalls.
+// last_seq has been written. Each phase runs on its own deadline context:
+// pickup on issued_at plus s.dbf.pickup, the owner's on its deadline plus
+// s.dbf.grace, never later than s.dbf.bound after issued_at. It ends without
+// a result, and writes cancel, when either deadline passes, a result
+// included, or when the client goes or stalls. Its last log line is the
+// relay's own measure (plan review finding 21).
func (s *Server) streamDBRead(w http.ResponseWriter, r *http.Request, rc *http.ResponseController, a dbReadAdmitted) {
ctx := r.Context()
+ bound := a.issued.Add(s.dbf.bound)
w.Header().Set("Content-Type", "application/x-ndjson")
w.Header().Set("Cache-Control", "no-store")
w.WriteHeader(http.StatusOK)
- lw := &dbLineWriter{w: w, rc: rc, stall: s.dbf.stall}
- end := func(why string) {
- if why != "" {
- lw.write(dbreads.Line{Ended: why})
+ st := &dbStream{s: s, lw: &dbLineWriter{w: w, rc: rc, stall: s.dbf.stall, until: bound.Add(dbfEndSlack)}}
+ start, why := time.Now(), "client gone"
+ defer func() {
+ log.Printf("console: db read %s: relayed %d chunks (%d bytes) in %s, ended %s",
+ a.id, st.chunks, st.bytes, time.Since(start).Round(time.Millisecond), why)
+ }()
+ end := func(ended string) {
+ if ended != "" {
+ why = ended
+ st.lw.write(dbreads.Line{Ended: ended})
}
s.cancelDBRead(a.id)
}
- if lw.write(dbreads.Line{Admitted: a.id}) != nil {
+ // stop ends a phase: a client that went or stalled gets no line, a phase
+ // whose deadline passed gets its ended word.
+ stop := func(err error, ended string) {
+ if errors.Is(err, errDBClientGone) || ctx.Err() != nil {
+ end("")
+ return
+ }
+ end(ended)
+ }
+ if st.lw.write(dbreads.Line{Admitted: a.id}) != nil {
end("")
return
}
store := s.cfg.Releases
+
+ pickCtx, cancelPick := context.WithDeadline(ctx, a.issued.Add(s.dbf.pickup))
+ defer cancelPick()
var picked dbreads.Picked
for {
- raw, _, err := store.Get(ctx, dbreads.PickedKey(a.id))
+ raw, err := dbfCall(st, pickCtx, dbreads.WaitPickup, func(c context.Context) ([]byte, error) {
+ b, _, err := store.Get(c, dbreads.PickedKey(a.id))
+ return b, err
+ })
if err == nil && json.Unmarshal(raw, &picked) == nil {
break
}
- if time.Now().After(a.issued.Add(s.dbf.pickup)) {
- end(dbreads.EndedNotPicked)
+ if pickCtx.Err() != nil || errors.Is(err, errDBClientGone) {
+ stop(err, dbreads.EndedNotPicked)
return
}
- if !lw.tick(ctx, dbreads.WaitPickup, s.dbf.poll, s.dbf.wait) {
- end("")
+ if err := st.lw.tick(pickCtx, dbreads.WaitPickup, s.dbf.poll, s.dbf.wait); err != nil {
+ stop(err, dbreads.EndedNotPicked)
return
}
}
- if lw.write(dbreads.Line{Picked: picked.Box}) != nil {
+ if st.lw.write(dbreads.Line{Picked: picked.Box}) != nil {
end("")
return
}
deadline := picked.Deadline.Add(s.dbf.grace)
- if bound := a.issued.Add(s.dbf.bound); deadline.After(bound) {
+ if deadline.After(bound) {
deadline = bound
}
+ ownCtx, cancelOwn := context.WithDeadline(ctx, deadline)
+ defer cancelOwn()
next := int64(1)
for {
- res := s.readDBResult(ctx, a.id) // first
- if keys, err := store.List(ctx, dbreads.OutPrefix(a.id)); err == nil {
+ res, err := dbfCall(st, ownCtx, dbreads.WaitReading, func(c context.Context) (*dbreads.Result, error) {
+ return s.readDBResult(c, a.id), nil
+ }) // first
+ if ownCtx.Err() != nil || errors.Is(err, errDBClientGone) {
+ stop(err, dbreads.EndedNoResult)
+ return
+ }
+ keys, err := dbfCall(st, ownCtx, dbreads.WaitReading, func(c context.Context) ([]string, error) {
+ return store.List(c, dbreads.OutPrefix(a.id))
+ })
+ if ownCtx.Err() != nil || errors.Is(err, errDBClientGone) {
+ stop(err, dbreads.EndedNoResult)
+ return
+ }
+ if err == nil {
listed := map[int64]bool{}
for _, k := range keys {
if seq, ok := dbreads.ParseChunkKey(k, dbreads.OutPrefix(a.id)); ok {
@@ -1847,27 +2394,40 @@ func (s *Server) streamDBRead(w http.ResponseWriter, r *http.Request, rc *http.R
}
}
for listed[next] {
- sealed, _, err := store.Get(ctx, dbreads.ChunkKey(a.id, next))
+ seq := next
+ sealed, err := dbfCall(st, ownCtx, dbreads.WaitReading, func(c context.Context) ([]byte, error) {
+ b, _, err := store.Get(c, dbreads.ChunkKey(a.id, seq))
+ return b, err
+ })
+ if ownCtx.Err() != nil || errors.Is(err, errDBClientGone) {
+ stop(err, dbreads.EndedNoResult)
+ return
+ }
if err != nil {
break
}
- if lw.write(dbreads.Line{Chunk: next, Data: sealed}) != nil {
+ if st.lw.write(dbreads.Line{Chunk: seq, Data: sealed}) != nil {
end("")
return
}
+ st.chunks++
+ st.bytes += len(sealed)
next++
}
}
if res != nil && next > res.LastSeq {
- lw.write(dbreads.Line{Result: res})
+ // The bound holds for a result too: one that is complete only
+ // after the deadline is not relayed (plan review finding 12).
+ if ownCtx.Err() != nil {
+ stop(nil, dbreads.EndedNoResult)
+ return
+ }
+ why = res.State
+ st.lw.write(dbreads.Line{Result: res})
return
}
- if time.Now().After(deadline) {
- end(dbreads.EndedNoResult)
- return
- }
- if !lw.tick(ctx, dbreads.WaitReading, s.dbf.poll, s.dbf.wait) {
- end("")
+ if err := st.lw.tick(ownCtx, dbreads.WaitReading, s.dbf.poll, s.dbf.wait); err != nil {
+ stop(err, dbreads.EndedNoResult)
return
}
}
@@ -3164,8 +3724,8 @@ git commit -m "edge: a forwarded read's stream survives the backhaul hop (T7)"
- Modify: `.amadan/ci.d/30-minio-contract` (buckets, users, the run and its PASS check)
**Interfaces:**
-- Consumes: `dbfFixture`, `queryBody`, `openRead`, `readLines`, `openAnswer`, `putEdgesFile`, `putBoxStatus`, `runBox`, `strictStore`, `scriptFor` (Tasks 4 to 7); `newConsoleFixtureKeyed`'s body; `release.S3Store`; `lss3.NewReplicaClient`; `dbdoortest.Replicate`; `dbreadstest.Start`.
-- Produces: `newConsoleFixtureOn(t *testing.T, withKey bool, rel release.Store) *consoleFixture`; `uploadReplica(t, from *lsfile.ReplicaClient, to *lss3.ReplicaClient)`; `TestDBForwardMinio`.
+- Consumes: `dbfFixture`, `queryBody`, `openRead`, `readLines`, `openAnswer`, `putEdgesFile`, `putBoxStatus`, `runBox`, `strictStore` (Tasks 4 to 7); `newConsoleFixtureKeyed`'s body; `release.S3Store`; `lss3.NewReplicaClient`; `dbdoortest.Replicate`; `dbreadstest.Start`.
+- Produces: `newConsoleFixtureOn(t *testing.T, withKey bool, rel release.Store) *consoleFixture`; `uploadReplica(t, from *lsfile.ReplicaClient, to *lss3.ReplicaClient)`; `dbfGateStore`; `TestDBForwardMinio`.
- [ ] **Step 1: Let the console fixture run over another store**
@@ -3193,8 +3753,11 @@ import (
"bufio"
"context"
"encoding/json"
+ "net/http"
"os"
"strconv"
+ "strings"
+ "sync"
"testing"
"time"
@@ -3238,6 +3801,60 @@ func uploadReplica(t *testing.T, from *lsfile.ReplicaClient, to *lss3.ReplicaCli
}
}
+// dbfGateStore holds the box's chunk and result creates, once armed, until
+// release closes, and records when each landed: T16 needs a box that
+// publishes only after the console has ended the request (plan review
+// finding 19).
+type dbfGateStore struct {
+ release.Store
+ mu sync.Mutex
+ open chan struct{}
+ reached chan struct{}
+ once *sync.Once
+ landed []time.Time
+}
+
+func (g *dbfGateStore) arm() {
+ g.mu.Lock()
+ defer g.mu.Unlock()
+ g.open, g.reached, g.once, g.landed = make(chan struct{}), make(chan struct{}), &sync.Once{}, nil
+}
+
+func (g *dbfGateStore) release() {
+ g.mu.Lock()
+ defer g.mu.Unlock()
+ close(g.open)
+}
+
+func (g *dbfGateStore) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
+ g.mu.Lock()
+ open, reached, once := g.open, g.reached, g.once
+ g.mu.Unlock()
+ if open == nil || !(strings.Contains(key, "/out/") || strings.HasSuffix(key, "/result.json")) {
+ return g.Store.Put(ctx, key, body, ifMatch)
+ }
+ once.Do(func() { close(reached) })
+ <-open
+ etag, err := g.Store.Put(ctx, key, body, ifMatch)
+ if err == nil {
+ g.mu.Lock()
+ g.landed = append(g.landed, time.Now())
+ g.mu.Unlock()
+ }
+ return etag, err
+}
+
+func (g *dbfGateStore) landedAfter(t time.Time) (n int, early bool) {
+ g.mu.Lock()
+ defer g.mu.Unlock()
+ for _, at := range g.landed {
+ if !at.After(t) {
+ early = true
+ }
+ }
+ return len(g.landed), early
+}
+
// TestDBForwardMinio is T1 and T16 against a real S3 server: a console in ie
// and a box in uk sharing one deployment bucket (versioned), each with its
// own data bucket and its own credentials, on one endpoint (spec §12). The
@@ -3314,8 +3931,9 @@ func TestDBForwardMinio(t *testing.T) {
}
putEdgesFile(t, fx.rel, "uk-"+run, "uk", []string{dbreads.Capability}, false)
putBoxStatus(t, fx.rel, "uk-"+run, time.Now())
+ gate := &dbfGateStore{Store: store(boxKey, boxSecret, "dbf-deploy")}
box := dbreadstest.Start(t, dbreadstest.Options{
- Store: strictStore{Store: store(boxKey, boxSecret, "dbf-deploy"), t: t}, Region: "uk", Box: "uk-" + run,
+ Store: strictStore{Store: gate, t: t}, Region: "uk", Box: "uk-" + run,
Pub: fx.pub, HibernatePrefix: "instances",
OpenReplica: func(_ context.Context, prefix string) (dbdoor.ReplicaReader, error) {
return replica(boxKey, boxSecret, "dbf-data-uk", prefix), nil
@@ -3335,55 +3953,79 @@ func TestDBForwardMinio(t *testing.T) {
}
}
- // T16, by disconnect: the CLI goes after admitted; the box still serves
- // the read and writes its chunks and result after the request ended.
- body, _ := queryBody(t, "SELECT count(*) AS n FROM t")
- res := dfx.openRead(t, shop, member, body)
- line, _ := bufio.NewReader(res.Body).ReadBytes('\n')
- res.Body.Close()
- var adm dbreads.Line
- json.Unmarshal(line, &adm)
- deadline := time.Now().Add(30 * time.Second)
- for {
- _, _, rerr := deploy.Get(context.Background(), dbreads.ResultKey(adm.Admitted))
- _, _, cerr := deploy.Get(context.Background(), dbreads.CancelKey(adm.Admitted))
- if rerr == nil && cerr == nil {
- break
+ // T16 (plan review finding 19): the real box reaches publication and is
+ // held there; the console's request ends, by disconnect and then by its
+ // bound; only then does the box publish. Its late chunks and result land
+ // after the console ended, change nothing the console sends, and the
+ // strict stores see no delete and no replacing write from either side.
+ late := func(t *testing.T, end func(res *http.Response, br *bufio.Reader)) {
+ gate.arm()
+ body, _ := queryBody(t, "SELECT count(*) AS n FROM t")
+ res := dfx.openRead(t, shop, member, body)
+ br := bufio.NewReader(res.Body)
+ line, _ := br.ReadBytes('\n')
+ var adm dbreads.Line
+ json.Unmarshal(line, &adm)
+ end(res, br)
+ // The handler's last act is the cancel: once it is there, the
+ // console has ended the request.
+ waitKey := func(key string) {
+ deadline := time.Now().Add(60 * time.Second)
+ for {
+ if _, _, err := deploy.Get(context.Background(), key); err == nil {
+ return
+ }
+ if time.Now().After(deadline) {
+ t.Fatalf("no %s", key)
+ }
+ time.Sleep(100 * time.Millisecond)
+ }
+ }
+ waitKey(dbreads.CancelKey(adm.Admitted))
+ consoleEnded := time.Now()
+ select {
+ case <-gate.reached:
+ case <-time.After(60 * time.Second):
+ t.Fatal("the box never reached publication")
+ }
+ if keys, _ := deploy.List(context.Background(), dbreads.OutPrefix(adm.Admitted)); len(keys) != 0 {
+ t.Fatalf("output published before the console ended: %v", keys)
+ }
+ gate.release()
+ waitKey(dbreads.ResultKey(adm.Admitted))
+ // At least the result lands late; a held chunk lands too unless the
+ // owner's cancel watch ended its create first (plan 1's watchCancel).
+ if n, early := gate.landedAfter(consoleEnded); n < 1 || early {
+ t.Fatalf("%d late writes, any before the console ended: %v", n, early)
}
- if time.Now().After(deadline) {
- t.Fatalf("after a disconnect: result %v, cancel %v", rerr, cerr)
- }
- time.Sleep(200 * time.Millisecond)
- }
-
- // T16, by timeout: a third tenant with nothing replicated. A scripted
- // box (a stand-in) races the real one for picked (J3). If the scripted
- // box wins, it picks with a short deadline, the console gives up at that
- // deadline, and the scripted box writes its chunk and result afterwards.
- // If the real box wins, it answers nothing-replicated. Either way the
- // strict stores see no delete and no replacing write.
- late := "late-" + run + ".example.com"
- putShopRecord(t, fx, provision.Record{Host: late, Region: "uk", CreatedAt: time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC)})
- fx.srv.dbf.grace = 0
- body, _ = queryBody(t, "SELECT 1")
- res = dfx.openRead(t, late, member, body)
- br := bufio.NewReader(res.Body)
- line, _ = br.ReadBytes('\n')
- json.Unmarshal(line, &adm)
- script := scriptFor(t, deploy, adm.Admitted)
- script.picked(time.Now().Add(300 * time.Millisecond))
- rest := readLines(t, br)
- res.Body.Close()
- switch last := rest[len(rest)-1]; {
- case last.Ended == dbreads.EndedNoResult:
- // The scripted box owned the read and writes after the request ended.
- script.chunk(1, true, "late\n")
- script.result(1, "late\n")
- case last.Result != nil && last.Result.Code == dbreads.CodeNothingReplicated:
- // The real box owned it; the scripted create of picked lost.
- default:
- t.Fatalf("by timeout: %+v", last)
}
+ t.Run("by disconnect", func(t *testing.T) {
+ late(t, func(res *http.Response, _ *bufio.Reader) {
+ select {
+ case <-gate.reached:
+ case <-time.After(60 * time.Second):
+ t.Fatal("the box never reached publication")
+ }
+ res.Body.Close()
+ })
+ })
+ t.Run("by timeout", func(t *testing.T) {
+ // The request's own bound, well past the box's restore and query of
+ // three batches, well short of the box's 20-minute owner deadline.
+ fx.srv.dbf.bound, fx.srv.dbf.grace = 20*time.Second, 0
+ late(t, func(res *http.Response, br *bufio.Reader) {
+ defer res.Body.Close()
+ lines := readLines(t, br)
+ if last := lines[len(lines)-1]; last.Ended != dbreads.EndedNoResult {
+ t.Fatalf("by timeout: %+v", last)
+ }
+ for _, l := range lines {
+ if l.Chunk != 0 || l.Result != nil {
+ t.Fatalf("a held box's output reached the console: %+v", l)
+ }
+ }
+ })
+ })
}
```
@@ -3672,16 +4314,25 @@ In a copy of the tree (`cp -a . "$TMPDIR/dbf2-mut"`), make each mutation, run it
|---|---|
| in `dbreads` `attempt`, open the replica at a prefix from anything but `rec.Host` (for example the first host the agent ever checked) | `TestDBForwardMinio` (the second tenant's rows), in the MinIO step; `TestDBReadsParityWithTheInRegionDoor` |
| `openDBForward` skips the byte count and SHA-256 comparison | `TestForwardedQueryRefusesAnAnswerThatDoesNotCheckOut`, `TestDBForwardPrintsNothingForABadAnswer` |
-| `admitDBRead` releases the counts before returning, or `handleAPIDBRead` drops its `defer` | `TestDBReadAdmissionBounds`, `TestDBReadDisconnectWritesCancelAndFreesTheSlot` |
+| `admitDBRead` releases the counts before returning, or `handleAPIDBRead` drops its `defer` | `TestDBReadAdmissionBounds` |
| `handleAPIDBRead` drops `SetReadDeadline` | `TestDBForwardLongRequestThroughTheEdge` |
| `dbLineWriter.write` drops `Flush` | `TestDBReadLinesAreFlushedAsWritten`, `TestDBForwardLongRequestThroughTheEdge` |
-| `streamDBRead` writes the result line without waiting for `next > res.LastSeq` | `TestDBReadResultWaitsForItsLastChunk` |
-| `streamDBRead` lists chunks before reading `result.json` and drops the `next > res.LastSeq` wait | `TestDBReadResultWaitsForItsLastChunk` (with the wait in place, listing first only costs a round; the result-first order is kept as the run follow door keeps it, and the wait is what T18 pins) |
+| `streamDBRead` writes the result line without waiting for `next > res.LastSeq` | `TestDBReadResultWaitsForItsLastChunk` (the chunk-count check) |
+| `streamDBRead` lists chunks before reading `result.json`, the wait left in place (plan review finding 20) | `TestDBReadResultWaitsForItsLastChunk` (the operation-order check: a `list` not preceded by its round's `result`) |
+| `admitDBRead` writes the audit row after the creates (plan review finding 11) | `TestDBReadEndedDuringAdmissionWritesOnlyTheCancel` ("the audit row is down before the request") |
+| `admitDBRead` calls `s.bumpSerial` instead of `announceDBRead`, or drops the `ctx.Err()` check after the queue create | `TestDBReadEndedDuringAdmissionWritesOnlyTheCancel` ("ended after the queue entry": the serial moves) |
+| `admitDBRead` drops `cancelDBRead` on the ended path | `TestDBReadEndedDuringAdmissionWritesOnlyTheCancel` (no cancel) |
+| `handleAPIDBRead` drops the `preHeader` read deadline (plan review finding 13) | `TestDBReadStalledBodyIsCutAtThePreHeaderBound` |
+| `streamDBRead` calls the store on `r.Context()` without `dbfCall`, or drops either phase's deadline context (plan review finding 12) | `TestDBReadStalledStoreKeepsTheStreamAndItsBound` |
+| `streamDBRead` relays without the owner's deadline (chunk Gets on `r.Context()`, no `ownCtx` check after them) | `TestDBReadSlowRelayEndsAtTheBound` |
+| a startup pass that writes under `db-reads/` | `TestDBReadAdmissionBounds` (the restart) |
| `cancelDBRead` deletes the read's objects, or any console `Put` under `db-reads/` passes an etag | `TestDBReadLateWritesChangeNothing`, `TestDBForwardMinio` |
| `dbForwardEvidence` checks `Complete` and not `BoxesComplete` | `TestDBForwardEvidenceRefusesOnlyOnCompleteEvidence` |
Record the table with every row red in the branch description.
+One check is not in the table on purpose: the `ownCtx.Err()` check just before the result line. `dbfCall` already cuts every store call once the owner's deadline passes, so that check guards only a line write that outlasts the deadline, which no in-process client can force. It stays as defence in depth, and the branch description says it has no witness of its own.
+
- [ ] **Step 2: Merge main and run the census**
```bash
@@ -3692,9 +4343,9 @@ Then add the temporary dump test from plan 1 Task 17 Step 2 (prefixes: `db.go`,
- [ ] **Step 3: Write the overlay**
-`internal/provision/testdata/plan14-root-reads-db-region-forward-2.tsv`, in plan 1 Task 17 Step 3's formats, with `rev3=PROPOSED-db-region-forward-2` and these bases by identity:
+`internal/provision/testdata/plan14-root-reads-db-region-forward-2.tsv`, in plan 1 Task 17 Step 3's formats (each `@legacy` fingerprint computed by plan 1's `TestZZDBForwardFingerprints`, in the temporary dump test, from the row's own fields), with `rev3=PROPOSED-db-region-forward-2` and these bases by identity:
-- console units (`Server.handleAPIDBRead`, `Server.admitDBRead`, `Server.streamDBRead`, `dbLineWriter.write`, `Server.readDBResult`, `Server.cancelDBRead`, `Server.dbResidency`, `dbForwardEvidence`, `NewHTTPServer`, and any changed console legacy unit the census names): `accepted-legacy;rev3=PROPOSED-db-region-forward-2;basis=route-and-input (J1,J4, as main's db door units); the console (cmd/console serve under console.service, user carlos-console): <one clause on what the unit reads: the instance record and the edges and status listing for the gate, or the read's picked, result.json and chunk objects in the deployment bucket, relayed to the member as ciphertext>; never reads another region's data bucket, never runs SQL, makes no root decision`;
+- console units (`Server.handleAPIDBRead`, `Server.admitDBRead`, `Server.streamDBRead`, `dbLineWriter.write`, `dbLineWriter.tick`, `dbfCall`, `Server.readDBResult`, `Server.cancelDBRead`, `Server.announceDBRead`, `Server.dbResidency`, `dbForwardEvidence`, `NewHTTPServer`, and any changed console legacy unit the census names): `accepted-legacy;rev3=PROPOSED-db-region-forward-2;basis=route-and-input (J1,J4, as main's db door units); the console (cmd/console serve under console.service, user carlos-console): <one clause on what the unit reads: the instance record and the edges and status listing for the gate, or the read's picked, result.json and chunk objects in the deployment bucket, relayed to the member as ciphertext>; never reads another region's data bucket, never runs SQL, makes no root decision`;
- CLI units (`dbQueryForward`, `readDBForward`, `openDBForward`, and the changed `dbRestore`, `dbQuery`, `dbPull`): `accepted-legacy;rev3=PROPOSED-db-region-forward-2;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): <one clause>`;
- `internal/dbdoor` `doorError` (changed): re-frozen at main's disposition with "and the db region forward marker's three additive fields" appended;
- `cmd/console` `serve` (changed): re-frozen at its disposition with "CARLOS_DB_FORWARD and console.NewHTTPServer" appended;
diff --git a/docs/superpowers/plans/2026-10-22-db-region-forward-plan-3-pull.md b/docs/superpowers/plans/2026-10-22-db-region-forward-plan-3-pull.md
index 387c3c2fc..253a9221d 100644
--- a/docs/superpowers/plans/2026-10-22-db-region-forward-plan-3-pull.md
+++ b/docs/superpowers/plans/2026-10-22-db-region-forward-plan-3-pull.md
@@ -8,7 +8,7 @@
**Tech Stack:** Go 1.26, plans 1 and 2's `internal/dbreads`, `internal/dbdoor`, console and CLI code, superfly/ltx's decoder (`Verify`), litestream's file replica client.
-**Spec:** `docs/superpowers/specs/2026-10-22-db-region-forward-design.md` (rev 5; Paul approved building it on 2026-10-22 with the rulings in §16, among them ruling 1: pull goes through the deployment bucket as designed, with the true storage statement, and a per-region bucket is plan 5). Read §2, §5.5 to §5.7, §6, §14 and §16. Plans 1 and 2 must have merged and plan 2 must have rolled to dev.
+**Spec:** `docs/superpowers/specs/2026-10-22-db-region-forward-design.md` (rev 6, which is rev 5 with the plan review's four clarifications; Paul approved building it on 2026-10-22 with the rulings in §16, among them ruling 1: pull goes through the deployment bucket as designed, with the true storage statement, and a per-region bucket is plan 5). Read §2, §5.5 to §5.7, §6, §14 and §16. Plans 1 and 2 must have merged and plan 2 must have rolled to dev.
**Branch:** `db-region-forward-3-pull`, cut from amadan `main` after plan 2 merged.
@@ -16,7 +16,7 @@
Every task's requirements include this section.
-- **Production-wired acceptance.** As plans 1 and 2: the real console server, the real CLI binary through `carlosTestEnv` with a scratch `cmd.Dir`, real stores (MinIO in CI), real verify, seal and restore code. The in-process box (`dbreadstest`) is not acceptance for the units; plan 1's root-runner tests are, and Task 4 extends them for the pull's worker mode. Scripted boxes are stand-ins and say so.
+- **Production-wired acceptance.** As plans 1 and 2: the real console server, the real CLI binary through `carlosTestEnv` with a scratch `cmd.Dir`, real stores (MinIO in CI), real verify, seal and restore code. The in-process box (`dbreadstest`) is not acceptance for the units; plan 1's root-runner tests are, and Task 5 extends them for the pull, once the agent can make one. Scripted boxes are stand-ins and say so.
- **Targeted local runs; CI runs the full suite.** Never `make test` or a whole console, host or root package locally; build, vet, gofmt and the cross-compiles run locally.
- **Cross-compile gate.** `GOOS=darwin GOARCH=arm64 go build ./...` and `GOOS=windows GOARCH=amd64 go build ./...` before every push. The CLI's pull path is portable.
- **Copy needs review before it ships, with no em dashes,** lower case after a full stop (ruling 4). Task 2 reviews the pull rows; no pull string reaches code first.
@@ -36,11 +36,23 @@ Every task's requirements include this section.
The five input classes the spec implies but no gate test names, most likely first. Each has its test in the owning task.
-1. **The member's disk fills mid-pull.** A pull spools up to the cap of ciphertext, then writes the files and the restored copy: about three times the cap under `os.TempDir()`. A person expects a clean failure, nothing at `--out`, and the spool directory gone. Test: Task 7, `TestForwardedPullLeavesNothingWhenTheSpoolFails`.
+1. **The member's disk fills mid-pull.** A pull spools up to the cap of ciphertext, then writes the files and the restored copy: about three times the cap under `os.TempDir()`. A person expects a clean failure, nothing at `--out`, and the spool directory gone. Test: Task 7, `TestForwardedPullLeavesNothingWhenTheSpoolFails` (through `dbPull`, the third spool write failing).
2. **A planned file that is corrupt in the bucket.** A person expects the box's worker to refuse it (`check-failed`, today's sentence), never to ship it, and never a half-written `--out`. Test: Task 4, `TestWorkerHoldRefusesACorruptFile`.
3. **Files that do not add up.** A summary whose file sizes do not sum to its byte count, or a file list that does not end at the summary's transaction. A person expects DBF-13 and nothing at `--out`. Test: Task 7, `TestForwardedPullRefusesFilesThatDoNotAddUp`.
-4. **Ctrl-C during a long pull.** A person expects `stopped`, nothing at `--out`, and the spool directory gone. Test: Task 7, `TestForwardedPullCtrlCLeavesNothing`.
-5. **A box whose cap is raised past the CLI's bound.** `CARLOS_DB_PULL_MAX_BYTES` above the built-in cap would ship more than any CLI accepts. A person expects the box to clamp to the built-in cap. Test: Task 5, `TestPullCapOnlyLowers`.
+4. **Ctrl-C during a long pull.** A person expects `stopped`, nothing at `--out`, and the spool directory gone. Tests: Task 7, `TestForwardedPullCtrlCLeavesNothing` (through `dbPull`), and Task 9, `TestDBForwardPullInterruptedLeavesNothing` (SIGINT to the binary).
+5. **A box whose cap is raised past the CLI's bound.** `CARLOS_DB_PULL_MAX_BYTES` above the built-in cap would ship more than any CLI accepts. A person expects the box to clamp to the built-in cap. Tests: Task 5, `TestPullCapOnlyLowers` and `TestNewAgentClampsMaxPull`.
+
+## Plan review, round 1
+
+Astra's findings (`docs/superpowers/specs/CODEX-FINDINGS-DB-REGION-FORWARD-PLANS.md`) for this plan, and where each landed:
+
+- **8** (the installed-unit pull test came before the agent's pull): it is Task 5's Step 5 now; Task 4 keeps the worker's own tests.
+- **21** (the measurement needed telemetry nobody built): plan 1's agent logs its planning-and-restore time and its confirmed output, plan 2's console logs its relay; Task 1 measures from those two lines.
+- **22** (malformed sizes could panic the CLI): `checkPullManifest` checks the whole file list before any file is created. Tests: four new malformed cases, each asserting DBF-13, no panic and no file.
+- **23** (the cleanup tests did not run cleanup): the spool failure and Ctrl-C run `dbPull` itself against a stand-in console, with `TMPDIR` scratch and a spool-write seam; Task 9 sends SIGINT to the binary mid-pull.
+- **24** (mutations that could not turn their tests red): `TestNewAgentClampsMaxPull` calls `NewAgent`; the malformed-manifest cases assert no file is created; the record-changed case asserts the worker never received `run`.
+
+None declined.
## Census impact
@@ -84,7 +96,7 @@ Drafted here for Task 2's review; none reaches code before Paul approves it:
Spec §14: "A 2 GiB pull in one request is the case most likely to need a longer request bound; plan 3 measures it before choosing." The choice rule here keeps the request bound (22 minutes, J1) and sizes the pull to fit it, because a longer request would reopen every timing argument of spec §5.2.
**Files:**
-- Modify: `docs/superpowers/specs/2026-10-22-db-region-forward-design.md` (§5.7's limits table and a short rev 6 note)
+- Modify: `docs/superpowers/specs/2026-10-22-db-region-forward-design.md` (§5.7's limits table and a short rev 7 note)
- [ ] **Step 1: Measure the forwarded path's throughput on dev**
@@ -96,7 +108,12 @@ time carlos db query --app <dev app> --instance <its uk instance> \
wc -c "$TMPDIR/dbf-measure.out"; rm "$TMPDIR/dbf-measure.out"
```
-That is about 64 MB of rows (the output cap; the command ends with DBF-6, which is expected). Run it three times. From the box's journal (`journalctl -u carlos-edge` on the uk box, read through the operator's usual access) take the time between the read's `picked` and its result write, and subtract the restore and query time the worker reports; from the console's journal take the same read's request duration. The relay rate R is the output bytes over the slower of the two streaming times, in MB/s. Record all three runs.
+That is about 64 MB of rows (the output cap; the command ends with DBF-6, which is expected). Run it three times. Each run leaves two lines that plans 1 and 2 write for this measurement (plan review finding 21):
+
+- the box's, in `journalctl -u carlos-edge` on the uk box (read through the operator's usual access): `dbreads: <id>: output <B> bytes in <N> confirmed chunks in <T_box>`, the time from `run` to the last chunk's confirmed create, which excludes planning and restore (the line before it, `planned ... and restored to ... in ...`, is the restore's);
+- the console's, in `journalctl -u console`: `console: db read <id>: relayed <N> chunks (<B'> bytes) in <T_console>, ended <why>`, the whole stream from the 200 to its last line.
+
+The box's rate is B / T_box. The console's is B' / (T_console − T_plan), where T_plan is the `planned ... in <T_plan>` line's duration; that leaves the pickup inside the console's time, so it understates the console's rate, which only makes the cap smaller. R is the slower of the two, in MB/s. Record all three runs, with both lines of each.
- [ ] **Step 2: Apply the rule**
@@ -113,11 +130,11 @@ In §5.7's limits table add, under "Output":
| Pull chunk | 1 MiB or 1 s, whichever comes first | agent |
```
-and a line in the status header: "Rev 6 (plan 3, <date>): the pull's cap and chunk size, from a measurement on dev; the request bound is unchanged." Ask Paul to confirm the cap as a limit (ruling 2 covers the others), and write his answer into §16's ruling 2. Commit:
+and a line in the status header: "Rev 7 (plan 3, <date>): the pull's cap and chunk size, from a measurement on dev; the request bound is unchanged." Ask Paul to confirm the cap as a limit (ruling 2 covers the others), and write his answer into §16's ruling 2. Commit:
```bash
git add docs/superpowers/specs/2026-10-22-db-region-forward-design.md
-git commit -m "db-region-forward spec rev 6: the pull's cap and chunk size, measured on dev"
+git commit -m "db-region-forward spec rev 7: the pull's cap and chunk size, measured on dev"
```
Every later task uses the recorded numbers. Where this plan writes `2 << 30` for the cap, use Task 1's value if it is lower.
@@ -262,7 +279,8 @@ git commit -m "dbreads: pull in the envelope, the pull's cap and chunk size"
**Files:**
- Modify: `internal/dbreads/worker.go`
- Create: `internal/dbreads/pull_test.go` (this task's tests)
-- Modify: `internal/host/dbreads_rootrunner_linux_test.go` (a pull through the installed units)
+
+The installed-unit pull test needs the agent's pull and is Task 5's (plan review finding 8).
**Interfaces:**
- Consumes: `receiveFile`, `frameWriter`, the frames (plan 1 Task 7); `ltx.NewDecoder`, `(*ltx.Decoder).Verify`.
@@ -461,19 +479,15 @@ func verifyLTX(path string) error {
}
```
-- [ ] **Step 4: Add the pull to the root-runner tests**
+- [ ] **Step 4: Run the unit tests**
-In `internal/host/dbreads_rootrunner_linux_test.go`, add `TestRootDBReadPullThroughTheInstalledUnits`: as `TestRootDBReadServesThroughTheInstalledUnits`, but `r.submit` builds a pull request (`Op: dbreads.OpPull`, empty `SQL`; give `submit` an `op` parameter and pass `dbreads.OpQuery` at every existing call), and the answer is checked by opening every chunk with `dbreads.StreamLTX`, comparing the concatenation's SHA-256 with the summary's, and comparing it byte for byte with the planned files read from `r.dir/replica/instances/shop.example.com` in plan order. After it, `privateTmpGone(t, 10*time.Second)`.
-
-- [ ] **Step 5: Run the unit tests and build the root-runner binary**
-
-Run (sandbox off): `go test ./internal/dbreads/ -run 'TestWorker' -count=1 -race && GOOS=linux GOARCH=arm64 go test -c -tags rootrunner -o "$TMPDIR/dbf3-rr.test" ./internal/host && rm "$TMPDIR/dbf3-rr.test"`
-Expected: PASS and a clean build. Run `TestRootDBReadPullThroughTheInstalledUnits` on the root runner (plan 1 Task 15 Step 4's command) before the branch is reviewed, and record the run id.
+Run (sandbox off): `go test ./internal/dbreads/ -run 'TestWorker' -count=1 -race`
+Expected: PASS.
-- [ ] **Step 6: Commit**
+- [ ] **Step 5: Commit**
```bash
-git add internal/dbreads internal/host/dbreads_rootrunner_linux_test.go
+git add internal/dbreads
git commit -m "dbreads: the worker's hold mode for a pull: verify, hold, send back after check B"
```
@@ -484,7 +498,8 @@ git commit -m "dbreads: the worker's hold mode for a pull: verify, hold, send ba
**Files:**
- Modify: `internal/dbreads/read.go` (`attempt` branches on the op), `internal/dbreads/agent.go` (`AgentConfig.MaxPull`)
- Modify: `internal/host/dbreads.go` (`CARLOS_DB_PULL_MAX_BYTES`)
-- Modify: `internal/dbreads/pull_test.go`, `internal/host/dbreads_test.go`
+- Modify: `internal/dbreads/pull_test.go`, `internal/dbreads/fixture_test.go` (plan 1's: `inProcWorkers` records the frames the agent sends), `internal/host/dbreads_test.go`
+- Modify: `internal/host/dbreads_rootrunner_linux_test.go` (a pull through the installed units, after the agent can make one: plan review finding 8)
**Interfaces:**
- Consumes: `attempt`, `newChunkWriter` with a size (Task 3), `relay`, `ended`, the worker's hold mode (Task 4).
@@ -571,7 +586,7 @@ func TestPullTooLargeAndNoChunkBeforeCheckB(t *testing.T) {
last := plan.Planned[len(plan.Planned)-1]
hook := &stageHook{ReplicaReader: lsfile.NewReplicaClient(fx2.replica), stage: "after the last byte",
lastSpan: dbdoor.Span(last.MinTXID, last.MaxTXID), change: func() { r := fx2.record(); r.Region = "ie"; fx2.putRecord(r) }}
- b, _ := fx2.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) {
+ b, w := fx2.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) {
c.OpenReplica = func(context.Context, string) (dbdoor.ReplicaReader, error) { return hook, nil }
})
req2 := fx2.pullRequest("")
@@ -581,9 +596,58 @@ func TestPullTooLargeAndNoChunkBeforeCheckB(t *testing.T) {
if _, _, r := fx2.answerPull(req2.ID); r.Code != CodeRecordChanged || len(fx2.chunkKeys(req2.ID)) != 0 {
t.Fatalf("%+v, chunks %v", r, fx2.chunkKeys(req2.ID))
}
+ // The worker never heard run: check B failed, so the agent said stop
+ // (plan review finding 24). Without this, an agent that sent run first
+ // and refused the files afterwards would pass the two checks above.
+ for _, typ := range w.sentFrames() {
+ if typ == frameRun {
+ t.Fatalf("the agent sent run before check B passed: %v", w.sentFrames())
+ }
+ }
+}
+
+// Plan review finding 24: NewAgent itself clamps the pull's cap, whatever a
+// box's settings hand it.
+func TestNewAgentClampsMaxPull(t *testing.T) {
+ t.Parallel()
+ fx := newFixture(t, 1)
+ for give, want := range map[int64]int64{0: MaxPullBytes, -1: MaxPullBytes, 1 << 20: 1 << 20, MaxPullBytes + 1: MaxPullBytes} {
+ a, _ := fx.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) { c.MaxPull = give })
+ if a.cfg.MaxPull != want {
+ t.Errorf("MaxPull %d became %d, want %d", give, a.cfg.MaxPull, want)
+ }
+ }
+}
+```
+
+In plan 1's `internal/dbreads/fixture_test.go`, let `inProcWorkers` record the type of every frame the agent writes. `writeFrame` writes each frame in a single `Write`, so the first byte of each `Write` on the agent's end is a frame type:
+
+```go
+// frameTap records the first byte of each Write the agent makes: one frame
+// type per frame (writeFrame writes a frame in one Write).
+type frameTap struct {
+ net.Conn
+ w *inProcWorkers
+}
+
+func (f frameTap) Write(p []byte) (int, error) {
+ if len(p) > 0 {
+ f.w.mu.Lock()
+ f.w.sent = append(f.w.sent, p[0])
+ f.w.mu.Unlock()
+ }
+ return f.Conn.Write(p)
+}
+
+func (w *inProcWorkers) sentFrames() []byte {
+ w.mu.Lock()
+ defer w.mu.Unlock()
+ return append([]byte(nil), w.sent...)
}
```
+with `sent []byte` added to `inProcWorkers` beside `liveAtDial`, and both of `dial`'s `return agentEnd, nil` lines becoming `return frameTap{Conn: agentEnd, w: w}, nil`.
+
Append to `internal/host/dbreads_test.go`:
```go
@@ -614,7 +678,7 @@ func TestPullCapOnlyLowers(t *testing.T) {
- [ ] **Step 2: Run them to verify they fail**
-Run (sandbox off): `go test ./internal/dbreads/ -run 'TestPull' -count=1 && go test ./internal/host/ -run 'TestPullCapOnlyLowers' -count=1`
+Run (sandbox off): `go test ./internal/dbreads/ -run 'TestPull|TestNewAgentClampsMaxPull' -count=1 && go test ./internal/host/ -run 'TestPullCapOnlyLowers' -count=1`
Expected: FAIL.
- [ ] **Step 3: Write the agent's pull**
@@ -680,10 +744,16 @@ In `internal/host/dbreads.go`, add `maxPull int64` to `dbReadsEnv`, parse `CARLO
Run (sandbox off): `go test ./internal/dbreads/ -count=1 -race && go test ./internal/host/ -run 'TestDBReads|TestPullCap' -count=1`
Expected: PASS.
-- [ ] **Step 5: Commit**
+- [ ] **Step 5: Add the pull to the root-runner tests**
+
+Now that the agent makes a pull, in `internal/host/dbreads_rootrunner_linux_test.go` add `TestRootDBReadPullThroughTheInstalledUnits`: as `TestRootDBReadServesThroughTheInstalledUnits`, but `r.submit(dbreads.OpPull, "")` (plan 1's `submit` already takes the op), and the answer is checked by opening every chunk with `dbreads.StreamLTX`, comparing the concatenation's SHA-256 with the summary's, and comparing it byte for byte with the planned files read from `r.dir/replica/instances/shop.example.com` in plan order. After it, `privateTmpGone(t, 10*time.Second)`.
+
+Build it (sandbox off): `GOOS=linux GOARCH=arm64 go test -c -tags rootrunner -o "$TMPDIR/dbf3-rr.test" ./internal/host && rm "$TMPDIR/dbf3-rr.test"`. Expected: a clean build. Run `TestRootDBReadPullThroughTheInstalledUnits` on the root runner (plan 1 Task 15 Step 4's command) before the branch is reviewed, and record the run id.
+
+- [ ] **Step 6: Commit**
```bash
-git add internal/dbreads internal/host/dbreads.go internal/host/dbreads_test.go
+git add internal/dbreads internal/host/dbreads.go internal/host/dbreads_test.go internal/host/dbreads_rootrunner_linux_test.go
git commit -m "dbreads: the agent's pull: its cap, its reservation, the ltx stream and the file list"
```
@@ -818,16 +888,22 @@ Waits for Task 2 (DBF-17, DBF-18, DBF-24 approved).
- Modify: `approved_db_forward_copy_test.go` (DBF-17, DBF-18, DBF-24)
**Interfaces:**
-- Consumes: plan 2's `readDBForward`, `openDBForward`, `errDBForward`; `dbdoor.RestoreLocal`, `dbdoor.Restored`, `lsfile.NewReplicaClient`, `litestream.LTXFilePath`; `copyOut`, `dbPullHolds`, `dbRestoreError`, `humanBytes`.
-- Produces: `type dbChunkSink interface { add([]byte) error; count() int; chunk(i int) ([]byte, error) }`; `memSink`, `fileSink{dir string}`; `readDBForward(ctx, body io.Reader, key *ecdh.PrivateKey, host, region, reading string, status io.Writer, sink dbChunkSink, max int64) (string, *dbreads.Result, error)` (the id and the result; `openDBForward` is called by the query path); `openDBPull(ctx, id string, key *ecdh.PrivateKey, sink dbChunkSink, res dbreads.Result, host, region, dir string) (*dbdoor.Restored, error)`; `dbPullForward(ctx context.Context, t dbTarget, fwd dbdoor.Forward) (*dbdoor.Restored, error)`; `dbfPullTooLarge`, `dbfFetching`, `dbfPullNoRoom`.
+- Consumes: plan 2's `readDBForward`, `openDBForward`, `errDBForward`, `newStream`, `fwdID`; `newConsoleClient`, `apiSession`, `apiAcct`, `dbTarget` (main); `dbdoor.ForwardBody`; `ltx.HeaderSize`, `ltx.TrailerSize`, `litestream.SnapshotLevel`; `dbdoor.RestoreLocal`, `dbdoor.Restored`, `lsfile.NewReplicaClient`, `litestream.LTXFilePath`; `copyOut`, `dbPullHolds`, `dbRestoreError`, `humanBytes`.
+- Produces: `type dbChunkSink interface { add([]byte) error; count() int; chunk(i int) ([]byte, error) }`; `memSink`, `fileSink{dir string}`; `readDBForward(ctx, body io.Reader, key *ecdh.PrivateKey, host, region, reading string, status io.Writer, sink dbChunkSink, max int64) (string, *dbreads.Result, error)` (the id and the result; `openDBForward` is called by the query path); `openDBPull(ctx, id string, key *ecdh.PrivateKey, sink dbChunkSink, res dbreads.Result, host, region, dir string) (*dbdoor.Restored, error)`; `dbPullForward(ctx context.Context, t dbTarget, fwd dbdoor.Forward) (*dbdoor.Restored, error)`; `checkPullManifest(s dbreads.Summary) error`; `var dbPullSpoolWrite = os.WriteFile`; `dbfPullTooLarge`, `dbfFetching`, `dbfPullNoRoom`; test helpers `pullStream`, `pullStreamN`, `readPull`, `readPullIn`, `standInConsole`, `leftBehind`.
- [ ] **Step 1: Write the failing tests**
Append to `dbforward_test.go` (crafted streams with real sealing: unit tests; Task 9 runs the binary):
```go
-// pullStream is a pull's body for the files of dir's replica.
+// pullStream is a pull's body for the files of dir's replica, in one chunk.
func pullStream(t *testing.T, key *ecdh.PrivateKey, dir string, mutate func(s *dbreads.Summary, files [][]byte)) *stream {
+ t.Helper()
+ return pullStreamN(t, key, dir, 1, mutate)
+}
+
+// pullStreamN is pullStream with the files' bytes split across n chunks.
+func pullStreamN(t *testing.T, key *ecdh.PrivateKey, dir string, n int, mutate func(s *dbreads.Summary, files [][]byte)) *stream {
t.Helper()
rd := lsfile.NewReplicaClient(dir)
plan, err := dbdoor.PlanFrom(context.Background(), rd, time.Time{})
@@ -849,15 +925,24 @@ func pullStream(t *testing.T, key *ecdh.PrivateKey, dir string, mutate func(s *d
mutate(&s, files)
}
st := newStream(t, key)
- sealed, _ := dbreads.SealChunk(key.PublicKey(), fwdID, 1, dbreads.StreamLTX, true, all.Bytes())
- st.line(dbreads.Line{Chunk: 1, Data: sealed})
- st.result(dbreads.StateDone, "", 1, s)
+ b := all.Bytes()
+ per := (len(b) + n - 1) / n
+ for seq := 1; seq <= n; seq++ {
+ part := b[min((seq-1)*per, len(b)):min(seq*per, len(b))]
+ sealed, _ := dbreads.SealChunk(key.PublicKey(), fwdID, int64(seq), dbreads.StreamLTX, seq == n, part)
+ st.line(dbreads.Line{Chunk: int64(seq), Data: sealed})
+ }
+ st.result(dbreads.StateDone, "", int64(n), s)
return st
}
func readPull(t *testing.T, st *stream, key *ecdh.PrivateKey) (*dbdoor.Restored, error) {
t.Helper()
- dir := t.TempDir()
+ return readPullIn(t, st, key, t.TempDir())
+}
+
+func readPullIn(t *testing.T, st *stream, key *ecdh.PrivateKey, dir string) (*dbdoor.Restored, error) {
+ t.Helper()
sink := &fileSink{dir: filepath.Join(dir, "sealed")}
id, res, err := readDBForward(context.Background(), &st.buf, key, "shop.example.com", "uk", dbfFetching, io.Discard, sink, dbreads.MaxPullBytes)
if err != nil {
@@ -883,55 +968,152 @@ func TestForwardedPullRestoresTheFiles(t *testing.T) {
}
}
-// Review Focus 3: files that do not add up are DBF-13, with nothing
-// restored.
+// Review Focus 3 and plan review findings 22 and 24: a file list that does
+// not add up is DBF-13, never a panic, and no file is created before the
+// list is checked whole. Mutations: drop checkPullManifest's per-file size
+// bounds (red: "sizes -1 and N+1" panics or creates files); drop its sum
+// (red: "sizes that do not sum" creates the first file before failing);
+// drop the duplicate check (red: "a file twice" creates a file); drop the
+// level bound (red: "level 12").
func TestForwardedPullRefusesFilesThatDoNotAddUp(t *testing.T) {
t.Parallel()
dir := t.TempDir()
dbdoortest.Replicate(t, dir, 3)
for name, mutate := range map[string]func(s *dbreads.Summary, files [][]byte){
- "sizes that do not sum": func(s *dbreads.Summary, _ [][]byte) { s.Files[0].Size++ },
+ "sizes that do not sum": func(s *dbreads.Summary, _ [][]byte) { s.Files[0].Size++ },
"a list that ends elsewhere": func(s *dbreads.Summary, _ [][]byte) { s.TXID++ },
- "no files": func(s *dbreads.Summary, _ [][]byte) { s.Files = nil },
+ "no files": func(s *dbreads.Summary, _ [][]byte) { s.Files = nil },
+ "sizes -1 and N+1": func(s *dbreads.Summary, _ [][]byte) {
+ s.Files[1].Size += s.Files[0].Size + 1
+ s.Files[0].Size = -1
+ },
+ "a file twice": func(s *dbreads.Summary, _ [][]byte) {
+ s.Files[1] = s.Files[0]
+ s.Bytes = 0
+ for _, f := range s.Files {
+ s.Bytes += f.Size
+ }
+ },
+ "level 12": func(s *dbreads.Summary, _ [][]byte) { s.Files[0].Level = 12 },
+ "a range that runs back": func(s *dbreads.Summary, _ [][]byte) { s.Files[0].MinTXID = s.Files[0].MaxTXID + 1 },
+ "a size past the pull's cap": func(s *dbreads.Summary, _ [][]byte) { s.Files[0].Size = dbreads.MaxPullBytes + 1 },
} {
key, _ := dbreads.NewReplyKey()
- if _, err := readPull(t, pullStream(t, key, dir, mutate), key); err == nil || err.Error() != fmt.Sprintf(dbfUnchecked, "uk") {
- t.Errorf("%s: %v", name, err)
+ out := t.TempDir()
+ func() {
+ defer func() {
+ if p := recover(); p != nil {
+ t.Errorf("%s: panic %v", name, p)
+ }
+ }()
+ if _, err := readPullIn(t, pullStream(t, key, dir, mutate), key, out); err == nil || err.Error() != fmt.Sprintf(dbfUnchecked, "uk") {
+ t.Errorf("%s: %v", name, err)
+ }
+ }()
+ if _, err := os.Stat(filepath.Join(out, "replica")); !os.IsNotExist(err) {
+ t.Errorf("%s: a file was created before the list was checked (%v)", name, err)
}
}
}
-// Review Focus 1: a spool that cannot be written fails cleanly and leaves
-// nothing behind.
+// standInConsole answers the list door with the forward marker offering a
+// pull, and the reads route with body (then holds the request open until
+// the client goes when hold is set). It is a stand-in for the console, so
+// that these tests run dbPull itself.
+func standInConsole(t *testing.T, body []byte, hold bool) dbTarget {
+ t.Helper()
+ ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if !strings.HasSuffix(r.URL.Path, "/reads") {
+ w.Header().Set("Content-Type", "application/json")
+ w.WriteHeader(http.StatusConflict)
+ json.NewEncoder(w).Encode(dbdoor.ForwardBody{Error: "shop.example.com keeps its data in region uk.",
+ Forward: 1, Region: "uk", ForwardOps: []string{dbreads.OpQuery, dbreads.OpPull}})
+ return
+ }
+ io.Copy(io.Discard, r.Body)
+ w.Header().Set("Content-Type", "application/x-ndjson")
+ w.WriteHeader(http.StatusOK)
+ w.Write(body)
+ w.(http.Flusher).Flush()
+ if hold {
+ <-r.Context().Done()
+ }
+ }))
+ t.Cleanup(ts.Close)
+ return dbTarget{sess: &apiSession{c: newConsoleClient(ts.URL, "tok"), console: ts.URL}, acct: apiAcct{Seg: "acme"},
+ app: "shop", host: "shop.example.com"}
+}
+
+// leftBehind is what a pull left in its temp dir and at --out.
+func leftBehind(t *testing.T, tmp, out string) {
+ t.Helper()
+ if _, err := os.Stat(out); !os.IsNotExist(err) {
+ t.Errorf("something is at --out: %v", err)
+ }
+ if left, _ := filepath.Glob(filepath.Join(tmp, "carlos-db-*")); len(left) != 0 {
+ t.Errorf("the pull's directory is still there: %v", left)
+ }
+}
+
+// Review Focus 1 and plan review finding 23: a spool that fails part way,
+// through dbPull itself, leaves nothing at --out and removes its directory.
+// Not parallel: it sets TMPDIR and dbPullSpoolWrite. Mutation: drop
+// dbPullForward's deferred RemoveAll (red: carlos-db-* is left).
func TestForwardedPullLeavesNothingWhenTheSpoolFails(t *testing.T) {
- t.Parallel()
+ tmp := t.TempDir()
+ t.Setenv("TMPDIR", tmp)
+ writes := 0
+ dbPullSpoolWrite = func(name string, b []byte, mode os.FileMode) error {
+ if writes++; writes == 3 {
+ return errors.New("no space left on device")
+ }
+ return os.WriteFile(name, b, mode)
+ }
+ t.Cleanup(func() { dbPullSpoolWrite = os.WriteFile })
dir := t.TempDir()
- dbdoortest.Replicate(t, dir, 1)
+ dbdoortest.Replicate(t, dir, 3)
+ // The stand-in's chunks are sealed to a key the CLI does not hold; the
+ // pull fails at the spool, before anything is opened.
key, _ := dbreads.NewReplyKey()
- st := pullStream(t, key, dir, nil)
- blocked := filepath.Join(t.TempDir(), "sealed")
- if err := os.WriteFile(blocked, nil, 0o600); err != nil { // a file where the spool's directory should be
- t.Fatal(err)
+ st := pullStreamN(t, key, dir, 4, nil)
+ out := filepath.Join(t.TempDir(), "copy.db")
+ if err := dbPull(context.Background(), standInConsole(t, st.buf.Bytes(), false), out); err == nil {
+ t.Fatal("a spool that failed was accepted")
}
- _, _, err := readDBForward(context.Background(), &st.buf, key, "shop.example.com", "uk", dbfFetching, io.Discard, &fileSink{dir: blocked}, dbreads.MaxPullBytes)
- if err == nil {
- t.Fatal("a spool that cannot be written was accepted")
+ if writes != 3 {
+ t.Fatalf("%d spool writes: the failure was not part way", writes)
}
+ leftBehind(t, tmp, out)
}
-// Review Focus 4: Ctrl-C during a pull is "stopped", and dbPullForward's
-// spool directory is removed on that path (its deferred removal; checked
-// end to end in Task 9's binary test).
+// Review Focus 4 and plan review finding 23: Ctrl-C while chunks are
+// spooling, through dbPull itself, is "stopped", with nothing at --out and
+// the directory removed. Not parallel: it sets TMPDIR and dbPullSpoolWrite.
+// Mutation: drop dbPullForward's deferred RemoveAll (red). The binary's
+// SIGINT is Task 9's TestDBForwardPullInterruptedLeavesNothing.
func TestForwardedPullCtrlCLeavesNothing(t *testing.T) {
- t.Parallel()
- key, _ := dbreads.NewReplyKey()
- st := newStream(t, key)
+ tmp := t.TempDir()
+ t.Setenv("TMPDIR", tmp)
ctx, cancel := context.WithCancel(context.Background())
- cancel()
- _, _, err := readDBForward(ctx, &cutReader{r: &st.buf}, key, "shop.example.com", "uk", dbfFetching, io.Discard, &memSink{}, dbreads.MaxPullBytes)
+ defer cancel()
+ dbPullSpoolWrite = func(name string, b []byte, mode os.FileMode) error {
+ err := os.WriteFile(name, b, mode)
+ cancel() // Ctrl-C once the first chunk is on disk
+ return err
+ }
+ t.Cleanup(func() { dbPullSpoolWrite = os.WriteFile })
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ key, _ := dbreads.NewReplyKey()
+ st := pullStreamN(t, key, dir, 4, nil)
+ // Admitted and the first chunk only, then the stand-in holds the request.
+ lines := bytes.SplitAfter(st.buf.Bytes(), []byte("\n"))
+ out := filepath.Join(t.TempDir(), "copy.db")
+ err := dbPull(ctx, standInConsole(t, bytes.Join(lines[:2], nil), true), out)
if err == nil || err.Error() != "stopped" {
t.Fatalf("%v", err)
}
+ leftBehind(t, tmp, out)
}
```
@@ -951,7 +1133,7 @@ func readFwd(t *testing.T, s *stream, key *ecdh.PrivateKey) (dbForwardAnswer, st
}
```
-and `TestForwardedQueryCtrlCPrintsNothing` calls `readDBForward` with `dbfReading, &status, &memSink{}, dbreads.MaxOutputBytes` and checks only its error. Add imports `"database/sql"`, `"io"`, `"path/filepath"`, `lsfile`, `ltx`, `dbdoortest`.
+and `TestForwardedQueryCtrlCPrintsNothing` calls `readDBForward` with `dbfReading, &status, &memSink{}, dbreads.MaxOutputBytes` and checks only its error. Add imports `"database/sql"`, `"errors"`, `"io"`, `"net/http"`, `"net/http/httptest"`, `"path/filepath"`, `lsfile`, `ltx`, `dbdoortest`. Check before writing `standInConsole` that `newStream` (plan 2) writes the `admitted` line first, so `lines[:2]` is admitted and chunk 1; and that `dbRestore` turns the marker into `errDBForward` from the list door's first GET, whatever its path (plan 2 Task 8).
- [ ] **Step 2: Run them to verify they fail**
@@ -1000,7 +1182,7 @@ func (f *fileSink) add(b []byte) error {
}
}
f.n++
- return os.WriteFile(filepath.Join(f.dir, strconv.Itoa(f.n)), b, 0o600)
+ return dbPullSpoolWrite(filepath.Join(f.dir, strconv.Itoa(f.n)), b, 0o600)
}
func (f *fileSink) count() int { return f.n }
@@ -1087,11 +1269,7 @@ func openDBPull(ctx context.Context, id string, key *ecdh.PrivateKey, sink dbChu
_, err := openDBForward(id, key, sink, res, host, region)
return nil, err
}
- var sizes int64
- for _, f := range s.Files {
- sizes += f.Size
- }
- if res.LastSeq != int64(sink.count()) || len(s.Files) == 0 || sizes != s.Bytes || s.Files[len(s.Files)-1].MaxTXID != s.TXID {
+ if res.LastSeq != int64(sink.count()) || checkPullManifest(s) != nil {
return nil, unchecked
}
root := filepath.Join(dir, "replica")
@@ -1152,6 +1330,44 @@ func openDBPull(ctx context.Context, id string, key *ecdh.PrivateKey, sink dbChu
return &dbdoor.Restored{Path: db, Dir: dir, TXID: s.TXID, At: s.At}, nil
}
+// checkPullManifest is the summary's file list checked whole before any
+// file is created (plan review finding 22): every size a possible LTX file
+// and within the cap, every level one litestream writes, every range a
+// range, no file twice, sizes that sum without overflow to the summary's
+// byte count, and a last file that ends at its transaction.
+func checkPullManifest(s dbreads.Summary) error {
+ bad := errors.New("the file list does not add up")
+ if len(s.Files) == 0 || s.Bytes <= 0 || s.Bytes > dbreads.MaxPullBytes {
+ return bad
+ }
+ type coord struct {
+ level int
+ min, max uint64
+ }
+ seen := make(map[coord]bool, len(s.Files))
+ var sum int64
+ for _, f := range s.Files {
+ c := coord{f.Level, f.MinTXID, f.MaxTXID}
+ switch {
+ case f.Size < ltx.HeaderSize+ltx.TrailerSize, f.Size > dbreads.MaxPullBytes,
+ f.Level < 0, f.Level > litestream.SnapshotLevel,
+ f.MinTXID < 1, f.MaxTXID < f.MinTXID, seen[c],
+ sum > dbreads.MaxPullBytes-f.Size: // sum + f.Size would pass the cap
+ return bad
+ }
+ seen[c] = true
+ sum += f.Size
+ }
+ if sum != s.Bytes || s.Files[len(s.Files)-1].MaxTXID != s.TXID {
+ return bad
+ }
+ return nil
+}
+
+// dbPullSpoolWrite writes one spooled chunk: os.WriteFile, a seam so a test
+// can fail the spool part way (plan review finding 23).
+var dbPullSpoolWrite = os.WriteFile
+
// dbPullForward is a pull through the forwarded read. The spool, the files
// and the copy live in one private 0700 directory under the system temp
// dir, removed on every path but success, where the caller removes it after
@@ -1364,6 +1580,59 @@ func TestDBForwardPullSavesNothingForABadAnswer(t *testing.T) {
}
}
+// Review Focus 4 and plan review finding 23, through the binary: SIGINT
+// while the CLI is spooling a pull prints "stopped", saves nothing at --out
+// and removes the pull's directory. The scripted box (a stand-in) writes two
+// chunks and never a result, so the CLI is mid-pull when the signal lands.
+// Mutation: drop dbPullForward's deferred RemoveAll (red: carlos-db-* is
+// left in the CLI's TMPDIR).
+func TestDBForwardPullInterruptedLeavesNothing(t *testing.T) {
+ e := newDBFE2E(t, dbfE2EOptions{forward: true, noBox: true})
+ stop := e.scriptBox(t, func(b *e2eScript) {
+ b.picked(time.Now().Add(time.Minute))
+ for seq := int64(1); seq <= 2; seq++ {
+ sealed, _ := dbreads.SealChunk(b.pub, b.id, seq, dbreads.StreamLTX, false, bytes.Repeat([]byte{'x'}, 4096))
+ b.store.Put(context.Background(), dbreads.ChunkKey(b.id, seq), sealed, "")
+ }
+ })
+ defer stop()
+ spool := t.TempDir()
+ out := filepath.Join(t.TempDir(), "far.db")
+ cmd := exec.Command(e.bin, "db", "pull", "--app", "shop", "--instance", dbfE2EFar, "--out", out)
+ // The last TMPDIR wins (os/exec keeps the last of a duplicated key).
+ cmd.Env = append(carlosTestEnv(t, append([]string{"CARLOS_CREDENTIALS=" + e.creds}, e.env...)...), "TMPDIR="+spool)
+ cmd.Dir = t.TempDir()
+ var stderr bytes.Buffer
+ cmd.Stderr = &stderr
+ if err := cmd.Start(); err != nil {
+ t.Fatal(err)
+ }
+ deadline := time.Now().Add(30 * time.Second)
+ for {
+ if got, _ := filepath.Glob(filepath.Join(spool, "carlos-db-*", "sealed", "2")); len(got) == 1 {
+ break
+ }
+ if time.Now().After(deadline) {
+ cmd.Process.Kill()
+ cmd.Wait()
+ t.Fatalf("the CLI never spooled two chunks: %s", stderr.String())
+ }
+ time.Sleep(50 * time.Millisecond)
+ }
+ if err := cmd.Process.Signal(os.Interrupt); err != nil {
+ t.Fatal(err)
+ }
+ if err := cmd.Wait(); err == nil || !strings.Contains(stderr.String(), "stopped") {
+ t.Fatalf("%v: %s", err, stderr.String())
+ }
+ if _, err := os.Stat(out); !os.IsNotExist(err) {
+ t.Fatalf("something was saved at --out: %v", err)
+ }
+ if left, _ := filepath.Glob(filepath.Join(spool, "carlos-db-*")); len(left) != 0 {
+ t.Fatalf("the pull's directory is still there: %v", left)
+ }
+}
+
// A console from plan 2 forwards queries and not pulls. Its marker is
// played here by a wrapper that drops "pull" from forward_ops on the way
// out: the new CLI says DBF-8 and saves nothing.
@@ -1437,6 +1706,8 @@ with `"crypto/sha256"` and `"encoding/hex"` added to that file's imports.
- [ ] **Step 2: Run them**
+Add `"os/exec"` to the file's imports if plan 2's harness does not already import it.
+
Run (sandbox off): `go test . -run 'TestDBForwardPull' -count=1 -v`
Expected: PASS. The MinIO subtest runs in CI's MinIO step.
@@ -1468,14 +1739,18 @@ In a tree copy under `$TMPDIR`, as plans 1 and 2 did; every row must turn red:
| Mutation | Test that must fail |
|---|---|
| `holdAndSend` skips `verifyLTX` | `TestWorkerHoldRefusesACorruptFile` |
-| the agent's pull sends `run` before check B | `TestPullTooLargeAndNoChunkBeforeCheckB` |
+| the agent's pull sends `run` before check B | `TestPullTooLargeAndNoChunkBeforeCheckB` (the recorded frames hold a `run`) |
| `openDBPull` skips the hash comparison | `TestDBForwardPullSavesNothingForABadAnswer` |
-| `openDBPull` skips the file-size sum | `TestForwardedPullRefusesFilesThatDoNotAddUp` |
-| `NewAgent` keeps a `MaxPull` above `MaxPullBytes` | `TestPullCapOnlyLowers` (through `dbReadsSettings`) |
+| `checkPullManifest` drops the sum | `TestForwardedPullRefusesFilesThatDoNotAddUp` ("sizes that do not sum": a file is created before the failure) |
+| `checkPullManifest` drops the per-file size bounds | `TestForwardedPullRefusesFilesThatDoNotAddUp` ("sizes -1 and N+1": a panic, or a file created) |
+| `checkPullManifest` drops the duplicate or the level check | `TestForwardedPullRefusesFilesThatDoNotAddUp` ("a file twice", "level 12") |
+| `NewAgent` keeps a `MaxPull` above `MaxPullBytes` | `TestNewAgentClampsMaxPull` (plan review finding 24: `TestPullCapOnlyLowers` reads only `dbReadsSettings`, which clamps on its own) |
+| `dbReadsSettings` keeps a `CARLOS_DB_PULL_MAX_BYTES` above the cap | `TestPullCapOnlyLowers` |
+| `dbPullForward` drops its deferred `RemoveAll` | `TestForwardedPullLeavesNothingWhenTheSpoolFails`, `TestForwardedPullCtrlCLeavesNothing`, `TestDBForwardPullInterruptedLeavesNothing` |
- [ ] **Step 2: Merge main, census, 9B8**
-Merge main; run plan 1 Task 17's dump with the prefixes `dbforward.go`, `db.go`, `internal/dbreads/`, `internal/console/apidbforward`, `internal/host/dbreads`; write `plan14-root-reads-db-region-forward-3.tsv` (`rev3=PROPOSED-db-region-forward-3`) with plans 1 and 2's basis templates: worker units as the db-read worker at J4 ("verifies and holds the planned files in its private scratch and sends them back after the agent's check B"), agent and console units re-frozen at their plan 1 and plan 2 dispositions with the pull appended, CLI units as the carlos CLI at J4 ("spools sealed chunks to a 0700 directory under the system temp dir, writes the opened files as a litestream file replica there, restores and checks them with dbdoor.RestoreLocal, and removes the directory on every path"); envelopes re-pinned. Load it last with the previous seal checked first; set the new seal. Run `TestPlan9B8RecordWriteCensus`; re-pin any changed entry at its class in a `dbRegionForward3WriteCensusInventory` layered on plan 2's. Write the census proposal and post it on #280.
+Merge main; run plan 1 Task 17's dump with the prefixes `dbforward.go`, `db.go`, `internal/dbreads/`, `internal/console/apidbforward`, `internal/host/dbreads`; write `plan14-root-reads-db-region-forward-3.tsv` (`rev3=PROPOSED-db-region-forward-3`; each `@legacy` fingerprint computed by plan 1's `TestZZDBForwardFingerprints` from the row's own fields) with plans 1 and 2's basis templates: worker units as the db-read worker at J4 ("verifies and holds the planned files in its private scratch and sends them back after the agent's check B"), agent and console units re-frozen at their plan 1 and plan 2 dispositions with the pull appended, CLI units as the carlos CLI at J4 ("spools sealed chunks to a 0700 directory under the system temp dir, writes the opened files as a litestream file replica there, restores and checks them with dbdoor.RestoreLocal, and removes the directory on every path"); envelopes re-pinned. Load it last with the previous seal checked first; set the new seal. Run `TestPlan9B8RecordWriteCensus`; re-pin any changed entry at its class in a `dbRegionForward3WriteCensusInventory` layered on plan 2's. Write the census proposal and post it on #280.
- [ ] **Step 3: Test weights**
diff --git a/docs/superpowers/specs/2026-10-22-db-region-forward-design.md b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md
index b471c9f76..0aaf635e8 100644
--- a/docs/superpowers/specs/2026-10-22-db-region-forward-design.md
+++ b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md
@@ -1,6 +1,13 @@
# `carlos db` for instances whose data lives in another region
-**Status:** rev 5, 2026-10-22. Rev 5 answers Astra's round 4
+**Status:** rev 6, 2026-10-22. Rev 6 changes no mechanism: it states four
+things the plan review (`CODEX-FINDINGS-DB-REGION-FORWARD-PLANS.md`) found
+the plans had to decide, each marked **(plan review N)**: what a result
+names when a chunk cannot be confirmed (§5.4, 9), a box's `cancel` read
+before it elects (§5.5 step 4, §5.8, 11), a retry's reservation (§5.7, 10),
+and the order of the audit row, the creates and the serial bump, with
+nothing but `cancel` once the request has ended (§5.5 step 3, §9.3, 11).
+Rev 5 answers Astra's round 4
(`CODEX-FINDINGS-DB-REGION-FORWARD-R4.md`), which found J1 to J5 fixed, with
four direct fixes and no new mechanism: flushed progress lines and the
backhaul header bound (codex 19), the result held until its chunks are out
@@ -262,7 +269,11 @@ read, anything else means it steps away, and a read that still fails is
treated as not owned (J3, codex 16). The box keeps each sealed chunk's
bytes until its create is confirmed; a retry that answers "already exists"
reads the object back and compares bytes, equal meaning its earlier attempt
-landed. It writes `result.json` only when every chunk is confirmed.
+landed. It writes `result.json` only when every chunk is confirmed. A chunk
+it cannot confirm before its deadline is never named: the box stops the
+read and writes a `failed` result with `interrupted` whose `last_seq` is
+the last confirmed chunk, so the result names only objects that exist
+**(plan review 9)**.
### 5.5 The request and its response
@@ -317,13 +328,18 @@ The envelope is unchanged from rev 3:
rail, pooled, fleet), refuses a moving record, checks `checkQuerySQL`
again, checks capability (§7.1), and takes its admission counts (J5).
Any refusal so far is an ordinary status and JSON error. Then it mints
- the id, creates the request and the queue entry, bumps the serial,
- writes the audit row (§9.3), and starts a `200` response of
+ the id, writes the audit row (§9.3), creates the request and the queue
+ entry, bumps the serial, and starts a `200` response of
newline-delimited JSON. From here every outcome is a line in that body.
+ If the request has ended by the time the queue entry is down, the
+ console writes `cancel` and nothing else: no serial bump, no `200`
+ **(plan review 11)**.
4. Each capable box lists its region's queue for this hour and the last
when the serial moves, and every 30 s if it does not. For an in-window
- entry it has not decided, it reads and verifies the request, checks its
- own admission (§5.7), waits a random 0 to 500 ms and creates `picked`.
+ entry it has not decided, it reads and verifies the request, skips it if
+ `cancel` exists **(plan review 11)**, skips it while the box is draining
+ (§7.1), checks its own admission (§5.7), waits a random 0 to 500 ms,
+ checks draining again, and creates `picked`.
5. The owner runs §8.2's identity bracket around planning and download,
runs the query in a worker, and writes sealed chunks as the output
arrives. It checks for `cancel` every 5 s. When the worker ends, it
@@ -508,6 +524,12 @@ gone. There is no timeout escape: an orphan worker stops within 5 s of
losing its connection (§8.3), and `RuntimeMaxSec` stops it in the end
regardless.
+A read's one retry (a planned file that vanished, §8.2) is a second
+worker. Its first attempt's reservation stays in the table, under its own
+key, until that attempt's unit has stopped, and the retry dials only once
+the unit count says so; the retry takes a fresh reservation from the floor.
+Closing the first connection is not the unit's stop **(plan review 10)**.
+
**Restart reconciliation (codex 8).** A restarted agent's table is empty,
and it cannot know the sizes of workers it did not start. So it reconciles
by refusal, against the units actually running: at start, and before every
@@ -520,8 +542,9 @@ before the restart has stopped and systemd has removed its scratch.
### 5.8 Cost
Each capable box does one LIST of its region's current and previous hour of
-queue entries per serial move and every 30 s, and one GET per new in-window
-entry. A read adds two console PUTs, one `picked`, one PUT per chunk and one
+queue entries per serial move and every 30 s, and two GETs per new
+in-window entry (the request, and `cancel` before electing, plan review 11).
+A read adds two console PUTs, one `picked`, one PUT per chunk and one
result, and the console's own GETs and LISTs about once a second while the
request is open. There are no deletes. The October bill was LIST and GET
polling, so plan 1 measures the LIST rate on dev before the flagship roll
@@ -837,7 +860,12 @@ A console with `CARLOS_DB_FORWARD` unset answers byte-identically to main.
### 9.3 Audit
One `db-read` activity row at admission, in the run's manner, with the
-detail DBF-21 or DBF-22. It records the SQL's length in bytes and the first
+detail DBF-21 or DBF-22. It is written before the request and the queue
+entry, while the request is open, so a read a box can see always has its
+row; a row the console cannot write refuses the read. A request that ends
+after the row and before the `200` leaves the row of an admission, and at
+most the request, the queue entry and `cancel`; never a serial bump
+**(plan review 11)**. It records the SQL's length in bytes and the first
16 hex digits of its SHA-256, not its text (ruling 3). A member
who kept the statement can match it to the row; the row itself holds none
of the data a query's literals can carry. That differs from `carlos run`,
diff --git a/docs/superpowers/specs/CODEX-REVIEW-BRIEF-DB-REGION-FORWARD-PLANS.md b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-DB-REGION-FORWARD-PLANS.md
index 292a9b584..fe1006065 100644
--- a/docs/superpowers/specs/CODEX-REVIEW-BRIEF-DB-REGION-FORWARD-PLANS.md
+++ b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-DB-REGION-FORWARD-PLANS.md
@@ -1,10 +1,12 @@
-# Review brief: the implementation plans for `carlos db` region forward
+# Review brief, round 2: the implementation plans for `carlos db` region forward
You are an independent second reader of three implementation plans for the
-CARLOS platform (this repository). Read them cold and adversarially, against
-the spec they implement and against the code on `main`. Nothing is built.
+CARLOS platform (this repository). Round 1
+(`CODEX-FINDINGS-DB-REGION-FORWARD-PLANS.md`) found 24 findings (8 P0, 4 P1,
+12 P2) and asked for all three plans to be revised. This round checks the
+revisions. Nothing is built.
-**Plans:**
+**Plans**, at the commit named in the prompt:
- `docs/superpowers/plans/2026-10-22-db-region-forward-plan-1-box.md`: the
box side and the wire format, shipped dark behind the `db-read-1`
@@ -14,159 +16,102 @@ the spec they implement and against the code on `main`. Nothing is built.
- `docs/superpowers/plans/2026-10-22-db-region-forward-plan-3-pull.md`:
`pull` and `pull --at` (10 tasks).
-**Spec:** `docs/superpowers/specs/2026-10-22-db-region-forward-design.md`,
-rev 5 plus the ruling commit `fe27bd9c`, at the commit named in the prompt.
-Astra accepted rev 5 in round 5 (`CODEX-FINDINGS-DB-REGION-FORWARD-R5.md`).
-Read `AGENTS.md` first, then the spec, then the plans.
+Each plan now opens with a "Plan review, round 1" section saying where each
+of its findings landed.
-Each plan is written to be executed task by task by an engineer who knows
-Go and nothing about this repository. A task's code is meant to compile
-and its tests to pass as written, against `main` plus the plans before it.
+**Spec:** `docs/superpowers/specs/2026-10-22-db-region-forward-design.md`,
+rev 6. Rev 6 is rev 5 (accepted in round 5, approved by Paul on 2026-10-22,
+rulings in §16) plus four clarifications the plan review forced, each marked
+**(plan review N)**: §5.4 (what a result names when a chunk cannot be
+confirmed), §5.5 steps 3 and 4 and §5.8 (the audit row first, `cancel` only
+once a request has ended, and a box's `cancel` read before it elects), §5.7
+(a retry's reservation), §9.3 (the audit row's order). No mechanism changed.
+Read `AGENTS.md`, then the spec's status header and the four marked places,
+then the plans.
## Fixed: do not re-argue
-- **The spec's design.** It was reviewed over five rounds and accepted. Do
- not propose a different design. Report a plan that fails to build what the
- spec says, or builds something the spec rules out.
-- **Paul's rulings (spec §16, 2026-10-22).** Ruling 1: the feature ships
- through the deployment bucket as designed, with the true storage
- statement; a per-region bucket for this traffic is plan 5, a follow-up
- after the feature works. Rulings 2 to 5 stand as written (the limits, the
- audit row, the copy casing, in-region reads moving to the box path in plan
- 4). Plans 4 and 5 are listed, not written; do not review their absence.
-- **The repository's rules**, which every plan carries as Global
- Constraints: production-wired acceptance (real constructors, the real
- units through the real `ensureSystemUnits`, real stores; stand-ins only in
- unit tests and labelled); targeted local test runs and CI as the full run;
- `//go:build linux` plus a `!linux` stub for Linux-only code; copy reviewed
- before it ships, with no em dashes; no tenant names; the default
- `GOCACHE`; a census overlay per branch, loaded last after main's; merge
- main first; test weights for new slow console or root tests; no CI-only
- scripts under `scripts/`; `t.Parallel()` in new host tests unless they
- touch env, cwd, umask or package vars.
+- **The spec's design**, rev 6 included. Report a plan that fails to build
+ what the spec says, or builds something the spec rules out.
+- **Paul's rulings (spec §16).** Plans 4 and 5 are listed, not written.
+- **The repository's rules** each plan carries as Global Constraints.
+- **Round 1's findings as disposed below**, unless a fix is wrong or
+ incomplete: then say so as a new finding, naming the round 1 number.
+
+## Round 1 dispositions
+
+| # | Sev | Plan, task | Disposition |
+|---|---|---|---|
+| 1 | P0 | 1: 7, 12, 13 | Fixed. `SelfView` reads `/proc/self/net/dev` (`parseNetDev`); `net.Interfaces` needs `AF_NETLINK`, refused by `RestrictAddressFamilies=AF_UNIX`. Unobserved facts go in `HelloReply.Errors`; the probe fails on any. Tests `TestParseNetDev`, the probe's "an observation that failed". |
+| 2 | P0 | 1: 15; 2: 7 | Fixed. `waitFor` (`internal/host/housekeeper_test.go:139`) became `dbreadWaitFor`; `hookStore` (`internal/console/temphook_test.go:16`) became `dbReadHookStore`. Checked with `git grep` on `origin/main`. |
+| 3 | P0 | 1: 15 | Fixed. The payload prints one `DBREAD-REPORT ` line; the parent scans for it. |
+| 4 | P0 | 1: 15 | Fixed. `dbreadServiceFile` is `/etc/systemd/system/carlos-dbread@.service`, errors checked, restored in cleanup through `mustSystemctl`. |
+| 5 | P0 | 1: 17 (2, 3: census) | Fixed. `TestZZDBForwardFingerprints` computes each `@legacy` line from the row's seven fields; plans 2 and 3 use it. |
+| 6 | P0 | 2: 4 to 6 | Fixed. `putFixtureObject` reads the etag and replaces with it; `putEdgesFile` and `putBoxStatus` use it. |
+| 7 | P0 | 2: 6 | Fixed. `TestDBReadAdmissionBounds` moved to Task 7, where the stream holds requests open. |
+| 8 | P0 | 3: 4 | Fixed. The installed-unit pull test is Task 5 Step 5, after the agent's pull; `submit` already takes the op (plan 1). |
+| 9 | P1 | 1: 11 | Fixed. `flushLocked` advances `seq` only after the create is confirmed. Test `TestResultNamesOnlyConfirmedChunks` (a chunk store that never confirms; result `failed`/`interrupted`, `last_seq` 1). Spec §5.4 rev 6. |
+| 10 | P1 | 1: 9 to 11 | Fixed. `admission.retry` keeps the first attempt's reservation under `id#attempt`; `settle` waits for the unit count before the retry dials. Tests `TestRetryWaitsForTheFirstWorkerToStop` (a lingering unit, space for one), `TestAdmissionRetryHoldsTheOldReservation`. Spec §5.7 rev 6. |
+| 11 | P1 | 2: 6 (and 1: 12) | Fixed in part. The audit row is written first, while the request is open and before any box can see the read; after the queue entry, an ended request writes `cancel` and returns, with no serial bump; the bump runs on the request's context (`announceDBRead`, not the detached `bumpSerial`). Boxes read `cancel` before electing. Tests `TestDBReadEndedDuringAdmissionWritesOnlyTheCancel`, `TestPassSkipsACancelledRequest`. Spec §5.5 and §9.3 rev 6. Declined: the audit insert's own cancellation (below). |
+| 12 | P1 | 2: 7 | Fixed. Phase deadline contexts; every store call through `dbfCall` (5 s, wait lines while it blocks); the bound checked before the result line; write deadlines clamped to the bound plus 10 s. Tests `TestDBReadStalledStoreKeepsTheStreamAndItsBound`, `TestDBReadSlowRelayEndsAtTheBound`. |
+| 13 | P2 | 2: 6 | Fixed. Read deadline `preHeader` until admission, then the read's bound. Test `TestDBReadStalledBodyIsCutAtThePreHeaderBound`. |
+| 14 | P2 | 1: 10, 12, 13 | Fixed. `AgentConfig.Draining`, from the box's own edges record, checked before admission and after the jitter. Test `TestADrainingBoxDoesNotPick`. Nothing on `main` sets `Draining` yet; the plan says so. |
+| 15 | P2 | 1: 11 | Fixed. Every pre-ready transport failure asks for a probe. Tests `TestAWorkerThatNeverStartsAsksForAProbe`, T13's fail-start case through the installed unit. |
+| 16 | P2 | 1: 15 | Fixed. T12 submits the second read once the first's unit is running. |
+| 17 | P2 | 1: 15, 17 | Fixed. The witness reports the interfaces the unit sees and fails on any but loopback (removing `PrivateNetwork` turns it red); ATTACH targets a real SQLite file. |
+| 18 | P2 | 2: 6, 7 | Fixed. `TestDBReadAdmissionBounds`: both slots freed before pickup and again after, the seventeenth read across nine people, a restart under open reads that writes nothing and starts with empty counts. |
+| 19 | P2 | 2: 12 | Fixed. `dbfGateStore` holds the real box at publication until the console's request has ended, by disconnect and by the bound; late writes are asserted after the end; the nothing-replicated branch is gone. |
+| 20 | P2 | 2: 14 | Fixed. An operation-order check (each `list` preceded by its round's `result`) with its own mutation row, apart from the chunk-count check. |
+| 21 | P2 | 3: 1 | Fixed. Plan 1's agent logs planning-and-restore time and confirmed output; plan 2's console logs its relay; plan 3 Task 1 measures from those lines. |
+| 22 | P2 | 3: 7 | Fixed. `checkPullManifest` checks sizes, levels, ranges, duplicates and an overflow-safe sum before any file is created. Tests: four new malformed cases, each DBF-13, no panic, no file. |
+| 23 | P2 | 3: 7, 9 | Fixed. The spool failure and Ctrl-C run `dbPull` against a stand-in console with `TMPDIR` scratch and a spool-write seam; Task 9 sends SIGINT to the binary mid-pull. Each asserts no `--out` and no `carlos-db-*` left. |
+| 24 | P2 | 3: 10 | Fixed. `TestNewAgentClampsMaxPull` calls `NewAgent`; the malformed-manifest cases assert no file; the record-changed case asserts the worker never received `run` (frames recorded at the agent's end). |
+
+**Fixed:** 23 in full, 1 in part (11). **Declined:** the part of 11 below.
+
+## Declined plan findings
+
+- **11, in part: a cancellation-aware audit insert.** `RecordActivityContext`
+ takes a context, but its claim plumbing ends in `claimRawRecordActivity`'s
+ context-free `s.db.Exec` (`internal/console/store.go:2588`, called from
+ `:2668`), shared by every audited route. Changing it is a change to the
+ claim SQL layer and its census, for one local SQLite insert. The order
+ makes it unnecessary: the row is written while the request is open
+ (checked just before) and before any object a box can see, so a request
+ that ends during the insert leaves the row of an admission and no read.
+ The one bucket write that announced reads after the request ended, the
+ serial bump, is now request-bound.
## What to check
-Check against the code, not the plans' description of it. Cite `file:line`
-for every claim about existing code.
-
-1. **Coverage.** Every item of spec §14's plans 1, 2 and 3 has a task.
- Every test T1 to T18 of §11.4 is built by the plan §14 assigns it to, as
- the spec describes it, and every mutation check of §11.4 appears in a
- plan's mutation table with a test that would actually turn red. Every
- number in §5.7 reaches code verbatim. J1 to J5 (§5.3) each hold in the
- code the plans write: in particular nothing the console writes or keeps
- outlives the request except one `cancel` (J1), every protocol write is a
- conditional create (J2), the `picked` read-back (J3), no protocol delete
- (J4), admission released only by the handler's return and box
- reservations held until the worker unit is gone (J5).
-2. **Code that cannot build or does not do what it says.** Names and
- signatures the plans call on `main`: `release.Store` and `Put`'s
- create-only `ifMatch ""`, `release.SignRootEnvelope`,
- `release.VerifyRootEnvelope`, `release.MintKeyGrant`, `release.RootSigner`,
- `dbdoor`'s unexported `restoreOnce`, `removeCopy`, `checkDB`,
- `fileIterator`, `errReadOnly`, `litestream.CalcRestorePlan`,
- `lease.ReadReplicaClient`, `hostclaim.Incarnation`,
- `registry.FleetPlacement`, `edgesfile.ReadLiveSnapshot` and the
- `PlacementView` fields, the console's `dbGate`, `resolveDBSource`,
- `apiAccountApp`, `readJSONLimit`, `cliRoute`, `auditActivity`,
- `fx.st.Events`, `newConsoleFixture`, `cliToken`, `putShopRecord`,
- `rawReq`, the root package's `buildCarlos`, `useTempCredentials`,
- `e2eDeviceLogin`, `replicaS3`, `carlosTestEnv`, `edge.New`,
- `registry.Open`, and the edge package's backhaul harness
- (`newBackhaulEdge`, `refreshHomeMap`, `MintMeshKey`, `putJSON`). Where a
- plan's test or code would not compile, fail for the wrong reason, or
- pass while proving nothing, say which.
-3. **Consistency across plans.** A name, field or signature defined in one
- task and used in another, or in a later plan, must match: for example
- `newChunkWriter`'s parameters (plan 1 Task 11, plan 3 Task 3),
- `readDBForward` and `openDBForward` (plan 2 Task 8, plan 3 Task 7),
- `dbForwardAuditDetail` (plan 2 Task 6, plan 3 Task 6), the fixture
- helpers each plan's tests reuse, and the copy gate's inventory and built
- list as rows are added and swapped.
-4. **Isolation and the identity bracket, as built.** Does the unit file in
- plan 1 Task 13 match spec §8.3 option by option, and do the witnesses in
- Task 15 actually exercise each one from inside the installed unit? Does
- anything the agent sends the worker, or anything the worker can open,
- come from the request rather than the fresh record and the box's config?
- Is there any path where output (a chunk) can exist before check B, for a
- query or for a pull (plan 3's hold mode)?
-5. **The long request, as built.** Does plan 2's handler extend the read
- deadline before anything that could block, flush every line under a
- write deadline, bound everything before the `200` at 20 s, and give up at
- the owner's deadline plus a minute but never past 22 minutes? Does the
- CLI's 23-minute context and its line bound fit a pull's 1 MiB chunks
- (plan 3)?
-6. **Tests that prove what they claim.** For each acceptance test, is it
- production-wired where the spec says so, and does it fail if the
- property it names breaks? Are stand-ins (the in-process worker, scripted
- boxes, the edge test's upstream) labelled, and is each one where the
- spec allows a stand-in?
-7. **Census and CI mechanics.** Are the overlay rows in the format
- `loadRootReadInventory` accepts (`internal/provision/plan14_root_reads_test.go:399`),
- with legacy fingerprints and the seal handled as the existing overlays
- handle them, and the H3 removal in the form the loader allows? Is the
- 9B8 inventory layering right? Does the MinIO step addition in plan 2
- Task 12 work with the existing `.amadan/ci.d/30-minio-contract`?
-8. **The plans' interpretations of the spec.** These were judgement calls.
- Say whether each is sound; challenge any that changes behaviour the spec
- fixes:
- - an additive `ended` line (`not-picked`, `no-result`) so the CLI can
- tell DBF-3 from DBF-12 (plan 2 Task 2);
- - the reads route answers 404 for an in-region instance, and today's
- residency sentence when forwarding is off, on a host-claim console, with
- a key that is not the root's, or for a home-region record seen from a
- regional console (plan 2 Tasks 5 and 6);
- - a class, pool or fleet change seen by the box is `record-changed`
- (plan 1 Task 10);
- - a box that fails the disk floor at admission does not pick the read
- (spec §7.1), so `no-room` comes only from the plan's growth; the root
- runner's T12 shows the floor holding a second read back and admitting
- it once the first unit is gone, and the `no-room` code itself is a unit
- test (plan 1 Tasks 9, 11, 15);
- - a finished read's reservation is released only when no
- `carlos-dbread@` unit is left at all (plan 1 Task 9);
- - output past the cap is cut at the byte; the result is `done` with
- `output-cap`, and the CLI prints the kept rows and then fails with
- DBF-6 on stderr (plan 1 Task 11, plan 2 Task 8);
- - a planning or restore overrun, or a worker that fails to start, is
- `interrupted` (DBF-12);
- - T7 is split: a scaled run through the real edge and the console's real
- `http.Server` with `ReadTimeout` set to 2 s in CI, an unscaled
- 21-minute run with a scripted box behind `CARLOS_DBF_LONG=1` run once
- before handoff, and the backhaul hop in package `edge` with a stand-in
- upstream, because `internal/edge` tests cannot import the console
- (plan 2 Tasks 10 and 11);
- - T8's "old CLI" is `dbRestoreError` over the door's 409, the function
- every CLI before plan 2 runs on a door refusal;
- - the two unit files enter the root-read census as `external` rows, the
- form `carlos-instance-cmd@.service` took, which the plans read as spec
- §13's "Plan 14 write census"; and the overlay loads after whatever main
- loads last (at `5f341f33` that is `console-brand-name`, not
- `securelisten-337`);
- - the copy record is created whole in plan 1, and each plan reviews and
- builds its own rows;
- - plan 3 keeps the 22-minute bound and sizes the pull's cap to fit a
- measured relay rate (at most 2 GiB; `CARLOS_DB_PULL_MAX_BYTES` only
- lowers it), with 1 MiB pull chunks and a 2 MiB line bound;
- - plan 3's worker holds and verifies the files and sends them back only
- after check B, because the agent writes no files and no output may
- precede check B;
- - `LimitFSIZE` is fixed at the default 2 GiB cap in the unit file, and
- `InaccessiblePaths` marks `/run/carlos-control` optional.
-9. **Missing pieces.** Anything a plan relies on that does not exist on
- `main` or in an earlier task, or exists in a different shape.
+Report **new P0, P1 and P2 findings only**; leave P3s out of this round.
+
+1. **Each fix as built.** Does it do what the disposition says, against the
+ code on `main` (cite `file:line`), and does its test fail without it, by
+ the mutation the plan names? Where a plan says a check has no witness of
+ its own (plan 2 Task 14's note on the `ownCtx` check before the result
+ line), is that true and acceptable?
+2. **What the fixes broke.** Each fix moved code or tests: the stream's
+ rewrite (`dbfCall`, phase contexts, `stop`), the audit-first admission,
+ the moved tests, the gate store in the MinIO test, the retry's
+ reservation and `settle`, `SelfView`, the frame tap, the stand-in console
+ in plan 3. Do names, signatures and imports still line up across tasks
+ and plans, and does every task still build on the ones before it?
+3. **Rev 6 against the plans.** Do the four marked clarifications match
+ what the plans build, and does any of them contradict J1 to J5 or
+ another part of the spec?
+4. **Anything round 1 missed** that is P0 to P2 under the round 1
+ definitions below.
## Output
-A numbered list of findings. Each one gives:
+A numbered list of findings, numbered from 25. Each one gives:
- **severity:** P0 the plan cannot be executed as written (a task that
cannot build or whose gate cannot pass); P1 the built code would break a
spec invariant or ruling, cross a tenant, or expose box credentials; P2
- wrong behaviour, a missing spec item, or a test that proves nothing; P3
- clarity or a gap;
-- the plan and task;
+ wrong behaviour, a missing spec item, or a test that proves nothing;
+- the plan and task, and the round 1 number if it reopens one;
- the evidence, with `file:line` in the code or `plan:Task N` in the plans;
- a concrete fix to the plan.