amadan

carlos / platform Public

Merge remote-tracking branch 'origin/main' into sec-mail-user-boundary

# Conflicts:
#	internal/provision/plan14_root_reads_test.go
claude 7aad0c1806db394f5004876428bc579518072f10 parent 1a0f725parent fef28dc
46 files changed, +2320 −124
  • .amadan/test-weights.tsv +4 −0
  • approved_db_forward_copy_test.go +13 −7
  • db.go +7 −2
  • dbforward.go +315 −26
  • dbforward_e2e_test.go +184 −0
  • dbforward_test.go +394 −5
  • docs/site/cli.md +2 −2
  • docs/superpowers/notes/census-proposal-db-region-forward-3.md +76 −0
  • docs/superpowers/notes/census-proposal-sec-mail-user-boundary.md +4 −4
  • docs/superpowers/notes/census-proposal-wake-latency-1.md +64 −0
  • docs/superpowers/plans/2026-10-22-db-region-forward-copy.md +16 −2
  • docs/superpowers/specs/2026-10-22-db-region-forward-design.md +7 −1
  • internal/activator/activator.go +15 −0
  • internal/activator/backend_exec.go +24 −1
  • internal/activator/backend_unit.go +3 −1
  • internal/activator/healthpoll_test.go +24 −0
  • internal/activator/wakesteps.go +40 −0
  • internal/activator/wakesteps_test.go +30 −0
  • internal/console/apidbforward.go +36 −12
  • internal/console/apidbforward_test.go +59 −5
  • internal/console/dbforward_minio_test.go +33 −0
  • internal/dbdoor/plan_from.go +34 −1
  • internal/dbdoor/plan_from_test.go +66 −0
  • internal/dbreads/agent.go +15 −3
  • internal/dbreads/chunks.go +7 −6
  • internal/dbreads/envelope.go +12 −4
  • internal/dbreads/envelope_test.go +19 −0
  • internal/dbreads/fixture_test.go +26 −2
  • internal/dbreads/limits.go +7 −0
  • internal/dbreads/lines.go +3 −2
  • internal/dbreads/lines_test.go +10 −0
  • internal/dbreads/pull_test.go +325 −0
  • internal/dbreads/read.go +36 −8
  • internal/dbreads/read_test.go +1 −1
  • internal/dbreads/worker.go +86 −1
  • internal/host/dbreads.go +15 −3
  • internal/host/dbreads_rootrunner_linux_test.go +48 −0
  • internal/host/dbreads_test.go +30 −0
  • internal/instancectl/gen.go +44 −10
  • internal/instancectl/server_gen_test.go +86 −9
  • internal/provision/db_region_forward_3_census_test.go +28 −0
  • internal/provision/plan14_root_reads_test.go +36 −3
  • internal/provision/plan9_b8_census_test.go +1 −1
  • internal/provision/testdata/plan14-root-reads-db-region-forward-3.tsv +23 −0
  • internal/provision/testdata/plan14-root-reads-sec-mail-user-boundary.tsv +2 −2
  • internal/provision/testdata/plan14-root-reads-wake-latency-1.tsv +10 −0

CI failed — run details

CI log 
=== gate ===
.amadan/test-gate.sh
=== launchctx (1 s)
ok  	amadan.net/carlos/platform.git/launchctx	0.012s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
=== vet (19 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (173 s)
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.644s
ok  	github.com/carlosframework/platform/cmd/moveprobe	11.768s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.760s
ok  	github.com/carlosframework/platform/internal/activator	57.679s
ok  	github.com/carlosframework/platform/internal/agentsock	0.038s
ok  	github.com/carlosframework/platform/internal/alerting	0.016s
ok  	github.com/carlosframework/platform/internal/analytics	0.034s
ok  	github.com/carlosframework/platform/internal/appconfig	2.774s
ok  	github.com/carlosframework/platform/internal/apperr	0.117s
ok  	github.com/carlosframework/platform/internal/applog	0.166s
ok  	github.com/carlosframework/platform/internal/awsinv	0.015s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.015s
ok  	github.com/carlosframework/platform/internal/boxid	1.026s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.017s
ok  	github.com/carlosframework/platform/internal/caps	0.032s
ok  	github.com/carlosframework/platform/internal/certwarm	0.015s
ok  	github.com/carlosframework/platform/internal/cgroup	0.740s
ok  	github.com/carlosframework/platform/internal/changelog	0.442s
ok  	github.com/carlosframework/platform/internal/claimactivation	1.095s
ok  	github.com/carlosframework/platform/internal/clamd	0.067s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	8.585s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.019s
ok  	github.com/carlosframework/platform/internal/dbquery	0.016s
ok  	github.com/carlosframework/platform/internal/dbreads	2.116s
?   	github.com/carlosframework/platform/internal/dbreads/dbreadstest	[no test files]
ok  	github.com/carlosframework/platform/internal/dbverify	0.018s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.016s
ok  	github.com/carlosframework/platform/internal/docsite	0.020s
ok  	github.com/carlosframework/platform/internal/domains	2.697s
ok  	github.com/carlosframework/platform/internal/doorkey	0.017s
ok  	github.com/carlosframework/platform/internal/economics	0.010s
ok  	github.com/carlosframework/platform/internal/edge	45.875s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.016s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.161s
ok  	github.com/carlosframework/platform/internal/gate	1.664s
ok  	github.com/carlosframework/platform/internal/genselect	0.022s
ok  	github.com/carlosframework/platform/internal/geo	0.017s
ok  	github.com/carlosframework/platform/internal/geoip	0.049s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.269s
ok  	github.com/carlosframework/platform/internal/host	142.238s
ok  	github.com/carlosframework/platform/internal/hostactors	5.733s
ok  	github.com/carlosframework/platform/internal/hostactors/awsmembership	0.011s
ok  	github.com/carlosframework/platform/internal/hostbox	3.033s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.795s
ok  	github.com/carlosframework/platform/internal/hostenv	0.004s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.006s
ok  	github.com/carlosframework/platform/internal/hostexec	5.334s
ok  	github.com/carlosframework/platform/internal/hostfence	0.048s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.350s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.022s
ok  	github.com/carlosframework/platform/internal/hostguard	1.073s
ok  	github.com/carlosframework/platform/internal/hostops	1.969s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.004s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.004s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.663s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.211s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.004s
ok  	github.com/carlosframework/platform/internal/infraview	0.003s
ok  	github.com/carlosframework/platform/internal/instancectl	5.226s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.807s
ok  	github.com/carlosframework/platform/internal/instancefs	0.010s
ok  	github.com/carlosframework/platform/internal/instancekey	0.007s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.013s
ok  	github.com/carlosframework/platform/internal/layout	1.791s
ok  	github.com/carlosframework/platform/internal/lbhealth	0.003s
ok  	github.com/carlosframework/platform/internal/lease	10.160s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.514s
ok  	github.com/carlosframework/platform/internal/ledger	0.017s
ok  	github.com/carlosframework/platform/internal/mail	0.254s
ok  	github.com/carlosframework/platform/internal/maintenance	0.080s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.578s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.089s
ok  	github.com/carlosframework/platform/internal/metrics	0.009s
ok  	github.com/carlosframework/platform/internal/moves	22.526s
ok  	github.com/carlosframework/platform/internal/moves/archive	1.020s
ok  	github.com/carlosframework/platform/internal/objstore	0.255s
ok  	github.com/carlosframework/platform/internal/ops	0.209s
ok  	github.com/carlosframework/platform/internal/park	0.281s
ok  	github.com/carlosframework/platform/internal/plans	0.018s
ok  	github.com/carlosframework/platform/internal/privatefile	0.004s
ok  	github.com/carlosframework/platform/internal/provision	24.069s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.007s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.639s
ok  	github.com/carlosframework/platform/internal/relay	7.227s
ok  	github.com/carlosframework/platform/internal/release	26.449s
--- FAIL: TestAMoveOntoTheInstalledGoodBuildWaitsForASignedRollback (5.03s)
    --- FAIL: TestAMoveOntoTheInstalledGoodBuildWaitsForASignedRollback/worker (5.02s)
        holdran_test.go:796: timed out waiting for the move onto v1
FAIL
FAIL	github.com/carlosframework/platform/internal/rollout	8.651s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.004s
ok  	github.com/carlosframework/platform/internal/rootaws	0.010s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.062s
ok  	github.com/carlosframework/platform/internal/rootstate	0.028s
ok  	github.com/carlosframework/platform/internal/runs	0.632s
ok  	github.com/carlosframework/platform/internal/s3compat	0.006s
ok  	github.com/carlosframework/platform/internal/safefile	0.004s
ok  	github.com/carlosframework/platform/internal/schedule	2.608s
ok  	github.com/carlosframework/platform/internal/seeds	0.012s
ok  	github.com/carlosframework/platform/internal/sim/engines	2.592s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	11.395s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.666s
ok  	github.com/carlosframework/platform/internal/status	0.080s
ok  	github.com/carlosframework/platform/internal/steering	0.008s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.313s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.007s
ok  	github.com/carlosframework/platform/internal/unixdial	0.003s
ok  	github.com/carlosframework/platform/internal/updatewords	0.005s
ok  	github.com/carlosframework/platform/internal/usage	0.005s
ok  	github.com/carlosframework/platform/internal/version	0.003s
ok  	github.com/carlosframework/platform/internal/vet	1.715s
FAIL
=== internal_console@0 (81 s)
ok  	github.com/carlosframework/platform/internal/console	78.979s
=== internal_console@1 (44 s)
ok  	github.com/carlosframework/platform/internal/console	41.847s
=== internal_console@2 (42 s)
ok  	github.com/carlosframework/platform/internal/console	39.714s
=== internal_console@3 (41 s)
ok  	github.com/carlosframework/platform/internal/console	39.661s
=== internal_console@4 (43 s)
ok  	github.com/carlosframework/platform/internal/console	41.128s
=== internal_console@5 (42 s)
ok  	github.com/carlosframework/platform/internal/console	39.882s
=== internal_console@6 (38 s)
ok  	github.com/carlosframework/platform/internal/console	36.457s
=== internal_console@7 (47 s)
ok  	github.com/carlosframework/platform/internal/console	45.029s
=== root@0 (110 s)
ok  	github.com/carlosframework/platform	108.061s
=== root@1 (84 s)
ok  	github.com/carlosframework/platform	82.626s
=== root@2 (93 s)
ok  	github.com/carlosframework/platform	91.118s

FAILED: rest
make: *** [Makefile:19: test] Error 1

amadan: step gate failed: exit status 2

amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/.amadan/test-weights.tsv b/.amadan/test-weights.tsv
index 5a4a098fc..6241b5499 100644
--- a/.amadan/test-weights.tsv
+++ b/.amadan/test-weights.tsv
@@ -10,6 +10,7 @@ github.com/carlosframework/platform/internal/console TestClaimDeleteRetryWhilePa
github.com/carlosframework/platform/internal/console TestClaimProbeRunnerDetectsAHollowRequest 3.2
github.com/carlosframework/platform/internal/console TestDBReadEachTenantGetsItsOwnRows 3.2
github.com/carlosframework/platform/internal/console TestDBReadSlowRelayEndsAtTheBound 3.0
+github.com/carlosframework/platform/internal/console TestDBReadPullStreamsTheFiles 2.8
github.com/carlosframework/platform/internal/console TestClaimPurgeMailRecordRefusesBeforeEffects 2.5
github.com/carlosframework/platform/internal/console TestDBReadStalledStoreKeepsTheStreamAndItsBound 2.5
github.com/carlosframework/platform/internal/console TestDBReadEndedDuringAdmissionWritesOnlyTheCancel 2.4
@@ -39,6 +40,9 @@ github.com/carlosframework/platform/internal/console TestDBReadRequestBoundClamp
github.com/carlosframework/platform TestDBForwardPrintsNothingForABadAnswer 67.0
github.com/carlosframework/platform TestDBForwardLongRequestThroughTheEdge 44.5
github.com/carlosframework/platform TestDBForwardThroughTheBackhaulHop 44.1
+github.com/carlosframework/platform TestDBForwardPullParityThroughTheCLI 4.8
github.com/carlosframework/platform TestDBForwardParityThroughTheCLI 4.6
+github.com/carlosframework/platform TestDBForwardPullInterruptedLeavesNothing 2.1
github.com/carlosframework/platform TestDBForwardMixedVersions 1.9
+github.com/carlosframework/platform TestDBForwardPullSavesNothingForABadAnswer 1.8
github.com/carlosframework/platform TestDBForwardRefusesSQLThatIsNotUTF8 1.0
diff --git a/approved_db_forward_copy_test.go b/approved_db_forward_copy_test.go
index fc172bf0e..9eebffc01 100644
--- a/approved_db_forward_copy_test.go
+++ b/approved_db_forward_copy_test.go
@@ -24,11 +24,15 @@ const dbForwardCopyRecordPath = "docs/superpowers/plans/2026-10-22-db-region-for
var dbForwardCopyIDPattern = regexp.MustCompile(`^DBF-[0-9]+$`)
-// dbForwardCopyIDs is the record's inventory, pinned: DBF-1 to DBF-16,
-// DBF-21, DBF-22, DBF-30, DBF-31 and DBF-32. Plan 3 adds its pull rows here.
+// dbForwardCopyIDs is the record's inventory, pinned: DBF-1 to DBF-20,
+// DBF-23 and DBF-24 (plan 3's pull rows among them), then DBF-21, DBF-22,
+// DBF-30, DBF-31 and DBF-32.
var dbForwardCopyIDs = func() []string {
- ids := make([]string, 0, 21)
- for i := 1; i <= 16; i++ {
+ ids := make([]string, 0, 27)
+ for i := 1; i <= 24; i++ {
+ if i == 21 || i == 22 {
+ continue // appended below, in the record's order
+ }
ids = append(ids, "DBF-"+strconv.Itoa(i))
}
return append(ids, "DBF-21", "DBF-22", "DBF-30", "DBF-31", "DBF-32")
@@ -40,7 +44,9 @@ var dbForwardCopyIDs = func() []string {
var dbForwardCopyBuilt = []string{
"DBF-1", "DBF-2", "DBF-14", "DBF-9", "DBF-10", "DBF-21", "DBF-22", // the console
"DBF-3", "DBF-4", "DBF-5", "DBF-6", "DBF-7", "DBF-8", "DBF-11", "DBF-12", "DBF-13", "DBF-15", "DBF-16", "DBF-32", // the CLI
- "DBF-30", "DBF-31", // docs/site/cli.md
+ "DBF-19", "DBF-20", // the console's pull audit details (plan 3)
+ "DBF-17", "DBF-18", "DBF-24", // the CLI's pull sentences (plan 3)
+ "DBF-23", "DBF-31", // docs/site/cli.md (DBF-23 replaced DBF-30 there)
}
// dbForwardCopyWired is every id whose PENDING text the code already
@@ -53,8 +59,8 @@ var dbForwardCopyBuilt = []string{
// that has not turned forwarding on. The copy review moves each id from here
// to dbForwardCopyBuilt, applying Paul's edits in the same commit. Docs rows
// are public the moment the site syncs, so they are never wired. Luna
-// approved every row on 2026-10-23, so the list is empty and every row is
-// built.
+// approved every plan 2 row and every plan 3 pull row on 2026-10-23, so the
+// list is empty and every row is built.
var dbForwardCopyWired = []string{}
func mustParseDBForwardRecord(t *testing.T) []copyRow {
diff --git a/db.go b/db.go
index 0757bdde4..bf697ae3d 100644
--- a/db.go
+++ b/db.go
@@ -284,8 +284,13 @@ func dbPull(ctx context.Context, t dbTarget, out string) error {
r, err := dbRestore(ctx, t)
var fwd *errDBForward
if errors.As(err, &fwd) {
- // Phase 1 forwards queries only (db region forward spec §6): DBF-8.
- return fmt.Errorf(dbfNoPull, t.host, fwd.door.Forward.Region)
+ // The copy's files come from the region that holds them (db region
+ // forward spec §6), when the console forwards pulls there.
+ if !slices.Contains(fwd.door.Forward.Ops, dbreads.OpPull) {
+ // A console that forwards queries and not pulls (plan 2's): DBF-8.
+ return fmt.Errorf(dbfNoPull, t.host, fwd.door.Forward.Region)
+ }
+ r, err = dbPullForward(ctx, t, *fwd.door.Forward)
}
if err != nil {
return err
diff --git a/dbforward.go b/dbforward.go
index 491df7887..5bf8af68a 100644
--- a/dbforward.go
+++ b/dbforward.go
@@ -1,9 +1,11 @@
-// dbforward.go is the CLI's half of a forwarded `carlos db query` (spec
-// docs/superpowers/specs/2026-10-22-db-region-forward-design.md §5.5): when
-// the db door answers with the forward marker, the CLI makes an ephemeral
-// key, holds one request to the console's reads route open, buffers every
-// sealed chunk, and prints nothing until a done result's summary checks out
-// against the bytes (codex 6).
+// dbforward.go is the CLI's half of a forwarded `carlos db query` and
+// `carlos db pull` (spec
+// docs/superpowers/specs/2026-10-22-db-region-forward-design.md §5.5, §6):
+// when the db door answers with the forward marker, the CLI makes an
+// ephemeral key, holds one request to the console's reads route open, keeps
+// every sealed chunk (in memory for a query, spooled to a private directory
+// for a pull), and prints or saves nothing until a done result's summary
+// checks out against the bytes (codex 6).
package main
import (
@@ -20,10 +22,16 @@ import (
"net/http"
"net/url"
"os"
+ "path/filepath"
+ "strconv"
"strings"
"time"
"unicode/utf8"
+ "github.com/benbjohnson/litestream"
+ lsfile "github.com/benbjohnson/litestream/file"
+ "github.com/superfly/ltx"
+
"github.com/carlosframework/platform/internal/dbdoor"
"github.com/carlosframework/platform/internal/dbreads"
)
@@ -45,6 +53,14 @@ const (
dbfNotUTF8 = "this query cannot be sent to region %s, because it contains invalid UTF-8 or the replacement character U+FFFD. fix the query and run it again." // DBF-32
)
+// The forwarded pull's CLI sentences (copy record DBF-17, DBF-18, DBF-24),
+// approved by Luna on 2026-10-23.
+const (
+ dbfPullTooLarge = "the database for %s has %s of files to fetch. carlos db pull can fetch at most %s from region %s." // DBF-17
+ dbfFetching = "fetching a copy of %s's database from region %s." // DBF-18
+ dbfPullNoRoom = "the server in region %s does not have room to prepare a copy of %s's database right now. try again later." // DBF-24
+)
+
// errDBForward is the db door's forward marker, handed back by dbRestore so
// a verb can switch to the forwarded read.
type errDBForward struct {
@@ -98,7 +114,12 @@ func dbQueryForward(ctx context.Context, t dbTarget, sqlText string, fwd dbdoor.
raw, _ := io.ReadAll(io.LimitReader(res.Body, 64<<10))
return apiErrorFromBody(res.StatusCode, raw)
}
- ans, err := readDBForward(ctx, res.Body, key, t.host, fwd.Region, os.Stderr)
+ sink := &memSink{}
+ id, result, err := readDBForward(ctx, res.Body, key, t.host, fwd.Region, dbfReading, os.Stderr, sink, newDBFChunkBound(dbreads.MaxOutputBytes, dbreads.ChunkBytes))
+ if err != nil {
+ return err
+ }
+ ans, err := openDBForward(id, key, sink, *result, t.host, fwd.Region)
if err != nil {
return err
}
@@ -141,55 +162,98 @@ func (b *dbfChunkBound) admit(sealed []byte) bool {
return true
}
+// dbChunkSink holds a read's sealed chunks until its result arrives: in
+// memory for a query (at most 64 MiB), in the pull's private directory for a
+// pull (up to its cap).
+type dbChunkSink interface {
+ add(sealed []byte) error
+ count() int
+ chunk(i int) ([]byte, error) // 0-based
+}
+
+type memSink struct{ chunks [][]byte }
+
+func (m *memSink) add(b []byte) error { m.chunks = append(m.chunks, b); return nil }
+func (m *memSink) count() int { return len(m.chunks) }
+func (m *memSink) chunk(i int) ([]byte, error) { return m.chunks[i], nil }
+
+// fileSink spools sealed chunks to dir, one file each, 0600 in a 0700
+// directory the caller removes.
+type fileSink struct {
+ dir string
+ n int
+}
+
+func (f *fileSink) add(b []byte) error {
+ if f.n == 0 {
+ if err := os.MkdirAll(f.dir, 0o700); err != nil {
+ return err
+ }
+ }
+ f.n++
+ return dbPullSpoolWrite(filepath.Join(f.dir, strconv.Itoa(f.n)), b, 0o600)
+}
+
+func (f *fileSink) count() int { return f.n }
+
+func (f *fileSink) chunk(i int) ([]byte, error) {
+ return os.ReadFile(filepath.Join(f.dir, strconv.Itoa(i+1)))
+}
+
+// drop removes chunk i once its bytes are in the files, so the spool and
+// the files are never both whole on the member's disk.
+func (f *fileSink) drop(i int) { os.Remove(filepath.Join(f.dir, strconv.Itoa(i+1))) }
+
// readDBForward reads the reads route's lines (spec §5.5 steps 6 to 8). It
-// says DBF-11 once admitted, buffers every chunk in sequence, and opens and
-// checks the answer only when the result arrives.
-func readDBForward(ctx context.Context, body io.Reader, key *ecdh.PrivateKey, host, region string, status io.Writer) (dbForwardAnswer, error) {
+// says the reading sentence once admitted, keeps every chunk in sequence in
+// sink (each one checked against bound before the sink keeps anything, plan
+// review finding 37), and returns the read's id and its result unopened.
+func readDBForward(ctx context.Context, body io.Reader, key *ecdh.PrivateKey, host, region, reading string, status io.Writer, sink dbChunkSink, bound *dbfChunkBound) (string, *dbreads.Result, error) {
unchecked := fmt.Errorf(dbfUnchecked, region)
sc := bufio.NewScanner(body)
sc.Buffer(make([]byte, 64<<10), dbreads.MaxLineBytes)
var id, ended string
- var sealed [][]byte
- bound := newDBFChunkBound(dbreads.MaxOutputBytes, dbreads.ChunkBytes)
for sc.Scan() {
var l dbreads.Line
if err := json.Unmarshal(sc.Bytes(), &l); err != nil {
- return dbForwardAnswer{}, unchecked
+ return "", nil, unchecked
}
switch {
case l.Admitted != "":
if id != "" || !dbreads.ValidID(l.Admitted) {
- return dbForwardAnswer{}, unchecked
+ return "", nil, unchecked
}
id = l.Admitted
- fmt.Fprintln(status, fmt.Sprintf(dbfReading, host, region))
+ fmt.Fprintln(status, fmt.Sprintf(reading, host, region))
case l.Chunk != 0:
- if id == "" || l.Chunk != int64(len(sealed))+1 || !bound.admit(l.Data) {
- return dbForwardAnswer{}, unchecked
+ if id == "" || l.Chunk != int64(sink.count())+1 || !bound.admit(l.Data) {
+ return "", nil, unchecked
+ }
+ if err := sink.add(l.Data); err != nil {
+ return "", nil, err
}
- sealed = append(sealed, l.Data)
case l.Result != nil:
if id == "" {
- return dbForwardAnswer{}, unchecked
+ return "", nil, unchecked
}
- return openDBForward(id, key, sealed, *l.Result, host, region)
+ return id, l.Result, nil
case l.Ended != "":
ended = l.Ended
}
}
if errors.Is(ctx.Err(), context.Canceled) {
- return dbForwardAnswer{}, errors.New("stopped")
+ return "", nil, errors.New("stopped")
}
if ended == dbreads.EndedNotPicked {
- return dbForwardAnswer{}, fmt.Errorf(dbfNotPicked, region)
+ return "", nil, fmt.Errorf(dbfNotPicked, region)
}
- return dbForwardAnswer{}, fmt.Errorf(dbfUnfinished, host, region)
+ return "", nil, fmt.Errorf(dbfUnfinished, host, region)
}
// openDBForward opens a result: a done one's every chunk and its summary,
// checked against each other before anything is returned; any other state
// mapped to its sentence (spec §5.6, §10).
-func openDBForward(id string, key *ecdh.PrivateKey, sealed [][]byte, res dbreads.Result, host, region string) (dbForwardAnswer, error) {
+func openDBForward(id string, key *ecdh.PrivateKey, sink dbChunkSink, res dbreads.Result, host, region string) (dbForwardAnswer, error) {
unchecked := fmt.Errorf(dbfUnchecked, region)
if res.Valid() != nil {
return dbForwardAnswer{}, unchecked
@@ -200,11 +264,15 @@ func openDBForward(id string, key *ecdh.PrivateKey, sealed [][]byte, res dbreads
}
switch res.State {
case dbreads.StateDone:
- if res.LastSeq != int64(len(sealed)) {
+ if res.LastSeq != int64(sink.count()) {
return dbForwardAnswer{}, unchecked
}
var rows bytes.Buffer
- for i, c := range sealed {
+ for i := 0; i < sink.count(); i++ {
+ c, err := sink.chunk(i)
+ if err != nil {
+ return dbForwardAnswer{}, unchecked
+ }
seq := int64(i + 1)
plain, err := dbreads.OpenChunk(key, id, seq, dbreads.StreamRows, seq == res.LastSeq, c)
if err != nil {
@@ -246,3 +314,224 @@ func openDBForward(id string, key *ecdh.PrivateKey, sealed [][]byte, res dbreads
}
return dbForwardAnswer{}, fmt.Errorf(dbfUnfinished, host, region)
}
+
+// openDBPull opens a pull's result (spec §6, phase 2): a done one's chunks,
+// opened in order, are the planned files concatenated; they are checked
+// against the summary's count and hash and split by its file list into a
+// litestream file replica under dir, which today's restore and check turn
+// into the copy. Any other state maps to its sentence.
+func openDBPull(ctx context.Context, id string, key *ecdh.PrivateKey, sink dbChunkSink, res dbreads.Result, host, region, dir string) (*dbdoor.Restored, error) {
+ unchecked := fmt.Errorf(dbfUnchecked, region)
+ if res.Valid() != nil {
+ return nil, unchecked
+ }
+ s, err := dbreads.OpenSummary(key, id, res.Sealed)
+ if err != nil {
+ return nil, unchecked
+ }
+ if res.State != dbreads.StateDone {
+ switch res.Code {
+ case dbreads.CodeTooLarge:
+ return nil, fmt.Errorf(dbfPullTooLarge, host, humanBytes(&s.Restored), humanBytes(&s.Cap), region)
+ case dbreads.CodeNoRoom:
+ return nil, fmt.Errorf(dbfPullNoRoom, region, host)
+ }
+ _, err := openDBForward(id, key, sink, res, host, region)
+ return nil, err
+ }
+ // A done pull is the whole answer: no output cap applies to one.
+ if res.Code != "" || res.LastSeq != int64(sink.count()) || checkPullManifest(s) != nil {
+ return nil, unchecked
+ }
+ root := filepath.Join(dir, "replica")
+ h := sha256.New()
+ fi, written := 0, int64(0)
+ var head []byte // the current file's first ltx.HeaderSize bytes
+ var out *os.File
+ defer func() {
+ if out != nil {
+ out.Close()
+ }
+ }()
+ drop, _ := sink.(interface{ drop(int) })
+ for i := 0; i < sink.count(); i++ {
+ // Ctrl-C and the read's deadline stop the rebuild too (Astra round
+ // 1 on plan 3, finding 2); RestoreLocal's reads check ctx after it.
+ if err := ctx.Err(); err != nil {
+ return nil, dbPullStopped(err, host, region)
+ }
+ sealed, err := sink.chunk(i)
+ if err != nil {
+ return nil, unchecked
+ }
+ seq := int64(i + 1)
+ plain, err := dbreads.OpenChunk(key, id, seq, dbreads.StreamLTX, seq == res.LastSeq, sealed)
+ if err != nil {
+ return nil, unchecked
+ }
+ h.Write(plain)
+ for len(plain) > 0 {
+ if fi >= len(s.Files) {
+ return nil, unchecked
+ }
+ f := s.Files[fi]
+ if out == nil {
+ p := litestream.LTXFilePath(root, f.Level, ltx.TXID(f.MinTXID), ltx.TXID(f.MaxTXID))
+ if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {
+ return nil, err
+ }
+ if out, err = os.OpenFile(p, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600); err != nil {
+ return nil, err
+ }
+ }
+ n := min(int64(len(plain)), f.Size-written)
+ if _, err := out.Write(plain[:n]); err != nil {
+ return nil, err
+ }
+ if len(head) < ltx.HeaderSize {
+ head = append(head, plain[:min(n, int64(ltx.HeaderSize-len(head)))]...)
+ }
+ plain, written = plain[n:], written+n
+ if written == f.Size {
+ if err := out.Close(); err != nil {
+ out = nil
+ return nil, err
+ }
+ // litestream's restore picks files by name and compacts them
+ // by header: a file whose header is not its manifest entry
+ // would restore as a transaction it does not hold (Astra
+ // round 1 on plan 3, finding 1).
+ var lh ltx.Header
+ if lh.UnmarshalBinary(head) != nil || uint64(lh.MinTXID) != f.MinTXID || uint64(lh.MaxTXID) != f.MaxTXID {
+ out = nil
+ return nil, unchecked
+ }
+ out, fi, written, head = nil, fi+1, 0, nil
+ }
+ }
+ if drop != nil {
+ drop.drop(i)
+ }
+ }
+ if fi != len(s.Files) || hex.EncodeToString(h.Sum(nil)) != s.SHA256 {
+ return nil, unchecked
+ }
+ db := filepath.Join(dir, "replica.db")
+ if err := dbdoor.RestoreLocal(ctx, lsfile.NewReplicaClient(root), db, s.TXID); err != nil {
+ if cerr := ctx.Err(); cerr != nil {
+ return nil, dbPullStopped(cerr, host, region)
+ }
+ return nil, dbRestoreError(host, err)
+ }
+ os.RemoveAll(root)
+ return &dbdoor.Restored{Path: db, Dir: dir, TXID: s.TXID, At: s.At}, nil
+}
+
+// dbPullStopped is a pull that ended on its context: "stopped" for Ctrl-C,
+// DBF-12 for the read's own deadline.
+func dbPullStopped(err error, host, region string) error {
+ if errors.Is(err, context.Canceled) {
+ return errors.New("stopped")
+ }
+ return fmt.Errorf(dbfUnfinished, host, region)
+}
+
+// checkPullManifest is the summary's file list checked whole before any
+// file is created (plan review finding 22): every size a possible LTX file
+// and within the cap, every level one litestream writes, every range a
+// range, no file twice, sizes that sum without overflow to the summary's
+// byte count, and a last file that ends at its transaction.
+func checkPullManifest(s dbreads.Summary) error {
+ bad := errors.New("the file list does not add up")
+ if len(s.Files) == 0 || s.Bytes <= 0 || s.Bytes > dbreads.MaxPullBytes {
+ return bad
+ }
+ type coord struct {
+ level int
+ min, max uint64
+ }
+ seen := make(map[coord]bool, len(s.Files))
+ var sum int64
+ for _, f := range s.Files {
+ c := coord{f.Level, f.MinTXID, f.MaxTXID}
+ switch {
+ case f.Size < ltx.HeaderSize+ltx.TrailerSize, f.Size > dbreads.MaxPullBytes,
+ f.Level < 0, f.Level > litestream.SnapshotLevel,
+ f.MinTXID < 1, f.MaxTXID < f.MinTXID, seen[c],
+ sum > dbreads.MaxPullBytes-f.Size: // sum + f.Size would pass the cap
+ return bad
+ }
+ seen[c] = true
+ sum += f.Size
+ }
+ if sum != s.Bytes || s.Files[len(s.Files)-1].MaxTXID != s.TXID {
+ return bad
+ }
+ return nil
+}
+
+// dbPullSpoolWrite writes one spooled chunk: os.WriteFile, a seam so a test
+// can fail the spool part way (plan review finding 23).
+var dbPullSpoolWrite = os.WriteFile
+
+// dbPullForward is a pull through the forwarded read (spec §6). The spool,
+// the files and the copy live in one private 0700 directory under the system
+// temp dir, removed on every path but success, where the caller removes it
+// after copyOut, as it does today's restore directory.
+func dbPullForward(ctx context.Context, t dbTarget, fwd dbdoor.Forward) (*dbdoor.Restored, error) {
+ ctx, cancel := context.WithTimeout(ctx, dbreads.CLIBound)
+ defer cancel()
+ key, err := dbreads.NewReplyKey()
+ if err != nil {
+ return nil, err
+ }
+ at := ""
+ if !t.at.IsZero() {
+ at = t.at.UTC().Format(time.RFC3339Nano)
+ }
+ body, err := json.Marshal(dbreads.ReadBody{Op: dbreads.OpPull, At: at, ReplyKey: dbreads.EncodeReplyKey(key.PublicKey())})
+ if err != nil {
+ return nil, err
+ }
+ res, err := t.sess.c.doStream(ctx, http.MethodPost, apiPath(t.acct, t.app, "/db/"+url.PathEscape(t.host)+"/reads"),
+ bytes.NewReader(body), int64(len(body)), http.Header{"Content-Type": {"application/json"}})
+ if err != nil {
+ if errors.Is(ctx.Err(), context.Canceled) {
+ return nil, errors.New("stopped")
+ }
+ return nil, fmt.Errorf(dbfUnfinished, t.host, fwd.Region)
+ }
+ defer res.Body.Close()
+ if res.StatusCode != http.StatusOK {
+ raw, _ := io.ReadAll(io.LimitReader(res.Body, 64<<10))
+ return nil, apiErrorFromBody(res.StatusCode, raw)
+ }
+ dir, err := os.MkdirTemp("", "carlos-db-")
+ if err != nil {
+ return nil, err
+ }
+ // MkdirTemp already creates 0700; say it anyway, as the in-region
+ // restore does, because the copy's privacy rests on it.
+ if err := os.Chmod(dir, 0o700); err != nil {
+ os.RemoveAll(dir)
+ return nil, err
+ }
+ done := false
+ defer func() {
+ if !done {
+ os.RemoveAll(dir)
+ }
+ }()
+ sink := &fileSink{dir: filepath.Join(dir, "sealed")}
+ id, result, err := readDBForward(ctx, res.Body, key, t.host, fwd.Region, dbfFetching, os.Stderr, sink, newDBFChunkBound(dbreads.MaxPullBytes, dbreads.PullChunkBytes))
+ if err != nil {
+ return nil, err
+ }
+ r, err := openDBPull(ctx, id, key, sink, *result, t.host, fwd.Region, dir)
+ if err != nil {
+ return nil, err
+ }
+ os.RemoveAll(sink.dir)
+ done = true
+ return r, nil
+}
diff --git a/dbforward_e2e_test.go b/dbforward_e2e_test.go
index e94c1ac38..f509cbddd 100644
--- a/dbforward_e2e_test.go
+++ b/dbforward_e2e_test.go
@@ -33,6 +33,8 @@ import (
"time"
"github.com/benbjohnson/litestream"
+ lsfile "github.com/benbjohnson/litestream/file"
+ "github.com/superfly/ltx"
"github.com/carlosframework/platform/internal/console"
"github.com/carlosframework/platform/internal/dbdoor"
@@ -783,3 +785,185 @@ func TestDBForwardThroughTheBackhaulHop(t *testing.T) {
e := newDBFE2E(t, dbfE2EOptions{forward: true, serve: throughBackhaul(2*time.Second, &tap), worker: slowRestore(40 * time.Second)})
checkProgressAtTheCLI(t, e, tap, 40*time.Second)
}
+
+func sha256File(t *testing.T, path string) string {
+ t.Helper()
+ b, err := os.ReadFile(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ sum := sha256.Sum256(b)
+ return hex.EncodeToString(sum[:])
+}
+
+// The pull's parity (plan 3): a forwarded pull and an in-region pull of the
+// same replica save byte-identical copies, for the newest copy and as of a
+// time, and print the same lines but for the host and the path.
+func TestDBForwardPullParityThroughTheCLI(t *testing.T) {
+ e := newDBFE2E(t, dbfE2EOptions{forward: true})
+ for _, at := range []string{"", e.betweenTheSyncs(t)} {
+ dir := t.TempDir()
+ args := []string{"db", "pull", "--account", "ops", "--app", "shop"}
+ if at != "" {
+ args = append(args, "--at", at)
+ }
+ hereOut, farOut := filepath.Join(dir, "here.db"), filepath.Join(dir, "far.db")
+ here, hereErr, err := e.run(t, append(args, "--instance", dbfE2EHere, "--out", hereOut)...)
+ if err != nil {
+ t.Fatalf("in region: %v\n%s", err, hereErr)
+ }
+ far, farErr, err := e.run(t, append(args, "--instance", dbfE2EFar, "--out", farOut)...)
+ if err != nil {
+ t.Fatalf("forwarded: %v\n%s", err, farErr)
+ }
+ if sha256File(t, hereOut) != sha256File(t, farOut) {
+ t.Fatalf("at %q the copies differ", at)
+ }
+ if here == "" || strings.ReplaceAll(strings.ReplaceAll(far, dbfE2EFar, dbfE2EHere), farOut, hereOut) != here {
+ t.Fatalf("at %q: in region %q, forwarded %q", at, here, far)
+ }
+ if !strings.Contains(farErr, fmt.Sprintf(dbfFetching, dbfE2EFar, "uk")) {
+ t.Fatalf("no DBF-18: %s", farErr)
+ }
+ db, _ := sql.Open("sqlite", "file:"+farOut+"?mode=ro")
+ var later int
+ err = db.QueryRow(`SELECT count(*) FROM orders WHERE note = 'later'`).Scan(&later)
+ db.Close()
+ if err != nil || (at == "") != (later == 1) {
+ t.Fatalf("at %q the copy holds %d of the second sync's rows (%v)", at, later, err)
+ }
+ }
+}
+
+// T11 for a pull: a hash mismatch saves nothing at --out. Everything else
+// is a pull the CLI would accept: far's real planned files, whole and
+// valid, a manifest that passes checkPullManifest, and a summary whose only
+// fault is its SHA-256 (plan review finding 32). Mutation: openDBPull skips
+// the hash comparison (red: the files restore and a copy is saved).
+func TestDBForwardPullSavesNothingForABadAnswer(t *testing.T) {
+ e := newDBFE2E(t, dbfE2EOptions{forward: true, noBox: true})
+ stop := e.scriptBox(t, func(b *e2eScript) {
+ ctx := context.Background()
+ rd := lsfile.NewReplicaClient(filepath.Join(e.root, "instances", dbfE2EFar))
+ plan, err := dbdoor.PlanFrom(ctx, rd, time.Time{})
+ if err != nil {
+ t.Errorf("plan far's replica: %v", err)
+ return
+ }
+ var all bytes.Buffer
+ for _, f := range plan.Planned {
+ rc, err := rd.OpenLTXFile(ctx, f.Level, ltx.TXID(f.MinTXID), ltx.TXID(f.MaxTXID), 0, 0)
+ if err != nil {
+ t.Errorf("open %+v: %v", f, err)
+ return
+ }
+ io.Copy(&all, rc)
+ rc.Close()
+ }
+ b.picked(time.Now().Add(time.Minute))
+ sealed, _ := dbreads.SealChunk(b.pub, b.id, 1, dbreads.StreamLTX, true, all.Bytes())
+ b.store.Put(ctx, dbreads.ChunkKey(b.id, 1), sealed, "")
+ wrong := sha256.Sum256([]byte("not these bytes"))
+ summary, _ := dbreads.SealSummary(b.pub, b.id, dbreads.Summary{TXID: plan.TXID, At: plan.At, Files: plan.Planned,
+ Bytes: int64(all.Len()), SHA256: hex.EncodeToString(wrong[:])})
+ r, _ := json.Marshal(dbreads.Result{State: dbreads.StateDone, LastSeq: 1, Box: "uk-1", Sealed: summary})
+ b.store.Put(ctx, dbreads.ResultKey(b.id), r, "")
+ })
+ defer stop()
+ out := filepath.Join(t.TempDir(), "far.db")
+ _, stderr, err := e.run(t, "db", "pull", "--account", "ops", "--app", "shop", "--instance", dbfE2EFar, "--out", out)
+ if err == nil || !strings.Contains(stderr, fmt.Sprintf(dbfUnchecked, "uk")) {
+ t.Fatalf("%q %v", stderr, err)
+ }
+ if _, err := os.Stat(out); !os.IsNotExist(err) {
+ t.Fatalf("something was saved at --out: %v", err)
+ }
+}
+
+// Review Focus 4 and plan review finding 23, through the binary: SIGINT
+// while the CLI is spooling a pull prints "stopped", saves nothing at --out
+// and removes the pull's directory. The scripted box (a stand-in) writes two
+// chunks and never a result, so the CLI is mid-pull when the signal lands.
+// Mutation: drop dbPullForward's deferred RemoveAll (red: carlos-db-* is
+// left in the CLI's TMPDIR).
+func TestDBForwardPullInterruptedLeavesNothing(t *testing.T) {
+ e := newDBFE2E(t, dbfE2EOptions{forward: true, noBox: true})
+ stop := e.scriptBox(t, func(b *e2eScript) {
+ b.picked(time.Now().Add(time.Minute))
+ for seq := int64(1); seq <= 2; seq++ {
+ sealed, _ := dbreads.SealChunk(b.pub, b.id, seq, dbreads.StreamLTX, false, bytes.Repeat([]byte{'x'}, 4096))
+ b.store.Put(context.Background(), dbreads.ChunkKey(b.id, seq), sealed, "")
+ }
+ })
+ defer stop()
+ spool := t.TempDir()
+ out := filepath.Join(t.TempDir(), "far.db")
+ cmd := exec.Command(e.bin, "db", "pull", "--account", "ops", "--app", "shop", "--instance", dbfE2EFar, "--out", out)
+ // The last TMPDIR wins (os/exec keeps the last of a duplicated key).
+ cmd.Env = append(carlosTestEnv(t, append([]string{"CARLOS_CREDENTIALS=" + e.creds}, e.env...)...), "TMPDIR="+spool)
+ cmd.Dir = t.TempDir()
+ var stderr bytes.Buffer
+ cmd.Stderr = &stderr
+ if err := cmd.Start(); err != nil {
+ t.Fatal(err)
+ }
+ deadline := time.Now().Add(30 * time.Second)
+ for {
+ if got, _ := filepath.Glob(filepath.Join(spool, "carlos-db-*", "sealed", "2")); len(got) == 1 {
+ break
+ }
+ if time.Now().After(deadline) {
+ cmd.Process.Kill()
+ cmd.Wait()
+ t.Fatalf("the CLI never spooled two chunks: %s", stderr.String())
+ }
+ time.Sleep(50 * time.Millisecond)
+ }
+ if err := cmd.Process.Signal(os.Interrupt); err != nil {
+ t.Fatal(err)
+ }
+ if err := cmd.Wait(); err == nil || !strings.Contains(stderr.String(), "stopped") {
+ t.Fatalf("%v: %s", err, stderr.String())
+ }
+ if _, err := os.Stat(out); !os.IsNotExist(err) {
+ t.Fatalf("something was saved at --out: %v", err)
+ }
+ if left, _ := filepath.Glob(filepath.Join(spool, "carlos-db-*")); len(left) != 0 {
+ t.Fatalf("the pull's directory is still there: %v", left)
+ }
+}
+
+// A console from plan 2 forwards queries and not pulls. Its marker is
+// played here by a wrapper that drops "pull" from forward_ops on the way
+// out: the new CLI says DBF-8 and saves nothing. The wrapper buffers each
+// response, so it serves only where no response streams (the list door's
+// 409 is all this test reads).
+func TestDBForwardPullOnAQueryOnlyConsole(t *testing.T) {
+ queryOnly := func(t *testing.T, h http.Handler) (string, []string) {
+ ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ rec := httptest.NewRecorder()
+ h.ServeHTTP(rec, r)
+ body := rec.Body.Bytes()
+ if rec.Code == http.StatusConflict {
+ body = bytes.Replace(body, []byte(`"forward_ops":["query","pull"]`), []byte(`"forward_ops":["query"]`), 1)
+ }
+ for k, vs := range rec.Header() {
+ w.Header()[k] = vs
+ }
+ w.Header().Del("Content-Length")
+ w.WriteHeader(rec.Code)
+ w.Write(body)
+ }))
+ t.Cleanup(ts.Close)
+ return ts.URL, nil
+ }
+ e := newDBFE2E(t, dbfE2EOptions{forward: true, noBox: true, serve: queryOnly})
+ out := filepath.Join(t.TempDir(), "far.db")
+ _, stderr, err := e.run(t, "db", "pull", "--account", "ops", "--app", "shop", "--instance", dbfE2EFar, "--out", out)
+ if err == nil || !strings.Contains(stderr, fmt.Sprintf(dbfNoPull, dbfE2EFar, "uk")) {
+ t.Fatalf("%q %v", stderr, err)
+ }
+ if _, err := os.Stat(out); !os.IsNotExist(err) {
+ t.Fatalf("something was saved at --out: %v", err)
+ }
+}
diff --git a/dbforward_test.go b/dbforward_test.go
index e6710057c..74dd9c2b6 100644
--- a/dbforward_test.go
+++ b/dbforward_test.go
@@ -5,18 +5,26 @@ import (
"context"
"crypto/ecdh"
"crypto/sha256"
+ "database/sql"
"encoding/hex"
"encoding/json"
+ "errors"
"fmt"
+ "io"
"net/http"
"net/http/httptest"
"os"
+ "path/filepath"
"strings"
"sync/atomic"
"testing"
"time"
+ lsfile "github.com/benbjohnson/litestream/file"
+ "github.com/superfly/ltx"
+
"github.com/carlosframework/platform/internal/dbdoor"
+ "github.com/carlosframework/platform/internal/dbdoor/dbdoortest"
"github.com/carlosframework/platform/internal/dbreads"
)
@@ -65,7 +73,12 @@ func summaryOf(rows string) dbreads.Summary {
func readFwd(t *testing.T, s *stream, key *ecdh.PrivateKey) (dbForwardAnswer, string, error) {
t.Helper()
var status bytes.Buffer
- ans, err := readDBForward(context.Background(), &s.buf, key, "shop.example.com", "uk", &status)
+ sink := &memSink{}
+ id, res, err := readDBForward(context.Background(), &s.buf, key, "shop.example.com", "uk", dbfReading, &status, sink, newDBFChunkBound(dbreads.MaxOutputBytes, dbreads.ChunkBytes))
+ if err != nil {
+ return dbForwardAnswer{}, status.String(), err
+ }
+ ans, err := openDBForward(id, key, sink, *res, "shop.example.com", "uk")
return ans, status.String(), err
}
@@ -206,9 +219,9 @@ func TestForwardedQueryCtrlCPrintsNothing(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
var status bytes.Buffer
- ans, err := readDBForward(ctx, &cutReader{r: &s.buf}, key, "shop.example.com", "uk", &status)
- if err == nil || err.Error() != "stopped" || ans.rows != nil {
- t.Fatalf("%q %v", ans.rows, err)
+ _, _, err := readDBForward(ctx, &cutReader{r: &s.buf}, key, "shop.example.com", "uk", dbfReading, &status, &memSink{}, newDBFChunkBound(dbreads.MaxOutputBytes, dbreads.ChunkBytes))
+ if err == nil || err.Error() != "stopped" {
+ t.Fatalf("%v", err)
}
}
@@ -243,7 +256,12 @@ func TestForwardedQueryBoundsChunkCountAndSize(t *testing.T) {
s := newStream(t, key)
write(s)
var status bytes.Buffer
- if _, err := readDBForward(context.Background(), &s.buf, key, "shop.example.com", "uk", &status); err == nil || err.Error() != unchecked {
+ sink := &memSink{}
+ id, res, err := readDBForward(context.Background(), &s.buf, key, "shop.example.com", "uk", dbfReading, &status, sink, newDBFChunkBound(dbreads.MaxOutputBytes, dbreads.ChunkBytes))
+ if err == nil {
+ _, err = openDBForward(id, key, sink, *res, "shop.example.com", "uk")
+ }
+ if err == nil || err.Error() != unchecked {
t.Errorf("%s: %v", name, err)
}
}
@@ -286,3 +304,374 @@ func (c *cutReader) Read(p []byte) (int, error) {
func dbdoorErrNothing() error { return dbdoor.ErrNothingReplicated }
func errVanished() error { return os.ErrNotExist }
+
+// pullStream is a pull's body for the files of dir's replica, in one chunk.
+func pullStream(t *testing.T, key *ecdh.PrivateKey, dir string, mutate func(s *dbreads.Summary, files [][]byte)) *stream {
+ t.Helper()
+ return pullStreamN(t, key, dir, 1, mutate)
+}
+
+// pullStreamN is pullStream with the files' bytes split across n chunks.
+func pullStreamN(t *testing.T, key *ecdh.PrivateKey, dir string, n int, mutate func(s *dbreads.Summary, files [][]byte)) *stream {
+ t.Helper()
+ rd := lsfile.NewReplicaClient(dir)
+ plan, err := dbdoor.PlanFrom(context.Background(), rd, time.Time{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ var files [][]byte
+ var all bytes.Buffer
+ for _, f := range plan.Planned {
+ rc, err := rd.OpenLTXFile(context.Background(), f.Level, ltx.TXID(f.MinTXID), ltx.TXID(f.MaxTXID), 0, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ b, _ := io.ReadAll(rc)
+ rc.Close()
+ files = append(files, b)
+ all.Write(b)
+ }
+ sum := sha256.Sum256(all.Bytes())
+ s := dbreads.Summary{TXID: plan.TXID, At: plan.At, Files: plan.Planned, Bytes: int64(all.Len()), SHA256: hex.EncodeToString(sum[:])}
+ if mutate != nil {
+ mutate(&s, files)
+ }
+ st := newStream(t, key)
+ b := all.Bytes()
+ per := (len(b) + n - 1) / n
+ for seq := 1; seq <= n; seq++ {
+ part := b[min((seq-1)*per, len(b)):min(seq*per, len(b))]
+ sealed, err := dbreads.SealChunk(key.PublicKey(), fwdID, int64(seq), dbreads.StreamLTX, seq == n, part)
+ if err != nil {
+ t.Fatal(err)
+ }
+ st.line(dbreads.Line{Chunk: int64(seq), Data: sealed})
+ }
+ st.result(dbreads.StateDone, "", int64(n), s)
+ return st
+}
+
+func readPull(t *testing.T, st *stream, key *ecdh.PrivateKey) (*dbdoor.Restored, error) {
+ t.Helper()
+ return readPullIn(t, st, key, t.TempDir())
+}
+
+func readPullIn(t *testing.T, st *stream, key *ecdh.PrivateKey, dir string) (*dbdoor.Restored, error) {
+ t.Helper()
+ sink := &fileSink{dir: filepath.Join(dir, "sealed")}
+ id, res, err := readDBForward(context.Background(), &st.buf, key, "shop.example.com", "uk", dbfFetching, io.Discard, sink, newDBFChunkBound(dbreads.MaxPullBytes, dbreads.PullChunkBytes))
+ if err != nil {
+ return nil, err
+ }
+ return openDBPull(context.Background(), id, key, sink, *res, "shop.example.com", "uk", dir)
+}
+
+func TestForwardedPullRestoresTheFiles(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ for _, n := range []int{1, 4} {
+ key, _ := dbreads.NewReplyKey()
+ out := t.TempDir()
+ r, err := readPullIn(t, pullStreamN(t, key, dir, n, nil), key, out)
+ if err != nil {
+ t.Fatalf("%d chunks: %v", n, err)
+ }
+ db, _ := sql.Open("sqlite", "file:"+r.Path+"?mode=ro")
+ var rows int
+ err = db.QueryRow("SELECT count(*) FROM t").Scan(&rows)
+ db.Close()
+ if err != nil || rows != 30 {
+ t.Fatalf("%d chunks: %d rows, %v", n, rows, err)
+ }
+ // Only the copy is left: each spooled chunk went once its bytes were
+ // in the files (so the two are never both whole on the disk), and
+ // the files went once the copy was restored. dbPullForward removes
+ // the empty spool directory.
+ if left, _ := os.ReadDir(filepath.Join(out, "sealed")); len(left) != 0 {
+ t.Errorf("%d chunks: %d spooled chunks are still there", n, len(left))
+ }
+ if _, err := os.Stat(filepath.Join(out, "replica")); !os.IsNotExist(err) {
+ t.Errorf("%d chunks: the files are still there (%v)", n, err)
+ }
+ }
+}
+
+// Review Focus 3 and plan review findings 22 and 24: a file list that does
+// not add up is DBF-13, never a panic, and no file is created before the
+// list is checked whole. Mutations: drop checkPullManifest's per-file size
+// bounds (red: "sizes -1 and N+1" panics or creates files); drop its sum
+// (red: "sizes that do not sum" creates the first file before failing);
+// drop the duplicate check (red: "a file twice" creates a file); drop the
+// level bound (red: "level 12").
+func TestForwardedPullRefusesFilesThatDoNotAddUp(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ for name, mutate := range map[string]func(s *dbreads.Summary, files [][]byte){
+ "sizes that do not sum": func(s *dbreads.Summary, _ [][]byte) { s.Files[0].Size++ },
+ "a list that ends elsewhere": func(s *dbreads.Summary, _ [][]byte) { s.TXID++ },
+ "no files": func(s *dbreads.Summary, _ [][]byte) { s.Files = nil },
+ "sizes -1 and N+1": func(s *dbreads.Summary, _ [][]byte) {
+ s.Files[1].Size += s.Files[0].Size + 1
+ s.Files[0].Size = -1
+ },
+ "a file twice": func(s *dbreads.Summary, _ [][]byte) {
+ s.Files[1] = s.Files[0]
+ s.Bytes = 0
+ for _, f := range s.Files {
+ s.Bytes += f.Size
+ }
+ },
+ "level 12": func(s *dbreads.Summary, _ [][]byte) { s.Files[0].Level = 12 },
+ "a range that runs back": func(s *dbreads.Summary, _ [][]byte) { s.Files[0].MinTXID = s.Files[0].MaxTXID + 1 },
+ "a size past the pull's cap": func(s *dbreads.Summary, _ [][]byte) { s.Files[0].Size = dbreads.MaxPullBytes + 1 },
+ } {
+ key, _ := dbreads.NewReplyKey()
+ out := t.TempDir()
+ func() {
+ defer func() {
+ if p := recover(); p != nil {
+ t.Errorf("%s: panic %v", name, p)
+ }
+ }()
+ if _, err := readPullIn(t, pullStream(t, key, dir, mutate), key, out); err == nil || err.Error() != fmt.Sprintf(dbfUnchecked, "uk") {
+ t.Errorf("%s: %v", name, err)
+ }
+ }()
+ if _, err := os.Stat(filepath.Join(out, "replica")); !os.IsNotExist(err) {
+ t.Errorf("%s: a file was created before the list was checked (%v)", name, err)
+ }
+ }
+}
+
+// The other endings of a pull: too-large is DBF-17 with the planned bytes
+// and the cap, no-room DBF-24, and the rest the query's mapping.
+func TestForwardedPullEndings(t *testing.T) {
+ t.Parallel()
+ for name, c := range map[string]struct {
+ code string
+ sum dbreads.Summary
+ want string
+ }{
+ "too large": {dbreads.CodeTooLarge, dbreads.Summary{Restored: 3 << 30, Cap: 2 << 30},
+ fmt.Sprintf(dbfPullTooLarge, "shop.example.com", humanBytes(ptr(int64(3<<30))), humanBytes(ptr(int64(2<<30))), "uk")},
+ "no room": {dbreads.CodeNoRoom, dbreads.Summary{}, fmt.Sprintf(dbfPullNoRoom, "uk", "shop.example.com")},
+ "nothing replicated": {dbreads.CodeNothingReplicated, dbreads.Summary{}, dbRestoreError("shop.example.com", dbdoor.ErrNothingReplicated).Error()},
+ } {
+ key, _ := dbreads.NewReplyKey()
+ st := newStream(t, key)
+ st.result(dbreads.StateFailed, c.code, 0, c.sum)
+ if _, err := readPull(t, st, key); err == nil || err.Error() != c.want {
+ t.Errorf("%s: %v, want %q", name, err, c.want)
+ }
+ }
+}
+
+func ptr[T any](v T) *T { return &v }
+
+// Plan review finding 37, at the spool: a chunk shorter than a sealed
+// empty one creates no spool file, and a stream of tiny chunks stops at the
+// bound's count, so the spool never holds more files than a read can make.
+// (TestForwardedQueryBoundsChunkCountAndSize pins the count's derivation; a
+// bound of 10 keeps this test small.) Mutation: add to the sink before
+// admit (red: a file for the short chunk, eleven files for the tiny ones).
+func TestForwardedPullSpoolsNoMalformedOrExcessChunk(t *testing.T) {
+ t.Parallel()
+ key, _ := dbreads.NewReplyKey()
+ for name, write := range map[string]func(s *stream) int{
+ "a chunk shorter than its seal": func(s *stream) int {
+ s.line(dbreads.Line{Chunk: 1, Data: []byte("short")})
+ return 0
+ },
+ "tiny chunks past the count": func(s *stream) int {
+ for seq := int64(1); seq <= 11; seq++ {
+ sealed, _ := dbreads.SealChunk(key.PublicKey(), fwdID, seq, dbreads.StreamLTX, false, []byte("x"))
+ s.line(dbreads.Line{Chunk: seq, Data: sealed})
+ }
+ return 10
+ },
+ } {
+ s := newStream(t, key)
+ want := write(s)
+ dir := filepath.Join(t.TempDir(), "sealed")
+ sink := &fileSink{dir: dir}
+ _, _, err := readDBForward(context.Background(), &s.buf, key, "shop.example.com", "uk", dbfFetching, io.Discard, sink,
+ &dbfChunkBound{maxBytes: dbreads.MaxPullBytes, maxChunks: 10})
+ files, _ := os.ReadDir(dir)
+ if err == nil || err.Error() != fmt.Sprintf(dbfUnchecked, "uk") || len(files) != want {
+ t.Errorf("%s: %v, %d spool files, want %d", name, err, len(files), want)
+ }
+ }
+}
+
+// standInConsole answers the list door with the forward marker offering a
+// pull, and the reads route with body (then holds the request open until
+// the client goes when hold is set). It is a stand-in for the console, so
+// that these tests run dbPull itself.
+func standInConsole(t *testing.T, body []byte, hold bool) dbTarget {
+ t.Helper()
+ ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if !strings.HasSuffix(r.URL.Path, "/reads") {
+ w.Header().Set("Content-Type", "application/json")
+ w.WriteHeader(http.StatusConflict)
+ json.NewEncoder(w).Encode(dbdoor.ForwardBody{Error: "shop.example.com keeps its data in region uk.",
+ Forward: 1, Region: "uk", ForwardOps: []string{dbreads.OpQuery, dbreads.OpPull}})
+ return
+ }
+ io.Copy(io.Discard, r.Body)
+ w.Header().Set("Content-Type", "application/x-ndjson")
+ w.WriteHeader(http.StatusOK)
+ w.Write(body)
+ w.(http.Flusher).Flush()
+ if hold {
+ <-r.Context().Done()
+ }
+ }))
+ t.Cleanup(ts.Close)
+ return dbTarget{sess: &apiSession{c: newConsoleClient(ts.URL, "tok"), console: ts.URL}, acct: apiAcct{Seg: "acme"},
+ app: "shop", host: "shop.example.com"}
+}
+
+// leftBehind is what a pull left in its temp dir and at --out.
+func leftBehind(t *testing.T, tmp, out string) {
+ t.Helper()
+ if _, err := os.Stat(out); !os.IsNotExist(err) {
+ t.Errorf("something is at --out: %v", err)
+ }
+ if left, _ := filepath.Glob(filepath.Join(tmp, "carlos-db-*")); len(left) != 0 {
+ t.Errorf("the pull's directory is still there: %v", left)
+ }
+}
+
+// Review Focus 1 and plan review finding 23: a spool that fails part way,
+// through dbPull itself, leaves nothing at --out and removes its directory.
+// Not parallel: it sets TMPDIR and dbPullSpoolWrite. Mutation: drop
+// dbPullForward's deferred RemoveAll (red: carlos-db-* is left).
+func TestForwardedPullLeavesNothingWhenTheSpoolFails(t *testing.T) {
+ tmp := t.TempDir()
+ t.Setenv("TMPDIR", tmp)
+ writes := 0
+ dbPullSpoolWrite = func(name string, b []byte, mode os.FileMode) error {
+ if writes++; writes == 3 {
+ return errors.New("no space left on device")
+ }
+ return os.WriteFile(name, b, mode)
+ }
+ t.Cleanup(func() { dbPullSpoolWrite = os.WriteFile })
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ // The stand-in's chunks are sealed to a key the CLI does not hold; the
+ // pull fails at the spool, before anything is opened.
+ key, _ := dbreads.NewReplyKey()
+ st := pullStreamN(t, key, dir, 4, nil)
+ out := filepath.Join(t.TempDir(), "copy.db")
+ if err := dbPull(context.Background(), standInConsole(t, st.buf.Bytes(), false), out); err == nil {
+ t.Fatal("a spool that failed was accepted")
+ }
+ if writes != 3 {
+ t.Fatalf("%d spool writes: the failure was not part way", writes)
+ }
+ leftBehind(t, tmp, out)
+}
+
+// Review Focus 4 and plan review finding 23: Ctrl-C while chunks are
+// spooling, through dbPull itself, is "stopped", with nothing at --out and
+// the directory removed. Not parallel: it sets TMPDIR and dbPullSpoolWrite.
+// Mutation: drop dbPullForward's deferred RemoveAll (red). The binary's
+// SIGINT is Task 9's TestDBForwardPullInterruptedLeavesNothing.
+func TestForwardedPullCtrlCLeavesNothing(t *testing.T) {
+ tmp := t.TempDir()
+ t.Setenv("TMPDIR", tmp)
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ dbPullSpoolWrite = func(name string, b []byte, mode os.FileMode) error {
+ err := os.WriteFile(name, b, mode)
+ cancel() // Ctrl-C once the first chunk is on disk
+ return err
+ }
+ t.Cleanup(func() { dbPullSpoolWrite = os.WriteFile })
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ key, _ := dbreads.NewReplyKey()
+ st := pullStreamN(t, key, dir, 4, nil)
+ // Admitted, picked and the first chunk only (newStream writes admitted
+ // and picked first), then the stand-in holds the request.
+ lines := bytes.SplitAfter(st.buf.Bytes(), []byte("\n"))
+ out := filepath.Join(t.TempDir(), "copy.db")
+ err := dbPull(ctx, standInConsole(t, bytes.Join(lines[:3], nil), true), out)
+ if err == nil || err.Error() != "stopped" {
+ t.Fatalf("%v", err)
+ }
+ leftBehind(t, tmp, out)
+}
+
+// pullStreamOf is a pull's body for the given bytes and summary, one chunk.
+func pullStreamOf(t *testing.T, key *ecdh.PrivateKey, b []byte, s dbreads.Summary) *stream {
+ t.Helper()
+ st := newStream(t, key)
+ sealed, err := dbreads.SealChunk(key.PublicKey(), fwdID, 1, dbreads.StreamLTX, true, b)
+ if err != nil {
+ t.Fatal(err)
+ }
+ st.line(dbreads.Line{Chunk: 1, Data: sealed})
+ st.result(dbreads.StateDone, "", 1, s)
+ return st
+}
+
+// Astra round 1 on plan 3, finding 1, at the CLI: a valid snapshot of
+// transaction 1, listed as [1,2] with a summary at 2 that adds up in every
+// other way, is DBF-13 and restores nothing: the CLI checks each rebuilt
+// file's LTX header against its manifest entry. Mutation: drop the header
+// comparison (red: it restores and reports transaction 2).
+func TestForwardedPullRefusesARelabelledSnapshot(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 2)
+ rc, err := lsfile.NewReplicaClient(dir).OpenLTXFile(context.Background(), 0, 1, 1, 0, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ b, _ := io.ReadAll(rc)
+ rc.Close()
+ sum := sha256.Sum256(b)
+ s := dbreads.Summary{TXID: 2, Files: []dbdoor.LTXFile{{Level: 0, MinTXID: 1, MaxTXID: 2, Size: int64(len(b))}},
+ Bytes: int64(len(b)), SHA256: hex.EncodeToString(sum[:])}
+ key, _ := dbreads.NewReplyKey()
+ out := t.TempDir()
+ r, err := readPullIn(t, pullStreamOf(t, key, b, s), key, out)
+ if err == nil || err.Error() != fmt.Sprintf(dbfUnchecked, "uk") {
+ t.Fatalf("a relabelled snapshot gave %+v, %v", r, err)
+ }
+ if _, err := os.Stat(filepath.Join(out, "replica.db")); !os.IsNotExist(err) {
+ t.Fatalf("a copy was restored: %v", err)
+ }
+}
+
+// Astra round 1 on plan 3, finding 2, after the download: a pull cancelled
+// once every chunk has arrived is "stopped" and rebuilds no file. Mutation:
+// drop the rebuild loop's context check (red: the files are written and
+// restored before the cancel is seen).
+func TestForwardedPullStopsWhenCancelledAfterTheDownload(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ key, _ := dbreads.NewReplyKey()
+ st := pullStreamN(t, key, dir, 4, nil)
+ out := t.TempDir()
+ sink := &fileSink{dir: filepath.Join(out, "sealed")}
+ id, res, err := readDBForward(context.Background(), &st.buf, key, "shop.example.com", "uk", dbfFetching, io.Discard, sink, newDBFChunkBound(dbreads.MaxPullBytes, dbreads.PullChunkBytes))
+ if err != nil {
+ t.Fatal(err)
+ }
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+ if _, err := openDBPull(ctx, id, key, sink, *res, "shop.example.com", "uk", out); err == nil || err.Error() != "stopped" {
+ t.Fatalf("a cancelled pull answered %v", err)
+ }
+ for _, gone := range []string{"replica", "replica.db"} {
+ if _, err := os.Stat(filepath.Join(out, gone)); !os.IsNotExist(err) {
+ t.Errorf("%s was written after the cancel (%v)", gone, err)
+ }
+ }
+}
diff --git a/docs/site/cli.md b/docs/site/cli.md
index a7b3f8e27..0a8e384f2 100644
--- a/docs/site/cli.md
+++ b/docs/site/cli.md
@@ -574,9 +574,9 @@ own error, `attempt to write a readonly database`. That is on purpose.
Changing a copy would change nothing on the instance, and a command that
seemed to succeed would say otherwise.
-If the app has more than one instance, name one with `--instance`. Only sleeping instances have a database copy. An instance on your own fleet or in a pool cannot be read this way, and the command says which.
+If the app has more than one instance, name one with `--instance`. Only instances that sleep have a database copy. An instance on your own fleet or in a pool cannot be read this way, and the command says which.
-When an instance keeps its data in a different region from the console's, `carlos db query` runs in that region. A server there restores the copy, runs your query and sends back only the rows. The rows are encrypted for your terminal on the way, so nothing in between can read them, and nothing is printed until all of them have arrived and been checked. In that region, the copy can be at most 2 GB once restored, the query runs for at most 5 minutes, and the rows can total at most 64 MB. `carlos db pull` cannot fetch a copy from another region yet.
+When an instance keeps its data in a different region from the console's, `carlos db query` runs in that region. A server there restores the copy, runs your query and sends back only the rows. `carlos db pull` fetches the copy's files from there and restores them on your machine. Both are encrypted for your terminal on the way, so nothing in between can read them, and nothing is printed or saved until all of it has arrived and been checked. In that region, a query's copy can be at most 2 GB once restored, the query runs for at most 5 minutes, and the rows can total at most 64 MB. A pull can fetch at most 2 GB.
### carlos run
diff --git a/docs/superpowers/notes/census-proposal-db-region-forward-3.md b/docs/superpowers/notes/census-proposal-db-region-forward-3.md
new file mode 100644
index 000000000..4dcc5447a
--- /dev/null
+++ b/docs/superpowers/notes/census-proposal-db-region-forward-3.md
@@ -0,0 +1,76 @@
+# Census: db region forward, plan 3 (`carlos db pull` and `pull --at` for an instance in another region)
+
+Status: ACCEPTED in full by the census owner on #280, 2026-10-23, as
+`census-ruling-db-region-forward-3`, for every row below and the round 1 fix
+rows. Every row now carries `rev3=census-ruling-db-region-forward-3`; the
+`@legacy` fingerprints were recomputed and the seal moved to
+`79cb56b4c7d3a8e155d1eff1902715d818fcf4c09b1d8a82921c413b116938a9`.
+
+The census owner's rationale: the box-side reads stay the agent's or the
+sandboxed worker's; the CLI reads only its own spool; the `admitDBRead`
+re-pin is the same write class; Astra matched every digest.
+
+The branch is `db-region-forward-3`, cut from amadan `main` at 0d9e81dc. It
+builds plan 3 of `docs/superpowers/specs/2026-10-22-db-region-forward-design.md`
+(rev 7): the box's worker holds a pull's planned LTX files, verifies their
+checksums and sends them back after the agent's check B; the agent seals them
+as the `ltx` stream in 1 MiB chunks, with the plan in the summary; the console
+forwards and audits the new op; the CLI spools the sealed chunks to a private
+directory, checks the file list, the count and the SHA-256, restores locally
+with `dbdoor.RestoreLocal` and saves the copy with today's `copyOut`. The
+box never restores for a pull.
+
+It adds one root-read overlay,
+`internal/provision/testdata/plan14-root-reads-db-region-forward-3.tsv`,
+loaded last in `TestPlan14RootReadCensus`, after `static-redirects` (the
+overlay main loads last at 0d9e81dc). Its record-write entry is layered on
+`plans3aWriteCensusInventory` in
+`internal/provision/db_region_forward_3_census_test.go`.
+
+## Root reads
+
+Executing identities, named once:
+
+- **the worker**: `carlos system db-worker` under `carlos-dbread@.service`, a
+ DynamicUser with no network, its own PrivateTmp, `/data` and `/etc/carlos`
+ hidden.
+- **the CLI**: `carlos db pull` on a person's machine, as the person.
+
+The scanner names no agent, console or host unit: the agent's pull branch in
+`attempt`, `NewAgent`'s `MaxPull`, the console's op check and audit detail,
+and `CARLOS_DB_PULL_MAX_BYTES` in `dbReadsSettings` make no read it counts
+(plan 1's R8 again).
+
+| unit | change | executing identity | proposed disposition |
+|---|---|---|---|
+| `internal/dbreads/worker.go` `holdAndSend` | new: opens each planned LTX file it received and verified, in its own private `/var/tmp`, after check B, and copies it back over its socket | the worker | legacy, accepted-legacy, J4 (`root-input`, as `receiveFile`) |
+| `internal/dbreads/worker.go` `verifyLTX` | new: opens one received LTX file and decodes it whole to check its checksums, before ready | the worker | legacy, accepted-legacy, J4 |
+| `internal/dbreads/worker.go` `ServeWorker` | changed: accepts a pull header and hands it to `holdAndSend` after the same bounded receive; the `Stat` is unchanged | the worker | re-frozen at `census-ruling-db-region-forward-1` with this change appended |
+| `dbforward.go` `dbPullForward` | new: the same bounded 64 KiB read of a refused reads-route answer as `dbQueryForward` (`io.ReadAll` over the response body, not a file); creates and removes the pull's private 0700 directory | the CLI | legacy, accepted-legacy, J4 (main's footing for `db.go`'s units) |
+| `dbforward.go` `fileSink.chunk` | new: reads back one sealed chunk it spooled itself, 0600 in that directory, admitted by the chunk bound before it was written | the CLI | legacy, accepted-legacy, J4 |
+| `dbforward.go` `openDBPull` | new: creates each planned LTX file exclusively (`O_EXCL`, 0600) in that directory, only after the whole file list is checked; then restores, checks and removes them | the CLI | legacy, accepted-legacy, J4 |
+| `internal/dbdoor/plan_from.go` `ctxReadCloser.Read` | new (Astra round 1, finding 2): `RestoreLocal` wraps its local replica's reads so each checks the restore's context first; litestream's compaction checks none of its own | the CLI and the worker | legacy, accepted-legacy, J4 (as main's `downloaded.Read`) |
+| `dbforward.go` `dbQueryForward` | changed: keeps its chunks in a `memSink` and opens them after `readDBForward` returns the result; the same bounded refusal read | the CLI | re-frozen at `census-ruling-db-region-forward-2` with this change appended |
+
+Round 1 (Astra on cc77ca95): `ServeWorker` now checks every received file's
+LTX header against its plan entry through `verifyLTX`, for a query as well as a
+pull, and the plan's end against its target; `holdAndSend` only sends the checked
+files back; `openDBPull` compares each rebuilt file's header with its manifest
+entry and checks its context per chunk. Their rows carry a round 1 clause; no
+read is added to them.
+
+Envelopes re-pinned: `.`, `internal/console`, `internal/dbdoor`,
+`internal/dbreads`, `internal/host`. Frozen declarations are added, so the seal moves:
+static-redirects' `65857c29…` is checked first, the proposed seal was
+`36ecd2eca043f1d0056f086be47ba1fb887dcdc2eda67427e698c2b6d1b9031b` (79cb56b4… once the markers flipped).
+
+## Record writes (9B8)
+
+`internal/console/apidbforward.go` `Server.admitDBRead` is re-digested and
+stays `nonrecord`: it admits a pull beside a query and audits it with its own
+detail; its writes are the same conditional creates under `db-reads/` in the
+deployment bucket. No write site is added.
+
+## Unchanged
+
+Claim routes (no new route: a pull uses plan 2's reads route), `go.mod`.
diff --git a/docs/superpowers/notes/census-proposal-sec-mail-user-boundary.md b/docs/superpowers/notes/census-proposal-sec-mail-user-boundary.md
index 2594bfb12..38215b7ea 100644
--- a/docs/superpowers/notes/census-proposal-sec-mail-user-boundary.md
+++ b/docs/superpowers/notes/census-proposal-sec-mail-user-boundary.md
@@ -33,7 +33,7 @@ root `carlos` binary links neither `internal/console` nor `internal/mail`
## Root-read census (TestPlan14RootReadCensus)
Overlay `internal/provision/testdata/plan14-root-reads-sec-mail-user-boundary.tsv`,
-loaded last, after `plan14-root-reads-static-redirects.tsv`. Without it the
+loaded last, after `plan14-root-reads-wake-latency-1.tsv`. Without it the
test reports 4 discrepancies; with it, none.
### Changed unit, re-frozen at main's disposition
@@ -56,9 +56,9 @@ no rows.
| `cmd/console` | `serve`, the new `mailUserBoundary`, the header doc comment, `mailDeployment`'s doc comment, and `TestMailUserBoundary` |
| `internal/mail` | `aws.go`: the boundary field, `checkUserBoundary`/`CheckUserBoundary`, `ErrNoUserBoundary`, `CreateUser`'s field, `backfillBoundary`, `PutUserPermissionsBoundary` on `iamAPI`; and the new `aws_iam_test.go` |
-Seal: `65857c29501dc7676f52921bea79d08217542ecb1d775c67d20a3460c851d831`
-(static-redirects, checked first) to
-`d82a9191c6be745d34211a8add6ac0e63dc35b28c3f6bcde99cdbc3f67154f62`.
+Seal: `09e20ea4bbc534710334e632e903878656dabfb5817d4d96c6e7c56508172148`
+(wake-latency-1, checked first) to
+`7b0701e08da6210d02ed0f76eabe4548d7139a772ec78c358480e1586b29c9cd`.
## Claim authority census (TestClaimAuthoritySourceCensus)
diff --git a/docs/superpowers/notes/census-proposal-wake-latency-1.md b/docs/superpowers/notes/census-proposal-wake-latency-1.md
new file mode 100644
index 000000000..522f74564
--- /dev/null
+++ b/docs/superpowers/notes/census-proposal-wake-latency-1.md
@@ -0,0 +1,64 @@
+# Census: wake latency, pass 1
+
+Status: ACCEPTED as proposed by the census owner on #280, 2026-10-24, as
+`census-ruling-wake-latency-1`. Every row now carries
+`rev3=census-ruling-wake-latency-1`; the `@legacy` fingerprints were
+recomputed and the seal moved to
+`09e20ea4bbc534710334e632e903878656dabfb5817d4d96c6e7c56508172148`.
+
+The census owner's rationale (the recommendation, accepted): clock reads and
+journal lines only, no read added or moved, the same class as the accepted
+envelope re-pins.
+
+The branch is `wake-latency-1`, cut from amadan `main` at 64a002c8. A dev
+wake took 843 ms for an app that boots in milliseconds, and a box-wide
+`systemctl daemon-reload` inside instance-control's generation start took
+297 ms of it (dev box journal, 2026-10-24 11:16:31). The branch:
+
+- starts a generation without the reload when systemd already reads its
+ drop-in (`internal/instancectl/gen.go`, `startGen` and
+ `genDropInApplied`): one `systemctl show` of Slice, NeedDaemonReload and
+ the tier's three memory lines, and a reload only when any of them is not
+ the drop-in's, then the same check again before the start;
+- backs off health probes from 5 ms to 50 ms instead of a fixed 150 ms
+ (`waitHealthy`, `waitHealthyAddr`);
+- logs per-step wake timing to the journal (`wakeSteps`).
+
+It adds one root-read overlay,
+`internal/provision/testdata/plan14-root-reads-wake-latency-1.tsv`, loaded
+last in `TestPlan14RootReadCensus`, after `db-region-forward-3` (the overlay
+main loads last at 64a002c8).
+
+## Root reads
+
+Executing identity, for every unit: the agent (`/opt/carlos/carlos agent`,
+`carlos-edge.service`, user carlos).
+
+| unit | change | proposed disposition |
+|---|---|---|
+| `internal/activator/activator.go` `Service.doWake` | marks the end of each wake step on the wake's timer: `a.now()` and a slice append, journal only. No read added, no step moved | re-frozen at `census-ruling-292fix` with this change appended |
+| `internal/activator/activator.go` `Service.wake` | logs that timer's breakdown after a successful wake. No read added | re-frozen at `census-ruling-carlos-run` with this change appended |
+| `internal/activator/backend_exec.go` `ExecBackend.startUnitAt` | times the unit's launch and its health wait with `time.Now()`, and logs both after a successful health wait. The log names `rt.Host`, which is the new `registry:Host` site. No read added, no behaviour changed | re-frozen at J1 (`root-route`) with this change appended |
+
+Frozen declarations are added, so the seal moves from
+`79cb56b4c7d3a8e155d1eff1902715d818fcf4c09b1d8a82921c413b116938a9`
+(`db-region-forward-3`'s, checked before the overlay loads) to
+`09e20ea4bbc534710334e632e903878656dabfb5817d4d96c6e7c56508172148` (it
+was `5976190a…4b1a` while the rows were PROPOSED).
+
+Two envelopes are re-pinned:
+
+| envelope | why |
+|---|---|
+| `internal/activator` | the units above, `wakesteps.go`, and the health-probe backoff in `backend_exec.go` and `backend_unit.go` |
+| `internal/instancectl` | `startGen`'s drop-in check in `gen.go`, and the timings on its start log line |
+
+`startGen` is outside the scanner's units: its `systemctl` calls run as
+instance-control (`carlos-instance-control.service`, root), the same calls
+in a different order, and the reload is now conditional. Review: Astra round
+1 found one P2 (NeedDaemonReload is mtime-based, so a reused and
+still-loaded generation could read as applied). It was fixed by checking the
+tier's memory lines, and round 2 was clean.
+
+Record-write census (9B8): unchanged. Claim routes: unchanged. go.mod:
+unchanged.
diff --git a/docs/superpowers/plans/2026-10-22-db-region-forward-copy.md b/docs/superpowers/plans/2026-10-22-db-region-forward-copy.md
index 8c7a8e5d1..a4a1db96c 100644
--- a/docs/superpowers/plans/2026-10-22-db-region-forward-copy.md
+++ b/docs/superpowers/plans/2026-10-22-db-region-forward-copy.md
@@ -24,8 +24,8 @@ The rules:
`\n\n`. In `docs/site/cli.md` each paragraph of a row is one line.
- No em dashes. A full stop and a new sentence instead, continuing in lower
case. Docs prose keeps the CLI page's sentence case.
-- Ids are not renumbered. DBF-17 to DBF-20 and DBF-23 to DBF-29 are kept
- for plan 3's pull rows.
+- Ids are not renumbered. DBF-17 to DBF-20, DBF-23 and DBF-24 are plan 3's
+ pull rows; DBF-25 to DBF-29 stay free.
Placeholders: `<host>` the instance's hostname; `<region>` a placement code
such as uk; `<size>` and `<cap>` byte sizes as the CLI prints them; `<n>` a
@@ -61,6 +61,20 @@ RFC 3339 time.
| DBF-16 | dbforward.go | <host> is moving between regions. read its database once the move has finished. | approved |
| DBF-32 | dbforward.go | this query cannot be sent to region <region>, because it contains invalid UTF-8 or the replacement character U+FFFD. fix the query and run it again. | approved |
+## Pull (plan 3)
+
+Approved by Luna on 2026-10-23: DBF-18, DBF-19, DBF-20 and DBF-24 as drafted, DBF-17 and DBF-23 rewritten.
+DBF-23 replaces DBF-30 on the CLI page; DBF-30 stays here, approved, and leaves the built list.
+
+| id | file | text | status |
+|---|---|---|---|
+| DBF-17 | dbforward.go | the database for <host> has <size> of files to fetch. carlos db pull can fetch at most <cap> from region <region>. | approved |
+| DBF-18 | dbforward.go | fetching a copy of <host>'s database from region <region>. | approved |
+| DBF-19 | internal/console/apidbforward.go | forwarded a database pull to region <region> (cli) | approved |
+| DBF-20 | internal/console/apidbforward.go | forwarded a database pull to region <region>, as of <time> (cli) | approved |
+| DBF-23 | docs/site/cli.md | If the app has more than one instance, name one with `--instance`. Only instances that sleep have a database copy. An instance on your own fleet or in a pool cannot be read this way, and the command says which.\n\nWhen an instance keeps its data in a different region from the console's, `carlos db query` runs in that region. A server there restores the copy, runs your query and sends back only the rows. `carlos db pull` fetches the copy's files from there and restores them on your machine. Both are encrypted for your terminal on the way, so nothing in between can read them, and nothing is printed or saved until all of it has arrived and been checked. In that region, a query's copy can be at most 2 GB once restored, the query runs for at most 5 minutes, and the rows can total at most 64 MB. A pull can fetch at most 2 GB. | approved |
+| DBF-24 | dbforward.go | the server in region <region> does not have room to prepare a copy of <host>'s database right now. try again later. | approved |
+
## Docs
| id | file | text | status |
diff --git a/docs/superpowers/specs/2026-10-22-db-region-forward-design.md b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md
index 23500f4fe..238a09614 100644
--- a/docs/superpowers/specs/2026-10-22-db-region-forward-design.md
+++ b/docs/superpowers/specs/2026-10-22-db-region-forward-design.md
@@ -1,6 +1,8 @@
# `carlos db` for instances whose data lives in another region
-**Status:** rev 6, 2026-10-22. Rev 6 changes no mechanism: it states four
+**Status:** rev 7, 2026-10-23. Rev 7 (plan 3, 2026-10-23): the pull's cap and chunk size, from a measurement on dev
+(§5.7); the request bound is unchanged.
+Rev 6, 2026-10-22, changes no mechanism: it states four
things the plan review (`CODEX-FINDINGS-DB-REGION-FORWARD-PLANS.md`) found
the plans had to decide, each marked **(plan review N)**: what a result
names when a chunk cannot be confirmed (§5.4, 9), a box's `cancel` read
@@ -482,6 +484,8 @@ the output. The CLI maps codes to today's sentences where one exists
| Disk | a reservation per read (below) | agent |
| Output | 64 MiB (`runs.MaxOutputBytes`, written "64 MB" in copy as the run's docs do); the read ends `output-cap` | agent; the CLI's buffer is bounded by the same number |
| Chunk | 256 KiB or 1 s, whichever comes first (`runs.ChunkBytes`, `runs.ChunkEvery`) | agent |
+| Pull | at most 2 GiB of planned LTX files (measured on dev, 2026-10-23: relay rate 3.94 MB/s, the slowest of three runs; the rule gives 3.25 GiB, above the 2 GiB ceiling); `CARLOS_DB_PULL_MAX_BYTES` on a box can only lower it; the read ends `too-large` | agent, from the plan, before any byte moves; the CLI's spool is bounded by the same number |
+| Pull chunk | 1 MiB or 1 s, whichever comes first | agent |
| Reads per box | 2, plus one connection kept for the readiness probe (`MaxConnections=3`) | agent, systemd |
| Open reads per person | 2 | console, held by the open request (J5) |
| Open reads per console | 16 | console, held by the open request, 429 with `Retry-After` |
@@ -1179,6 +1183,8 @@ All five are now Paul's.
2. **Limits stand.** 2 GiB restored, 5 minutes, 64 MB of rows, two reads per
box. Real database sizes on the flagship are still unmeasured; plan 2's
dev check reports them.
+ The pull's cap, 2 GiB of planned LTX files (§5.7, rev 7, measured on dev
+ 2026-10-23), is put to Paul as a limit: **pending his confirmation.**
3. **Audit.** The row records the SQL's length and a SHA-256 prefix, not its
text (§9.3). The SQL itself rests in the request object until expiry.
4. **Copy casing.** New strings use Paul's voice rule: lower case after a
diff --git a/internal/activator/activator.go b/internal/activator/activator.go
index dbd881d66..8aa16e0f5 100644
--- a/internal/activator/activator.go
+++ b/internal/activator/activator.go
@@ -1799,6 +1799,9 @@ func (a *Service) wake(e *actEntry) {
}
} else {
emitEvent(a.Logs, e.route, "info", wakeEventMessage(w, e.worker))
+ if steps := w.steps.String(); steps != "" {
+ log.Printf("activator: %s: woke in %s: %s", e.host, w.wakeTook.Round(time.Millisecond), steps)
+ }
}
var (
@@ -2063,6 +2066,8 @@ type wokenInstance struct {
// `carlos logs` already carries the line.
spawnTook time.Duration
wakeTook time.Duration
+ // steps is the operator's breakdown of wakeTook, journal only.
+ steps wakeSteps
// planned is the generation a planned wake left serving; failed and
// failReason name a first wake's build that failed; fellBack says the
@@ -2098,6 +2103,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
// it was (seeded pre-land re-review, P2; census ruling 292fix).
w := &wokenInstance{stopOutcome: &stopObservation{}}
w.stopOutcome.notStarted()
+ w.steps.begin(a.now())
if ctx.Err() != nil || !a.wakeAllowed(host) {
return w, ErrFenced
}
@@ -2239,6 +2245,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
// single writer. Contention is not retried here — on one box it means a
// previous incarnation hasn't released; tell the client to come back after
// the TTL instead of stalling its request.
+ w.steps.mark(a.now(), "checks")
*w.stopOutcome = stopObservation{}
lease, err := a.backend.Acquire(ctx, rt)
if err != nil {
@@ -2253,6 +2260,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
}
return w, err
}
+ w.steps.mark(a.now(), "lease")
if ctx.Err() != nil || !a.wakeAllowed(host) {
w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
@@ -2295,6 +2303,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
}
return w, fmt.Errorf("restore %s: %w", host, err)
}
+ w.steps.mark(a.now(), "restore")
if ctx.Err() != nil || !a.wakeAllowed(host) {
w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
return w, ErrFenced
@@ -2354,6 +2363,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
ctx = wctx
}
}
+ w.steps.mark(a.now(), "identity")
// 3. First-boot seed, strictly between the restore and the spawn — so it
// runs before Start's first health probe, and a brand-new instance is born
@@ -2386,6 +2396,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
return w, fmt.Errorf("seed %s: %w", host, err)
}
w.stopOutcome.uncertain = false
+ w.steps.mark(a.now(), "seed")
}
// 4. Spawn before replication starts: a fresh instance creates its own DB
@@ -2527,6 +2538,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
spawnVer, w.gen = g.Build, g
}
spawnTook := a.now().Sub(spawnStart)
+ w.steps.mark(a.now(), "spawn")
w.proc = proc
if err != nil {
// A start whose kill was unproven may have left its generation
@@ -2575,6 +2587,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
a.cleanupFailedWake(ctx, rt, w, lease)
return w, fmt.Errorf("start %s: %w", host, serr)
}
+ w.steps.mark(a.now(), "settle")
}
if ctx.Err() != nil || !a.wakeAllowed(host) {
@@ -2601,6 +2614,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
log.Printf("activator: %s: seed placement not recorded yet: %v", host, err)
}
}
+ w.steps.mark(a.now(), "replicate")
// 6. Co-spawn the sidecar, last — only once the instance is healthy and
// replicating, so the sidecar always boots against a live instance (its
@@ -2633,6 +2647,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
} else {
w.sidecar = sc
}
+ w.steps.mark(a.now(), "sidecar")
}
return w, nil
}
diff --git a/internal/activator/backend_exec.go b/internal/activator/backend_exec.go
index 738ff0ccc..6656e8ee9 100644
--- a/internal/activator/backend_exec.go
+++ b/internal/activator/backend_exec.go
@@ -2547,11 +2547,13 @@ func (b *ExecBackend) spawnUnitFrom(ctx context.Context, rt registry.Route, sock
// recorded (StartGenExe), which root binds in place of the link. tier is
// exe's own; zero is the live tier (b.tier).
func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock string, n uint64, exe *instancectl.StagedExe, tier cgroup.Limit) (Proc, string, error) {
+ began := time.Now()
p, spawnVer, err := b.spawnUnitFrom(ctx, rt, sock, n, exe, tier)
if err != nil {
return p, "", err
}
up := p.(*unitProc)
+ launched := time.Now()
if err := waitHealthyCtx(ctx, sock, spawnHealthTimeout); err != nil {
// Set before the Stop call below, closing the same race Stop's own
// internal set leaves open by a few instructions: observe runs
@@ -2582,6 +2584,10 @@ func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock s
}
return nil, "", err
}
+ // The spawn's own split, for the journal: the unit's launch (instance
+ // control's half included) and then the app's boot to its socket.
+ log.Printf("activator: %s: %s launched in %s, healthy %s later", rt.Host, unitName(instancectl.RoleRun, rt.Host, n),
+ launched.Sub(began).Round(time.Millisecond), time.Since(launched).Round(time.Millisecond))
b.normalizeSocketMode(rt, sock, up)
return up, spawnVer, nil
}
@@ -3239,6 +3245,7 @@ func waitHealthy(socket string, timeout time.Duration) error {
deadline := time.Now().Add(timeout)
client := socketClient(socket)
var last error
+ var poll time.Duration
for time.Now().Before(deadline) {
resp, err := client.Get("http://carlos/")
if err == nil {
@@ -3250,11 +3257,27 @@ func waitHealthy(socket string, timeout time.Duration) error {
} else {
last = err
}
- time.Sleep(150 * time.Millisecond)
+ poll = nextHealthPoll(poll)
+ time.Sleep(poll)
}
return fmt.Errorf("instance not healthy within %s: %v", timeout, last)
}
+// The wait between health probes. A request is held open for the whole of
+// a wake, and a fixed 150 ms gap cost it 75 ms on average after the app was
+// already listening. A refused dial costs microseconds, so the wait starts
+// short and doubles to a ceiling that keeps a slow boot from being probed
+// in a busy loop.
+const (
+ healthPollMin = 5 * time.Millisecond
+ healthPollMax = 50 * time.Millisecond
+)
+
+// nextHealthPoll is the wait after one of d; zero gives the first.
+func nextHealthPoll(d time.Duration) time.Duration {
+ return min(max(2*d, healthPollMin), healthPollMax)
+}
+
// waitHealthyCtx is waitHealthy bounded by ctx as well as its own timeout, so
// an aborted wake doesn't keep probing a socket nobody wants anymore.
func waitHealthyCtx(ctx context.Context, socket string, timeout time.Duration) error {
diff --git a/internal/activator/backend_unit.go b/internal/activator/backend_unit.go
index b970b8c0b..e6b64abb3 100644
--- a/internal/activator/backend_unit.go
+++ b/internal/activator/backend_unit.go
@@ -251,6 +251,7 @@ func waitHealthyAddr(addr string, timeout time.Duration) error {
deadline := time.Now().Add(timeout)
client := &http.Client{Timeout: 3 * time.Second}
var last error
+ var poll time.Duration
for time.Now().Before(deadline) {
resp, err := client.Get("http://" + addr + "/")
if err == nil {
@@ -262,7 +263,8 @@ func waitHealthyAddr(addr string, timeout time.Duration) error {
} else {
last = err
}
- time.Sleep(150 * time.Millisecond)
+ poll = nextHealthPoll(poll)
+ time.Sleep(poll)
}
return fmt.Errorf("instance not healthy within %s: %v", timeout, last)
}
diff --git a/internal/activator/healthpoll_test.go b/internal/activator/healthpoll_test.go
new file mode 100644
index 000000000..0ad398ecc
--- /dev/null
+++ b/internal/activator/healthpoll_test.go
@@ -0,0 +1,24 @@
+package activator
+
+import (
+ "testing"
+ "time"
+)
+
+// The wait between health probes starts short, so a fast app is noticed
+// within milliseconds of binding, and doubles to a ceiling, so a slow one
+// is not probed in a busy loop.
+func TestHealthPollBacksOffFromShortToACeiling(t *testing.T) {
+ var got []time.Duration
+ d := time.Duration(0)
+ for range 7 {
+ d = nextHealthPoll(d)
+ got = append(got, d)
+ }
+ want := []time.Duration{5, 10, 20, 40, 50, 50, 50}
+ for i := range want {
+ if got[i] != want[i]*time.Millisecond {
+ t.Fatalf("delays = %v, want %v ms", got, want)
+ }
+ }
+}
diff --git a/internal/activator/wakesteps.go b/internal/activator/wakesteps.go
new file mode 100644
index 000000000..68dd3528b
--- /dev/null
+++ b/internal/activator/wakesteps.go
@@ -0,0 +1,40 @@
+package activator
+
+import (
+ "strings"
+ "time"
+)
+
+// wakeSteps times one wake step by step for the operator's journal. The
+// member's wake event carries the total and the spawn (wakeEventMessage);
+// this says where the rest went, which is what a slow wake's diagnosis
+// needs and what no single figure can show.
+type wakeSteps struct {
+ last time.Time
+ steps []wakeStep
+}
+
+type wakeStep struct {
+ name string
+ took time.Duration
+}
+
+// begin starts the clock. A mark before it records nothing.
+func (s *wakeSteps) begin(now time.Time) { s.last = now }
+
+// mark closes the step called name at now and starts the next.
+func (s *wakeSteps) mark(now time.Time, name string) {
+ if s.last.IsZero() {
+ return
+ }
+ s.steps = append(s.steps, wakeStep{name, now.Sub(s.last)})
+ s.last = now
+}
+
+func (s *wakeSteps) String() string {
+ parts := make([]string, len(s.steps))
+ for i, st := range s.steps {
+ parts[i] = st.name + " " + st.took.Round(time.Millisecond).String()
+ }
+ return strings.Join(parts, ", ")
+}
diff --git a/internal/activator/wakesteps_test.go b/internal/activator/wakesteps_test.go
new file mode 100644
index 000000000..1ac2c04c2
--- /dev/null
+++ b/internal/activator/wakesteps_test.go
@@ -0,0 +1,30 @@
+package activator
+
+import (
+ "testing"
+ "time"
+)
+
+// Each mark records the time since the one before it, under the step's
+// name, so the line reads as where the wake spent its time.
+func TestWakeStepsNameTheTimeSinceThePreviousMark(t *testing.T) {
+ t0 := time.Unix(1000, 0)
+ var s wakeSteps
+ s.begin(t0)
+ s.mark(t0.Add(3*time.Millisecond), "checks")
+ s.mark(t0.Add(83*time.Millisecond), "lease")
+ s.mark(t0.Add(83400*time.Microsecond), "restore")
+ s.mark(t0.Add(752*time.Millisecond), "spawn")
+ if got, want := s.String(), "checks 3ms, lease 80ms, restore 0s, spawn 669ms"; got != want {
+ t.Errorf("String = %q, want %q", got, want)
+ }
+}
+
+// A wake that never began (the claim path, a nil entry) has nothing to say.
+func TestWakeStepsWithoutABeginRecordNothing(t *testing.T) {
+ var s wakeSteps
+ s.mark(time.Unix(1000, 0), "lease")
+ if got := s.String(); got != "" {
+ t.Errorf("String = %q, want empty", got)
+ }
+}
diff --git a/internal/console/apidbforward.go b/internal/console/apidbforward.go
index 6a2181415..3b6a66777 100644
--- a/internal/console/apidbforward.go
+++ b/internal/console/apidbforward.go
@@ -144,7 +144,7 @@ func (s *Server) dbResidency(w http.ResponseWriter, r *http.Request, rec provisi
}
writeJSON(w, http.StatusConflict, dbdoor.ForwardBody{
Error: fmt.Sprintf(dbfUpdateCarlos, rec.Host, rec.Region), Forward: 1,
- Region: rec.Region, ForwardOps: []string{dbreads.OpQuery},
+ Region: rec.Region, ForwardOps: []string{dbreads.OpQuery, dbreads.OpPull},
})
}
@@ -157,11 +157,25 @@ const (
dbfAuditAt = "forwarded a database query to region %s, as of %s. the query was %d bytes, sha256 %s (cli)"
)
+// The pull's audit details (copy record DBF-19, DBF-20), approved by Luna
+// on 2026-10-23.
+const (
+ dbfAuditPull = "forwarded a database pull to region %s (cli)"
+ dbfAuditPullAt = "forwarded a database pull to region %s, as of %s (cli)"
+)
+
const dbReadRetryAfter = "30"
-// dbForwardAuditDetail is the db-read row's detail (spec §9.3): the SQL's
-// length and the first 16 hex digits of its SHA-256, never its text.
-func dbForwardAuditDetail(region, sqlText string, at time.Time) string {
+// dbForwardAuditDetail is the db-read row's detail (spec §9.3): for a query
+// the SQL's length and the first 16 hex digits of its SHA-256, never its
+// text; a pull carries no SQL.
+func dbForwardAuditDetail(op, region, sqlText string, at time.Time) string {
+ if op == dbreads.OpPull {
+ if at.IsZero() {
+ return fmt.Sprintf(dbfAuditPull, region)
+ }
+ return fmt.Sprintf(dbfAuditPullAt, region, at.UTC().Format(time.RFC3339))
+ }
sum := sha256.Sum256([]byte(sqlText))
sha := hex.EncodeToString(sum[:])[:16]
if at.IsZero() {
@@ -326,7 +340,15 @@ func (s *Server) admitDBRead(w http.ResponseWriter, r *http.Request) (dbReadAdmi
jsonError(w, http.StatusConflict, fmt.Sprintf(dbfMoving, host))
return none, "", false
}
- if body.Op != dbreads.OpQuery {
+ switch body.Op {
+ case dbreads.OpQuery:
+ case dbreads.OpPull:
+ // A pull runs nothing: SQL in one is a request no CLI builds.
+ if body.SQL != "" {
+ http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
+ return none, "", false
+ }
+ default:
http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
return none, "", false
}
@@ -340,13 +362,15 @@ func (s *Server) admitDBRead(w http.ResponseWriter, r *http.Request) (dbReadAdmi
http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
return none, "", false
}
- if err := dbquery.CheckSQL(body.SQL); err != nil || body.SQL == "" {
- msg := http.StatusText(http.StatusBadRequest)
- if err != nil {
- msg = err.Error()
+ if body.Op == dbreads.OpQuery {
+ if err := dbquery.CheckSQL(body.SQL); err != nil || body.SQL == "" {
+ msg := http.StatusText(http.StatusBadRequest)
+ if err != nil {
+ msg = err.Error()
+ }
+ jsonError(w, http.StatusBadRequest, msg)
+ return none, "", false
}
- jsonError(w, http.StatusBadRequest, msg)
- return none, "", false
}
var at time.Time
if body.At != "" {
@@ -430,7 +454,7 @@ func (s *Server) admitDBRead(w http.ResponseWriter, r *http.Request) (dbReadAdmi
return none, "", false
}
if err := s.store.RecordActivityContext(ctx, acct.ID, app, actor, Activity{Action: dbAuditAction,
- TargetType: "instance", Target: host, Detail: dbForwardAuditDetail(rec.Region, body.SQL, at)}, time.Now()); err != nil {
+ TargetType: "instance", Target: host, Detail: dbForwardAuditDetail(body.Op, rec.Region, body.SQL, at)}, time.Now()); err != nil {
return fail("audit", err)
}
if ended("request", false) {
diff --git a/internal/console/apidbforward_test.go b/internal/console/apidbforward_test.go
index 07c395eb5..cf69f06f2 100644
--- a/internal/console/apidbforward_test.go
+++ b/internal/console/apidbforward_test.go
@@ -206,7 +206,7 @@ func TestListDoorMarksACapableRegion(t *testing.T) {
var got dbdoor.ForwardBody
json.Unmarshal([]byte(body), &got)
if status != http.StatusConflict || got.Forward != 1 || got.Region != "uk" ||
- !slices.Equal(got.ForwardOps, []string{dbreads.OpQuery}) ||
+ !slices.Equal(got.ForwardOps, []string{dbreads.OpQuery, dbreads.OpPull}) ||
got.Error != fmt.Sprintf(dbfUpdateCarlos, dbfHost, "uk") {
t.Fatalf("%d %s", status, body)
}
@@ -297,8 +297,10 @@ func TestDBReadRouteRefusesBeforeAdmitting(t *testing.T) {
Move: &provision.MoveState{ID: "mv_x", To: "ie", State: "requested"}})
two := good
two.SQL = "SELECT 1; SELECT 2"
- pull := good
- pull.Op, pull.SQL = dbreads.OpPull, ""
+ shell := good
+ shell.Op = "shell"
+ pullSQL := good
+ pullSQL.Op = dbreads.OpPull
badKey := good
badKey.ReplyKey = "AAAA"
future := good
@@ -313,7 +315,8 @@ func TestDBReadRouteRefusesBeforeAdmitting(t *testing.T) {
"no such instance": {"nope.example.com", good, http.StatusNotFound, ""},
"moving": {"moving.example.com", good, http.StatusConflict, fmt.Sprintf(dbfMoving, "moving.example.com")},
"two statements": {dbfHost, two, http.StatusBadRequest, dbquery.ErrOneStatement.Error()},
- "an op this console does not forward": {dbfHost, pull, http.StatusBadRequest, ""},
+ "an op this console does not forward": {dbfHost, shell, http.StatusBadRequest, ""},
+ "a pull carrying SQL": {dbfHost, pullSQL, http.StatusBadRequest, ""},
"a reply key that is not a key": {dbfHost, badKey, http.StatusBadRequest, ""},
"a time in the future": {dbfHost, future, http.StatusBadRequest, ""},
} {
@@ -394,7 +397,7 @@ func TestDBReadQueuesASignedRequestAndAudits(t *testing.T) {
t.Fatalf("the serial was not bumped: %v", err)
}
evs, _ := fx.st.Events(OpsAccountID, "shop", 10)
- want := dbForwardAuditDetail("uk", body.SQL, time.Time{})
+ want := dbForwardAuditDetail(dbreads.OpQuery, "uk", body.SQL, time.Time{})
found := false
for _, e := range evs {
found = found || (e.Action == "db-read" && e.Target == dbfHost && e.Detail == want)
@@ -649,6 +652,57 @@ func openAnswer(t *testing.T, key *ecdh.PrivateKey, lines []dbreads.Line) ([]byt
return rows.Bytes(), s
}
+// A pull through the console: the route admits it, relays the box's ltx
+// stream and its summary's file list, and audits it with DBF-19; with --at,
+// DBF-20.
+func TestDBReadPullStreamsTheFiles(t *testing.T) {
+ fx := newDBFFixture(t, true)
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ fx.startBox(t, dir)
+ key, _ := dbreads.NewReplyKey()
+ res := fx.openRead(t, dbfHost, fx.member, dbreads.ReadBody{Op: dbreads.OpPull, ReplyKey: dbreads.EncodeReplyKey(key.PublicKey())})
+ lines := readLines(t, res.Body)
+ res.Body.Close()
+ last := lines[len(lines)-1].Result
+ if last == nil || last.State != dbreads.StateDone {
+ t.Fatalf("lines %+v", lines)
+ }
+ s, err := dbreads.OpenSummary(key, lines[0].Admitted, last.Sealed)
+ if err != nil || len(s.Files) == 0 || s.TXID != s.Files[len(s.Files)-1].MaxTXID {
+ t.Fatalf("summary %+v %v", s, err)
+ }
+ h := sha256.New()
+ for _, l := range lines {
+ if l.Chunk == 0 {
+ continue
+ }
+ plain, err := dbreads.OpenChunk(key, lines[0].Admitted, l.Chunk, dbreads.StreamLTX, l.Chunk == last.LastSeq, l.Data)
+ if err != nil {
+ t.Fatalf("chunk %d: %v", l.Chunk, err)
+ }
+ h.Write(plain)
+ }
+ if hex.EncodeToString(h.Sum(nil)) != s.SHA256 {
+ t.Fatal("the pulled bytes do not add up to the summary")
+ }
+ at := time.Now().Add(-time.Second).UTC().Truncate(time.Second)
+ res = fx.openRead(t, dbfHost, fx.member, dbreads.ReadBody{Op: dbreads.OpPull, At: at.Format(time.RFC3339),
+ ReplyKey: dbreads.EncodeReplyKey(key.PublicKey())})
+ readLines(t, res.Body)
+ res.Body.Close()
+ evs, _ := fx.st.Events(OpsAccountID, "shop", 10)
+ for _, want := range []string{fmt.Sprintf(dbfAuditPull, "uk"), fmt.Sprintf(dbfAuditPullAt, "uk", at.Format(time.RFC3339))} {
+ found := false
+ for _, e := range evs {
+ found = found || (e.Action == "db-read" && e.Target == dbfHost && e.Detail == want)
+ }
+ if !found {
+ t.Errorf("no %q audit row in %+v", want, evs)
+ }
+ }
+}
+
func TestDBReadStreamsAQueryEndToEnd(t *testing.T) {
fx := newDBFFixture(t, true)
dir := t.TempDir()
diff --git a/internal/console/dbforward_minio_test.go b/internal/console/dbforward_minio_test.go
index ce20a55dd..dc8d20633 100644
--- a/internal/console/dbforward_minio_test.go
+++ b/internal/console/dbforward_minio_test.go
@@ -3,6 +3,8 @@ package console
import (
"bufio"
"context"
+ "crypto/sha256"
+ "encoding/hex"
"encoding/json"
"net/http"
"os"
@@ -204,6 +206,37 @@ func TestDBForwardMinio(t *testing.T) {
}
}
+ // A pull through MinIO (plan 3): the planned files, sealed as the ltx
+ // stream, adding up to the summary, whose file list ends at its copy.
+ pkey, _ := dbreads.NewReplyKey()
+ pres := dfx.openRead(t, shop, member, dbreads.ReadBody{Op: dbreads.OpPull, ReplyKey: dbreads.EncodeReplyKey(pkey.PublicKey())})
+ plines := readLines(t, pres.Body)
+ pres.Body.Close()
+ presult := plines[len(plines)-1].Result
+ if presult == nil || presult.State != dbreads.StateDone {
+ t.Fatalf("pull: %+v", plines[len(plines)-1])
+ }
+ psum, err := dbreads.OpenSummary(pkey, plines[0].Admitted, presult.Sealed)
+ if err != nil || len(psum.Files) == 0 || psum.TXID != psum.Files[len(psum.Files)-1].MaxTXID {
+ t.Fatalf("pull summary %+v %v", psum, err)
+ }
+ ph := sha256.New()
+ var pn int64
+ for _, l := range plines {
+ if l.Chunk == 0 {
+ continue
+ }
+ plain, err := dbreads.OpenChunk(pkey, plines[0].Admitted, l.Chunk, dbreads.StreamLTX, l.Chunk == presult.LastSeq, l.Data)
+ if err != nil {
+ t.Fatalf("pull chunk %d: %v", l.Chunk, err)
+ }
+ ph.Write(plain)
+ pn += int64(len(plain))
+ }
+ if hex.EncodeToString(ph.Sum(nil)) != psum.SHA256 || pn != psum.Bytes {
+ t.Fatal("the pulled bytes do not add up to the summary")
+ }
+
// T16 (plan review finding 19): the real box reaches publication and is
// held there; the console's request ends, by disconnect and then by its
// bound; only then does the box publish. Its late chunks and result land
diff --git a/internal/dbdoor/plan_from.go b/internal/dbdoor/plan_from.go
index 23b378e61..a0764c837 100644
--- a/internal/dbdoor/plan_from.go
+++ b/internal/dbdoor/plan_from.go
@@ -144,7 +144,10 @@ func PlanFrom(ctx context.Context, rd ReplicaReader, at time.Time) (RestorePlan,
// pull), exactly to target, at out, and proves it with lease.CheckDBContext.
// On failure nothing is left at out.
func RestoreLocal(ctx context.Context, c litestream.ReplicaClient, out string, target uint64) error {
- if err := restoreOnce(ctx, c, out, target, time.Time{}); err != nil {
+ // litestream's compaction checks no context, so every read of the local
+ // files checks it here, as the door path's downloaded.Read does (Astra
+ // round 1 on plan 3, finding 2).
+ if err := restoreOnce(ctx, ctxReplica{ReplicaClient: c, ctx: ctx}, out, target, time.Time{}); err != nil {
removeCopy(out)
if cerr := ctx.Err(); cerr != nil {
return cerr
@@ -164,3 +167,33 @@ func RestoreLocal(ctx context.Context, c litestream.ReplicaClient, out string, t
}
return nil
}
+
+// ctxReplica is a local replica client whose file reads stop once ctx is
+// done.
+type ctxReplica struct {
+ litestream.ReplicaClient
+ ctx context.Context
+}
+
+func (c ctxReplica) OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error) {
+ if err := c.ctx.Err(); err != nil {
+ return nil, err
+ }
+ rc, err := c.ReplicaClient.OpenLTXFile(ctx, level, minTXID, maxTXID, offset, size)
+ if err != nil {
+ return nil, err
+ }
+ return ctxReadCloser{ReadCloser: rc, ctx: c.ctx}, nil
+}
+
+type ctxReadCloser struct {
+ io.ReadCloser
+ ctx context.Context
+}
+
+func (r ctxReadCloser) Read(p []byte) (int, error) {
+ if err := r.ctx.Err(); err != nil {
+ return 0, err
+ }
+ return r.ReadCloser.Read(p)
+}
diff --git a/internal/dbdoor/plan_from_test.go b/internal/dbdoor/plan_from_test.go
index e0503147c..95b907511 100644
--- a/internal/dbdoor/plan_from_test.go
+++ b/internal/dbdoor/plan_from_test.go
@@ -6,6 +6,7 @@ import (
"io"
"os"
"path/filepath"
+ "sync"
"testing"
"time"
@@ -152,3 +153,68 @@ func TestRestoreLocalRestoresThePlan(t *testing.T) {
t.Fatalf("restored %d bytes, planned %d", st.Size(), plan.RestoredBytes)
}
}
+
+// countingReads is a litestream.ReplicaClient over a file replica that
+// cancels a context on the first byte read and counts every read made after
+// it, so a test can see whether a restore keeps reading once cancelled.
+type countingReads struct {
+ litestream.ReplicaClient
+ cancel context.CancelFunc
+ mu sync.Mutex
+ reads int
+ after int
+}
+
+func (c *countingReads) OpenLTXFile(ctx context.Context, level int, lo, hi ltx.TXID, off, size int64) (io.ReadCloser, error) {
+ rc, err := c.ReplicaClient.OpenLTXFile(ctx, level, lo, hi, off, size)
+ if err != nil {
+ return nil, err
+ }
+ return &countedReader{ReadCloser: rc, c: c}, nil
+}
+
+type countedReader struct {
+ io.ReadCloser
+ c *countingReads
+}
+
+func (r *countedReader) Read(p []byte) (int, error) {
+ r.c.mu.Lock()
+ r.c.reads++
+ if r.c.reads == 1 {
+ r.c.cancel()
+ } else {
+ r.c.after++
+ }
+ r.c.mu.Unlock()
+ return r.ReadCloser.Read(p)
+}
+
+// Astra round 1 on plan 3, finding 2: litestream's local compaction checks
+// no context, so RestoreLocal's replica reads must: once the restore is
+// cancelled (Ctrl-C, or the read's deadline), no further byte is read and
+// nothing is left at out. Mutation: RestoreLocal reads the client without
+// its context wrapper (red: reads continue after the cancel).
+func TestRestoreLocalStopsReadingOnceCancelled(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 4)
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ c := &countingReads{ReplicaClient: lsfile.NewReplicaClient(dir), cancel: cancel}
+ plan, err := PlanFrom(context.Background(), lsfile.NewReplicaClient(dir), time.Time{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ db := filepath.Join(t.TempDir(), "copy.db")
+ err = RestoreLocal(ctx, c, db, plan.TXID)
+ if !errors.Is(err, context.Canceled) {
+ t.Fatalf("a cancelled restore answered %v", err)
+ }
+ if c.after != 0 {
+ t.Fatalf("%d replica reads after the cancel", c.after)
+ }
+ if _, err := os.Stat(db); !os.IsNotExist(err) {
+ t.Fatalf("something is at out: %v", err)
+ }
+}
diff --git a/internal/dbreads/agent.go b/internal/dbreads/agent.go
index 41e6ddb79..78ac62fee 100644
--- a/internal/dbreads/agent.go
+++ b/internal/dbreads/agent.go
@@ -43,9 +43,13 @@ type AgentConfig struct {
// Free reads the filesystem holding /var/tmp.
Free func() (avail, total uint64, err error)
MaxRestored int64 // DefaultMaxRestored when zero
- Now func() time.Time
- Jitter func() time.Duration // 0 to MaxJitter
- Logf func(format string, args ...any)
+ // MaxPull is a pull's cap on its planned LTX files: MaxPullBytes when
+ // zero or larger, so no box ships more than a CLI accepts, and never
+ // more than the worker holds (PlanBytesFactor times MaxRestored).
+ MaxPull int64
+ Now func() time.Time
+ Jitter func() time.Duration // 0 to MaxJitter
+ Logf func(format string, args ...any)
// Draining is this box's own published draining flag (its edges file's
// draining field); a draining box takes no new read. Never draining
// when nil.
@@ -80,6 +84,14 @@ func NewAgent(cfg AgentConfig) *Agent {
if cfg.MaxRestored <= 0 {
cfg.MaxRestored = DefaultMaxRestored
}
+ if cfg.MaxPull <= 0 || cfg.MaxPull > MaxPullBytes {
+ cfg.MaxPull = MaxPullBytes
+ }
+ // The worker refuses a plan whose files total more than this (its
+ // planWithinBounds), so a pull over it ends too-large here instead.
+ if held := PlanBytesFactor * cfg.MaxRestored; cfg.MaxPull > held {
+ cfg.MaxPull = held
+ }
if cfg.Now == nil {
cfg.Now = time.Now
}
diff --git a/internal/dbreads/chunks.go b/internal/dbreads/chunks.go
index d160a1557..f5df0cb94 100644
--- a/internal/dbreads/chunks.go
+++ b/internal/dbreads/chunks.go
@@ -45,7 +45,7 @@ func putCreateOnly(ctx context.Context, s release.Store, key string, body []byte
}
// chunkWriter seals a read's output into out/<seq> objects: one every
-// ChunkBytes, and whatever is pending every `every`; close seals what is
+// size bytes (ChunkBytes for a query, PullChunkBytes for a pull), and whatever is pending every `every`; close seals what is
// left, possibly nothing, as the last chunk. Past max nothing more is kept
// and Write answers errCapped. Each put is create-only and confirmed before
// the next is sealed, under the lock, so seqs are in order (the run
@@ -57,6 +57,7 @@ type chunkWriter struct {
pub *ecdh.PublicKey
stream string
max int64
+ size int
mu sync.Mutex
pending []byte
@@ -70,8 +71,8 @@ type chunkWriter struct {
done chan struct{}
}
-func newChunkWriter(ctx context.Context, s release.Store, id string, pub *ecdh.PublicKey, stream string, every time.Duration, max int64) *chunkWriter {
- c := &chunkWriter{ctx: ctx, store: s, id: id, pub: pub, stream: stream, max: max, sum: sha256.New(),
+func newChunkWriter(ctx context.Context, s release.Store, id string, pub *ecdh.PublicKey, stream string, every time.Duration, max int64, size int) *chunkWriter {
+ c := &chunkWriter{ctx: ctx, store: s, id: id, pub: pub, stream: stream, max: max, size: size, sum: sha256.New(),
stop: make(chan struct{}), done: make(chan struct{})}
go c.tick(every)
return c
@@ -112,12 +113,12 @@ func (c *chunkWriter) Write(p []byte) (int, error) {
c.sum.Write(p)
c.total += int64(len(p))
c.pending = append(c.pending, p...)
- for len(c.pending) >= ChunkBytes {
- if err := c.flushLocked(c.pending[:ChunkBytes], false); err != nil {
+ for len(c.pending) >= c.size {
+ if err := c.flushLocked(c.pending[:c.size], false); err != nil {
c.err = err
return 0, err
}
- c.pending = append([]byte(nil), c.pending[ChunkBytes:]...)
+ c.pending = append([]byte(nil), c.pending[c.size:]...)
}
if capped {
c.isCapped = true
diff --git a/internal/dbreads/envelope.go b/internal/dbreads/envelope.go
index 3fc536284..f44a583a0 100644
--- a/internal/dbreads/envelope.go
+++ b/internal/dbreads/envelope.go
@@ -45,8 +45,8 @@ func (r Request) AtTime() (time.Time, error) {
return time.Parse(time.RFC3339, r.At)
}
-// ops are the operations this build serves. Plan 3 adds OpPull.
-var ops = map[string]bool{OpQuery: true}
+// ops are the operations this build serves.
+var ops = map[string]bool{OpQuery: true, OpPull: true}
func (r Request) valid() bool {
if r.Purpose != Purpose || r.V != 1 || !ValidID(r.ID) || !ValidRegion(r.Region) ||
@@ -54,8 +54,16 @@ func (r Request) valid() bool {
r.Incarnation == "" || !ops[r.Op] || r.By == "" {
return false
}
- if r.Op == OpQuery && (r.SQL == "" || len(r.SQL) > MaxSQLBytes || !utf8.ValidString(r.SQL)) {
- return false
+ switch r.Op {
+ case OpQuery:
+ if r.SQL == "" || len(r.SQL) > MaxSQLBytes || !utf8.ValidString(r.SQL) {
+ return false
+ }
+ case OpPull:
+ // A pull runs nothing: SQL in one is a request nobody built.
+ if r.SQL != "" {
+ return false
+ }
}
if _, err := r.AtTime(); err != nil {
return false
diff --git a/internal/dbreads/envelope_test.go b/internal/dbreads/envelope_test.go
index 379f1323d..aa26621dc 100644
--- a/internal/dbreads/envelope_test.go
+++ b/internal/dbreads/envelope_test.go
@@ -175,3 +175,22 @@ func TestQueueMatchesNeedsEveryPart(t *testing.T) {
}
}
}
+
+// A pull is an op this build signs and serves (plan 3), and carries no SQL.
+func TestEnvelopeCarriesAPull(t *testing.T) {
+ t.Parallel()
+ pub, priv := rootKeys(t)
+ req := testRequest(t)
+ req.Op, req.SQL = OpPull, ""
+ wire, err := Sign(release.RootSigner(priv), req)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got, err := Verify(pub, wire, req.ID, "uk", testNow); err != nil || got.Op != OpPull {
+ t.Fatalf("%+v %v", got, err)
+ }
+ req.SQL = "SELECT 1"
+ if _, err := Sign(release.RootSigner(priv), req); !errors.Is(err, ErrEnvelope) {
+ t.Fatalf("a pull carrying SQL signed: %v", err)
+ }
+}
diff --git a/internal/dbreads/fixture_test.go b/internal/dbreads/fixture_test.go
index bcf0e7582..c4a481a30 100644
--- a/internal/dbreads/fixture_test.go
+++ b/internal/dbreads/fixture_test.go
@@ -46,6 +46,30 @@ type inProcWorkers struct {
// failStart makes the worker end at once without a word, as a unit
// whose service fails to start after the socket accepted does.
failStart atomic.Bool
+ // sent is the type of every frame the agent wrote, in order.
+ sent []byte
+}
+
+// frameTap records the first byte of each Write the agent makes: one frame
+// type per frame (writeFrame writes a frame in one Write).
+type frameTap struct {
+ net.Conn
+ w *inProcWorkers
+}
+
+func (f frameTap) Write(p []byte) (int, error) {
+ if len(p) > 0 {
+ f.w.mu.Lock()
+ f.w.sent = append(f.w.sent, p[0])
+ f.w.mu.Unlock()
+ }
+ return f.Conn.Write(p)
+}
+
+func (w *inProcWorkers) sentFrames() []byte {
+ w.mu.Lock()
+ defer w.mu.Unlock()
+ return append([]byte(nil), w.sent...)
}
func (w *inProcWorkers) dial(context.Context) (net.Conn, error) {
@@ -64,7 +88,7 @@ func (w *inProcWorkers) dial(context.Context) (net.Conn, error) {
time.Sleep(w.linger)
w.live.Add(-1)
}()
- return agentEnd, nil
+ return frameTap{Conn: agentEnd, w: w}, nil
}
opt := w.opt
if opt.Scratch == "" {
@@ -82,7 +106,7 @@ func (w *inProcWorkers) dial(context.Context) (net.Conn, error) {
time.Sleep(w.linger)
w.live.Add(-1)
}()
- return agentEnd, nil
+ return frameTap{Conn: agentEnd, w: w}, nil
}
func (w *inProcWorkers) units(context.Context) (int, error) { return int(w.live.Load()), nil }
diff --git a/internal/dbreads/limits.go b/internal/dbreads/limits.go
index 09a085f6a..613224574 100644
--- a/internal/dbreads/limits.go
+++ b/internal/dbreads/limits.go
@@ -51,6 +51,13 @@ const (
MaxOutputBytes = runs.MaxOutputBytes
ChunkBytes = runs.ChunkBytes
ChunkEvery = runs.ChunkEvery
+ // MaxPullBytes is a pull's cap on its planned LTX files (spec §5.7, rev
+ // 7: measured on dev, 2026-10-23). CARLOS_DB_PULL_MAX_BYTES on a box can
+ // only lower it, so no box ships more than a CLI accepts.
+ MaxPullBytes int64 = 2 << 30
+ // PullChunkBytes is a pull's chunk: larger than a query's, so a full
+ // pull is about two thousand objects, not eight.
+ PullChunkBytes = 1 << 20
ReadsPerBox = 2
MaxConnections = 3 // two reads and one probe (carlos-dbread.socket)
diff --git a/internal/dbreads/lines.go b/internal/dbreads/lines.go
index 20e1e4112..2b5a81c96 100644
--- a/internal/dbreads/lines.go
+++ b/internal/dbreads/lines.go
@@ -2,8 +2,9 @@ package dbreads
// The reads route's 200 body is newline-delimited JSON, one Line per line
// (spec §5.5 step 6). MaxLineBytes bounds one line: a ChunkBytes chunk,
-// sealed and base64-encoded, is about 350 KiB.
-const MaxLineBytes = 1 << 20
+// sealed and base64-encoded, is about 350 KiB, and a pull's PullChunkBytes
+// chunk about 1.4 MB.
+const MaxLineBytes = 2 << 20
// Wait lines name what the console is waiting on; ended lines say why the
// response ends without a result.
diff --git a/internal/dbreads/lines_test.go b/internal/dbreads/lines_test.go
index f693558d5..bb1265cba 100644
--- a/internal/dbreads/lines_test.go
+++ b/internal/dbreads/lines_test.go
@@ -39,3 +39,13 @@ func TestLinesAreOneOutcomeEach(t *testing.T) {
t.Fatalf("a full chunk line is %d bytes, over MaxLineBytes", len(full))
}
}
+
+// A full pull chunk, sealed and base64-encoded in its line, fits
+// MaxLineBytes (plan 3: a 1 MiB chunk's line is about 1.4 MB).
+func TestAFullPullChunkLineFits(t *testing.T) {
+ t.Parallel()
+ full, _ := json.Marshal(Line{Chunk: 1, Data: bytes.Repeat([]byte{0xff}, PullChunkBytes+SealOverhead)})
+ if len(full)+1 > MaxLineBytes {
+ t.Fatalf("a full pull chunk line is %d bytes", len(full))
+ }
+}
diff --git a/internal/dbreads/pull_test.go b/internal/dbreads/pull_test.go
new file mode 100644
index 000000000..421d05d1a
--- /dev/null
+++ b/internal/dbreads/pull_test.go
@@ -0,0 +1,325 @@
+package dbreads
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "io"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ lsfile "github.com/benbjohnson/litestream/file"
+ "github.com/superfly/ltx"
+
+ "github.com/carlosframework/platform/internal/dbdoor"
+ "github.com/carlosframework/platform/internal/dbdoor/dbdoortest"
+)
+
+// sendPullPlan sends a pull header and the planned files; it returns the
+// files' bytes concatenated, which is what the worker must send back.
+// net.Pipe to the real worker code: not acceptance (Task 5's root-runner
+// test is).
+func sendPullPlan(t *testing.T, w io.Writer, dir string, corrupt bool) (dbdoor.RestorePlan, []byte) {
+ t.Helper()
+ rd := lsfile.NewReplicaClient(dir)
+ plan, err := dbdoor.PlanFrom(context.Background(), rd, time.Time{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := writeJSONFrame(w, frameHeader, planHeader{V: ProtocolVersion, Op: OpPull, Files: plan.Planned, Target: plan.TXID, MaxRestored: DefaultMaxRestored}); err != nil {
+ t.Fatal(err)
+ }
+ var all bytes.Buffer
+ for i, f := range plan.Planned {
+ rc, err := rd.OpenLTXFile(context.Background(), f.Level, ltx.TXID(f.MinTXID), ltx.TXID(f.MaxTXID), 0, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ b, _ := io.ReadAll(rc)
+ rc.Close()
+ if corrupt && i == 0 {
+ b[len(b)/2] ^= 0xff
+ }
+ all.Write(b)
+ if err := writeFrame(w, frameFile, b); err != nil {
+ t.Fatal(err)
+ }
+ }
+ return plan, all.Bytes()
+}
+
+func TestWorkerHoldsThenSendsThePlannedFiles(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 3)
+ conn, done := workerPipe(t, WorkerOptions{})
+ plan, want := sendPullPlan(t, conn, dir, false)
+ var ready readyMsg
+ readTyped(t, conn, frameReady, &ready)
+ if ready.TXID != plan.TXID {
+ t.Fatalf("ready at %d", ready.TXID)
+ }
+ if err := writeJSONFrame(conn, frameRun, runMsg{}); err != nil {
+ t.Fatal(err)
+ }
+ var got bytes.Buffer
+ for {
+ typ, p, err := readFrame(conn)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if typ == frameOutput {
+ got.Write(p)
+ continue
+ }
+ var end endMsg
+ if typ != frameEnd || json.Unmarshal(p, &end) != nil || end.Code != "" {
+ t.Fatalf("end %q %s", typ, p)
+ }
+ break
+ }
+ if !bytes.Equal(got.Bytes(), want) {
+ t.Fatalf("sent back %d bytes, held %d", got.Len(), len(want))
+ }
+ if err := <-done; err != nil {
+ t.Fatal(err)
+ }
+}
+
+// Review Focus 2: a corrupt file is refused before ready, never shipped.
+// Mutation: holdAndSend skips verifyLTX (red: ready, not check-failed).
+func TestWorkerHoldRefusesACorruptFile(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 2)
+ conn, done := workerPipe(t, WorkerOptions{})
+ sendPullPlan(t, conn, dir, true)
+ var end endMsg
+ readTyped(t, conn, frameEnd, &end)
+ if end.Code != CodeCheckFailed {
+ t.Fatalf("end = %+v", end)
+ }
+ <-done
+}
+
+func TestWorkerHoldStopsWithoutSending(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 1)
+ conn, done := workerPipe(t, WorkerOptions{})
+ sendPullPlan(t, conn, dir, false)
+ readTyped(t, conn, frameReady, nil)
+ if err := writeFrame(conn, frameStop, nil); err != nil {
+ t.Fatal(err)
+ }
+ if err := <-done; err != nil {
+ t.Fatal(err)
+ }
+}
+
+func (fx *fixture) pullRequest(at string) Request {
+ req := fx.request("", at)
+ req.Op = OpPull
+ return req
+}
+
+// answerPull opens a pull's chunks as the CLI does: the ltx stream.
+func (fx *fixture) answerPull(id string) ([]byte, Summary, Result) {
+ fx.t.Helper()
+ raw, _, err := fx.store.Get(context.Background(), ResultKey(id))
+ if err != nil {
+ fx.t.Fatal(err)
+ }
+ var r Result
+ if err := json.Unmarshal(raw, &r); err != nil {
+ fx.t.Fatal(err)
+ }
+ s, err := OpenSummary(fx.reply, id, r.Sealed)
+ if err != nil {
+ fx.t.Fatal(err)
+ }
+ var out bytes.Buffer
+ for seq := int64(1); seq <= r.LastSeq; seq++ {
+ sealed, _, _ := fx.store.Get(context.Background(), ChunkKey(id, seq))
+ plain, err := OpenChunk(fx.reply, id, seq, StreamLTX, seq == r.LastSeq, sealed)
+ if err != nil {
+ fx.t.Fatalf("chunk %d: %v", seq, err)
+ }
+ out.Write(plain)
+ }
+ return out.Bytes(), s, r
+}
+
+// plannedBytes is dir's planned files, concatenated in plan order.
+func plannedBytes(t *testing.T, dir string) (dbdoor.RestorePlan, []byte) {
+ t.Helper()
+ rd := lsfile.NewReplicaClient(dir)
+ plan, err := dbdoor.PlanFrom(context.Background(), rd, time.Time{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ var want bytes.Buffer
+ for _, f := range plan.Planned {
+ rc, err := rd.OpenLTXFile(context.Background(), f.Level, ltx.TXID(f.MinTXID), ltx.TXID(f.MaxTXID), 0, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ io.Copy(&want, rc)
+ rc.Close()
+ }
+ return plan, want.Bytes()
+}
+
+// The agent's pull ships the planned files whole, in plan order, as the
+// ltx stream in chunks of at most PullChunkBytes, with the file list in the
+// summary, and logs plan 1's two timing lines (plan review finding 26).
+// Mutation: drop the timing lines (red).
+func TestPullShipsThePlannedFiles(t *testing.T) {
+ t.Parallel()
+ fx := newFixture(t, 4)
+ var logMu sync.Mutex
+ var logs []string
+ a, _ := fx.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) {
+ c.Logf = func(format string, args ...any) {
+ logMu.Lock()
+ defer logMu.Unlock()
+ logs = append(logs, fmt.Sprintf(format, args...))
+ }
+ })
+ req := fx.pullRequest("")
+ fx.submit(req)
+ a.Pass(context.Background())
+ a.Wait()
+ got, s, r := fx.answerPull(req.ID)
+ plan, want := plannedBytes(t, fx.replica)
+ if r.State != StateDone || !bytes.Equal(got, want) || s.TXID != plan.TXID || len(s.Files) != len(plan.Planned) || s.Bytes != plan.LTXBytes {
+ t.Fatalf("result %+v summary %+v: %d bytes, want %d", r, s, len(got), len(want))
+ }
+ for i, f := range s.Files {
+ if f.Level != plan.Planned[i].Level || f.MinTXID != plan.Planned[i].MinTXID || f.MaxTXID != plan.Planned[i].MaxTXID || f.Size != plan.Planned[i].Size {
+ t.Fatalf("file %d is %+v, planned %+v", i, f, plan.Planned[i])
+ }
+ }
+ for seq := int64(1); seq < r.LastSeq; seq++ {
+ sealed, _, _ := fx.store.Get(context.Background(), ChunkKey(req.ID, seq))
+ if len(sealed)-SealOverhead > PullChunkBytes {
+ t.Fatalf("chunk %d is %d bytes", seq, len(sealed))
+ }
+ }
+ // Task 1's measurement reads these two lines (plan review finding 26).
+ logMu.Lock()
+ all := strings.Join(logs, "\n")
+ logMu.Unlock()
+ if !strings.Contains(all, "and held through") || !strings.Contains(all, "confirmed chunks in") {
+ t.Fatalf("the pull's timing lines are missing:\n%s", all)
+ }
+}
+
+// A pull over the agent's cap ends too-large with the planned bytes and the
+// cap, and a record change after the last byte left the bucket (T9 for a
+// pull) writes no chunk and never sends the worker run. Mutation: the
+// agent's pull sends run before check B (red: the recorded frames hold a
+// run).
+func TestPullTooLargeAndNoChunkBeforeCheckB(t *testing.T) {
+ t.Parallel()
+ fx := newFixture(t, 2)
+ a, _ := fx.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) { c.MaxPull = 100 })
+ req := fx.pullRequest("")
+ fx.submit(req)
+ a.Pass(context.Background())
+ a.Wait()
+ _, s, r := fx.answerPull(req.ID)
+ if r.Code != CodeTooLarge || s.Cap != 100 || s.Restored <= 100 {
+ t.Fatalf("%+v %+v", r, s)
+ }
+ fx2 := newFixture(t, 2)
+ plan, _ := plannedBytes(t, fx2.replica)
+ last := plan.Planned[len(plan.Planned)-1]
+ hook := &stageHook{ReplicaReader: lsfile.NewReplicaClient(fx2.replica), stage: "after the last byte",
+ lastSpan: dbdoor.Span(last.MinTXID, last.MaxTXID), change: func() { r := fx2.record(); r.Region = "ie"; fx2.putRecord(r) }}
+ b, w := fx2.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) {
+ c.OpenReplica = func(context.Context, string) (dbdoor.ReplicaReader, error) { return hook, nil }
+ })
+ req2 := fx2.pullRequest("")
+ fx2.submit(req2)
+ b.Pass(context.Background())
+ b.Wait()
+ if _, _, r := fx2.answerPull(req2.ID); r.Code != CodeRecordChanged || len(fx2.chunkKeys(req2.ID)) != 0 {
+ t.Fatalf("%+v, chunks %v", r, fx2.chunkKeys(req2.ID))
+ }
+ // The worker never heard run: check B failed, so the agent said stop
+ // (plan review finding 24). Without this, an agent that sent run first
+ // and refused the files afterwards would pass the two checks above.
+ sent := w.sentFrames()
+ if !bytes.Contains(sent, []byte{frameStop}) {
+ t.Fatalf("the agent never said stop: %q", sent)
+ }
+ if bytes.Contains(sent, []byte{frameRun}) {
+ t.Fatalf("the agent sent run before check B passed: %q", sent)
+ }
+}
+
+// Plan review finding 24: NewAgent itself clamps the pull's cap, whatever a
+// box's settings hand it. Mutation: NewAgent keeps a MaxPull above
+// MaxPullBytes (red).
+func TestNewAgentClampsMaxPull(t *testing.T) {
+ t.Parallel()
+ fx := newFixture(t, 1)
+ for give, want := range map[int64]int64{0: MaxPullBytes, -1: MaxPullBytes, 1 << 20: 1 << 20, MaxPullBytes + 1: MaxPullBytes} {
+ a, _ := fx.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) { c.MaxPull = give })
+ if a.cfg.MaxPull != want {
+ t.Errorf("MaxPull %d became %d, want %d", give, a.cfg.MaxPull, want)
+ }
+ }
+ // Never more than the worker holds: PlanBytesFactor times MaxRestored.
+ a, _ := fx.agent("uk-1", func(c *AgentConfig, _ *inProcWorkers) { c.MaxRestored = 256 << 20 })
+ if a.cfg.MaxPull != PlanBytesFactor*(256<<20) {
+ t.Errorf("MaxPull %d with a 256 MiB restored cap", a.cfg.MaxPull)
+ }
+}
+
+// relabelled is the bytes of dir's first planned file, a valid snapshot at
+// transaction 1, under the coordinates of a file that ends at 2.
+func relabelled(t *testing.T, dir string) (dbdoor.LTXFile, []byte) {
+ t.Helper()
+ rd := lsfile.NewReplicaClient(dir)
+ rc, err := rd.OpenLTXFile(context.Background(), 0, 1, 1, 0, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ b, _ := io.ReadAll(rc)
+ rc.Close()
+ return dbdoor.LTXFile{Level: 0, MinTXID: 1, MaxTXID: 2, Size: int64(len(b))}, b
+}
+
+// Astra round 1 on plan 3, finding 1: a valid snapshot under another file's
+// coordinates is refused before ready, for a pull and for a query alike: the
+// worker checks each file's LTX header against its plan entry, so a copy
+// can never be reported at a transaction it does not hold. Mutation: drop
+// the header comparison (red: the query restores and answers ready at 2;
+// the pull verifies and answers ready).
+func TestWorkerRefusesARelabelledSnapshot(t *testing.T) {
+ t.Parallel()
+ dir := t.TempDir()
+ dbdoortest.Replicate(t, dir, 2)
+ for _, op := range []string{OpQuery, OpPull} {
+ lf, b := relabelled(t, dir)
+ conn, done := workerPipe(t, WorkerOptions{})
+ if err := writeJSONFrame(conn, frameHeader, planHeader{V: ProtocolVersion, Op: op, Files: []dbdoor.LTXFile{lf}, Target: 2, MaxRestored: DefaultMaxRestored}); err != nil {
+ t.Fatal(err)
+ }
+ if err := writeFrame(conn, frameFile, b); err != nil {
+ t.Fatal(err)
+ }
+ typ, p, err := readFrame(conn)
+ var end endMsg
+ if err != nil || typ != frameEnd || json.Unmarshal(p, &end) != nil || end.Code != CodeCheckFailed {
+ t.Errorf("%s: a relabelled snapshot answered %q %s %v", op, typ, p, err)
+ writeFrame(conn, frameStop, nil)
+ }
+ <-done
+ }
+}
diff --git a/internal/dbreads/read.go b/internal/dbreads/read.go
index 8ba085390..b3aafa8ce 100644
--- a/internal/dbreads/read.go
+++ b/internal/dbreads/read.go
@@ -183,10 +183,21 @@ func (a *Agent) attempt(ctx context.Context, req Request, pub *ecdh.PublicKey, a
a.logf("dbreads: %s: plan: %v", req.ID, err)
return fail(CodeInterrupted, Summary{})
}
- if plan.RestoredBytes > a.cfg.MaxRestored {
+ // A pull's cap is on the files it ships, a query's on the copy it
+ // restores (spec §5.7). Both endings are the replica's word, so both wait
+ // for check B (Astra round 1, finding 2).
+ pull := req.Op == OpPull
+ switch {
+ case pull && plan.LTXBytes > a.cfg.MaxPull:
+ return failB(CodeTooLarge, Summary{Restored: plan.LTXBytes, Cap: a.cfg.MaxPull})
+ case !pull && plan.RestoredBytes > a.cfg.MaxRestored:
return failB(CodeTooLarge, Summary{Restored: plan.RestoredBytes, Cap: a.cfg.MaxRestored})
}
- if err := a.adm.grow(req.ID, plan.LTXBytes+plan.RestoredBytes); err != nil {
+ need := plan.LTXBytes + plan.RestoredBytes
+ if pull {
+ need = plan.LTXBytes // the worker holds the files and restores nothing
+ }
+ if err := a.adm.grow(req.ID, need); err != nil {
return failB(CodeNoRoom, Summary{})
}
conn, err := a.dial(pctx)
@@ -242,20 +253,37 @@ func (a *Agent) attempt(ctx context.Context, req Request, pub *ecdh.PublicKey, a
}
return refuse(code)
}
- stopPlanBound() // the query is bounded by the worker and the owner's deadline
- // Phase timings, for operators and for plan 3's throughput measurement:
- // planning and the worker's restore, then the output's sealing and
- // confirmed creates.
- a.logf("dbreads: %s: planned %d files (%d bytes) and restored to %d in %s", req.ID, plan.Files, plan.LTXBytes, plan.TXID, time.Since(attemptStart).Round(time.Millisecond))
+ stopPlanBound() // the run is bounded by the worker and the owner's deadline
+ stream, size, max := StreamRows, ChunkBytes, a.cfg.MaxOutput
+ if pull {
+ stream, size, max = StreamLTX, PullChunkBytes, plan.LTXBytes
+ }
+ // Phase timings, for operators and for plan 3's throughput measurement
+ // (plan review finding 26): a query's first phase plans and restores, a
+ // pull's plans and has the worker verify and hold the files; then the
+ // output's sealing and confirmed creates.
+ held := "restored to"
+ if pull {
+ held = "held through"
+ }
+ a.logf("dbreads: %s: planned %d files (%d bytes) and %s %d in %s", req.ID, plan.Files, plan.LTXBytes, held, plan.TXID, time.Since(attemptStart).Round(time.Millisecond))
runStart := time.Now()
if err := writeJSONFrame(conn, frameRun, runMsg{SQL: req.SQL}); err != nil {
return fail(CodeInterrupted, Summary{})
}
- cw := newChunkWriter(ctx, a.cfg.Store, req.ID, pub, StreamRows, a.cfg.ChunkEvery, a.cfg.MaxOutput)
+ cw := newChunkWriter(ctx, a.cfg.Store, req.ID, pub, stream, a.cfg.ChunkEvery, max, size)
endCode, sqlErr, rerr := relay(conn, cw)
last, n, sum, werr := cw.close()
a.logf("dbreads: %s: output %d bytes in %d confirmed chunks in %s", req.ID, n, last, time.Since(runStart).Round(time.Millisecond))
s := Summary{TXID: plan.TXID, At: plan.At, Bytes: n, SHA256: sum, Error: sqlErr}
+ if pull {
+ s.Files = plan.Planned
+ // A worker that sends more or less than it held is a check
+ // failure, never an output cap.
+ if werr == nil && rerr == nil && endCode == "" && (cw.capped() || n != plan.LTXBytes) {
+ return a.ended(pub, req.ID, StateFailed, CodeCheckFailed, Summary{}, last), false
+ }
+ }
switch {
case werr != nil:
a.logf("dbreads: %s: write output: %v", req.ID, werr)
diff --git a/internal/dbreads/read_test.go b/internal/dbreads/read_test.go
index b754e860a..ad0bd5d50 100644
--- a/internal/dbreads/read_test.go
+++ b/internal/dbreads/read_test.go
@@ -265,7 +265,7 @@ func TestChunkWriterCapIsExact(t *testing.T) {
for _, extra := range []int{0, 1} {
fx := newFixture(t, 0)
max := int64(3*ChunkBytes + 7)
- cw := newChunkWriter(context.Background(), fx.store, "dbr_aaaaaaaaaaaaaaaaaaaaaaaaaa", fx.reply.PublicKey(), StreamRows, time.Hour, max)
+ cw := newChunkWriter(context.Background(), fx.store, "dbr_aaaaaaaaaaaaaaaaaaaaaaaaaa", fx.reply.PublicKey(), StreamRows, time.Hour, max, ChunkBytes)
big := bytes.Repeat([]byte("x"), 2*ChunkBytes+5)
rest := bytes.Repeat([]byte("y"), int(max)-len(big)+extra)
cw.Write(big)
diff --git a/internal/dbreads/worker.go b/internal/dbreads/worker.go
index f8b7d88cf..20b2cbf00 100644
--- a/internal/dbreads/worker.go
+++ b/internal/dbreads/worker.go
@@ -133,7 +133,7 @@ func ServeWorker(conn io.ReadWriter, opt WorkerOptions) error {
return errProtocol
}
var h planHeader
- if err := json.Unmarshal(f.p, &h); err != nil || h.V != ProtocolVersion || h.Op != OpQuery {
+ if err := json.Unmarshal(f.p, &h); err != nil || h.V != ProtocolVersion || (h.Op != OpQuery && h.Op != OpPull) {
return errProtocol
}
// Every byte this worker writes is bounded before the first is
@@ -157,6 +157,20 @@ func ServeWorker(conn io.ReadWriter, opt WorkerOptions) error {
}
return end(CodeCheckFailed, err)
}
+ // litestream's restore picks files by their names and compacts them
+ // by their headers, so a file whose header is not its plan entry
+ // would be reported at a transaction it does not hold (Astra round 1
+ // on plan 3, finding 1). A pull's files go back unrestored, so each is
+ // decoded whole as well.
+ if err := verifyLTX(path, lf, h.Op == OpPull); err != nil {
+ return end(CodeCheckFailed, err)
+ }
+ }
+ if len(h.Files) == 0 || h.Files[len(h.Files)-1].MaxTXID != h.Target {
+ return end(CodeCheckFailed, errors.New("dbreads: the plan does not end at its target"))
+ }
+ if h.Op == OpPull {
+ return holdAndSend(ctx, conn, next, end, root, h)
}
db := filepath.Join(dir, "copy.db")
if err := opt.Restore(ctx, lsfile.NewReplicaClient(root), db, h.Target); err != nil {
@@ -204,6 +218,77 @@ func ServeWorker(conn io.ReadWriter, opt WorkerOptions) error {
return end(CodeSQLError, qerr)
}
+// holdAndSend is a pull's whole worker job after the files are received and
+// checked (spec §6: the box never restores for a pull): the files are held
+// in this unit's private scratch, and after the agent's check B they go back
+// in the header's order, whole, as output.
+func holdAndSend(ctx context.Context, conn io.Writer, next func() (frame, error), end func(string, error) error, root string, h planHeader) error {
+ // ServeWorker has checked every file whole against its plan entry, and
+ // the plan's end against its target.
+ paths := make([]string, len(h.Files))
+ for i, lf := range h.Files {
+ paths[i] = litestream.LTXFilePath(root, lf.Level, ltx.TXID(lf.MinTXID), ltx.TXID(lf.MaxTXID))
+ }
+ if err := writeJSONFrame(conn, frameReady, readyMsg{TXID: h.Target}); err != nil {
+ return err
+ }
+ f, err := next()
+ if err != nil {
+ return err
+ }
+ switch f.typ {
+ case frameStop:
+ return nil
+ case frameRun:
+ default:
+ return errProtocol
+ }
+ out := &frameWriter{w: conn}
+ for _, p := range paths {
+ if ctx.Err() != nil {
+ return errAgentGone
+ }
+ in, err := os.Open(p)
+ if err != nil {
+ return end(CodeCheckFailed, err)
+ }
+ _, err = io.Copy(out, in)
+ in.Close()
+ if err != nil {
+ return err
+ }
+ }
+ if err := out.flush(); err != nil {
+ return err
+ }
+ return end("", nil)
+}
+
+// verifyLTX checks one received file against its plan entry: its LTX
+// header must name exactly the entry's transactions. With whole, every page
+// and the checksums are decoded too (the decode is the worker's, as every
+// decode of tenant bytes is).
+func verifyLTX(path string, lf dbdoor.LTXFile, whole bool) error {
+ f, err := os.Open(path)
+ if err != nil {
+ return err
+ }
+ defer f.Close()
+ dec := ltx.NewDecoder(f)
+ if whole {
+ err = dec.Verify()
+ } else {
+ err = dec.DecodeHeader()
+ }
+ if err != nil {
+ return err
+ }
+ if h := dec.Header(); uint64(h.MinTXID) != lf.MinTXID || uint64(h.MaxTXID) != lf.MaxTXID {
+ return fmt.Errorf("dbreads: %s holds transactions %s", filepath.Base(path), dbdoor.Span(uint64(h.MinTXID), uint64(h.MaxTXID)))
+ }
+ return nil
+}
+
// receiveFile writes exactly size bytes of file frames to path, refusing a
// file larger than its header said (spec §8.3).
func receiveFile(next func() (frame, error), path string, size int64) error {
diff --git a/internal/host/dbreads.go b/internal/host/dbreads.go
index a836646ff..777dc8a60 100644
--- a/internal/host/dbreads.go
+++ b/internal/host/dbreads.go
@@ -26,6 +26,8 @@ const (
// (carlos-dbread.socket says why).
dbReadSocketPath = "/run/carlos/dbread/worker.sock"
dbReadMaxBytesEnv = "CARLOS_DB_READ_MAX_BYTES"
+ // dbPullMaxBytesEnv can only lower a pull's cap (spec §5.7, rev 7).
+ dbPullMaxBytesEnv = "CARLOS_DB_PULL_MAX_BYTES"
dbReadsSwitchEnv = "CARLOS_DB_READS"
)
@@ -33,6 +35,7 @@ type dbReadsEnv struct {
region string
pub ed25519.PublicKey
maxRestored int64
+ maxPull int64
}
// dbReadsSettings is the environment half of spec §7.1's conditions:
@@ -57,7 +60,7 @@ func dbReadsSettings(getenv func(string) string) (dbReadsEnv, string) {
if err != nil {
return dbReadsEnv{}, err.Error()
}
- env := dbReadsEnv{region: region, pub: pub, maxRestored: dbreads.DefaultMaxRestored}
+ env := dbReadsEnv{region: region, pub: pub, maxRestored: dbreads.DefaultMaxRestored, maxPull: dbreads.MaxPullBytes}
if raw := strings.TrimSpace(getenv(dbReadMaxBytesEnv)); raw != "" {
n, err := strconv.ParseInt(raw, 10, 64)
if err != nil || n <= 0 {
@@ -65,6 +68,15 @@ func dbReadsSettings(getenv func(string) string) (dbReadsEnv, string) {
}
env.maxRestored = n
}
+ if raw := strings.TrimSpace(getenv(dbPullMaxBytesEnv)); raw != "" {
+ n, err := strconv.ParseInt(raw, 10, 64)
+ if err != nil || n <= 0 {
+ return dbReadsEnv{}, dbPullMaxBytesEnv + " is not a whole number of bytes"
+ }
+ // Lowering only: no box ships more than a CLI accepts (Review
+ // Focus 5). NewAgent clamps again.
+ env.maxPull = min(n, dbreads.MaxPullBytes)
+ }
return env, ""
}
@@ -143,7 +155,7 @@ func armDBReads(hib activator.Config) {
return lease.ReadReplicaClient(ctx, hib.Bucket, prefix, hib.Region)
},
DialWorker: dialDBReadWorker, Units: dbReadUnits, Free: varTmpFree,
- MaxRestored: env.maxRestored, Logf: log.Printf, Draining: dbReadsDraining.Load,
+ MaxRestored: env.maxRestored, MaxPull: env.maxPull, Logf: log.Printf, Draining: dbReadsDraining.Load,
})
armedDBReads.Store(a)
go a.Run(ctx, dbReadsNudge)
@@ -151,7 +163,7 @@ func armDBReads(hib activator.Config) {
}
// dbReadsHostEnvKeys are the host.env settings dbReadsSettings reads.
-var dbReadsHostEnvKeys = []string{"CARLOS_HIBERNATE_BUCKET", "CARLOS_REGION", "CARLOS_RELEASE_PUBKEY", dbReadsSwitchEnv, "CARLOS_HOST_CLAIM", dbReadMaxBytesEnv}
+var dbReadsHostEnvKeys = []string{"CARLOS_HIBERNATE_BUCKET", "CARLOS_REGION", "CARLOS_RELEASE_PUBKEY", dbReadsSwitchEnv, "CARLOS_HOST_CLAIM", dbReadMaxBytesEnv, dbPullMaxBytesEnv}
// dbReadsSocketWanted is root's half of the dark gate (Astra round 1,
// finding 1): the adopt pass arms carlos-dbread.socket only when the box's
diff --git a/internal/host/dbreads_rootrunner_linux_test.go b/internal/host/dbreads_rootrunner_linux_test.go
index 16d5b61d6..f6dd18cc5 100644
--- a/internal/host/dbreads_rootrunner_linux_test.go
+++ b/internal/host/dbreads_rootrunner_linux_test.go
@@ -9,8 +9,10 @@ import (
"crypto/ecdh"
"crypto/ed25519"
"crypto/rand"
+ "crypto/sha256"
"database/sql"
"encoding/base64"
+ "encoding/hex"
"encoding/json"
"fmt"
"io"
@@ -25,6 +27,7 @@ import (
"time"
lsfile "github.com/benbjohnson/litestream/file"
+ "github.com/superfly/ltx"
"github.com/carlosframework/platform/internal/dbdoor"
"github.com/carlosframework/platform/internal/dbdoor/dbdoortest"
@@ -342,6 +345,51 @@ func TestRootDBReadServesThroughTheInstalledUnits(t *testing.T) {
privateTmpGone(t, 10*time.Second)
}
+// A forwarded pull through the real socket and the real service (plan 3
+// Task 5): the installed worker verifies and holds the planned files and
+// sends them back after check B; the answer, opened as the ltx stream, is
+// the planned files byte for byte in plan order, and adds up to the sealed
+// summary. The worker's private tmp is gone afterwards.
+func TestRootDBReadPullThroughTheInstalledUnits(t *testing.T) {
+ r := newDBReadRunner(t, "40g")
+ id, reply := r.submit(dbreads.OpPull, "")
+ rep, _ := r.payload("serve", true, id)
+ if rep["capable"] != true {
+ t.Fatalf("the installed worker failed its probe: %v", rep)
+ }
+ got, res := r.answer(id, reply, dbreads.StreamLTX)
+ if res.State != dbreads.StateDone || res.Code != "" {
+ t.Fatalf("result %+v", res)
+ }
+ sum, err := dbreads.OpenSummary(reply, id, res.Sealed)
+ if err != nil {
+ t.Fatal(err)
+ }
+ h := sha256.Sum256(got)
+ if hex.EncodeToString(h[:]) != sum.SHA256 || int64(len(got)) != sum.Bytes {
+ t.Fatalf("%d bytes do not add up to the summary %+v", len(got), sum)
+ }
+ rd := lsfile.NewReplicaClient(filepath.Join(r.dir, "replica", "instances", dbreadRunnerHost))
+ plan, err := dbdoor.PlanFrom(context.Background(), rd, time.Time{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ var want bytes.Buffer
+ for _, f := range plan.Planned {
+ rc, err := rd.OpenLTXFile(context.Background(), f.Level, ltx.TXID(f.MinTXID), ltx.TXID(f.MaxTXID), 0, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ io.Copy(&want, rc)
+ rc.Close()
+ }
+ if !bytes.Equal(got, want.Bytes()) || sum.TXID != plan.TXID || len(sum.Files) != len(plan.Planned) {
+ t.Fatalf("pulled %d bytes at %d in %d files; planned %d bytes at %d in %d files",
+ len(got), sum.TXID, len(sum.Files), want.Len(), plan.TXID, len(plan.Planned))
+ }
+ privateTmpGone(t, 10*time.Second)
+}
+
// T3: from inside the worker, nothing outside its scratch is reachable,
// including any network interface but loopback; the private tmp is gone
// after a normal end, a cancel and kill -9 of the agent. Mutation: remove
diff --git a/internal/host/dbreads_test.go b/internal/host/dbreads_test.go
index 620804c23..542464139 100644
--- a/internal/host/dbreads_test.go
+++ b/internal/host/dbreads_test.go
@@ -252,3 +252,33 @@ func TestEnsureSystemUnitsKeepsTheDBReadSocketDarkWithoutTheGate(t *testing.T) {
}
}
}
+
+// Review Focus 5: the pull's cap can only be lowered on a box. Mutation:
+// dbReadsSettings keeps a CARLOS_DB_PULL_MAX_BYTES above the cap (red).
+func TestPullCapOnlyLowers(t *testing.T) {
+ t.Parallel()
+ pub, _, _ := ed25519.GenerateKey(rand.Reader)
+ base := map[string]string{"CARLOS_HIBERNATE_BUCKET": "d", "CARLOS_REGION": "uk", "CARLOS_RELEASE_PUBKEY": base64.StdEncoding.EncodeToString(pub)}
+ get := func(v string) func(string) string {
+ return func(k string) string {
+ if k == "CARLOS_DB_PULL_MAX_BYTES" {
+ return v
+ }
+ return base[k]
+ }
+ }
+ if env, why := dbReadsSettings(get("")); why != "" || env.maxPull != dbreads.MaxPullBytes {
+ t.Fatalf("default %d %q", env.maxPull, why)
+ }
+ if env, why := dbReadsSettings(get("1048576")); why != "" || env.maxPull != 1<<20 {
+ t.Fatalf("lowered %d %q", env.maxPull, why)
+ }
+ if env, why := dbReadsSettings(get("99999999999999")); why != "" || env.maxPull != dbreads.MaxPullBytes {
+ t.Fatalf("raised past the built-in cap: %d %q", env.maxPull, why)
+ }
+ for _, bad := range []string{"0", "-5", "2GB"} {
+ if _, why := dbReadsSettings(get(bad)); why == "" {
+ t.Fatalf("CARLOS_DB_PULL_MAX_BYTES=%s armed the loop", bad)
+ }
+ }
+}
diff --git a/internal/instancectl/gen.go b/internal/instancectl/gen.go
index 5adf3fb58..c23172173 100644
--- a/internal/instancectl/gen.go
+++ b/internal/instancectl/gen.go
@@ -18,8 +18,8 @@ import (
// template carlos-instance-gen@<iid>:<n>.service cannot key its slice and
// data mounts by itself: systemd has no specifier for part of an instance
// string. So before each start this service writes a runtime drop-in with
-// everything the template leaves out, reloads, checks that systemd took
-// it, and only then starts the unit.
+// everything the template leaves out, checks that systemd took it
+// (reloading first only when it has not), and only then starts the unit.
// ErrGenUnproven is StartGen's answer when the start failed after systemd
// may have started the unit and the stop that followed did not prove the
@@ -113,6 +113,26 @@ func (s *Server) slicePopulated(host string) bool {
return !strings.Contains(string(b), "populated 0")
}
+// genDropInApplied reports whether systemd's view of unit is the drop-in
+// just written (GenDropIn). The unit's name fixes the host and generation,
+// and so every line of the drop-in but the tier's: the slice says systemd
+// read a drop-in at all, and the memory lines say it was this tier's.
+// NeedDaemonReload=no says no file changed since systemd loaded the unit,
+// though only by mtime, which is why the tier is checked as well (Astra
+// review: a reused, still-loaded generation and a clock that went back).
+// Any failure to answer is a no.
+func (s *Server) genDropInApplied(ctx context.Context, host, unit string, tier cgroup.Limit) (slice string, ok bool) {
+ out, err := s.Systemctl(ctx, "show", "-p", "Slice,NeedDaemonReload,MemoryMax,MemoryHigh,MemorySwapMax", unit)
+ if err != nil {
+ return "", false
+ }
+ m := parseShow(out)
+ max := strconv.FormatInt(tier.MemoryMax, 10)
+ return m["Slice"], m["Slice"] == SliceName(host) && m["NeedDaemonReload"] == "no" &&
+ m["MemoryMax"] == max && m["MemorySwapMax"] == max &&
+ m["MemoryHigh"] == strconv.FormatInt(cgroup.HighOf(tier.MemoryMax), 10)
+}
+
// startGen is /start with a generation. The host slice is the parent (spec
// §7.3): a cold start sets it to the tier. A start that finds the slice
// populated leaves it alone, because a swap raised it with /parent before
@@ -126,6 +146,8 @@ func (s *Server) startGen(w http.ResponseWriter, r *http.Request, req request) {
}
s.markGenStarting(unit, true)
defer s.markGenStarting(unit, false)
+ // Timed for the start's log line: a wake waits on all of this.
+ began, reload := time.Now(), "skipped"
// A refusal for the disk writes nothing. An exe layout that fails
// half-way may have made its directory, so that failure cleans up too.
// A staged executable is validated before anything is laid out
@@ -150,13 +172,23 @@ func (s *Server) startGen(w http.ResponseWriter, r *http.Request, req request) {
fail(http.StatusBadGateway, "drop-in: "+err.Error())
return
}
- if out, err := s.Systemctl(ctx, "daemon-reload"); err != nil {
- fail(http.StatusBadGateway, "daemon-reload: "+strings.TrimSpace(out+" "+err.Error()))
- return
- }
- if got := s.showValue(ctx, "Slice", unit); got != SliceName(req.host) {
- fail(http.StatusBadGateway, "not starting "+unit+": its drop-in did not apply (slice "+strconv.Quote(got)+")")
- return
+ // The reload is only for a unit systemd already holds loaded from older
+ // files. A generation's unit is new and an inactive one is unloaded
+ // straight away, so the next reference reads the drop-in fresh: the
+ // common case needs no reload, which on a small box was a third of the
+ // whole wake. The check is the same either way (genDropInApplied).
+ slice, applied := s.genDropInApplied(ctx, req.host, unit, req.tier)
+ if !applied {
+ reloadStart := time.Now()
+ if out, err := s.Systemctl(ctx, "daemon-reload"); err != nil {
+ fail(http.StatusBadGateway, "daemon-reload: "+strings.TrimSpace(out+" "+err.Error()))
+ return
+ }
+ if slice, applied = s.genDropInApplied(ctx, req.host, unit, req.tier); !applied {
+ fail(http.StatusBadGateway, "not starting "+unit+": its drop-in did not apply (slice "+strconv.Quote(slice)+")")
+ return
+ }
+ reload = time.Since(reloadStart).Round(time.Millisecond).String()
}
if !s.slicePopulated(req.host) {
if err := s.setTier(ctx, req.host, req.tier); err != nil {
@@ -168,6 +200,7 @@ func (s *Server) startGen(w http.ResponseWriter, r *http.Request, req request) {
// answer says. A failure past this point kills it and proves it empty
// before its files go; one that cannot prove that keeps the files and
// says so (statusGenUnproven).
+ startAt := time.Now()
if out, err := s.Systemctl(ctx, "start", unit); err != nil {
s.abandonGen(ctx, w, req.host, unit, "start: "+strings.TrimSpace(out+" "+err.Error()))
return
@@ -177,7 +210,8 @@ func (s *Server) startGen(w http.ResponseWriter, r *http.Request, req request) {
s.abandonGen(ctx, w, req.host, unit, "show: "+err.Error())
return
}
- s.logf("instance-control: %s: started %s memory.max=%d", req.host, unit, req.tier.MemoryMax)
+ s.logf("instance-control: %s: started %s memory.max=%d (prepared in %s, reload %s, start %s)", req.host, unit, req.tier.MemoryMax,
+ startAt.Sub(began).Round(time.Millisecond), reload, time.Since(startAt).Round(time.Millisecond))
writeJSON(w, http.StatusOK, StartResponse{InvocationID: strings.TrimSpace(inv)})
}
diff --git a/internal/instancectl/server_gen_test.go b/internal/instancectl/server_gen_test.go
index 3ec774378..6f2f61937 100644
--- a/internal/instancectl/server_gen_test.go
+++ b/internal/instancectl/server_gen_test.go
@@ -19,6 +19,9 @@ const genHost = "bill-2.example.com"
var genUnit = GenUnitName(genHost, 7)
+// genTierShow is systemd's answer for the default tier's memory lines.
+const genTierShow = "MemoryMax=536870912\nMemoryHigh=483183819\nMemorySwapMax=536870912\n"
+
func genServer(t *testing.T, ctl *fakeCtl) (*Server, string, string) {
t.Helper()
dropins, exe := t.TempDir(), t.TempDir()
@@ -28,11 +31,11 @@ func genServer(t *testing.T, ctl *fakeCtl) (*Server, string, string) {
func genCtl() *fakeCtl {
return &fakeCtl{answers: map[string]string{
- "show -p Slice --value": SliceName(genHost) + "\n",
- "show -p InvocationID --value": "abc123\n",
- "list-units": "",
- "list-jobs": "",
- "show -p ActiveState --value": "inactive\n",
+ "show -p Slice,NeedDaemonReload,MemoryMax,MemoryHigh,MemorySwapMax": "Slice=" + SliceName(genHost) + "\nNeedDaemonReload=no\n" + genTierShow,
+ "show -p InvocationID --value": "abc123\n",
+ "list-units": "",
+ "list-jobs": "",
+ "show -p ActiveState --value": "inactive\n",
}}
}
@@ -54,7 +57,9 @@ MemorySwapMax=536870912
}
}
-func TestGenStartWritesTheDropInReloadsChecksTheSliceThenStarts(t *testing.T) {
+// The wake path's case: a generation's unit is new, so systemd reads its
+// drop-in on the first reference and the box-wide reload is skipped.
+func TestGenStartWritesTheDropInChecksItThenStartsWithoutAReload(t *testing.T) {
ctl := genCtl()
s, sock, dropins := genServer(t, ctl)
inv, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax)
@@ -69,8 +74,7 @@ func TestGenStartWritesTheDropInReloadsChecksTheSliceThenStarts(t *testing.T) {
t.Fatalf("exe app link = %q, %v", target, err)
}
want := strings.Join([]string{
- "daemon-reload",
- "show -p Slice --value " + genUnit,
+ "show -p Slice,NeedDaemonReload,MemoryMax,MemoryHigh,MemorySwapMax " + genUnit,
`set-property --runtime carlos-instances-bill\x2d2.example.com.slice MemoryMax=536870912 MemoryHigh=483183819 MemorySwapMax=536870912 CPUWeight=100 CPUQuota=100% IOWeight=100`,
"start " + genUnit,
"show -p InvocationID --value " + genUnit,
@@ -80,6 +84,78 @@ func TestGenStartWritesTheDropInReloadsChecksTheSliceThenStarts(t *testing.T) {
}
}
+// staleUntilReload makes systemd hold a stale copy of the unit (answer)
+// until a daemon-reload, after which the drop-in reads as applied.
+func staleUntilReload(ctl *fakeCtl, answer string) {
+ ctl.answers["show -p Slice,NeedDaemonReload,MemoryMax,MemoryHigh,MemorySwapMax"] = answer
+ ctl.before = map[string]func(){"daemon-reload": func() {
+ ctl.mu.Lock()
+ defer ctl.mu.Unlock()
+ ctl.answers["show -p Slice,NeedDaemonReload,MemoryMax,MemoryHigh,MemorySwapMax"] = "Slice=" + SliceName(genHost) + "\nNeedDaemonReload=no\n" + genTierShow
+ }}
+}
+
+// systemd still holding the unit loaded from older files says so with
+// NeedDaemonReload, or shows a slice the drop-in does not name. Either way
+// the start reloads, checks again, and only then starts.
+func TestGenStartReloadsWhenSystemdHoldsAStaleCopy(t *testing.T) {
+ for name, answer := range map[string]string{
+ "files changed since load": "Slice=" + SliceName(genHost) + "\nNeedDaemonReload=yes\n" + genTierShow,
+ "loaded without drop-in": `Slice=system-carlos\x2dinstance\x2dgen.slice` + "\nNeedDaemonReload=no\n" + genTierShow,
+ "show failed to answer": "",
+ } {
+ t.Run(name, func(t *testing.T) {
+ ctl := genCtl()
+ staleUntilReload(ctl, answer)
+ _, sock, _ := genServer(t, ctl)
+ if _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax); err != nil {
+ t.Fatal(err)
+ }
+ want := strings.Join([]string{
+ "show -p Slice,NeedDaemonReload,MemoryMax,MemoryHigh,MemorySwapMax " + genUnit,
+ "daemon-reload",
+ "show -p Slice,NeedDaemonReload,MemoryMax,MemoryHigh,MemorySwapMax " + genUnit,
+ }, "\n")
+ if got := ctl.joined(); !strings.HasPrefix(got, want+"\n") || !strings.Contains(got, "\nstart "+genUnit) {
+ t.Errorf("calls =\n%s\nwant them to begin\n%s\nand then start", got, want)
+ }
+ })
+ }
+}
+
+// systemd's change check is by mtime, so a reused generation still loaded
+// with an older tier, whose rewritten drop-in is not newer (a clock that
+// went back), reads NeedDaemonReload=no. The drop-in's other lines follow
+// from the unit's name; its memory lines are checked against the tier.
+func TestGenStartReloadsWhenSystemdHoldsAnotherTier(t *testing.T) {
+ ctl := genCtl()
+ staleUntilReload(ctl, "Slice="+SliceName(genHost)+"\nNeedDaemonReload=no\nMemoryMax=1073741824\nMemoryHigh=966367641\nMemorySwapMax=1073741824\n")
+ _, sock, _ := genServer(t, ctl)
+ if _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax); err != nil {
+ t.Fatal(err)
+ }
+ if got := ctl.joined(); !strings.Contains(got, "daemon-reload\n") {
+ t.Errorf("started on systemd's stale tier without a reload:\n%s", got)
+ }
+}
+
+// A reload that leaves systemd still saying its files changed proves
+// nothing about what it loaded: refuse rather than start on a guess.
+func TestGenStartRefusesWhenTheReloadDidNotSettle(t *testing.T) {
+ ctl := genCtl()
+ ctl.answers["show -p Slice,NeedDaemonReload,MemoryMax,MemoryHigh,MemorySwapMax"] = "Slice=" + SliceName(genHost) + "\nNeedDaemonReload=yes\n" + genTierShow
+ s, sock, dropins := genServer(t, ctl)
+ if _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax); err == nil {
+ t.Fatal("started a generation systemd still held a stale copy of")
+ }
+ if strings.Contains(ctl.joined(), "start "+genUnit) {
+ t.Errorf("the unit was started:\n%s", ctl.joined())
+ }
+ if dropIn, exe := genFilesPresent(t, s, dropins); dropIn || exe {
+ t.Errorf("files left behind: drop-in %v exe %v", dropIn, exe)
+ }
+}
+
// A successor's start during an overlap: the slice holds the incumbent,
// and its raised limit is the swap's. The start leaves it alone.
func TestGenStartLeavesAPopulatedSliceAlone(t *testing.T) {
@@ -123,7 +199,7 @@ func TestGenStartResetsAnEmptySlice(t *testing.T) {
// and an empty /data. Refuse, and leave nothing behind.
func TestGenStartRefusesWhenTheDropInDidNotApply(t *testing.T) {
ctl := genCtl()
- ctl.answers["show -p Slice --value"] = `system-carlos\x2dinstance\x2dgen.slice` + "\n"
+ ctl.answers["show -p Slice,NeedDaemonReload,MemoryMax,MemoryHigh,MemorySwapMax"] = `Slice=system-carlos\x2dinstance\x2dgen.slice` + "\nNeedDaemonReload=no\n" + genTierShow
s, sock, dropins := genServer(t, ctl)
if _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax); err == nil {
t.Fatal("started a generation whose drop-in did not apply")
@@ -388,6 +464,7 @@ func TestGenStartDropInWriteFailureRemovesTheFiles(t *testing.T) {
func TestGenStartDaemonReloadFailureRemovesTheFiles(t *testing.T) {
ctl := genCtl()
+ ctl.answers["show -p Slice,NeedDaemonReload,MemoryMax,MemoryHigh,MemorySwapMax"] = "Slice=" + SliceName(genHost) + "\nNeedDaemonReload=yes\n" + genTierShow
ctl.fail = map[string]error{"daemon-reload": errors.New("bus gone")}
s, sock, dropins := genServer(t, ctl)
_, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax)
diff --git a/internal/provision/db_region_forward_3_census_test.go b/internal/provision/db_region_forward_3_census_test.go
new file mode 100644
index 000000000..4cbd070d2
--- /dev/null
+++ b/internal/provision/db_region_forward_3_census_test.go
@@ -0,0 +1,28 @@
+package provision
+
+// Record-write census entries for db region forward plan 3 (carlos db
+// pull), layered on plans3aWriteCensusInventory (main's last at 0d9e81dc).
+// census-ruling-db-region-forward-3, accepted in full by the census owner on
+// #280, 2026-10-23 (docs/superpowers/notes/census-proposal-db-region-forward-3.md).
+// admitDBRead now admits a pull beside a query and audits it with its own
+// detail; its writes are the same conditional creates under db-reads/ in
+// the deployment bucket (the request, the queue entry), never an instance
+// or sidecar record: still nonrecord. No write site is added.
+func dbRegionForward3WriteCensusInventory() []censusEntry {
+ entries := append([]censusEntry(nil), plans3aWriteCensusInventory()...)
+ replacements := []censusEntry{
+ {"internal/console/apidbforward.go", "Server.admitDBRead", "nonrecord", "sel:Put", "3f657a9ad3b11512944ad557573bd59adf257e3fd3fe9cad9fc10a974658328b"},
+ }
+ for _, r := range replacements {
+ found := false
+ for i, e := range entries {
+ if e.File == r.File && e.Unit == r.Unit {
+ entries[i], found = r, true
+ }
+ }
+ if !found {
+ entries = append(entries, r)
+ }
+ }
+ return entries
+}
diff --git a/internal/provision/plan14_root_reads_test.go b/internal/provision/plan14_root_reads_test.go
index 09ed98fcf..9b80f6e30 100644
--- a/internal/provision/plan14_root_reads_test.go
+++ b/internal/provision/plan14_root_reads_test.go
@@ -1423,22 +1423,55 @@ func TestPlan14RootReadCensus(t *testing.T) {
t.Fatal(err)
}
seal = "65857c29501dc7676f52921bea79d08217542ecb1d775c67d20a3460c851d831"
+ // Census overlay db-region-forward-3 (db region forward plan 3: carlos
+ // db pull and pull --at for an instance in another region), loaded last,
+ // after static-redirects. census-ruling-db-region-forward-3, accepted in
+ // full by the census owner on #280, 2026-10-23
+ // (docs/superpowers/notes/census-proposal-db-region-forward-3.md). The
+ // worker's pull units (holdAndSend, verifyLTX) are accepted legacy at J4
+ // inside its sandbox, the CLI's (dbPullForward, fileSink.chunk,
+ // openDBPull) at J4 as the person, RestoreLocal's context-checked reads
+ // (ctxReadCloser.Read, round 1) at J4 as downloaded.Read is;
+ // ServeWorker and dbQueryForward are re-frozen with the pull appended. Frozen declarations are added, so
+ // the seal moves; static-redirects' is checked first.
+ if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 {
+ t.Fatal(errs)
+ }
+ if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-db-region-forward-3.tsv"), &inv, true); err != nil {
+ t.Fatal(err)
+ }
+ seal = "79cb56b4c7d3a8e155d1eff1902715d818fcf4c09b1d8a82921c413b116938a9"
+ // Census overlay wake-latency-1 (wake latency pass 1), loaded last,
+ // after db-region-forward-3. census-ruling-wake-latency-1, accepted as
+ // proposed by the census owner on #280, 2026-10-24
+ // (docs/superpowers/notes/census-proposal-wake-latency-1.md). doWake,
+ // wake and startUnitAt are re-frozen with per-step wake timing appended:
+ // clock reads and journal lines, no read added. The internal/activator
+ // and internal/instancectl envelopes are re-pinned. Frozen declarations
+ // are added, so the seal moves; db-region-forward-3's is checked first.
+ if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 {
+ t.Fatal(errs)
+ }
+ if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-wake-latency-1.tsv"), &inv, true); err != nil {
+ t.Fatal(err)
+ }
+ seal = "09e20ea4bbc534710334e632e903878656dabfb5817d4d96c6e7c56508172148"
// Census overlay sec-mail-user-boundary (email sending users are created
// under CARLOS_MAIL_USER_BOUNDARY and existing ones backfilled), loaded
- // last, after static-redirects. Proposed; ruling asked of the census
+ // last, after wake-latency-1. Proposed; ruling asked of the census
// owner on #280
// (docs/superpowers/notes/census-proposal-sec-mail-user-boundary.md).
// cmd/console serve is re-frozen (one more environment variable, the
// same three /data literals, moved); the cmd/console and internal/mail
// envelopes are re-pinned. No read is added. The serve declaration is
- // re-frozen, so the seal moves; static-redirects's is checked first.
+ // re-frozen, so the seal moves; wake-latency-1's is checked first.
if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 {
t.Fatal(errs)
}
if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-sec-mail-user-boundary.tsv"), &inv, true); err != nil {
t.Fatal(err)
}
- seal = "d82a9191c6be745d34211a8add6ac0e63dc35b28c3f6bcde99cdbc3f67154f62"
+ seal = "7b0701e08da6210d02ed0f76eabe4548d7139a772ec78c358480e1586b29c9cd"
errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...)
errs = append(errs, checkRootReadDeclarations(got, inv)...)
if len(errs) > 0 {
diff --git a/internal/provision/plan9_b8_census_test.go b/internal/provision/plan9_b8_census_test.go
index 389ad2e5c..525edfb2c 100644
--- a/internal/provision/plan9_b8_census_test.go
+++ b/internal/provision/plan9_b8_census_test.go
@@ -366,7 +366,7 @@ func TestPlan9B8RecordWriteCensus(t *testing.T) {
if len(got) < 100 {
t.Fatalf("census found only %d units under %s; the walk is broken", len(got), root)
}
- inventory := plans3aWriteCensusInventory()
+ inventory := dbRegionForward3WriteCensusInventory()
problems := append(censusDiff(inventory, got), censusPolicy(inventory)...)
if len(problems) > 0 {
t.Fatalf("record-write census rejected %d site(s); update the explicit inventory only after reviewing the source:\n %s", len(problems), strings.Join(problems, "\n "))
diff --git a/internal/provision/testdata/plan14-root-reads-db-region-forward-3.tsv b/internal/provision/testdata/plan14-root-reads-db-region-forward-3.tsv
new file mode 100644
index 000000000..806049068
--- /dev/null
+++ b/internal/provision/testdata/plan14-root-reads-db-region-forward-3.tsv
@@ -0,0 +1,23 @@
+# Census overlay db-region-forward-3 (db region forward plan 3: carlos db pull and pull --at for an instance in another region; spec docs/superpowers/specs/2026-10-22-db-region-forward-design.md rev 7, §6, §13), loaded LAST in TestPlan14RootReadCensus, after static-redirects, the overlay main loads last at 0d9e81dc. Every row carries rev3=census-ruling-db-region-forward-3, accepted in full by the census owner on #280, 2026-10-23 (docs/superpowers/notes/census-proposal-db-region-forward-3.md).
+# The scanner names two new worker units (holdAndSend, verifyLTX: the pull's hold mode, inside the db-read worker's sandbox, over the files the agent sent), three new CLI units (dbPullForward, fileSink.chunk, openDBPull: the pull's spool, its file replica, as the person), one new dbdoor unit (ctxReadCloser.Read, round 1: RestoreLocal's context-checked reads, for the CLI and the worker), and two changed legacy units (ServeWorker, dbQueryForward), re-frozen at their accepted dispositions with plan 3's change appended. No agent, console or host unit is named: the agent's pull, the console's forward of it and CARLOS_DB_PULL_MAX_BYTES add no read the scanner sees. Frozen declarations are added, so the seal moves; static-redirects' is checked first. Envelopes . (the CLI), internal/console, internal/dbdoor, internal/dbreads and internal/host are re-pinned.
+@legacy 7b7f70fe5b2c4b215433a00e1742b99d4e95df82d429b7845f9a59f493c0ce76 internal/dbreads/worker.go ServeWorker legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-1;basis=root-input (J4); the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network, PrivateTmp, /data and /etc/carlos hidden): new, stats the copy it restored in its own private /var/tmp for the restored-size cap; reads only the files the agent sent and its own scratch; round 1: it refuses, before writing anything, a plan whose restored-size cap exceeds its own LimitFSIZE or whose files total more than twice that cap;rev3=census-ruling-db-region-forward-3;basis=accepts a pull header beside a query's and hands a pull, after the same bounded receive, to holdAndSend before any restore; the Stat is unchanged, now reached by a query only; no read added Round 1 (Astra on plan 3, finding 1): after each receive it calls verifyLTX, for a query as for a pull, and refuses a plan that does not end at its target; no read added here. file:Stat@3347 37763b2fe7774cb2ad602559c5c55ac7b3021b3abad89d3c3c6cbb8e59faa6c9
+internal/dbreads/worker.go ServeWorker legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-1;basis=root-input (J4); the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network, PrivateTmp, /data and /etc/carlos hidden): new, stats the copy it restored in its own private /var/tmp for the restored-size cap; reads only the files the agent sent and its own scratch; round 1: it refuses, before writing anything, a plan whose restored-size cap exceeds its own LimitFSIZE or whose files total more than twice that cap;rev3=census-ruling-db-region-forward-3;basis=accepts a pull header beside a query's and hands a pull, after the same bounded receive, to holdAndSend before any restore; the Stat is unchanged, now reached by a query only; no read added Round 1 (Astra on plan 3, finding 1): after each receive it calls verifyLTX, for a query as for a pull, and refuses a plan that does not end at its target; no read added here. file:Stat@3347 37763b2fe7774cb2ad602559c5c55ac7b3021b3abad89d3c3c6cbb8e59faa6c9
+@legacy 9df9b8719b2f4669d689736ab4f50b32f7eed042e8756f3d7b96448f7377316c internal/dbreads/worker.go holdAndSend legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4); the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network, PrivateTmp, /data and /etc/carlos hidden): new (plan 3, a pull), opens each planned LTX file it received and verified, in its own private /var/tmp, once check B has passed, and copies it back to the agent over its socket; reads only the files the agent sent and its own scratch; decides nothing for root Round 1: the per-file verification moved to ServeWorker; this unit only opens the checked files to send them back. file:Open@796 a084883e6c399444fd88f81b22b739819eb305f85975a280210ba749f595d84a
+internal/dbreads/worker.go holdAndSend legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4); the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network, PrivateTmp, /data and /etc/carlos hidden): new (plan 3, a pull), opens each planned LTX file it received and verified, in its own private /var/tmp, once check B has passed, and copies it back to the agent over its socket; reads only the files the agent sent and its own scratch; decides nothing for root Round 1: the per-file verification moved to ServeWorker; this unit only opens the checked files to send them back. file:Open@796 a084883e6c399444fd88f81b22b739819eb305f85975a280210ba749f595d84a
+@legacy 2ec89dd852f01aaac42d7ea8142198d065a0aa399cd69423136e4f9b7c048538 internal/dbreads/worker.go verifyLTX legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4); the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network, PrivateTmp, /data and /etc/carlos hidden): new (plan 3, a pull), opens one planned LTX file it received, in its own private /var/tmp, and decodes it whole to check its LTX checksums before ready (the worker's decode of tenant bytes, as its restore is); reads only the files the agent sent and its own scratch Round 1: also compares the decoded header's transaction range with the file's plan entry, for a query (header only) as for a pull (decoded whole), so a file is never restored or shipped as a transaction it does not hold. file:Open@78 76da616e0866788c581e9eb70c648595ffc63010501efb4a6f9ee8cdf4ab1442
+internal/dbreads/worker.go verifyLTX legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4); the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network, PrivateTmp, /data and /etc/carlos hidden): new (plan 3, a pull), opens one planned LTX file it received, in its own private /var/tmp, and decodes it whole to check its LTX checksums before ready (the worker's decode of tenant bytes, as its restore is); reads only the files the agent sent and its own scratch Round 1: also compares the decoded header's transaction range with the file's plan entry, for a query (header only) as for a pull (decoded whole), so a file is never restored or shipped as a transaction it does not hold. file:Open@78 76da616e0866788c581e9eb70c648595ffc63010501efb4a6f9ee8cdf4ab1442
+@legacy 8b7f718c59b7d328dbc383b0cfdb2e54540b695cc66890d7e8cf7fe4f5f1d912 dbforward.go dbQueryForward legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-2;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new, reads at most 64 KiB of a refused reads-route answer from the console (io.ReadAll over the HTTP response body, not a file) to print the console's own sentence; decides nothing for root;rev3=census-ruling-db-region-forward-3;basis=keeps its chunks in a memSink and opens them with openDBForward after readDBForward returns the result, the same bytes and checks as before; the same bounded refusal read; no read added file:ReadAll@1371 757573f0b60a4b2e4eafdef63b4d1157363768dcbc9354e1317f52e886394dc8
+dbforward.go dbQueryForward legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-2;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new, reads at most 64 KiB of a refused reads-route answer from the console (io.ReadAll over the HTTP response body, not a file) to print the console's own sentence; decides nothing for root;rev3=census-ruling-db-region-forward-3;basis=keeps its chunks in a memSink and opens them with openDBForward after readDBForward returns the result, the same bytes and checks as before; the same bounded refusal read; no read added file:ReadAll@1371 757573f0b60a4b2e4eafdef63b4d1157363768dcbc9354e1317f52e886394dc8
+@legacy a076452b9b26a17382535b2f5145f84742c8324457827f20b107dccbef494555 dbforward.go dbPullForward legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new (plan 3), reads at most 64 KiB of a refused reads-route answer from the console (io.ReadAll over the HTTP response body, not a file) to print the console's own sentence, as dbQueryForward does; creates the pull's one private 0700 directory under the system temp dir and removes it on every path but success, where db.go's dbPull removes it after copyOut; decides nothing for root file:ReadAll@939 9f9d794b2d34a5613098f39edc8aa2e1d572a95ef41baa6c621173b3911833d2
+dbforward.go dbPullForward legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new (plan 3), reads at most 64 KiB of a refused reads-route answer from the console (io.ReadAll over the HTTP response body, not a file) to print the console's own sentence, as dbQueryForward does; creates the pull's one private 0700 directory under the system temp dir and removes it on every path but success, where db.go's dbPull removes it after copyOut; decides nothing for root file:ReadAll@939 9f9d794b2d34a5613098f39edc8aa2e1d572a95ef41baa6c621173b3911833d2
+@legacy 2425eaa4ccaa924c1796aa2c4489494da992fffb0c3c0a3a4ceaa883fe9f4067 dbforward.go fileSink.chunk legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new (plan 3), reads back one sealed chunk it spooled itself, 0600 in the pull's private 0700 directory under the system temp dir, each admitted by the chunk bound before it was written; the bytes open only with the CLI's own ephemeral key; decides nothing for root file:ReadFile@58 6a1f96c3ac25abc41412246cb50d6412d61fd91fcd1e7ba9d8ef6be6c97d6e8b
+dbforward.go fileSink.chunk legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new (plan 3), reads back one sealed chunk it spooled itself, 0600 in the pull's private 0700 directory under the system temp dir, each admitted by the chunk bound before it was written; the bytes open only with the CLI's own ephemeral key; decides nothing for root file:ReadFile@58 6a1f96c3ac25abc41412246cb50d6412d61fd91fcd1e7ba9d8ef6be6c97d6e8b
+@legacy 9a11fd8342a7c74d49979995cd603bc2d62329005f580eb0dc5ce3bda9ddfac0 dbforward.go openDBPull legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new (plan 3), creates each planned LTX file exclusively (O_EXCL, 0600) as a litestream file replica in the pull's private 0700 directory, only after the summary's whole file list has been checked, from chunks opened with the CLI's own key; then restores and checks them with dbdoor.RestoreLocal and removes them; the count and SHA-256 are checked before the restore; decides nothing for root Round 1 (Astra on plan 3, findings 1 and 2): compares each rebuilt file's LTX header, from the bytes it wrote, with its manifest entry, and checks the context before each chunk; no read added. file:OpenFile@2009 51fc426dbd5d8730bbd15c92fb023fa9af85c999eee769decfd63bb97635f66a
+dbforward.go openDBPull legacy accepted-legacy accepted-legacy;rev3=census-ruling-db-region-forward-3;basis=root-input (J4, as main's db.go units); the carlos CLI on a person's machine, as the person (no box unit): new (plan 3), creates each planned LTX file exclusively (O_EXCL, 0600) as a litestream file replica in the pull's private 0700 directory, only after the summary's whole file list has been checked, from chunks opened with the CLI's own key; then restores and checks them with dbdoor.RestoreLocal and removes them; the count and SHA-256 are checked before the restore; decides nothing for root Round 1 (Astra on plan 3, findings 1 and 2): compares each rebuilt file's LTX header, from the bytes it wrote, with its manifest entry, and checks the context before each chunk; no read added. file:OpenFile@2009 51fc426dbd5d8730bbd15c92fb023fa9af85c999eee769decfd63bb97635f66a
+@legacy f89c10f1e2034212c6b36f0ea099bd5933f0a5e24621c3374479a9829c568bbb internal/dbdoor/plan_from.go ctxReadCloser.Read legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input (J4, as main's downloaded.Read);rev3=census-ruling-db-region-forward-3;identity=the carlos CLI on a person's machine, as the person, and the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network): new (Astra round 1 on plan 3, finding 2), RestoreLocal's reads of the local replica files it was handed, each checking the restore's context first so litestream's compaction stops on Ctrl-C or the deadline; reads only those files; decides nothing for root file:Read@117,file:Read@23 e6bf708f2f7a6643b55b6afc64bd94f65cd60e4235f99a5a91ced9d353a6e67b
+internal/dbdoor/plan_from.go ctxReadCloser.Read legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input (J4, as main's downloaded.Read);rev3=census-ruling-db-region-forward-3;identity=the carlos CLI on a person's machine, as the person, and the db-read worker (carlos system db-worker, carlos-dbread@.service, a DynamicUser with no network): new (Astra round 1 on plan 3, finding 2), RestoreLocal's reads of the local replica files it was handed, each checking the restore's context first so litestream's compaction stops on Ctrl-C or the deadline; reads only those files; decides nothing for root file:Read@117,file:Read@23 e6bf708f2f7a6643b55b6afc64bd94f65cd60e4235f99a5a91ced9d353a6e67b
+@coverage . cc17fa0a3414d5c8811d5728f339a9a54b0e9fe0aee0a547f9bc677f1ddd1ab7
+@coverage internal/console a09cd3a5db072ff86040580fb038d723c9a114e72a15a39db688bd2523f6bcb4
+@coverage internal/dbreads 64d1eb057fc028865c5b99b2b9bef2714a95eac75ae711f84018980f077d9403
+@coverage internal/host 218f8d6a50eb852a22424236fa89427b4b5509e08fd956ce778d95bc5a63c9a0
+@coverage internal/dbdoor e705cafd886bc300e3bacdf65bcfcf09e0e824c30967fd8d82c36238a2ad8573
diff --git a/internal/provision/testdata/plan14-root-reads-sec-mail-user-boundary.tsv b/internal/provision/testdata/plan14-root-reads-sec-mail-user-boundary.tsv
index 76e3a4523..4c10ecf71 100644
--- a/internal/provision/testdata/plan14-root-reads-sec-mail-user-boundary.tsv
+++ b/internal/provision/testdata/plan14-root-reads-sec-mail-user-boundary.tsv
@@ -1,5 +1,5 @@
-# Census overlay sec-mail-user-boundary (email sending users are created under a permissions boundary, CARLOS_MAIL_USER_BOUNDARY, and existing users are backfilled with iam:PutUserPermissionsBoundary), loaded last, after static-redirects. PROPOSED-sec-mail-user-boundary; the ruling is asked of the census owner on #280 (docs/superpowers/notes/census-proposal-sec-mail-user-boundary.md).
-# cmd/console serve is re-frozen at main's disposition with this change appended: it reads one more environment variable and fails boot without it; its three /data literals move but are the same three. No read, file or record write is added. The cmd/console and internal/mail envelopes are re-pinned (main.go's mailUserBoundary and doc comment; aws.go's boundary check, CreateUser field and backfill; their tests). The serve declaration is re-frozen, so the seal moves; static-redirects's is checked first.
+# Census overlay sec-mail-user-boundary (email sending users are created under a permissions boundary, CARLOS_MAIL_USER_BOUNDARY, and existing users are backfilled with iam:PutUserPermissionsBoundary), loaded last, after wake-latency-1. PROPOSED-sec-mail-user-boundary; the ruling is asked of the census owner on #280 (docs/superpowers/notes/census-proposal-sec-mail-user-boundary.md).
+# cmd/console serve is re-frozen at main's disposition with this change appended: it reads one more environment variable and fails boot without it; its three /data literals move but are the same three. No read, file or record write is added. The cmd/console and internal/mail envelopes are re-pinned (main.go's mailUserBoundary and doc comment; aws.go's boundary check, CreateUser field and backfill; their tests). The serve declaration is re-frozen, so the seal moves; wake-latency-1's is checked first.
@legacy 041b03c43c4c7ab2f7c07c3db18540abd86d776900b0d42c884a838de68bdb0b cmd/console/main.go serve legacy accepted-legacy accepted-legacy;rev3=PROPOSED-scan-demand-revival;basis=Console service wires the optional callback signer and starts the claim-guarded queue worker. The same three data path literals move; it does not add a privileged reader.;rev3=PROPOSED-email-signin;identity=the console (cmd/console serve under console.service, user carlos-console);basis=passes CARLOS_CONSOLE_NAME to NewServer as Config.ConsoleName, the name the console's emails use; the same three /data literals at the same offsets; no read added;rev3=PROPOSED-s3-change-index;identity=the console (cmd/console serve under console.service, user carlos-console);basis=arms the record view when CARLOS_CHANGE_INDEX=read (armChangeIndex) for the run and seed sweeps and starts its backstop loop; the same three /data literals at the same offsets; no read added;rev3=census-ruling-db-region-forward-2;identity=the console (cmd/console serve under console.service, user carlos-console);basis=parses CARLOS_DB_FORWARD (dbForward) into Config.DBForward and builds its http.Server with console.NewHTTPServer, the same two timeouts; the same three /data literals at the same offsets; no read added;rev3=PROPOSED-sec-mail-user-boundary;identity=the console (cmd/console serve under console.service, user carlos-console);basis=reads CARLOS_MAIL_USER_BOUNDARY (an environment variable, not a file) through mailUserBoundary, stops with log.Fatal when CARLOS_MAIL_REGION is set and it is empty or not an IAM policy ARN, and passes it to mail.NewAWSProvisioner through a closure handed to mailBackends; the same three /data literals, each moved 282 bytes; no read added data-literal@4359,data-literal@5460,data-literal@5697 2fbc4e6d3ad3999eeacd7eeb880d14f9dac4e70df75f192649cba2bf4da0f548
cmd/console/main.go serve legacy accepted-legacy accepted-legacy;rev3=PROPOSED-scan-demand-revival;basis=Console service wires the optional callback signer and starts the claim-guarded queue worker. The same three data path literals move; it does not add a privileged reader.;rev3=PROPOSED-email-signin;identity=the console (cmd/console serve under console.service, user carlos-console);basis=passes CARLOS_CONSOLE_NAME to NewServer as Config.ConsoleName, the name the console's emails use; the same three /data literals at the same offsets; no read added;rev3=PROPOSED-s3-change-index;identity=the console (cmd/console serve under console.service, user carlos-console);basis=arms the record view when CARLOS_CHANGE_INDEX=read (armChangeIndex) for the run and seed sweeps and starts its backstop loop; the same three /data literals at the same offsets; no read added;rev3=census-ruling-db-region-forward-2;identity=the console (cmd/console serve under console.service, user carlos-console);basis=parses CARLOS_DB_FORWARD (dbForward) into Config.DBForward and builds its http.Server with console.NewHTTPServer, the same two timeouts; the same three /data literals at the same offsets; no read added;rev3=PROPOSED-sec-mail-user-boundary;identity=the console (cmd/console serve under console.service, user carlos-console);basis=reads CARLOS_MAIL_USER_BOUNDARY (an environment variable, not a file) through mailUserBoundary, stops with log.Fatal when CARLOS_MAIL_REGION is set and it is empty or not an IAM policy ARN, and passes it to mail.NewAWSProvisioner through a closure handed to mailBackends; the same three /data literals, each moved 282 bytes; no read added data-literal@4359,data-literal@5460,data-literal@5697 2fbc4e6d3ad3999eeacd7eeb880d14f9dac4e70df75f192649cba2bf4da0f548
@coverage cmd/console 19d46a41e903849729d7c33dd049d5752fc03b80b7f93d393472b346324172cf
diff --git a/internal/provision/testdata/plan14-root-reads-wake-latency-1.tsv b/internal/provision/testdata/plan14-root-reads-wake-latency-1.tsv
new file mode 100644
index 000000000..d03f7208f
--- /dev/null
+++ b/internal/provision/testdata/plan14-root-reads-wake-latency-1.tsv
@@ -0,0 +1,10 @@
+# Census overlay wake-latency-1 (wake latency pass 1: a generation starts without a box-wide daemon-reload when systemd already reads its drop-in; health probes back off from 5 ms; per-step wake timing in the journal), loaded LAST in TestPlan14RootReadCensus, after db-region-forward-3, the overlay main loads last at 64a002c8. census-ruling-wake-latency-1, accepted as proposed by the census owner on #280, 2026-10-24 (docs/superpowers/notes/census-proposal-wake-latency-1.md).
+# Service.doWake, Service.wake and ExecBackend.startUnitAt are re-frozen at their accepted dispositions with the timing appended: clock reads and journal lines only, no read added, removed or reordered. Frozen declarations are added, so the seal moves; db-region-forward-3's is checked first. Envelopes internal/activator (the timing, the health backoff) and internal/instancectl (startGen's drop-in check) are re-pinned.
+@legacy 9e3f18aa71907fbaa1bd94c6d420cceac6c3b5f809915a0ba86bc231d729b212 internal/activator/activator.go Service.doWake legacy accepted-legacy accepted-legacy;rev3=census-ruling-292fix;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): main's body (census ruling 292's legacy start mark and lease give-back; census ruling owed's pending record) with the seeded change ruled under census-ruling-seeded-instances (the refreshed row checked against the row the wake began from, twice; the seed's first upload recorded after the wake), and now one more line on a failed start: a start its pre-exec gate refused (launchNotStarted: the exec gate, the pending record or its recheck answered no before root was asked) is recorded as known not started (stopObservation.notStarted) instead of uncertain, only when the wake has started nothing else (wakeStartedNothing); otherwise it goes through cleanupFailedWake and its proofs. A refusal before its lease (step 1), which has started nothing, is recorded the same way. Read sites unchanged; no new read and no root decision;rev3=census-ruling-wake-latency-1;identity=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos);basis=marks the end of each wake step on the woken instance's journal timer (wakeSteps: a.now() and a slice append, journal only); no read added, no step reordered registry:DBPath@1300,registry:DBPath@1316,registry:DBPath@1443,registry:DBPath@1454,registry:DBPath@22081,registry:Host@1129,registry:Host@1204,registry:Host@1282,registry:Host@1434,registry:Host@174,registry:Host@1768,registry:Host@2628,registry:Host@4603,registry:Replication@22116,registry:Seed@12122,registry:Socket@1329,registry:Socket@1345,registry:Socket@1468,registry:Socket@1479,whole:Route@10056,whole:Route@10248,whole:Route@10938,whole:Route@12223,whole:Route@1227,whole:Route@12294,whole:Route@14710,whole:Route@159,whole:Route@16568,whole:Route@1663,whole:Route@1668,whole:Route@17214,whole:Route@1722,whole:Route@1728,whole:Route@1794,whole:Route@1802,whole:Route@18521,whole:Route@18675,whole:Route@19031,whole:Route@19125,whole:Route@19841,whole:Route@20496,whole:Route@21495,whole:Route@21677,whole:Route@22190,whole:Route@22549,whole:Route@23521,whole:Route@23616,whole:Route@2518,whole:Route@3972,whole:Route@4231,whole:Route@4493,whole:Route@4498,whole:Route@4555,whole:Route@4561,whole:Route@4630,whole:Route@4638,whole:Route@5426,whole:Route@6494,whole:Route@7559,whole:Route@8440 28b4040841dc7824fd2f2bf259446d17ea66ca2932b67521b38d337c55fd50a8
+@legacy 7377249f1ac50bddc95e84c96bf641513f8bbfd489731d6b4ad627d4e9b8d098 internal/activator/activator.go Service.wake legacy accepted-legacy accepted-legacy;rev3=census-ruling-carlos-run;basis=Re-digested at the pre-land merge: the body includes main's own change under census ruling 292, with no further change from this branch. the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): main's body; the stop now carries its reason for a run it interrupts (beginStopLocked's why argument); read sites unchanged, offsets moved;rev3=census-ruling-wake-latency-1;identity=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos);basis=logs that timer's breakdown to the journal after a successful wake (w.steps.String(), e.host, w.wakeTook); no read added registry:Socket@6955,registry:Socket@7116,whole:Route@2091,whole:Route@2148,whole:Route@5492,whole:Route@6908,whole:Route@6957,whole:Route@7118,whole:Route@8441,whole:Route@8462 bce391d35720996a8c650b24f350959f7294171f76071bd0037ac801124e3d9a
+@legacy fa32fa2f7102b48f474e81a5d426d5025ba77eb5a0612bdc7f7ef4f14a946d4f internal/activator/backend_exec.go ExecBackend.startUnitAt legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route;rev3=census-ruling-wake-latency-1;identity=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos);basis=times the unit's launch and its health wait with time.Now() and logs both, with rt.Host and the unit name, after a successful health wait (the new registry:Host site); no read added, no behaviour changed registry:Host@1384,registry:Host@2097,registry:Host@2136,whole:Route@2265,whole:Route@231 4ea2538e564e29f9ea7ba935775fbeef8cb602d7f0d00ad04bef84792d3b2803
+internal/activator/activator.go Service.doWake legacy accepted-legacy accepted-legacy;rev3=census-ruling-292fix;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): main's body (census ruling 292's legacy start mark and lease give-back; census ruling owed's pending record) with the seeded change ruled under census-ruling-seeded-instances (the refreshed row checked against the row the wake began from, twice; the seed's first upload recorded after the wake), and now one more line on a failed start: a start its pre-exec gate refused (launchNotStarted: the exec gate, the pending record or its recheck answered no before root was asked) is recorded as known not started (stopObservation.notStarted) instead of uncertain, only when the wake has started nothing else (wakeStartedNothing); otherwise it goes through cleanupFailedWake and its proofs. A refusal before its lease (step 1), which has started nothing, is recorded the same way. Read sites unchanged; no new read and no root decision;rev3=census-ruling-wake-latency-1;identity=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos);basis=marks the end of each wake step on the woken instance's journal timer (wakeSteps: a.now() and a slice append, journal only); no read added, no step reordered registry:DBPath@1300,registry:DBPath@1316,registry:DBPath@1443,registry:DBPath@1454,registry:DBPath@22081,registry:Host@1129,registry:Host@1204,registry:Host@1282,registry:Host@1434,registry:Host@174,registry:Host@1768,registry:Host@2628,registry:Host@4603,registry:Replication@22116,registry:Seed@12122,registry:Socket@1329,registry:Socket@1345,registry:Socket@1468,registry:Socket@1479,whole:Route@10056,whole:Route@10248,whole:Route@10938,whole:Route@12223,whole:Route@1227,whole:Route@12294,whole:Route@14710,whole:Route@159,whole:Route@16568,whole:Route@1663,whole:Route@1668,whole:Route@17214,whole:Route@1722,whole:Route@1728,whole:Route@1794,whole:Route@1802,whole:Route@18521,whole:Route@18675,whole:Route@19031,whole:Route@19125,whole:Route@19841,whole:Route@20496,whole:Route@21495,whole:Route@21677,whole:Route@22190,whole:Route@22549,whole:Route@23521,whole:Route@23616,whole:Route@2518,whole:Route@3972,whole:Route@4231,whole:Route@4493,whole:Route@4498,whole:Route@4555,whole:Route@4561,whole:Route@4630,whole:Route@4638,whole:Route@5426,whole:Route@6494,whole:Route@7559,whole:Route@8440 28b4040841dc7824fd2f2bf259446d17ea66ca2932b67521b38d337c55fd50a8
+internal/activator/activator.go Service.wake legacy accepted-legacy accepted-legacy;rev3=census-ruling-carlos-run;basis=Re-digested at the pre-land merge: the body includes main's own change under census ruling 292, with no further change from this branch. the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): main's body; the stop now carries its reason for a run it interrupts (beginStopLocked's why argument); read sites unchanged, offsets moved;rev3=census-ruling-wake-latency-1;identity=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos);basis=logs that timer's breakdown to the journal after a successful wake (w.steps.String(), e.host, w.wakeTook); no read added registry:Socket@6955,registry:Socket@7116,whole:Route@2091,whole:Route@2148,whole:Route@5492,whole:Route@6908,whole:Route@6957,whole:Route@7118,whole:Route@8441,whole:Route@8462 bce391d35720996a8c650b24f350959f7294171f76071bd0037ac801124e3d9a
+internal/activator/backend_exec.go ExecBackend.startUnitAt legacy accepted-legacy accepted-legacy;rev3=J1;basis=root-route;rev3=census-ruling-wake-latency-1;identity=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos);basis=times the unit's launch and its health wait with time.Now() and logs both, with rt.Host and the unit name, after a successful health wait (the new registry:Host site); no read added, no behaviour changed registry:Host@1384,registry:Host@2097,registry:Host@2136,whole:Route@2265,whole:Route@231 4ea2538e564e29f9ea7ba935775fbeef8cb602d7f0d00ad04bef84792d3b2803
+@coverage internal/activator cd29cfafc2b10f03be81bec326b082219b615d25512d02bf941ae35e56483f52
+@coverage internal/instancectl f0e91529abcdeb4ead7761ac3017c5ff75a0d809f5b09ea25c9681535fa88d6c