amadan

carlos / platform Public

WIP checkpoint (not for merge): files seed checkpoint 8: restore compatibility, recovery tests, root-runner tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paul Campbell pushed by claude 403986097a066e1d084029ac3cdc58cfcf29bd97 parent 58716b1
7 files changed, +693 −10
  • internal/host/claimfilesseed_recovery_linux_test.go +67 −0
  • internal/host/claimfilesseed_restore_linux_test.go +123 −7
  • internal/host/claimservingpark_rootrunner_linux_test.go +391 −0
  • internal/host/plan12_restore_linux_test.go +6 −1
  • internal/hostbox/restore.go +29 −0
  • internal/hostbox/restore_test.go +74 −0
  • internal/provision/testdata/plan14-root-reads-seed.tsv +3 −2

CI passed — run details

CI log 
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	347.123s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.276s
ok  	github.com/carlosframework/platform/cmd/moveprobe	93.271s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.709s
ok  	github.com/carlosframework/platform/internal/activator	235.148s
ok  	github.com/carlosframework/platform/internal/agentsock	0.017s
ok  	github.com/carlosframework/platform/internal/alerting	0.007s
ok  	github.com/carlosframework/platform/internal/analytics	0.028s
ok  	github.com/carlosframework/platform/internal/appconfig	2.333s
ok  	github.com/carlosframework/platform/internal/apperr	0.075s
ok  	github.com/carlosframework/platform/internal/applog	0.129s
ok  	github.com/carlosframework/platform/internal/awsinv	0.004s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.005s
ok  	github.com/carlosframework/platform/internal/boxid	0.657s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.008s
ok  	github.com/carlosframework/platform/internal/caps	0.013s
ok  	github.com/carlosframework/platform/internal/certwarm	0.004s
ok  	github.com/carlosframework/platform/internal/cgroup	0.712s
ok  	github.com/carlosframework/platform/internal/changelog	0.314s
ok  	github.com/carlosframework/platform/internal/clamd	0.054s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	365.265s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.109s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.009s
ok  	github.com/carlosframework/platform/internal/dbverify	0.004s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.002s
ok  	github.com/carlosframework/platform/internal/docsite	0.004s
ok  	github.com/carlosframework/platform/internal/domains	2.114s
ok  	github.com/carlosframework/platform/internal/doorkey	0.006s
ok  	github.com/carlosframework/platform/internal/economics	0.004s
ok  	github.com/carlosframework/platform/internal/edge	26.217s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.007s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.122s
ok  	github.com/carlosframework/platform/internal/gate	1.657s
ok  	github.com/carlosframework/platform/internal/genselect	0.014s
ok  	github.com/carlosframework/platform/internal/geo	0.014s
ok  	github.com/carlosframework/platform/internal/geoip	0.029s
ok  	github.com/carlosframework/platform/internal/host	139.194s
ok  	github.com/carlosframework/platform/internal/hostactors	0.373s
ok  	github.com/carlosframework/platform/internal/hostbox	2.185s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.535s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.014s
ok  	github.com/carlosframework/platform/internal/hostexec	4.886s
ok  	github.com/carlosframework/platform/internal/hostfence	0.032s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.526s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.015s
ok  	github.com/carlosframework/platform/internal/hostguard	1.013s
ok  	github.com/carlosframework/platform/internal/hostops	1.423s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.014s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.014s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.455s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.128s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.014s
ok  	github.com/carlosframework/platform/internal/infraview	0.015s
ok  	github.com/carlosframework/platform/internal/instancectl	1.825s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.725s
ok  	github.com/carlosframework/platform/internal/instancefs	0.013s
ok  	github.com/carlosframework/platform/internal/instancekey	0.015s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.020s
ok  	github.com/carlosframework/platform/internal/layout	1.448s
ok  	github.com/carlosframework/platform/internal/lease	9.472s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.019s
ok  	github.com/carlosframework/platform/internal/ledger	0.011s
ok  	github.com/carlosframework/platform/internal/mail	0.166s
ok  	github.com/carlosframework/platform/internal/maintenance	0.088s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.546s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.070s
ok  	github.com/carlosframework/platform/internal/metrics	0.011s
ok  	github.com/carlosframework/platform/internal/moves	20.938s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.785s
ok  	github.com/carlosframework/platform/internal/objstore	0.167s
ok  	github.com/carlosframework/platform/internal/ops	0.157s
ok  	github.com/carlosframework/platform/internal/park	0.200s
ok  	github.com/carlosframework/platform/internal/privatefile	0.007s
ok  	github.com/carlosframework/platform/internal/provision	13.583s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.006s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	4.006s
ok  	github.com/carlosframework/platform/internal/relay	7.248s
ok  	github.com/carlosframework/platform/internal/release	19.566s
ok  	github.com/carlosframework/platform/internal/rollout	3.783s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.003s
ok  	github.com/carlosframework/platform/internal/rootaws	0.006s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.007s
ok  	github.com/carlosframework/platform/internal/rootstate	0.027s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.004s
ok  	github.com/carlosframework/platform/internal/schedule	2.288s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	9.723s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.501s
ok  	github.com/carlosframework/platform/internal/status	0.040s
ok  	github.com/carlosframework/platform/internal/steering	0.005s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.229s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.003s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.004s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.572s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: cached, digest verified
  mc: cached, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.023s
=== RUN   TestT4StorageProbeMinio
2026/10/14 17:16:01 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DE74D55DE7B91D, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.022s
minio contract: PASS
diff --git a/internal/host/claimfilesseed_recovery_linux_test.go b/internal/host/claimfilesseed_recovery_linux_test.go
new file mode 100644
index 000000000..9f5d136d4
--- /dev/null
+++ b/internal/host/claimfilesseed_recovery_linux_test.go
@@ -0,0 +1,67 @@
+//go:build linux
+
+package host
+
+import (
+ "os"
+ "path/filepath"
+ "reflect"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/hostbox"
+)
+
+func TestServingParkIncompleteRecoveryRechecksEnrollment(t *testing.T) {
+ for _, change := range []string{"enrollment", "revision"} {
+ t.Run(change, func(t *testing.T) {
+ f := newServingRootFixture(t, true)
+ a, err := f.park()
+ if err != nil {
+ t.Fatal(err)
+ }
+ f.tags(a.Successor.Stamp)
+ before, err := f.p.ledger.read(a.ID)
+ if err != nil || before.Complete || !before.Parked {
+ t.Fatal("expected pending completion", err)
+ }
+ actor := f.actor
+ if change == "enrollment" {
+ f.actor.Enrollment = "new-enrollment"
+ } else {
+ f.actor.Revision++
+ }
+ // Reconstruct the authority object, retaining only durable journals.
+ root := &servingParkRoot{root: f.p.root, ledger: &servingParkLedger{state: f.p.ledger.state, fs: f.p.ledger.fs, journal: f.f.journal, now: f.p.ledger.now}}
+ if err := root.completeServingDatabasePark(t.Context(), f.conn, a.ID, a.Successor); err == nil {
+ t.Fatal("changed deployment identity completed old park")
+ }
+ if _, err := root.parkServingDatabase(t.Context(), f.conn, a.Successor, hostbox.ServingArtifactIdentity{}); err == nil {
+ t.Fatal("changed deployment identity replayed old park")
+ }
+ after, err := root.ledger.read(a.ID)
+ if err != nil || !reflect.DeepEqual(before, after) {
+ t.Fatal("refusal rewrote pending evidence", err)
+ }
+ // Restoring the recorded identity permits the ordinary completion.
+ // A real enrollment change needs separately established root evidence.
+ f.actor = actor
+ if err := root.completeServingDatabasePark(t.Context(), f.conn, a.ID, a.Successor); err != nil {
+ t.Fatal(err)
+ }
+ after, err = root.ledger.read(a.ID)
+ if err != nil || !after.Complete {
+ t.Fatal("completion did not recover", err)
+ }
+ after.Complete = before.Complete
+ if !reflect.DeepEqual(before, after) {
+ t.Fatal("completion changed source, receipts or retention clock")
+ }
+ for suffix, want := range map[string]string{"": "served bytes", ".gen": "G", "-wal": "WAL", "-shm": "SHM"} {
+ body, err := os.ReadFile(filepath.Join(f.f.guard.DataDir, "instances", ".serving-parked", a.ID, filepath.Base(f.db)+suffix))
+ if err != nil || string(body) != want {
+ t.Fatalf("retained %s: %q %v", suffix, body, err)
+ }
+ }
+ })
+ }
+}
diff --git a/internal/host/claimfilesseed_restore_linux_test.go b/internal/host/claimfilesseed_restore_linux_test.go
index b28b7d688..43b70d92d 100644
--- a/internal/host/claimfilesseed_restore_linux_test.go
+++ b/internal/host/claimfilesseed_restore_linux_test.go
@@ -11,6 +11,7 @@ import (
"path/filepath"
"reflect"
"strings"
+ "sync/atomic"
"syscall"
"testing"
"time"
@@ -201,7 +202,7 @@ func (p *servingPreparation) prepare(t *testing.T) (hostclaim.Observed, activato
}
func TestFilesSeedOrdinaryPreparationRestore(t *testing.T) {
- for _, failure := range []string{"", "enrolled-refusal", "missing-marker", "cancel-after-enrollment", "disk-after-enrollment", "invalid-sqlite"} {
+ for _, failure := range []string{"", "enrolled-refusal", "missing-marker", "cancel-after-enrollment", "disk-after-enrollment", "invalid-sqlite", "cancel-after-output", "restore-checksum"} {
for _, restart := range []string{"same-controller", "controller", "reboot"} {
t.Run(failure+"/"+restart, func(t *testing.T) { runFilesSeedOrdinaryPreparation(t, failure, restart) })
}
@@ -257,9 +258,30 @@ func runFilesSeedOrdinaryPreparation(t *testing.T, failure, restart string, moun
t.Error(err)
}
})
+ if err := unix.Mount("", instances, "", unix.MS_BIND|unix.MS_REMOUNT|unix.MS_NOSUID|unix.MS_NODEV|unix.MS_NOEXEC, ""); err != nil {
+ t.Fatal(err)
+ }
+ if parent, _, err := newServingParkFS(f.f.guard.DataDir, state).parent(); err == nil {
+ parent.Close()
+ t.Fatal("production filesystem accepted a mount without nosymfollow")
+ }
if err := unix.Mount("", instances, "", unix.MS_BIND|unix.MS_REMOUNT|unix.MS_NOSYMFOLLOW|unix.MS_NOSUID|unix.MS_NODEV|unix.MS_NOEXEC, ""); err != nil {
t.Fatal(err)
}
+ link := filepath.Join(instances, ".nosymfollow-witness")
+ if err := os.Symlink(f.db, link); err != nil {
+ t.Fatal(err)
+ }
+ fd, err := unix.Open(link, unix.O_RDONLY|unix.O_CLOEXEC, 0)
+ if fd >= 0 {
+ unix.Close(fd)
+ }
+ if !errors.Is(err, syscall.ELOOP) {
+ t.Fatalf("kernel followed symlink on nosymfollow mount: %v", err)
+ }
+ if err := os.Remove(link); err != nil {
+ t.Fatal(err)
+ }
// Use the production UID and mount checks in this isolated namespace.
f.p.ledger.state = state
f.p.ledger.fs = newServingParkFS(f.f.guard.DataDir, state)
@@ -290,7 +312,11 @@ func runFilesSeedOrdinaryPreparation(t *testing.T, failure, restart string, moun
} else {
f, _, _ = newServingLegacyRootSetup(t, false)
}
- selected := plan12RestoreBaseline(t, f.f)
+ extraBytes := 0
+ if failure == "cancel-after-output" {
+ extraBytes = 2 << 20
+ }
+ selected := plan12RestoreBaseline(t, f.f, extraBytes)
raw, err := f.f.data.GetVersion(t.Context(), f.f.bucket, selected.Files[0].Key, selected.Files[0].VersionID)
if err != nil {
t.Fatal(err)
@@ -313,6 +339,43 @@ func runFilesSeedOrdinaryPreparation(t *testing.T, failure, restart string, moun
}
version = bucket.Put(f.f.bucket, key, encoded.Bytes())
}
+ var deltaKey, deltaVersion string
+ var deltaGood []byte
+ if failure == "restore-checksum" {
+ var spec ltx.FileSpec
+ if _, err := spec.ReadFrom(bytes.NewReader(raw)); err != nil {
+ t.Fatal(err)
+ }
+ checksum := ltx.ChecksumFlag
+ for _, page := range spec.Pages {
+ checksum ^= ltx.ChecksumPage(page.Header.Pgno, page.Data)
+ }
+ checksum |= ltx.ChecksumFlag
+ // Delta trailers cannot validate the resulting database without applying
+ // the snapshot. Keep the envelope valid and fail only the final check.
+ spec.Header.Flags = 0
+ spec.Trailer.PostApplyChecksum = checksum
+ var snapshot bytes.Buffer
+ if _, err := spec.WriteTo(&snapshot); err != nil {
+ t.Fatal(err)
+ }
+ raw = append([]byte(nil), snapshot.Bytes()...)
+ version = bucket.Put(f.f.bucket, key, raw)
+ spec.Header.MinTXID, spec.Header.MaxTXID = 2, 2
+ spec.Header.PreApplyChecksum = checksum
+ var encoded bytes.Buffer
+ if _, err := spec.WriteTo(&encoded); err != nil {
+ t.Fatal(err)
+ }
+ deltaGood = append([]byte(nil), encoded.Bytes()...)
+ encoded.Reset()
+ spec.Trailer.PostApplyChecksum ^= 1
+ if _, err := spec.WriteTo(&encoded); err != nil {
+ t.Fatal(err)
+ }
+ deltaKey = filepath.Dir(filepath.Dir(key)) + "/0000/" + ltx.FormatFilename(2, 2)
+ deltaVersion = bucket.Put(f.f.bucket, deltaKey, encoded.Bytes())
+ }
handler, _ := leasetest.NewS3Server(bucket)
server := httptest.NewServer(handler)
t.Cleanup(server.Close)
@@ -336,19 +399,35 @@ func runFilesSeedOrdinaryPreparation(t *testing.T, failure, restart string, moun
if failure == "invalid-sqlite" {
version = bucket.Put(f.f.bucket, key, raw)
}
+ if failure == "restore-checksum" {
+ deltaVersion = bucket.Put(f.f.bucket, deltaKey, deltaGood)
+ }
f.f.now = f.f.now.Add(time.Minute)
}
p.events = nil
admitted, start := p.prepare(t)
- if len(p.baseline.Files) != 1 || p.baseline.Files[0].Key != key || p.baseline.Files[0].VersionID != version {
+ wantFiles := 1
+ if failure == "restore-checksum" {
+ wantFiles = 2
+ }
+ if len(p.baseline.Files) != wantFiles || p.baseline.Files[0].Key != key || p.baseline.Files[0].VersionID != version {
t.Fatalf("wrong exact baseline: %+v", p.baseline)
}
- if baseline != nil && failure != "invalid-sqlite" && !reflect.DeepEqual(baseline, p.baseline) {
+ if failure == "restore-checksum" && (p.baseline.Files[1].Key != deltaKey || p.baseline.Files[1].VersionID != deltaVersion) {
+ t.Fatalf("wrong exact delta: %+v", p.baseline)
+ }
+ if baseline != nil && failure != "restore-checksum" && failure != "invalid-sqlite" && !reflect.DeepEqual(baseline, p.baseline) {
t.Fatal("baseline changed")
}
baseline = p.baseline
p.enrollFailure = nil
- restoreCtx, cancel := context.WithCancel(t.Context())
+ cancelCtx, cancel := context.WithCancel(t.Context())
+ var restoreCtx context.Context = cancelCtx
+ var outputCancel *servingRestoreOutputCancel
+ if i == 0 && failure == "cancel-after-output" {
+ outputCancel = &servingRestoreOutputCancel{Context: cancelCtx, cancel: cancel, directory: filepath.Dir(f.db)}
+ restoreCtx = outputCancel
+ }
p.afterEnroll = nil
if i == 0 {
switch failure {
@@ -376,6 +455,9 @@ func runFilesSeedOrdinaryPreparation(t *testing.T, failure, restart string, moun
t.Fatal(err)
}
}
+ if outputCancel != nil && !outputCancel.observed.Load() {
+ t.Fatalf("cancellation never observed real restore output: %v", restoreErr)
+ }
cancel()
failed := i == 0 && failure != "" && failure != "missing-marker"
if failed {
@@ -385,12 +467,15 @@ func runFilesSeedOrdinaryPreparation(t *testing.T, failure, restart string, moun
if p.enrollFailure != nil && !errors.Is(restoreErr, p.enrollFailure) {
t.Fatalf("wrong failure: %v", restoreErr)
}
- if failure == "cancel-after-enrollment" && !errors.Is(restoreErr, context.Canceled) {
+ if (failure == "cancel-after-enrollment" || failure == "cancel-after-output") && !errors.Is(restoreErr, context.Canceled) {
t.Fatalf("wrong cancellation: %v", restoreErr)
}
if failure == "invalid-sqlite" && !strings.Contains(restoreErr.Error(), "check restored database") {
t.Fatalf("failure before database check: %v", restoreErr)
}
+ if failure == "restore-checksum" && !strings.Contains(restoreErr.Error(), "restored database checksum") {
+ t.Fatalf("failure before restored checksum: %v", restoreErr)
+ }
if _, err = os.Stat(f.db); !os.IsNotExist(err) {
t.Fatalf("failed restore left database: %v", err)
}
@@ -514,9 +599,40 @@ func TestServingParkNestedBindMount(t *testing.T) {
if !plan14RootFixture(t) {
return
}
- for _, failure := range []string{"", "enrolled-refusal", "missing-marker", "cancel-after-enrollment", "disk-after-enrollment", "invalid-sqlite", "restore-enospc"} {
+ for _, failure := range []string{"", "enrolled-refusal", "missing-marker", "cancel-after-enrollment", "disk-after-enrollment", "invalid-sqlite", "cancel-after-output", "restore-checksum", "restore-enospc"} {
for _, restart := range []string{"same-controller", "controller", "reboot"} {
t.Run(failure+"/"+restart, func(t *testing.T) { runFilesSeedOrdinaryPreparation(t, failure, restart, true) })
}
}
}
+
+// Cancel only once the real restore has written a temporary database. This
+// context seam injects a fault without replacing Restore or its filesystem.
+type servingRestoreOutputCancel struct {
+ context.Context
+ cancel context.CancelFunc
+ directory string
+ observed atomic.Bool
+}
+
+func (c *servingRestoreOutputCancel) Err() error {
+ if !c.observed.Load() {
+ names, _ := filepath.Glob(filepath.Join(c.directory, ".restore-*"))
+ for _, name := range names {
+ if !strings.HasSuffix(name, ".db") && !strings.HasSuffix(name, ".db.tmp") {
+ continue
+ }
+ if info, err := os.Stat(name); err == nil && info.Mode().IsRegular() && info.Size() > 0 {
+ c.observed.Store(true)
+ c.cancel()
+ break
+ }
+ }
+ }
+ return c.Context.Err()
+}
+
+func (c *servingRestoreOutputCancel) Done() <-chan struct{} {
+ _ = c.Err()
+ return c.Context.Done()
+}
diff --git a/internal/host/claimservingpark_rootrunner_linux_test.go b/internal/host/claimservingpark_rootrunner_linux_test.go
new file mode 100644
index 000000000..139330d79
--- /dev/null
+++ b/internal/host/claimservingpark_rootrunner_linux_test.go
@@ -0,0 +1,391 @@
+//go:build linux && rootrunner
+
+package host
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "syscall"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostbox"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/rootstate"
+ "golang.org/x/sys/unix"
+)
+
+func servingRootRunner(t *testing.T) {
+ t.Helper()
+ mapping, err := os.ReadFile("/proc/self/uid_map")
+ if err != nil || os.Geteuid() != 0 || strings.Join(strings.Fields(string(mapping)), " ") != "0 0 4294967295" {
+ t.Fatalf("root runner needs real root in the initial user namespace: %v", err)
+ }
+}
+
+// This includes loopback S3 and Unix-socket fixtures; it makes no remote calls.
+// Installation, peer/census admission and prelaunch cgroups remain test seams.
+func TestRootServingParkNestedBindRestore(t *testing.T) {
+ servingRootRunner(t)
+ if os.Getenv("PLAN14_ROOT_FIXTURE") == t.Name() {
+ if err := unix.Mount("", "/", "", unix.MS_REC|unix.MS_PRIVATE, ""); err != nil {
+ t.Fatal(err)
+ }
+ if !plan14RootFixture(t) {
+ t.Fatal("root child did not enter its scratch root")
+ }
+ for _, failure := range []string{"", "enrolled-refusal", "missing-marker", "cancel-after-enrollment", "disk-after-enrollment", "invalid-sqlite", "cancel-after-output", "restore-checksum", "restore-enospc"} {
+ for _, restart := range []string{"same-controller", "controller", "reboot"} {
+ t.Run(failure+"/"+restart, func(t *testing.T) { runFilesSeedOrdinaryPreparation(t, failure, restart, true) })
+ }
+ }
+ return
+ }
+ root := t.TempDir()
+ for _, name := range []string{"scratch", "proc", "foreign-dir"} {
+ if err := os.Mkdir(filepath.Join(root, name), 0755); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if err := os.WriteFile(filepath.Join(root, "foreign"), nil, 0644); err != nil {
+ t.Fatal(err)
+ }
+ binary, err := os.Executable()
+ if err != nil {
+ t.Fatal(err)
+ }
+ cmd := exec.CommandContext(t.Context(), binary, "-test.run=^"+t.Name()+"$", "-test.v", "-test.timeout=3m")
+ cmd.Env = append(os.Environ(), "PLAN14_ROOT_FIXTURE="+t.Name(), "PLAN14_ROOT_DIR="+root)
+ cmd.SysProcAttr = &syscall.SysProcAttr{Cloneflags: unix.CLONE_NEWNS}
+ if out, err := cmd.CombinedOutput(); err != nil {
+ t.Fatalf("real-root mounted restore: %v\n%s", err, out)
+ } else {
+ t.Log(string(out))
+ }
+}
+
+func servingRunnerMount(t *testing.T, data string) {
+ t.Helper()
+ instances := filepath.Join(data, "instances")
+ if err := os.MkdirAll(instances, 0700); err != nil {
+ t.Fatal(err)
+ }
+ if err := unix.Mount(instances, instances, "", unix.MS_BIND, ""); err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() {
+ if err := unix.Unmount(instances, 0); err != nil {
+ t.Error(err)
+ }
+ })
+ if err := unix.Mount("", instances, "", unix.MS_BIND|unix.MS_REMOUNT|unix.MS_NOSYMFOLLOW|unix.MS_NOSUID|unix.MS_NODEV|unix.MS_NOEXEC, ""); err != nil {
+ t.Fatal(err)
+ }
+}
+
+// Exercise the production filesystem and private store. The authorization is
+// a fixture, so this helper proves no peer, census or installation admission.
+func servingRunnerPark(t *testing.T, data string, state rootstate.Store) servingParkRecord {
+ t.Helper()
+ current := servingArtifactIdentity{Host: "runner.example", Box: "runner", Region: "test", Incarnation: "inc", Location: "instances/runner.example", Generation: "G2", Stamp: 2, Session: 2}
+ db := provision.NestedDBPath(data, current.Host)
+ if err := os.MkdirAll(filepath.Dir(db), 0700); err != nil {
+ t.Fatal(err)
+ }
+ for suffix, body := range map[string]string{"": "database", "-wal": "WAL", "-shm": "SHM"} {
+ if err := os.WriteFile(db+suffix, []byte(body), 0600); err != nil {
+ t.Fatal(err)
+ }
+ }
+ dir, err := os.Open(filepath.Dir(db))
+ if err != nil {
+ t.Fatal(err)
+ }
+ id, err := servingParkDirID(dir)
+ dir.Close()
+ if err != nil {
+ t.Fatal(err)
+ }
+ fs := newServingParkFS(data, state)
+ a := servingParkAuthorization{Successor: current, OriginalSuccessor: current, Directory: id, Legacy: true}
+ a.ID = servingRetirementID(a.Predecessor, current, true)
+ source, err := fs.Capture(a)
+ if err != nil {
+ t.Fatal(err)
+ }
+ r := servingParkRecord{Version: 1, Authorization: a, Source: source, Disposition: servingParkRetain}
+ save := func() error { return (&servingParkLedger{state: state}).write("fixture.json", r) }
+ if err = save(); err != nil {
+ t.Fatal(err)
+ }
+ if err = fs.Prepare(&r, save); err != nil {
+ t.Fatal(err)
+ }
+ if err = fs.Move(&r, save); err != nil {
+ t.Fatal(err)
+ }
+ if err = fs.CheckParked(&r); err != nil {
+ t.Fatal(err)
+ }
+ return r
+}
+
+// This child runs only through the installed test unit below. The production
+// hardening is retained, with a test binary and scratch deployment roots.
+func TestRootServingParkUnitPayload(t *testing.T) {
+ servingRootRunner(t)
+ data, authority := os.Getenv("SERVING_RUNNER_DATA"), os.Getenv("SERVING_RUNNER_AUTHORITY")
+ mode := os.Getenv("SERVING_RUNNER_MODE")
+ if data == "" || authority == "" || (mode != "on" && mode != "off") {
+ t.Fatal("invoke through TestRootServingParkInstalledUnit")
+ }
+ if mode == "off" {
+ if err := os.WriteFile(filepath.Join(data, "must-not-write"), nil, 0600); !errors.Is(err, syscall.EROFS) {
+ t.Fatalf("claims-off data write: %v", err)
+ }
+ if err := os.WriteFile(filepath.Join(authority, "must-not-write"), nil, 0600); !errors.Is(err, syscall.EROFS) {
+ t.Fatalf("claims-off authority write: %v", err)
+ }
+ return
+ }
+ state, err := rootstate.Open(authority, false, false)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer state.Close()
+ r := servingRunnerPark(t, data, state)
+ fs := newServingParkFS(data, state)
+ r.Deleting, r.Cause = true, "expiry"
+ save := func() error { return (&servingParkLedger{state: state}).write("fixture.json", r) }
+ if err := fs.Delete(&r, save); err != nil {
+ t.Fatal(err)
+ }
+ for _, item := range r.Source.Files {
+ if item.Present && !item.Deleted {
+ t.Fatal("installed unit did not delete recorded artifact")
+ }
+ }
+}
+
+func servingRunnerSystemctl(t *testing.T, args ...string) string {
+ t.Helper()
+ ctx, cancel := context.WithTimeout(t.Context(), 45*time.Second)
+ defer cancel()
+ out, err := exec.CommandContext(ctx, "systemctl", args...).CombinedOutput()
+ if err != nil {
+ t.Fatalf("systemctl %v: %v\n%s", args, err, out)
+ }
+ return string(out)
+}
+
+func TestRootServingParkInstalledUnit(t *testing.T) {
+ servingRootRunner(t)
+ pid1, err := os.ReadFile("/proc/1/comm")
+ if err != nil || strings.TrimSpace(string(pid1)) != "systemd" {
+ t.Fatal("root runner needs systemd as PID 1")
+ }
+ root, err := os.MkdirTemp("/var/lib", "carlos-rootrunner-")
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() {
+ if err := os.RemoveAll(root); err != nil {
+ t.Error(err)
+ }
+ })
+ data := filepath.Join(root, "data")
+ opt := filepath.Join(root, "opt")
+ // Create only absent deployment directories and remove only those on exit.
+ for _, path := range []string{data, opt, "/run/carlos-control", "/run/carlos", "/var/lib/carlos-claims"} {
+ if err := os.Mkdir(path, 0700); err == nil {
+ if path == data || path == opt {
+ continue
+ }
+ p := path
+ t.Cleanup(func() {
+ if err := os.Remove(p); err != nil && !errors.Is(err, os.ErrNotExist) {
+ t.Error(err)
+ }
+ })
+ } else if !errors.Is(err, os.ErrExist) {
+ t.Fatal(err)
+ }
+ }
+ authority := filepath.Join("/var/lib/carlos-claims", filepath.Base(root))
+ if err := os.Mkdir(authority, 0700); err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() {
+ if err := os.RemoveAll(authority); err != nil {
+ t.Error(err)
+ }
+ })
+ servingRunnerMount(t, data)
+ binary, err := os.Executable()
+ if err != nil {
+ t.Fatal(err)
+ }
+ body, err := os.ReadFile(binary)
+ if err != nil {
+ t.Fatal(err)
+ }
+ installed := filepath.Join(root, "runner.test")
+ if err := os.WriteFile(installed, body, 0700); err != nil {
+ t.Fatal(err)
+ }
+ raw, err := unitsFS.ReadFile("units/" + claimControlUnit)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, mode := range []string{"off", "on"} {
+ t.Run(mode, func(t *testing.T) {
+ unit := filepath.Base(root) + "-" + mode + ".service"
+ content := claimUnitContent(claimControlUnit, raw, claimUnitSettings{enabled: mode == "on", data: data, opt: opt})
+ var lines []string
+ for _, line := range strings.Split(string(content), "\n") {
+ if strings.HasPrefix(line, "Requires=") || strings.HasPrefix(line, "After=") || strings.HasPrefix(line, "ExecStart=") || strings.HasPrefix(line, "EnvironmentFile=") {
+ continue
+ }
+ lines = append(lines, line)
+ }
+ content = []byte(strings.Join(lines, "\n") + fmt.Sprintf("\nType=oneshot\nTimeoutStartSec=45s\nExecStart=%s -test.run=^TestRootServingParkUnitPayload$ -test.v -test.timeout=30s\nEnvironment=SERVING_RUNNER_DATA=%s SERVING_RUNNER_AUTHORITY=%s SERVING_RUNNER_MODE=%s\nIPAddressDeny=any\n", installed, data, authority, mode))
+ path := filepath.Join("/run/systemd/system", unit)
+ file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0644)
+ if err != nil {
+ t.Fatal(err)
+ }
+ _, writeErr := file.Write(content)
+ closeErr := file.Close()
+ t.Cleanup(func() {
+ ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+ defer cancel()
+ if out, err := exec.CommandContext(ctx, "systemctl", "stop", unit).CombinedOutput(); err != nil {
+ t.Errorf("stop test unit: %v %s", err, out)
+ }
+ if err := os.Remove(path); err != nil {
+ t.Error(err)
+ }
+ if out, err := exec.CommandContext(ctx, "systemctl", "daemon-reload").CombinedOutput(); err != nil {
+ t.Errorf("reload after cleanup: %v %s", err, out)
+ }
+ })
+ if err := errors.Join(writeErr, closeErr); err != nil {
+ t.Fatal(err)
+ }
+ servingRunnerSystemctl(t, "daemon-reload")
+ servingRunnerSystemctl(t, "start", unit)
+ if got := strings.TrimSpace(servingRunnerSystemctl(t, "show", unit, "--property=Result", "--value")); got != "success" {
+ t.Fatal(got)
+ }
+ if got := strings.TrimSpace(servingRunnerSystemctl(t, "show", unit, "--property=ExecMainStatus", "--value")); got != "0" {
+ t.Fatal(got)
+ }
+ })
+ }
+}
+
+const servingRebootRoot = "/var/lib/carlos-rootrunner-serving-reboot"
+
+type servingRebootWitness struct {
+ Boot string
+ Record servingParkRecord
+}
+
+// The runner reboots between these phases. This currently proves refusal and
+// byte preservation on changed topology; automatic reboot recovery is not built.
+func TestRootRebootServingParkChangedMountPhase1(t *testing.T) {
+ servingRootRunner(t)
+ if err := os.Mkdir(servingRebootRoot, 0700); err != nil {
+ t.Fatal(err)
+ }
+ data := filepath.Join(servingRebootRoot, "data")
+ servingRunnerMount(t, data)
+ state, err := rootstate.Open(filepath.Join(servingRebootRoot, "authority"), true, false)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer state.Close()
+ r := servingRunnerPark(t, data, state)
+ boot, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := (&servingParkLedger{state: state}).write("reboot.json", servingRebootWitness{Boot: strings.TrimSpace(string(boot)), Record: r}); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func TestRootRebootServingParkChangedMountPhase2(t *testing.T) {
+ servingRootRunner(t)
+ state, err := rootstate.Open(filepath.Join(servingRebootRoot, "authority"), false, false)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer state.Close()
+ body, err := state.Read("reboot.json")
+ if err != nil {
+ t.Fatal(err)
+ }
+ var saved servingRebootWitness
+ if err := json.Unmarshal(body, &saved); err != nil {
+ t.Fatal(err)
+ }
+ boot, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
+ if err != nil || saved.Boot == "" || strings.TrimSpace(string(boot)) == saved.Boot {
+ t.Fatal("phase 2 requires an actual intervening reboot")
+ }
+ t.Cleanup(func() {
+ if err := os.RemoveAll(servingRebootRoot); err != nil {
+ t.Error(err)
+ }
+ })
+ data := filepath.Join(servingRebootRoot, "data")
+ // Mount IDs can be reused across boots. Add bind layers until this fixture
+ // has a different ID, instead of assuming reboot alone changes the number.
+ fs := newServingParkFS(data, state)
+ for i := 0; i < 8; i++ {
+ servingRunnerMount(t, data)
+ parent, id, err := fs.parent()
+ if err != nil {
+ t.Fatal(err)
+ }
+ parent.Close()
+ if id.Mount != saved.Record.Source.Parent.Mount {
+ break
+ }
+ if i == 7 {
+ t.Fatal("could not create changed mount topology")
+ }
+ }
+ before, err := state.Read("namespace.json")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := fs.CheckParked(&saved.Record); !errors.Is(err, hostbox.ErrChanged) {
+ t.Fatalf("old mount authority reused: %v", err)
+ }
+ r := saved.Record
+ r.Deleting, r.Cause = true, "expiry"
+ writes := 0
+ if err := fs.Delete(&r, func() error { writes++; return nil }); !errors.Is(err, hostbox.ErrChanged) || writes != 0 {
+ t.Fatalf("stale mount deleted data: writes=%d err=%v", writes, err)
+ }
+ after, err := state.Read("namespace.json")
+ if err != nil || !bytes.Equal(before, after) {
+ t.Fatal("refusal changed namespace authority", err)
+ }
+ for suffix, want := range map[string]string{"": "database", "-wal": "WAL", "-shm": "SHM"} {
+ path := filepath.Join(data, "instances", ".serving-parked", r.Authorization.ID, r.Authorization.Successor.Host+".db"+suffix)
+ got, err := os.ReadFile(path)
+ if err != nil || string(got) != want {
+ t.Fatalf("reboot lost retained %s: %q %v", suffix, got, err)
+ }
+ }
+}
diff --git a/internal/host/plan12_restore_linux_test.go b/internal/host/plan12_restore_linux_test.go
index ef479bea6..ea171df75 100644
--- a/internal/host/plan12_restore_linux_test.go
+++ b/internal/host/plan12_restore_linux_test.go
@@ -369,7 +369,7 @@ func (in plan12ReturnInputs) Select(ctx context.Context, rt registry.Route, o ho
return selected, err
}
-func plan12RestoreBaseline(t *testing.T, f *recoveryFixture) *genselect.Plan {
+func plan12RestoreBaseline(t *testing.T, f *recoveryFixture, extraBytes ...int) *genselect.Plan {
t.Helper()
path := filepath.Join(t.TempDir(), "selected.db")
db, err := sql.Open("sqlite", path)
@@ -379,6 +379,11 @@ func plan12RestoreBaseline(t *testing.T, f *recoveryFixture) *genselect.Plan {
if _, err := db.Exec("CREATE TABLE fixture(value TEXT); INSERT INTO fixture VALUES ('selected recovery baseline')"); err != nil {
t.Fatal(err)
}
+ if len(extraBytes) > 0 && extraBytes[0] > 0 {
+ if _, err := db.Exec("CREATE TABLE padding(value BLOB); INSERT INTO padding VALUES (zeroblob(?))", extraBytes[0]); err != nil {
+ t.Fatal(err)
+ }
+ }
if err := db.Close(); err != nil {
t.Fatal(err)
}
diff --git a/internal/hostbox/restore.go b/internal/hostbox/restore.go
index b0b024668..46df73186 100644
--- a/internal/hostbox/restore.go
+++ b/internal/hostbox/restore.go
@@ -362,6 +362,24 @@ func loadPinned(ctx context.Context, reader genselect.Reader, bucket string, pla
if uint64(h.MinTXID) != selected.MinTXID || uint64(h.MaxTXID) != selected.MaxTXID {
return nil, 0, fmt.Errorf("selected %s range mismatch", selected.Key)
}
+ // Litestream emits HeaderFlagNoChecksum but copies the last input's
+ // database checksum. Re-encode one page at a time for that format,
+ // retaining the original checksum for the final restored database.
+ // Nothing uses the converted bytes until the original proof passes.
+ var encoded bytes.Buffer
+ var enc *ltx.Encoder
+ if !h.NoChecksum() {
+ enc, err = ltx.NewEncoder(&encoded)
+ if err != nil {
+ return nil, 0, err
+ }
+ normalized := h
+ normalized.Flags |= ltx.HeaderFlagNoChecksum
+ normalized.PreApplyChecksum = 0
+ if err = enc.EncodeHeader(normalized); err != nil {
+ return nil, 0, err
+ }
+ }
buf := make([]byte, h.PageSize)
var ph ltx.PageHeader
for {
@@ -372,6 +390,11 @@ func loadPinned(ctx context.Context, reader genselect.Reader, bucket string, pla
if err != nil {
return nil, 0, fmt.Errorf("decode %s: %w", selected.Key, err)
}
+ if enc != nil {
+ if err := enc.EncodePage(ph, buf); err != nil {
+ return nil, 0, err
+ }
+ }
}
if err := dec.Close(); err != nil {
return nil, 0, fmt.Errorf("verify %s trailer: %w", selected.Key, err)
@@ -380,6 +403,12 @@ func loadPinned(ctx context.Context, reader genselect.Reader, bucket string, pla
if i == 0 && plan.Pin != nil && plan.Pin.SnapshotCRC64 != 0 && uint64(t.FileChecksum) != plan.Pin.SnapshotCRC64 {
return nil, 0, genselect.ErrPinnedMismatch
}
+ if enc != nil {
+ if err := enc.Close(); err != nil {
+ return nil, 0, err
+ }
+ data = encoded.Bytes()
+ }
info := &ltx.FileInfo{Level: selected.Level, MinTXID: h.MinTXID, MaxTXID: h.MaxTXID, PreApplyChecksum: h.PreApplyChecksum, PostApplyChecksum: t.PostApplyChecksum, Size: int64(len(data))}
files = append(files, pinnedFile{plan: selected, data: data, info: info, infoPageSize: int(h.PageSize)})
last, previous = t.PostApplyChecksum, selected.MaxTXID
diff --git a/internal/hostbox/restore_test.go b/internal/hostbox/restore_test.go
index 0f9b4d943..d67dbbb70 100644
--- a/internal/hostbox/restore_test.go
+++ b/internal/hostbox/restore_test.go
@@ -346,3 +346,77 @@ func TestGenerationMarkerIsSharedWithTheColdRestore(t *testing.T) {
t.Fatalf("a refused write changed the marker: %q", got)
}
}
+
+func TestLoadPinnedNormalizesOnlyAfterOriginalProof(t *testing.T) {
+ key, original, _ := snapshotFixture(t, "checksummed")
+ var spec ltx.FileSpec
+ if _, err := spec.ReadFrom(bytes.NewReader(original)); err != nil {
+ t.Fatal(err)
+ }
+ checksum := ltx.Checksum(0)
+ for _, page := range spec.Pages {
+ checksum ^= ltx.ChecksumPage(page.Header.Pgno, page.Data)
+ }
+ checksum |= ltx.ChecksumFlag
+ spec.Header.Flags = 0
+ spec.Trailer.PostApplyChecksum = checksum
+ var encoded bytes.Buffer
+ if _, err := spec.WriteTo(&encoded); err != nil {
+ t.Fatal(err)
+ }
+ original = append([]byte(nil), encoded.Bytes()...)
+ for _, fault := range []string{"valid", "pin", "file-checksum", "database-checksum"} {
+ t.Run(fault, func(t *testing.T) {
+ body := append([]byte(nil), original...)
+ pin := uint64(spec.Trailer.FileChecksum)
+ switch fault {
+ case "pin":
+ pin ^= 1
+ case "file-checksum":
+ body[len(body)-1] ^= 1
+ case "database-checksum":
+ bad := spec
+ bad.Trailer.PostApplyChecksum ^= 1
+ var out bytes.Buffer
+ if _, err := bad.WriteTo(&out); err != nil {
+ t.Fatal(err)
+ }
+ body, pin = out.Bytes(), uint64(bad.Trailer.FileChecksum)
+ }
+ reader := &restoreReader{data: map[string][]byte{key + "@selected": body}}
+ plan := &genselect.Plan{Generation: "one", MaxTXID: 1, Pin: &genselect.Pin{SnapshotCRC64: pin}, Files: []genselect.File{{Key: key, VersionID: "selected", Level: 9, MinTXID: 1, MaxTXID: 1, Size: int64(len(body))}}}
+ files, final, err := loadPinned(t.Context(), reader, "data", plan)
+ if fault != "valid" {
+ if err == nil {
+ t.Fatal("normalization bypassed original proof")
+ }
+ return
+ }
+ if err != nil {
+ t.Fatal(err)
+ }
+ if final != checksum || len(files) != 1 || !bytes.Equal(reader.data[key+"@selected"], original) {
+ t.Fatal("normalization changed selected bytes or lost database checksum")
+ }
+ var converted ltx.FileSpec
+ if _, err := converted.ReadFrom(bytes.NewReader(files[0].data)); err != nil {
+ t.Fatal(err)
+ }
+ if !converted.Header.NoChecksum() || converted.Header.PreApplyChecksum != 0 || converted.Trailer.PostApplyChecksum != 0 {
+ t.Fatal("incompatible restore stream")
+ }
+ if len(converted.Pages) != len(spec.Pages) {
+ t.Fatal("page loss")
+ }
+ for i := range converted.Pages {
+ if converted.Pages[i].Header != spec.Pages[i].Header || !bytes.Equal(converted.Pages[i].Data, spec.Pages[i].Data) {
+ t.Fatal("normalization changed database pages")
+ }
+ }
+ path := filepath.Join(t.TempDir(), "restored.db")
+ if err := (ColdRestore{Reader: reader, Bucket: "data", Generation: "successor", Work: &restoreWork{}}).Install(t.Context(), path, plan); err != nil {
+ t.Fatal(err)
+ }
+ })
+ }
+}
diff --git a/internal/provision/testdata/plan14-root-reads-seed.tsv b/internal/provision/testdata/plan14-root-reads-seed.tsv
index 28581a042..d48ae235b 100644
--- a/internal/provision/testdata/plan14-root-reads-seed.tsv
+++ b/internal/provision/testdata/plan14-root-reads-seed.tsv
@@ -1,4 +1,4 @@
-# Files seed checkpoint 7 overlay; pending Paul’s census ruling.
+# Files seed checkpoint 8 overlay; pending Paul’s census ruling.
# Generation, legacy and empty-bootstrap root authority have test callers only; production binding remains refused.
# Load after zddr4 and its explicit seal check. No legacy fingerprints are added or removed.
# Every classification below is rev3=PROPOSED-seed, not an approved disposition.
@@ -76,7 +76,7 @@ internal/host/claimservingpark_records.go servingParkRecord.activeAuthorization
internal/host/claimservingpark_records.go servingParkLedger.continueLocked authority C,J4 rev3=PROPOSED-seed; namespace/host-protected slot and record locks; pending original retirement, absent source and retained parked identities before durable bounded continuation 87f297a924f7f153624304233218d8b5548a016fc7dbf0f7cb8dedb789324009
internal/host/claimservingpark_records.go servingParkLedger.replaceLocked authority C,J4 rev3=PROPOSED-seed; completed incumbent record under its record lock; resumes its recorded deletion cause before slot replacement 3e67316e959c9a901c32a2a61fda4e6ea5e92fe51bdb7fdb5f5c53d9d5960aad
@coverage internal/host 2d65820f0366a1ee92490ef9af8888cfda33292f5a5367d35d353e320f031a85
-@coverage internal/hostbox d48f1a63963edb2551230477c8162f7ac182761c80571c3d753fe19dd7acacba
+@coverage internal/hostbox 4b68d7fc25b29963dcf1b7fe92b047fd2229515d83e5cf424c0835d6e690859d
@coverage internal/hostguard 94912485e1e5fad4bad56985dd48ede776cfcb5fe18e0f4a59953584ddab06c0
internal/host/claimservinglegacy.go servingLegacyKey authority C,J4 rev3=PROPOSED-seed; box, host and incarnation derive private stop and consumption keys; never a caller path 93c10e40202ab23986b54a366edd7abfab9f8d942736305b5732e8741e96f3ab
internal/host/claimservinglegacy.go validateServingLegacyStop authority C,J4 rev3=PROPOSED-seed; original session-zero serving observation and exact root no-process bytes; claim ETag, receipt digest, kernel evidence and empty fence identity file:Exec@413 88bf4d578819b8d3f62b80e2068a4c574a1aaa8118a8461bbeee99569e035ed0
@@ -121,3 +121,4 @@ internal/hostbox/servingpark_transport.go servingParkResponse authority C,J4 rev
internal/hostbox/servingpark_transport.go validateServingParkRequest authority C,J4 rev3=PROPOSED-seed; v4 envelope, nonce, digest and identities; rejects mixed operation fields and unsupported pressure before any effect 2c87ae130afc670925c87f52b9688fdc24d940d0a5fcdbf09667df2b662f7f59
internal/host/claimservingpark_transport.go var openClaimServingParkAuthority authority C,J4 rev3=PROPOSED-seed; descriptor-relative private rootstate at /var/lib/carlos-claims/authority/serving-parks with creation enabled and public reads disabled file:Open@78 7a1db088a7240e6a65ba0bed96fc5f973a69f70c2273c4475c486a6e0359fb23
internal/host/claimwakestop.go ClaimAssembly.collectCapturedStop authority C,G,J4 rev3=PROPOSED-seed; fresh claim and verified local stop identity; session-zero legacy holder may name only the local writer box, with no admissions; root independently proves and retains no-process evidence file:Exec@2994,file:Read@605 d4cd80663dfae5f9737a74e319d3423f4ef853e1843666b02b4cd2341e869d72
+internal/hostbox/restore.go loadPinned authority C,J2,J4 rev3=PROPOSED-seed; exact selected LTX versions and original trailer/pin proof; validated bytes normalized for Litestream while retaining original database checksum for final comparison file:ReadAll@662 1e2b22c7c501d31cd1f2cb3be0fe419fa7e25afde87c6ecb43ca85edb7dbc3a4