WIP files seed checkpoint 2 (not for merge): serving retirement and enrollment contract, launch-evidence checks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9 files changed,
+1483
−3
internal/host/claimservingartifact.go+5 −0internal/host/claimservingartifact_linux.go+41 −0internal/host/claimservingartifact_other.go+9 −0internal/host/claimservingpark.go+69 −0internal/host/claimservingpark_test.go+79 −0internal/host/claimservingretire.go+477 −0internal/host/claimservingretire_linux_test.go+767 −0internal/hostbox/servingpark.go+12 −0internal/provision/testdata/plan14-root-reads-seed.tsv+24 −3
CI passed — run details
CI log
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok amadan.net/carlos/platform.git/launchctx 0.011s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok github.com/carlosframework/platform 342.506s
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.046s
ok github.com/carlosframework/platform/cmd/moveprobe 92.292s
ok github.com/carlosframework/platform/cmd/zddprobe 0.705s
ok github.com/carlosframework/platform/internal/activator 233.545s
ok github.com/carlosframework/platform/internal/agentsock 0.025s
ok github.com/carlosframework/platform/internal/alerting 0.007s
ok github.com/carlosframework/platform/internal/analytics 0.024s
ok github.com/carlosframework/platform/internal/appconfig 2.382s
ok github.com/carlosframework/platform/internal/apperr 0.083s
ok github.com/carlosframework/platform/internal/applog 0.191s
ok github.com/carlosframework/platform/internal/awsinv 0.006s
ok github.com/carlosframework/platform/internal/boxchoice 0.009s
ok github.com/carlosframework/platform/internal/boxid 0.546s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.011s
ok github.com/carlosframework/platform/internal/caps 0.020s
ok github.com/carlosframework/platform/internal/certwarm 0.003s
ok github.com/carlosframework/platform/internal/cgroup 0.695s
ok github.com/carlosframework/platform/internal/changelog 0.344s
ok github.com/carlosframework/platform/internal/clamd 0.058s
? github.com/carlosframework/platform/internal/cliflag [no test files]
ok github.com/carlosframework/platform/internal/console 367.025s
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 5.973s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.009s
ok github.com/carlosframework/platform/internal/dbverify 0.006s
ok github.com/carlosframework/platform/internal/dnsrec 0.003s
ok github.com/carlosframework/platform/internal/docsite 0.005s
ok github.com/carlosframework/platform/internal/domains 2.133s
ok github.com/carlosframework/platform/internal/doorkey 0.006s
ok github.com/carlosframework/platform/internal/economics 0.005s
ok github.com/carlosframework/platform/internal/edge 26.115s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.009s
ok github.com/carlosframework/platform/internal/fleetapi 0.128s
ok github.com/carlosframework/platform/internal/gate 1.656s
ok github.com/carlosframework/platform/internal/genselect 0.019s
ok github.com/carlosframework/platform/internal/geo 0.028s
ok github.com/carlosframework/platform/internal/geoip 0.031s
ok github.com/carlosframework/platform/internal/host 127.787s
ok github.com/carlosframework/platform/internal/hostactors 0.332s
ok github.com/carlosframework/platform/internal/hostbox 2.176s
ok github.com/carlosframework/platform/internal/hostclaim 0.497s
ok github.com/carlosframework/platform/internal/hostevidence 0.005s
ok github.com/carlosframework/platform/internal/hostexec 4.877s
ok github.com/carlosframework/platform/internal/hostfence 0.031s
ok github.com/carlosframework/platform/internal/hostfiles 10.538s
ok github.com/carlosframework/platform/internal/hostfloor 0.012s
ok github.com/carlosframework/platform/internal/hostguard 0.972s
ok github.com/carlosframework/platform/internal/hostops 1.431s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.002s
ok github.com/carlosframework/platform/internal/hostquarantine 0.002s
ok github.com/carlosframework/platform/internal/hostrelocate 0.445s
ok github.com/carlosframework/platform/internal/hostresolve 0.144s
ok github.com/carlosframework/platform/internal/hostwrite 0.002s
ok github.com/carlosframework/platform/internal/infraview 0.002s
ok github.com/carlosframework/platform/internal/instancectl 1.819s
ok github.com/carlosframework/platform/internal/instanceexec 2.745s
ok github.com/carlosframework/platform/internal/instancefs 0.004s
ok github.com/carlosframework/platform/internal/instancekey 0.017s
ok github.com/carlosframework/platform/internal/instanceupdates 0.026s
ok github.com/carlosframework/platform/internal/layout 1.424s
ok github.com/carlosframework/platform/internal/lease 9.450s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.020s
ok github.com/carlosframework/platform/internal/ledger 0.011s
ok github.com/carlosframework/platform/internal/mail 0.154s
ok github.com/carlosframework/platform/internal/maintenance 0.077s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.550s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.077s
ok github.com/carlosframework/platform/internal/metrics 0.007s
ok github.com/carlosframework/platform/internal/moves 21.110s
ok github.com/carlosframework/platform/internal/moves/archive 0.756s
ok github.com/carlosframework/platform/internal/objstore 0.165s
ok github.com/carlosframework/platform/internal/ops 0.161s
ok github.com/carlosframework/platform/internal/park 0.210s
ok github.com/carlosframework/platform/internal/privatefile 0.006s
ok github.com/carlosframework/platform/internal/provision 13.206s
ok github.com/carlosframework/platform/internal/publicrelease 0.012s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 4.062s
ok github.com/carlosframework/platform/internal/relay 7.235s
ok github.com/carlosframework/platform/internal/release 20.211s
ok github.com/carlosframework/platform/internal/rollout 3.756s
ok github.com/carlosframework/platform/internal/rolloutwords 0.005s
ok github.com/carlosframework/platform/internal/rootaws 0.007s
ok github.com/carlosframework/platform/internal/rootparticipation 0.007s
ok github.com/carlosframework/platform/internal/rootstate 0.024s
ok github.com/carlosframework/platform/internal/s3compat 0.004s
ok github.com/carlosframework/platform/internal/safefile 0.003s
ok github.com/carlosframework/platform/internal/schedule 2.301s
ok github.com/carlosframework/platform/internal/sim/hostlife 9.478s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.434s
ok github.com/carlosframework/platform/internal/status 0.045s
ok github.com/carlosframework/platform/internal/steering 0.005s
ok github.com/carlosframework/platform/internal/stopevidence 0.227s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.002s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.004s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.573s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
minio: downloaded, digest verified
mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
mc: downloaded, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.03s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.02s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.030s
=== RUN TestT4StorageProbeMinio
2026/10/14 06:28:10 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DE517AF267171C, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.00s)
PASS
ok github.com/carlosframework/platform/internal/host 0.021s
minio contract: PASS
diff --git a/internal/host/claimservingartifact.go b/internal/host/claimservingartifact.go| new file mode 100644 |
| index 000000000..008c0cda9 |
| --- /dev/null |
| +++ b/internal/host/claimservingartifact.go |
| @@ -0,0 +1,5 @@ |
| +package host |
| + |
| +type servingArtifactDirectory struct { |
| + Device, Inode, Mount uint64 |
| +} |
diff --git a/internal/host/claimservingartifact_linux.go b/internal/host/claimservingartifact_linux.go| new file mode 100644 |
| index 000000000..54b22c879 |
| --- /dev/null |
| +++ b/internal/host/claimservingartifact_linux.go |
| @@ -0,0 +1,41 @@ |
| +//go:build linux |
| + |
| +package host |
| + |
| +import ( |
| + "errors" |
| + "path/filepath" |
| + |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/instancefs" |
| + "golang.org/x/sys/unix" |
| +) |
| + |
| +// checkServingArtifactAbsence checks the root-recorded source directory and |
| +// all four source names through one descriptor. Missing parents, links and |
| +// replacement directories are errors, never evidence of absent artifacts. |
| +func checkServingArtifactAbsence(db string, expected servingArtifactDirectory) error { |
| + dir, err := instancefs.OpenDirBeneath(filepath.Dir(db), ".") |
| + if err != nil { |
| + return err |
| + } |
| + defer dir.Close() |
| + var stat unix.Statx_t |
| + if err = unix.Statx(int(dir.Fd()), "", unix.AT_EMPTY_PATH|unix.AT_SYMLINK_NOFOLLOW, unix.STATX_BASIC_STATS|unix.STATX_MNT_ID, &stat); err != nil { |
| + return err |
| + } |
| + if stat.Mask&unix.STATX_MNT_ID == 0 || unix.Mkdev(stat.Dev_major, stat.Dev_minor) != expected.Device || stat.Ino != expected.Inode || stat.Mnt_id != expected.Mount { |
| + return hostbox.ErrChanged |
| + } |
| + for _, suffix := range []string{"", ".gen", "-wal", "-shm"} { |
| + var item unix.Stat_t |
| + err = unix.Fstatat(int(dir.Fd()), filepath.Base(db)+suffix, &item, unix.AT_SYMLINK_NOFOLLOW) |
| + if err == nil { |
| + return hostbox.ErrChanged |
| + } |
| + if !errors.Is(err, unix.ENOENT) { |
| + return err |
| + } |
| + } |
| + return nil |
| +} |
diff --git a/internal/host/claimservingartifact_other.go b/internal/host/claimservingartifact_other.go| new file mode 100644 |
| index 000000000..a69c868f3 |
| --- /dev/null |
| +++ b/internal/host/claimservingartifact_other.go |
| @@ -0,0 +1,9 @@ |
| +//go:build !linux |
| + |
| +package host |
| + |
| +import "github.com/carlosframework/platform/internal/hostbox" |
| + |
| +func checkServingArtifactAbsence(string, servingArtifactDirectory) error { |
| + return hostbox.ErrUnsupported |
| +} |
diff --git a/internal/host/claimservingpark.go b/internal/host/claimservingpark.go| new file mode 100644 |
| index 000000000..a1469ae54 |
| --- /dev/null |
| +++ b/internal/host/claimservingpark.go |
| @@ -0,0 +1,69 @@ |
| +package host |
| + |
| +import ( |
| + "context" |
| + |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostguard" |
| +) |
| + |
| +type servingParkDisposition uint8 |
| + |
| +const ( |
| + servingParkRefuse servingParkDisposition = iota |
| + servingParkArtifactFree |
| + servingParkDiscard |
| + servingParkRetain |
| +) |
| + |
| +// servingGenerationParkDisposition is a root-side decision made after source |
| +// capture and stop authentication. The database must exist to own the slot. |
| +// A prelaunch record alone proves no tenant started. Once a main-role launch |
| +// was intended, missing start evidence is ambiguous: StartedUnit is recorded |
| +// after the process may have run. Preserve that source and refuse retirement. |
| +// Task 0R's filesystem driver must persist this decision with its source |
| +// identities before either replacing the slot or discarding a restore. |
| +func servingGenerationParkDisposition(ctx context.Context, journal *hostbox.LaunchJournal, prior servingArtifactIdentity, receipts []hostguard.FenceReceipt, databasePresent bool) (servingParkDisposition, error) { |
| + if ctx == nil || journal == nil || prior.Host == "" || prior.Box == "" || prior.Region == "" || prior.Incarnation == "" || prior.Generation == "" || prior.Stamp == 0 || prior.Session == 0 || len(receipts) == 0 { |
| + return servingParkRefuse, hostbox.ErrChanged |
| + } |
| + launched := false |
| + for _, receipt := range receipts { |
| + t := receipt.Target |
| + if t.Host != prior.Host || t.Box != prior.Box || t.Region != prior.Region || t.Incarnation != prior.Incarnation || t.Stamp != prior.Stamp || t.Session != prior.Session || t.Op != "" || t.Phase != "" || t.ObligationID != "" || verifyRetirementReceipt(ctx, receipt) != nil { |
| + return servingParkRefuse, ErrClaimStopReceipt |
| + } |
| + // Artifact-free retirement never replaces or refreshes an existing park. |
| + if !databasePresent { |
| + continue |
| + } |
| + records, err := journal.RootSessionLaunches(ctx, prior.Host, prior.Incarnation, prior.Stamp, prior.Session, prior.Box, t.Enrollment) |
| + if err != nil { |
| + return servingParkRefuse, err |
| + } |
| + if len(records) != 1 || !hostbox.SameLaunchOwner(records[0].Target, t) { |
| + return servingParkRefuse, hostbox.ErrChanged |
| + } |
| + seen := make(map[string]bool) |
| + for _, reservation := range records[0].Reservations { |
| + if reservation.Role != "main" { |
| + continue |
| + } |
| + if reservation.ID == "" || seen[reservation.ID] || !hostbox.SameLaunchOwner(reservation.Target, t) { |
| + return servingParkRefuse, hostbox.ErrChanged |
| + } |
| + seen[reservation.ID] = true |
| + if !hostbox.IsUnitReservation(reservation) || reservation.State != hostbox.UnitSettled || reservation.PID <= 0 || reservation.ProcessStart == "" || reservation.Group.InvocationID == "" || reservation.Group.Device == 0 || reservation.Group.Inode == 0 || reservation.Executable.Device == 0 || reservation.Executable.Inode == 0 { |
| + return servingParkRefuse, hostbox.ErrChanged |
| + } |
| + launched = true |
| + } |
| + } |
| + if !databasePresent { |
| + return servingParkArtifactFree, nil |
| + } |
| + if launched { |
| + return servingParkRetain, nil |
| + } |
| + return servingParkDiscard, nil |
| +} |
diff --git a/internal/host/claimservingpark_test.go b/internal/host/claimservingpark_test.go| new file mode 100644 |
| index 000000000..e7bfa2733 |
| --- /dev/null |
| +++ b/internal/host/claimservingpark_test.go |
| @@ -0,0 +1,79 @@ |
| +//go:build linux |
| + |
| +package host |
| + |
| +import ( |
| + "context" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostguard" |
| + "github.com/carlosframework/platform/internal/instancectl" |
| +) |
| + |
| +func TestServingParkRequiresRootObservedTenantStart(t *testing.T) { |
| + for _, state := range []string{"prelaunch", "intended", "started", "settled-unobserved", "settled-started", "tombstoned", "missing", "remote"} { |
| + for _, database := range []bool{true, false} { |
| + t.Run(state+map[bool]string{true: "/database", false: "/no-database"}[database], func(t *testing.T) { |
| + f, _, _, in := servingRetirementFixture(t) |
| + admitted := servingAdmit(t, f, &in, "G") |
| + receipt := servingStop(t, f, admitted) |
| + target := receipt.Target |
| + if state != "missing" { |
| + if err := f.journal.Prelaunch(t.Context(), hostbox.LaunchInventory{Target: target, ControllerPID: 77, ProcessStart: "controller-start", NoState: true, Groups: []hostbox.LaunchGroup{{RelativePath: "carlos/parent", Device: 1, Inode: 2}}}); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + reservation := hostbox.LaunchReservation{ID: "attempt:main", Role: "main", InvocationID: "attempt-invocation", Inputs: "inputs", Target: target, Group: hostbox.LaunchGroup{RelativePath: instancectl.ClaimUnitControlGroup(target.Host)}, Executable: hostbox.LaunchFile{Device: 1, Inode: 3}} |
| + switch state { |
| + case "intended", "started", "settled-unobserved", "settled-started": |
| + if err := f.journal.IntendUnit(t.Context(), reservation); err != nil { |
| + t.Fatal(err) |
| + } |
| + case "tombstoned": |
| + if err := f.journal.TombstoneUnit(t.Context(), reservation); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if state == "started" || state == "settled-started" { |
| + group := hostbox.LaunchGroup{RelativePath: reservation.Group.RelativePath, Device: 1, Inode: 4, InvocationID: "unit-invocation"} |
| + if err := f.journal.StartedUnit(t.Context(), reservation, group, 99, "tenant-start"); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if state == "settled-started" || state == "settled-unobserved" { |
| + if err := f.journal.SettleUnit(t.Context(), target, reservation.ID); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + journal := f.journal |
| + remoteCalls := 0 |
| + if state == "remote" { |
| + journal, _ = hostbox.RemoteJournals(func(context.Context, hostbox.JournalRequest) (hostbox.JournalResponse, error) { |
| + remoteCalls++ |
| + return hostbox.JournalResponse{}, nil |
| + }) |
| + } |
| + prior, err := servingWriterIdentity(admitted, "box-1", "ie") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + got, err := servingGenerationParkDisposition(t.Context(), journal, prior, []hostguard.FenceReceipt{receipt}, database) |
| + want := servingParkRefuse |
| + if !database { |
| + want = servingParkArtifactFree |
| + } else if state == "prelaunch" { |
| + want = servingParkDiscard |
| + } else if state == "settled-started" { |
| + want = servingParkRetain |
| + } |
| + if got != want || (err != nil) != (want == servingParkRefuse) { |
| + t.Fatalf("got %v,%v want %v", got, err, want) |
| + } |
| + if remoteCalls != 0 { |
| + t.Fatal("root retirement trusted agent journal") |
| + } |
| + }) |
| + } |
| + } |
| +} |
diff --git a/internal/host/claimservingretire.go b/internal/host/claimservingretire.go| new file mode 100644 |
| index 000000000..540e63a85 |
| --- /dev/null |
| +++ b/internal/host/claimservingretire.go |
| @@ -0,0 +1,477 @@ |
| +package host |
| + |
| +import ( |
| + "context" |
| + "crypto/sha256" |
| + "encoding/hex" |
| + "encoding/json" |
| + "path/filepath" |
| + "sort" |
| + "strings" |
| + "sync" |
| + |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/hostguard" |
| + "github.com/carlosframework/platform/internal/hostwrite" |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// These selectors contain no local paths. Root must independently resolve the |
| +// installation from its state inventory before returning an authorization. |
| +type servingArtifactIdentity struct { |
| + Host, Box, Region, Incarnation, Location, Generation string |
| + Stamp, Session uint64 |
| +} |
| + |
| +type servingLegacyPredecessor struct { |
| + Session uint64 |
| + Generation string |
| + Empty bool |
| +} |
| + |
| +type servingParkAuthorization struct { |
| + TagsFrom uint64 |
| + ID string |
| + Legacy bool |
| + Predecessor servingArtifactIdentity |
| + OriginalSuccessor servingArtifactIdentity |
| + Successor servingArtifactIdentity |
| + Directory servingArtifactDirectory |
| +} |
| + |
| +// A zero predecessor asks Park to resume an existing record for successor; |
| +// it must never authorize a new retirement. Complete independently rereads |
| +// the claim's paired tags. Neither call may run while Work is held. |
| +type servingDatabaseParker interface { |
| + Park(context.Context, servingArtifactIdentity, servingArtifactIdentity) (servingParkAuthorization, error) |
| + ParkLegacy(context.Context, servingArtifactIdentity, servingLegacyPredecessor) (servingParkAuthorization, error) |
| + Complete(context.Context, string, servingArtifactIdentity) error |
| +} |
| + |
| +type claimServingArtifactRetirement struct { |
| + Claims release.Store // The assembly's authorized claim writer, never a new bucket client. |
| + Guard hostguard.Guard |
| + DataDir, Box, Region string |
| + Root servingDatabaseParker |
| + mu sync.Mutex |
| + completed map[string]servingParkAuthorization |
| +} |
| + |
| +func servingRetirementID(prior, original servingArtifactIdentity, legacy bool) string { |
| + body, _ := json.Marshal(struct { |
| + Prior, Successor servingArtifactIdentity |
| + Legacy bool |
| + }{prior, original, legacy}) |
| + sum := sha256.Sum256(body) |
| + return hex.EncodeToString(sum[:]) |
| +} |
| + |
| +func servingWriterIdentity(o hostclaim.Observed, box, region string) (servingArtifactIdentity, error) { |
| + if o.Absent || o.Host == "" || o.Claim == nil || o.Claim.Host != o.Host || o.Claim.Holder == nil || box == "" || region == "" { |
| + return servingArtifactIdentity{}, hostbox.ErrChanged |
| + } |
| + c, h := o.Claim, o.Claim.Holder |
| + if c.Op.Kind != hostclaim.KindServing || c.Op.ID != "" || c.Op.Phase != "" || c.Op.Exec != box || h.WriterBox != box || h.Incarnation == "" || h.Location == "" || h.Generation == "" || h.Session == 0 || h.Stamp != h.Session { |
| + return servingArtifactIdentity{}, hostbox.ErrChanged |
| + } |
| + found := 0 |
| + for _, a := range h.Admissions { |
| + if a.Session != h.Session { |
| + continue |
| + } |
| + if a.Box != box || a.Region != region || a.Generation != h.Generation || a.Mode != "cold" { |
| + return servingArtifactIdentity{}, hostbox.ErrChanged |
| + } |
| + found++ |
| + } |
| + if found != 1 { |
| + return servingArtifactIdentity{}, hostbox.ErrChanged |
| + } |
| + return servingArtifactIdentity{o.Host, box, region, h.Incarnation, h.Location, h.Generation, h.Stamp, h.Session}, nil |
| +} |
| + |
| +func (r *claimServingArtifactRetirement) snapshot(ctx context.Context, admitted hostclaim.Observed, in ClaimLaunchInputs) (hostclaim.Observed, servingArtifactIdentity, string, uint64, error) { |
| + fail := func(err error) (hostclaim.Observed, servingArtifactIdentity, string, uint64, error) { |
| + return hostclaim.Observed{}, servingArtifactIdentity{}, "", 0, err |
| + } |
| + if ctx == nil || r == nil || r.Claims == nil || r.Root == nil || !filepath.IsAbs(r.DataDir) || r.Guard.DataDir != r.DataDir { |
| + return fail(hostbox.ErrUnsupported) |
| + } |
| + expected, err := servingWriterIdentity(admitted, r.Box, r.Region) |
| + if err != nil { |
| + return fail(err) |
| + } |
| + if in.DBPath != provision.NestedDBPath(r.DataDir, admitted.Host) { |
| + return fail(hostbox.ErrChanged) |
| + } |
| + db, err := filepath.Rel(r.DataDir, in.DBPath) |
| + if err != nil || !filepath.IsLocal(db) { |
| + return fail(hostbox.ErrChanged) |
| + } |
| + db = filepath.ToSlash(db) |
| + fresh, err := hostclaim.Read(ctx, r.Claims, admitted.Host) |
| + if err != nil { |
| + return fail(err) |
| + } |
| + actual, err := servingWriterIdentity(fresh, r.Box, r.Region) |
| + if err != nil || actual != expected { |
| + return fail(hostbox.ErrChanged) |
| + } |
| + a, err := servingInventoryPair(fresh.Claim.Inventory[r.Box], db) |
| + if err != nil || a > actual.Stamp { |
| + return fail(hostbox.ErrChanged) |
| + } |
| + return fresh, actual, db, a, nil |
| +} |
| + |
| +func servingPredecessor(c *hostclaim.Claim, current servingArtifactIdentity, stamp uint64, generation string) (servingArtifactIdentity, error) { |
| + if stamp == 0 || stamp >= current.Stamp { |
| + return servingArtifactIdentity{}, hostbox.ErrChanged |
| + } |
| + var out servingArtifactIdentity |
| + count := 0 |
| + for _, a := range c.Holder.Admissions { |
| + if a.Session != stamp { |
| + continue |
| + } |
| + if a.Mode != "cold" || a.Box != current.Box || a.Region != current.Region || a.Generation == "" || a.Generation == current.Generation || (generation != "" && a.Generation != generation) { |
| + return out, hostbox.ErrChanged |
| + } |
| + out = current |
| + out.Session, out.Stamp, out.Generation = a.Session, stamp, a.Generation |
| + count++ |
| + } |
| + if count != 1 { |
| + return servingArtifactIdentity{}, hostbox.ErrChanged |
| + } |
| + return out, nil |
| +} |
| + |
| +func (r *claimServingArtifactRetirement) stopped(ctx context.Context, prior servingArtifactIdentity) error { |
| + receipts, err := r.Guard.CompletedServingStops(ctx, prior.Host, hostguard.Captured{Incarnation: prior.Incarnation, Stamp: prior.Stamp}, prior.Session, prior.Box) |
| + if err != nil { |
| + return err |
| + } |
| + if len(receipts) == 0 { |
| + return hostguard.ErrNoFenceReceipt |
| + } |
| + for _, receipt := range receipts { |
| + if receipt.Target.Region != prior.Region || verifyRetirementReceipt(ctx, receipt) != nil { |
| + return ErrClaimStopReceipt |
| + } |
| + } |
| + return nil |
| +} |
| + |
| +func (r *claimServingArtifactRetirement) Retire(ctx context.Context, admitted hostclaim.Observed, in ClaimLaunchInputs, generation string) error { |
| + if generation == "" { |
| + return hostbox.ErrChanged |
| + } |
| + return r.recover(ctx, admitted, in, generation) |
| +} |
| + |
| +func (r *claimServingArtifactRetirement) Recover(ctx context.Context, admitted hostclaim.Observed, in ClaimLaunchInputs) error { |
| + return r.recover(ctx, admitted, in, "") |
| +} |
| + |
| +func (r *claimServingArtifactRetirement) recover(ctx context.Context, admitted hostclaim.Observed, in ClaimLaunchInputs, generation string) error { |
| + r.forgetCompletion(admitted.Host) |
| + fresh, current, _, stamp, err := r.snapshot(ctx, admitted, in) |
| + if err != nil { |
| + return err |
| + } |
| + if stamp == 0 { |
| + // Legacy authority is available only after PrepareState. In particular, |
| + // missing tags never authorize deleting an existing generation marker. |
| + if generation != "" { |
| + return hostbox.ErrChanged |
| + } |
| + return nil |
| + } |
| + var prior servingArtifactIdentity |
| + if stamp != current.Stamp { |
| + prior, err = servingPredecessor(fresh.Claim, current, stamp, generation) |
| + if err != nil { |
| + return err |
| + } |
| + if err = r.stopped(ctx, prior); err != nil { |
| + return err |
| + } |
| + } |
| + authorization, err := r.Root.Park(ctx, current, prior) |
| + if err != nil { |
| + return err |
| + } |
| + if generation != "" && authorization.Predecessor.Generation != generation { |
| + return hostbox.ErrChanged |
| + } |
| + return r.finish(ctx, admitted, in, authorization, nil) |
| +} |
| + |
| +func (r *claimServingArtifactRetirement) BeforeLegacyInstall(ctx context.Context, admitted hostclaim.Observed, in ClaimLaunchInputs, work hostwrite.Work) error { |
| + r.forgetCompletion(admitted.Host) |
| + _, current, _, stamp, err := r.snapshot(ctx, admitted, in) |
| + if err != nil { |
| + return err |
| + } |
| + if work == nil { |
| + return hostbox.ErrUnsupported |
| + } |
| + if stamp != 0 { |
| + return r.Recover(ctx, admitted, in) |
| + } |
| + prior := servingLegacyPredecessor{Session: in.Admission.ObservedSession, Generation: in.Admission.ObservedGeneration, Empty: in.Empty} |
| + if (prior.Session == 0) != (prior.Generation == "") || prior.Session >= current.Session || (prior.Empty && prior.Session != 0) { |
| + return hostbox.ErrChanged |
| + } |
| + authorization, err := r.Root.ParkLegacy(ctx, current, prior) |
| + if err != nil { |
| + return err |
| + } |
| + if !authorization.Legacy { |
| + return hostbox.ErrChanged |
| + } |
| + return r.finish(ctx, admitted, in, authorization, work) |
| +} |
| + |
| +func (r *claimServingArtifactRetirement) validateAuthorization(ctx context.Context, fresh hostclaim.Observed, current servingArtifactIdentity, a servingParkAuthorization) error { |
| + original := a.OriginalSuccessor |
| + if a.TagsFrom > current.Stamp { |
| + return hostbox.ErrChanged |
| + } |
| + if a.TagsFrom != 0 && a.TagsFrom != current.Stamp && a.TagsFrom != a.Predecessor.Stamp { |
| + if a.TagsFrom < original.Stamp { |
| + return hostbox.ErrChanged |
| + } |
| + if _, err := servingPredecessor(fresh.Claim, current, a.TagsFrom, ""); err != nil { |
| + return err |
| + } |
| + } |
| + if !a.Legacy && a.TagsFrom == 0 { |
| + return hostbox.ErrChanged |
| + } |
| + if a.Successor != current || original.Host != current.Host || original.Box != current.Box || original.Region != current.Region || original.Incarnation != current.Incarnation || original.Location != current.Location || original.Stamp == 0 || original.Session != original.Stamp || original.Generation == "" || original.Session > current.Session || a.Directory.Device == 0 || a.Directory.Inode == 0 || a.Directory.Mount == 0 || a.ID != servingRetirementID(a.Predecessor, original, a.Legacy) { |
| + return hostbox.ErrChanged |
| + } |
| + if original.Session == current.Session && original != current { |
| + return hostbox.ErrChanged |
| + } |
| + // A continuation preserves the original record identity; it cannot rename |
| + // that record to the next admission. Root authenticates its own continuation. |
| + if original.Session < current.Session { |
| + if _, err := servingPredecessor(fresh.Claim, current, original.Stamp, original.Generation); err != nil { |
| + return err |
| + } |
| + } |
| + if !a.Legacy { |
| + prior, err := servingPredecessor(fresh.Claim, current, a.Predecessor.Stamp, a.Predecessor.Generation) |
| + if err != nil || prior != a.Predecessor || prior.Stamp >= original.Stamp { |
| + return hostbox.ErrChanged |
| + } |
| + return r.stopped(ctx, prior) |
| + } |
| + if a.Predecessor != (servingArtifactIdentity{}) { |
| + return hostbox.ErrChanged |
| + } |
| + return nil |
| +} |
| + |
| +func (r *claimServingArtifactRetirement) finish(ctx context.Context, admitted hostclaim.Observed, in ClaimLaunchInputs, a servingParkAuthorization, work hostwrite.Work) error { |
| + fresh, current, _, _, err := r.snapshot(ctx, admitted, in) |
| + if err != nil { |
| + return err |
| + } |
| + if err = r.validateAuthorization(ctx, fresh, current, a); err != nil { |
| + return err |
| + } |
| + enroll := func() error { return r.enrollPairLocked(ctx, admitted, in, a) } |
| + if work != nil { |
| + err = work.With(ctx, in.Identity, enroll) |
| + } else { |
| + err = r.Guard.With(ctx, current.Host, hostguard.Captured{Incarnation: current.Incarnation, Stamp: current.Stamp}, func(c *hostclaim.Claim) error { |
| + identity, err := servingWriterIdentity(hostclaim.Observed{Host: current.Host, Claim: c}, r.Box, r.Region) |
| + if err != nil || identity != current { |
| + return hostbox.ErrChanged |
| + } |
| + return nil |
| + }, enroll) |
| + } |
| + if err != nil { |
| + return err |
| + } |
| + if err = r.Root.Complete(ctx, a.ID, current); err != nil { |
| + return err |
| + } |
| + r.mu.Lock() |
| + if r.completed == nil { |
| + r.completed = make(map[string]servingParkAuthorization) |
| + } |
| + r.completed[current.Host] = a |
| + r.mu.Unlock() |
| + return nil |
| +} |
| + |
| +// enrollPairLocked runs only under the selected Work or the matching Guard. |
| +// The root authorization proves the old sources have been consumed. The |
| +// descriptor check prevents a replacement inode from reaching purgeDB. |
| +func (r *claimServingArtifactRetirement) enrollPairLocked(ctx context.Context, admitted hostclaim.Observed, in ClaimLaunchInputs, a servingParkAuthorization) error { |
| + fresh, current, db, stamp, err := r.snapshot(ctx, admitted, in) |
| + if err != nil { |
| + return err |
| + } |
| + if a.Successor != current { |
| + return hostbox.ErrChanged |
| + } |
| + prior := a.TagsFrom |
| + if stamp != prior && stamp != current.Stamp { |
| + return hostbox.ErrChanged |
| + } |
| + if err = checkServingArtifactAbsence(in.DBPath, a.Directory); err != nil { |
| + return err |
| + } |
| + transition := servingInventoryTransition(current, db, prior) |
| + if stamp == current.Stamp { |
| + return transition(fresh.Claim) |
| + } |
| + _, updateErr := hostclaim.Update(ctx, r.Claims, fresh, transition) |
| + // An uncertain CAS is resolved only by a fresh, exact pair and the same |
| + // writer. A refusal never permits a database or marker write. |
| + after, same, _, tagged, readErr := r.snapshot(ctx, admitted, in) |
| + if readErr != nil { |
| + return readErr |
| + } |
| + if same != current || tagged != current.Stamp { |
| + if updateErr != nil { |
| + return updateErr |
| + } |
| + return hostbox.ErrChanged |
| + } |
| + if err = transition(after.Claim); err != nil { |
| + return err |
| + } |
| + return checkServingArtifactAbsence(in.DBPath, a.Directory) |
| +} |
| + |
| +func servingInventoryTransition(current servingArtifactIdentity, db string, prior uint64) func(*hostclaim.Claim) error { |
| + return func(c *hostclaim.Claim) error { |
| + actual, err := servingWriterIdentity(hostclaim.Observed{Host: current.Host, Claim: c}, current.Box, current.Region) |
| + if err != nil || actual != current || !filepath.IsLocal(db) || filepath.ToSlash(filepath.Clean(db)) != db || strings.ContainsAny(db, "\\\x00") { |
| + return hostbox.ErrChanged |
| + } |
| + entry := c.Inventory[current.Box] |
| + stamp, err := servingInventoryPair(entry, db) |
| + if err != nil || (stamp != prior && stamp != current.Stamp) { |
| + return hostbox.ErrChanged |
| + } |
| + // Copy before editing: a rejected transition cannot mutate its observation. |
| + paths := make(map[string]uint64, len(entry.Paths)+2) |
| + for p, s := range entry.Paths { |
| + paths[p] = s |
| + } |
| + paths[db], paths[db+".gen"] = current.Stamp, current.Stamp |
| + if c.Inventory == nil { |
| + c.Inventory = make(map[string]hostclaim.InventoryEntry) |
| + } |
| + c.Inventory[current.Box] = hostclaim.InventoryEntry{Layout: "nested", Paths: paths} |
| + return nil |
| + } |
| +} |
| + |
| +func (r *claimServingArtifactRetirement) EnrollLocked(ctx context.Context, admitted hostclaim.Observed, in ClaimLaunchInputs) error { |
| + if r == nil { |
| + return hostbox.ErrUnsupported |
| + } |
| + r.mu.Lock() |
| + a, ok := r.completed[admitted.Host] |
| + r.mu.Unlock() |
| + if !ok { |
| + return hostbox.ErrChanged |
| + } |
| + // This final hook can only revalidate a pair completed outside Work. It |
| + // cannot finish an incomplete root record or enroll an unparked legacy DB. |
| + _, current, _, stamp, err := r.snapshot(ctx, admitted, in) |
| + if err != nil { |
| + return err |
| + } |
| + if a.Successor != current || stamp != current.Stamp { |
| + return hostbox.ErrChanged |
| + } |
| + return r.enrollPairLocked(ctx, admitted, in, a) |
| +} |
| + |
| +// servingInventoryPair preserves RecordInventory's normalized-path and |
| +// ancestor rules without taking its lock again inside Work. |
| +func servingInventoryPair(entry hostclaim.InventoryEntry, db string) (uint64, error) { |
| + if !filepath.IsLocal(db) || filepath.ToSlash(filepath.Clean(db)) != db || strings.ContainsAny(db, "\\\x00") { |
| + return 0, hostbox.ErrChanged |
| + } |
| + if len(entry.Paths) != 0 && entry.Layout != "nested" { |
| + return 0, hostbox.ErrChanged |
| + } |
| + left, leftOK := entry.Paths[db] |
| + right, rightOK := entry.Paths[db+".gen"] |
| + if leftOK != rightOK || left != right || (leftOK && left == 0) { |
| + return 0, hostbox.ErrChanged |
| + } |
| + all := make([]string, 0, len(entry.Paths)+2) |
| + for p, stamp := range entry.Paths { |
| + if !filepath.IsLocal(p) || filepath.ToSlash(filepath.Clean(p)) != p || strings.ContainsAny(p, "\\\x00") || stamp == 0 { |
| + return 0, hostbox.ErrChanged |
| + } |
| + all = append(all, p) |
| + } |
| + if !leftOK { |
| + all = append(all, db, db+".gen") |
| + } |
| + sort.Strings(all) |
| + for i, p := range all { |
| + for _, next := range all[i+1:] { |
| + if strings.HasPrefix(next, p+"/") { |
| + return 0, hostbox.ErrChanged |
| + } |
| + } |
| + } |
| + return left, nil |
| +} |
| + |
| +// bindServingArtifactRetirement composes the serving contract with whichever |
| +// operation owners the assembly already enabled. A missing root client leaves |
| +// the existing production refusal in place. |
| +func bindServingArtifactRetirement(w *ClaimWakeAssembly, r *claimServingArtifactRetirement) error { |
| + if w == nil || r == nil || r.Root == nil || r.Claims == nil { |
| + return hostbox.ErrUnsupported |
| + } |
| + operationRetire, operationRecover := w.RetireArtifacts, w.RecoverArtifacts |
| + w.RetireArtifacts = func(ctx context.Context, o hostclaim.Observed, in ClaimLaunchInputs, generation string) error { |
| + if o.Claim != nil && o.Claim.Op.Kind == hostclaim.KindServing { |
| + return r.Retire(ctx, o, in, generation) |
| + } |
| + if operationRetire == nil { |
| + return hostbox.ErrChanged |
| + } |
| + return operationRetire(ctx, o, in, generation) |
| + } |
| + w.RecoverArtifacts = func(ctx context.Context, o hostclaim.Observed, in ClaimLaunchInputs) error { |
| + if o.Claim != nil && o.Claim.Op.Kind == hostclaim.KindServing { |
| + return r.Recover(ctx, o, in) |
| + } |
| + if operationRecover == nil { |
| + return nil |
| + } |
| + return operationRecover(ctx, o, in) |
| + } |
| + w.BeforeLegacyInstall, w.BeforeInstallLocked = r.BeforeLegacyInstall, r.EnrollLocked |
| + w.RequireServingEnrollment = true |
| + return nil |
| +} |
| + |
| +func (r *claimServingArtifactRetirement) forgetCompletion(host string) { |
| + if r == nil { |
| + return |
| + } |
| + r.mu.Lock() |
| + delete(r.completed, host) |
| + r.mu.Unlock() |
| +} |
diff --git a/internal/host/claimservingretire_linux_test.go b/internal/host/claimservingretire_linux_test.go| new file mode 100644 |
| index 000000000..fa65d2e68 |
| --- /dev/null |
| +++ b/internal/host/claimservingretire_linux_test.go |
| @@ -0,0 +1,767 @@ |
| +//go:build linux |
| + |
| +package host |
| + |
| +import ( |
| + "bytes" |
| + "context" |
| + "encoding/json" |
| + "errors" |
| + "fmt" |
| + "os" |
| + "path/filepath" |
| + "reflect" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/activator" |
| + "github.com/carlosframework/platform/internal/genselect" |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/hostguard" |
| + "github.com/carlosframework/platform/internal/hostwrite" |
| + "github.com/carlosframework/platform/internal/lease/leasetest" |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| + "github.com/superfly/ltx" |
| + "golang.org/x/sys/unix" |
| +) |
| + |
| +// This is the explicit Task 0 root seam, not a production parking witness. |
| +// It retains immutable replies across controller restarts so the agent's |
| +// enrollment/completion order can be tested independently of Task 0R. |
| +type servingParkFixture struct { |
| + continuePending bool |
| + t *testing.T |
| + f *recoveryFixture |
| + db string |
| + records map[string]servingParkAuthorization |
| + calls []string |
| + inside bool |
| + failComplete bool |
| + afterPark func() |
| +} |
| + |
| +func (p *servingParkFixture) park(current, prior servingArtifactIdentity, legacy bool) (servingParkAuthorization, error) { |
| + if p.inside { |
| + p.t.Fatal("root called while Work held") |
| + } |
| + p.calls = append(p.calls, "park") |
| + for _, a := range p.records { |
| + if a.Successor == current { |
| + return a, nil |
| + } |
| + } |
| + if p.continuePending { |
| + for id, a := range p.records { |
| + a.Successor = current |
| + db, _ := filepath.Rel(p.f.guard.DataDir, p.db) |
| + a.TagsFrom = p.f.read().Claim.Inventory[current.Box].Paths[db] |
| + p.records[id] = a |
| + return a, nil |
| + } |
| + } |
| + if !legacy && prior == (servingArtifactIdentity{}) { |
| + return servingParkAuthorization{}, hostbox.ErrChanged |
| + } |
| + a := servingParkAuthorization{TagsFrom: prior.Stamp, Predecessor: prior, Successor: current, OriginalSuccessor: current, Legacy: legacy} |
| + a.ID = servingRetirementID(prior, current, legacy) |
| + var stat unix.Statx_t |
| + if err := unix.Statx(unix.AT_FDCWD, filepath.Dir(p.db), unix.AT_SYMLINK_NOFOLLOW, unix.STATX_BASIC_STATS|unix.STATX_MNT_ID, &stat); err != nil { |
| + return a, err |
| + } |
| + a.Directory = servingArtifactDirectory{unix.Mkdev(stat.Dev_major, stat.Dev_minor), stat.Ino, stat.Mnt_id} |
| + dest := filepath.Join(p.f.guard.DataDir, "test-parks", a.ID) |
| + if err := os.MkdirAll(dest, 0700); err != nil { |
| + return a, err |
| + } |
| + for _, suffix := range []string{"", ".gen", "-wal", "-shm"} { |
| + if err := os.Rename(p.db+suffix, filepath.Join(dest, filepath.Base(p.db)+suffix)); err != nil && !errors.Is(err, os.ErrNotExist) { |
| + return a, err |
| + } |
| + } |
| + p.records[a.ID] = a |
| + if p.afterPark != nil { |
| + p.afterPark() |
| + } |
| + return a, nil |
| +} |
| +func (p *servingParkFixture) Park(_ context.Context, current, prior servingArtifactIdentity) (servingParkAuthorization, error) { |
| + return p.park(current, prior, false) |
| +} |
| +func (p *servingParkFixture) ParkLegacy(_ context.Context, current servingArtifactIdentity, _ servingLegacyPredecessor) (servingParkAuthorization, error) { |
| + return p.park(current, servingArtifactIdentity{}, true) |
| +} |
| +func (p *servingParkFixture) Complete(ctx context.Context, id string, current servingArtifactIdentity) error { |
| + if p.inside { |
| + p.t.Fatal("completion under Work") |
| + } |
| + p.calls = append(p.calls, "complete") |
| + a, ok := p.records[id] |
| + if !ok || a.Successor != current { |
| + return hostbox.ErrChanged |
| + } |
| + o, err := hostclaim.Read(ctx, p.f.claims, current.Host) |
| + if err != nil { |
| + return err |
| + } |
| + db, _ := filepath.Rel(p.f.guard.DataDir, p.db) |
| + if o.Claim.Inventory[current.Box].Paths[db] != current.Stamp || o.Claim.Inventory[current.Box].Paths[db+".gen"] != current.Stamp { |
| + p.t.Fatal("root completion before paired enrollment") |
| + } |
| + if p.failComplete { |
| + return hostbox.ErrChanged |
| + } |
| + return nil |
| +} |
| + |
| +type servingRetirementWork struct { |
| + p *servingParkFixture |
| + r *claimServingArtifactRetirement |
| + admitted hostclaim.Observed |
| +} |
| + |
| +func (w servingRetirementWork) With(ctx context.Context, _ hostwrite.Identity, fn func() error) error { |
| + if w.p.inside { |
| + w.p.t.Fatal("recursive Work") |
| + } |
| + h := w.admitted.Claim.Holder |
| + return w.r.Guard.With(ctx, w.admitted.Host, hostguard.Captured{Incarnation: h.Incarnation, Stamp: h.Stamp}, func(c *hostclaim.Claim) error { |
| + if !claimSameWriter(w.admitted, hostclaim.Observed{Host: w.admitted.Host, Claim: c}, nil) { |
| + return hostbox.ErrChanged |
| + } |
| + return nil |
| + }, func() error { w.p.inside = true; defer func() { w.p.inside = false }(); return fn() }) |
| +} |
| + |
| +func servingRetirementFixture(t *testing.T) (*recoveryFixture, *claimServingArtifactRetirement, *servingParkFixture, ClaimLaunchInputs) { |
| + t.Helper() |
| + f := newRecoveryFixture(t) |
| + db := provision.NestedDBPath(f.guard.DataDir, recoveryHost) |
| + if err := os.MkdirAll(filepath.Dir(db), 0700); err != nil { |
| + t.Fatal(err) |
| + } |
| + p := &servingParkFixture{t: t, f: f, db: db, records: make(map[string]servingParkAuthorization)} |
| + r := &claimServingArtifactRetirement{Claims: f.claims, Guard: f.guard, DataDir: f.guard.DataDir, Box: "box-1", Region: "ie", Root: p} |
| + return f, r, p, ClaimLaunchInputs{DBPath: db} |
| +} |
| +func servingAdmit(t *testing.T, f *recoveryFixture, in *ClaimLaunchInputs, generation string) hostclaim.Observed { |
| + t.Helper() |
| + o := f.read() |
| + in.Admission = hostclaim.Admission{ObservedSession: o.Claim.Holder.Session, ObservedGeneration: o.Claim.Holder.Generation, Box: "box-1", Region: "ie", PredecessorRegion: "ie", Mode: "cold", NewGeneration: generation} |
| + admitted, err := hostclaim.Update(t.Context(), f.claims, o, hostclaim.AdmitWriter(in.Admission)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := f.guard.Activate(t.Context(), recoveryHost, "box-1", nil); err != nil { |
| + t.Fatal(err) |
| + } |
| + return admitted |
| +} |
| +func servingStop(t *testing.T, f *recoveryFixture, o hostclaim.Observed) hostguard.FenceReceipt { |
| + t.Helper() |
| + target := recoveryTuple(o, claimStopLocal{Actor: hostbox.Actor{Box: "box-1", Region: "ie", Enrollment: "enrollment", Revision: 7}, BootID: "boot", ControllerID: "controller"}) |
| + target.FenceID, target.ClaimGen = "stop-"+o.Claim.Holder.Generation, o.Claim.Gen |
| + if _, err := f.guard.BeginFence(t.Context(), target); err != nil { |
| + t.Fatal(err) |
| + } |
| + s := StopReceipt{V: 1, ID: target.FenceID, Kind: StopReceiptStopped, Region: target.Region, ClaimETag: target.ClaimETag, ClaimGen: target.ClaimGen, FenceID: target.FenceID, Host: target.Host, Incarnation: target.Incarnation, InstallationStamp: target.Stamp, WriterSession: target.Session, Box: target.Box, Enrollment: target.Enrollment, BootID: target.BootID, ControllerID: target.ControllerID, DirectoryRevision: target.DirectoryRevision, EvidenceAt: f.now} |
| + body := mustMarshalStopReceipt(s) |
| + receipt := hostguard.FenceReceipt{Target: target, Bytes: body, EvidenceAt: f.now} |
| + if err := f.guard.CompleteFence(t.Context(), target, body, f.now, func(ctx context.Context, _ hostguard.FenceTarget, _ []byte, _ time.Time) error { |
| + return verifyRetirementReceipt(ctx, receipt) |
| + }); err != nil { |
| + t.Fatal(err) |
| + } |
| + return receipt |
| +} |
| + |
| +func TestServingRetirementEnrollmentAndProductionRestore(t *testing.T) { |
| + f, r, p, in := servingRetirementFixture(t) |
| + plan := plan12RestoreBaseline(t, f) |
| + if err := os.WriteFile(in.DBPath, []byte("legacy source"), 0600); err != nil { |
| + t.Fatal(err) |
| + } |
| + files := filepath.Join(filepath.Dir(in.DBPath), "files") |
| + if err := os.Mkdir(files, 0700); err != nil { |
| + t.Fatal(err) |
| + } |
| + env := filepath.Join(filepath.Dir(in.DBPath), "env") |
| + if err := os.WriteFile(env, []byte("KEEP=1"), 0600); err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, generation := range []string{"G", "G2"} { |
| + admitted := servingAdmit(t, f, &in, generation) |
| + work := servingRetirementWork{p: p, r: r, admitted: admitted} |
| + restore := hostbox.ColdRestore{Reader: f.data, Bucket: f.bucket, Generation: generation, Work: work, |
| + BeforeLegacyInstall: func(ctx context.Context, _, _ string) error { return r.BeforeLegacyInstall(ctx, admitted, in, work) }, |
| + BeforeInstallLocked: func(ctx context.Context, _, _ string) error { return r.EnrollLocked(ctx, admitted, in) }, |
| + RetireArtifacts: func(ctx context.Context, generation string) error { return r.Retire(ctx, admitted, in, generation) }, |
| + } |
| + if err := r.Recover(t.Context(), admitted, in); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := restore.Install(t.Context(), in.DBPath, plan); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got := mwReadDB(t, in.DBPath); got != "selected recovery baseline" { |
| + t.Fatal(got) |
| + } |
| + marker, err := hostbox.ReadGenerationMarker(in.DBPath + ".gen") |
| + if err != nil || marker != generation { |
| + t.Fatalf("marker %q: %v", marker, err) |
| + } |
| + servingStop(t, f, f.read()) |
| + } |
| + if body, err := os.ReadFile(env); err != nil || string(body) != "KEEP=1" { |
| + t.Fatalf("env: %s %v", body, err) |
| + } |
| + if _, err := os.Stat(files); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(p.records) != 2 { |
| + t.Fatalf("park records: %v", p.records) |
| + } |
| +} |
| + |
| +func TestServingRetirementRefusesUnprovedEnrollment(t *testing.T) { |
| + f, r, p, in := servingRetirementFixture(t) |
| + admitted := servingAdmit(t, f, &in, "G") |
| + if err := r.EnrollLocked(t.Context(), admitted, in); !errors.Is(err, hostbox.ErrChanged) { |
| + t.Fatal(err) |
| + } |
| + if len(p.calls) != 0 { |
| + t.Fatal(p.calls) |
| + } |
| + for _, a := range []string{"", ".gen", "-wal", "-shm"} { |
| + t.Run("replacement"+a, func(t *testing.T) { |
| + if err := r.BeforeLegacyInstall(t.Context(), admitted, in, servingRetirementWork{p, r, admitted}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(in.DBPath+a, []byte("unexpected"), 0600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := r.EnrollLocked(t.Context(), admitted, in); !errors.Is(err, hostbox.ErrChanged) { |
| + t.Fatal(err) |
| + } |
| + if err := os.Remove(in.DBPath + a); err != nil { |
| + t.Fatal(err) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestServingRetirementCrashBetweenTagsAndCompletion(t *testing.T) { |
| + f, r, p, in := servingRetirementFixture(t) |
| + admitted := servingAdmit(t, f, &in, "G") |
| + p.failComplete = true |
| + if err := r.BeforeLegacyInstall(t.Context(), admitted, in, servingRetirementWork{p, r, admitted}); !errors.Is(err, hostbox.ErrChanged) { |
| + t.Fatal(err) |
| + } |
| + if err := r.EnrollLocked(t.Context(), admitted, in); !errors.Is(err, hostbox.ErrChanged) { |
| + t.Fatal(err) |
| + } |
| + p.failComplete = false |
| + restarted := &claimServingArtifactRetirement{Claims: r.Claims, Guard: r.Guard, DataDir: r.DataDir, Box: r.Box, Region: r.Region, Root: p} |
| + before := f.read().ETag |
| + if err := restarted.Recover(t.Context(), admitted, in); err != nil { |
| + t.Fatal(err) |
| + } |
| + if f.read().ETag != before { |
| + t.Fatal("recovery rewrote current tags") |
| + } |
| + if err := restarted.EnrollLocked(t.Context(), admitted, in); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(p.records) != 1 { |
| + t.Fatal("recovery invented another retirement") |
| + } |
| +} |
| + |
| +func TestServingRetirementAuthenticatesEveryReceiptField(t *testing.T) { |
| + fields := []string{"Region", "ClaimETag", "ClaimGen", "ObligationID", "FenceID", "Host", "Incarnation", "InstallationStamp", "WriterSession", "OperationID", "Phase", "Box", "Enrollment", "BootID", "ControllerID", "DirectoryRevision", "EvidenceAt"} |
| + for _, field := range fields { |
| + t.Run(field, func(t *testing.T) { |
| + f, r, p, in := servingRetirementFixture(t) |
| + old := servingAdmit(t, f, &in, "G") |
| + if err := r.BeforeLegacyInstall(t.Context(), old, in, servingRetirementWork{p, r, old}); err != nil { |
| + t.Fatal(err) |
| + } |
| + receipt := servingStop(t, f, f.read()) |
| + admitted := servingAdmit(t, f, &in, "G2") |
| + var body StopReceipt |
| + if err := json.Unmarshal(receipt.Bytes, &body); err != nil { |
| + t.Fatal(err) |
| + } |
| + v := reflect.ValueOf(&body).Elem().FieldByName(field) |
| + switch v.Kind() { |
| + case reflect.String: |
| + v.SetString(v.String() + "bad") |
| + case reflect.Uint64: |
| + v.SetUint(v.Uint() + 1) |
| + default: |
| + v.Set(reflect.ValueOf(f.now.Add(time.Second))) |
| + } |
| + receipt.Bytes = mustMarshalStopReceipt(body) |
| + r.Guard.FenceView = func(_ context.Context, host string, c hostguard.Captured) (hostguard.FenceView, error) { |
| + return hostguard.FenceView{Schema: 1, Host: host, Captured: c, Installation: hostguard.RootInstallation{Host: host, Migration: 1}, Receipts: []hostguard.FenceReceipt{receipt}}, nil |
| + } |
| + p.calls = nil |
| + before := f.read().ETag |
| + if err := r.Retire(t.Context(), admitted, in, "G"); !errors.Is(err, ErrClaimStopReceipt) { |
| + t.Fatalf("forged %s: %v", field, err) |
| + } |
| + if len(p.calls) != 0 || f.read().ETag != before { |
| + t.Fatal("forged receipt caused effects") |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestServingRetirementRefusesWrongTagsAndWriter(t *testing.T) { |
| + for _, name := range []string{"missing-one", "zero", "mixed", "future", "foreign-box", "foreign-generation", "stale-location", "flat", "ancestor", "outside-path", "noncold", "duplicate-admission", "missing-receipt", "wrong-session"} { |
| + t.Run(name, func(t *testing.T) { |
| + f, r, p, in := servingRetirementFixture(t) |
| + old := servingAdmit(t, f, &in, "G") |
| + if err := r.BeforeLegacyInstall(t.Context(), old, in, servingRetirementWork{p, r, old}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if name != "missing-receipt" { |
| + servingStop(t, f, f.read()) |
| + } |
| + admitted := servingAdmit(t, f, &in, "G2") |
| + db, _ := filepath.Rel(r.DataDir, in.DBPath) |
| + if name == "outside-path" { |
| + in.DBPath = filepath.Join(r.DataDir, "foreign.db") |
| + } else if name == "stale-location" { |
| + admitted.Claim.Holder.Location += "/stale" |
| + } else { |
| + o := f.read() |
| + _, err := hostclaim.Update(t.Context(), f.claims, o, func(c *hostclaim.Claim) error { |
| + e := c.Inventory[r.Box] |
| + switch name { |
| + case "missing-one": |
| + delete(e.Paths, db+".gen") |
| + case "zero": |
| + e.Paths[db], e.Paths[db+".gen"] = 0, 0 |
| + case "mixed": |
| + e.Paths[db]++ |
| + case "future": |
| + e.Paths[db], e.Paths[db+".gen"] = c.Gen+50, c.Gen+50 |
| + case "foreign-box": |
| + c.Holder.Admissions[0].Box = "foreign" |
| + case "foreign-generation": |
| + c.Holder.Admissions[0].Generation = "foreign" |
| + case "flat": |
| + e.Layout = "flat" |
| + case "ancestor": |
| + e.Paths[filepath.Dir(db)] = old.Claim.Holder.Stamp |
| + case "noncold": |
| + c.Holder.Admissions[0].Mode = "warm" |
| + case "duplicate-admission": |
| + c.Holder.Admissions = append(c.Holder.Admissions, c.Holder.Admissions[0]) |
| + case "wrong-session": |
| + c.Holder.Admissions[0].Session++ |
| + } |
| + c.Inventory[r.Box] = e |
| + return nil |
| + }) |
| + // Some invalid claims are refused by the writer itself. That also |
| + // prevents retirement; never persist malformed bytes to bypass it. |
| + if err != nil { |
| + return |
| + } |
| + } |
| + before := f.read().ETag |
| + p.calls = nil |
| + if err := r.Retire(t.Context(), admitted, in, "G"); err == nil { |
| + t.Fatal("invalid authority accepted") |
| + } |
| + if len(p.calls) != 0 { |
| + t.Fatal(p.calls) |
| + } |
| + if f.read().ETag != before { |
| + t.Fatal("invalid authority retagged inventory") |
| + } |
| + }) |
| + } |
| +} |
| + |
| +type servingCASFailure struct { |
| + release.Store |
| + afterCommit bool |
| + calls int |
| +} |
| + |
| +func (s *servingCASFailure) Put(ctx context.Context, key string, b []byte, etag string) (string, error) { |
| + s.calls++ |
| + if s.afterCommit { |
| + if _, err := s.Store.Put(ctx, key, b, etag); err != nil { |
| + return "", err |
| + } |
| + } |
| + return "", fmt.Errorf("injected lost CAS reply") |
| +} |
| +func TestServingEnrollmentResolvesOnlyExactCommittedCAS(t *testing.T) { |
| + for _, committed := range []bool{false, true} { |
| + t.Run(fmt.Sprint(committed), func(t *testing.T) { |
| + f, r, p, in := servingRetirementFixture(t) |
| + admitted := servingAdmit(t, f, &in, "G") |
| + r.Claims = &servingCASFailure{Store: f.claims, afterCommit: committed} |
| + err := r.BeforeLegacyInstall(t.Context(), admitted, in, servingRetirementWork{p, r, admitted}) |
| + if (err == nil) != committed { |
| + t.Fatalf("committed %v: %v", committed, err) |
| + } |
| + if !committed { |
| + if err := r.EnrollLocked(t.Context(), admitted, in); err == nil { |
| + t.Fatal("uncommitted enrollment admitted") |
| + } |
| + r.Claims = f.claims |
| + if err := r.BeforeLegacyInstall(t.Context(), admitted, in, servingRetirementWork{p, r, admitted}); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if len(p.records) != 1 { |
| + t.Fatal("CAS retry reparks legacy source") |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestServingRetirementPreservesOperationDispatch(t *testing.T) { |
| + _, r, _, _ := servingRetirementFixture(t) |
| + var calls []string |
| + w := &ClaimWakeAssembly{ |
| + RetireArtifacts: func(_ context.Context, o hostclaim.Observed, _ ClaimLaunchInputs, _ string) error { |
| + calls = append(calls, "retire:"+o.Claim.Op.Kind) |
| + return nil |
| + }, |
| + RecoverArtifacts: func(_ context.Context, o hostclaim.Observed, _ ClaimLaunchInputs) error { |
| + calls = append(calls, "recover:"+o.Claim.Op.Kind) |
| + return nil |
| + }, |
| + } |
| + if err := bindServingArtifactRetirement(w, r); err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, kind := range []string{hostclaim.KindLanding, hostclaim.KindRestoring, hostclaim.KindReverting, hostclaim.KindRelocating} { |
| + o := hostclaim.Observed{Claim: &hostclaim.Claim{Op: hostclaim.Op{Kind: kind}}} |
| + if err := w.RetireArtifacts(t.Context(), o, ClaimLaunchInputs{}, "G"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := w.RecoverArtifacts(t.Context(), o, ClaimLaunchInputs{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if len(calls) != 8 { |
| + t.Fatal(calls) |
| + } |
| + o := hostclaim.Observed{Claim: &hostclaim.Claim{Op: hostclaim.Op{Kind: hostclaim.KindServing}}} |
| + if err := w.RetireArtifacts(t.Context(), o, ClaimLaunchInputs{}, "G"); err == nil { |
| + t.Fatal("invalid serving authority accepted") |
| + } |
| + if len(calls) != 8 { |
| + t.Fatal("serving fell through to operations") |
| + } |
| + if !w.RequireServingEnrollment || w.BeforeInstallLocked == nil || w.BeforeLegacyInstall == nil { |
| + t.Fatal("missing serving hooks") |
| + } |
| +} |
| + |
| +func TestServingArtifactDirectoryRefusesReplacement(t *testing.T) { |
| + for _, kind := range []string{"missing", "replacement", "symlink", "mount", "device", "inode", "leaf-symlink"} { |
| + t.Run(kind, func(t *testing.T) { |
| + f, r, p, in := servingRetirementFixture(t) |
| + admitted := servingAdmit(t, f, &in, "G") |
| + if err := r.BeforeLegacyInstall(t.Context(), admitted, in, servingRetirementWork{p, r, admitted}); err != nil { |
| + t.Fatal(err) |
| + } |
| + a := r.completed[admitted.Host] |
| + switch kind { |
| + case "missing", "replacement", "symlink": |
| + dir := filepath.Dir(in.DBPath) |
| + if err := os.Rename(dir, dir+".original"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if kind == "replacement" { |
| + if err := os.Mkdir(dir, 0700); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if kind == "symlink" { |
| + if err := os.Symlink(dir+".original", dir); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + case "mount": |
| + a.Directory.Mount++ |
| + case "device": |
| + a.Directory.Device++ |
| + case "inode": |
| + a.Directory.Inode++ |
| + case "leaf-symlink": |
| + if err := os.Symlink(filepath.Join(t.TempDir(), "absent"), in.DBPath); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if err := checkServingArtifactAbsence(in.DBPath, a.Directory); err == nil { |
| + t.Fatal("unsafe directory accepted") |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestServingInventoryPairValidationAndPreservation(t *testing.T) { |
| + f, _, _, in := servingRetirementFixture(t) |
| + o := servingAdmit(t, f, &in, "G") |
| + current, err := servingWriterIdentity(o, "box-1", "ie") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + db, _ := filepath.Rel(f.guard.DataDir, in.DBPath) |
| + for _, name := range []string{"absent", "current", "older", "missing-one", "mixed", "zero", "future", "ancestor", "descendant", "unrelated-overlap", "unnormalized", "flat"} { |
| + t.Run(name, func(t *testing.T) { |
| + c := *o.Claim |
| + paths := map[string]uint64{"unrelated": 1} |
| + entry := hostclaim.InventoryEntry{Layout: "nested", Paths: paths} |
| + expected := false |
| + prior := uint64(0) |
| + switch name { |
| + case "absent": |
| + expected = true |
| + case "current": |
| + paths[db], paths[db+".gen"] = current.Stamp, current.Stamp |
| + expected = true |
| + case "older": |
| + paths[db], paths[db+".gen"] = 1, 1 |
| + prior = 1 |
| + expected = true |
| + case "missing-one": |
| + paths[db] = 1 |
| + case "mixed": |
| + paths[db], paths[db+".gen"] = 1, 2 |
| + case "zero": |
| + paths[db], paths[db+".gen"] = 0, 0 |
| + case "future": |
| + paths[db], paths[db+".gen"] = current.Stamp+1, current.Stamp+1 |
| + case "ancestor": |
| + paths[filepath.Dir(db)] = 1 |
| + case "descendant": |
| + paths[db+"/child"] = 1 |
| + case "unrelated-overlap": |
| + paths["unrelated/child"] = 1 |
| + case "unnormalized": |
| + paths["a/../b"] = 1 |
| + case "flat": |
| + entry.Layout = "flat" |
| + } |
| + c.Inventory = map[string]hostclaim.InventoryEntry{"box-1": entry, "other": {Layout: "flat", Paths: map[string]uint64{"other-db": 99}}} |
| + before, _ := json.Marshal(c) |
| + err := servingInventoryTransition(current, db, prior)(&c) |
| + if (err == nil) != expected { |
| + t.Fatalf("accepted=%v: %v", expected, err) |
| + } |
| + if !expected { |
| + after, _ := json.Marshal(c) |
| + if string(after) != string(before) { |
| + t.Fatal("refused transition mutated claim") |
| + } |
| + return |
| + } |
| + if c.Inventory["box-1"].Paths[db] != current.Stamp || c.Inventory["box-1"].Paths[db+".gen"] != current.Stamp || c.Inventory["box-1"].Paths["unrelated"] != 1 || c.Inventory["other"].Paths["other-db"] != 99 { |
| + t.Fatal(c.Inventory) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestServingRetirementParkThenClaimRaceRefusesRetag(t *testing.T) { |
| + f, r, p, in := servingRetirementFixture(t) |
| + admitted := servingAdmit(t, f, &in, "G") |
| + p.afterPark = func() { |
| + p.afterPark = nil |
| + newer := in |
| + servingAdmit(t, f, &newer, "racer") |
| + } |
| + if err := r.BeforeLegacyInstall(t.Context(), admitted, in, servingRetirementWork{p, r, admitted}); err == nil { |
| + t.Fatal("stale writer enrolled") |
| + } |
| + if len(f.read().Claim.Inventory) != 0 { |
| + t.Fatal("stale writer tagged new installation") |
| + } |
| + if err := r.EnrollLocked(t.Context(), admitted, in); err == nil { |
| + t.Fatal("stale writer permitted restore") |
| + } |
| +} |
| + |
| +type servingBaselineInputs struct{ recoveryInputs } |
| + |
| +func (s servingBaselineInputs) Select(ctx context.Context, rt registry.Route, o hostclaim.Observed, policy activator.StartPolicy) (ClaimLaunchInputs, error) { |
| + in, err := s.recoveryInputs.Select(ctx, rt, o, policy) |
| + if err != nil { |
| + return in, err |
| + } |
| + in.DBPath = provision.NestedDBPath(s.f.guard.DataDir, o.Host) |
| + in.Empty = false |
| + in.Baseline, err = genselect.Select(ctx, s.f.data, genselect.Request{Bucket: s.f.bucket, Region: "ie", Host: o.Host, Holder: *o.Claim.Holder}) |
| + if err == nil { |
| + in.Admission.RemoteMaxTXID = in.Baseline.MaxTXID |
| + } |
| + return in, err |
| +} |
| + |
| +func TestServingRetirementOrdinaryPreparationThenSuccessor(t *testing.T) { |
| + f, r, p, in := servingRetirementFixture(t) |
| + bucket := leasetest.NewVersionedBucket() |
| + bucket.AddBucket("data") |
| + key := f.read().Claim.Holder.Location + "/0009/" + ltx.FormatFilename(1, 1) |
| + payload := mwSnapshot(t, 1, "ordinary serving baseline") |
| + version := bucket.Put("data", key, payload) |
| + f.data, f.bucket = bucket, "data" |
| + if err := os.WriteFile(in.DBPath, []byte("unmarked legacy"), 0600); err != nil { |
| + t.Fatal(err) |
| + } |
| + controller := f.controller("boot-1", "controller-1", 4101, "start-1") |
| + controller.wake.Inputs = servingBaselineInputs{recoveryInputs{f}} |
| + if err := bindServingArtifactRetirement(controller.wake, r); err != nil { |
| + t.Fatal(err) |
| + } |
| + var generations []string |
| + for i := 0; i < 2; i++ { |
| + admitted := controller.admit(t, f) |
| + generations = append(generations, admitted.Claim.Holder.Generation) |
| + start, err := controller.runtime.Prepare(t.Context(), registry.Route{Host: recoveryHost}, admitted, recoveryPolicy) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err = start.Restore(t.Context()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got := mwReadDB(t, in.DBPath); got != "ordinary serving baseline" { |
| + t.Fatal(got) |
| + } |
| + // Stop before Materialize: an unseeded ordinary C8 correctly refuses. |
| + // FailedAdmission drives the real serving stop and retry settlement. |
| + if err = controller.runtime.FailedAdmission(t.Context(), admitted, hostbox.ErrChanged); err != nil { |
| + t.Fatal(err) |
| + } |
| + f.now = f.now.Add(time.Minute) |
| + } |
| + if generations[0] == generations[1] { |
| + t.Fatal("successor reused generation") |
| + } |
| + if len(f.receipts()) != 2 { |
| + t.Fatal("missing exact serving stops") |
| + } |
| + if f.retry().AttemptsReserved != 2 { |
| + t.Fatal("retirement changed retry accounting") |
| + } |
| + remote, err := bucket.GetVersion(t.Context(), "data", key, version) |
| + if err != nil || !bytes.Equal(remote, payload) { |
| + t.Fatal("baseline changed") |
| + } |
| + if len(p.records) != 2 { |
| + t.Fatal("retirement did not park both sources") |
| + } |
| +} |
| + |
| +func TestServingRetirementContinuationKeepsOriginalIdentity(t *testing.T) { |
| + f, r, p, in := servingRetirementFixture(t) |
| + original := servingAdmit(t, f, &in, "G") |
| + p.failComplete = true |
| + if err := r.BeforeLegacyInstall(t.Context(), original, in, servingRetirementWork{p, r, original}); err == nil { |
| + t.Fatal("incomplete park accepted") |
| + } |
| + var old servingParkAuthorization |
| + for _, a := range p.records { |
| + old = a |
| + } |
| + servingStop(t, f, f.read()) |
| + successor := servingAdmit(t, f, &in, "G2") |
| + p.failComplete = false |
| + p.continuePending = true |
| + if err := r.Recover(t.Context(), successor, in); err != nil { |
| + t.Fatal(err) |
| + } |
| + a := r.completed[successor.Host] |
| + if a.ID != old.ID || a.OriginalSuccessor != old.OriginalSuccessor || a.Successor.Generation != "G2" || len(p.records) != 1 { |
| + t.Fatal("continuation relabeled immutable retirement") |
| + } |
| + if err := r.EnrollLocked(t.Context(), successor, in); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := r.EnrollLocked(t.Context(), original, in); err == nil { |
| + t.Fatal("old admission reused continuation") |
| + } |
| +} |
| + |
| +func TestServingRetirementAuthorizationBinding(t *testing.T) { |
| + f, r, p, in := servingRetirementFixture(t) |
| + admitted := servingAdmit(t, f, &in, "G") |
| + if err := r.BeforeLegacyInstall(t.Context(), admitted, in, servingRetirementWork{p, r, admitted}); err != nil { |
| + t.Fatal(err) |
| + } |
| + a := r.completed[admitted.Host] |
| + fresh := f.read() |
| + current, err := servingWriterIdentity(fresh, r.Box, r.Region) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, field := range []string{"Host", "Box", "Region", "Incarnation", "Location", "Generation", "Stamp", "Session"} { |
| + for _, target := range []string{"Successor", "OriginalSuccessor"} { |
| + t.Run(target+"/"+field, func(t *testing.T) { |
| + changed := a |
| + v := reflect.ValueOf(&changed).Elem().FieldByName(target).FieldByName(field) |
| + if v.Kind() == reflect.String { |
| + v.SetString(v.String() + "wrong") |
| + } else { |
| + v.SetUint(v.Uint() + 1) |
| + } |
| + // Even a recomputed journal ID cannot bind a foreign original tuple. |
| + changed.ID = servingRetirementID(changed.Predecessor, changed.OriginalSuccessor, changed.Legacy) |
| + if err := r.validateAuthorization(t.Context(), fresh, current, changed); err == nil { |
| + t.Fatal("foreign authorization accepted") |
| + } |
| + }) |
| + } |
| + } |
| + for _, name := range []string{"id", "mount", "inode", "device", "future-tags"} { |
| + t.Run(name, func(t *testing.T) { |
| + changed := a |
| + switch name { |
| + case "id": |
| + changed.ID = "wrong" |
| + case "mount": |
| + changed.Directory.Mount = 0 |
| + case "inode": |
| + changed.Directory.Inode = 0 |
| + case "device": |
| + changed.Directory.Device = 0 |
| + case "future-tags": |
| + changed.TagsFrom = current.Stamp + 1 |
| + } |
| + if err := r.validateAuthorization(t.Context(), fresh, current, changed); err == nil { |
| + t.Fatal("invalid authorization accepted") |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestServingRetirementFailedReplayInvalidatesPermission(t *testing.T) { |
| + f, r, p, in := servingRetirementFixture(t) |
| + admitted := servingAdmit(t, f, &in, "G") |
| + if err := r.BeforeLegacyInstall(t.Context(), admitted, in, servingRetirementWork{p, r, admitted}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := r.EnrollLocked(t.Context(), admitted, in); err != nil { |
| + t.Fatal(err) |
| + } |
| + p.failComplete = true |
| + if err := r.Recover(t.Context(), admitted, in); err == nil { |
| + t.Fatal("refused completion accepted") |
| + } |
| + if err := r.EnrollLocked(t.Context(), admitted, in); err == nil { |
| + t.Fatal("old completion survived failed replay") |
| + } |
| +} |
diff --git a/internal/hostbox/servingpark.go b/internal/hostbox/servingpark.go| new file mode 100644 |
| index 000000000..dba21f155 |
| --- /dev/null |
| +++ b/internal/hostbox/servingpark.go |
| @@ -0,0 +1,12 @@ |
| +package hostbox |
| + |
| +import "context" |
| + |
| +// RootSessionLaunches reads launch history for retirement decisions. A remote |
| +// agent journal view cannot establish root authority to discard a database. |
| +func (j *LaunchJournal) RootSessionLaunches(ctx context.Context, host, incarnation string, stamp, session uint64, box, enrollment string) ([]LaunchInventory, error) { |
| + if j == nil || j.remote != nil || j.db == nil { |
| + return nil, ErrUnsupported |
| + } |
| + return j.SessionLaunches(ctx, host, incarnation, stamp, session, box, enrollment) |
| +} |
diff --git a/internal/provision/testdata/plan14-root-reads-seed.tsv b/internal/provision/testdata/plan14-root-reads-seed.tsv| index 6fd9e187d..4de2c4271 100644 |
| --- a/internal/provision/testdata/plan14-root-reads-seed.tsv |
| +++ b/internal/provision/testdata/plan14-root-reads-seed.tsv |
| @@ -1,4 +1,4 @@ |
| -# Files seed checkpoint overlay; pending Paul’s census ruling. |
| +# Files seed checkpoint 2 overlay; pending Paul’s census ruling. |
| # No root parking or deletion authority is implemented in this checkpoint. |
| # Load after zddr4 and its explicit seal check. No legacy fingerprints are added or removed. |
| # Every classification below is rev3=PROPOSED-seed, not an approved disposition. |
| @@ -10,6 +10,27 @@ internal/host/claimwakeassembly.go ClaimWakeAssembly.Build authority C,G,J1,J4 r |
| internal/hostbox/restore.go ColdRestore.Install authority C,J2,J4 rev3=PROPOSED-seed; selected database versions, local generation marker and restored SQLite data; legacy callback outside Work and enrollment callback before any local write file:CreateTemp@2200,file:Open@3189 7054cf1c887f4b6b4e20b62fd94e17949770c675cbe7e8efe2accf7f20b1c7ba |
| internal/hostbox/restore.go ColdRestore.InstallEmpty authority C rev3=PROPOSED-seed; existing empty-bootstrap proof and local marker; legacy callback outside Work and enrollment callback before purge or marker creation f8c0ea4eaf3fa692abe52dd3b0799c9239bcb0d761c69644cc4be78d51d74c04 |
| internal/hostguard/writerstop.go Guard.CompletedServingStops authority C rev3=PROPOSED-seed; root fence journal through the existing Guard backend; exact host, incarnation, stamp, session and box, with empty operation, phase and obligation; no mixed-session or pending-only evidence 79260151b0fe4f86c59cbe65d727cdceb56028bdfdb208fc9f02cebba1473a98 |
| -@coverage internal/host fde50d7114d67d82f5cebef8e36c26020e2b29f5987b0b6bd818fe7c17b5476c |
| -@coverage internal/hostbox fa8a9baca13bbb0f7fbc2f47e24e2e5187a728372b247359a3f61d455e7e4cd5 |
| +internal/host/claimservingartifact_linux.go checkServingArtifactAbsence authority C,J4 rev3=PROPOSED-seed; source directory opened without symlinks; descriptor statx mount/device/inode and no-follow lookup of database, marker, WAL and SHM file:Fstatat@680,file:Statx@234 ab45d3f0651d25e8a6d1c48def6cc222c35dc152aca477ebd1555a67c9b89881 |
| +internal/host/claimservingartifact_other.go checkServingArtifactAbsence authority C,J4 rev3=PROPOSED-seed; non-Linux refusal; no source inspection or restore permission 27a7173119b5af75be262e9fcc56fddadb84a539f69fe5882f8c6613cfbc88c9 |
| +internal/host/claimservingpark.go servingGenerationParkDisposition authority C,J4 rev3=PROPOSED-seed; exact serving stop bytes and root-local launch journal; only settled root-observed tenant start permits slot retention; absent database preserves old slot; ambiguous intended launch refuses a11aaa2f4bb12cd7f4c52a0a38fb3b9cfe98714cea5ca3cc75f0e95053aca63d |
| +internal/host/claimservingretire.go bindServingArtifactRetirement authority C,J4 rev3=PROPOSED-seed; in-process root-client dependency and existing operation callbacks; serving dispatch does not replace landing/recovery dispatch 19ade85561b59f2f0b75f4f50b33333ab2598cba7d20e596893530c6a2f8102a |
| +internal/host/claimservingretire.go claimServingArtifactRetirement.BeforeLegacyInstall authority C,J4 rev3=PROPOSED-seed; admitted tuple and observed predecessor selection; root legacy authorization outside Work followed by guarded enrollment e8c23b09776dd4367e2a0fb5b85102747c3fe14cd45581f24e502e61b7fbcfd0 |
| +internal/host/claimservingretire.go claimServingArtifactRetirement.EnrollLocked authority C,J4 rev3=PROPOSED-seed; completed root authorization cached for exact writer, fresh current tags and source absence; missing or stale permission refuses local mutation cf33600f5c3faaa0cde5658e53b1657ceadb7340244a68c9791441834a372729 |
| +internal/host/claimservingretire.go claimServingArtifactRetirement.Recover authority C,J4 rev3=PROPOSED-seed; admitted serving observation and launch inputs; resumes retirement without treating absent marker as enrollment authority 674a5a3c598cbdc29494a3cfc0859dd632c41be154a966ca7684f4440cbccdcb |
| +internal/host/claimservingretire.go claimServingArtifactRetirement.Retire authority C,J4 rev3=PROPOSED-seed; captured predecessor generation; delegates only to serving recovery and root parking contract 6cf2e52f7e2aa511622268dd0a8dd37f1713e5beb4eba920e70057c2c90c32a3 |
| +internal/host/claimservingretire.go claimServingArtifactRetirement.enrollPairLocked authority C,J4 rev3=PROPOSED-seed; fresh claim and root-bound source directory; descriptor-relative absence before and after paired tag CAS; exact readback resolves uncertain CAS 351faf39946ade5f9166b79c0a177001c8b4287ee698ce37c2c1e8337b534c30 |
| +internal/host/claimservingretire.go claimServingArtifactRetirement.finish authority C,J4 rev3=PROPOSED-seed; root authorization; guarded paired-tag update before root completion; in-memory final-install permission only after completion 4ecf52c799a177f0df88d0429eb5bffc8d485045be065eb0107d0821772b0a29 |
| +internal/host/claimservingretire.go claimServingArtifactRetirement.forgetCompletion authority C,J4 rev3=PROPOSED-seed; in-memory completed authorization for the selected host; failed replay must invalidate prior final-install permission 771a11b74e74ca98cbb51a1da72ecffea91801c2b622448a559eb1bdc6f7b3b5 |
| +internal/host/claimservingretire.go claimServingArtifactRetirement.recover authority C,J4 rev3=PROPOSED-seed; fresh paired tags and predecessor admission; stop evidence before path-free root request; existing root record required for current-tag replay 00755911d37655eea3449e3fd69c884dd955a4f7d67487efa7cc6b9b9c7514f9 |
| +internal/host/claimservingretire.go claimServingArtifactRetirement.snapshot authority C,J4 rev3=PROPOSED-seed; fresh claim and nested database path; paired inventory stamps, normalized paths and ancestor conflicts before root requests file:Read@908 202138f761733b9853154f6b9382e68d74e4e933a600949cdd284a9cf6df765a |
| +internal/host/claimservingretire.go claimServingArtifactRetirement.stopped authority C,J4 rev3=PROPOSED-seed; completed serving fence receipts from the existing Guard backend; exact predecessor and byte-identical stop evidence ea358c5594a469b11d6255df9da2f0567f5d911fb94e1a3702f7cdc4f60c7751 |
| +internal/host/claimservingretire.go claimServingArtifactRetirement.validateAuthorization authority C,J4 rev3=PROPOSED-seed; root-returned retirement identity, original and continuing successor tuples, tag source stamp and source-directory identity 8324a81ca00f5d6cf79a38560b7a99f4de65d203545074b24bc2144fc26ac52d |
| +internal/host/claimservingretire.go servingInventoryPair authority C,J4 rev3=PROPOSED-seed; normalized inventory paths and stamps; paired ownership and ancestor-conflict validation shared by preflight and CAS 700cd20c58f0bfb130c54810a5baf1f1df0f89f8e4cce16716d7961cde4d554a |
| +internal/host/claimservingretire.go servingInventoryTransition authority C,J4 rev3=PROPOSED-seed; claim writer identity and complete local inventory; changes only database/marker pair and preserves other paths and boxes 2f4024d4d05b348e4d2b9e2fa4e25bbda69575fa8ac495c4f986444527425d48 |
| +internal/host/claimservingretire.go servingPredecessor authority C,J4 rev3=PROPOSED-seed; retained local cold admission at the strictly older installation stamp; exact generation, box and region e5d5534da88e3bbbc051c8f3c19c9cfd14918d84bd6a0206faf87a2a612c3a57 |
| +internal/host/claimservingretire.go servingRetirementID authority C,J4 rev3=PROPOSED-seed; immutable predecessor and original successor tuples; deterministic journal identity independent of claim operation IDs c984974d17e621d7fe9c276a03ae789b86789abbccbb82a09d06f55c3bf13819 |
| +internal/host/claimservingretire.go servingWriterIdentity authority C,J4 rev3=PROPOSED-seed; authoritative serving holder and unique cold admission; exact local executor, stamp, session, generation, incarnation, location and region file:Exec@403 ae2fa116ece3bbd596baa02971ef69fba597ce2ab4135abb6bf9caae6d326202 |
| +internal/hostbox/servingpark.go LaunchJournal.RootSessionLaunches authority C,J4 rev3=PROPOSED-seed; root-local persisted launch records for exact host/incarnation/stamp/session/box/enrollment; refuses agent journal transport cbbb9378bc53ac499972bed5fb654e05856a9afd7256d55323a717140ffcba18 |
| +@coverage internal/host 0e48977bb98fe6f8c707566b661fd034cda83afeb408512820466e0d31f972e7 |
| +@coverage internal/hostbox 72c9b640d98c779a044579f70a103a68cd9819c794a0bb3e53f378d39ad11419 |
| @coverage internal/hostguard 51408b42f01be94a232e395494e5d154cc57736e32a6519ac6c03e9ff3d091e6 |