amadan

carlos / platform Public

WIP files seed checkpoint 27 (not for merge)

Second isolated launch PID race fixed; two-host isolation across retirement and reboot; whole-executor kill and rerun; Task 7 runbook strings drafted for review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paul Campbell pushed by claude 2a8d29f2cbedebde33c69645e6ad5568a8b40149 parent a107047
18 files changed, +1182 −82
  • internal/host/claimfilesseed_cli_rootrunner_linux_test.go +315 −0
  • internal/host/claimfilesseed_crash_rootrunner_linux_test.go +3 −0
  • internal/host/claimfilesseed_integration_linux_test.go +5 −2
  • internal/host/claimfilesseed_isolated_rootrunner_linux_test.go +92 −12
  • internal/host/claimfilesseed_lifetime_rootrunner_linux_test.go +181 −0
  • internal/host/claimfilesseed_live_rootrunner_linux_test.go +10 −1
  • internal/host/claimfilesseed_reboot_rootrunner_linux_test.go +294 −0
  • internal/host/claimfilesseed_restore_linux_test.go +58 −45
  • internal/host/claimfilesseed_twohost_rootrunner_linux_test.go +157 −0
  • internal/host/claimretryrecovery_test.go +18 −6
  • internal/host/claimservinglegacyroot_linux_test.go +6 −2
  • internal/host/claimservingpeer_rootrunner_linux_test.go +8 −0
  • internal/host/claimservingretire_linux_test.go +3 −3
  • internal/host/plan12_restore_linux_test.go +1 −1
  • internal/instancectl/claimexec.go +16 −3
  • internal/instancectl/claimexec_rev3_linux_test.go +10 −5
  • internal/instancectl/claimunit.go +1 −1
  • internal/provision/testdata/plan14-root-reads-seed.tsv +4 −1

CI passed — run details

CI log 
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	282.185s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	2.544s
ok  	github.com/carlosframework/platform/cmd/moveprobe	93.157s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.702s
ok  	github.com/carlosframework/platform/internal/activator	250.773s
ok  	github.com/carlosframework/platform/internal/agentsock	0.014s
ok  	github.com/carlosframework/platform/internal/alerting	0.003s
ok  	github.com/carlosframework/platform/internal/analytics	0.020s
ok  	github.com/carlosframework/platform/internal/appconfig	2.438s
ok  	github.com/carlosframework/platform/internal/apperr	0.069s
ok  	github.com/carlosframework/platform/internal/applog	0.127s
ok  	github.com/carlosframework/platform/internal/awsinv	0.004s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.004s
ok  	github.com/carlosframework/platform/internal/boxid	0.992s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.008s
ok  	github.com/carlosframework/platform/internal/caps	0.015s
ok  	github.com/carlosframework/platform/internal/certwarm	0.002s
ok  	github.com/carlosframework/platform/internal/cgroup	0.691s
ok  	github.com/carlosframework/platform/internal/changelog	0.300s
ok  	github.com/carlosframework/platform/internal/clamd	0.054s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	385.576s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	5.977s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.010s
ok  	github.com/carlosframework/platform/internal/dbverify	0.004s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.003s
ok  	github.com/carlosframework/platform/internal/docsite	0.005s
ok  	github.com/carlosframework/platform/internal/domains	2.312s
ok  	github.com/carlosframework/platform/internal/doorkey	0.006s
ok  	github.com/carlosframework/platform/internal/economics	0.004s
ok  	github.com/carlosframework/platform/internal/edge	26.591s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.008s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.129s
ok  	github.com/carlosframework/platform/internal/gate	1.655s
ok  	github.com/carlosframework/platform/internal/genselect	0.015s
ok  	github.com/carlosframework/platform/internal/geo	0.019s
ok  	github.com/carlosframework/platform/internal/geoip	0.034s
ok  	github.com/carlosframework/platform/internal/host	157.708s
ok  	github.com/carlosframework/platform/internal/hostactors	0.467s
ok  	github.com/carlosframework/platform/internal/hostbox	2.281s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.483s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.005s
ok  	github.com/carlosframework/platform/internal/hostexec	4.695s
ok  	github.com/carlosframework/platform/internal/hostfence	0.039s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.486s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.014s
ok  	github.com/carlosframework/platform/internal/hostguard	0.969s
ok  	github.com/carlosframework/platform/internal/hostops	1.532s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.002s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.002s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.433s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.124s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.002s
ok  	github.com/carlosframework/platform/internal/infraview	0.001s
ok  	github.com/carlosframework/platform/internal/instancectl	4.165s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.721s
ok  	github.com/carlosframework/platform/internal/instancefs	0.005s
ok  	github.com/carlosframework/platform/internal/instancekey	0.015s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.021s
ok  	github.com/carlosframework/platform/internal/layout	1.510s
ok  	github.com/carlosframework/platform/internal/lease	9.797s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.017s
ok  	github.com/carlosframework/platform/internal/ledger	0.014s
ok  	github.com/carlosframework/platform/internal/mail	0.148s
ok  	github.com/carlosframework/platform/internal/maintenance	0.075s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.505s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.066s
ok  	github.com/carlosframework/platform/internal/metrics	0.004s
ok  	github.com/carlosframework/platform/internal/moves	20.922s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.763s
ok  	github.com/carlosframework/platform/internal/objstore	0.160s
ok  	github.com/carlosframework/platform/internal/ops	0.174s
ok  	github.com/carlosframework/platform/internal/park	0.197s
ok  	github.com/carlosframework/platform/internal/privatefile	0.003s
ok  	github.com/carlosframework/platform/internal/provision	15.766s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.006s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	4.910s
ok  	github.com/carlosframework/platform/internal/relay	7.233s
ok  	github.com/carlosframework/platform/internal/release	19.702s
ok  	github.com/carlosframework/platform/internal/rollout	3.650s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.003s
ok  	github.com/carlosframework/platform/internal/rootaws	0.006s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.007s
ok  	github.com/carlosframework/platform/internal/rootstate	0.024s
ok  	github.com/carlosframework/platform/internal/runs	0.569s
ok  	github.com/carlosframework/platform/internal/s3compat	0.004s
ok  	github.com/carlosframework/platform/internal/safefile	0.004s
ok  	github.com/carlosframework/platform/internal/schedule	2.448s
ok  	github.com/carlosframework/platform/internal/seeds	0.011s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	8.889s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.354s
ok  	github.com/carlosframework/platform/internal/status	0.037s
ok  	github.com/carlosframework/platform/internal/steering	0.005s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.216s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.002s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.004s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.535s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 46989474 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: cached, digest verified
  mc: cached, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.022s
=== RUN   TestT4StorageProbeMinio
2026/10/15 17:56:03 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DEC59926FAF3E7, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.020s
minio contract: PASS
diff --git a/internal/host/claimfilesseed_cli_rootrunner_linux_test.go b/internal/host/claimfilesseed_cli_rootrunner_linux_test.go
new file mode 100644
index 000000000..97ca3440a
--- /dev/null
+++ b/internal/host/claimfilesseed_cli_rootrunner_linux_test.go
@@ -0,0 +1,315 @@
+//go:build linux && rootrunner
+
+package host
+
+import (
+ "context"
+ "crypto/ed25519"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "net"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "strconv"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/cgroup"
+ "github.com/carlosframework/platform/internal/controlsock"
+ "github.com/carlosframework/platform/internal/hostactors"
+ "github.com/carlosframework/platform/internal/hostbox"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostfiles"
+ "github.com/carlosframework/platform/internal/hostguard"
+ "github.com/carlosframework/platform/internal/hostops"
+ "github.com/carlosframework/platform/internal/instancectl"
+ "github.com/carlosframework/platform/internal/lease"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+type seedCLIConfiguration struct {
+ Actor hostbox.Actor
+ Claims, Executable string
+}
+
+func TestRootFilesSeedDelegatedOneShot(t *testing.T) {
+ runServingParkPeerNamespace(t, false, false, "seed-cli")
+}
+
+func rootFilesSeedDelegatedCLI(t *testing.T, binary string) {
+ for _, p := range []string{"/data", "/var/lib/carlos-claims", "/run/carlos-control", "/run/carlos", "/opt/carlos"} {
+ if err := os.Mkdir(p, 0755); err != nil {
+ t.Fatal("existing CLI fixture", p, err)
+ }
+ t.Cleanup(func() { os.RemoveAll(p) })
+ }
+ var captured *servingPreparation
+ var executor *filesSeedExecutor
+ capture := filesSeedLiveWitness{seed: func(_ *testing.T, p *servingPreparation, e *filesSeedExecutor) { captured, executor = p, e }}
+ runFilesSeedPreparationWithLive(t, "real-seed", "same-controller", []bool{true}, []filesSeedLiveWitness{capture})
+ f := captured.f
+ f.actor.ControlBucket = "deployment"
+ cfg := seedCLIConfiguration{Actor: f.actor, Claims: "/data/cli-claims", Executable: "/opt/carlos/versions/fixture/app"}
+ claims := release.NewDirStore(cfg.Claims)
+ keys, err := f.f.claims.List(t.Context(), "")
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, key := range keys {
+ b, _, err := f.f.claims.Get(t.Context(), key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := claims.Put(t.Context(), key, b, ""); err != nil {
+ t.Fatal(err)
+ }
+ }
+ f.f.claims, f.f.evidence = claims, claims
+ f.f.guard.Claims = hostguard.StoreReader{Store: claims}
+ f.f.guard.LockDir = ""
+ f.p.root.Stop.Coordinator.Claims = hostguard.StoreReader{Store: claims}
+ f.p.root.Stop.Coordinator.Guard = f.f.guard
+ f.p.root.Stop.Coordinator.Evidence = claims
+ f.p.root.Stop.Coordinator.Now = time.Now
+ production, err := NewProductionClaimStopHandler(f.p.root.Stop.Coordinator, f.f.journal, productionStopControllers{}, f.p.root.Roots)
+ if err != nil {
+ t.Fatal(err)
+ }
+ f.p.root.Stop = production.(ClaimStopHTTP)
+ // Initial installation/roster and the no-process kernel census are fixtures.
+ f.f.kernel.noProcess = true
+ f.p.root.Stop.Coordinator.Kernel = f.f.kernel
+ f.p.root.Stop.Coordinator.RetainNoProcess = f.p.retainLegacyNoProcess
+ h := servingTransportRoot(t, f)
+ capability := newLaunchFixture(t, hostclaim.Op{Kind: hostclaim.KindServing, Exec: f.actor.Box}).root
+ capability.Authenticate = h.Stop.Authenticate
+ capability.SeedController = verifyFilesSeedDelegation
+ listener, err := net.Listen("unix", controlsock.Path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Chmod(controlsock.Path, 0666); err != nil {
+ t.Fatal(err)
+ }
+ server := &instancectl.Server{ClaimStop: ClaimRootWithLaunch{Root: h, Launch: capability}, PeerOK: func(net.Conn) error { return nil }, Logf: t.Logf}
+ ctx, cancel := context.WithCancel(context.Background())
+ done := make(chan error, 1)
+ go func() { done <- server.Serve(ctx, listener) }()
+ t.Cleanup(func() {
+ cancel()
+ if err := <-done; err != nil {
+ t.Error(err)
+ }
+ })
+ reg, err := registry.Open("/data/registry.db")
+ if err != nil {
+ t.Fatal(err)
+ }
+ row, err := executor.registry.Get(recoveryHost)
+ if err != nil || row == nil {
+ t.Fatal(err)
+ }
+ if err := reg.Upsert(*row); err != nil {
+ t.Fatal(err)
+ }
+ reg.Close()
+ if err := os.MkdirAll(filepath.Dir(cfg.Executable), 0755); err != nil {
+ t.Fatal(err)
+ }
+ image, err := os.ReadFile(binary)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(cfg.Executable, image, 0755); err != nil {
+ t.Fatal(err)
+ }
+ raw, _ := json.Marshal(cfg)
+ if err := os.WriteFile("/data/seed-cli.json", raw, 0600); err != nil {
+ t.Fatal(err)
+ }
+ account, err := user.Lookup("carlos")
+ if err != nil {
+ t.Fatal(err)
+ }
+ uid, _ := strconv.Atoi(account.Uid)
+ gid, _ := strconv.Atoi(account.Gid)
+ seedCLISharedLocks(t, "/run/carlos/hostguard", uid, gid)
+ if err := filepath.Walk("/data", func(path string, info os.FileInfo, err error) error {
+ if err != nil {
+ return err
+ }
+ return os.Chown(path, uid, gid)
+ }); err != nil {
+ t.Fatal(err)
+ }
+ raw, err = unitsFS.ReadFile("units/carlos-edge.service")
+ if err != nil {
+ t.Fatal(err)
+ }
+ var lines []string
+ for _, line := range strings.Split(string(raw), "\n") {
+ if strings.HasPrefix(line, "Requires=") || strings.HasPrefix(line, "After=") || strings.HasPrefix(line, "Wants=") || strings.HasPrefix(line, "ExecStart=") || strings.HasPrefix(line, "ExecReload=") || strings.HasPrefix(line, "EnvironmentFile=") || strings.HasPrefix(line, "Restart=") {
+ continue
+ }
+ lines = append(lines, line)
+ }
+ lines = append(lines, "[Service]", "Type=oneshot", "RemainAfterExit=yes", "Restart=no", "TimeoutStartSec=60s", "TimeoutStopSec=45s", "ExecStart="+binary+" -test.run=^TestRootFilesSeedCLIPayload$ -test.v -test.timeout=50s", "Environment=FILES_SEED_CLI=1 TMPDIR=/data CARLOS_HOST_CLAIM=1 CARLOS_HOST_CLAIM_ORIGINS=1 CARLOS_DATA_DIR=/data CARLOS_DEPLOYMENT_DIR=/data/cli-claims AWS_REGION=eu-west-1 AWS_DEFAULT_REGION=eu-west-1 AWS_ACCESS_KEY_ID=test-key AWS_SECRET_ACCESS_KEY=test-secret AWS_EC2_METADATA_DISABLED=true CARLOS_S3_ENDPOINT="+os.Getenv("CARLOS_S3_ENDPOINT"), "IPAddressDeny=any", "IPAddressAllow=127.0.0.1")
+ unit := "/run/systemd/system/carlos-edge.service"
+ file, err := os.OpenFile(unit, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0644)
+ if err != nil {
+ t.Fatal(err)
+ }
+ _, err = file.WriteString(strings.Join(lines, "\n") + "\n")
+ file.Close()
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() {
+ exec.Command("systemctl", "stop", "carlos-edge.service").Run()
+ os.Remove(unit)
+ exec.Command("systemctl", "daemon-reload").Run()
+ })
+ servingRunnerSystemctl(t, "daemon-reload")
+ run := func() {
+ out, err := exec.CommandContext(t.Context(), "systemctl", "start", "carlos-edge.service").CombinedOutput()
+ logs, _ := exec.Command("journalctl", "-u", "carlos-edge.service", "--no-pager").CombinedOutput()
+ t.Log(string(logs))
+ if err != nil {
+ t.Fatal("delegated CLI", err, string(out))
+ }
+ state, err := exec.Command("systemctl", "show", "carlos-edge.service", "-p", "ExecMainStatus", "-p", "Result", "-p", "SubState").Output()
+ if err != nil || !strings.Contains(string(state), "ExecMainStatus=0") || !strings.Contains(string(state), "Result=success") {
+ t.Fatal("one-shot status", string(state), err)
+ }
+ }
+ run()
+ observed := f.f.read()
+ if observed.Claim.Origins == nil || observed.Claim.Holder.WakeRetry == nil || observed.Claim.Holder.WakeRetry.AttemptsReserved != 1 || observed.Claim.Holder.WakeRetry.State != "open" {
+ t.Fatal("one-shot did not settle publication", observed)
+ }
+ files, err := executor.source.Open(t.Context(), claims, executor.assembly.Data, f.actor)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, found, err := hostfiles.Load(t.Context(), files.(*movesFiles).Files, movesFilesPrefix(observed.Claim.Holder.Location, observed.Claim.Holder.Generation)); err != nil || !found {
+ t.Fatal("CLI did not publish", err)
+ }
+ servingRunnerSystemctl(t, "stop", "carlos-edge.service")
+ etag := observed.ETag
+ run()
+ if f.f.read().ETag != etag {
+ t.Fatal("delegated established rerun mutated claim")
+ }
+}
+
+func TestRootFilesSeedCLIPayload(t *testing.T) {
+ if os.Getenv("FILES_SEED_CLI") != "1" || os.Geteuid() == 0 {
+ t.Fatal("invoke through delegated CLI fixture")
+ }
+ body, err := os.ReadFile("/data/seed-cli.json")
+ if err != nil {
+ t.Fatal(err)
+ }
+ var cfg seedCLIConfiguration
+ if err := json.Unmarshal(body, &cfg); err != nil {
+ t.Fatal(err)
+ }
+ claimOwnerRegistration.Provider = seedCLIOwners{t: t, cfg: cfg}
+ RunSystemFiles([]string{"seed", "--host", recoveryHost})
+}
+
+type seedCLIOwners struct {
+ t *testing.T
+ cfg seedCLIConfiguration
+}
+
+func (o seedCLIOwners) OpenClaimOwners(ctx context.Context, store release.Store, r ClaimRuntimeResources) (ClaimOwnerBindings, error) {
+ _, _, defaults, _ := fix4JournalFixture(o.t)
+ owners := defaults.ClaimOwnerBindings
+ owners.Identity = handoverRecoveryIdentity(o.t, o.cfg.Actor)
+ owners.EvidenceKey = nil
+ owners.RetirementAuthority = &hostops.ProductionAuthority{Store: store, Directory: hostactors.Directory{Store: store, PublicKey: ed25519.NewKeyFromSeed(make([]byte, 32)).Public().(ed25519.PublicKey), Bucket: o.cfg.Actor.ControlBucket}, Now: time.Now}
+ data, err := lease.NewVersionedS3(ctx, map[string]string{o.cfg.Actor.Bucket: "eu-west-1"})
+ if err != nil {
+ return owners, err
+ }
+ owners.Data = data
+ owners.Local = &rev2Local{}
+ owners.Controllers = productionStopControllers{}
+ fixture := plan9D5Fixture(o.t)
+ fixture.launch.Executable = o.cfg.Executable
+ image, err := os.ReadFile(o.cfg.Executable)
+ if err != nil {
+ return owners, err
+ }
+ sum := sha256.Sum256(image)
+ fixture.launch.ExecutableSHA256 = hex.EncodeToString(sum[:])
+ fixture.launch.Args = []string{"--socket", provision.NestedSocket(recoveryHost), "--db", filepath.Join(r.DataDir, "instances", recoveryHost, recoveryHost+".db")}
+ fixture.launch.Limit = cgroup.TierDefault
+ fixture.launch.Env = []string{"CARLOS_ADMIN_TOKEN=test"}
+ owners.Serving = fixture.owners.Sources
+ files, err := productionClaimFilesSource().Open(ctx, store, data, o.cfg.Actor)
+ if err != nil {
+ return owners, err
+ }
+ owners.Serving.Files = files
+ return owners, nil
+}
+
+func TestFilesSeedCLIStopFixture(t *testing.T) {
+ var p *servingPreparation
+ capture := filesSeedLiveWitness{seed: func(_ *testing.T, got *servingPreparation, _ *filesSeedExecutor) { p = got }}
+ runFilesSeedPreparationWithLive(t, "real-seed", "same-controller", nil, []filesSeedLiveWitness{capture})
+ f := p.f
+ claims := release.NewDirStore(filepath.Join(t.TempDir(), "claims"))
+ keys, err := f.f.claims.List(t.Context(), "")
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, key := range keys {
+ body, _, err := f.f.claims.Get(t.Context(), key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := claims.Put(t.Context(), key, body, ""); err != nil {
+ t.Fatal(err)
+ }
+ }
+ f.f.claims, f.f.evidence = claims, claims
+ f.f.guard.Claims = hostguard.StoreReader{Store: claims}
+ f.f.guard.LockDir = filepath.Join(t.TempDir(), "hostguard")
+ seedCLISharedLocks(t, f.f.guard.LockDir, os.Getuid(), os.Getgid())
+ c := f.p.root.Stop.Coordinator
+ c.Claims, c.Guard, c.Evidence = hostguard.StoreReader{Store: claims}, f.f.guard, claims
+ c.Now = time.Now
+ f.f.kernel.noProcess = true
+ c.Kernel = f.f.kernel
+ f.p.root.Stop.Coordinator = c
+ c.RetainNoProcess = f.p.retainLegacyNoProcess
+ o := f.f.read()
+ h := o.Claim.Holder
+ req := StopRequest{Region: f.actor.Region, Host: o.Host, Incarnation: h.Incarnation, InstallationStamp: h.Stamp, Box: f.actor.Box, Enrollment: f.actor.Enrollment, BootID: f.f.peer.BootID, ControllerID: f.f.peer.ControllerID, ExpectedClaimETag: o.ETag, ExpectedDirectoryRevision: strconv.FormatUint(f.actor.Revision, 10)}
+ if _, err := c.FenceAndStop(t.Context(), req); err != nil {
+ t.Fatal("CLI fixture no-process", err)
+ }
+}
+
+func seedCLISharedLocks(t *testing.T, path string, uid, gid int) {
+ t.Helper()
+ // Match bootstrap's carlos-owned, setgid shared-lock directory.
+ if err := os.Mkdir(path, 0770); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Chown(path, uid, gid); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Chmod(path, os.ModeSetgid|0770); err != nil {
+ t.Fatal(err)
+ }
+}
diff --git a/internal/host/claimfilesseed_crash_rootrunner_linux_test.go b/internal/host/claimfilesseed_crash_rootrunner_linux_test.go
index c46a85a9a..6262e5f6d 100644
--- a/internal/host/claimfilesseed_crash_rootrunner_linux_test.go
+++ b/internal/host/claimfilesseed_crash_rootrunner_linux_test.go
@@ -36,6 +36,9 @@ type seedCrashFixture struct {
}
func rootFilesSeedWitnessFor(t *testing.T, binary string) filesSeedLiveWitness {
+ if strings.Contains(t.Name(), "Socket") {
+ return filesSeedLiveWitness{seed: rootFilesSeedLifetime}
+ }
if strings.Contains(t.Name(), "WholeExecutorKilledRerun") {
return filesSeedLiveWitness{seed: func(t *testing.T, p *servingPreparation, e *filesSeedExecutor) { rootFilesSeedCrash(t, binary, p, e) }}
}
diff --git a/internal/host/claimfilesseed_integration_linux_test.go b/internal/host/claimfilesseed_integration_linux_test.go
index 0d16d5265..d696aac5c 100644
--- a/internal/host/claimfilesseed_integration_linux_test.go
+++ b/internal/host/claimfilesseed_integration_linux_test.go
@@ -28,12 +28,12 @@ func runFilesSeedRealPreparation(t *testing.T, p *servingPreparation, live ...fi
f.actor.HomeRegion = "ie"
f.actor.FleetHomeRegion = "ie"
p.local.Actor = f.actor
- rt := registry.Route{Host: recoveryHost, Kind: "instance", Hibernate: true, Account: "bab", App: "jam", DBPath: f.db, Socket: provision.NestedSocket(recoveryHost)}
+ rt := registry.Route{Host: f.f.hostName(), Kind: "instance", Hibernate: true, Account: "bab", App: "jam", DBPath: f.db, Socket: provision.NestedSocket(f.f.hostName())}
reg, err := registry.Open(filepath.Join(t.TempDir(), "seed.db"))
if err != nil {
t.Fatal(err)
}
- defer reg.Close()
+ t.Cleanup(func() { reg.Close() })
if err := reg.Upsert(rt); err != nil {
t.Fatal(err)
}
@@ -198,6 +198,9 @@ func runFilesSeedRealPreparation(t *testing.T, p *servingPreparation, live ...fi
}
type filesSeedLiveWitness struct {
+ host string
+ shared bool
+ start func(*testing.T, *servingPreparation, hostclaim.Observed, activator.ClaimStart) func()
healthy bool
prepare func(*servingPreparation)
seed func(*testing.T, *servingPreparation, *filesSeedExecutor)
diff --git a/internal/host/claimfilesseed_isolated_rootrunner_linux_test.go b/internal/host/claimfilesseed_isolated_rootrunner_linux_test.go
index c420d79b2..78608e3cf 100644
--- a/internal/host/claimfilesseed_isolated_rootrunner_linux_test.go
+++ b/internal/host/claimfilesseed_isolated_rootrunner_linux_test.go
@@ -70,8 +70,27 @@ func (c seedIsolatedClient) Stop(ctx context.Context, q hostbox.RoleLaunchReques
}
func rootFilesSeedIsolated(t *testing.T, binary string) {
+ witness := rootFilesSeedIsolatedWitness(t, binary)
+ runFilesSeedPreparationWithLive(t, "real-seed", "same-controller", []bool{true}, []filesSeedLiveWitness{witness})
+}
+
+func rootFilesSeedIsolatedWitness(t *testing.T, binary string, twoHosts ...bool) filesSeedLiveWitness {
+ hosts := []string{"seed-isolated.bab.oncarlos.com"}
+ if len(twoHosts) > 0 {
+ hosts = seedTwoHostNames()
+ }
+ for _, host := range hosts {
+ path := "/run/systemd/system.control/" + instancectl.SliceName(host) + ".d"
+ if _, err := os.Lstat(path); !os.IsNotExist(err) {
+ t.Fatal("refuse existing slice properties", path, err)
+ }
+ }
for _, dir := range []string{"/data", "/var/lib/carlos-claims", "/opt/carlos", "/run/carlos", "/run/carlos-control", "/etc/carlos"} {
- if err := os.Mkdir(dir, 0755); err != nil {
+ if len(twoHosts) > 1 && (dir == "/data" || dir == "/var/lib/carlos-claims" || dir == "/opt/carlos" || dir == "/etc/carlos") {
+ if _, err := os.Stat(dir); err != nil {
+ t.Fatal("missing reboot fixture", dir, err)
+ }
+ } else if err := os.Mkdir(dir, 0755); err != nil {
t.Fatal("refuse existing fixture path", dir, err)
}
t.Cleanup(func() { os.RemoveAll(dir) })
@@ -108,6 +127,9 @@ func rootFilesSeedIsolated(t *testing.T, binary string) {
}
lines = append(lines, line)
}
+ if os.Getenv("FILES_SEED_ISOLATION_MUTATION") == "1" {
+ lines = append(lines, "[Service]", "BindReadOnlyPaths=/etc/carlos/fixture-secret:/tmp/exposed-root")
+ }
unit := "/run/systemd/system/carlos-instance@.service"
file, err := os.OpenFile(unit, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0644)
if err != nil {
@@ -120,11 +142,38 @@ func rootFilesSeedIsolated(t *testing.T, binary string) {
t.Cleanup(func() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
- if out, err := exec.CommandContext(ctx, "systemctl", "stop", instancectl.UnitName(instancectl.RoleRun, recoveryHost)).CombinedOutput(); err != nil {
- t.Errorf("stop isolated tenant: %v %s", err, out)
+ for _, host := range hosts {
+ if out, err := exec.CommandContext(ctx, "systemctl", "stop", instancectl.UnitName(instancectl.RoleRun, host)).CombinedOutput(); err != nil {
+ t.Errorf("stop isolated tenant: %v %s", err, out)
+ }
+ os.RemoveAll("/run/systemd/system/" + instancectl.UnitName(instancectl.RoleRun, host) + ".d")
+ slice := instancectl.SliceName(host)
+ if out, err := exec.CommandContext(ctx, "systemctl", "stop", slice).CombinedOutput(); err != nil {
+ t.Errorf("stop isolated slice: %v %s", err, out)
+ }
+ state, err := exec.CommandContext(ctx, "systemctl", "show", slice, "--property=ActiveState", "--value").Output()
+ if err != nil || strings.TrimSpace(string(state)) != "inactive" {
+ t.Errorf("isolated slice survived cleanup: %s %v", state, err)
+ }
+ exec.CommandContext(ctx, "systemctl", "reset-failed", instancectl.UnitName(instancectl.RoleRun, host)).Run()
+ path := "/run/systemd/system.control/" + slice + ".d"
+ if err := os.RemoveAll(path); err != nil {
+ t.Error("remove fixture slice properties", err)
+ }
+ if _, err := os.Lstat(path); !os.IsNotExist(err) {
+ t.Error("fixture slice properties survived", err)
+ }
+ }
+ for _, slice := range []string{"carlos-instances.slice", "carlos.slice"} {
+ if out, err := exec.CommandContext(ctx, "systemctl", "stop", slice).CombinedOutput(); err != nil {
+ t.Errorf("stop isolated parent slice: %v %s", err, out)
+ }
+ state, err := exec.CommandContext(ctx, "systemctl", "show", slice, "--property=ActiveState", "--value").Output()
+ if err != nil || strings.TrimSpace(string(state)) != "inactive" {
+ t.Errorf("isolated parent slice survived cleanup: %s %v", state, err)
+ }
}
os.Remove(unit)
- os.RemoveAll("/run/systemd/system/" + instancectl.UnitName(instancectl.RoleRun, recoveryHost) + ".d")
if out, err := exec.CommandContext(ctx, "systemctl", "daemon-reload").CombinedOutput(); err != nil {
t.Errorf("reload isolated unit: %v %s", err, out)
}
@@ -149,6 +198,7 @@ func rootFilesSeedIsolated(t *testing.T, binary string) {
}
})
witness := rootFilesSeedLiveWitness(t, app)
+ witness.host = "seed-isolated.bab.oncarlos.com"
configure := witness.configure
witness.configure = func(w *ClaimWakeAssembly) {
configure(w)
@@ -156,7 +206,20 @@ func rootFilesSeedIsolated(t *testing.T, binary string) {
selected := owners.Sources.Launch
owners.Sources.Launch = func(ctx context.Context, rt registry.Route) (ClaimServingLaunch, error) {
in, err := selected(ctx, rt)
- in.Env = append(in.Env, "FILES_SEED_DENIED="+strings.Join(denied, ":"))
+ paths := append([]string(nil), denied...)
+ if os.Getenv("FILES_SEED_ISOLATION_MUTATION") == "1" {
+ paths = append(paths, "/tmp/exposed-root")
+ }
+ if len(twoHosts) > 0 {
+ pair := seedTwoHostNames()
+ other := pair[1]
+ if rt.Host == other {
+ other = pair[0]
+ }
+ paths = append(paths, provision.NestedDBPath("/data", other), filepath.Join("/data/instances", other, other, "payload"))
+ in.Env = append(in.Env, "FILES_SEED_EXPECTED=real seeded bytes:"+rt.Host)
+ }
+ in.Env = append(in.Env, "FILES_SEED_DENIED="+strings.Join(paths, ":"))
return in, err
}
}
@@ -209,6 +272,7 @@ func rootFilesSeedIsolated(t *testing.T, binary string) {
}
units := instancectl.ClaimUnits{Systemctl: func(ctx context.Context, args ...string) (string, error) {
out, err := exec.CommandContext(ctx, "systemctl", args...).CombinedOutput()
+ t.Logf("isolated systemctl %q: %s (%v)", args, out, err)
return string(out), err
}, CgroupRoot: "/sys/fs/cgroup", ExeRoot: "/run/carlos-control/exe", UnitRoot: "/run/systemd/system", DataRoot: "/data"}
root := &ClaimLaunchRoot{Guard: p.f.f.guard, Journal: p.f.f.journal, DataRoot: "/data", ExecLockDir: t.TempDir(), Local: &rev2Local{}, Clock: execlease.SystemClock{}, Actor: func(context.Context) (hostbox.Actor, error) { return p.f.actor, nil }, Units: units, Kernel: linuxClaimLaunchKernel{OptRoot: "/opt/carlos", CgroupRoot: "/sys/fs/cgroup", ProcRoot: "/proc"}, ValidateExecutable: func(string, string, string) error { return nil }, executionState: state}
@@ -248,7 +312,7 @@ func rootFilesSeedIsolated(t *testing.T, binary string) {
t.Fatal(err)
}
}
- witness.serve = func(t *testing.T, p *servingPreparation, o hostclaim.Observed, start activator.ClaimStart) {
+ witness.start = func(t *testing.T, p *servingPreparation, o hostclaim.Observed, start activator.ClaimStart) func() {
launch, err := start.PrepareLaunch(t.Context())
if err != nil {
t.Fatal("prepare isolated launch", err)
@@ -258,14 +322,22 @@ func rootFilesSeedIsolated(t *testing.T, binary string) {
out, _ := exec.Command("journalctl", "-u", instancectl.UnitName(instancectl.RoleRun, o.Host), "--no-pager", "-n", "30").CombinedOutput()
t.Fatalf("isolated launch: %v\n%s", err, out)
}
- defer func() {
- if err := settle(t.Context()); err != nil {
- t.Error(err)
+ stopped := false
+ stop := func() {
+ if !stopped {
+ stopped = true
+ if err := settle(context.Background()); err != nil {
+ t.Error(err)
+ }
}
- }()
+ }
+ t.Cleanup(stop)
ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
defer cancel()
if err := start.Ready(ctx, cmd); err != nil {
+ if body, readErr := os.ReadFile(filepath.Join(filepath.Dir(p.f.db), o.Host, "payload.isolation-violation")); readErr == nil {
+ t.Fatalf("isolation mutation detected: %s", body)
+ }
t.Fatal("isolated readiness", err)
}
wrong, _ := os.FindProcess(os.Getpid())
@@ -285,7 +357,11 @@ func rootFilesSeedIsolated(t *testing.T, binary string) {
}
body, err := io.ReadAll(response.Body)
response.Body.Close()
- if err != nil || response.StatusCode != 200 || string(body) != "real seeded bytes" {
+ expected := "real seeded bytes"
+ if len(twoHosts) > 0 {
+ expected += ":" + o.Host
+ }
+ if err != nil || response.StatusCode != 200 || string(body) != expected {
t.Fatal("isolated payload", string(body), err)
}
for _, path := range denied {
@@ -294,6 +370,10 @@ func rootFilesSeedIsolated(t *testing.T, binary string) {
}
}
t.Log(fmt.Sprintf("healthy isolated wake: host=%s session=%d pid=%d", o.Host, o.Claim.Holder.Session, cmd.Process.Pid))
+ return stop
}
- runFilesSeedPreparationWithLive(t, "real-seed", "same-controller", []bool{true}, []filesSeedLiveWitness{witness})
+ witness.serve = func(t *testing.T, p *servingPreparation, o hostclaim.Observed, start activator.ClaimStart) {
+ witness.start(t, p, o, start)()
+ }
+ return witness
}
diff --git a/internal/host/claimfilesseed_lifetime_rootrunner_linux_test.go b/internal/host/claimfilesseed_lifetime_rootrunner_linux_test.go
new file mode 100644
index 000000000..469c8dbdb
--- /dev/null
+++ b/internal/host/claimfilesseed_lifetime_rootrunner_linux_test.go
@@ -0,0 +1,181 @@
+//go:build linux && rootrunner
+
+package host
+
+import (
+ "context"
+ "encoding/json"
+ "io"
+ "net"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/activator"
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostfiles"
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+func TestRootFilesSeedSocketDisconnectExecutor(t *testing.T) {
+ runRootFilesSeedRealOriginPreparation(t, true)
+}
+func TestRootFilesSeedSocketShutdownExecutor(t *testing.T) {
+ runRootFilesSeedRealOriginPreparation(t, true)
+}
+func TestRootFilesSeedSocketCompleteExecutor(t *testing.T) {
+ runRootFilesSeedRealOriginPreparation(t, true)
+}
+
+func rootFilesSeedLifetime(t *testing.T, p *servingPreparation, e *filesSeedExecutor) {
+ blocked := !strings.Contains(t.Name(), "Complete")
+ entered := make(chan struct{})
+ factory := e.assembly.Wake.seedFactory
+ e.assembly.Wake.seedFactory = func(ctx context.Context, op *filesSeedOperation, token *activator.SeedExclusion) (*activator.ClaimRuntime, *ClaimWakeAssembly, error) {
+ runtime, wake, err := factory(ctx, op, token)
+ if err != nil {
+ return runtime, wake, err
+ }
+ prepare := runtime.Prepare
+ runtime.Prepare = func(ctx context.Context, rt registry.Route, o hostclaim.Observed, policy activator.StartPolicy) (activator.ClaimStart, error) {
+ start, err := prepare(ctx, rt, o, policy)
+ if err == nil && blocked {
+ start.Materialize = func(ctx context.Context) error { close(entered); <-ctx.Done(); return ctx.Err() }
+ }
+ return start, err
+ }
+ return runtime, wake, nil
+ }
+ ready := make(chan struct{})
+ close(ready)
+ path := filepath.Join(t.TempDir(), "seed.sock")
+ if err := os.Chmod(filepath.Dir(path), 0700); err != nil {
+ t.Fatal(err)
+ }
+ // Peer and delegated service authentication have their own installed witness.
+ // Here the real executor is attached to the production connection lifetime.
+ open := func() *filesSeedServer {
+ s, err := startFilesSeedSocket(t.Context(), ready, path, func(context.Context, *net.UnixConn) error { return nil }, e.run)
+ if err != nil {
+ t.Fatal(err)
+ }
+ s.finalize = e.close
+ return s
+ }
+ s := open()
+ shutdown := func() {
+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+ defer cancel()
+ if err := s.close(ctx); err != nil {
+ t.Fatal("seed server did not join executor", err)
+ }
+ }
+ defer func() {
+ ctx, cancel := context.WithTimeout(context.Background(), time.Second)
+ defer cancel()
+ s.close(ctx)
+ }()
+ request := func() net.Conn {
+ c, err := net.DialTimeout("unix", path, time.Second)
+ if err != nil {
+ t.Fatal(err)
+ }
+ c.SetDeadline(time.Now().Add(10 * time.Second))
+ if err := json.NewEncoder(c).Encode(filesSeedRequest{Verb: "seed", Host: recoveryHost}); err != nil {
+ t.Fatal(err)
+ }
+ return c
+ }
+ c := request()
+ defer c.Close()
+ if blocked {
+ select {
+ case <-entered:
+ case <-time.After(10 * time.Second):
+ t.Fatal("seed did not reach precommit")
+ }
+ if err := e.run(t.Context(), recoveryHost, io.Discard); err != activator.ErrFenceHeld {
+ t.Fatal("second executor escaped exclusion", err)
+ }
+ if strings.Contains(t.Name(), "Disconnect") {
+ c.Close()
+ deadline := time.Now().Add(5 * time.Second)
+ for {
+ e.mu.Lock()
+ running := e.running
+ e.mu.Unlock()
+ if !running {
+ break
+ }
+ if time.Now().After(deadline) {
+ t.Fatal("executor survived client disconnect")
+ }
+ time.Sleep(10 * time.Millisecond)
+ }
+ }
+ shutdown()
+ e.mu.Lock()
+ retained, running := len(e.pending), e.running
+ e.mu.Unlock()
+ if retained != 0 || running {
+ t.Fatal("executor outlived server", retained, running)
+ }
+ o := p.f.f.read()
+ if o.Claim.Holder.WakeRetry == nil || o.Claim.Holder.WakeRetry.State != "open" || o.Claim.Holder.WakeRetry.AttemptsReserved != 1 {
+ t.Fatal("cancellation lost settlement", o)
+ }
+ files, err := e.source.Open(t.Context(), e.assembly.Claims, e.assembly.Data, p.f.actor)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, found, err := hostfiles.Load(t.Context(), files.(*movesFiles).Files, movesFilesPrefix(o.Claim.Holder.Location, o.Claim.Holder.Generation)); err != nil || found {
+ t.Fatal("canceled seed committed", found, err)
+ }
+ blocked = false
+ p.f.f.now = p.f.f.now.Add(time.Minute)
+ s = open()
+ c = request()
+ defer c.Close()
+ }
+ decoder := json.NewDecoder(c)
+ var text strings.Builder
+ for {
+ var reply filesSeedReply
+ if err := decoder.Decode(&reply); err != nil {
+ t.Fatal("seed reply", err, text.String())
+ }
+ text.WriteString(reply.Text)
+ if reply.Code != nil {
+ if *reply.Code != 0 {
+ t.Fatal("seed failed", *reply.Code, text.String())
+ }
+ break
+ }
+ }
+ shutdown()
+ o := p.f.f.read()
+ files, err := e.source.Open(t.Context(), e.assembly.Claims, e.assembly.Data, p.f.actor)
+ if err != nil {
+ t.Fatal(err)
+ }
+ manifest, found, err := hostfiles.Load(t.Context(), files.(*movesFiles).Files, movesFilesPrefix(o.Claim.Holder.Location, o.Claim.Holder.Generation))
+ if err != nil || !found || manifest.Entries["payload"].SHA256 == "" {
+ t.Fatal("seed did not publish", found, err)
+ }
+ if !strings.Contains(text.String(), "seeded.") {
+ t.Fatal(text.String())
+ }
+ if _, err := os.Lstat(path); !os.IsNotExist(err) {
+ t.Fatal("socket outlived server", err)
+ }
+}
+
+func TestFilesSeedExecutorLifetime(t *testing.T) {
+ for _, mode := range []string{"Complete", "Disconnect", "Shutdown"} {
+ t.Run(mode, func(t *testing.T) {
+ runFilesSeedPreparationWithLive(t, "real-seed", "same-controller", nil, []filesSeedLiveWitness{{seed: rootFilesSeedLifetime}})
+ })
+ }
+}
diff --git a/internal/host/claimfilesseed_live_rootrunner_linux_test.go b/internal/host/claimfilesseed_live_rootrunner_linux_test.go
index f777eab11..e297bb4d5 100644
--- a/internal/host/claimfilesseed_live_rootrunner_linux_test.go
+++ b/internal/host/claimfilesseed_live_rootrunner_linux_test.go
@@ -45,8 +45,13 @@ func TestRootFilesSeedTenantPayload(t *testing.T) {
for _, path := range strings.Split(os.Getenv("FILES_SEED_DENIED"), ":") {
if path != "" {
if body, err := os.ReadFile(path); err == nil {
+ _ = os.WriteFile(os.Getenv("FILES_SEED_PAYLOAD")+".isolation-violation", []byte(path), 0600)
t.Fatalf("tenant read foreign path %s: %q", path, body)
}
+ if f, err := os.OpenFile(path, os.O_WRONLY, 0); err == nil {
+ f.Close()
+ t.Fatalf("tenant opened foreign path for write: %s", path)
+ }
}
}
db, err := sql.Open("sqlite", *seedTenantDB)
@@ -59,7 +64,11 @@ func TestRootFilesSeedTenantPayload(t *testing.T) {
t.Fatal(integrity, err)
}
body, err := os.ReadFile(os.Getenv("FILES_SEED_PAYLOAD"))
- if err != nil || string(body) != "real seeded bytes" {
+ expected := os.Getenv("FILES_SEED_EXPECTED")
+ if expected == "" {
+ expected = "real seeded bytes"
+ }
+ if err != nil || string(body) != expected {
t.Fatal(string(body), err)
}
listener, err := net.Listen("unix", *seedTenantSocket)
diff --git a/internal/host/claimfilesseed_reboot_rootrunner_linux_test.go b/internal/host/claimfilesseed_reboot_rootrunner_linux_test.go
new file mode 100644
index 000000000..e3a9c890b
--- /dev/null
+++ b/internal/host/claimfilesseed_reboot_rootrunner_linux_test.go
@@ -0,0 +1,294 @@
+//go:build linux && rootrunner
+
+package host
+
+import (
+ "context"
+ "crypto/ed25519"
+ "encoding/json"
+ "fmt"
+ "net"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "sort"
+ "strconv"
+ "strings"
+ "syscall"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostbox"
+ "github.com/carlosframework/platform/internal/hostguard"
+ "github.com/carlosframework/platform/internal/lease"
+ "github.com/carlosframework/platform/internal/lease/leasetest"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+ "github.com/carlosframework/platform/internal/rootstate"
+)
+
+const seedRebootPath = "/var/lib/carlos-rootrunner-files-seed-reboot"
+
+type seedRebootObject struct {
+ Version lease.ObjectVersion
+ Body []byte
+}
+type seedRebootHost struct {
+ Host string
+ Fixture seedCrashFixture
+ Objects []seedRebootObject
+ Snapshot string
+ Records []servingParkRecord
+}
+type seedRebootSave struct {
+ Boot, Scratch, Install string
+ Hosts []seedRebootHost
+}
+
+func TestRootRebootFilesSeedTwoHostPhase1(t *testing.T) {
+ if err := os.Mkdir(seedRebootPath, 0700); err != nil {
+ t.Fatal("refuse existing reboot fixture", err)
+ }
+ runServingParkPeerNamespace(t, false, false, "seed-two-host")
+}
+
+func saveFilesSeedReboot(t *testing.T, binary string, hosts []*seedLiveHost) {
+ boot, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
+ if err != nil {
+ t.Fatal(err)
+ }
+ saved := seedRebootSave{Boot: strings.TrimSpace(string(boot)), Scratch: os.Getenv("TMPDIR"), Install: filepath.Dir(binary)}
+ for _, h := range hosts {
+ f := h.p.f
+ host := f.f.hostName()
+ cfg := seedCrashFixture{Claims: filepath.Join(seedRebootPath, host, "claims"), Data: f.f.guard.DataDir, Locks: f.f.guard.LockDir, Root: f.rootDisk.dir, Parks: "/var/lib/carlos-claims/authority/serving-parks", Journal: f.f.journalPath, Registry: f.p.root.Reconcile.Registry.Path(), Retirement: f.p.root.Reconcile.Retirement.JournalDir, Actor: f.actor, Now: f.f.now}
+ claims := release.NewDirStore(cfg.Claims)
+ keys, err := f.f.claims.List(t.Context(), "")
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, key := range keys {
+ body, _, err := f.f.claims.Get(t.Context(), key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := claims.Put(t.Context(), key, body, ""); err != nil {
+ t.Fatal(err)
+ }
+ }
+ sh := seedRebootHost{Host: host, Fixture: cfg, Snapshot: f.f.data.(*leasetest.VersionedBucket).Snapshot()}
+ versions, err := f.f.data.(*leasetest.VersionedBucket).ListVersions(t.Context(), f.actor.Bucket, "")
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, v := range versions {
+ var body []byte
+ if !v.DeleteMarker {
+ body, err = f.f.data.GetVersion(t.Context(), f.actor.Bucket, v.Key, v.VersionID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ }
+ sh.Objects = append(sh.Objects, seedRebootObject{v, body})
+ }
+ names, err := f.p.ledger.state.Names()
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, name := range names {
+ if !strings.HasPrefix(name, "retirement-") || !strings.HasSuffix(name, ".json") {
+ continue
+ }
+ r, err := f.p.ledger.read(strings.TrimSuffix(strings.TrimPrefix(name, "retirement-"), ".json"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if r.Authorization.Successor.Host == host && r.Parked && !r.Closed {
+ sh.Records = append(sh.Records, r)
+ }
+ }
+ if len(sh.Records) == 0 {
+ t.Fatal("no retirement retained", host)
+ }
+ if err := h.executor.registry.Close(); err != nil {
+ t.Fatal(err)
+ }
+ if err := f.f.journal.Close(); err != nil {
+ t.Fatal(err)
+ }
+ if err := f.p.root.Reconcile.Registry.Close(); err != nil {
+ t.Fatal(err)
+ }
+ saved.Hosts = append(saved.Hosts, sh)
+ }
+ raw, err := json.Marshal(saved)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(filepath.Join(seedRebootPath, "fixture.json"), raw, 0600); err != nil {
+ t.Fatal(err)
+ }
+ syscall.Sync()
+ // Phase 2 owns these exact durable fixtures. All tenant units have stopped;
+ // process exit closes the loopback servers without running TempDir removal.
+ t.Log("two live hosts retired independently; durable fixture ready for actual reboot")
+ os.Exit(0)
+}
+
+func TestRootRebootFilesSeedTwoHostPhase2(t *testing.T) {
+ servingRootRunner(t)
+ raw, err := os.ReadFile(filepath.Join(seedRebootPath, "fixture.json"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var saved seedRebootSave
+ if err := json.Unmarshal(raw, &saved); err != nil {
+ t.Fatal(err)
+ }
+ boot, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
+ if err != nil || strings.TrimSpace(string(boot)) == saved.Boot {
+ t.Fatal("actual reboot required", err)
+ }
+ t.Cleanup(func() {
+ for _, p := range []string{saved.Scratch, saved.Install, seedRebootPath} {
+ if !strings.HasPrefix(p, "/var/lib/carlos-rootrunner-") && !strings.HasPrefix(p, "/var/lib/carlos-serving-test-") {
+ t.Errorf("refuse cleanup path %q", p)
+ continue
+ }
+ if err := os.RemoveAll(p); err != nil {
+ t.Error(err)
+ }
+ }
+ })
+ binary, err := os.Executable()
+ if err != nil {
+ t.Fatal(err)
+ }
+ witness := rootFilesSeedIsolatedWitness(t, binary, true, true)
+ servingRunnerMount(t, "/data")
+ state, err := openClaimServingParkAuthority()
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { state.Close() })
+ fs := newServingParkFS("/data", state)
+ ledger := &servingParkLedger{state: state, fs: fs, now: time.Now}
+ for _, h := range saved.Hosts {
+ for _, r := range h.Records {
+ if err := ledger.fs.CheckParked(&r); err == nil {
+ t.Fatal("old boot authority accepted", h.Host)
+ }
+ if !r.Complete {
+ t.Fatal("unfinished reboot fixture", h.Host)
+ }
+ if err := ledger.withNamespace(t.Context(), func() error { return ledger.prepareRebootLocked(t.Context(), r, nil) }); err != nil {
+ t.Fatal("prepare completed park", h.Host, err)
+ }
+ if err := ledger.withNamespace(t.Context(), func() error { return ledger.commitRebootLocked(t.Context(), r, nil) }); err != nil {
+ t.Fatal("recover park", h.Host, err)
+ }
+ }
+ }
+ var hosts []*seedLiveHost
+ for _, savedHost := range saved.Hosts {
+ h := reopenSeedLiveHost(t, savedHost, witness, state)
+ hosts = append(hosts, h)
+ h.wake(t)
+ }
+ for _, h := range hosts {
+ h.probe(t)
+ }
+ for i, h := range hosts {
+ hosts[1-i].probe(t)
+ h.stop()
+ h.wake(t)
+ hosts[1-i].probe(t)
+ }
+ for _, h := range hosts {
+ h.probe(t)
+ h.stop()
+ }
+}
+
+func reopenSeedLiveHost(t *testing.T, saved seedRebootHost, witness filesSeedLiveWitness, state rootstate.Store) *seedLiveHost {
+ t.Helper()
+ cfg := saved.Fixture
+ data := leasetest.NewVersionedBucket()
+ data.AddBucket(cfg.Actor.Bucket)
+ sort.Slice(saved.Objects, func(i, j int) bool { return saved.Objects[i].Version.VersionID < saved.Objects[j].Version.VersionID })
+ next := 1
+ for _, o := range saved.Objects {
+ n, err := strconv.Atoi(strings.TrimPrefix(o.Version.VersionID, "v"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ for next < n {
+ id := data.Put(cfg.Actor.Bucket, "reboot-gap", nil)
+ if err := data.DeleteVersion(t.Context(), cfg.Actor.Bucket, "reboot-gap", id); err != nil {
+ t.Fatal(err)
+ }
+ next++
+ }
+ var id string
+ if o.Version.DeleteMarker {
+ id = data.BareDelete(cfg.Actor.Bucket, o.Version.Key)
+ } else {
+ id = data.Put(cfg.Actor.Bucket, o.Version.Key, o.Body)
+ }
+ if id != o.Version.VersionID {
+ t.Fatal("version identity changed", id, o.Version.VersionID)
+ }
+ next++
+ }
+ if data.Snapshot() != saved.Snapshot {
+ t.Fatal("remote fixture version snapshot changed")
+ }
+ handler, _ := leasetest.NewS3Server(data)
+ server := httptest.NewServer(handler)
+ t.Cleanup(server.Close)
+ for k, v := range map[string]string{"AWS_ACCESS_KEY_ID": "test-key", "AWS_SECRET_ACCESS_KEY": "test-secret", "AWS_SESSION_TOKEN": "", "AWS_EC2_METADATA_DISABLED": "true", "AWS_REGION": "eu-west-1", "AWS_DEFAULT_REGION": "eu-west-1", "CARLOS_HOST_CLAIM": "1", "CARLOS_HOST_CLAIM_ORIGINS": "1", "CARLOS_MOVES_PHASE1": "", "CARLOS_S3_ENDPOINT": server.URL} {
+ t.Setenv(k, v)
+ }
+ claims := release.NewDirStore(cfg.Claims)
+ journal, err := hostbox.OpenLaunchJournal(cfg.Journal)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { journal.Close() })
+ roots := &servingAuthorityDisk{rollbackDisk: &rollbackDisk{dir: cfg.Root}}
+ rf := &recoveryFixture{host: saved.Host, t: t, claims: claims, evidence: claims, journal: journal, journalPath: cfg.Journal, data: data, bucket: cfg.Actor.Bucket, signer: ed25519.NewKeyFromSeed(make([]byte, ed25519.SeedSize)), now: cfg.Now.Add(time.Minute), seq: 100000}
+ rf.guard = hostguard.Guard{DataDir: cfg.Data, LockDir: cfg.Locks, Claims: hostguard.StoreReader{Store: claims}, Root: hostguard.NewRootAuthority(roots)}
+ rf.kernel = &recoveryKernel{journal: journal}
+ reg, err := registry.Open(cfg.Registry)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { reg.Close() })
+ row, err := reg.Get(saved.Host)
+ if err != nil || row == nil {
+ t.Fatal("route", err)
+ }
+ journals, err := hostbox.NewJournalService(journal)
+ if err != nil {
+ t.Fatal(err)
+ }
+ f := &servingRootFixture{t: t, f: rf, rootDisk: roots, db: row.DBPath, actor: cfg.Actor}
+ f.p = &servingParkRoot{ledger: &servingParkLedger{state: state, fs: newServingParkFS(cfg.Data, state), journal: journal, now: func() time.Time { return rf.now }}, empty: plan9D5Fixture(t).owners}
+ coordinator := rf.root(nil)
+ coordinator.Kernel = &noProcessKernelFake{}
+ coordinator.Inventory = journals.Inventory
+ f.p.root = ClaimRootHTTP{Roots: map[string]string{"data": cfg.Data}, Journals: journals, Launches: journal, Stop: ClaimStopHTTP{Coordinator: coordinator, Authenticate: func(context.Context, net.Conn) (StopPeer, error) { return rf.peer, nil }}, InventoryActor: func(context.Context) (hostbox.Actor, error) { return f.actor, nil }, RequireNewWork: func(context.Context) error { return nil }}
+ f.p.root.Reconcile.Registry = reg
+ f.p.root.Reconcile.Inventory = journals.Inventory
+ f.p.root.Reconcile.Box, f.p.root.Reconcile.Enrollment = f.actor.Box, f.actor.Enrollment
+ f.p.root.Reconcile.Retirement.JournalDir = cfg.Retirement
+ source := &movesFilesSource{DataDir: cfg.Data, Routes: reg, newFiles: func(context.Context, string, string) (release.Store, error) {
+ return casBucketStore{b: data, bucket: cfg.Actor.Bucket}, nil
+ }}
+ opened, err := source.Open(t.Context(), claims, data, cfg.Actor)
+ if err != nil {
+ t.Fatal(err)
+ }
+ p := newServingPreparation(t, f, "boot", fmt.Sprintf("reboot-%d", os.Getpid()))
+ return &seedLiveHost{p: p, endpoint: server.URL, files: opened.(*movesFiles), witness: witness}
+}
diff --git a/internal/host/claimfilesseed_restore_linux_test.go b/internal/host/claimfilesseed_restore_linux_test.go
index 15bd3a35b..6c6949c8c 100644
--- a/internal/host/claimfilesseed_restore_linux_test.go
+++ b/internal/host/claimfilesseed_restore_linux_test.go
@@ -55,7 +55,7 @@ func newServingPreparation(t *testing.T, f *servingRootFixture, boot, controller
p := &servingPreparation{f: f, local: claimStopLocal{Actor: f.actor, BootID: boot, ControllerID: controller, PID: 77, ProcessStart: controller + "-start"}}
f.f.peer = StopPeer{UID: 1001, PID: p.local.PID, ProcessStart: p.local.ProcessStart, BootID: boot, ControllerID: controller}
launchFixture := plan9D5Fixture(t)
- launchFixture.launch.Args = []string{"--socket", provision.NestedSocket(recoveryHost), "--db", f.db}
+ launchFixture.launch.Args = []string{"--socket", provision.NestedSocket(f.f.hostName()), "--db", f.db}
launchFixture.launch.Limit = cgroup.TierDefault
launchFixture.launch.Env = []string{"CARLOS_ADMIN_TOKEN=test"}
sources := launchFixture.owners.Sources
@@ -151,7 +151,7 @@ func newServingPreparation(t *testing.T, f *servingRootFixture, boot, controller
func (p *servingPreparation) prepare(t *testing.T) (hostclaim.Observed, activator.ClaimStart) {
t.Helper()
f := p.f
- rt := registry.Route{Host: recoveryHost, DBPath: f.db, Socket: provision.NestedSocket(recoveryHost)}
+ rt := registry.Route{Host: f.f.hostName(), DBPath: f.db, Socket: provision.NestedSocket(f.f.hostName())}
o := f.f.read()
// Prerequisites' isolated launch preflight requires an installed /data
// namespace. Drive its selection, stop and real lease in the same order;
@@ -228,17 +228,22 @@ func runFilesSeedPreparationWithLive(t *testing.T, failure, restart string, moun
t.Setenv("CARLOS_MOVES_PHASE1", "")
var f *servingRootFixture
if len(mounted) > 0 && mounted[0] {
-
- t.Cleanup(func() {
- if err := os.RemoveAll("/data"); err != nil {
- t.Error(err)
- }
- })
- t.Cleanup(func() {
- if err := os.RemoveAll("/var/lib/carlos-claims"); err != nil {
- t.Error(err)
- }
- })
+ shared := false
+ for _, witness := range live {
+ shared = shared || witness.shared
+ }
+ if !shared {
+ t.Cleanup(func() {
+ if err := os.RemoveAll("/data"); err != nil {
+ t.Error(err)
+ }
+ })
+ t.Cleanup(func() {
+ if err := os.RemoveAll("/var/lib/carlos-claims"); err != nil {
+ t.Error(err)
+ }
+ })
+ }
if failure == "restore-enospc" {
if err := os.Mkdir("/data", 0700); err != nil {
t.Fatal(err)
@@ -261,39 +266,47 @@ func runFilesSeedPreparationWithLive(t *testing.T, failure, restart string, moun
t.Error(err)
}
})
- f, _, _ = newServingLegacyRootSetup(t, false, "/data")
- instances := filepath.Join(f.f.guard.DataDir, "instances")
- if err := unix.Mount(instances, instances, "", unix.MS_BIND, ""); err != nil {
- t.Fatal(err)
- }
- t.Cleanup(func() {
- if err := unix.Unmount(instances, 0); err != nil {
- t.Error(err)
+ host := recoveryHost
+ for _, witness := range live {
+ if witness.host != "" {
+ host = witness.host
}
- })
- if err := unix.Mount("", instances, "", unix.MS_BIND|unix.MS_REMOUNT|unix.MS_NOSUID|unix.MS_NODEV|unix.MS_NOEXEC, ""); err != nil {
- t.Fatal(err)
}
- if parent, _, err := newServingParkFS(f.f.guard.DataDir, state).parent(); err == nil {
- parent.Close()
- t.Fatal("production filesystem accepted a mount without nosymfollow")
- }
- if err := unix.Mount("", instances, "", unix.MS_BIND|unix.MS_REMOUNT|unix.MS_NOSYMFOLLOW|unix.MS_NOSUID|unix.MS_NODEV|unix.MS_NOEXEC, ""); err != nil {
- t.Fatal(err)
- }
- link := filepath.Join(instances, ".nosymfollow-witness")
- if err := os.Symlink(f.db, link); err != nil {
- t.Fatal(err)
- }
- fd, err := unix.Open(link, unix.O_RDONLY|unix.O_CLOEXEC, 0)
- if fd >= 0 {
- unix.Close(fd)
- }
- if !errors.Is(err, syscall.ELOOP) {
- t.Fatalf("kernel followed symlink on nosymfollow mount: %v", err)
- }
- if err := os.Remove(link); err != nil {
- t.Fatal(err)
+ f, _, _ = newServingLegacyRootSetup(t, false, "/data", host)
+ if !shared {
+ instances := filepath.Join(f.f.guard.DataDir, "instances")
+ if err := unix.Mount(instances, instances, "", unix.MS_BIND, ""); err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() {
+ if err := unix.Unmount(instances, 0); err != nil {
+ t.Error(err)
+ }
+ })
+ if err := unix.Mount("", instances, "", unix.MS_BIND|unix.MS_REMOUNT|unix.MS_NOSUID|unix.MS_NODEV|unix.MS_NOEXEC, ""); err != nil {
+ t.Fatal(err)
+ }
+ if parent, _, err := newServingParkFS(f.f.guard.DataDir, state).parent(); err == nil {
+ parent.Close()
+ t.Fatal("production filesystem accepted a mount without nosymfollow")
+ }
+ if err := unix.Mount("", instances, "", unix.MS_BIND|unix.MS_REMOUNT|unix.MS_NOSYMFOLLOW|unix.MS_NOSUID|unix.MS_NODEV|unix.MS_NOEXEC, ""); err != nil {
+ t.Fatal(err)
+ }
+ link := filepath.Join(instances, ".nosymfollow-witness")
+ if err := os.Symlink(f.db, link); err != nil {
+ t.Fatal(err)
+ }
+ fd, err := unix.Open(link, unix.O_RDONLY|unix.O_CLOEXEC, 0)
+ if fd >= 0 {
+ unix.Close(fd)
+ }
+ if !errors.Is(err, syscall.ELOOP) {
+ t.Fatalf("kernel followed symlink on nosymfollow mount: %v", err)
+ }
+ if err := os.Remove(link); err != nil {
+ t.Fatal(err)
+ }
}
// Use the production UID and mount checks in this isolated namespace.
f.p.ledger.state = state
@@ -566,7 +579,7 @@ func runFilesSeedPreparationWithLive(t *testing.T, failure, restart string, moun
nextBoot = "boot-2"
}
next := newServingPreparation(t, f, nextBoot, "controller-2")
- err = next.runtime.Prerequisites(t.Context(), registry.Route{Host: recoveryHost, DBPath: f.db}, f.f.read(), recoveryPolicy)
+ err = next.runtime.Prerequisites(t.Context(), registry.Route{Host: f.f.hostName(), DBPath: f.db}, f.f.read(), recoveryPolicy)
if !errors.Is(err, hostbox.ErrChanged) {
t.Fatalf("retry settlement: %v", err)
}
diff --git a/internal/host/claimfilesseed_twohost_rootrunner_linux_test.go b/internal/host/claimfilesseed_twohost_rootrunner_linux_test.go
new file mode 100644
index 000000000..c187f62e5
--- /dev/null
+++ b/internal/host/claimfilesseed_twohost_rootrunner_linux_test.go
@@ -0,0 +1,157 @@
+//go:build linux && rootrunner
+
+package host
+
+import (
+ "context"
+ "io"
+ "net"
+ "net/http"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/lease/leasetest"
+ "github.com/carlosframework/platform/internal/provision"
+)
+
+func seedTwoHostNames() []string {
+ if os.Getenv("FILES_SEED_ISOLATION_MUTATION") == "1" {
+ return []string{"seed-negative-a.bab.oncarlos.com", "seed-negative-b.bab.oncarlos.com"}
+ }
+ return []string{"seed-a.bab.oncarlos.com", "seed-b.bab.oncarlos.com"}
+}
+
+func TestRootFilesSeedTwoHostIsolation(t *testing.T) {
+ runServingParkPeerNamespace(t, false, false, "seed-two-host")
+}
+
+type seedLiveHost struct {
+ p *servingPreparation
+ executor *filesSeedExecutor
+ endpoint string
+ files *movesFiles
+ witness filesSeedLiveWitness
+ stop func()
+ observed hostclaim.Observed
+}
+
+func rootFilesSeedTwoHost(t *testing.T, binary string) {
+ witness := rootFilesSeedIsolatedWitness(t, binary, true)
+ hosts := make([]*seedLiveHost, 0, 2)
+ for i, host := range seedTwoHostNames() {
+ h := &seedLiveHost{witness: witness}
+ capture := filesSeedLiveWitness{host: host, shared: i > 0, seed: func(t *testing.T, p *servingPreparation, e *filesSeedExecutor) {
+ h.p, h.executor, h.endpoint = p, e, os.Getenv("CARLOS_S3_ENDPOINT")
+ if err := os.WriteFile(filepath.Join(filepath.Dir(p.f.db), host, "payload"), []byte("real seeded bytes:"+host), 0600); err != nil {
+ t.Fatal(err)
+ }
+ }}
+ runFilesSeedPreparationWithLive(t, "real-seed", "same-controller", []bool{true}, []filesSeedLiveWitness{capture})
+ hosts = append(hosts, h)
+ }
+ for _, h := range hosts {
+ t.Setenv("CARLOS_S3_ENDPOINT", h.endpoint)
+ if err := h.executor.run(t.Context(), h.p.f.f.hostName(), io.Discard); err != nil {
+ t.Fatal("seed", err)
+ }
+ f := h.p.f
+ opened, err := h.executor.source.Open(t.Context(), f.f.claims, f.f.data.(*leasetest.VersionedBucket), f.actor)
+ if err != nil {
+ t.Fatal(err)
+ }
+ h.files = opened.(*movesFiles)
+ h.wake(t)
+ }
+ for _, h := range hosts {
+ h.probe(t)
+ }
+ // Retire each real launched generation while the other tenant stays live.
+ for i, h := range hosts {
+ other := hosts[1-i]
+ other.probe(t)
+ h.stop()
+ previous := h.observed.Claim.Holder.Generation
+ h.wake(t)
+ if h.observed.Claim.Holder.Generation == previous {
+ t.Fatal("cold wake reused generation")
+ }
+ other.probe(t)
+ h.probe(t)
+ }
+ for _, h := range hosts {
+ h.stop()
+ }
+ if t.Name() == "TestRootRebootFilesSeedTwoHostPhase1" {
+ saveFilesSeedReboot(t, binary, hosts)
+ }
+}
+
+func (h *seedLiveHost) wake(t *testing.T) {
+ t.Helper()
+ t.Setenv("CARLOS_S3_ENDPOINT", h.endpoint)
+ f := h.p.f
+ f.f.now = f.f.now.Add(time.Minute)
+ p := newServingPreparation(t, f, "boot", "controller", func(w *ClaimWakeAssembly) {
+ w.Inputs.(*claimServingOwners).Sources.Files = h.files
+ w.ServingPrerequisites = servingFilesPrerequisites(f.f.claims, h.files)
+ h.witness.configure(w)
+ })
+ h.witness.prepare(p)
+ admitted, start := p.prepare(t)
+ if err := start.Restore(t.Context()); err != nil {
+ t.Fatal("restore", err)
+ }
+ if err := start.Reserve(t.Context(), 0); err != nil {
+ t.Fatal("reserve", err)
+ }
+ if err := start.Materialize(t.Context()); err != nil {
+ t.Fatal("materialize", err)
+ }
+ stop := h.witness.start(t, p, admitted, start)
+ h.observed = admitted
+ h.stop = func() {
+ stop()
+ t.Setenv("CARLOS_S3_ENDPOINT", h.endpoint)
+ if err := p.runtime.FailedAdmission(t.Context(), admitted, hostclaim.ErrWrongState); err != nil {
+ t.Fatal("release", err)
+ }
+ }
+}
+
+func (h *seedLiveHost) probe(t *testing.T) {
+ t.Helper()
+ host := h.p.f.f.hostName()
+ transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+ return (&net.Dialer{}).DialContext(ctx, "unix", provision.NestedSocket(host))
+ }}
+ defer transport.CloseIdleConnections()
+ response, err := (&http.Client{Transport: transport, Timeout: time.Second}).Get("http://tenant/")
+ if err != nil {
+ t.Fatal("sibling unavailable", host, err)
+ }
+ defer response.Body.Close()
+ body, err := io.ReadAll(response.Body)
+ if err != nil || response.StatusCode != 200 || string(body) != "real seeded bytes:"+host {
+ t.Fatal("sibling changed", host, string(body), err)
+ }
+}
+
+func TestRootFilesSeedIsolationNegativeControl(t *testing.T) {
+ servingRootRunner(t)
+ binary, err := os.Executable()
+ if err != nil {
+ t.Fatal(err)
+ }
+ cmd := exec.CommandContext(t.Context(), binary, "-test.run=^TestRootFilesSeedTwoHostIsolation$", "-test.v", "-test.timeout=2m")
+ cmd.Env = append(os.Environ(), "FILES_SEED_ISOLATION_MUTATION=1")
+ output, err := cmd.CombinedOutput()
+ if err == nil || !strings.Contains(string(output), "isolation mutation detected: /tmp/exposed-root") {
+ t.Fatalf("isolation mutant not caught: %v\n%s", err, output)
+ }
+ t.Log("root-only mutation caught: exposed private-root file was read by the tenant")
+}
diff --git a/internal/host/claimretryrecovery_test.go b/internal/host/claimretryrecovery_test.go
index cbd0f51e3..a0e0d5d75 100644
--- a/internal/host/claimretryrecovery_test.go
+++ b/internal/host/claimretryrecovery_test.go
@@ -111,6 +111,7 @@ func (k *recoveryKernel) RecoverStopped(ctx context.Context, t hostguard.FenceTa
}
type recoveryFixture struct {
+ host string
journalPath string
data genselect.Reader
bucket string
@@ -138,14 +139,18 @@ type recoveryController struct {
assembly ClaimAssembly
}
-func newRecoveryFixture(t *testing.T) *recoveryFixture {
+func newRecoveryFixture(t *testing.T, hosts ...string) *recoveryFixture {
+ host := recoveryHost
+ if len(hosts) > 0 {
+ host = hosts[0]
+ }
t.Helper()
ctx := t.Context()
st := release.NewDirStore(t.TempDir())
now := time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
- in := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/jam/instances/" + recoveryHost + ".json", Account: "bab", App: "jam", Host: recoveryHost, Region: "ie", RecordRegion: "ie", CreatedAt: now}
- body := []byte(`{"schema":1,"host":"shop.bab.oncarlos.com","account":"bab","app":"jam","region":"ie","created_at":"2026-10-06T12:00:00Z"}`)
- absent, err := hostclaim.Read(ctx, st, recoveryHost)
+ in := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/jam/instances/" + host + ".json", Account: "bab", App: "jam", Host: host, Region: "ie", RecordRegion: "ie", CreatedAt: now}
+ body := []byte(strings.ReplaceAll(`{"schema":1,"host":"shop.bab.oncarlos.com","account":"bab","app":"jam","region":"ie","created_at":"2026-10-06T12:00:00Z"}`, recoveryHost, host))
+ absent, err := hostclaim.Read(ctx, st, host)
if err != nil {
t.Fatal(err)
}
@@ -166,14 +171,21 @@ func newRecoveryFixture(t *testing.T) *recoveryFixture {
t.Fatal(err)
}
t.Cleanup(func() { journal.Close() })
- return &recoveryFixture{journalPath: journalPath, t: t, claims: st, evidence: release.NewDirStore(t.TempDir()), signer: ed25519.NewKeyFromSeed(make([]byte, ed25519.SeedSize)), guard: hostguard.Guard{DataDir: data, LockDir: t.TempDir(), Claims: hostguard.StoreReader{Store: st}}, journal: journal, kernel: &recoveryKernel{journal: journal}, now: now, recordPrelaunch: true}
+ return &recoveryFixture{host: host, journalPath: journalPath, t: t, claims: st, evidence: release.NewDirStore(t.TempDir()), signer: ed25519.NewKeyFromSeed(make([]byte, ed25519.SeedSize)), guard: hostguard.Guard{DataDir: data, LockDir: t.TempDir(), Claims: hostguard.StoreReader{Store: st}}, journal: journal, kernel: &recoveryKernel{journal: journal}, now: now, recordPrelaunch: true}
}
func (f *recoveryFixture) newID() string { f.seq++; return fmt.Sprintf("id-%d", f.seq) }
+func (f *recoveryFixture) hostName() string {
+ if f.host != "" {
+ return f.host
+ }
+ return recoveryHost
+}
+
func (f *recoveryFixture) read() hostclaim.Observed {
f.t.Helper()
- o, err := hostclaim.Read(f.t.Context(), f.claims, recoveryHost)
+ o, err := hostclaim.Read(f.t.Context(), f.claims, f.hostName())
if err != nil {
f.t.Fatal(err)
}
diff --git a/internal/host/claimservinglegacyroot_linux_test.go b/internal/host/claimservinglegacyroot_linux_test.go
index ccecb2bd8..3dcdaa62c 100644
--- a/internal/host/claimservinglegacyroot_linux_test.go
+++ b/internal/host/claimservinglegacyroot_linux_test.go
@@ -25,10 +25,14 @@ import (
func newServingLegacyRootSetup(t *testing.T, empty bool, dataRoot ...string) (*servingRootFixture, ClaimLaunchInputs, ClaimStopCoordinator) {
t.Helper()
- f, _, _, in := servingRetirementFixture(t)
+ var hosts []string
+ if len(dataRoot) > 1 {
+ hosts = dataRoot[1:]
+ }
+ f, _, _, in := servingRetirementFixture(t, hosts...)
if len(dataRoot) != 0 {
f.guard.DataDir = dataRoot[0]
- in.DBPath = provision.NestedDBPath(f.guard.DataDir, recoveryHost)
+ in.DBPath = provision.NestedDBPath(f.guard.DataDir, f.hostName())
for _, dir := range []string{filepath.Join(f.guard.DataDir, "hostguard"), filepath.Dir(in.DBPath)} {
if err := os.MkdirAll(dir, 0700); err != nil {
t.Fatal(err)
diff --git a/internal/host/claimservingpeer_rootrunner_linux_test.go b/internal/host/claimservingpeer_rootrunner_linux_test.go
index a731b71e3..ca10ef57a 100644
--- a/internal/host/claimservingpeer_rootrunner_linux_test.go
+++ b/internal/host/claimservingpeer_rootrunner_linux_test.go
@@ -324,6 +324,14 @@ func runServingParkPeerNamespace(t *testing.T, handover, writer bool, namespace
if err := os.WriteFile(installed, body, 0755); err != nil {
t.Fatal(err)
}
+ if namespace == "seed-cli" {
+ rootFilesSeedDelegatedCLI(t, installed)
+ return
+ }
+ if namespace == "seed-two-host" {
+ rootFilesSeedTwoHost(t, installed)
+ return
+ }
if namespace == "seed-isolated" {
rootFilesSeedIsolated(t, installed)
return
diff --git a/internal/host/claimservingretire_linux_test.go b/internal/host/claimservingretire_linux_test.go
index 98ea1c3e7..995368673 100644
--- a/internal/host/claimservingretire_linux_test.go
+++ b/internal/host/claimservingretire_linux_test.go
@@ -135,10 +135,10 @@ func (w servingRetirementWork) With(ctx context.Context, _ hostwrite.Identity, f
}, func() error { w.p.inside = true; defer func() { w.p.inside = false }(); return fn() })
}
-func servingRetirementFixture(t *testing.T) (*recoveryFixture, *claimServingArtifactRetirement, *servingParkFixture, ClaimLaunchInputs) {
+func servingRetirementFixture(t *testing.T, hosts ...string) (*recoveryFixture, *claimServingArtifactRetirement, *servingParkFixture, ClaimLaunchInputs) {
t.Helper()
- f := newRecoveryFixture(t)
- db := provision.NestedDBPath(f.guard.DataDir, recoveryHost)
+ f := newRecoveryFixture(t, hosts...)
+ db := provision.NestedDBPath(f.guard.DataDir, f.hostName())
if err := os.MkdirAll(filepath.Dir(db), 0700); err != nil {
t.Fatal(err)
}
diff --git a/internal/host/plan12_restore_linux_test.go b/internal/host/plan12_restore_linux_test.go
index ea171df75..52fcf9faf 100644
--- a/internal/host/plan12_restore_linux_test.go
+++ b/internal/host/plan12_restore_linux_test.go
@@ -406,7 +406,7 @@ func plan12RestoreBaseline(t *testing.T, f *recoveryFixture, extraBytes ...int)
if _, err := spec.WriteTo(&out); err != nil {
t.Fatal(err)
}
- key := "instances/" + recoveryHost + "/g-src/0009/" + ltx.FormatFilename(1, 1)
+ key := "instances/" + f.hostName() + "/g-src/0009/" + ltx.FormatFilename(1, 1)
bucket := leasetest.NewVersionedBucket()
bucket.AddBucket("data")
version := bucket.Put("data", key, out.Bytes())
diff --git a/internal/instancectl/claimexec.go b/internal/instancectl/claimexec.go
index 528584872..ad13c1c3b 100644
--- a/internal/instancectl/claimexec.go
+++ b/internal/instancectl/claimexec.go
@@ -8,6 +8,7 @@ import (
"net"
"os"
"path/filepath"
+ "strconv"
"time"
"github.com/carlosframework/platform/internal/safefile"
@@ -84,7 +85,7 @@ func (u ClaimUnits) prepareExecGate(ctx context.Context, host string, p ClaimExe
return gate, nil
}
-func (g *claimExecGate) authorize(ctx context.Context, u ClaimUnits, s ClaimUnitStarted, check func() error) error {
+func (g *claimExecGate) authorize(ctx context.Context, u ClaimUnits, s *ClaimUnitStarted, check func() error) error {
deadline := time.Now().Add(u.budget())
if d, ok := ctx.Deadline(); ok && d.Before(deadline) {
deadline = d
@@ -104,10 +105,22 @@ func (g *claimExecGate) authorize(ctx context.Context, u ClaimUnits, s ClaimUnit
if err = conn.SetDeadline(deadline); err != nil {
return err
}
+ // PrivatePIDs startup can hand MainPID from systemd's setup process to
+ // the namespace init after start returns. Pin the current main process
+ // only within the same invocation and cgroup, then authenticate the peer.
+ state, err := u.show(ctx, s.Unit, "InvocationID", "MainPID", "ControlGroup", "ActiveState")
+ if err != nil {
+ return err
+ }
+ pid, err := strconv.Atoi(state["MainPID"])
+ if err != nil || pid <= 0 || state["InvocationID"] != s.InvocationID || state["ControlGroup"] != "/"+s.ControlGroup || state["ActiveState"] != "active" {
+ return ErrClaimUnit
+ }
+ s.MainPID = pid
if err = claimExecPeer(conn, s.MainPID); err != nil {
return err
}
- if err = u.Same(ctx, s); err != nil {
+ if err = u.Same(ctx, *s); err != nil {
return err
}
var asked ClaimExecPermit
@@ -121,7 +134,7 @@ func (g *claimExecGate) authorize(ctx context.Context, u ClaimUnits, s ClaimUnit
if err = check(); err != nil {
return err
}
- if err = u.Same(ctx, s); err != nil {
+ if err = u.Same(ctx, *s); err != nil {
return err
}
// One connection, one permit. Closing/restarting root cannot replay it.
diff --git a/internal/instancectl/claimexec_rev3_linux_test.go b/internal/instancectl/claimexec_rev3_linux_test.go
index 753c1705b..5a88e9f54 100644
--- a/internal/instancectl/claimexec_rev3_linux_test.go
+++ b/internal/instancectl/claimexec_rev3_linux_test.go
@@ -13,7 +13,7 @@ import (
)
func TestRev3ReservedUnitChecksAtTrampolineBoundary(t *testing.T) {
- for _, name := range []string{"valid", "expired during setup", "wrong peer", "wrong reservation", "replacement invocation"} {
+ for _, name := range []string{"valid", "expired during setup", "wrong peer", "wrong reservation", "replacement invocation", "main PID handed over"} {
t.Run(name, func(t *testing.T) {
u, ctl, in, _ := claimUnitFixture(t)
dir, err := os.MkdirTemp("", "rg")
@@ -31,13 +31,18 @@ func TestRev3ReservedUnitChecksAtTrampolineBoundary(t *testing.T) {
original := u.Systemctl
u.Systemctl = func(ctx context.Context, args ...string) (string, error) {
out, err := original(ctx, args...)
+ if name == "main PID handed over" && args[0] == "show" && strings.Contains(strings.Join(args, ","), "MainPID") {
+ ctl.mu.Lock()
+ ctl.pid = os.Getpid()
+ ctl.mu.Unlock()
+ }
if args[0] != "start" || err != nil {
return out, err
}
setup = true
ctl.mu.Lock()
ctl.pid = os.Getpid()
- if name == "wrong peer" {
+ if name == "wrong peer" || name == "main PID handed over" {
ctl.pid++
}
ctl.mu.Unlock()
@@ -66,7 +71,7 @@ func TestRev3ReservedUnitChecksAtTrampolineBoundary(t *testing.T) {
return out, nil
}
l := Launch{Schema: LaunchSchema, Host: in.Host, Argv: []string{LiveDir + "/" + in.Host}}
- _, err = u.StartReserved(t.Context(), in, l, p, func() error {
+ started, err := u.StartReserved(t.Context(), in, l, p, func() error {
checks++
if setup && name == "expired during setup" {
return ErrClaimUnit
@@ -79,8 +84,8 @@ func TestRev3ReservedUnitChecksAtTrampolineBoundary(t *testing.T) {
return nil
})
clientErr := <-received
- if name == "valid" {
- if err != nil || clientErr != nil || checks != 2 {
+ if name == "valid" || name == "main PID handed over" {
+ if err != nil || clientErr != nil || checks != 2 || started.MainPID != os.Getpid() {
t.Fatalf("start=%v gate=%v checks=%d", err, clientErr, checks)
}
source, _, _ := u.launchPaths(in.Host)
diff --git a/internal/instancectl/claimunit.go b/internal/instancectl/claimunit.go
index aa6832f46..e5e14ba65 100644
--- a/internal/instancectl/claimunit.go
+++ b/internal/instancectl/claimunit.go
@@ -203,7 +203,7 @@ func (u ClaimUnits) start(ctx context.Context, in ClaimUnit, launch *Launch, che
}
started := ClaimUnitStarted{Unit: unit, InvocationID: id, ControlGroup: ClaimUnitControlGroup(in.Host), MainPID: pid}
if gate != nil {
- if err := gate.authorize(ready, u, started, check); err != nil {
+ if err := gate.authorize(ready, u, &started, check); err != nil {
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), u.budget())
defer cancel()
if sameErr := u.Same(cleanup, started); sameErr != nil {
diff --git a/internal/provision/testdata/plan14-root-reads-seed.tsv b/internal/provision/testdata/plan14-root-reads-seed.tsv
index 5a2a248fe..090a298df 100644
--- a/internal/provision/testdata/plan14-root-reads-seed.tsv
+++ b/internal/provision/testdata/plan14-root-reads-seed.tsv
@@ -1,4 +1,4 @@
-# Files seed overlay loads LAST. Run 26; FS-R6 legacy dispositions explicitly ruled; other rows remain proposals.
+# Files seed overlay loads LAST. Run 27; FS-R6 legacy dispositions explicitly ruled; other rows remain proposals.
@legacy c4009ab5fb7d91ab65fde5cda910d796b2a98109831c7e90e1bef18c1a77ff48 internal/activator/activator.go Service.sweepOnce legacy accepted-legacy accepted-legacy;rev3=PROPOSED-seed;basis=census-ruling-FS-R6; diagnostic throttling only; registry remains legacy registry:Hibernate@12209,registry:Hibernate@21380,registry:Hibernate@5124,registry:Hibernate@6794,registry:Hibernate@9200,registry:Host@21449,registry:Host@21507,registry:Host@21879,registry:Host@21972,registry:Host@22707,registry:Host@22789,registry:Host@23854,registry:Host@24033,registry:Host@24082,registry:NextDue@22662,registry:Status@21396,whole:Route@12211,whole:Route@14144,whole:Route@15854,whole:Route@16044,whole:Route@16337,whole:Route@16509,whole:Route@16656,whole:Route@20867,whole:Route@22376,whole:Route@22716,whole:Route@22723,whole:Route@23669,whole:Route@24066,whole:Route@5126,whole:Route@6796,whole:Route@9202 74872baf2607ef6e24d8466dca33aa9f449b45c712377d3474f7d5c87368f62f
@legacy fae097bc67df1153c4fe96526cdbbdc0800fe9f989edae9c34e243962fb5c19d internal/activator/servingretry.go ServingRetries.Wrap legacy accepted-legacy accepted-legacy;rev3=PROPOSED-seed;basis=census-ruling-FS-R6; BeforeReserve refuses only; cleanup deadline bounds lifetime; route identity remains legacy file:Prepare@128,file:Prepare@4325,file:Prepare@4455,file:Read@2888,file:Read@3531,file:Read@5054,file:Read@5991,file:Read@6656,whole:Route@1212,whole:Route@1550,whole:Route@1648,whole:Route@1948,whole:Route@2360,whole:Route@4473,whole:Route@5421,whole:Route@6522,whole:Route@876,whole:Route@954 e0fe1c8acbe4b1631ff9c3ce63dceba8297fbeab55fc6e4ace72be1d2d9d5bfa
@legacy e84357c7d4acebbf3a83055c10796752ad67b32f98248f54361a387d2682d7d6 internal/host/relocationflush.go ReleaseFlushFiles.Flush legacy accepted-legacy accepted-legacy;rev3=PROPOSED-seed;basis=census-ruling-FS-R6; tenant files and env are published data, never authority file:Open@772 7cc12267c16d9d1ae1116c9d10528f97d3983c77fb6486e0aac73142d8999c4b
@@ -332,3 +332,6 @@ system.go systemVerb authority B,C,D,F,J4 rev3=PROPOSED-seed; command arguments,
internal/host/rootexecution_recovery.go recoverRootExecutionBeforeUpdate authority D rev3=PROPOSED-seed; extracted update recovery ordering; root instance-control identity; broker durable controller state then best-effort local park and retirement journals; retained park failures cannot gate update eb97b162b0d712dd2fe8971984353ebc10efe1142077a3c2531781dd969cbbf5
internal/lease/leasetest/s3server.go NewS3Server authority C,J4 rev3=PROPOSED-seed; test-process S3 fixture only, including privileged runner; HTTP query selects simulated versions, GET exposes stored ETag and conditional PUT refuses stale owners under the fixture mutex; not called by production root paths file:Query@479 ac2435bd4d5114736ba9b7e964f62d5fac3dd97eee5f11cbadd4facc342a9583
@coverage internal/lease/leasetest a589d673ded02391c4dfbbe6a9b86890f44522d3985bfa219c19d6ed40a0fd07
+internal/instancectl/claimexec.go claimExecGate.authorize authority C,G,J2,J4 rev3=PROPOSED-seed; root instance-control; systemd MainPID refreshed only within the same active invocation and cgroup at accepted trampoline connection; exact kernel peer match and repeated Same checks before permit 2c7461c0ce1f033f163bb660214b9e906945e75a8067bb1d6b1faa3b6916649d
+internal/instancectl/claimunit.go ClaimUnits.start authority C,G,J2,J4 rev3=PROPOSED-seed; root instance-control; retain the refreshed authenticated trampoline MainPID for post-start kernel launch verification and exact-invocation cleanup 7f01cf7ac8fd9db0f74f7b08c412ee7d229ff2e097ae1d69706d8b3c87f1e30d
+@coverage internal/instancectl a9b87eaf6abd540b9e80a4f2fccbce59877e4cf9049b47fbf8829b348d6ef14d