amadan
WIP files seed checkpoint 26 (not for merge)
Reboot-pair regression fixed (private fixture binds); FS-R2 combined 200-record recovery and update; claims-on instance-control construction; killed-executor settlement; direct real-origin launch and readiness.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
13 files changed,
+1027
−20
internal/host/claimconstruction_rootrunner_linux_test.go+136 −0internal/host/claimfilesseed_crash_rootrunner_linux_test.go+249 −0internal/host/claimfilesseed_integration_linux_test.go+51 −3internal/host/claimfilesseed_isolated_rootrunner_linux_test.go+299 −0internal/host/claimfilesseed_live_rootrunner_linux_test.go+8 −0internal/host/claimrecovery_combined_linux_test.go+149 −0internal/host/claimrecovery_combined_rootrunner_linux_test.go+37 −0internal/host/claimservingpark_rootrunner_linux_test.go+8 −1internal/host/claimservingpeer_rootrunner_linux_test.go+40 −12internal/host/rootexecution_recovery.go+5 −1internal/lease/leasetest/s3server.go+10 −0internal/lease/leasetest/s3server_test.go+29 −0internal/provision/testdata/plan14-root-reads-seed.tsv+6 −3
CI passed — run details
CI log
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok amadan.net/carlos/platform.git/launchctx 0.011s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok github.com/carlosframework/platform 280.254s
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 2.776s
ok github.com/carlosframework/platform/cmd/moveprobe 93.195s
ok github.com/carlosframework/platform/cmd/zddprobe 0.702s
ok github.com/carlosframework/platform/internal/activator 249.735s
ok github.com/carlosframework/platform/internal/agentsock 0.014s
ok github.com/carlosframework/platform/internal/alerting 0.003s
ok github.com/carlosframework/platform/internal/analytics 0.021s
ok github.com/carlosframework/platform/internal/appconfig 2.457s
ok github.com/carlosframework/platform/internal/apperr 0.071s
ok github.com/carlosframework/platform/internal/applog 0.122s
ok github.com/carlosframework/platform/internal/awsinv 0.003s
ok github.com/carlosframework/platform/internal/boxchoice 0.004s
ok github.com/carlosframework/platform/internal/boxid 0.898s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.008s
ok github.com/carlosframework/platform/internal/caps 0.011s
ok github.com/carlosframework/platform/internal/certwarm 0.002s
ok github.com/carlosframework/platform/internal/cgroup 0.687s
ok github.com/carlosframework/platform/internal/changelog 0.302s
ok github.com/carlosframework/platform/internal/clamd 0.054s
? github.com/carlosframework/platform/internal/cliflag [no test files]
ok github.com/carlosframework/platform/internal/console 385.560s
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.045s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.008s
ok github.com/carlosframework/platform/internal/dbverify 0.004s
ok github.com/carlosframework/platform/internal/dnsrec 0.002s
ok github.com/carlosframework/platform/internal/docsite 0.004s
ok github.com/carlosframework/platform/internal/domains 2.314s
ok github.com/carlosframework/platform/internal/doorkey 0.006s
ok github.com/carlosframework/platform/internal/economics 0.004s
ok github.com/carlosframework/platform/internal/edge 26.529s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.007s
ok github.com/carlosframework/platform/internal/fleetapi 0.128s
ok github.com/carlosframework/platform/internal/gate 1.654s
ok github.com/carlosframework/platform/internal/genselect 0.014s
ok github.com/carlosframework/platform/internal/geo 0.010s
ok github.com/carlosframework/platform/internal/geoip 0.029s
ok github.com/carlosframework/platform/internal/host 158.880s
ok github.com/carlosframework/platform/internal/hostactors 0.383s
ok github.com/carlosframework/platform/internal/hostbox 2.375s
ok github.com/carlosframework/platform/internal/hostclaim 0.495s
ok github.com/carlosframework/platform/internal/hostevidence 0.008s
ok github.com/carlosframework/platform/internal/hostexec 4.667s
ok github.com/carlosframework/platform/internal/hostfence 0.035s
ok github.com/carlosframework/platform/internal/hostfiles 10.469s
ok github.com/carlosframework/platform/internal/hostfloor 0.016s
ok github.com/carlosframework/platform/internal/hostguard 1.017s
ok github.com/carlosframework/platform/internal/hostops 1.510s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.006s
ok github.com/carlosframework/platform/internal/hostquarantine 0.002s
ok github.com/carlosframework/platform/internal/hostrelocate 0.440s
ok github.com/carlosframework/platform/internal/hostresolve 0.151s
ok github.com/carlosframework/platform/internal/hostwrite 0.002s
ok github.com/carlosframework/platform/internal/infraview 0.002s
ok github.com/carlosframework/platform/internal/instancectl 4.173s
ok github.com/carlosframework/platform/internal/instanceexec 2.703s
ok github.com/carlosframework/platform/internal/instancefs 0.005s
ok github.com/carlosframework/platform/internal/instancekey 0.003s
ok github.com/carlosframework/platform/internal/instanceupdates 0.008s
ok github.com/carlosframework/platform/internal/layout 1.476s
ok github.com/carlosframework/platform/internal/lease 9.791s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.007s
ok github.com/carlosframework/platform/internal/ledger 0.022s
ok github.com/carlosframework/platform/internal/mail 0.162s
ok github.com/carlosframework/platform/internal/maintenance 0.082s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.488s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.068s
ok github.com/carlosframework/platform/internal/metrics 0.016s
ok github.com/carlosframework/platform/internal/moves 20.905s
ok github.com/carlosframework/platform/internal/moves/archive 0.746s
ok github.com/carlosframework/platform/internal/objstore 0.165s
ok github.com/carlosframework/platform/internal/ops 0.169s
ok github.com/carlosframework/platform/internal/park 0.205s
ok github.com/carlosframework/platform/internal/privatefile 0.004s
ok github.com/carlosframework/platform/internal/provision 14.495s
ok github.com/carlosframework/platform/internal/publicrelease 0.011s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 4.877s
ok github.com/carlosframework/platform/internal/relay 7.230s
ok github.com/carlosframework/platform/internal/release 19.478s
ok github.com/carlosframework/platform/internal/rollout 3.602s
ok github.com/carlosframework/platform/internal/rolloutwords 0.003s
ok github.com/carlosframework/platform/internal/rootaws 0.008s
ok github.com/carlosframework/platform/internal/rootparticipation 0.006s
ok github.com/carlosframework/platform/internal/rootstate 0.024s
ok github.com/carlosframework/platform/internal/runs 0.561s
ok github.com/carlosframework/platform/internal/s3compat 0.005s
ok github.com/carlosframework/platform/internal/safefile 0.003s
ok github.com/carlosframework/platform/internal/schedule 2.443s
ok github.com/carlosframework/platform/internal/seeds 0.011s
ok github.com/carlosframework/platform/internal/sim/hostlife 8.611s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.951s
ok github.com/carlosframework/platform/internal/status 0.039s
ok github.com/carlosframework/platform/internal/steering 0.005s
ok github.com/carlosframework/platform/internal/stopevidence 0.213s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.003s
ok github.com/carlosframework/platform/internal/termfence 0.003s
ok github.com/carlosframework/platform/internal/tunnelclient 0.003s
ok github.com/carlosframework/platform/internal/unixdial 0.003s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.007s
ok github.com/carlosframework/platform/internal/version 0.001s
ok github.com/carlosframework/platform/internal/vet 1.526s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 46989474 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.021s
=== RUN TestT4StorageProbeMinio
2026/10/15 16:56:05 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DEC25371102410, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.00s)
PASS
ok github.com/carlosframework/platform/internal/host 0.020s
minio contract: PASS
diff --git a/internal/host/claimconstruction_rootrunner_linux_test.go b/internal/host/claimconstruction_rootrunner_linux_test.go| new file mode 100644 |
| index 000000000..9c669fe11 |
| --- /dev/null |
| +++ b/internal/host/claimconstruction_rootrunner_linux_test.go |
| @@ -0,0 +1,136 @@ |
| +//go:build linux && rootrunner |
| + |
| +package host |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "fmt" |
| + "net" |
| + "os" |
| + "os/exec" |
| + "os/user" |
| + "path/filepath" |
| + "strconv" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/hostactors" |
| + "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/rootparticipation" |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +func init() { |
| + if len(os.Args) > 1 && os.Args[1] == registryChildCommand { |
| + if err := RunRegistryChild(os.Args[2:]); err != nil { |
| + fmt.Fprintln(os.Stderr, err) |
| + os.Exit(1) |
| + } |
| + os.Exit(0) |
| + } |
| +} |
| + |
| +func TestRootFilesSeedInstanceControlConstruction(t *testing.T) { |
| + runServingParkPeerNamespace(t, false, false, "seed-construction") |
| +} |
| + |
| +func filesSeedInstanceControlChild(t *testing.T, binary, root string, listener *net.UnixListener) { |
| + t.Helper() |
| + for _, path := range []string{"/data", "/var/lib/carlos-claims"} { |
| + if err := os.Mkdir(path, 0755); err != nil { |
| + t.Fatal("refuse existing fixture path", path, err) |
| + } |
| + t.Cleanup(func() { os.RemoveAll(path) }) |
| + } |
| + account, err := user.Lookup("carlos") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + uid, _ := strconv.Atoi(account.Uid) |
| + gid, _ := strconv.Atoi(account.Gid) |
| + if err := os.Chown("/data", uid, gid); err != nil { |
| + t.Fatal(err) |
| + } |
| + file, err := listener.File() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer file.Close() |
| + listener.SetUnlinkOnClose(false) |
| + if err := listener.Close(); err != nil { |
| + t.Fatal(err) |
| + } |
| + output, err := os.Create(filepath.Join(root, "control.log")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + cmd := exec.Command(binary, "-test.run=^TestRootFilesSeedInstanceControlPayload$", "-test.v", "-test.timeout=90s") |
| + cmd.Env = append(os.Environ(), "FILES_SEED_CONTROL=1", "TMPDIR="+root) |
| + cmd.ExtraFiles = []*os.File{file} |
| + cmd.Stdout, cmd.Stderr = output, output |
| + if err := cmd.Start(); err != nil { |
| + output.Close() |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { |
| + cmd.Process.Kill() |
| + cmd.Wait() |
| + output.Close() |
| + body, _ := os.ReadFile(filepath.Join(root, "control.log")) |
| + t.Log(string(body)) |
| + }) |
| + // The inherited listener stays open while RunInstanceControl constructs its owners. |
| +} |
| + |
| +func TestRootFilesSeedInstanceControlPayload(t *testing.T) { |
| + if os.Getenv("FILES_SEED_CONTROL") != "1" { |
| + t.Fatal("invoke through construction witness") |
| + } |
| + t.Setenv("CARLOS_HOST_CLAIM", "1") |
| + t.Setenv("CARLOS_HOST_CLAIM_ORIGINS", "1") |
| + t.Setenv("CARLOS_MOVES_PHASE1", "") |
| + t.Setenv("CARLOS_REGION_HA", "") |
| + t.Setenv("CARLOS_DEPLOYMENT_BUCKET", "") |
| + t.Setenv("CARLOS_DATA_DIR", "/data") |
| + t.Setenv("LISTEN_FDS", "1") |
| + t.Setenv("LISTEN_PID", strconv.Itoa(os.Getpid())) |
| + store, resources, owners, _ := fix4JournalFixture(t) |
| + dir := t.TempDir() |
| + t.Setenv("CARLOS_DEPLOYMENT_DIR", dir) |
| + t.Setenv("CARLOS_OPT_DIR", resources.OptDir) |
| + body, _, err := store.Get(t.Context(), hostactors.Key) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := release.NewDirStore(dir).Put(t.Context(), hostactors.Key, body, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + owners.Serving = plan9D5Fixture(t).owners.Sources |
| + owners.Controllers = productionStopControllers{} |
| + claimOwnerRegistration.Provider = seedConstructionOwners{owners} |
| + state, err := rootstate.Open(rootparticipation.Namespace, true, false) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + raw, _ := json.Marshal(rootparticipation.Census{Schema: 1, Box: "box", Enrollment: "en", Migration: 1, Revision: 1, LaunchRevision: 1, Complete: true}) |
| + if err := state.Write("census.json", raw); err != nil { |
| + t.Fatal(err) |
| + } |
| + state.Close() |
| + if err := rootparticipation.RegisterLaunchVerifier(func(context.Context, uint64) error { return nil }); err != nil { |
| + t.Fatal(err) |
| + } |
| + RunInstanceControl(nil) |
| + t.Fatal("claims-on instance-control returned before serving") |
| +} |
| + |
| +type seedConstructionOwners struct{ fix4Owners } |
| + |
| +func (o seedConstructionOwners) OpenClaimOwners(_ context.Context, store release.Store, _ ClaimRuntimeResources) (ClaimOwnerBindings, error) { |
| + owners := o.ClaimOwnerBindings |
| + authority := *owners.RetirementAuthority |
| + authority.Store = store |
| + authority.Directory.Store = store |
| + owners.RetirementAuthority = &authority |
| + return owners, nil |
| +} |
diff --git a/internal/host/claimfilesseed_crash_rootrunner_linux_test.go b/internal/host/claimfilesseed_crash_rootrunner_linux_test.go| new file mode 100644 |
| index 000000000..c46a85a9a |
| --- /dev/null |
| +++ b/internal/host/claimfilesseed_crash_rootrunner_linux_test.go |
| @@ -0,0 +1,249 @@ |
| +//go:build linux && rootrunner |
| + |
| +package host |
| + |
| +import ( |
| + "bytes" |
| + "context" |
| + "crypto/ed25519" |
| + "encoding/json" |
| + "fmt" |
| + "net" |
| + "os" |
| + "os/exec" |
| + "path/filepath" |
| + "strings" |
| + "syscall" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/activator" |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/hostfiles" |
| + "github.com/carlosframework/platform/internal/hostguard" |
| + "github.com/carlosframework/platform/internal/lease" |
| + "github.com/carlosframework/platform/internal/lease/leasetest" |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +type seedCrashFixture struct { |
| + Claims, Data, Locks, Root, Parks, Journal, Registry, Retirement, Signal string |
| + Actor hostbox.Actor |
| + Now time.Time |
| +} |
| + |
| +func rootFilesSeedWitnessFor(t *testing.T, binary string) filesSeedLiveWitness { |
| + if strings.Contains(t.Name(), "WholeExecutorKilledRerun") { |
| + return filesSeedLiveWitness{seed: func(t *testing.T, p *servingPreparation, e *filesSeedExecutor) { rootFilesSeedCrash(t, binary, p, e) }} |
| + } |
| + return rootFilesSeedLiveWitness(t, binary) |
| +} |
| + |
| +func rootFilesSeedCrash(t *testing.T, binary string, p *servingPreparation, e *filesSeedExecutor) { |
| + f := p.f |
| + root := t.TempDir() |
| + saved := seedCrashFixture{Claims: filepath.Join(root, "claims"), Data: f.f.guard.DataDir, Locks: f.f.guard.LockDir, Root: f.rootDisk.dir, Parks: "/var/lib/carlos-claims/authority/serving-parks", Journal: f.f.journalPath, Registry: f.p.root.Reconcile.Registry.Path(), Retirement: f.p.root.Reconcile.Retirement.JournalDir, Signal: filepath.Join(root, "precommit"), Actor: f.actor, Now: f.f.now} |
| + claims := release.NewDirStore(saved.Claims) |
| + keys, err := f.f.claims.List(t.Context(), "") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, key := range keys { |
| + body, _, err := f.f.claims.Get(t.Context(), key) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err = claims.Put(t.Context(), key, body, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + // The seed selection registry and the handover registry share the same route here. |
| + rt, err := e.registry.Get(recoveryHost) |
| + if err != nil || rt == nil { |
| + t.Fatal(err) |
| + } |
| + savedBody, _ := json.Marshal(saved) |
| + config := filepath.Join(root, "fixture.json") |
| + if err := os.WriteFile(config, savedBody, 0600); err != nil { |
| + t.Fatal(err) |
| + } |
| + run := func(crash bool) error { |
| + cmd := exec.Command(binary, "-test.run=^TestRootFilesSeedWholeExecutorPayload$", "-test.v", "-test.timeout=40s") |
| + cmd.Env = append(os.Environ(), "FILES_SEED_CRASH_CONFIG="+config, fmt.Sprintf("FILES_SEED_CRASH=%v", crash)) |
| + cmd.Stdin = os.Stdin |
| + var out bytes.Buffer |
| + cmd.Stdout, cmd.Stderr = &out, &out |
| + err := cmd.Run() |
| + t.Log(out.String()) |
| + return err |
| + } |
| + err = run(true) |
| + var status syscall.WaitStatus |
| + if exit, ok := err.(*exec.ExitError); ok { |
| + status = exit.Sys().(syscall.WaitStatus) |
| + } |
| + if !status.Signaled() || status.Signal() != syscall.SIGKILL { |
| + t.Fatal("seed did not die at precommit", err) |
| + } |
| + if _, err := os.Stat(saved.Signal); err != nil { |
| + t.Fatal("never reached materialization", err) |
| + } |
| + before, err := hostclaim.Read(t.Context(), claims, recoveryHost) |
| + if err != nil || before.Claim.Holder.WakeRetry.State != "attempting" || before.Claim.Holder.WakeRetry.AttemptsReserved != 1 { |
| + t.Fatal("crash lost retry debt", err) |
| + } |
| + data := f.f.data.(*leasetest.VersionedBucket) |
| + first := movesFilesPrefix(before.Claim.Holder.Location, before.Claim.Holder.Generation) |
| + if data.LatestID(f.actor.Bucket, hostfiles.ManifestKey(first)) != "" { |
| + t.Fatal("crashed seed committed") |
| + } |
| + // Model expiry of the dead owner's remote lease; no production TTL is changed. |
| + versions, err := data.ListVersions(t.Context(), f.actor.Bucket, before.Claim.Holder.Location+"/") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, v := range versions { |
| + if strings.HasSuffix(v.Key, "/lock.json") && v.IsLatest { |
| + body, err := data.GetVersion(t.Context(), f.actor.Bucket, v.Key, v.VersionID) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + var lock map[string]any |
| + if err := json.Unmarshal(body, &lock); err != nil { |
| + t.Fatal(err) |
| + } |
| + lock["expires_at"] = time.Now().Add(-time.Minute).UTC().Format(time.RFC3339Nano) |
| + expired, _ := json.Marshal(lock) |
| + data.Put(f.actor.Bucket, v.Key, expired) |
| + } |
| + } |
| + if err := run(false); err == nil { |
| + t.Fatal("rerun bypassed settlement backoff") |
| + } |
| + settled, err := hostclaim.Read(t.Context(), claims, recoveryHost) |
| + if err != nil || settled.Claim.Holder.WakeRetry.State != "open" || settled.Claim.Holder.WakeRetry.AttemptsReserved != 1 || settled.Claim.Holder.WakeRetry.Attempt != nil { |
| + t.Fatal("rerun did not settle crashed attempt before backoff", err) |
| + } |
| + saved.Now = saved.Now.Add(time.Minute) |
| + savedBody, _ = json.Marshal(saved) |
| + if err := os.WriteFile(config, savedBody, 0600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := run(false); err != nil { |
| + t.Fatal("rerun after backoff", err) |
| + } |
| + after, err := hostclaim.Read(t.Context(), claims, recoveryHost) |
| + if err != nil || after.Claim.Holder.WakeRetry.State != "open" || after.Claim.Holder.WakeRetry.AttemptsReserved != 2 || after.Claim.Holder.Generation == before.Claim.Holder.Generation { |
| + t.Fatal("rerun did not settle and reserve once", err) |
| + } |
| + final := movesFilesPrefix(after.Claim.Holder.Location, after.Claim.Holder.Generation) |
| + if data.LatestID(f.actor.Bucket, hostfiles.ManifestKey(final)) == "" { |
| + t.Fatal("rerun did not commit real admitted prefix") |
| + } |
| + etag := after.ETag |
| + if err := run(false); err != nil { |
| + t.Fatal(err) |
| + } |
| + after, err = hostclaim.Read(t.Context(), claims, recoveryHost) |
| + if err != nil || after.ETag != etag { |
| + t.Fatal("established rerun changed claim", err) |
| + } |
| +} |
| + |
| +func TestRootFilesSeedWholeExecutorPayload(t *testing.T) { |
| + path := os.Getenv("FILES_SEED_CRASH_CONFIG") |
| + if path == "" { |
| + t.Fatal("invoke through killed executor witness") |
| + } |
| + body, err := os.ReadFile(path) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + var saved seedCrashFixture |
| + if err := json.Unmarshal(body, &saved); err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Setenv("AWS_REGION", "eu-west-1") |
| + t.Setenv("AWS_DEFAULT_REGION", "eu-west-1") |
| + claims := release.NewDirStore(saved.Claims) |
| + data, err := lease.NewVersionedS3(t.Context(), map[string]string{saved.Actor.Bucket: "eu-west-1"}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + journal, err := hostbox.OpenLaunchJournal(saved.Journal) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer journal.Close() |
| + roots := &servingAuthorityDisk{rollbackDisk: &rollbackDisk{dir: saved.Root}} |
| + rf := &recoveryFixture{t: t, claims: claims, evidence: claims, journal: journal, journalPath: saved.Journal, data: data, bucket: saved.Actor.Bucket, signer: ed25519.NewKeyFromSeed(make([]byte, ed25519.SeedSize)), now: saved.Now, seq: os.Getpid() * 100000} |
| + rf.guard = hostguard.Guard{DataDir: saved.Data, LockDir: saved.Locks, Claims: hostguard.StoreReader{Store: claims}, Root: hostguard.NewRootAuthority(roots)} |
| + rf.kernel = &recoveryKernel{journal: journal} |
| + crash := os.Getenv("FILES_SEED_CRASH") == "true" |
| + if !crash { |
| + rf.now = rf.now.Add(time.Minute) |
| + } |
| + reg, err := registry.Open(saved.Registry) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer reg.Close() |
| + row, err := reg.Get(recoveryHost) |
| + if err != nil || row == nil { |
| + t.Fatal(err) |
| + } |
| + state, err := rootstate.Open(saved.Parks, false, false) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer state.Close() |
| + journals, err := hostbox.NewJournalService(journal) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + f := &servingRootFixture{t: t, f: rf, rootDisk: roots, db: row.DBPath, actor: saved.Actor} |
| + f.p = &servingParkRoot{ledger: &servingParkLedger{state: state, fs: newServingParkFS(saved.Data, state), journal: journal, now: func() time.Time { return rf.now }}, empty: plan9D5Fixture(t).owners} |
| + coordinator := rf.root(nil) |
| + coordinator.Kernel = &noProcessKernelFake{} |
| + coordinator.Inventory = journals.Inventory |
| + f.p.root = ClaimRootHTTP{Roots: map[string]string{"data": saved.Data}, Journals: journals, Launches: journal, Stop: ClaimStopHTTP{Coordinator: coordinator, Authenticate: func(context.Context, net.Conn) (StopPeer, error) { return rf.peer, nil }}, InventoryActor: func(context.Context) (hostbox.Actor, error) { return f.actor, nil }, RequireNewWork: func(context.Context) error { return nil }} |
| + f.p.root.Reconcile.Registry = reg |
| + f.p.root.Reconcile.Inventory = journals.Inventory |
| + f.p.root.Reconcile.Box, f.p.root.Reconcile.Enrollment = f.actor.Box, f.actor.Enrollment |
| + f.p.root.Reconcile.Retirement.JournalDir = saved.Retirement |
| + p := newServingPreparation(t, f, "boot", fmt.Sprintf("seed-%d", os.Getpid())) |
| + a := p.assembly |
| + a.Data = data |
| + a.Identity = handoverRecoveryIdentity(t, saved.Actor) |
| + source := &movesFilesSource{DataDir: saved.Data, Routes: reg, newFiles: func(ctx context.Context, _, _ string) (release.Store, error) { |
| + return release.NewS3Store(ctx, saved.Actor.Bucket) |
| + }} |
| + a.Wake.seedFactory = func(ctx context.Context, op *filesSeedOperation, token *activator.SeedExclusion) (*activator.ClaimRuntime, *ClaimWakeAssembly, error) { |
| + runtime, wake, err := newFilesSeedRuntime(ctx, a, p.source, p.writer, op, token) |
| + if err != nil { |
| + return runtime, wake, err |
| + } |
| + prepare := runtime.Prepare |
| + runtime.Prepare = func(ctx context.Context, rt registry.Route, o hostclaim.Observed, policy activator.StartPolicy) (activator.ClaimStart, error) { |
| + start, err := prepare(ctx, rt, o, policy) |
| + if err == nil && crash { |
| + start.Materialize = func(context.Context) error { |
| + if err := os.WriteFile(saved.Signal, []byte(o.Claim.Holder.Generation), 0600); err != nil { |
| + return err |
| + } |
| + syscall.Kill(os.Getpid(), syscall.SIGKILL) |
| + select {} |
| + } |
| + } |
| + return start, err |
| + } |
| + return runtime, wake, nil |
| + } |
| + executor := &filesSeedExecutor{assembly: &a, service: &activator.Service{}, registry: reg, source: source} |
| + if err := executor.run(t.Context(), recoveryHost, os.Stdout); err != nil { |
| + t.Fatal(err) |
| + } |
| +} |
diff --git a/internal/host/claimfilesseed_integration_linux_test.go b/internal/host/claimfilesseed_integration_linux_test.go| index aad01e295..0d16d5265 100644 |
| --- a/internal/host/claimfilesseed_integration_linux_test.go |
| +++ b/internal/host/claimfilesseed_integration_linux_test.go |
| @@ -99,6 +99,12 @@ func runFilesSeedRealPreparation(t *testing.T, p *servingPreparation, live ...fi |
| t.Fatal("input selection", err) |
| } |
| executor := &filesSeedExecutor{assembly: &a, service: &activator.Service{}, registry: reg, source: source} |
| + for _, witness := range live { |
| + if witness.seed != nil { |
| + witness.seed(t, p, executor) |
| + return |
| + } |
| + } |
| var output bytes.Buffer |
| if err := executor.run(t.Context(), rt.Host, &output); err != nil { |
| t.Fatalf("seed: %v\n%s", err, output.String()) |
| @@ -131,6 +137,11 @@ func runFilesSeedRealPreparation(t *testing.T, p *servingPreparation, live ...fi |
| witness.configure(w) |
| } |
| }) |
| + for _, witness := range live { |
| + if witness.prepare != nil { |
| + witness.prepare(normal) |
| + } |
| + } |
| admitted, start := normal.prepare(t) |
| if err := start.Restore(t.Context()); err != nil { |
| t.Fatal("cold restore", wake, err) |
| @@ -156,8 +167,14 @@ func runFilesSeedRealPreparation(t *testing.T, p *servingPreparation, live ...fi |
| for _, witness := range live { |
| witness.serve(t, normal, admitted, start) |
| } |
| - if err := start.Failed(t.Context(), 0); err != nil { |
| - t.Fatal(err) |
| + healthy := false |
| + for _, witness := range live { |
| + healthy = healthy || witness.healthy |
| + } |
| + if !healthy { |
| + if err := start.Failed(t.Context(), 0); err != nil { |
| + t.Fatal(err) |
| + } |
| } |
| if err := normal.runtime.FailedAdmission(t.Context(), admitted, hostclaim.ErrWrongState); err != nil { |
| t.Fatal(err) |
| @@ -166,13 +183,24 @@ func runFilesSeedRealPreparation(t *testing.T, p *servingPreparation, live ...fi |
| t.Fatal("ordinary provider armed") |
| } |
| } |
| - if f.f.read().Claim.Holder.WakeRetry.AttemptsReserved != 3 { |
| + healthy := false |
| + for _, witness := range live { |
| + healthy = healthy || witness.healthy |
| + } |
| + if healthy { |
| + if f.f.read().Claim.Holder.WakeRetry != nil { |
| + t.Fatal("health did not clear retry debt") |
| + } |
| + } else if f.f.read().Claim.Holder.WakeRetry.AttemptsReserved != 3 { |
| t.Fatal("attempt budget changed") |
| } |
| t.Log(output.String()) |
| } |
| type filesSeedLiveWitness struct { |
| + healthy bool |
| + prepare func(*servingPreparation) |
| + seed func(*testing.T, *servingPreparation, *filesSeedExecutor) |
| configure func(*ClaimWakeAssembly) |
| serve func(*testing.T, *servingPreparation, hostclaim.Observed, activator.ClaimStart) |
| } |
| @@ -180,3 +208,23 @@ type filesSeedLiveWitness struct { |
| func TestFilesSeedRealOriginPreparation(t *testing.T) { |
| runFilesSeedOrdinaryPreparation(t, "real-seed", "same-controller") |
| } |
| + |
| +func TestFilesSeedPreparationChangedController(t *testing.T) { |
| + witness := filesSeedLiveWitness{configure: func(*ClaimWakeAssembly) {}, serve: func(*testing.T, *servingPreparation, hostclaim.Observed, activator.ClaimStart) {}} |
| + witness.prepare = func(p *servingPreparation) { |
| + p.local.BootID, p.local.ControllerID, p.local.PID, p.local.ProcessStart = "new-boot", "new-controller", 88, "new-start" |
| + p.f.f.peer = StopPeer{UID: 2001, PID: 88, ProcessStart: "new-start", BootID: "new-boot", ControllerID: "new-controller"} |
| + p.f.p.root.Stop.Coordinator.CarlosUID = 2001 |
| + p.wake.Retries.BootID, p.wake.Retries.ControllerID = "new-boot", "new-controller" |
| + if len(p.wake.pending) != 0 { |
| + t.Fatal("rebuilding active fixture") |
| + } |
| + p.wake.pending = nil |
| + var err error |
| + p.runtime, err = p.wake.Build(t.Context(), p.source, p.writer) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + runFilesSeedPreparationWithLive(t, "real-seed", "same-controller", nil, []filesSeedLiveWitness{witness}) |
| +} |
diff --git a/internal/host/claimfilesseed_isolated_rootrunner_linux_test.go b/internal/host/claimfilesseed_isolated_rootrunner_linux_test.go| new file mode 100644 |
| index 000000000..c420d79b2 |
| --- /dev/null |
| +++ b/internal/host/claimfilesseed_isolated_rootrunner_linux_test.go |
| @@ -0,0 +1,299 @@ |
| +//go:build linux && rootrunner |
| + |
| +package host |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "fmt" |
| + "io" |
| + "net" |
| + "net/http" |
| + "os" |
| + "os/exec" |
| + "os/user" |
| + "path/filepath" |
| + "strconv" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/activator" |
| + "github.com/carlosframework/platform/internal/hostbox" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + execlease "github.com/carlosframework/platform/internal/hostexec" |
| + "github.com/carlosframework/platform/internal/hostwrite" |
| + "github.com/carlosframework/platform/internal/instancectl" |
| + "github.com/carlosframework/platform/internal/instanceexec" |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/rootparticipation" |
| + "github.com/carlosframework/platform/internal/rootstate" |
| + "golang.org/x/sys/unix" |
| +) |
| + |
| +func init() { |
| + if len(os.Args) > 1 { |
| + switch os.Args[1] { |
| + case "instance-exec": |
| + instanceexec.RunExec(os.Args[2:]) |
| + os.Exit(0) |
| + case "instance-exit": |
| + instanceexec.RunExit(os.Args[2:]) |
| + os.Exit(0) |
| + } |
| + } |
| +} |
| + |
| +func TestRootFilesSeedIsolatedTenant(t *testing.T) { |
| + runServingParkPeerNamespace(t, false, false, "seed-isolated") |
| +} |
| + |
| +type seedIsolatedClient struct { |
| + root *ClaimLaunchRoot |
| + peer StopPeer |
| +} |
| + |
| +func (c seedIsolatedClient) Capability(context.Context) (string, error) { |
| + return hostbox.RoleLaunchProtocol, nil |
| +} |
| +func (c seedIsolatedClient) Launch(ctx context.Context, q hostbox.RoleLaunchRequest) (hostbox.RoleLaunchGrant, error) { |
| + out, err := c.root.Serve(ctx, c.peer, q) |
| + if err != nil { |
| + return hostbox.RoleLaunchGrant{}, err |
| + } |
| + return *out.Grant, nil |
| +} |
| +func (c seedIsolatedClient) Stop(ctx context.Context, q hostbox.RoleLaunchRequest) error { |
| + _, err := c.root.Serve(ctx, c.peer, q) |
| + return err |
| +} |
| + |
| +func rootFilesSeedIsolated(t *testing.T, binary string) { |
| + for _, dir := range []string{"/data", "/var/lib/carlos-claims", "/opt/carlos", "/run/carlos", "/run/carlos-control", "/etc/carlos"} { |
| + if err := os.Mkdir(dir, 0755); err != nil { |
| + t.Fatal("refuse existing fixture path", dir, err) |
| + } |
| + t.Cleanup(func() { os.RemoveAll(dir) }) |
| + } |
| + account, err := user.Lookup("carlos") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + uid, _ := strconv.Atoi(account.Uid) |
| + gid, _ := strconv.Atoi(account.Gid) |
| + for _, dir := range []string{"/opt/carlos/versions/fixture", "/run/carlos/instances", "/run/carlos/logs"} { |
| + if err := os.MkdirAll(dir, 0755); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + image, err := os.ReadFile(binary) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + app := "/opt/carlos/versions/fixture/app" |
| + for _, path := range []string{"/opt/carlos/carlos", app} { |
| + if err := os.WriteFile(path, image, 0755); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + raw, err := unitsFS.ReadFile("units/carlos-instance@.service") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + lines := []string{} |
| + for _, line := range strings.Split(string(raw), "\n") { |
| + if strings.HasPrefix(line, "BindsTo=") || strings.HasPrefix(line, "Requires=") || strings.HasPrefix(line, "After=") { |
| + continue |
| + } |
| + lines = append(lines, line) |
| + } |
| + unit := "/run/systemd/system/carlos-instance@.service" |
| + file, err := os.OpenFile(unit, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0644) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := file.WriteString(strings.Join(lines, "\n") + "\nIPAddressDeny=any\n"); err != nil { |
| + t.Fatal(err) |
| + } |
| + file.Close() |
| + t.Cleanup(func() { |
| + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) |
| + defer cancel() |
| + if out, err := exec.CommandContext(ctx, "systemctl", "stop", instancectl.UnitName(instancectl.RoleRun, recoveryHost)).CombinedOutput(); err != nil { |
| + t.Errorf("stop isolated tenant: %v %s", err, out) |
| + } |
| + os.Remove(unit) |
| + os.RemoveAll("/run/systemd/system/" + instancectl.UnitName(instancectl.RoleRun, recoveryHost) + ".d") |
| + if out, err := exec.CommandContext(ctx, "systemctl", "daemon-reload").CombinedOutput(); err != nil { |
| + t.Errorf("reload isolated unit: %v %s", err, out) |
| + } |
| + }) |
| + servingRunnerSystemctl(t, "daemon-reload") |
| + denied := []string{"/etc/carlos/fixture-secret", "/var/lib/carlos-claims/fixture-secret", "/data/instances/foreign.example/fixture-secret"} |
| + for _, path := range denied { |
| + if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(path, []byte("foreign sentinel"), 0644); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + agent := "/sys/fs/cgroup/files-seed-controller" |
| + if err := os.Mkdir(agent, 0755); err != nil { |
| + t.Fatal("refuse existing controller cgroup", err) |
| + } |
| + t.Cleanup(func() { |
| + if err := os.Remove(agent); err != nil { |
| + t.Error(err) |
| + } |
| + }) |
| + witness := rootFilesSeedLiveWitness(t, app) |
| + configure := witness.configure |
| + witness.configure = func(w *ClaimWakeAssembly) { |
| + configure(w) |
| + owners := w.Inputs.(*claimServingOwners) |
| + selected := owners.Sources.Launch |
| + owners.Sources.Launch = func(ctx context.Context, rt registry.Route) (ClaimServingLaunch, error) { |
| + in, err := selected(ctx, rt) |
| + in.Env = append(in.Env, "FILES_SEED_DENIED="+strings.Join(denied, ":")) |
| + return in, err |
| + } |
| + } |
| + |
| + witness.healthy = true |
| + witness.prepare = func(p *servingPreparation) { |
| + boot, err := execlease.BootID() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + start, err := claimLaunchProcessStart("/proc", os.Getpid()) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + p.local.BootID, p.local.ControllerID, p.local.PID, p.local.ProcessStart = boot, "isolated-controller", os.Getpid(), start |
| + p.f.f.peer = StopPeer{UID: uint32(uid), PID: os.Getpid(), ProcessStart: start, BootID: boot, ControllerID: p.local.ControllerID} |
| + p.wake.Retries.BootID, p.wake.Retries.ControllerID = boot, p.local.ControllerID |
| + p.f.p.root.Stop.Coordinator.CarlosUID = uint32(uid) |
| + prior := claimContainmentAgentRoot |
| + claimContainmentAgentRoot = func(context.Context) (string, error) { return agent, nil } |
| + t.Cleanup(func() { claimContainmentAgentRoot = prior }) |
| + p.wake.Prelaunch = func(ctx context.Context, o hostclaim.Observed, work hostwrite.Work, in ClaimLaunchInputs) (*activator.RoleLaunch, error) { |
| + target := recoveryTuple(o, p.local) |
| + group, err := claimContainmentParent(agent, target) |
| + if err != nil { |
| + return nil, err |
| + } |
| + path := filepath.Join("/sys/fs/cgroup", group) |
| + if err := os.Mkdir(path, 0755); err != nil { |
| + return nil, err |
| + } |
| + t.Cleanup(func() { os.Remove(path) }) |
| + var st unix.Stat_t |
| + if err := unix.Stat(path, &st); err != nil { |
| + return nil, err |
| + } |
| + err = work.With(ctx, in.Identity, func() error { |
| + return p.f.f.journal.Prelaunch(ctx, hostbox.LaunchInventory{Target: target, ControllerPID: p.local.PID, ProcessStart: start, NoState: true, Groups: []hostbox.LaunchGroup{{RelativePath: group, Device: uint64(st.Dev), Inode: st.Ino}}}) |
| + }) |
| + return &activator.RoleLaunch{Target: target, Journal: p.f.f.journal}, err |
| + } |
| + state, err := rootstate.Open(filepath.Join(t.TempDir(), "execution"), true, false) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { state.Close() }) |
| + census, _ := json.Marshal(rootparticipation.Census{Schema: 1, Box: p.f.actor.Box, Enrollment: p.f.actor.Enrollment, Migration: 1, Revision: 1, LaunchRevision: 1, Complete: true}) |
| + if err := state.Write("census.json", census); err != nil { |
| + t.Fatal(err) |
| + } |
| + units := instancectl.ClaimUnits{Systemctl: func(ctx context.Context, args ...string) (string, error) { |
| + out, err := exec.CommandContext(ctx, "systemctl", args...).CombinedOutput() |
| + return string(out), err |
| + }, CgroupRoot: "/sys/fs/cgroup", ExeRoot: "/run/carlos-control/exe", UnitRoot: "/run/systemd/system", DataRoot: "/data"} |
| + root := &ClaimLaunchRoot{Guard: p.f.f.guard, Journal: p.f.f.journal, DataRoot: "/data", ExecLockDir: t.TempDir(), Local: &rev2Local{}, Clock: execlease.SystemClock{}, Actor: func(context.Context) (hostbox.Actor, error) { return p.f.actor, nil }, Units: units, Kernel: linuxClaimLaunchKernel{OptRoot: "/opt/carlos", CgroupRoot: "/sys/fs/cgroup", ProcRoot: "/proc"}, ValidateExecutable: func(string, string, string) error { return nil }, executionState: state} |
| + client := seedIsolatedClient{root: root, peer: p.f.f.peer} |
| + p.wake.Launch = func(ctx context.Context, o hostclaim.Observed, work hostwrite.Work, in ClaimLaunchInputs, pre *activator.RoleLaunch) (*activator.RoleLaunch, error) { |
| + paths, err := claimIsolatedPaths(map[string]string{"data": "/data"}, o.Host, in) |
| + if err != nil { |
| + return nil, err |
| + } |
| + l := *pre |
| + l.Work, l.Identity, l.Root, l.CgroupRoot = work, in.Identity, client, "/sys/fs/cgroup" |
| + l.Spec = &hostbox.LaunchSpec{Identity: in.Identity, Executable: in.Executable, Digest: in.ExecutableSHA256, Args: in.Args, Env: in.Env} |
| + l.LaunchDir, l.RunDir, l.LogSocket, l.MemoryMax = paths.LaunchDir, paths.RunDir, paths.LogSocket, paths.MemoryMax |
| + l.Executable = func(context.Context, *exec.Cmd) (*os.File, error) { return os.Open(app) } |
| + l.Initialize = func(context.Context) error { |
| + if err := filepath.Walk(l.LaunchDir, func(path string, info os.FileInfo, err error) error { |
| + if err != nil { |
| + return err |
| + } |
| + return os.Chown(path, uid, gid) |
| + }); err != nil { |
| + return err |
| + } |
| + if err := os.MkdirAll(l.RunDir, 0750); err != nil { |
| + return err |
| + } |
| + return os.Chown(l.RunDir, uid, gid) |
| + } |
| + return &l, nil |
| + } |
| + if len(p.wake.pending) != 0 { |
| + t.Fatal("fixture rebuilt active wake assembly") |
| + } |
| + p.wake.pending = nil |
| + p.runtime, err = p.wake.Build(t.Context(), p.source, p.writer) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + witness.serve = func(t *testing.T, p *servingPreparation, o hostclaim.Observed, start activator.ClaimStart) { |
| + launch, err := start.PrepareLaunch(t.Context()) |
| + if err != nil { |
| + t.Fatal("prepare isolated launch", err) |
| + } |
| + cmd, _, settle, _, err := launch.Start(t.Context(), "main", start.CurrentAttempt(), start.Build, func() error { return nil }) |
| + if err != nil { |
| + out, _ := exec.Command("journalctl", "-u", instancectl.UnitName(instancectl.RoleRun, o.Host), "--no-pager", "-n", "30").CombinedOutput() |
| + t.Fatalf("isolated launch: %v\n%s", err, out) |
| + } |
| + defer func() { |
| + if err := settle(t.Context()); err != nil { |
| + t.Error(err) |
| + } |
| + }() |
| + ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second) |
| + defer cancel() |
| + if err := start.Ready(ctx, cmd); err != nil { |
| + t.Fatal("isolated readiness", err) |
| + } |
| + wrong, _ := os.FindProcess(os.Getpid()) |
| + if err := start.Healthy(ctx, 0, &exec.Cmd{Process: wrong}); err == nil { |
| + t.Fatal("controller accepted as healthy tenant") |
| + } |
| + if err := start.Healthy(ctx, 0, cmd); err != nil { |
| + t.Fatal("isolated exact health", err) |
| + } |
| + transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) { |
| + return (&net.Dialer{}).DialContext(ctx, "unix", provision.NestedSocket(o.Host)) |
| + }} |
| + defer transport.CloseIdleConnections() |
| + response, err := (&http.Client{Transport: transport, Timeout: time.Second}).Get("http://tenant/") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + body, err := io.ReadAll(response.Body) |
| + response.Body.Close() |
| + if err != nil || response.StatusCode != 200 || string(body) != "real seeded bytes" { |
| + t.Fatal("isolated payload", string(body), err) |
| + } |
| + for _, path := range denied { |
| + if body, err := os.ReadFile(path); err != nil || string(body) != "foreign sentinel" { |
| + t.Fatal("foreign sentinel changed", path, err) |
| + } |
| + } |
| + t.Log(fmt.Sprintf("healthy isolated wake: host=%s session=%d pid=%d", o.Host, o.Claim.Holder.Session, cmd.Process.Pid)) |
| + } |
| + runFilesSeedPreparationWithLive(t, "real-seed", "same-controller", []bool{true}, []filesSeedLiveWitness{witness}) |
| +} |
diff --git a/internal/host/claimfilesseed_live_rootrunner_linux_test.go b/internal/host/claimfilesseed_live_rootrunner_linux_test.go| index 0e69299b7..f777eab11 100644 |
| --- a/internal/host/claimfilesseed_live_rootrunner_linux_test.go |
| +++ b/internal/host/claimfilesseed_live_rootrunner_linux_test.go |
| @@ -15,6 +15,7 @@ import ( |
| "os" |
| "os/exec" |
| "path/filepath" |
| + "strings" |
| "testing" |
| "time" |
| @@ -41,6 +42,13 @@ func TestRootFilesSeedTenantPayload(t *testing.T) { |
| if os.Getenv("FILES_SEED_TENANT") != "1" { |
| t.Fatal("invoke through real-origin tenant witness") |
| } |
| + for _, path := range strings.Split(os.Getenv("FILES_SEED_DENIED"), ":") { |
| + if path != "" { |
| + if body, err := os.ReadFile(path); err == nil { |
| + t.Fatalf("tenant read foreign path %s: %q", path, body) |
| + } |
| + } |
| + } |
| db, err := sql.Open("sqlite", *seedTenantDB) |
| if err != nil { |
| t.Fatal(err) |
diff --git a/internal/host/claimrecovery_combined_linux_test.go b/internal/host/claimrecovery_combined_linux_test.go| new file mode 100644 |
| index 000000000..580134366 |
| --- /dev/null |
| +++ b/internal/host/claimrecovery_combined_linux_test.go |
| @@ -0,0 +1,149 @@ |
| +//go:build linux |
| + |
| +package host |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "fmt" |
| + "os" |
| + "os/exec" |
| + "path/filepath" |
| + "testing" |
| + "time" |
| + |
| + execlease "github.com/carlosframework/platform/internal/hostexec" |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/rootparticipation" |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +func filesSeedCombinedRecovery(t *testing.T, install func(*servingParkDiskFixture) rootstate.Store) { |
| + t.Helper() |
| + f := newServingParkDiskFixture(t) |
| + state := install(f) |
| + var parks []servingParkRecord |
| + for i := 0; i < 200; i++ { |
| + f.base.Host = fmt.Sprintf("park-%03d.example.com", i) |
| + f.target.Host = f.base.Host |
| + if err := os.MkdirAll(filepath.Dir(provision.NestedDBPath(f.data, f.base.Host)), 0700); err != nil { |
| + t.Fatal(err) |
| + } |
| + f.put("", fmt.Sprintf("retained-%03d", i)) |
| + auth, receipts := f.evidence(10, true, "missing") |
| + r, err := f.park(auth, receipts) |
| + if err != nil { |
| + t.Fatal(i, err) |
| + } |
| + if i != 0 { |
| + f.complete(auth) |
| + } |
| + parks = append(parks, r) |
| + } |
| + ctx, cancel := context.WithTimeout(t.Context(), 20*time.Second) |
| + defer cancel() |
| + maintenance := &servingParkRuntime{authority: &servingParkRoot{ledger: f.ledger}, ctx: ctx, cancel: cancel, idle: make(chan struct{})} |
| + root := ClaimRootHTTP{servingParks: maintenance} |
| + ledger := rootparticipation.Ledger{State: state, VerifyLaunches: func(context.Context, uint64) error { return nil }} |
| + raw, _ := json.Marshal(rootparticipation.Census{Schema: 1, Box: "box", Enrollment: "en", Migration: 1, Revision: 1, LaunchRevision: 1, Complete: true}) |
| + if err := state.Write("census.json", raw); err != nil { |
| + t.Fatal(err) |
| + } |
| + child := exec.Command("sleep", "60") |
| + if err := child.Start(); err != nil { |
| + t.Fatal(err) |
| + } |
| + defer func() { child.Process.Kill(); child.Wait() }() |
| + boot, err := execlease.BootID() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + start, err := claimLaunchProcessStart("/proc", child.Process.Pid) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + peer := execlease.ControllerProcess{KernelBootID: boot, PID: child.Process.Pid, Start: start} |
| + q := execlease.ExecutionRequest{Schema: 1, ID: "begin", Operation: "executionBegin", Host: "crashed.example.com", Incarnation: "inc", Stamp: 1, Session: 1, Op: "move"} |
| + cleaned, settled := 0, 0 |
| + cfg := execlease.RootBrokerConfig{State: state, Remote: release.NewDirStore(t.TempDir()), Clock: execlease.SystemClock{}, BootID: boot, Identity: "old-root", Alive: execlease.ControllerAlive, |
| + Authority: func(context.Context, execlease.ExecutionRequest, execlease.ControllerProcess) (execlease.ExecutionAuthority, error) { |
| + return execlease.ExecutionAuthority{Box: "box", Enrollment: "en", Account: "bab", App: "jam", Installation: "inc/1/box/en", Migration: 1, ClaimRevision: "claim", RecordRevision: "record", DirectoryRevision: "1", Kind: "landing", Phase: "checking"}, nil |
| + }, |
| + Cleanup: func(ctx context.Context, r execlease.ExecutionRecord) error { |
| + if execlease.ControllerAlive(ctx, r.Peer) == nil { |
| + return execlease.ErrState |
| + } |
| + cleaned++ |
| + return nil |
| + }, |
| + Settle: func(context.Context, execlease.ExecutionRecord) error { settled++; return nil }, Exclude: ledger.Acquire, Recovered: ledger.RecoverAcquisition} |
| + broker, err := execlease.NewRootBroker(cfg) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := broker.Execute(ctx, peer, q); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := ledger.BeginUpdate(ctx, "update"); err == nil { |
| + t.Fatal("live participation allowed update") |
| + } |
| + if err := child.Process.Kill(); err != nil { |
| + t.Fatal(err) |
| + } |
| + child.Wait() |
| + if err := broker.Close(ctx); err != nil { |
| + t.Fatal(err) |
| + } |
| + cleaned = 0 |
| + cfg.Identity = "replacement-root" |
| + broker, err = execlease.NewRootBroker(cfg) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer broker.Close(context.Background()) |
| + root.Execution = broker |
| + update, _ := applyHappy(t) |
| + update.a.participation = &ledger |
| + originalRecovery := recoverPendingControllers |
| + t.Cleanup(func() { recoverPendingControllers = originalRecovery }) |
| + recoverPendingControllers = func(call context.Context, _, _ string) error { |
| + return recoverRootExecutionBeforeUpdate(call, broker, root) |
| + } |
| + if err := update.a.controllersRecovered(); err != nil { |
| + t.Fatal(err) |
| + } |
| + if cleaned != 1 || settled != 1 { |
| + t.Fatal("controller debt not settled exactly once", cleaned, settled) |
| + } |
| + if maintenance.recoveryAfter == "" { |
| + t.Fatal("first page lost its cursor") |
| + } |
| + if err := maintenance.recoverPending(ctx); err != nil { |
| + t.Fatal(err) |
| + } |
| + if maintenance.recoveryAfter != "" { |
| + t.Fatal("second page did not complete") |
| + } |
| + if err := ledger.BeginUpdate(ctx, "update"); err != nil { |
| + t.Fatal("retained parks blocked update", err) |
| + } |
| + if err := ledger.EndUpdate(ctx, "update"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := update.a.Run(ctx, false); err != nil { |
| + t.Fatal(err) |
| + } |
| + if r := update.result(); r == nil || r.Outcome != systemOutcomeApplied { |
| + t.Fatal("update did not apply", r) |
| + } |
| + for i, r := range parks { |
| + f.retained(r.Authorization, fmt.Sprintf("retained-%03d", i), r.ReapAfter) |
| + } |
| +} |
| + |
| +func TestFilesSeedCombinedRecoveryAllowsUpdate(t *testing.T) { |
| + filesSeedCombinedRecovery(t, func(*servingParkDiskFixture) rootstate.Store { |
| + return &servingAuthorityDisk{rollbackDisk: &rollbackDisk{dir: t.TempDir()}} |
| + }) |
| +} |
diff --git a/internal/host/claimrecovery_combined_rootrunner_linux_test.go b/internal/host/claimrecovery_combined_rootrunner_linux_test.go| new file mode 100644 |
| index 000000000..fccb84cf1 |
| --- /dev/null |
| +++ b/internal/host/claimrecovery_combined_rootrunner_linux_test.go |
| @@ -0,0 +1,37 @@ |
| +//go:build linux && rootrunner |
| + |
| +package host |
| + |
| +import ( |
| + "os" |
| + "path/filepath" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/rootstate" |
| +) |
| + |
| +func TestRootFilesSeedCombinedRecoveryAllowsUpdate(t *testing.T) { |
| + servingRootRunner(t) |
| + root, err := os.MkdirTemp("/var/lib", "carlos-rootrunner-combined-") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { os.RemoveAll(root) }) |
| + t.Setenv("TMPDIR", root) |
| + filesSeedCombinedRecovery(t, func(f *servingParkDiskFixture) rootstate.Store { |
| + servingRunnerMount(t, f.data) |
| + parks, err := rootstate.Open(filepath.Join(root, "parks"), true, false) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { parks.Close() }) |
| + f.fs = newServingParkFS(f.data, parks) |
| + f.ledger.fs, f.ledger.state = f.fs, parks |
| + state, err := rootstate.Open(filepath.Join(root, "execution"), true, false) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { state.Close() }) |
| + return state |
| + }) |
| +} |
diff --git a/internal/host/claimservingpark_rootrunner_linux_test.go b/internal/host/claimservingpark_rootrunner_linux_test.go| index e348c8fc0..1d5e2df4b 100644 |
| --- a/internal/host/claimservingpark_rootrunner_linux_test.go |
| +++ b/internal/host/claimservingpark_rootrunner_linux_test.go |
| @@ -97,6 +97,10 @@ func servingRunnerMount(t *testing.T, data string) { |
| if err := unix.Mount(instances, instances, "", unix.MS_BIND, ""); err != nil { |
| t.Fatal(err) |
| } |
| + if err := unix.Mount("", instances, "", unix.MS_PRIVATE|unix.MS_REC, ""); err != nil { |
| + _ = unix.Unmount(instances, 0) |
| + t.Fatal(err) |
| + } |
| t.Cleanup(func() { |
| if err := unix.Unmount(instances, 0); err != nil { |
| t.Error(err) |
| @@ -685,6 +689,9 @@ func TestRootFilesSeedRealOriginPreparation(t *testing.T) { |
| runRootFilesSeedRealOriginPreparation(t, false) |
| } |
| func TestRootFilesSeedRealOriginTenant(t *testing.T) { runRootFilesSeedRealOriginPreparation(t, true) } |
| +func TestRootFilesSeedWholeExecutorKilledRerun(t *testing.T) { |
| + runRootFilesSeedRealOriginPreparation(t, true) |
| +} |
| func runRootFilesSeedRealOriginPreparation(t *testing.T, live bool) { |
| servingRootRunner(t) |
| if os.Getenv("PLAN14_ROOT_FIXTURE") == t.Name() { |
| @@ -695,7 +702,7 @@ func runRootFilesSeedRealOriginPreparation(t *testing.T, live bool) { |
| t.Fatal("root child did not enter its scratch root") |
| } |
| if live { |
| - runFilesSeedPreparationWithLive(t, "real-seed", "same-controller", []bool{true}, []filesSeedLiveWitness{rootFilesSeedLiveWitness(t, "/tenant.test")}) |
| + runFilesSeedPreparationWithLive(t, "real-seed", "same-controller", []bool{true}, []filesSeedLiveWitness{rootFilesSeedWitnessFor(t, "/tenant.test")}) |
| } else { |
| runFilesSeedOrdinaryPreparation(t, "real-seed", "same-controller", true) |
| } |
diff --git a/internal/host/claimservingpeer_rootrunner_linux_test.go b/internal/host/claimservingpeer_rootrunner_linux_test.go| index 6d6883a2c..a731b71e3 100644 |
| --- a/internal/host/claimservingpeer_rootrunner_linux_test.go |
| +++ b/internal/host/claimservingpeer_rootrunner_linux_test.go |
| @@ -115,6 +115,20 @@ func TestRootServingParkPeerPayload(t *testing.T) { |
| var reply servingPeerReply |
| var callErr error |
| switch q.Action { |
| + case "control-request": |
| + transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) { |
| + return (&net.Dialer{}).DialContext(ctx, "unix", socket) |
| + }} |
| + response, err := (&http.Client{Transport: transport, Timeout: 3 * time.Second}).Get("http://root/claim/retire") |
| + if err != nil { |
| + callErr = err |
| + } else { |
| + response.Body.Close() |
| + if response.StatusCode != http.StatusMethodNotAllowed { |
| + callErr = fmt.Errorf("control status %d", response.StatusCode) |
| + } |
| + } |
| + transport.CloseIdleConnections() |
| case "seed-listen": |
| ready := make(chan struct{}) |
| close(ready) |
| @@ -310,6 +324,10 @@ func runServingParkPeerNamespace(t *testing.T, handover, writer bool, namespace |
| if err := os.WriteFile(installed, body, 0755); err != nil { |
| t.Fatal(err) |
| } |
| + if namespace == "seed-isolated" { |
| + rootFilesSeedIsolated(t, installed) |
| + return |
| + } |
| f := newServingRootFixture(t, true) |
| servingRunnerMount(t, f.f.guard.DataDir) |
| for name, body := range map[string]string{"files/upload": "tenant upload", "env": "tenant configuration"} { |
| @@ -380,19 +398,23 @@ func runServingParkPeerNamespace(t *testing.T, handover, writer bool, namespace |
| capability.SeedController = verifyFilesSeedDelegation |
| handler = ClaimRootWithLaunch{Root: h, Launch: capability} |
| } |
| - server := &instancectl.Server{ClaimStop: handler, PeerOK: func(net.Conn) error { return nil }, Logf: t.Logf} |
| - ctx, cancel := context.WithCancel(context.Background()) |
| - if lifecycle, ok := handler.(interface{ Start(context.Context) }); ok { |
| - lifecycle.Start(ctx) |
| - } |
| - done := make(chan error, 1) |
| - go func() { done <- server.Serve(ctx, listener) }() |
| - t.Cleanup(func() { |
| - cancel() |
| - if err := <-done; err != nil { |
| - t.Error(err) |
| + if namespace == "seed-construction" { |
| + filesSeedInstanceControlChild(t, installed, root, listener.(*net.UnixListener)) |
| + } else { |
| + server := &instancectl.Server{ClaimStop: handler, PeerOK: func(net.Conn) error { return nil }, Logf: t.Logf} |
| + ctx, cancel := context.WithCancel(context.Background()) |
| + if lifecycle, ok := handler.(interface{ Start(context.Context) }); ok { |
| + lifecycle.Start(ctx) |
| } |
| - }) |
| + done := make(chan error, 1) |
| + go func() { done <- server.Serve(ctx, listener) }() |
| + t.Cleanup(func() { |
| + cancel() |
| + if err := <-done; err != nil { |
| + t.Error(err) |
| + } |
| + }) |
| + } |
| controlPath := filepath.Join(root, "control.sock") |
| control, err := net.ListenUnix("unix", &net.UnixAddr{Name: controlPath, Net: "unix"}) |
| if err != nil { |
| @@ -499,6 +521,12 @@ func runServingParkPeerNamespace(t *testing.T, handover, writer bool, namespace |
| } |
| return reply |
| } |
| + if namespace == "seed-construction" { |
| + command(servingPeerCommand{Action: "control-request"}, false) |
| + command(servingPeerCommand{Action: "seed-manager"}, false) |
| + command(servingPeerCommand{Action: "seed-capability"}, false) |
| + return |
| + } |
| if strings.HasPrefix(namespace, "seed-") { |
| command(servingPeerCommand{Action: "seed-capability"}, true) |
| command(servingPeerCommand{Action: "seed-manager"}, false) |
diff --git a/internal/host/rootexecution_recovery.go b/internal/host/rootexecution_recovery.go| index 3b87f649d..501ef0302 100644 |
| --- a/internal/host/rootexecution_recovery.go |
| +++ b/internal/host/rootexecution_recovery.go |
| @@ -72,7 +72,11 @@ func recoverRootExecution(ctx context.Context, dataDir, guardDir string) (result |
| return err |
| } |
| root := handler.(ClaimRootWithLaunch).Root |
| - if err := runtime.broker.Recover(ctx); err != nil { |
| + return recoverRootExecutionBeforeUpdate(ctx, runtime.broker, root) |
| +} |
| + |
| +func recoverRootExecutionBeforeUpdate(ctx context.Context, broker *execlease.RootBroker, root ClaimRootHTTP) error { |
| + if err := broker.Recover(ctx); err != nil { |
| return err |
| } |
| if err := root.RecoverPending(ctx); err != nil { |
diff --git a/internal/lease/leasetest/s3server.go b/internal/lease/leasetest/s3server.go| index 9ccf76364..0b07ae29d 100644 |
| --- a/internal/lease/leasetest/s3server.go |
| +++ b/internal/lease/leasetest/s3server.go |
| @@ -160,6 +160,9 @@ func NewS3Server(f *VersionedBucket, serves ...string) (http.Handler, func() int |
| writeErr(w, err) |
| return |
| } |
| + if v, ok := f.find(bucket, key, versionID); ok { |
| + w.Header().Set("ETag", `"`+v.etag+`"`) |
| + } |
| if hdr := r.Header.Get("Range"); hdr != "" { |
| span, ok := strings.CutPrefix(hdr, "bytes=") |
| start, end, dash := strings.Cut(span, "-") |
| @@ -187,6 +190,13 @@ func NewS3Server(f *VersionedBucket, serves ...string) (http.Handler, func() int |
| case r.Method == http.MethodPut && r.Header.Get("x-amz-copy-source") != "": |
| f.serveCopy(w, r, bucket, key) |
| case r.Method == http.MethodPut: |
| + if match := r.Header.Get("If-Match"); match != "" { |
| + current := f.buckets[bucket][key] |
| + if len(current) == 0 || current[0].deleteMarker || current[0].etag != strings.Trim(match, `"`) { |
| + s3err(w, http.StatusPreconditionFailed, "PreconditionFailed") |
| + return |
| + } |
| + } |
| var id string |
| var err error |
| if r.Header.Get("If-None-Match") == "*" { |
diff --git a/internal/lease/leasetest/s3server_test.go b/internal/lease/leasetest/s3server_test.go| index 5defbec0e..b4546f938 100644 |
| --- a/internal/lease/leasetest/s3server_test.go |
| +++ b/internal/lease/leasetest/s3server_test.go |
| @@ -226,3 +226,32 @@ func TestS3ServerValidatesCopyBody(t *testing.T) { |
| } |
| } |
| } |
| + |
| +func TestS3ServerConditionalReplaceUsesReadETag(t *testing.T) { |
| + m := NewVersionedBucket() |
| + m.AddBucket("data") |
| + h, _ := NewS3Server(m) |
| + first := do(t, h, http.MethodPut, "/data/lock.json", "expired", map[string]string{"If-None-Match": "*"}) |
| + read := do(t, h, http.MethodGet, "/data/lock.json", "", nil) |
| + etag := read.Header().Get("ETag") |
| + if etag == "" || etag != first.Header().Get("ETag") { |
| + t.Fatal("GET did not return current ETag", etag) |
| + } |
| + if got := do(t, h, http.MethodPut, "/data/lock.json", "new-owner", map[string]string{"If-Match": etag}); got.Code != 200 { |
| + t.Fatal("takeover", got.Code) |
| + } |
| + before := m.Snapshot() |
| + if got := do(t, h, http.MethodPut, "/data/lock.json", "stale-owner", map[string]string{"If-Match": etag}); got.Code != http.StatusPreconditionFailed { |
| + t.Fatal("stale CAS accepted", got.Code) |
| + } |
| + if m.Snapshot() != before { |
| + t.Fatal("stale CAS mutated versions") |
| + } |
| + if got := do(t, h, http.MethodPut, "/data/missing", "new", map[string]string{"If-Match": etag}); got.Code != http.StatusPreconditionFailed { |
| + t.Fatal("CAS on missing accepted", got.Code) |
| + } |
| + do(t, h, http.MethodDelete, "/data/lock.json", "", nil) |
| + if got := do(t, h, http.MethodPut, "/data/lock.json", "new", map[string]string{"If-Match": etag}); got.Code != http.StatusPreconditionFailed { |
| + t.Fatal("CAS over delete marker accepted", got.Code) |
| + } |
| +} |
diff --git a/internal/provision/testdata/plan14-root-reads-seed.tsv b/internal/provision/testdata/plan14-root-reads-seed.tsv| index 51c29f1e7..5a2a248fe 100644 |
| --- a/internal/provision/testdata/plan14-root-reads-seed.tsv |
| +++ b/internal/provision/testdata/plan14-root-reads-seed.tsv |
| @@ -1,4 +1,4 @@ |
| -# Files seed overlay loads LAST. Run 25; FS-R6 legacy dispositions explicitly ruled; other rows remain proposals. |
| +# Files seed overlay loads LAST. Run 26; FS-R6 legacy dispositions explicitly ruled; other rows remain proposals. |
| @legacy c4009ab5fb7d91ab65fde5cda910d796b2a98109831c7e90e1bef18c1a77ff48 internal/activator/activator.go Service.sweepOnce legacy accepted-legacy accepted-legacy;rev3=PROPOSED-seed;basis=census-ruling-FS-R6; diagnostic throttling only; registry remains legacy registry:Hibernate@12209,registry:Hibernate@21380,registry:Hibernate@5124,registry:Hibernate@6794,registry:Hibernate@9200,registry:Host@21449,registry:Host@21507,registry:Host@21879,registry:Host@21972,registry:Host@22707,registry:Host@22789,registry:Host@23854,registry:Host@24033,registry:Host@24082,registry:NextDue@22662,registry:Status@21396,whole:Route@12211,whole:Route@14144,whole:Route@15854,whole:Route@16044,whole:Route@16337,whole:Route@16509,whole:Route@16656,whole:Route@20867,whole:Route@22376,whole:Route@22716,whole:Route@22723,whole:Route@23669,whole:Route@24066,whole:Route@5126,whole:Route@6796,whole:Route@9202 74872baf2607ef6e24d8466dca33aa9f449b45c712377d3474f7d5c87368f62f |
| @legacy fae097bc67df1153c4fe96526cdbbdc0800fe9f989edae9c34e243962fb5c19d internal/activator/servingretry.go ServingRetries.Wrap legacy accepted-legacy accepted-legacy;rev3=PROPOSED-seed;basis=census-ruling-FS-R6; BeforeReserve refuses only; cleanup deadline bounds lifetime; route identity remains legacy file:Prepare@128,file:Prepare@4325,file:Prepare@4455,file:Read@2888,file:Read@3531,file:Read@5054,file:Read@5991,file:Read@6656,whole:Route@1212,whole:Route@1550,whole:Route@1648,whole:Route@1948,whole:Route@2360,whole:Route@4473,whole:Route@5421,whole:Route@6522,whole:Route@876,whole:Route@954 e0fe1c8acbe4b1631ff9c3ce63dceba8297fbeab55fc6e4ace72be1d2d9d5bfa |
| @legacy e84357c7d4acebbf3a83055c10796752ad67b32f98248f54361a387d2682d7d6 internal/host/relocationflush.go ReleaseFlushFiles.Flush legacy accepted-legacy accepted-legacy;rev3=PROPOSED-seed;basis=census-ruling-FS-R6; tenant files and env are published data, never authority file:Open@772 7cc12267c16d9d1ae1116c9d10528f97d3983c77fb6486e0aac73142d8999c4b |
| @@ -286,7 +286,7 @@ internal/host/claimwakeassembly.go type ClaimWakeAssembly authority C,J4 rev3=PR |
| internal/host/claimwakestop.go ClaimAssembly.collectCapturedStop authority C,G,J4 rev3=PROPOSED-seed; fresh claim and verified local stop identity; session-zero legacy holder may name only the local writer box, with no admissions; root independently proves and retains no-process evidence file:Exec@2994,file:Read@605 d4cd80663dfae5f9737a74e319d3423f4ef853e1843666b02b4cd2341e869d72 |
| internal/host/movesfiles.go movesFiles.Establish authority C,H3 rev3=PROPOSED-seed; ordinary origin proof and single C8 commit; private seed checks exact admitted writer and stages shared snapshot; no armed live provider 172fe8888da88c249b7990bd9fb4bd2821bd402fb96ab52af13c58500f31d50f |
| internal/host/relocationflush.go ReleaseFlushFiles.Flush legacy accepted-legacy accepted-legacy;rev3=PROPOSED-seed;basis=census-ruling-FS-R6; tenant files and env are published data, never authority file:Open@772 7cc12267c16d9d1ae1116c9d10528f97d3983c77fb6486e0aac73142d8999c4b |
| -internal/host/rootexecution_recovery.go recoverRootExecution authority D rev3=PROPOSED-seed; root lifecycle and retained recovery; independent crashed-controller recovery, per-journal errors retained, no agent evidence elevated 44d102ddc175fb2df17832a64b34063cc17e6abf4fe8779dd3843344336dd5c0 |
| +internal/host/rootexecution_recovery.go recoverRootExecution authority D rev3=PROPOSED-seed; root lifecycle and retained recovery; independent crashed-controller recovery, per-journal errors retained, no agent evidence elevated b9dfb1821d4bd2cfe67db3c2dcae872930c17f4e31d3994ecc6e4b0cd855b518 |
| internal/host/rootexecution_runtime.go claimExecutionRuntime.Close authority C,J4 rev3=PROPOSED-seed; existing execution monitor lifetime plus serving-park lifetime; both share caller deadline before authority stores close 08352d84ef3a8cd55924338abc3d33229f2216a608c0daef707aec8c472e2a8e |
| internal/hostbox/coldhandover.go ColdHandover.Run authority C,J4 rev3=PROPOSED-seed; production operation-only gate refuses destructive serving handover, including old pending journals; existing move effects unchanged file:Exec@1027,file:Lstat@9838,file:OpenRoot@9760,file:Read@1379,registry:Addr@3654,registry:Addrs@3672,registry:Backing@3691,registry:Canary@3748,registry:ClaimIncarnation@3532,registry:ClaimIncarnation@4034,registry:ClaimOwner@10588,registry:ClaimOwner@4116,registry:ClaimOwner@4136,registry:ClaimOwner@6011,registry:ClaimOwner@7439,registry:ClaimOwnership@10180,registry:ClaimOwnership@3502,registry:ClaimOwnership@4480,registry:ClaimOwnership@4698,registry:ClaimOwnership@4731,registry:ClaimRowID@10478,registry:ClaimRowID@10559,registry:ClaimRowID@4154,registry:ClaimRowID@4762,registry:ClaimRowID@7464,registry:ClaimStamp@3577,registry:ClaimStamp@4081,registry:Host@7401,registry:Kind@3610,registry:Name@4816,registry:Name@7420,registry:Platform@3732,registry:Registration@4180,registry:Registration@7489,registry:Replica@3762,registry:Route@10180,registry:Route@4480,registry:Route@4731,registry:Route@5262,registry:Route@6011,registry:Timers@4679,registry:Timers@4698,registry:Timers@4762,registry:Timers@4816,registry:Timers@7276,registry:Tunnel@3712,registry:Unit@3636,whole:ClaimOwnership@10194,whole:ClaimOwnership@3506,whole:ClaimOwnership@4492,whole:ClaimOwnership@4714,whole:ClaimOwnership@4743,whole:ColdRows@10182,whole:ColdRows@11294,whole:ColdRows@11304,whole:ColdRows@4977,whole:ColdRows@4985,whole:ColdRows@4993,whole:ColdRows@5000,whole:ColdRows@5264,whole:ColdRows@6013,whole:ColdRows@7278,whole:Route@10188,whole:Route@3487,whole:Route@3847,whole:Route@4279,whole:Route@4287,whole:Route@4338,whole:Route@4346,whole:Route@4486,whole:Route@4737,whole:Route@5271,whole:Route@6020,whole:Timer@4018,whole:Timer@4292,whole:Timer@4300,whole:Timer@4351,whole:Timer@4389,whole:Timer@4685,whole:Timer@4704,whole:Timer@4768,whole:Timer@4822,whole:Timer@7285 050ae1e6818687e19e7a0cd443c09e837e54bdc00f4ea1feb6430cfedd2b5343 |
| internal/hostbox/coldhandover.go type ColdHandover authority C,J4 rev3=PROPOSED-seed; operation-only production cold-handover policy; no source-preserving serving authority from operation cleanup 874b2b74ea349dd93655d4e11f4d9e97752c3f216593ebb922d819caba7981c4 |
| @@ -325,7 +325,10 @@ system.go runSystem authority B,C,D,F,J4 rev3=PROPOSED-seed; command arguments, |
| system.go systemVerb authority B,C,D,F,J4 rev3=PROPOSED-seed; command arguments, claim flags, service identity and root delegation capability before constructing agent-mode resources; one-shot skips Configure, Start, duties and listeners; close resources only after private runtime joins ee25513ed72f9b8338d95682c4c76b4ced26ec1f060528fc5804da9afbfbfd83 |
| @coverage . bf3d70b6a2c5d769a3c580c9d5511231499b788d9c08a542895de8381e5a39c9 |
| @coverage internal/activator 41f0ad01ac54bae63f089ef8ec5bbfa9155bb76720d72b2d88563481de40fe5c |
| -@coverage internal/host 455146512e142affa2ba23b8cf3c49ad2028e78901c23e5ceaac96a1edddbad4 |
| +@coverage internal/host 276b5d95e109772c8af5bc7408bf4d888656ad032ef5d552e169203f55be023e |
| @coverage internal/hostbox db890260533bdace6ad9a7c23dc1538cb919962894f005fc7c5a59a18e969429 |
| @coverage internal/hostfiles a38dbcd9789ba4acd237e88c834f2b2bfca9847b0d43c32b0116dc8996d7a142 |
| @coverage internal/hostguard a721142f4f3ac5fa17b6b1e2ccfc66ca809cfdcbf2733fdc00fbd0efaa90aeb6 |
| +internal/host/rootexecution_recovery.go recoverRootExecutionBeforeUpdate authority D rev3=PROPOSED-seed; extracted update recovery ordering; root instance-control identity; broker durable controller state then best-effort local park and retirement journals; retained park failures cannot gate update eb97b162b0d712dd2fe8971984353ebc10efe1142077a3c2531781dd969cbbf5 |
| +internal/lease/leasetest/s3server.go NewS3Server authority C,J4 rev3=PROPOSED-seed; test-process S3 fixture only, including privileged runner; HTTP query selects simulated versions, GET exposes stored ETag and conditional PUT refuses stale owners under the fixture mutex; not called by production root paths file:Query@479 ac2435bd4d5114736ba9b7e964f62d5fac3dd97eee5f11cbadd4facc342a9583 |
| +@coverage internal/lease/leasetest a589d673ded02391c4dfbbe6a9b86890f44522d3985bfa219c19d6ed40a0fd07 |