carlos / platform Public

Plan 14 integrated: cancellation retries; lifecycle E2E and root suite green (not for merge)

Producer cancellation no longer discards a retryable root-stop failure or leaves its producer unjoined. Lifecycle E2E with the real guard passes; full root suite green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Paul Campbell pushed by claude 1 week ago 43ff0259dc3977ea977ba6d1903a04d9a68fea4b parent c221196

4 files changed, +45 −6

internal/activator/claimproducer_test.go +29 −0
internal/activator/claimwake.go +10 −1
internal/host/plan14_lifecycle_root_linux_test.go +5 −4
internal/provision/testdata/plan14-root-reads-p3.tsv +1 −1

CI passed — run details

CI log

=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.012s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	333.557s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	2.867s
ok  	github.com/carlosframework/platform/cmd/moveprobe	93.435s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.712s
ok  	github.com/carlosframework/platform/internal/activator	251.818s
ok  	github.com/carlosframework/platform/internal/agentsock	0.014s
ok  	github.com/carlosframework/platform/internal/alerting	0.005s
ok  	github.com/carlosframework/platform/internal/analytics	0.020s
ok  	github.com/carlosframework/platform/internal/appconfig	2.496s
ok  	github.com/carlosframework/platform/internal/apperr	0.076s
ok  	github.com/carlosframework/platform/internal/applog	0.157s
ok  	github.com/carlosframework/platform/internal/awsinv	0.004s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.005s
ok  	github.com/carlosframework/platform/internal/boxid	0.504s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.011s
ok  	github.com/carlosframework/platform/internal/caps	0.013s
ok  	github.com/carlosframework/platform/internal/certwarm	0.005s
ok  	github.com/carlosframework/platform/internal/cgroup	0.722s
ok  	github.com/carlosframework/platform/internal/changelog	0.311s
ok  	github.com/carlosframework/platform/internal/claimactivation	0.025s
ok  	github.com/carlosframework/platform/internal/clamd	0.054s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	416.044s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.103s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.009s
ok  	github.com/carlosframework/platform/internal/dbverify	0.004s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.003s
ok  	github.com/carlosframework/platform/internal/docsite	0.005s
ok  	github.com/carlosframework/platform/internal/domains	2.404s
ok  	github.com/carlosframework/platform/internal/doorkey	0.006s
ok  	github.com/carlosframework/platform/internal/economics	0.004s
ok  	github.com/carlosframework/platform/internal/edge	45.047s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.009s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.133s
ok  	github.com/carlosframework/platform/internal/gate	1.655s
ok  	github.com/carlosframework/platform/internal/genselect	0.015s
ok  	github.com/carlosframework/platform/internal/geo	0.013s
ok  	github.com/carlosframework/platform/internal/geoip	0.031s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.158s
ok  	github.com/carlosframework/platform/internal/host	186.331s
ok  	github.com/carlosframework/platform/internal/hostactors	0.547s
ok  	github.com/carlosframework/platform/internal/hostbox	2.874s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.585s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.006s
ok  	github.com/carlosframework/platform/internal/hostexec	5.720s
ok  	github.com/carlosframework/platform/internal/hostfence	0.043s
ok  	github.com/carlosframework/platform/internal/hostfiles	11.012s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.012s
ok  	github.com/carlosframework/platform/internal/hostguard	1.042s
ok  	github.com/carlosframework/platform/internal/hostops	1.641s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.004s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.002s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.499s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.160s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.002s
ok  	github.com/carlosframework/platform/internal/infraview	0.002s
ok  	github.com/carlosframework/platform/internal/instancectl	5.260s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.783s
ok  	github.com/carlosframework/platform/internal/instancefs	0.006s
ok  	github.com/carlosframework/platform/internal/instancekey	0.013s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.024s
ok  	github.com/carlosframework/platform/internal/layout	1.564s
ok  	github.com/carlosframework/platform/internal/lbhealth	0.013s
ok  	github.com/carlosframework/platform/internal/lease	9.868s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.006s
ok  	github.com/carlosframework/platform/internal/ledger	0.018s
ok  	github.com/carlosframework/platform/internal/mail	0.169s
ok  	github.com/carlosframework/platform/internal/maintenance	0.087s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.574s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.073s
ok  	github.com/carlosframework/platform/internal/metrics	0.006s
ok  	github.com/carlosframework/platform/internal/moves	21.441s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.829s
ok  	github.com/carlosframework/platform/internal/objstore	0.191s
ok  	github.com/carlosframework/platform/internal/ops	0.192s
ok  	github.com/carlosframework/platform/internal/park	0.229s
ok  	github.com/carlosframework/platform/internal/privatefile	0.004s
ok  	github.com/carlosframework/platform/internal/provision	35.311s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.007s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.578s
ok  	github.com/carlosframework/platform/internal/relay	7.301s
ok  	github.com/carlosframework/platform/internal/release	22.964s
ok  	github.com/carlosframework/platform/internal/rollout	4.221s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.004s
ok  	github.com/carlosframework/platform/internal/rootaws	0.007s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.061s
ok  	github.com/carlosframework/platform/internal/rootstate	0.026s
ok  	github.com/carlosframework/platform/internal/runs	0.570s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.005s
ok  	github.com/carlosframework/platform/internal/schedule	2.602s
ok  	github.com/carlosframework/platform/internal/seeds	0.011s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	13.505s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	4.094s
ok  	github.com/carlosframework/platform/internal/status	0.039s
ok  	github.com/carlosframework/platform/internal/steering	0.006s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.301s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.008s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.003s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.004s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.588s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 48824482 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: cached, digest verified
  mc: cached, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.023s
=== RUN   TestT4StorageProbeMinio
2026/10/17 17:34:12 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DF619102D646D1, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.021s
minio contract: PASS

diff --git a/internal/activator/claimproducer_test.go b/internal/activator/claimproducer_test.go

index cb955d5b7..e908d6504 100644

--- a/internal/activator/claimproducer_test.go

+++ b/internal/activator/claimproducer_test.go

@@ -59,3 +59,32 @@ func TestClaimProducerCancellationStopsAndJoinsProcess(t *testing.T) {

t.Fatalf("containment closed %d times", containment.closed)

}

+func TestClaimProducerCancellationRetriesRootSettlement(t *testing.T) {

+ p := &claimProcess{root: true, done: make(chan struct{})}

+ close(p.done)

+ var attempts int

+ p.settle = func(context.Context) error {

+ attempts++

+ if attempts < 3 {

+ return errors.New("root stop temporarily refused")

+ }

+ return nil

+ }

+ ctx, cancel := context.WithCancel(t.Context())

+ defer cancel()

+ finished := make(chan error, 1)

+ p.bindProducer(RoleProducer{Context: ctx, Finish: func(err error) { finished <- err }})

+ cancel()

+ select {

+ case err := <-finished:

+ if err != nil || attempts != 3 {

+ t.Fatalf("completion before root settlement: attempts=%d error=%v", attempts, err)

+ }

+ case <-time.After(3 * time.Second):

+ t.Fatal("producer did not retry root settlement")

+ }

+ if err := p.Stop(t.Context()); err != nil || attempts != 3 {

+ t.Fatalf("settled stop was not idempotent: attempts=%d error=%v", attempts, err)

+ }

diff --git a/internal/activator/claimwake.go b/internal/activator/claimwake.go

index 28344498f..77ee02923 100644

--- a/internal/activator/claimwake.go

+++ b/internal/activator/claimwake.go

@@ -851,6 +851,15 @@ func (p *claimProcess) bindProducer(producer RoleProducer) {

}

cleanup, cancel := context.WithTimeout(context.Background(), 30*time.Second)

defer cancel()

- _ = p.Stop(cleanup)

+ for {

+ if err := p.Stop(cleanup); err == nil {

+ return

+ }

+ select {

+ case <-cleanup.Done():

+ return

+ case <-time.After(100 * time.Millisecond):

+ }

}()

}

diff --git a/internal/host/plan14_lifecycle_root_linux_test.go b/internal/host/plan14_lifecycle_root_linux_test.go

index 59b764815..016362681 100644

--- a/internal/host/plan14_lifecycle_root_linux_test.go

+++ b/internal/host/plan14_lifecycle_root_linux_test.go

@@ -510,15 +510,16 @@ func lifecycleAgent(t *testing.T) {

assembly, err := OpenProductionClaimAssembly(ctx, store, resources, lifecycleOwners{t})

lifecycleCheck(t, "agent assembly", err)

defer func() {

- deadline := time.Now().Add(30 * time.Second)

+ cleanup, cancel := context.WithTimeout(context.Background(), 30*time.Second)

+ defer cancel()

for {

t.Log("agent assembly close enter")

- err := assembly.Close(context.Background())

+ err := assembly.Close(cleanup)

t.Log("agent assembly close returned", err)

if err == nil {

return

}

- if time.Now().After(deadline) {

+ if cleanup.Err() != nil {

t.Error("agent cleanup", err)

return

}

@@ -753,7 +754,7 @@ func lifecycleLaunch(t *testing.T, ctx context.Context, runtime *activator.Claim

}

cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)

defer cancel()

- _ = stop(cleanup)

+ _ = lifecycleRetry(cleanup, stop)

}()

}

lifecycleCheck(t, "tenant ready", start.Ready(ctx, cmd))

diff --git a/internal/provision/testdata/plan14-root-reads-p3.tsv b/internal/provision/testdata/plan14-root-reads-p3.tsv

index 039408f30..3f1fb8d32 100644

--- a/internal/provision/testdata/plan14-root-reads-p3.tsv

+++ b/internal/provision/testdata/plan14-root-reads-p3.tsv

@@ -198,7 +198,7 @@ internal/host/rootmigration_timers.go type rootMigrationTimerRecord authority G,

internal/host/rootmigration_timers.go type rootMigrationTimers authority G,J4 rev3=PROPOSED-p3;identity=offline migration root uid 0;basis=holds root-private migration state, root inventory, root fence authority and isolated registry interface under offline exclusion dc83f25eaae718327601d7d19c2c93f56e589cc176405c6249e9ea4e3c864c34

internal/hostguard/rootmigration.go RootAuthority.CompleteMigrationInventory authority G,J4 rev3=PROPOSED-p3;identity=offline migration root uid 0;basis=root fence completion binds the migration inventory revision; timer adoption permits only its exact previous revision plus one and recommits visible failed writes 2cae20b131f8095abb56be863ed457b0cb22c42d28d486488edc83ba9d519986

@coverage . 2145f21807743eb683387a61738d6bb16d9c41d7d69a9071dbac765ccd9ec2d6

-@coverage internal/activator 0b9179cf42fbbe917aa7e3dd5b1b03eb67a5f77689bc99dc47e8183b735174c1

+@coverage internal/activator 6c7d3e5f221891dde6ea156c12c8d04fd5a13e377cedb660eca0ccb7f67ff693

@coverage internal/console 7702e7033dee36caa5129394955cb70d09ed5b3fd81dbf752ff37fdf2ed4edef

@coverage internal/edgesfile e0a7dd2795beedde4d17389b96b562f3f9438ad8505ea19a62d1e9c4b921aefb

@coverage internal/host/units bf6ae516012701e0ca41908167e4efd60dd3c2c24e9af1555dec077e8be44461