carlos / platform Public

Claim activation tools, run 7: invariant coverage (not for merge)

Restore admission and ownership fixes; 105 real reboots and 30 post-done pending-journal cases on production units.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Paul Campbell pushed by claude 1 week ago d474c7cd888490b5a94e2fd939b61b5ba13fc578 parent 310e4ca

4 files changed, +312 −29

internal/host/activation_invariants_rootrunner_linux_test.go +36 −0
internal/host/activation_matrix_rootrunner_linux_test.go +243 −6
internal/host/activation_offline_linux.go +31 −21
internal/provision/testdata/plan14-root-reads-activation-tools.tsv +2 −2

CI passed — run details

CI log

=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	288.250s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	2.743s
ok  	github.com/carlosframework/platform/cmd/moveprobe	92.190s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.704s
ok  	github.com/carlosframework/platform/internal/activator	250.467s
ok  	github.com/carlosframework/platform/internal/agentsock	0.017s
ok  	github.com/carlosframework/platform/internal/alerting	0.008s
ok  	github.com/carlosframework/platform/internal/analytics	0.019s
ok  	github.com/carlosframework/platform/internal/appconfig	2.398s
ok  	github.com/carlosframework/platform/internal/apperr	0.075s
ok  	github.com/carlosframework/platform/internal/applog	0.138s
ok  	github.com/carlosframework/platform/internal/awsinv	0.007s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.009s
ok  	github.com/carlosframework/platform/internal/boxid	0.843s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.011s
ok  	github.com/carlosframework/platform/internal/caps	0.015s
ok  	github.com/carlosframework/platform/internal/certwarm	0.003s
ok  	github.com/carlosframework/platform/internal/cgroup	0.706s
ok  	github.com/carlosframework/platform/internal/changelog	0.311s
ok  	github.com/carlosframework/platform/internal/claimactivation	0.026s
ok  	github.com/carlosframework/platform/internal/clamd	0.056s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	389.351s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.123s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.012s
ok  	github.com/carlosframework/platform/internal/dbverify	0.006s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.004s
ok  	github.com/carlosframework/platform/internal/docsite	0.006s
ok  	github.com/carlosframework/platform/internal/domains	2.333s
ok  	github.com/carlosframework/platform/internal/doorkey	0.006s
ok  	github.com/carlosframework/platform/internal/economics	0.004s
ok  	github.com/carlosframework/platform/internal/edge	26.628s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.009s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.133s
ok  	github.com/carlosframework/platform/internal/gate	1.657s
ok  	github.com/carlosframework/platform/internal/genselect	0.017s
ok  	github.com/carlosframework/platform/internal/geo	0.013s
ok  	github.com/carlosframework/platform/internal/geoip	0.034s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.156s
ok  	github.com/carlosframework/platform/internal/host	173.669s
ok  	github.com/carlosframework/platform/internal/hostactors	0.297s
ok  	github.com/carlosframework/platform/internal/hostbox	2.490s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.514s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.015s
ok  	github.com/carlosframework/platform/internal/hostexec	4.698s
ok  	github.com/carlosframework/platform/internal/hostfence	0.034s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.547s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.015s
ok  	github.com/carlosframework/platform/internal/hostguard	1.034s
ok  	github.com/carlosframework/platform/internal/hostops	1.550s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.015s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.015s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.451s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.140s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.015s
ok  	github.com/carlosframework/platform/internal/infraview	0.015s
ok  	github.com/carlosframework/platform/internal/instancectl	5.220s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.708s
ok  	github.com/carlosframework/platform/internal/instancefs	0.015s
ok  	github.com/carlosframework/platform/internal/instancekey	0.017s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.021s
ok  	github.com/carlosframework/platform/internal/layout	1.529s
ok  	github.com/carlosframework/platform/internal/lease	9.796s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.021s
ok  	github.com/carlosframework/platform/internal/ledger	0.035s
ok  	github.com/carlosframework/platform/internal/mail	0.172s
ok  	github.com/carlosframework/platform/internal/maintenance	0.081s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.458s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.074s
ok  	github.com/carlosframework/platform/internal/metrics	0.006s
ok  	github.com/carlosframework/platform/internal/moves	21.084s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.774s
ok  	github.com/carlosframework/platform/internal/objstore	0.190s
ok  	github.com/carlosframework/platform/internal/ops	0.180s
ok  	github.com/carlosframework/platform/internal/park	0.217s
ok  	github.com/carlosframework/platform/internal/privatefile	0.006s
ok  	github.com/carlosframework/platform/internal/provision	28.412s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.014s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.479s
ok  	github.com/carlosframework/platform/internal/relay	7.232s
ok  	github.com/carlosframework/platform/internal/release	21.695s
ok  	github.com/carlosframework/platform/internal/rollout	3.614s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.003s
ok  	github.com/carlosframework/platform/internal/rootaws	0.007s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.060s
ok  	github.com/carlosframework/platform/internal/rootstate	0.026s
ok  	github.com/carlosframework/platform/internal/runs	0.577s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.003s
ok  	github.com/carlosframework/platform/internal/schedule	2.475s
ok  	github.com/carlosframework/platform/internal/seeds	0.010s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	8.820s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	4.272s
ok  	github.com/carlosframework/platform/internal/status	0.038s
ok  	github.com/carlosframework/platform/internal/steering	0.005s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.363s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.004s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.006s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.536s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 48693410 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: cached, digest verified
  mc: cached, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.022s
=== RUN   TestT4StorageProbeMinio
2026/10/16 22:02:45 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DF21A431D7DAD3, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.020s
minio contract: PASS

diff --git a/internal/host/activation_invariants_rootrunner_linux_test.go b/internal/host/activation_invariants_rootrunner_linux_test.go

index 007ca683c..1a417d909 100644

--- a/internal/host/activation_invariants_rootrunner_linux_test.go

+++ b/internal/host/activation_invariants_rootrunner_linux_test.go

@@ -246,3 +246,39 @@ func TestRootActivationConfiguredUpdateRoot(t *testing.T) {

t.Fatal("relative root accepted")

}

+func TestRootActivationDoneUpdateRace(t *testing.T) {

+ tx, _ := offlineFixture(t)

+ if err := tx.begin(t.Context(), false); err != nil {

+ t.Fatal(err)

+ }

+ var held func()

+ defer func() {

+ if held != nil {

+ held()

+ }

+ }()

+ tx.checkpoint = func(point string) error {

+ r := offlineRecord(t, tx)

+ if point == "journal-done" && !r.EdgePending && r.TriggersPending {

+ var err error

+ held, err = tx.settled(t.Context(), false)

+ return err

+ }

+ return nil

+ }

+ err := tx.restore(t.Context(), false)

+ if err == nil || !strings.Contains(err.Error(), "settle the update first") {

+ t.Fatal("update race admitted", err)

+ }

+ r := offlineRecord(t, tx)

+ if r.Phase != "done" || r.EdgePending || !r.TriggersPending {

+ t.Fatal("lost continuation", r)

+ }

+ held()

+ held = nil

+ tx.checkpoint = nil

+ if err := tx.restore(t.Context(), false); err != nil {

+ t.Fatal("settled retry", err)

+ }

diff --git a/internal/host/activation_matrix_rootrunner_linux_test.go b/internal/host/activation_matrix_rootrunner_linux_test.go

index 608199fae..99b3ce629 100644

--- a/internal/host/activation_matrix_rootrunner_linux_test.go

+++ b/internal/host/activation_matrix_rootrunner_linux_test.go

@@ -5,14 +5,20 @@ package host

import (

"bytes"

"context"

+ "crypto/ed25519"

"encoding/json"

"errors"

"fmt"

"os"

"path/filepath"

+ "reflect"

"strings"

"testing"

"time"

+ "github.com/carlosframework/platform/internal/hostactors"

+ "github.com/carlosframework/platform/internal/hostfloor"

+ "github.com/carlosframework/platform/internal/release"

)

const activationMatrixDir = "/var/lib/carlos-activation-matrix"

@@ -117,6 +123,17 @@ func activationMatrixCheck(t *testing.T, tx activationOffline, point string) {

if strings.HasPrefix(point, "start-") && strings.HasSuffix(point, ".timer") && r.Phase != "done" {

t.Fatal("timer activated before closure")

}

+ if r.Phase == "done" && (point == "stage-"+activationEdgeUnit || point == "replace-"+activationEdgeUnit || strings.HasPrefix(point, "start-") && (strings.HasSuffix(point, ".timer") || strings.HasSuffix(point, ".path"))) {

+ lock, err := os.OpenFile(filepath.Join(tx.updateDir(), systemLockFile), os.O_RDWR, 0)

+ if err != nil {

+ t.Fatal(err)

+ }

+ err = flockExclusiveNonblock(lock)

+ lock.Close()

+ if err == nil {

+ t.Fatal("continuation effect without updater exclusion", point)

+ }

}

func activationMatrixInterrupt(t *testing.T, tx activationOffline, cut int) (string, bool) {

@@ -143,6 +160,29 @@ func activationMatrixInterrupt(t *testing.T, tx activationOffline, cut int) (str

func activationMatrixResume(t *testing.T, tx activationOffline, point string) {

t.Helper()

before := offlineRecord(t, tx)

+ foreign := map[string]activationOfflineItem{}

+ if before.Phase == "done" {

+ for _, item := range before.Items {

+ path := filepath.Join(tx.units, item.Name)

+ if sameActivationFile(path, filepath.Join(tx.state, item.Name+".original")) || sameActivationFile(path, filepath.Join(tx.state, item.Name+".mask")) {

+ continue

+ }

+ snapshot, err := activationUnitSnapshot(path, item.Name)

+ if err != nil {

+ t.Fatal(err)

+ }

+ foreign[item.Name] = snapshot

+ }

+ real := tx.systemctl

+ tx.systemctl = func(ctx context.Context, args ...string) ([]byte, error) {

+ if len(args) > 1 && (args[0] == "start" || args[0] == "stop" || args[0] == "reset-failed") {

+ if _, exists := foreign[args[1]]; exists {

+ t.Fatal("closed continuation reclaimed service state", args)

+ }

+ return real(ctx, args...)

+ }

if before.Phase != "done" {

activationMatrixPending(t, tx, tx.restore)

}

@@ -171,10 +211,16 @@ func activationMatrixResume(t *testing.T, tx activationOffline, point string) {

if r.Phase != "done" || r.EdgePending || r.TriggersPending {

t.Fatal("unfinished continuation", r.Phase, r.EdgePending, r.TriggersPending)

}

+ for name, snapshot := range foreign {

+ got, err := activationUnitSnapshot(filepath.Join(tx.units, name), name)

+ if err != nil || !reflect.DeepEqual(got, snapshot) {

+ t.Fatal("closed continuation overwrote recovered unit", name, err)

+ }

// Once done is durable, timer-driven adoption is ordinary operation. The

// close checkpoint, not a later live service observation, checks equality.

for _, item := range r.Items[:len(activationTriggers)] {

- if item.Active {

+ if item.Active && sameActivationFile(filepath.Join(tx.units, item.Name), filepath.Join(tx.state, item.Name+".original")) {

_, active, err := tx.status(t.Context(), item.Name)

if err != nil || active != "active" {

t.Fatal("active trigger not restored", item.Name, active, err)

@@ -193,7 +239,7 @@ func TestRootActivationProductionInvariantMatrix(t *testing.T) {

activationMatrixBaseline(t, tx, timer)

point, interrupted := activationMatrixInterrupt(t, tx, cut)

activationMatrixResume(t, tx, point)

- t.Logf("matrix reboot=false timer=%t pending={present:refused,absent:restored} cut=%d point=%s", timer, cut, point)

+ t.Logf("matrix reboot=false timer=%t pending={open:refused,closed:ordinary-recovery,absent:restored} cut=%d point=%s", timer, cut, point)

if !interrupted {

complete = true

break

@@ -216,6 +262,7 @@ func TestRootActivationProductionInvariantBootStep(t *testing.T) {

if os.IsNotExist(err) {

tx = setupActivationProduction(t)

activationMatrixTimer(t)

+ activationMatrixOwnTmpfiles(t)

c.Cut = 1

} else if err != nil || json.Unmarshal(raw, &c) != nil {

t.Fatal("cursor", err)

@@ -229,14 +276,16 @@ func TestRootActivationProductionInvariantBootStep(t *testing.T) {

t.Fatal("no actual reboot")

}

activationMatrixCheck(t, tx, "boot-resume")

- if c.Pending {

+ if c.Pending && offlineRecord(t, tx).Phase == "done" {

+ activationMatrixRecovered(t, tx)

+ } else if c.Pending {

activationMatrixRefusePending(t, tx, tx.restore)

if err := os.Remove(filepath.Join(tx.updateDir(), "journal.json")); err != nil {

t.Fatal(err)

}

activationMatrixResume(t, tx, c.Point)

- t.Logf("matrix reboot=true timer=%t pending={present:refused,absent:restored} cut=%d point=%s boot_pending=%t boot=%s", c.Timer, c.Cut, c.Point, c.Pending, strings.TrimSpace(string(boot)))

+ t.Logf("matrix reboot=true timer=%t pending={open:refused,closed:ordinary-recovery,absent:restored} cut=%d point=%s boot_pending=%t boot=%s", c.Timer, c.Cut, c.Point, c.Pending, strings.TrimSpace(string(boot)))

c.Cut++

}

if c.Cut > 120 {

@@ -249,6 +298,9 @@ func TestRootActivationProductionInvariantBootStep(t *testing.T) {

activationMatrixResume(t, tx, point)

if c.Timer {

cleanupActivationProduction(t, tx)

+ if err := os.Remove("/etc/tmpfiles.d/carlos.conf"); err != nil && !os.IsNotExist(err) {

+ t.Fatal(err)

+ }

activationProductionWrite(t, activationMatrixDir+"/control", []byte("done\n"), 0600)

return

}

@@ -256,8 +308,10 @@ func TestRootActivationProductionInvariantBootStep(t *testing.T) {

c.Cut = 1

continue

}

- c.Pending = offlineRecord(t, tx).Phase != "done" && os.Getenv("ACTIVATION_MATRIX_PENDING") != "absent"

- if c.Pending {

+ c.Pending = os.Getenv("ACTIVATION_MATRIX_PENDING") != "absent"

+ if c.Pending && offlineRecord(t, tx).Phase == "done" {

+ activationMatrixClosedJournal(t, tx)

+ } else if c.Pending {

raw, err := json.Marshal(systemJournal{Phase: systemPhaseInstalled, Target: "v2.0.0", PrevBuild: "v1.0.0", StartedAt: time.Now()})

if err != nil {

t.Fatal(err)

@@ -290,3 +344,186 @@ func activationMatrixTimer(t *testing.T) {

activationProductionWrite(t, "/etc/systemd/system/carlos-adopt.timer.d/10-runner-deadline.conf", []byte("[Timer]\nOnBootSec=\nOnBootSec=1us\nRandomizedDelaySec=0\n"), 0644)

servingRunnerSystemctl(t, "daemon-reload")

}

+// Exercise recovery's real unit replacement at every closed continuation cut.

+// The staged rollback set contains the production units this fixture owns.

+func activationMatrixClosedJournal(t *testing.T, tx activationOffline) {

+ t.Helper()

+ if offlineRecord(t, tx).Phase != "done" {

+ t.Fatal("pending recovery fixture before closure")

+ }

+ prev := filepath.Join(tx.updateDir(), "staging", "prev")

+ tmp, err := os.ReadFile("/etc/tmpfiles.d/carlos-activation-runner.conf")

+ if err != nil {

+ t.Fatal(err)

+ }

+ activationProductionWrite(t, filepath.Join(prev, "etc/tmpfiles.d/carlos.conf"), tmp, 0644)

+ for _, name := range activationProductionNames() {

+ raw, err := unitsFS.ReadFile("units/" + name)

+ if err != nil {

+ t.Fatal(err)

+ }

+ activationProductionWrite(t, filepath.Join(prev, "etc/systemd/system", name), raw, 0644)

+ }

+ keyBytes, err := os.ReadFile(offlineRebootState + "/signing-key")

+ if err != nil {

+ t.Fatal(err)

+ }

+ key := ed25519.PrivateKey(keyBytes)

+ disk := hostfloor.Disk{Dir: "/opt/carlos/system/guard", Bucket: "activation-seed-fixture", PublicKey: key.Public().(ed25519.PublicKey)}

+ _, floor, err := disk.Load()

+ if err != nil {

+ t.Fatal(err)

+ }

+ id := fmt.Sprintf("%032x", time.Now().UnixNano())

+ enrollment := strings.Repeat("a", 32)

+ hash := hostfloor.Digest(activationCarlosImage)

+ actor := "box:seed-fixture"

+ box := hostactors.Box{Enrollment: enrollment, Namespace: "test", BoxID: "seed-fixture", Label: "fixture", Region: "ie", Pool: "default", Cloud: hostactors.Cloud{Provider: "aws", Account: "fixture", Region: "eu-west-1", InstanceID: "i-fixture"}, EvidenceSHA256: strings.Repeat("b", 64), State: "active"}

+ reservation := hostactors.Reservation{ID: id, FloorSHA256: floor.Digest, Actors: map[string]string{actor: enrollment}, Releases: map[string]hostactors.Report{actor: {Enrollment: enrollment}}, Hashes: map[string][]string{actor: {hash, hash, hash}}}

+ roster := hostactors.Roster{V: 1, DeploymentBucket: disk.Bucket, Revision: 1, Boxes: map[string]hostactors.Box{"seed-fixture": box}, Consoles: map[string]hostactors.Console{}, Holds: map[string]hostactors.Reservation{}, Activations: map[string]hostactors.Reservation{id: reservation}}

+ if err := roster.Validate(disk.Bucket); err != nil {

+ t.Fatal(err)

+ }

+ wire, err := release.SignRootEnvelope(release.RootSigner(key), roster)

+ if err != nil {

+ t.Fatal(err)

+ }

+ guard, err := json.Marshal(guardTransaction{Roster: wire, ID: id, Actor: actor, Enrollment: enrollment})

+ if err != nil {

+ t.Fatal(err)

+ }

+ if err := hostfloor.Durable(filepath.Join(disk.Dir, "activation.json"), guard, 0644); err != nil {

+ t.Fatal(err)

+ }

+ if err := os.MkdirAll("/data/.hostexec", 0755); err != nil {

+ t.Fatal(err)

+ }

+ if err := disk.Begin(t.Context(), id, "/data"); err != nil {

+ t.Fatal(err)

+ }

+ raw, err := json.Marshal(bootstrapPlan{Enabled: []string{"carlos-edge.service", "carlos-system-recover.service", "carlos-adopt.timer"}})

+ if err != nil {

+ t.Fatal(err)

+ }

+ activationProductionWrite(t, filepath.Join(prev, "plan.json"), raw, 0600)

+ activationProductionWrite(t, prevVersionPath("/opt/carlos"), []byte("v1.0.0\n"), 0600)

+ raw, err = json.Marshal(systemJournal{Phase: systemPhaseInstalled, Target: "v2.0.0", PrevBuild: "v1.0.0", StartedAt: time.Now()})

+ if err != nil {

+ t.Fatal(err)

+ }

+ if err := writeFileDurable(filepath.Join(tx.updateDir(), "journal.json"), raw, 0600); err != nil {

+ t.Fatal(err)

+ }

+func activationMatrixRecovered(t *testing.T, tx activationOffline) {

+ t.Helper()

+ deadline := time.Now().Add(20 * time.Second)

+ for {

+ servingRunnerSystemctl(t, "start", "carlos-system-recover.service")

+ _, err := os.Lstat(filepath.Join(tx.updateDir(), "journal.json"))

+ if os.IsNotExist(err) {

+ break

+ }

+ if err != nil || time.Now().After(deadline) {

+ t.Fatal("recovery left pending journal", err)

+ }

+ // The overdue adoption timer can briefly hold the shared update lock.

+ // A successful deferred start is not evidence that recovery completed.

+ r := offlineRecord(t, tx)

+ if r.EdgePending || r.TriggersPending {

+ activationMatrixRefusePending(t, tx, tx.restore)

+ }

+ t.Log("recovery deferred; journal retained; retrying within deadline")

+ time.Sleep(100 * time.Millisecond)

+ }

+ raw, err := os.ReadFile(filepath.Join(tx.updateDir(), "result.json"))

+ if err != nil || !bytes.Contains(raw, []byte(systemOutcomeRolledBack)) {

+ t.Fatal("recovery did not record rollback", string(raw), err)

+ }

+ if err := tx.updateSettled(); err != nil {

+ t.Fatal("recovery did not settle update state", err)

+ }

+ guardRaw, err := os.ReadFile("/opt/carlos/system/guard/activation.json")

+ var guard guardTransaction

+ if err != nil || json.Unmarshal(guardRaw, &guard) != nil {

+ t.Fatal("recovered guard transaction", err)

+ }

+ completed, err := os.ReadFile("/opt/carlos/system/guard/completed.json")

+ if err != nil || string(completed) != guard.ID {

+ t.Fatal("recovery did not complete this reservation", err)

+ }

+ for _, item := range offlineRecord(t, tx).Items {

+ if sameActivationFile(filepath.Join(tx.units, item.Name), filepath.Join(tx.state, item.Name+".original")) {

+ t.Fatal("recovery did not replace production unit", item.Name)

+ }

+func TestRootActivationProductionDonePendingMatrix(t *testing.T) {

+ for _, order := range []string{"pending-first", "recovery-first"} {

+ t.Run(order, func(t *testing.T) {

+ tx := setupActivationProduction(t)

+ defer cleanupActivationProduction(t, tx)

+ activationMatrixOwnTmpfiles(t)

+ defer os.Remove("/etc/tmpfiles.d/carlos.conf")

+ activationMatrixTimer(t)

+ for _, timer := range []bool{false, true} {

+ activationMatrixBaseline(t, tx, timer)

+ var cuts []int

+ count := 0

+ tx.checkpoint = func(point string) error {

+ count++

+ activationMatrixCheck(t, tx, point)

+ if offlineRecord(t, tx).Phase == "done" {

+ cuts = append(cuts, count)

+ }

+ return nil

+ }

+ if err := tx.restore(t.Context(), false); err != nil {

+ t.Fatal(err)

+ }

+ tx.checkpoint = nil

+ cuts = append(cuts, count+1)

+ for _, cut := range cuts {

+ activationMatrixBaseline(t, tx, timer)

+ point, _ := activationMatrixInterrupt(t, tx, cut)

+ r := offlineRecord(t, tx)

+ if r.Phase != "done" {

+ t.Fatal("closed cut changed", cut, point, r.Phase)

+ }

+ activationMatrixClosedJournal(t, tx)

+ if order == "pending-first" {

+ if r.EdgePending || r.TriggersPending {

+ activationMatrixRefusePending(t, tx, tx.restore)

+ } else {

+ saved, err := os.ReadFile(filepath.Join(tx.state, "transaction.json"))

+ if err != nil {

+ t.Fatal(err)

+ }

+ if err := tx.restore(t.Context(), false); err != nil {

+ t.Fatal("finished transaction resumed work", err)

+ }

+ after, err := os.ReadFile(filepath.Join(tx.state, "transaction.json"))

+ if err != nil || !bytes.Equal(saved, after) {

+ t.Fatal("finished transaction changed", err)

+ }

+ activationMatrixRecovered(t, tx)

+ activationMatrixResume(t, tx, point)

+ t.Logf("post-done reboot=false timer=%t pending=recovered order=%s cut=%d point=%s", timer, order, cut, point)

+ }

+ })

+ }

+func activationMatrixOwnTmpfiles(t *testing.T) {

+ t.Helper()

+ if _, err := os.Lstat("/etc/tmpfiles.d/carlos.conf"); !os.IsNotExist(err) {

+ t.Fatal("pre-existing tmpfiles configuration", err)

+ }

diff --git a/internal/host/activation_offline_linux.go b/internal/host/activation_offline_linux.go

index cb9c9a3cf..e3f628327 100644

--- a/internal/host/activation_offline_linux.go

+++ b/internal/host/activation_offline_linux.go

@@ -439,13 +439,12 @@ func (t activationOffline) restore(ctx context.Context, dry bool) error {

if r.Phase == "done" && !r.EdgePending && !r.TriggersPending {

return nil

}

- release := func() {}

+ release, err := t.settled(ctx, dry)

+ if err != nil {

+ return err

+ }

+ defer func() { release() }()

if r.Phase != "done" {

- release, err = t.settled(ctx, dry)

- if err != nil {

- return err

- }

- defer func() { release() }()

for _, item := range r.Items {

if err = t.checkItem(item); err != nil {

return fmt.Errorf("%s: recorded file inode, content, or metadata changed: %w", item.Name, err)

@@ -553,23 +552,29 @@ func (t activationOffline) restore(ctx context.Context, dry bool) error {

if err = t.save(state, r); err != nil {

return err

}

- // Recovery can take the update lock only after closure.

- release()

- release = func() {}

}

- // Release the edge file before triggers: ordinary adoption may rewrite units

- // once done is durable. No transaction-owned replacement follows a trigger.

+ // After closure, another root writer may have replaced any unit. Only an

+ // owned mask can still authorize replacement; retain updater exclusion

+ // across that last rename, then release it before the edge's recovery job.

+ edgeOwned := t.checkItem(edge) == nil

if r.EdgePending {

- if err = t.replace(units, state, edge, true); err != nil {

- return err

- }

- if _, err = t.systemctl(ctx, "daemon-reload"); err != nil {

- return err

- }

- if err = t.point("edge-reload"); err != nil {

- return err

+ if sameActivationFile(filepath.Join(t.units, edge.Name), filepath.Join(t.state, edge.Name+".mask")) || edgeOwned {

+ if err = t.replace(units, state, edge, true); err != nil {

+ return err

+ }

+ if _, err = t.systemctl(ctx, "daemon-reload"); err != nil {

+ return err

+ }

+ if err = t.point("edge-reload"); err != nil {

+ return err

+ }

}

+ }

+ release()

+ release = func() {}

+ // Inactive state was established before done. Never stop an edge that

+ // ordinary boot or recovery has since started. Only active startup is owed.

+ if r.EdgePending && edgeOwned && edge.Active && !edge.Masked && t.checkItem(edge) == nil {

if err = t.reconcile(ctx, edge); err != nil {

return err

}

@@ -584,8 +589,13 @@ func (t activationOffline) restore(ctx context.Context, dry bool) error {

return err

}

if r.TriggersPending {

+ nextRelease, e := t.settled(ctx, false)

+ if e != nil {

+ return e

+ }

+ release = nextRelease

for _, item := range r.Items[:len(activationTriggers)] {

- if item.Active && !item.Masked {

+ if item.Active && !item.Masked && t.checkItem(item) == nil {

if _, err = t.systemctl(ctx, "start", item.Name); err != nil {

return err

}

diff --git a/internal/provision/testdata/plan14-root-reads-activation-tools.tsv b/internal/provision/testdata/plan14-root-reads-activation-tools.tsv

index fe9b6a020..d35e657a2 100644

--- a/internal/provision/testdata/plan14-root-reads-activation-tools.tsv

+++ b/internal/provision/testdata/plan14-root-reads-activation-tools.tsv

@@ -1,7 +1,7 @@

# Activation-tools runs 1–6: proposals only; loaded after all existing overlays.

@coverage . efeeda08e816fc2c7e3aa451b06cce95ab5778af79682290cf98a60d8b9a841a

@coverage internal/claimactivation 625d82dce14bae92f619077ca92a8c9066a9446545dfcc96a6f818e89e20d4c2

-@coverage internal/host e9a3134343f930cb50566629d2c2d8e3613e75557036efa67158af6a52c113f2

+@coverage internal/host 0b153b028de260c2050dfd4087bf4ba16eedb7dfc685797d24ed8edad86d9c9b

@coverage internal/hostfence 552570510fb62d2a8822a3f24fa76aeb1437c0a9516966dc5ce15ef58e3fb4c6

internal/claimactivation/runner.go Runner.Run authority F,G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=trusted operator phases bind expected signed roster revision, exact deployment and actors; verify each write; dry-run overlay never calls backing Put file:Read@310,file:Read@5986,file:Read@6318 683059aedf27656d4b49c88102089b2bb2f2ce7fcc108ff8545e52a86a0f8385

internal/claimactivation/runner.go checkedStore.Delete authority F,G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=trusted operator phases bind expected signed roster revision, exact deployment and actors; verify each write; dry-run overlay never calls backing Put 6bc9103a30cfd53cea59d611ccacb7dcc0ca0ea620b403497ef543f5bdaa0217

@@ -30,7 +30,7 @@ internal/host/activation_offline_linux.go activationOffline.prepare authority G,

internal/host/activation_offline_linux.go activationOffline.preview authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit root operator or root updater;basis=settled update lock and journal precondition; all writers excluded until done; bounded full close verification with held triggers and masked edge; pending trigger and edge continuation after durable done; absent transaction retains legacy behavior a092bb6fb96f64e406a266cef59a4516145a7f58935ff8d1bbc42a96a653aab1

internal/host/activation_offline_linux.go activationOffline.read authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit root operator or root updater;basis=settled update lock and journal precondition; all writers excluded until done; bounded full close verification with held triggers and masked edge; pending trigger and edge continuation after durable done; absent transaction retains legacy behavior 6ffaa998c2441b9a2d7889275b526bace11feb269df6856527b199a40cfa5bec

internal/host/activation_offline_linux.go activationOffline.replace authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit root operator or root updater;basis=settled update lock and journal precondition; all writers excluded until done; bounded full close verification with held triggers and masked edge; pending trigger and edge continuation after durable done; absent transaction retains legacy behavior 2a64167ac7053a5c73fe64aa9fea7e4f8db3b1409c7e34fa78c1b842328ccdbb

-internal/host/activation_offline_linux.go activationOffline.restore authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit root operator or root updater;basis=settled update lock and journal precondition; all writers excluded until done; bounded full close verification with held triggers and masked edge; pending trigger and edge continuation after durable done; absent transaction retains legacy behavior; only recorded-active unmasked triggers defer; all other trigger states compare exactly df2ea187884d439b7b6cac083b0ff53224bd680ab21d3f2deced4296cbcb41e0

+internal/host/activation_offline_linux.go activationOffline.restore authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit root operator or root updater;basis=settled update lock and journal precondition; all writers excluded until done; bounded full close verification with held triggers and masked edge; pending trigger and edge continuation after durable done; absent transaction retains legacy behavior; only recorded-active unmasked triggers defer; all other trigger states compare exactly; run7=unfinished done resumes require settled updater exclusion; owned edge mask replacement holds updater lock; replaced units relinquished without file or state mutation; inactive edge never stopped after closure; active owned triggers start under updater lock bb95e89d1bf5363702accd83c21eacefd91f6659b6da62be3f5c39574c907091

internal/host/activation_offline_linux.go activationOffline.save authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit root operator or root updater;basis=settled update lock and journal precondition; all writers excluded until done; bounded full close verification with held triggers and masked edge; pending trigger and edge continuation after durable done; absent transaction retains legacy behavior 3082a1585eb287687e05756f495aae7de7e11aab6e96bbee920c50b4807ab9b7

internal/host/activation_offline_linux.go activationOffline.status authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit root operator or root updater;basis=settled update lock and journal precondition; all writers excluded until done; bounded full close verification with held triggers and masked edge; pending trigger and edge continuation after durable done; absent transaction retains legacy behavior 616c3058401766ada7c16395412aaca50b4d68be8d3a2e57ae969a53e21292ff

internal/host/activation_offline_linux.go activationOfflineUnits authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit root operator or root updater;basis=settled update lock and journal precondition; all writers excluded until done; bounded full close verification with held triggers and masked edge; pending trigger and edge continuation after durable done; absent transaction retains legacy behavior 372c65c44288c766e9b2aa0ece8cbc55c95fb1ccae7e9bae80a5bfcda2550b4b