amadan

carlos / platform Public

Claim activation tools, run 5: restore closes before edge start (not for merge)

Restore durably closes the transaction before starting the edge, so guarded recovery cannot deadlock; resuming reconciles every recorded service and trigger; witnesses on production units (guard-armed edge, real recovery unit) across reboots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paul Campbell pushed by claude a93395a3b833f7c0418705c3289bf1138d24568e parent ac70dab
13 files changed, +900 −1061
  • internal/host/activation_adoption_linux.go +6 −2
  • internal/host/activation_adoption_other.go +2 −0
  • internal/host/activation_install_rootrunner_linux_test.go +2 −14
  • internal/host/activation_offline_linux.go +55 −32
  • internal/host/activation_offline_rootrunner_linux_test.go +5 −192
  • internal/host/activation_production_reboot_rootrunner_linux_test.go +191 −0
  • internal/host/activation_production_rootrunner_linux_test.go +488 −0
  • internal/host/activation_restore_order_rootrunner_linux_test.go +6 −398
  • internal/host/activation_seed_cli_rootrunner_linux_test.go +67 −100
  • internal/host/activation_seed_rootrunner_linux_test.go +48 −314
  • internal/host/systemapply.go +1 −1
  • internal/provision/testdata/plan14-root-reads-activation-tools.tsv +9 −7
  • scripts/activation-tools-root-runner.sh +20 −1

CI passed — run details

CI log 
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	303.867s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	2.820s
ok  	github.com/carlosframework/platform/cmd/moveprobe	92.228s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.704s
ok  	github.com/carlosframework/platform/internal/activator	250.856s
ok  	github.com/carlosframework/platform/internal/agentsock	0.033s
ok  	github.com/carlosframework/platform/internal/alerting	0.008s
ok  	github.com/carlosframework/platform/internal/analytics	0.029s
ok  	github.com/carlosframework/platform/internal/appconfig	2.451s
ok  	github.com/carlosframework/platform/internal/apperr	0.095s
ok  	github.com/carlosframework/platform/internal/applog	0.139s
ok  	github.com/carlosframework/platform/internal/awsinv	0.004s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.005s
ok  	github.com/carlosframework/platform/internal/boxid	0.706s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.008s
ok  	github.com/carlosframework/platform/internal/caps	0.013s
ok  	github.com/carlosframework/platform/internal/certwarm	0.002s
ok  	github.com/carlosframework/platform/internal/cgroup	0.693s
ok  	github.com/carlosframework/platform/internal/changelog	0.305s
ok  	github.com/carlosframework/platform/internal/claimactivation	0.029s
ok  	github.com/carlosframework/platform/internal/clamd	0.055s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	393.293s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.259s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.010s
ok  	github.com/carlosframework/platform/internal/dbverify	0.005s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.004s
ok  	github.com/carlosframework/platform/internal/docsite	0.005s
ok  	github.com/carlosframework/platform/internal/domains	2.363s
ok  	github.com/carlosframework/platform/internal/doorkey	0.010s
ok  	github.com/carlosframework/platform/internal/economics	0.005s
ok  	github.com/carlosframework/platform/internal/edge	26.826s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.009s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.143s
ok  	github.com/carlosframework/platform/internal/gate	1.656s
ok  	github.com/carlosframework/platform/internal/genselect	0.024s
ok  	github.com/carlosframework/platform/internal/geo	0.012s
ok  	github.com/carlosframework/platform/internal/geoip	0.034s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.159s
ok  	github.com/carlosframework/platform/internal/host	174.168s
ok  	github.com/carlosframework/platform/internal/hostactors	0.354s
ok  	github.com/carlosframework/platform/internal/hostbox	2.628s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.552s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.013s
ok  	github.com/carlosframework/platform/internal/hostexec	4.715s
ok  	github.com/carlosframework/platform/internal/hostfence	0.039s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.464s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.013s
ok  	github.com/carlosframework/platform/internal/hostguard	1.029s
ok  	github.com/carlosframework/platform/internal/hostops	1.568s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.013s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.011s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.446s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.144s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.014s
ok  	github.com/carlosframework/platform/internal/infraview	0.012s
ok  	github.com/carlosframework/platform/internal/instancectl	5.220s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.749s
ok  	github.com/carlosframework/platform/internal/instancefs	0.016s
ok  	github.com/carlosframework/platform/internal/instancekey	0.013s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.017s
ok  	github.com/carlosframework/platform/internal/layout	1.567s
ok  	github.com/carlosframework/platform/internal/lease	9.809s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.017s
ok  	github.com/carlosframework/platform/internal/ledger	0.015s
ok  	github.com/carlosframework/platform/internal/mail	0.154s
ok  	github.com/carlosframework/platform/internal/maintenance	0.084s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.516s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.082s
ok  	github.com/carlosframework/platform/internal/metrics	0.006s
ok  	github.com/carlosframework/platform/internal/moves	21.044s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.758s
ok  	github.com/carlosframework/platform/internal/objstore	0.183s
ok  	github.com/carlosframework/platform/internal/ops	0.171s
ok  	github.com/carlosframework/platform/internal/park	0.215s
ok  	github.com/carlosframework/platform/internal/privatefile	0.004s
ok  	github.com/carlosframework/platform/internal/provision	28.839s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.014s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.516s
ok  	github.com/carlosframework/platform/internal/relay	7.243s
ok  	github.com/carlosframework/platform/internal/release	21.384s
ok  	github.com/carlosframework/platform/internal/rollout	3.653s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.004s
ok  	github.com/carlosframework/platform/internal/rootaws	0.007s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.057s
ok  	github.com/carlosframework/platform/internal/rootstate	0.026s
ok  	github.com/carlosframework/platform/internal/runs	0.577s
ok  	github.com/carlosframework/platform/internal/s3compat	0.006s
ok  	github.com/carlosframework/platform/internal/safefile	0.006s
ok  	github.com/carlosframework/platform/internal/schedule	2.492s
ok  	github.com/carlosframework/platform/internal/seeds	0.011s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	9.086s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.978s
ok  	github.com/carlosframework/platform/internal/status	0.040s
ok  	github.com/carlosframework/platform/internal/steering	0.006s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.387s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.005s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.005s
ok  	github.com/carlosframework/platform/internal/unixdial	0.002s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.004s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.548s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 48693410 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: cached, digest verified
  mc: cached, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.025s
=== RUN   TestT4StorageProbeMinio
2026/10/16 18:29:45 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DF160475C19EF7, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.020s
minio contract: PASS
diff --git a/internal/host/activation_adoption_linux.go b/internal/host/activation_adoption_linux.go
index 91c261a2c..e41f9708c 100644
--- a/internal/host/activation_adoption_linux.go
+++ b/internal/host/activation_adoption_linux.go
@@ -8,9 +8,13 @@ import (
"path/filepath"
)
-// An explicit offline transaction excludes automatic unit repair through edge startup.
+// An explicit offline transaction excludes automatic unit repair until closure.
// A process lock alone would release that exclusion on interruption or reboot.
func activationAdoptionAllowed(etc string) error {
+ return activationTransactionAllowed(etc, false)
+}
+
+func activationTransactionAllowed(etc string, recovery bool) error {
statePath := filepath.Join(etc, "carlos", "claim-activation", "offline")
if _, err := os.Lstat(statePath); errors.Is(err, os.ErrNotExist) {
return nil
@@ -31,7 +35,7 @@ func activationAdoptionAllowed(etc string) error {
if err != nil {
return err
}
- if r.Phase != "done" {
+ if r.Phase != "done" && !(recovery && r.Phase == "complete") {
return errActivation
}
return nil
diff --git a/internal/host/activation_adoption_other.go b/internal/host/activation_adoption_other.go
index 40ec0c8bb..7c83f2fc7 100644
--- a/internal/host/activation_adoption_other.go
+++ b/internal/host/activation_adoption_other.go
@@ -3,3 +3,5 @@
package host
func activationAdoptionAllowed(string) error { return nil }
+
+func activationTransactionAllowed(string, bool) error { return nil }
diff --git a/internal/host/activation_install_rootrunner_linux_test.go b/internal/host/activation_install_rootrunner_linux_test.go
index 6c24b128b..d5286e5a3 100644
--- a/internal/host/activation_install_rootrunner_linux_test.go
+++ b/internal/host/activation_install_rootrunner_linux_test.go
@@ -214,23 +214,11 @@ func TestRootActivationConsoleInstaller(t *testing.T) {
}
func TestRootActivationControlSystemd(t *testing.T) {
- tx := setupOfflineReal(t)
- defer cleanupOfflineReal(t, tx)
+ tx := setupActivationProduction(t)
+ defer cleanupActivationProduction(t, tx)
if err := tx.begin(t.Context(), false); err != nil {
t.Fatal(err)
}
- if _, err := os.Lstat("/etc/carlos"); !os.IsNotExist(err) {
- t.Fatal("pre-existing config directory", err)
- }
- if err := os.Mkdir("/etc/carlos", 0700); err != nil {
- t.Fatal(err)
- }
- defer os.RemoveAll("/etc/carlos")
- unit := "/etc/systemd/system/" + claimControlUnit
- if _, err := os.Lstat(unit); !os.IsNotExist(err) {
- t.Fatal("pre-existing control unit", err)
- }
- defer func() { os.Remove(unit); offlineRealSystemctl(t.Context(), "daemon-reload") }()
if err := os.WriteFile("/etc/carlos/host.env", []byte("CARLOS_HOST_CLAIM=1\nCARLOS_HOST_CLAIM_ORIGINS=1\n"), 0600); err != nil {
t.Fatal(err)
}
diff --git a/internal/host/activation_offline_linux.go b/internal/host/activation_offline_linux.go
index 5b68264e0..d2f7a35a7 100644
--- a/internal/host/activation_offline_linux.go
+++ b/internal/host/activation_offline_linux.go
@@ -45,11 +45,12 @@ type activationOfflineItem struct {
Digest string
}
type activationOfflineRecord struct {
- V int
- Units string
- Phase string
- Items []activationOfflineItem
- SeedGuard *activationOfflineItem `json:",omitempty"`
+ V int
+ Units string
+ Phase string
+ Items []activationOfflineItem
+ SeedGuard *activationOfflineItem `json:",omitempty"`
+ EdgePending bool `json:",omitempty"`
}
func (t activationOffline) point(name string) error {
@@ -416,12 +417,14 @@ func (t activationOffline) restore(ctx context.Context, dry bool) error {
if r.Phase == "prepared" || r.Phase == "seed-prepared" {
return errActivation
}
- if r.Phase == "done" {
+ if r.Phase == "done" && !r.EdgePending {
return nil
}
- for _, item := range r.Items {
- if err = t.checkItem(item); err != nil {
- return err
+ if r.Phase != "done" {
+ for _, item := range r.Items {
+ if err = t.checkItem(item); err != nil {
+ return err
+ }
}
}
if dry {
@@ -442,6 +445,15 @@ func (t activationOffline) restore(ctx context.Context, dry bool) error {
if err != nil {
return err
}
+ if verb == "stop" && active == "failed" {
+ if _, err = t.systemctl(ctx, "reset-failed", item.Name); err != nil {
+ return err
+ }
+ load, active, err = t.status(ctx, item.Name)
+ if err != nil {
+ return err
+ }
+ }
wantLoad, wantActive := "loaded", "inactive"
if item.Masked {
wantLoad = "masked"
@@ -455,15 +467,17 @@ func (t activationOffline) restore(ctx context.Context, dry bool) error {
return nil
}
edge := r.Items[len(r.Items)-1]
- // The edge mask survives every pre-completion checkpoint, including reboot.
- // Adoption entrypoints also honor this journal until the final done record.
- if r.Phase != "complete" {
+ if r.Phase != "done" {
+ // A resumed complete record still owns every unit. Reconcile them all:
+ // boot may have started an enabled service or left recovery failed.
if err = t.replace(units, state, edge, false); err != nil {
return err
}
- r.Phase = "restoring"
- if err = t.save(state, r); err != nil {
- return err
+ if r.Phase != "complete" {
+ r.Phase = "restoring"
+ if err = t.save(state, r); err != nil {
+ return err
+ }
}
for _, item := range r.Items[:len(activationTriggers)] {
if _, err = t.systemctl(ctx, "stop", item.Name); err != nil {
@@ -484,15 +498,12 @@ func (t activationOffline) restore(ctx context.Context, dry bool) error {
if err = t.point("restore-reload"); err != nil {
return err
}
- for _, item := range r.Items[len(activationTriggers) : len(r.Items)-1] {
- if err = reconcile(item); err != nil {
+ if r.Phase != "complete" {
+ r.Phase = "released"
+ if err = t.save(state, r); err != nil {
return err
}
}
- r.Phase = "released"
- if err = t.save(state, r); err != nil {
- return err
- }
for _, item := range r.Items[:len(activationTriggers)] {
if err = t.replace(units, state, item, true); err != nil {
return err
@@ -504,19 +515,31 @@ func (t activationOffline) restore(ctx context.Context, dry bool) error {
if err = t.point("trigger-reload"); err != nil {
return err
}
- }
- for _, active := range []bool{false, true} {
- for _, item := range r.Items[:len(activationTriggers)] {
- if (item.Active && !item.Masked) == active {
- if err = reconcile(item); err != nil {
- return err
+ if r.Phase != "complete" {
+ // Recovery must be callable while service states are finalized.
+ // Adoption and new updates remain excluded until done.
+ r.Phase = "complete"
+ if err = t.save(state, r); err != nil {
+ return err
+ }
+ }
+ for _, item := range r.Items[len(activationTriggers) : len(r.Items)-1] {
+ if err = reconcile(item); err != nil {
+ return err
+ }
+ }
+ for _, active := range []bool{false, true} {
+ for _, item := range r.Items[:len(activationTriggers)] {
+ if (item.Active && !item.Masked) == active {
+ if err = reconcile(item); err != nil {
+ return err
+ }
}
}
}
- }
- if r.Phase != "complete" {
- // Save completion before releasing the persistent edge mask.
- r.Phase = "complete"
+ // Close durably before the edge can request its recovery dependency.
+ // EdgePending resumes only the ordinary edge step, never the transaction.
+ r.Phase, r.EdgePending = "done", true
if err = t.save(state, r); err != nil {
return err
}
@@ -533,7 +556,7 @@ func (t activationOffline) restore(ctx context.Context, dry bool) error {
if err = reconcile(edge); err != nil {
return err
}
- r.Phase = "done"
+ r.EdgePending = false
return t.save(state, r)
}
diff --git a/internal/host/activation_offline_rootrunner_linux_test.go b/internal/host/activation_offline_rootrunner_linux_test.go
index d71932438..8d64c0ee5 100644
--- a/internal/host/activation_offline_rootrunner_linux_test.go
+++ b/internal/host/activation_offline_rootrunner_linux_test.go
@@ -37,7 +37,7 @@ func offlineFixture(t *testing.T) (activationOffline, string) {
}
return activationOffline{units: units, state: state, systemctl: ctl, drained: func(context.Context) error { return nil }}, d
}
-func TestRootActivationOfflineInterruptions(t *testing.T) {
+func TestRootActivationModelOfflineInterruptions(t *testing.T) {
for _, phase := range []string{"begin", "restore"} {
for cut := 1; cut < 100; cut++ {
t.Run(fmt.Sprintf("%s/%d", phase, cut), func(t *testing.T) {
@@ -114,7 +114,7 @@ func TestRootActivationOfflineInterruptions(t *testing.T) {
}
}
}
-func TestRootActivationOfflineReplacementRefused(t *testing.T) {
+func TestRootActivationModelReplacementRefused(t *testing.T) {
tx, _ := offlineFixture(t)
if err := tx.begin(t.Context(), true); err != nil {
t.Fatal(err)
@@ -141,104 +141,7 @@ const offlineRebootState = "/etc/carlos-activation-rootrunner"
func offlineRealSystemctl(ctx context.Context, args ...string) ([]byte, error) {
return exec.CommandContext(ctx, "systemctl", args...).CombinedOutput()
}
-func setupOfflineReal(t *testing.T) activationOffline {
- t.Helper()
- if os.Geteuid() != 0 {
- t.Fatal("root required")
- }
- if err := os.Mkdir(offlineRebootState, 0700); err != nil {
- t.Fatal(err)
- }
- tx := activationOffline{units: "/etc/systemd/system", state: offlineRebootState + "/transaction", systemctl: offlineRealSystemctl, drained: quiesceLegacyRootMutators}
- for _, name := range activationOfflineUnits() {
- p := filepath.Join(tx.units, name)
- if _, err := os.Lstat(p); !os.IsNotExist(err) {
- t.Fatal("pre-existing unit", name, err)
- }
- raw := "[Unit]\nDescription=activation root runner\n[Service]\nType=simple\nExecStart=/usr/bin/sleep infinity\n[Install]\nWantedBy=multi-user.target\n"
- if strings.HasSuffix(name, ".timer") {
- raw = "[Unit]\nDescription=activation timer\n[Timer]\nOnActiveSec=1h\n[Install]\nWantedBy=timers.target\n"
- }
- if strings.HasSuffix(name, ".path") {
- raw = "[Unit]\nDescription=activation path\n[Path]\nPathExists=" + offlineRebootState + "/never\n[Install]\nWantedBy=multi-user.target\n"
- }
- if err := os.WriteFile(p, []byte(raw), 0644); err != nil {
- t.Fatal(err)
- }
- }
- for _, args := range [][]string{{"daemon-reload"}, {"enable", "--now", "carlos-edge.service", "carlos-adopt.timer"}} {
- if out, err := tx.systemctl(t.Context(), args...); err != nil {
- t.Fatal(string(out), err)
- }
- }
- return tx
-}
-func cleanupOfflineReal(t *testing.T, tx activationOffline) {
- t.Helper()
- for _, name := range activationOfflineUnits() {
- tx.systemctl(t.Context(), "disable", "--now", name)
- os.Remove(filepath.Join(tx.units, name))
- }
- tx.systemctl(t.Context(), "daemon-reload")
- os.RemoveAll(offlineRebootState)
-}
-func TestRootActivationOfflineSystemd(t *testing.T) {
- tx := setupOfflineReal(t)
- defer cleanupOfflineReal(t, tx)
- if err := tx.begin(t.Context(), false); err != nil {
- t.Fatal(err)
- }
- if err := verifyOfflineRootMigration(t.Context()); err != nil {
- t.Fatal(err)
- }
- if err := tx.restore(t.Context(), false); err != nil {
- t.Fatal(err)
- }
- out, err := tx.systemctl(t.Context(), "is-active", "carlos-edge.service")
- if err != nil || strings.TrimSpace(string(out)) != "active" {
- t.Fatal(string(out), err)
- }
-}
-func TestRootRebootActivationOfflinePhase1(t *testing.T) {
- tx := setupOfflineReal(t)
- if err := tx.begin(t.Context(), false); err != nil {
- t.Fatal(err)
- }
- boot, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
- if err != nil {
- t.Fatal(err)
- }
- if err := os.WriteFile(offlineRebootState+"/boot", boot, 0600); err != nil {
- t.Fatal(err)
- }
-}
-func TestRootRebootActivationOfflinePhase2(t *testing.T) {
- tx := activationOffline{units: "/etc/systemd/system", state: offlineRebootState + "/transaction", systemctl: offlineRealSystemctl, drained: quiesceLegacyRootMutators}
- defer cleanupOfflineReal(t, tx)
- before, err := os.ReadFile(offlineRebootState + "/boot")
- if err != nil {
- t.Fatal(err)
- }
- after, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
- if err != nil || string(before) == string(after) {
- t.Fatal("no actual reboot", err)
- }
- if err := verifyOfflineRootMigration(t.Context()); err != nil {
- t.Fatal(err)
- }
- if err := tx.begin(t.Context(), false); err != nil {
- t.Fatal("resume after reboot", err)
- }
- if err := tx.restore(t.Context(), false); err != nil {
- t.Fatal("restore after reboot", err)
- }
- out, err := tx.systemctl(t.Context(), "is-active", "carlos-edge.service")
- if err != nil || strings.TrimSpace(string(out)) != "active" {
- t.Fatal(string(out), err)
- }
-}
-
-func TestRootActivationOfflineEdgeLast(t *testing.T) {
+func TestRootActivationModelEdgeLast(t *testing.T) {
tx, _ := offlineFixture(t)
original := tx.systemctl
restoring := false
@@ -281,7 +184,7 @@ func TestRootActivationOfflineEdgeLast(t *testing.T) {
}
}
}
- if args[1] == "carlos-edge.service" && offlineRecord(t, tx).Phase != "complete" {
+ if args[1] == "carlos-edge.service" && offlineRecord(t, tx).Phase != "done" {
t.Fatal("edge started before durable completion")
}
if args[1] != "carlos-edge.service" {
@@ -325,43 +228,7 @@ func TestRootActivationProcessDrain(t *testing.T) {
}
}
-func TestRootActivationRestoreInactiveTrigger(t *testing.T) {
- for _, point := range []string{"restore-reload", "journal-released"} {
- t.Run(point, func(t *testing.T) {
- tx := setupOfflineReal(t)
- defer cleanupOfflineReal(t, tx)
- if out, err := tx.systemctl(t.Context(), "stop", "carlos-adopt.timer"); err != nil {
- t.Fatal(string(out), err)
- }
- if err := tx.begin(t.Context(), false); err != nil {
- t.Fatal(err)
- }
- injected := errors.New("restore interruption")
- tx.checkpoint = func(name string) error {
- if name == point {
- return injected
- }
- return nil
- }
- if err := tx.restore(t.Context(), false); !errors.Is(err, injected) {
- t.Fatal(err)
- }
- tx.checkpoint = nil
- if out, err := tx.systemctl(t.Context(), "start", "carlos-adopt.timer"); err == nil {
- t.Fatal("adoption timer unmasked before service reconciliation", string(out))
- }
- if err := tx.restore(t.Context(), false); err != nil {
- t.Fatal(err)
- }
- _, active, err := tx.status(t.Context(), "carlos-adopt.timer")
- if err != nil || active != "inactive" {
- t.Fatal("originally inactive timer still running", active, err)
- }
- })
- }
-}
-
-func TestRootActivationRestoreReleasedTriggerOwnership(t *testing.T) {
+func TestRootActivationModelReleasedTriggerOwnership(t *testing.T) {
tx, _ := offlineFixture(t)
if err := tx.begin(t.Context(), false); err != nil {
t.Fatal(err)
@@ -392,57 +259,3 @@ func TestRootActivationRestoreReleasedTriggerOwnership(t *testing.T) {
t.Fatal("replacement changed")
}
}
-
-func TestRootRebootActivationRestorePhase1(t *testing.T) {
- tx := setupOfflineReal(t)
- if out, err := tx.systemctl(t.Context(), "stop", "carlos-adopt.timer"); err != nil {
- t.Fatal(string(out), err)
- }
- if err := tx.begin(t.Context(), false); err != nil {
- t.Fatal(err)
- }
- injected := errors.New("restore reboot window")
- tx.checkpoint = func(name string) error {
- if name == "journal-released" {
- return injected
- }
- return nil
- }
- if err := tx.restore(t.Context(), false); !errors.Is(err, injected) {
- t.Fatal(err)
- }
- boot, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
- if err != nil {
- t.Fatal(err)
- }
- if err := os.WriteFile(offlineRebootState+"/boot", boot, 0600); err != nil {
- t.Fatal(err)
- }
-}
-func TestRootRebootActivationRestorePhase2(t *testing.T) {
- tx := activationOffline{units: "/etc/systemd/system", state: offlineRebootState + "/transaction", systemctl: offlineRealSystemctl}
- defer cleanupOfflineReal(t, tx)
- before, err := os.ReadFile(offlineRebootState + "/boot")
- if err != nil {
- t.Fatal(err)
- }
- after, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
- if err != nil || string(before) == string(after) {
- t.Fatal("no actual reboot", err)
- }
- _, active, err := tx.status(t.Context(), "carlos-adopt.timer")
- if err != nil || active != "inactive" {
- t.Fatal("held timer ran on reboot", active, err)
- }
- if err := tx.restore(t.Context(), false); err != nil {
- t.Fatal(err)
- }
- _, active, err = tx.status(t.Context(), "carlos-adopt.timer")
- if err != nil || active != "inactive" {
- t.Fatal("inactive state lost", active, err)
- }
- _, active, err = tx.status(t.Context(), "carlos-edge.service")
- if err != nil || active != "active" {
- t.Fatal("edge not restored", active, err)
- }
-}
diff --git a/internal/host/activation_production_reboot_rootrunner_linux_test.go b/internal/host/activation_production_reboot_rootrunner_linux_test.go
new file mode 100644
index 000000000..ee913fc60
--- /dev/null
+++ b/internal/host/activation_production_reboot_rootrunner_linux_test.go
@@ -0,0 +1,191 @@
+//go:build linux && rootrunner
+
+package host
+
+import (
+ "bytes"
+ "context"
+ "errors"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+)
+
+func activationProductionRebootPhase1(t *testing.T, phase string) {
+ tx := setupActivationProduction(t)
+ if phase == "complete" {
+ servingRunnerSystemctl(t, "stop", "carlos-edge.service")
+ }
+ stop := errors.New("reboot checkpoint")
+ point := "journal-" + phase
+ tx.checkpoint = func(name string) error {
+ if name == point {
+ return stop
+ }
+ return nil
+ }
+ err := tx.begin(t.Context(), false)
+ if phase != "prepared" && phase != "held" {
+ if err != nil {
+ t.Fatal(err)
+ }
+ err = tx.restore(t.Context(), false)
+ }
+ if !errors.Is(err, stop) {
+ t.Fatal("checkpoint not reached", point, err)
+ }
+ r := offlineRecord(t, tx)
+ if r.Phase != phase {
+ t.Fatal("phase", r.Phase, phase)
+ }
+ if phase != "prepared" {
+ load, active, err := tx.status(t.Context(), activationEdgeUnit)
+ if err != nil || load != "masked" || active != "inactive" {
+ t.Fatal("edge escaped before release", load, active, err)
+ }
+ }
+ if phase == "done" {
+ if !r.EdgePending {
+ t.Fatal("missing ordinary edge continuation")
+ }
+ for _, item := range r.Items[:len(r.Items)-1] {
+ _, active, err := tx.status(t.Context(), item.Name)
+ want := "inactive"
+ if item.Active && !item.Masked {
+ want = "active"
+ }
+ if err != nil || active != want {
+ t.Fatal("closed before reconciliation", item.Name, active, want, err)
+ }
+ }
+ }
+ boot, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
+ if err != nil {
+ t.Fatal(err)
+ }
+ activationProductionWrite(t, offlineRebootState+"/boot", boot, 0600)
+ activationProductionWrite(t, offlineRebootState+"/phase", []byte(phase), 0600)
+ t.Log("rebooting production units at", point)
+}
+
+func activationProductionRebootPhase2(t *testing.T, phase string) {
+ tx := activationOffline{units: "/etc/systemd/system", state: "/etc/carlos/claim-activation/offline", systemctl: offlineRealSystemctl, drained: quiesceLegacyRootMutators}
+ defer cleanupActivationProduction(t, tx)
+ before, err := os.ReadFile(offlineRebootState + "/boot")
+ if err != nil {
+ t.Fatal(err)
+ }
+ after, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
+ if err != nil || bytes.Equal(before, after) {
+ t.Fatal("no actual reboot", err)
+ }
+ r := offlineRecord(t, tx)
+ if r.Phase != phase {
+ t.Fatal("durable phase changed", r.Phase, phase)
+ }
+ if phase != "prepared" {
+ load, active, err := tx.status(t.Context(), activationEdgeUnit)
+ if err != nil || load != "masked" || active != "inactive" {
+ t.Fatal("edge escaped on reboot", load, active, err)
+ }
+ }
+ if phase == "prepared" {
+ if err := tx.begin(t.Context(), false); err != nil {
+ t.Fatal("resume begin", err)
+ }
+ }
+ if phase == "complete" {
+ // Real services, started after the actual reboot with deliberately missing
+ // configuration. Every unmasked service must be reconciled, including failed.
+ if err := os.Rename("/etc/carlos/host.env", "/etc/carlos/host.env.saved"); err != nil {
+ t.Fatal(err)
+ }
+ for _, name := range rootMigrationOfflineUnits {
+ tx.systemctl(t.Context(), "start", name)
+ }
+ if err := os.Rename("/etc/carlos/host.env.saved", "/etc/carlos/host.env"); err != nil {
+ t.Fatal(err)
+ }
+ servingRunnerSystemctl(t, "start", "carlos-adopt.timer")
+ }
+ if phase == "done" {
+ // Closure releases ordinary boot behavior. The pending edge step must not
+ // restore old service files or roll an enabled timer back to its old state.
+ original := tx.systemctl
+ tx.systemctl = func(ctx context.Context, args ...string) ([]byte, error) {
+ if (args[0] == "start" || args[0] == "stop") && args[1] != activationEdgeUnit {
+ t.Fatalf("closed transaction changed %s", args[1])
+ }
+ return original(ctx, args...)
+ }
+ }
+ tx.checkpoint = func(point string) error {
+ if point == "replace-"+activationEdgeUnit || point == "start-"+activationEdgeUnit {
+ if offlineRecord(t, tx).Phase != "done" {
+ t.Fatal("edge released before done", point)
+ }
+ }
+ return nil
+ }
+ if err := tx.restore(t.Context(), false); err != nil {
+ t.Fatal("resume after actual reboot", phase, err)
+ }
+ if phase != "done" {
+ assertActivationProductionRecorded(t, tx, true)
+ }
+ if phase != "complete" {
+ assertActivationProductionEdge(t)
+ }
+ if got := offlineRecord(t, tx); got.Phase != "done" || got.EdgePending {
+ t.Fatal("unfinished", got.Phase, got.EdgePending)
+ }
+ for _, name := range []string{"carlos-edge.service", "carlos-system-recover.service"} {
+ raw, err := os.ReadFile(filepath.Join(tx.units, name+".d/90-host-floor.conf"))
+ if err != nil || !strings.Contains(string(raw), "guard-exec") {
+ t.Fatal("lost production guard", name, err)
+ }
+ }
+ t.Log("actual reboot and production restore passed", phase)
+ if phase == "restoring" {
+ cleanupActivationProduction(t, tx)
+ t.Run("seed-production-units", TestRootActivationSeedCLI)
+ }
+}
+
+func TestRootRebootActivationProductionPreparedPhase1(t *testing.T) {
+ activationProductionRebootPhase1(t, "prepared")
+}
+func TestRootRebootActivationProductionPreparedPhase2(t *testing.T) {
+ activationProductionRebootPhase2(t, "prepared")
+}
+func TestRootRebootActivationProductionHeldPhase1(t *testing.T) {
+ activationProductionRebootPhase1(t, "held")
+}
+func TestRootRebootActivationProductionHeldPhase2(t *testing.T) {
+ activationProductionRebootPhase2(t, "held")
+}
+func TestRootRebootActivationProductionRestoringPhase1(t *testing.T) {
+ activationProductionRebootPhase1(t, "restoring")
+}
+func TestRootRebootActivationProductionRestoringPhase2(t *testing.T) {
+ activationProductionRebootPhase2(t, "restoring")
+}
+func TestRootRebootActivationProductionReleasedPhase1(t *testing.T) {
+ activationProductionRebootPhase1(t, "released")
+}
+func TestRootRebootActivationProductionReleasedPhase2(t *testing.T) {
+ activationProductionRebootPhase2(t, "released")
+}
+func TestRootRebootActivationProductionCompletePhase1(t *testing.T) {
+ activationProductionRebootPhase1(t, "complete")
+}
+func TestRootRebootActivationProductionCompletePhase2(t *testing.T) {
+ activationProductionRebootPhase2(t, "complete")
+}
+func TestRootRebootActivationProductionDonePhase1(t *testing.T) {
+ activationProductionRebootPhase1(t, "done")
+}
+func TestRootRebootActivationProductionDonePhase2(t *testing.T) {
+ activationProductionRebootPhase2(t, "done")
+}
diff --git a/internal/host/activation_production_rootrunner_linux_test.go b/internal/host/activation_production_rootrunner_linux_test.go
new file mode 100644
index 000000000..d390c7680
--- /dev/null
+++ b/internal/host/activation_production_rootrunner_linux_test.go
@@ -0,0 +1,488 @@
+//go:build linux && rootrunner
+
+package host
+
+import (
+ "context"
+ "crypto/ed25519"
+ "encoding/base64"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+ "os/exec"
+ "os/user"
+ "path/filepath"
+ "strconv"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostactors"
+ "github.com/carlosframework/platform/internal/hostfloor"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func activationProductionWrite(t *testing.T, path string, raw []byte, mode os.FileMode) {
+ t.Helper()
+ if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(path, raw, mode); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func activationProductionNames() []string {
+ return append(activationOfflineUnits(), "data-instances.mount", "run-carlos-instances.mount", "run-carlos-logs.mount", "carlos.slice", "carlos-instance-control.service", "carlos-instance-control.socket")
+}
+
+func installActivationProductionUnits(t *testing.T, uid, gid int) activationOffline {
+ t.Helper()
+ if err := os.Mkdir(offlineRebootState, 0755); err != nil {
+ t.Fatal(err)
+ }
+ for _, path := range []string{"/data", "/run/carlos", "/run/carlos-control"} {
+ if _, err := os.Lstat(path); !os.IsNotExist(err) {
+ t.Fatal("pre-existing", path, err)
+ }
+ }
+ for _, name := range activationProductionNames() {
+ path := "/etc/systemd/system/" + name
+ if _, err := os.Lstat(path); !os.IsNotExist(err) {
+ t.Fatal("pre-existing", path, err)
+ }
+ raw, err := unitsFS.ReadFile("units/" + name)
+ if err != nil {
+ t.Fatal(err)
+ }
+ activationProductionWrite(t, path, raw, 0644)
+ if strings.HasSuffix(name, ".service") {
+ activationProductionWrite(t, path+".d/10-runner-network.conf", []byte("[Service]\nIPAddressDeny=any\nIPAddressAllow=127.0.0.1\n"), 0644)
+ }
+ }
+ for _, path := range []string{"/data", "/data/.instances", "/data/instances", "/data/layout", "/run/carlos", "/run/carlos/.instances", "/run/carlos/instances", "/run/carlos/.logs", "/run/carlos/logs"} {
+ if err := os.MkdirAll(path, 0755); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Chown(path, uid, gid); err != nil {
+ t.Fatal(err)
+ }
+ }
+ var tmp strings.Builder
+ for _, path := range []string{"/run/carlos", "/run/carlos/.instances", "/run/carlos/instances", "/run/carlos/.logs", "/run/carlos/logs"} {
+ fmt.Fprintf(&tmp, "d %s 0755 carlos carlos -\n", path)
+ }
+ fmt.Fprintln(&tmp, "d /run/carlos-control 0750 root carlos -")
+ activationProductionWrite(t, "/etc/tmpfiles.d/carlos-activation-runner.conf", []byte(tmp.String()), 0644)
+ return activationOffline{units: "/etc/systemd/system", state: "/etc/carlos/claim-activation/offline", systemctl: offlineRealSystemctl, drained: quiesceLegacyRootMutators}
+}
+
+func cleanupActivationProduction(t *testing.T, tx activationOffline) {
+ t.Helper()
+ if t.Failed() {
+ out, _ := exec.Command("journalctl", "-u", "carlos-edge.service", "-u", "carlos-system-recover.service", "-u", "data-instances.mount", "-u", "run-carlos-instances.mount", "-u", "run-carlos-logs.mount", "--no-pager", "-n", "80").CombinedOutput()
+ t.Log(string(out))
+ }
+ for _, name := range activationProductionNames() {
+ offlineRealSystemctl(context.Background(), "disable", "--now", name)
+ os.Remove("/etc/systemd/system/" + name)
+ os.RemoveAll("/etc/systemd/system/" + name + ".d")
+ }
+ offlineRealSystemctl(context.Background(), "daemon-reload")
+ offlineRealSystemctl(context.Background(), "reset-failed")
+ for _, path := range []string{"/data", "/run/carlos", "/run/carlos-control", "/etc/carlos", "/opt/carlos", "/etc/tmpfiles.d/carlos-activation-runner.conf", offlineRebootState} {
+ os.RemoveAll(path)
+ }
+ if _, err := os.Stat("/etc/carlos-activation-runner-user"); err == nil {
+ if out, err := exec.Command("userdel", serviceUser).CombinedOutput(); err != nil {
+ t.Errorf("remove fixture user: %s %v", out, err)
+ return
+ }
+ if _, err := user.LookupGroup(serviceUser); err == nil {
+ if out, err := exec.Command("groupdel", serviceUser).CombinedOutput(); err != nil {
+ t.Errorf("remove fixture group: %s %v", out, err)
+ return
+ }
+ }
+ os.Remove("/etc/carlos-activation-runner-user")
+ }
+
+}
+
+func assertActivationProductionEdge(t *testing.T) {
+ t.Helper()
+ ctx, cancel := context.WithTimeout(t.Context(), 20*time.Second)
+ defer cancel()
+ for {
+ out, err := exec.CommandContext(ctx, "curl", "-s", "--max-time", "1", "-o", "/dev/null", "http://127.0.0.1:18080").CombinedOutput()
+ if err == nil {
+ break
+ }
+ if ctx.Err() != nil {
+ t.Fatal("real edge did not listen", string(out), err)
+ }
+ time.Sleep(100 * time.Millisecond)
+ }
+ raw, err := offlineRealSystemctl(ctx, "show", "carlos-edge.service", "--property=Requires,ExecStart")
+ if err != nil || !strings.Contains(string(raw), "carlos-system-recover.service") || !strings.Contains(string(raw), "guard-exec edge") {
+ t.Fatal("production edge dependency missing", string(raw), err)
+ }
+ raw, err = offlineRealSystemctl(ctx, "show", "carlos-system-recover.service", "--property=ExecStart,Result")
+ if err != nil || !strings.Contains(string(raw), "guard-exec recover") || !strings.Contains(string(raw), "Result=success") {
+ t.Fatal("real guarded recovery did not succeed", string(raw), err)
+ }
+}
+
+func setupActivationProduction(t *testing.T) activationOffline {
+ t.Helper()
+ if len(activationCarlosImage) == 0 {
+ t.Fatal("requires embedded real CLI")
+ }
+ uid, gid := activationProductionUser(t)
+ for _, path := range []string{"/etc/carlos", "/opt/carlos", "/etc/systemd/system/carlos-edge.service.d", "/etc/systemd/system/carlos-instance-control.service", "/etc/systemd/system/carlos-system-update.service.d", "/etc/systemd/system/carlos-system-recover.service.d"} {
+ if _, err := os.Lstat(path); !os.IsNotExist(err) {
+ t.Fatal("existing fixture path", path, err)
+ }
+ }
+ tx := installActivationProductionUnits(t, uid, gid)
+ t.Cleanup(func() {
+ if t.Failed() {
+ cleanupActivationProduction(t, tx)
+ }
+ })
+ tx.state = "/etc/carlos/claim-activation/offline"
+ write := func(path string, b []byte, mode os.FileMode) {
+ t.Helper()
+ if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(path, b, mode); err != nil {
+ t.Fatal(err)
+ }
+ }
+ write(offlineRebootState+"/carlos-cli", activationCarlosImage, 0755)
+ write("/opt/carlos/carlos", activationCarlosImage, 0755)
+ write("/opt/carlos/carlos.prev", activationCarlosImage, 0755)
+ pub, key, err := ed25519.GenerateKey(nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ write(offlineRebootState+"/signing-key", key, 0600)
+ signer := release.RootSigner(key)
+ const bucket = "activation-seed-fixture"
+ store := release.NewDirStore(offlineRebootState + "/bucket")
+ rel := release.SystemRelease{Version: "v1.0.0", Source: "release:v1.0.0", ReleaseTag: "v1.0.0", MovesProtocol: 1, Artifacts: map[string]release.SystemArtifact{"linux-arm64": {Key: release.SystemArtifactKey("v1.0.0", "linux-arm64"), SHA256: hostfloor.Digest(activationCarlosImage)}}}
+ if _, err := release.PromoteSystem(t.Context(), store, signer, "router", "stable", rel, release.SystemPromoteOptions{}); err != nil {
+ t.Fatal(err)
+ }
+ identity, _, err := store.Get(t.Context(), release.SystemPointerKey("router", "stable"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ floor, _ := json.Marshal(hostfloor.Floor{Min: 1, MinRelease: "v1.0.0"})
+ auth, err := release.SignRootEnvelope(signer, hostfloor.Authorization{Schema: 1, Purpose: "moves-protocol", DeploymentBucket: bucket, Key: hostfloor.Key, SHA256: hostfloor.Digest(floor)})
+ if err != nil {
+ t.Fatal(err)
+ }
+ policy, _ := json.Marshal(hostfloor.Policy{Schema: 1, DeploymentBucket: bucket, Mode: "armed", Floor: floor, Authorization: auth})
+ write("/etc/carlos/host.env", []byte("CARLOS_HIBERNATE_BUCKET=activation-seed-fixture\nCARLOS_HIBERNATE_REGION=eu-west-1\nAWS_EC2_METADATA_DISABLED=true\nAWS_REGION=eu-west-1\nAWS_ACCESS_KEY_ID=fixture\nAWS_SECRET_ACCESS_KEY=fixture\nCARLOS_S3_ENDPOINT=http://127.0.0.1:1\nCARLOS_DEPLOYMENT_BUCKET="+bucket+"\nCARLOS_RELEASE_PUBKEY="+base64.StdEncoding.EncodeToString(pub)+"\n"), 0600)
+ if err := (hostactors.Directory{Store: store, PublicKey: pub, Bucket: bucket}).Initialize(t.Context(), signer); err != nil {
+ t.Fatal(err)
+ }
+ local := hostfloor.Enrollment{Box: "seed-fixture", Enrollment: strings.Repeat("a", 32), InstanceID: "i-fixture"}
+ invitation, err := release.SignRootEnvelope(signer, hostactors.Invitation{V: 1, Bucket: bucket, Reference: local.Box, Enrollment: local.Enrollment, EvidenceSHA256: strings.Repeat("b", 64)})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := store.Put(t.Context(), "control/host-actor-invitations/"+local.Box+".json", invitation, ""); err != nil {
+ t.Fatal(err)
+ }
+ localRaw, _ := json.Marshal(local)
+ write(offlineRebootState+"/enrollment.json", localRaw, 0600)
+ write(offlineRebootState+"/policy.json", policy, 0600)
+ write(offlineRebootState+"/identity.json", identity, 0600)
+ // The real guard-arm installs the dependency-bearing drop-in. Its signed
+ // roster invitation and loopback S3 endpoint are fixture authorization.
+ if err := os.Mkdir(offlineRebootState+"/data", 0755); err != nil {
+ t.Fatal(err)
+ }
+ endpoint := activationS3Fixture(t, store)
+ arm := exec.CommandContext(t.Context(), offlineRebootState+"/carlos-cli", "system", "guard-arm", "--policy", offlineRebootState+"/policy.json", "--identity", offlineRebootState+"/identity.json", "--enrollment", offlineRebootState+"/enrollment.json")
+ arm.Dir = offlineRebootState
+ arm.Env = []string{"PATH=/usr/sbin:/usr/bin:/sbin:/bin", "HOME=" + offlineRebootState, "CARLOS_DEPLOYMENT_BUCKET=" + bucket, "CARLOS_RELEASE_PUBKEY=" + base64.StdEncoding.EncodeToString(pub), "CARLOS_DATA_DIR=" + offlineRebootState + "/data", "AWS_REGION=eu-west-1", "AWS_ACCESS_KEY_ID=fixture", "AWS_SECRET_ACCESS_KEY=fixture", "AWS_EC2_METADATA_DISABLED=true", "CARLOS_S3_ENDPOINT=" + endpoint}
+ if out, err := arm.CombinedOutput(); err != nil {
+ t.Fatal("guard-arm", string(out), err)
+ }
+ guard, err := os.ReadFile(filepath.Join(tx.units, "carlos-edge.service.d/90-host-floor.conf"))
+ if err != nil || string(guard) != activationSeedGuardOriginal {
+ t.Fatal("unexpected production guard", err)
+ }
+ servingRunnerSystemctl(t, "daemon-reload")
+ servingRunnerSystemctl(t, "enable", "carlos-edge.service", "carlos-system-recover.service", "carlos-adopt.timer")
+ servingRunnerSystemctl(t, "start", "carlos-edge.service")
+ assertActivationProductionEdge(t)
+ return tx
+}
+
+func TestRootActivationProductionRestore(t *testing.T) {
+ tx := setupActivationProduction(t)
+ defer cleanupActivationProduction(t, tx)
+ if err := tx.begin(t.Context(), false); err != nil {
+ t.Fatal(err)
+ }
+ tx.checkpoint = func(point string) error {
+ if point == "replace-carlos-edge.service" || point == "start-carlos-edge.service" {
+ if phase := offlineRecord(t, tx).Phase; phase != "done" {
+ t.Fatalf("edge released before closed: %s at %s", phase, point)
+ }
+ }
+ return nil
+ }
+ if err := tx.restore(t.Context(), false); err != nil {
+ t.Fatal("guarded restore", err)
+ }
+ assertActivationProductionEdge(t)
+}
+
+func TestRootActivationProductionCompleteResume(t *testing.T) {
+ tx := setupActivationProduction(t)
+ defer cleanupActivationProduction(t, tx)
+ servingRunnerSystemctl(t, "stop", "carlos-edge.service")
+ if err := tx.begin(t.Context(), false); err != nil {
+ t.Fatal(err)
+ }
+ stop := errors.New("complete interruption")
+ tx.checkpoint = func(point string) error {
+ if point == "journal-complete" {
+ return stop
+ }
+ return nil
+ }
+ if err := tx.restore(t.Context(), false); !errors.Is(err, stop) {
+ t.Fatal(err)
+ }
+ tx.checkpoint = nil
+ // An actual failed recovery unit models the state left by the previous gate.
+ activationProductionWrite(t, "/etc/carlos/host.env.bad", []byte("bad"), 0600)
+ if err := os.Rename("/etc/carlos/host.env", "/etc/carlos/host.env.saved"); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := tx.systemctl(t.Context(), "start", "carlos-system-recover.service"); err == nil {
+ t.Fatal("failure injection did not fail")
+ }
+ if err := os.Rename("/etc/carlos/host.env.saved", "/etc/carlos/host.env"); err != nil {
+ t.Fatal(err)
+ }
+ if err := tx.restore(t.Context(), false); err != nil {
+ t.Fatal("resume", err)
+ }
+ assertActivationProductionRecorded(t, tx, true)
+}
+
+func assertActivationProductionRecorded(t *testing.T, tx activationOffline, files bool) {
+ t.Helper()
+ for _, item := range offlineRecord(t, tx).Items {
+ load, active, err := tx.status(t.Context(), item.Name)
+ wantLoad, wantActive := "loaded", "inactive"
+ if item.Masked {
+ wantLoad = "masked"
+ }
+ if item.Active && !item.Masked {
+ wantActive = "active"
+ }
+ if err != nil || load != wantLoad || active != wantActive {
+ t.Errorf("%s recorded %s/%s got %s/%s: %v", item.Name, wantLoad, wantActive, load, active, err)
+ }
+ if files {
+ if !sameActivationFile(filepath.Join(tx.units, item.Name), filepath.Join(tx.state, item.Name+".original")) {
+ t.Errorf("original inode changed: %s", item.Name)
+ }
+ if err := tx.checkItem(item); err != nil {
+ t.Errorf("original bytes or metadata changed: %s: %v", item.Name, err)
+ }
+ }
+ }
+}
+
+func TestRootActivationProductionEdgeFailureClosed(t *testing.T) {
+ tx := setupActivationProduction(t)
+ defer cleanupActivationProduction(t, tx)
+ if err := tx.begin(t.Context(), false); err != nil {
+ t.Fatal(err)
+ }
+ bad := filepath.Join(tx.units, "carlos-edge.service.d/91-runner-failure.conf")
+ activationProductionWrite(t, bad, []byte("[Service]\nExecStartPre=/usr/bin/false\nRestart=no\n"), 0644)
+ if err := tx.restore(t.Context(), false); err == nil {
+ t.Fatal("real failed edge start succeeded")
+ }
+ if r := offlineRecord(t, tx); r.Phase != "done" || !r.EdgePending {
+ t.Fatal("edge failure reopened transaction", r.Phase, r.EdgePending)
+ }
+ if err := activationAdoptionAllowed("/etc"); err != nil {
+ t.Fatal("failed ordinary edge held adoption", err)
+ }
+ if err := os.Remove(bad); err != nil {
+ t.Fatal(err)
+ }
+ servingRunnerSystemctl(t, "daemon-reload")
+ servingRunnerSystemctl(t, "reset-failed", "carlos-edge.service")
+ servingRunnerSystemctl(t, "start", "carlos-edge.service")
+ assertActivationProductionEdge(t)
+ if err := tx.restore(t.Context(), false); err != nil {
+ t.Fatal("ordinary edge retry", err)
+ }
+}
+
+func TestRootActivationProductionRecoveryGate(t *testing.T) {
+ tx := setupActivationProduction(t)
+ defer cleanupActivationProduction(t, tx)
+ if err := tx.begin(t.Context(), false); err != nil {
+ t.Fatal(err)
+ }
+ stop := errors.New("complete")
+ tx.checkpoint = func(point string) error {
+ if point == "journal-complete" {
+ return stop
+ }
+ return nil
+ }
+ if err := tx.restore(t.Context(), false); !errors.Is(err, stop) {
+ t.Fatal(err)
+ }
+ if err := activationAdoptionAllowed("/etc"); err == nil {
+ t.Fatal("complete allowed adoption")
+ }
+ if err := (&systemApplier{etcDir: "/etc"}).Run(t.Context(), false); !errors.Is(err, errActivation) {
+ t.Fatal("complete allowed update", err)
+ }
+ servingRunnerSystemctl(t, "start", "carlos-system-recover.service")
+ raw, err := tx.systemctl(t.Context(), "show", "carlos-system-recover.service", "--property=Result", "--value")
+ if err != nil || strings.TrimSpace(string(raw)) != "success" {
+ t.Fatal("complete blocked real recovery", string(raw), err)
+ }
+ tx.checkpoint = nil
+ if err := tx.restore(t.Context(), false); err != nil {
+ t.Fatal(err)
+ }
+ assertActivationProductionEdge(t)
+}
+
+func TestRootActivationProductionRestoreBoundaries(t *testing.T) {
+ tx := setupActivationProduction(t)
+ defer cleanupActivationProduction(t, tx)
+ for cut := 1; cut < 100; cut++ {
+ if err := tx.begin(t.Context(), false); err != nil {
+ t.Fatal(cut, err)
+ }
+ count := 0
+ point := ""
+ stop := errors.New("interruption")
+ tx.checkpoint = func(name string) error {
+ count++
+ r := offlineRecord(t, tx)
+ if r.Phase != "done" {
+ target, _ := os.Readlink(filepath.Join(tx.units, activationEdgeUnit))
+ if target != "/dev/null" {
+ t.Fatal("edge released before closure", name, r.Phase)
+ }
+ }
+ if count == cut {
+ point = name
+ return stop
+ }
+ return nil
+ }
+ err := tx.restore(t.Context(), false)
+ if err != nil && !errors.Is(err, stop) {
+ t.Fatal(cut, point, err)
+ }
+ tx.checkpoint = nil
+ if err := tx.restore(t.Context(), false); err != nil {
+ t.Fatal("resume", cut, point, err)
+ }
+ assertActivationProductionRecorded(t, tx, true)
+ assertActivationProductionEdge(t)
+ t.Log("production restore boundary passed", cut, point)
+ if err == nil {
+ return
+ }
+ if err := os.RemoveAll(tx.state); err != nil {
+ t.Fatal(err)
+ }
+ }
+ t.Fatal("boundary loop did not complete")
+}
+
+func activationProductionUser(t *testing.T) (int, int) {
+ t.Helper()
+ if os.Geteuid() != 0 {
+ t.Fatal("root required")
+ }
+ u, err := user.Lookup(serviceUser)
+ if _, unknown := err.(user.UnknownUserError); unknown {
+ if _, err := os.Lstat("/etc/carlos-activation-runner-user"); !os.IsNotExist(err) {
+ t.Fatal("existing fixture user marker", err)
+ }
+ if _, err := user.LookupGroup(serviceUser); err == nil {
+ t.Fatal("pre-existing group without user")
+ }
+ if out, err := exec.Command("useradd", "--system", "--user-group", "--no-create-home", "--shell", "/usr/sbin/nologin", serviceUser).CombinedOutput(); err != nil {
+ t.Fatal(string(out), err)
+ }
+ activationProductionWrite(t, "/etc/carlos-activation-runner-user", nil, 0600)
+ u, err = user.Lookup(serviceUser)
+ }
+ if err != nil {
+ t.Fatal(err)
+ }
+ uid, err := strconv.Atoi(u.Uid)
+ if err != nil {
+ t.Fatal(err)
+ }
+ gid, err := strconv.Atoi(u.Gid)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return uid, gid
+}
+
+func TestRootActivationProductionOperatorMask(t *testing.T) {
+ tx := setupActivationProduction(t)
+ defer cleanupActivationProduction(t, tx)
+ path := filepath.Join(tx.units, "carlos-stage.service")
+ if err := os.Rename(path, offlineRebootState+"/stage.original"); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Symlink("/dev/null", path); err != nil {
+ t.Fatal(err)
+ }
+ servingRunnerSystemctl(t, "daemon-reload")
+ if err := tx.begin(t.Context(), false); err != nil {
+ t.Fatal(err)
+ }
+ if err := verifyOfflineRootMigration(t.Context()); err != nil {
+ t.Fatal(err)
+ }
+ edge := filepath.Join(tx.units, activationEdgeUnit)
+ if err := os.Remove(edge); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Symlink("/dev/null", edge); err != nil {
+ t.Fatal(err)
+ }
+ if err := tx.restore(t.Context(), false); err == nil {
+ t.Fatal("unowned edge replacement accepted")
+ }
+ if err := os.Remove(edge); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Link(filepath.Join(tx.state, activationEdgeUnit+".mask"), edge); err != nil {
+ t.Fatal(err)
+ }
+ if err := tx.restore(t.Context(), false); err != nil {
+ t.Fatal(err)
+ }
+ assertActivationProductionRecorded(t, tx, true)
+ assertActivationProductionEdge(t)
+}
diff --git a/internal/host/activation_restore_order_rootrunner_linux_test.go b/internal/host/activation_restore_order_rootrunner_linux_test.go
index 6e8ea3c6f..3cf264aa0 100644
--- a/internal/host/activation_restore_order_rootrunner_linux_test.go
+++ b/internal/host/activation_restore_order_rootrunner_linux_test.go
@@ -26,7 +26,7 @@ func offlineRecord(t *testing.T, tx activationOffline) activationOfflineRecord {
return r
}
-func TestRootActivationRestoreEveryBoundaryBoot(t *testing.T) {
+func TestRootActivationModelRestoreEveryBoundaryBoot(t *testing.T) {
for cut := 1; cut < 160; cut++ {
reached := false
t.Run(fmt.Sprint(cut), func(t *testing.T) {
@@ -96,7 +96,7 @@ func TestRootActivationRestoreEveryBoundaryBoot(t *testing.T) {
}
}
-func TestRootActivationRestoreRealEnsureInterrupted(t *testing.T) {
+func TestRootActivationModelRepairExclusion(t *testing.T) {
tx, d := offlineFixture(t)
etc := filepath.Join(d, "etc")
units := filepath.Join(etc, "systemd", "system")
@@ -170,314 +170,6 @@ func TestRootActivationRestoreRealEnsureInterrupted(t *testing.T) {
}
}
-func offlineMatrixNames(t *testing.T, n int) {
- t.Helper()
- activationEdgeUnit = fmt.Sprintf("carlos-activation-%03d-edge.service", n)
- activationTriggers = []string{}
- for _, name := range []string{"adopt.timer", "adopt.path", "stage.path", "restart.path", "system-update.path"} {
- activationTriggers = append(activationTriggers, fmt.Sprintf("carlos-activation-%03d-%s", n, name))
- }
- rootMigrationOfflineUnits = []string{}
- for _, name := range []string{"adopt.service", "stage.service", "config-sync.service", "restart.service", "system-update.service", "system-recover.service"} {
- rootMigrationOfflineUnits = append(rootMigrationOfflineUnits, fmt.Sprintf("carlos-activation-%03d-%s", n, name))
- }
-}
-
-func offlineMatrixGlobals(t *testing.T) {
- t.Helper()
- edge, triggers, services := activationEdgeUnit, activationTriggers, rootMigrationOfflineUnits
- t.Cleanup(func() { activationEdgeUnit, activationTriggers, rootMigrationOfflineUnits = edge, triggers, services })
-}
-
-type offlineMatrixCut struct {
- N int
- Point, Phase string
-}
-
-func TestRootRebootActivationRestoreMatrixPhase1(t *testing.T) {
- offlineMatrixGlobals(t)
- if err := os.Mkdir(offlineRebootState, 0700); err != nil {
- t.Fatal(err)
- }
- boot, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
- if err != nil {
- t.Fatal(err)
- }
- if err := os.WriteFile(offlineRebootState+"/boot", boot, 0600); err != nil {
- t.Fatal(err)
- }
- cuts := []offlineMatrixCut{}
- for n := 1; n < 160; n++ {
- offlineMatrixNames(t, n)
- tx := activationOffline{units: "/etc/systemd/system", state: fmt.Sprintf("%s/case-%03d", offlineRebootState, n), systemctl: offlineRealSystemctl, drained: func(context.Context) error { return nil }}
- for _, name := range activationOfflineUnits() {
- path := filepath.Join(tx.units, name)
- if _, err := os.Lstat(path); !os.IsNotExist(err) {
- t.Fatal("pre-existing", path, err)
- }
- raw := "[Unit]\nDescription=activation reboot matrix\n[Service]\nExecStart=/usr/bin/sleep infinity\n[Install]\nWantedBy=multi-user.target\n"
- if strings.HasSuffix(name, ".timer") {
- raw = "[Unit]\nDescription=activation reboot timer\n[Timer]\nOnActiveSec=1h\n[Install]\nWantedBy=timers.target\n"
- }
- if strings.HasSuffix(name, ".path") {
- target := strings.TrimSuffix(name, ".path") + ".service"
- if strings.HasSuffix(name, "adopt.path") {
- target = strings.TrimSuffix(name, "adopt.path") + "config-sync.service"
- }
- raw = "[Unit]\nDescription=activation reboot path\n[Path]\nPathExists=" + offlineRebootState + "/never\nUnit=" + target + "\n[Install]\nWantedBy=multi-user.target\n"
- }
- if strings.HasSuffix(name, "-stage.service") {
- err = os.Symlink("/dev/null", path)
- } else {
- err = os.WriteFile(path, []byte(raw), 0644)
- }
- if err != nil {
- t.Fatal(err)
- }
- }
- for _, args := range [][]string{{"daemon-reload"}, {"enable", "--now", activationEdgeUnit}, {"enable", activationTriggers[0]}} {
- if out, err := tx.systemctl(t.Context(), args...); err != nil {
- t.Fatal(string(out), err)
- }
- }
- if err := tx.begin(t.Context(), false); err != nil {
- t.Fatal(n, err)
- }
- count := 0
- cut := offlineMatrixCut{N: n}
- stop := errors.New("reboot cut")
- tx.checkpoint = func(name string) error {
- count++
- r := offlineRecord(t, tx)
- if r.Phase != "complete" && r.Phase != "done" {
- if target, _ := os.Readlink(filepath.Join(tx.units, activationEdgeUnit)); target != "/dev/null" {
- t.Fatalf("%d: edge unmasked at %s before complete", n, name)
- }
- }
- if count == n {
- cut.Point = name
- return stop
- }
- return nil
- }
- err = tx.restore(t.Context(), false)
- if err != nil && !errors.Is(err, stop) {
- t.Fatal(n, err)
- }
- cut.Phase = offlineRecord(t, tx).Phase
- cuts = append(cuts, cut)
- t.Logf("actual reboot fixture %d: %s phase=%s", n, cut.Point, cut.Phase)
- if err == nil {
- break
- }
- }
- raw, err := json.Marshal(cuts)
- if err != nil {
- t.Fatal(err)
- }
- if err = os.WriteFile(offlineRebootState+"/cuts.json", raw, 0600); err != nil {
- t.Fatal(err)
- }
-}
-
-func TestRootRebootActivationRestoreMatrixPhase2(t *testing.T) {
- offlineMatrixGlobals(t)
- before, err := os.ReadFile(offlineRebootState + "/boot")
- if err != nil {
- t.Fatal(err)
- }
- after, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
- if err != nil || string(before) == string(after) {
- t.Fatal("no actual reboot", err)
- }
- raw, err := os.ReadFile(offlineRebootState + "/cuts.json")
- if err != nil {
- t.Fatal(err)
- }
- var cuts []offlineMatrixCut
- if err = json.Unmarshal(raw, &cuts); err != nil {
- t.Fatal(err)
- }
- sawEdge := false
- for _, cut := range cuts {
- t.Run(fmt.Sprintf("%d-%s", cut.N, cut.Point), func(t *testing.T) {
- offlineMatrixNames(t, cut.N)
- tx := activationOffline{units: "/etc/systemd/system", state: fmt.Sprintf("%s/case-%03d", offlineRebootState, cut.N), systemctl: offlineRealSystemctl}
- defer func() {
- for _, name := range activationOfflineUnits() {
- tx.systemctl(t.Context(), "disable", "--now", name)
- os.Remove(filepath.Join(tx.units, name))
- }
- tx.systemctl(t.Context(), "daemon-reload")
- os.RemoveAll(tx.state)
- }()
- load, active, err := tx.status(t.Context(), activationEdgeUnit)
- if err != nil {
- t.Fatal(err)
- }
- if cut.Phase != "complete" && cut.Phase != "done" && (load != "masked" || active != "inactive") {
- t.Fatalf("edge served before complete after %s: %s/%s", cut.Point, load, active)
- }
- if cut.Point == "replace-"+activationEdgeUnit {
- sawEdge = true
- if cut.Phase != "complete" {
- t.Fatal("edge replacement preceded durable completion")
- }
- if load != "loaded" || active != "active" {
- t.Fatalf("completed edge not booted: %s/%s", load, active)
- }
- }
- if err := tx.restore(t.Context(), false); err != nil {
- t.Fatal("resume", err)
- }
- _, active, err = tx.status(t.Context(), activationEdgeUnit)
- if err != nil || active != "active" {
- t.Fatal("edge not restored", active, err)
- }
- if cut.Phase != "done" {
- _, active, err = tx.status(t.Context(), activationTriggers[0])
- if err != nil || active != "inactive" {
- t.Fatal("enabled but inactive timer not reconciled", active, err)
- }
- }
- t.Logf("actual reboot and restore passed: %s", cut.Point)
- })
- }
- if !sawEdge {
- t.Fatal("missing immediate edge-replacement reboot")
- }
- os.RemoveAll(offlineRebootState)
-}
-
-func TestRootActivationRestoreEnsurePayload(t *testing.T) {
- if os.Getenv("CARLOS_ACTIVATION_ENSURE_PAYLOAD") != "1" {
- t.Fatal("invoke only through the adoption fixture")
- }
- if _, err := os.Stat(offlineRebootState + "/fire"); os.IsNotExist(err) {
- return
- }
- ensureSystemUnits("/etc", offlineRebootState+"/apply.lock", productionSysOps().systemctl, systemctlProbe, t.Logf)
- boot, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
- if err != nil {
- t.Fatal(err)
- }
- if err := os.WriteFile(offlineRebootState+"/fired", boot, 0600); err != nil {
- t.Fatal(err)
- }
-}
-
-func TestRootActivationRestoreSystemdAdoption(t *testing.T) {
- tx := setupOfflineReal(t)
- defer cleanupOfflineReal(t, tx)
- tx.state = "/etc/carlos/claim-activation/offline"
- if _, err := os.Lstat("/etc/carlos/claim-activation"); !os.IsNotExist(err) {
- t.Fatal("pre-existing activation state", err)
- }
- defer os.RemoveAll("/etc/carlos/claim-activation")
- for _, name := range []string{instanceControlSocket, claimControlUnit} {
- if _, err := os.Lstat(filepath.Join(tx.units, name)); !os.IsNotExist(err) {
- t.Fatal("pre-existing control unit", name, err)
- }
- }
- if _, err := os.Lstat("/run/carlos-control"); !os.IsNotExist(err) {
- t.Fatal("pre-existing control socket directory", err)
- }
- generationRunnerRegistryUser(t)
- defer func() {
- for _, name := range []string{instanceControlSocket, claimControlUnit} {
- tx.systemctl(t.Context(), "disable", "--now", name)
- os.Remove(filepath.Join(tx.units, name))
- }
- tx.systemctl(t.Context(), "daemon-reload")
- os.RemoveAll("/run/carlos-control")
- }()
- bin, err := os.Executable()
- if err != nil {
- t.Fatal(err)
- }
- raw := "[Unit]\nDescription=real activation adoption repair\n[Service]\nType=oneshot\nEnvironment=CARLOS_ACTIVATION_ENSURE_PAYLOAD=1\nExecStart=" + bin + " -test.run=^TestRootActivationRestoreEnsurePayload$ -test.v\nRestrictAddressFamilies=AF_UNIX\nIPAddressDeny=any\n"
- if err = os.WriteFile(filepath.Join(tx.units, "carlos-adopt.service"), []byte(raw), 0644); err != nil {
- t.Fatal(err)
- }
- timer := "[Timer]\nOnBootSec=1ms\nOnUnitActiveSec=1h\n[Install]\nWantedBy=timers.target\n"
- if err = os.WriteFile(filepath.Join(tx.units, "carlos-adopt.timer"), []byte(timer), 0644); err != nil {
- t.Fatal(err)
- }
- for _, args := range [][]string{{"daemon-reload"}, {"restart", "carlos-adopt.timer"}, {"start", "carlos-adopt.service"}} {
- if out, err := tx.systemctl(t.Context(), args...); err != nil {
- t.Fatal(string(out), err)
- }
- }
- if err := tx.begin(t.Context(), false); err != nil {
- t.Fatal(err)
- }
- if err := os.WriteFile(offlineRebootState+"/fire", nil, 0600); err != nil {
- t.Fatal(err)
- }
- stop := errors.New("interrupted after adoption activation")
- tx.checkpoint = func(name string) error {
- if name == "start-carlos-adopt.timer" {
- return stop
- }
- return nil
- }
- if err := tx.restore(t.Context(), false); !errors.Is(err, stop) {
- t.Fatal(err)
- }
- if err := runAdoptUnderRootRelease(nil, nil, nil, nil); !errors.Is(err, errActivation) {
- t.Fatal("root adoption did not honor transaction", err)
- }
- for _, recover := range []bool{false, true} {
- if err := (&systemApplier{etcDir: "/etc"}).Run(t.Context(), recover); !errors.Is(err, errActivation) {
- t.Fatal("system apply did not honor transaction", recover, err)
- }
- }
- path := filepath.Join(tx.units, "carlos-system-update.path")
- before, err := os.Lstat(path)
- if err != nil {
- t.Fatal(err)
- }
- if out, err := tx.systemctl(t.Context(), "start", "carlos-adopt.service"); err != nil {
- t.Fatal(string(out), err)
- }
- if _, err := os.Stat(offlineRebootState + "/fired"); err != nil {
- t.Fatal("real ensure did not run", err)
- }
- after, err := os.Lstat(path)
- if err != nil {
- t.Fatal(err)
- }
- if !os.SameFile(before, after) {
- t.Fatal("adoption replaced the held path")
- }
- load, active, err := tx.status(t.Context(), "carlos-edge.service")
- if err != nil || load != "masked" || active != "inactive" {
- t.Fatal("edge escaped", load, active, err)
- }
- tx.checkpoint = nil
- if err := tx.restore(t.Context(), false); err != nil {
- t.Fatal(err)
- }
- _, active, err = tx.status(t.Context(), "carlos-system-update.path")
- if err != nil || active != "inactive" {
- t.Fatal("inactive path was not restored", active, err)
- }
- if out, err := tx.systemctl(t.Context(), "start", "carlos-adopt.service"); err != nil {
- t.Fatal(string(out), err)
- }
- after, err = os.Lstat(path)
- if err != nil {
- t.Fatal(err)
- }
- if os.SameFile(before, after) {
- t.Fatal("ordinary adoption still blocked after done")
- }
- _, active, err = tx.status(t.Context(), "carlos-system-update.path")
- if err != nil || active != "active" {
- t.Fatal("ordinary adoption did not arm path", active, err)
- }
-}
-
func TestRootActivationAdoptionGate(t *testing.T) {
d := activationRootFixture(t)
etc := filepath.Join(d, "etc")
@@ -512,6 +204,10 @@ func TestRootActivationAdoptionGate(t *testing.T) {
if (err == nil) != (phase == "done") {
t.Fatal("gate phase", phase, err)
}
+ err = activationTransactionAllowed(etc, true)
+ if (err == nil) != (phase == "complete" || phase == "done") {
+ t.Fatal("recovery gate phase", phase, err)
+ }
}
if err := os.Chmod(tx.state, 0777); err != nil {
t.Fatal(err)
@@ -544,91 +240,3 @@ func TestRootActivationAdoptionGate(t *testing.T) {
t.Fatal("invalid journal permitted adoption")
}
}
-
-func TestRootRebootActivationRestoreAdoptionPhase1(t *testing.T) {
- tx := setupOfflineReal(t)
- tx.state = "/etc/carlos/claim-activation/offline"
- if _, err := os.Lstat("/etc/carlos/claim-activation"); !os.IsNotExist(err) {
- t.Fatal("pre-existing activation state", err)
- }
- bin, err := os.Executable()
- if err != nil {
- t.Fatal(err)
- }
- unit := "[Unit]\nDescription=activation adoption reboot\n[Service]\nType=oneshot\nEnvironment=CARLOS_ACTIVATION_ENSURE_PAYLOAD=1\nExecStart=" + bin + " -test.run=^TestRootActivationRestoreEnsurePayload$ -test.v\nRestrictAddressFamilies=AF_UNIX\nIPAddressDeny=any\n"
- if err := os.WriteFile(filepath.Join(tx.units, "carlos-adopt.service"), []byte(unit), 0644); err != nil {
- t.Fatal(err)
- }
- timer := "[Timer]\nOnBootSec=1ms\nOnUnitActiveSec=1h\n[Install]\nWantedBy=timers.target\n"
- if err := os.WriteFile(filepath.Join(tx.units, "carlos-adopt.timer"), []byte(timer), 0644); err != nil {
- t.Fatal(err)
- }
- for _, args := range [][]string{{"daemon-reload"}, {"restart", "carlos-adopt.timer"}, {"start", "carlos-adopt.service"}} {
- if out, err := tx.systemctl(t.Context(), args...); err != nil {
- t.Fatal(string(out), err)
- }
- }
- if err := tx.begin(t.Context(), false); err != nil {
- t.Fatal(err)
- }
- if err := os.WriteFile(offlineRebootState+"/fire", nil, 0600); err != nil {
- t.Fatal(err)
- }
- stop := errors.New("reboot during adoption restoration")
- tx.checkpoint = func(name string) error {
- if name == "start-carlos-adopt.timer" {
- return stop
- }
- return nil
- }
- if err := tx.restore(t.Context(), false); !errors.Is(err, stop) {
- t.Fatal(err)
- }
- boot, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
- if err != nil {
- t.Fatal(err)
- }
- if err := os.WriteFile(offlineRebootState+"/boot", boot, 0600); err != nil {
- t.Fatal(err)
- }
-}
-
-func TestRootRebootActivationRestoreAdoptionPhase2(t *testing.T) {
- tx := activationOffline{units: "/etc/systemd/system", state: "/etc/carlos/claim-activation/offline", systemctl: offlineRealSystemctl}
- defer cleanupOfflineReal(t, tx)
- defer os.RemoveAll("/etc/carlos/claim-activation")
- old, err := os.ReadFile(offlineRebootState + "/boot")
- if err != nil {
- t.Fatal(err)
- }
- boot, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
- if err != nil || string(old) == string(boot) {
- t.Fatal("no actual reboot", err)
- }
- fired, err := os.ReadFile(offlineRebootState + "/fired")
- if err != nil || string(fired) != string(boot) {
- t.Fatal("overdue boot timer did not invoke real ensureSystemUnits this boot", string(fired), err)
- }
- path := filepath.Join(tx.units, "carlos-system-update.path")
- if !sameActivationFile(path, filepath.Join(tx.state, "carlos-system-update.path.original")) {
- t.Fatal("boot adoption replaced held path")
- }
- if _, err := os.Stat(offlineRebootState + "/apply.lock"); !os.IsNotExist(err) {
- t.Fatal("adoption crossed persistent gate", err)
- }
- load, active, err := tx.status(t.Context(), "carlos-edge.service")
- if err != nil || load != "masked" || active != "inactive" {
- t.Fatal("edge escaped on reboot", load, active, err)
- }
- if err := tx.restore(t.Context(), false); err != nil {
- t.Fatal("resume", err)
- }
- _, active, err = tx.status(t.Context(), "carlos-system-update.path")
- if err != nil || active != "inactive" {
- t.Fatal("recorded path state lost", active, err)
- }
- _, active, err = tx.status(t.Context(), "carlos-edge.service")
- if err != nil || active != "active" {
- t.Fatal("edge not restored", active, err)
- }
-}
diff --git a/internal/host/activation_seed_cli_rootrunner_linux_test.go b/internal/host/activation_seed_cli_rootrunner_linux_test.go
index 0a0fb6855..2bc6c2bd5 100644
--- a/internal/host/activation_seed_cli_rootrunner_linux_test.go
+++ b/internal/host/activation_seed_cli_rootrunner_linux_test.go
@@ -4,10 +4,7 @@ package host
import (
"bytes"
- "context"
"crypto/ed25519"
- "encoding/base64"
- "encoding/json"
"os"
"os/exec"
"path/filepath"
@@ -16,7 +13,6 @@ import (
"testing"
"time"
- "github.com/carlosframework/platform/internal/hostactors"
"github.com/carlosframework/platform/internal/hostfloor"
"github.com/carlosframework/platform/internal/release"
)
@@ -26,45 +22,32 @@ import (
// never contacts a deployment. Files-seed data effects have separate witnesses.
const activationSeedCLIPayload = `#!/bin/sh
set -eu
-test "$*" = 'system files seed'
-echo started > /etc/carlos-activation-rootrunner/payload-started
-while test -f /etc/carlos-activation-rootrunner/payload-wait; do sleep 0.1; done
+if test "$*" != 'system files seed'; then
+ exec /etc/carlos-activation-rootrunner/carlos-cli "$@"
+fi
+test ! -f /data/activation-payload-fail
+echo started > /data/activation-payload-started
+while test -f /data/activation-payload-wait; do sleep 0.1; done
exit 0
`
func setupActivationSeedCLI(t *testing.T) activationOffline {
t.Helper()
- if len(activationCarlosImage) == 0 {
- t.Fatal("requires embedded real CLI")
- }
- generationRunnerRegistryUser(t)
- for _, path := range []string{"/etc/carlos", "/opt/carlos", "/etc/systemd/system/carlos-edge.service.d", "/etc/systemd/system/carlos-instance-control.service", "/etc/systemd/system/carlos-system-update.service.d", "/etc/systemd/system/carlos-system-recover.service.d"} {
- if _, err := os.Lstat(path); !os.IsNotExist(err) {
- t.Fatal("existing fixture path", path, err)
- }
- }
- tx := setupOfflineReal(t)
- tx.state = "/etc/carlos/claim-activation/offline"
- write := func(path string, b []byte, mode os.FileMode) {
- t.Helper()
- if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil {
- t.Fatal(err)
- }
- if err := os.WriteFile(path, b, mode); err != nil {
- t.Fatal(err)
- }
+ tx := setupActivationProduction(t)
+ servingRunnerSystemctl(t, "stop", "carlos-edge.service")
+ servingRunnerSystemctl(t, "start", "carlos-instance-control.service")
+ raw, err := os.ReadFile("/etc/carlos/host.env")
+ if err != nil {
+ t.Fatal(err)
}
- write(offlineRebootState+"/carlos-cli", activationCarlosImage, 0755)
- write("/opt/carlos/carlos", activationCarlosImage, 0755)
- write("/opt/carlos/carlos.prev", activationCarlosImage, 0755)
- pub, key, err := ed25519.GenerateKey(nil)
+ activationProductionWrite(t, "/etc/carlos/host.env", append(raw, []byte("CARLOS_HOST_CLAIM=1\nCARLOS_HOST_CLAIM_ORIGINS=1\n")...), 0600)
+ key, err := os.ReadFile(offlineRebootState + "/signing-key")
if err != nil {
t.Fatal(err)
}
- signer := release.RootSigner(key)
- const bucket = "activation-seed-fixture"
+ signer := release.RootSigner(ed25519.PrivateKey(key))
store := release.NewDirStore(offlineRebootState + "/bucket")
- rel := release.SystemRelease{Version: "v1.0.0", Source: "release:v1.0.0", ReleaseTag: "v1.0.0", MovesProtocol: 1, Artifacts: map[string]release.SystemArtifact{"linux-arm64": {Key: release.SystemArtifactKey("v1.0.0", "linux-arm64"), SHA256: hostfloor.Digest(activationCarlosImage)}}}
+ rel := release.SystemRelease{Version: "v1.0.1", Source: "release:v1.0.1", ReleaseTag: "v1.0.1", MovesProtocol: 1, Artifacts: map[string]release.SystemArtifact{"linux-arm64": {Key: release.SystemArtifactKey("v1.0.1", "linux-arm64"), SHA256: hostfloor.Digest([]byte(activationSeedCLIPayload))}}}
if _, err := release.PromoteSystem(t.Context(), store, signer, "router", "stable", rel, release.SystemPromoteOptions{}); err != nil {
t.Fatal(err)
}
@@ -72,63 +55,22 @@ func setupActivationSeedCLI(t *testing.T) activationOffline {
if err != nil {
t.Fatal(err)
}
- floor, _ := json.Marshal(hostfloor.Floor{Min: 1, MinRelease: "v1.0.0"})
- auth, err := release.SignRootEnvelope(signer, hostfloor.Authorization{Schema: 1, Purpose: "moves-protocol", DeploymentBucket: bucket, Key: hostfloor.Key, SHA256: hostfloor.Digest(floor)})
- if err != nil {
- t.Fatal(err)
- }
- policy, _ := json.Marshal(hostfloor.Policy{Schema: 1, DeploymentBucket: bucket, Mode: "armed", Floor: floor, Authorization: auth})
- write("/etc/carlos/host.env", []byte("CARLOS_HOST_CLAIM=1\nCARLOS_HOST_CLAIM_ORIGINS=1\nCARLOS_DEPLOYMENT_BUCKET="+bucket+"\nCARLOS_RELEASE_PUBKEY="+base64.StdEncoding.EncodeToString(pub)+"\n"), 0600)
- if err := (hostactors.Directory{Store: store, PublicKey: pub, Bucket: bucket}).Initialize(t.Context(), signer); err != nil {
- t.Fatal(err)
- }
- local := hostfloor.Enrollment{Box: "seed-fixture", Enrollment: strings.Repeat("a", 32), InstanceID: "i-fixture"}
- invitation, err := release.SignRootEnvelope(signer, hostactors.Invitation{V: 1, Bucket: bucket, Reference: local.Box, Enrollment: local.Enrollment, EvidenceSHA256: strings.Repeat("b", 64)})
- if err != nil {
- t.Fatal(err)
- }
- if _, err := store.Put(t.Context(), "control/host-actor-invitations/"+local.Box+".json", invitation, ""); err != nil {
- t.Fatal(err)
- }
- localRaw, _ := json.Marshal(local)
- write(offlineRebootState+"/enrollment.json", localRaw, 0600)
- write(offlineRebootState+"/policy.json", policy, 0600)
- write(offlineRebootState+"/identity.json", identity, 0600)
- // The real guard-arm installs the dependency-bearing drop-in. Its signed
- // roster invitation and loopback S3 endpoint are fixture authorization.
- if err := os.Mkdir(offlineRebootState+"/data", 0755); err != nil {
- t.Fatal(err)
- }
- endpoint := activationS3Fixture(t, store)
- arm := exec.CommandContext(t.Context(), offlineRebootState+"/carlos-cli", "system", "guard-arm", "--policy", offlineRebootState+"/policy.json", "--identity", offlineRebootState+"/identity.json", "--enrollment", offlineRebootState+"/enrollment.json")
- arm.Dir = offlineRebootState
- arm.Env = []string{"PATH=/usr/sbin:/usr/bin:/sbin:/bin", "HOME=" + offlineRebootState, "CARLOS_DEPLOYMENT_BUCKET=" + bucket, "CARLOS_RELEASE_PUBKEY=" + base64.StdEncoding.EncodeToString(pub), "CARLOS_DATA_DIR=" + offlineRebootState + "/data", "AWS_REGION=eu-west-1", "AWS_ACCESS_KEY_ID=fixture", "AWS_SECRET_ACCESS_KEY=fixture", "AWS_EC2_METADATA_DISABLED=true", "CARLOS_S3_ENDPOINT=" + endpoint}
- if out, err := arm.CombinedOutput(); err != nil {
- t.Fatal("guard-arm", string(out), err)
- }
- guard, err := os.ReadFile(filepath.Join(tx.units, "carlos-edge.service.d/90-host-floor.conf"))
- if err != nil || string(guard) != activationSeedGuardOriginal {
- t.Fatal("unexpected production guard", err)
- }
- // Authorize a harmless payload separately; the operator and installed guard
- // remain real binaries. No production signing key or deployment is used.
- rel.Version, rel.Source, rel.ReleaseTag = "v1.0.1", "release:v1.0.1", "v1.0.1"
- rel.Artifacts["linux-arm64"] = release.SystemArtifact{Key: release.SystemArtifactKey("v1.0.1", "linux-arm64"), SHA256: hostfloor.Digest([]byte(activationSeedCLIPayload))}
- if _, err := release.PromoteSystem(t.Context(), store, signer, "router", "stable", rel, release.SystemPromoteOptions{}); err != nil {
- t.Fatal(err)
- }
- payloadIdentity, _, err := store.Get(t.Context(), release.SystemPointerKey("router", "stable"))
+ activationProductionWrite(t, "/opt/carlos/system/guard/identities/v1.0.1.json", identity, 0644)
+ activationProductionWrite(t, "/opt/carlos/carlos", []byte(activationSeedCLIPayload), 0755)
+ activationSeedCLI(t, "offline-begin")
+ return tx
+}
+
+func activationSeedPrepareRestore(t *testing.T) {
+ t.Helper()
+ activationProductionWrite(t, "/opt/carlos/carlos", activationCarlosImage, 0755)
+ raw, err := os.ReadFile("/etc/carlos/host.env")
if err != nil {
t.Fatal(err)
}
- write("/opt/carlos/system/guard/identities/v1.0.1.json", payloadIdentity, 0644)
- write("/opt/carlos/carlos", []byte(activationSeedCLIPayload), 0755)
- write(filepath.Join(tx.units, claimControlUnit), []byte("[Service]\nExecStart=/usr/bin/sleep infinity\n[Install]\nWantedBy=multi-user.target\n"), 0644)
- servingRunnerSystemctl(t, "stop", "carlos-edge.service")
- servingRunnerSystemctl(t, "daemon-reload")
- servingRunnerSystemctl(t, "enable", "--now", claimControlUnit)
- activationSeedCLI(t, "offline-begin")
- return tx
+ raw = bytes.ReplaceAll(raw, []byte("CARLOS_HOST_CLAIM=1\nCARLOS_HOST_CLAIM_ORIGINS=1\n"), nil)
+ activationProductionWrite(t, "/etc/carlos/host.env", raw, 0600)
+ servingRunnerSystemctl(t, "stop", "carlos-instance-control.service")
}
func activationSeedCLICommand(t *testing.T, args ...string) *exec.Cmd {
t.Helper()
@@ -145,13 +87,7 @@ func activationSeedCLI(t *testing.T, args ...string) {
}
}
func cleanupActivationSeedCLI(t *testing.T, tx activationOffline) {
- t.Helper()
- offlineRealSystemctl(context.Background(), "disable", "--now", claimControlUnit)
- cleanupOfflineReal(t, tx)
- for _, path := range []string{"/etc/carlos", "/opt/carlos", filepath.Join(tx.units, "carlos-edge.service.d"), filepath.Join(tx.units, claimControlUnit), filepath.Join(tx.units, "carlos-system-update.service.d"), filepath.Join(tx.units, "carlos-system-recover.service.d")} {
- os.RemoveAll(path)
- }
- offlineRealSystemctl(context.Background(), "daemon-reload")
+ cleanupActivationProduction(t, tx)
}
func assertActivationSeedCLIHeld(t *testing.T, tx activationOffline) {
t.Helper()
@@ -179,24 +115,26 @@ func assertActivationSeedCLIHeld(t *testing.T, tx activationOffline) {
}
func startActivationSeedCLIBlocked(t *testing.T) *exec.Cmd {
t.Helper()
- if err := os.WriteFile(offlineRebootState+"/payload-wait", nil, 0600); err != nil {
+ if err := os.WriteFile("/data/activation-payload-wait", nil, 0600); err != nil {
t.Fatal(err)
}
- os.Remove(offlineRebootState + "/payload-started")
+ os.Remove("/data/activation-payload-started")
cmd := activationSeedCLICommand(t, "offline-seed")
+ var output bytes.Buffer
+ cmd.Stdout, cmd.Stderr = &output, &output
if err := cmd.Start(); err != nil {
t.Fatal(err)
}
deadline := time.Now().Add(20 * time.Second)
for time.Now().Before(deadline) {
- if _, err := os.Stat(offlineRebootState + "/payload-started"); err == nil {
+ if _, err := os.Stat("/data/activation-payload-started"); err == nil {
return cmd
}
time.Sleep(50 * time.Millisecond)
}
cmd.Process.Kill()
cmd.Wait()
- t.Fatal("seed payload did not start")
+ t.Fatal("seed payload did not start", output.String())
return nil
}
func TestRootActivationSeedCLI(t *testing.T) {
@@ -210,6 +148,7 @@ func TestRootActivationSeedCLI(t *testing.T) {
t.Fatal("missing successful seed", string(result), err)
}
interruptActivationSeedCLI(t, tx)
+ activationSeedPrepareRestore(t)
activationSeedCLI(t, "offline-restore")
}
func interruptActivationSeedCLI(t *testing.T, tx activationOffline) {
@@ -225,11 +164,38 @@ func interruptActivationSeedCLI(t *testing.T, tx activationOffline) {
if err != nil || load != "masked" || active != "inactive" && active != "failed" {
t.Fatal("cancel did not remask", load, active, err)
}
- os.Remove(offlineRebootState + "/payload-wait")
+ os.Remove("/data/activation-payload-wait")
activationSeedCLI(t, "offline-seed")
assertActivationSeedCLIHeld(t, tx)
}
func TestRootRebootActivationSeedCLIPhase1(t *testing.T) {
+ for _, test := range []struct {
+ name string
+ run func(*testing.T)
+ }{
+ {"guarded-restore", TestRootActivationProductionRestore},
+ {"complete-resume", TestRootActivationProductionCompleteResume},
+ {"recovery-gate", TestRootActivationProductionRecoveryGate},
+ {"ordinary-edge-failure", TestRootActivationProductionEdgeFailureClosed},
+ {"operator-mask", TestRootActivationProductionOperatorMask},
+ {"restore-boundaries", TestRootActivationProductionRestoreBoundaries},
+ {"seed-cli", TestRootActivationSeedCLI},
+ {"seed-boundaries", TestRootActivationSeedInterruptions},
+ {"seed-refusals", TestRootActivationSeedProductionFailures},
+ {"gate-shapes", TestRootActivationAdoptionGate},
+ {"model-offline", TestRootActivationModelOfflineInterruptions},
+ {"model-ownership", TestRootActivationModelReplacementRefused},
+ {"model-edge-order", TestRootActivationModelEdgeLast},
+ {"model-trigger-ownership", TestRootActivationModelReleasedTriggerOwnership},
+ {"model-boot", TestRootActivationModelRestoreEveryBoundaryBoot},
+ {"model-repair", TestRootActivationModelRepairExclusion},
+ } {
+ t.Run(test.name, test.run)
+ }
+ if t.Failed() {
+ return
+ }
+
tx := setupActivationSeedCLI(t)
cmd := startActivationSeedCLIBlocked(t)
if err := cmd.Process.Kill(); err != nil {
@@ -289,9 +255,10 @@ func TestRootRebootActivationSeedCLIPhase2(t *testing.T) {
t.Fatal("seed not running after reboot", pid, err)
}
t.Log("reboot started a new seed invocation", strings.TrimSpace(string(current)), "pid", pid)
- os.Remove(offlineRebootState + "/payload-wait")
+ os.Remove("/data/activation-payload-wait")
activationSeedCLI(t, "offline-seed")
assertActivationSeedCLIHeld(t, tx)
interruptActivationSeedCLI(t, tx)
+ activationSeedPrepareRestore(t)
activationSeedCLI(t, "offline-restore")
}
diff --git a/internal/host/activation_seed_rootrunner_linux_test.go b/internal/host/activation_seed_rootrunner_linux_test.go
index 3ea6beeb4..7945354d3 100644
--- a/internal/host/activation_seed_rootrunner_linux_test.go
+++ b/internal/host/activation_seed_rootrunner_linux_test.go
@@ -3,352 +3,86 @@
package host
import (
- "bytes"
- "context"
"errors"
"os"
"path/filepath"
- "strings"
"testing"
)
-func TestRootActivationSeedTransaction(t *testing.T) {
- tx := setupOfflineReal(t)
- defer cleanupOfflineReal(t, tx)
- if err := tx.begin(t.Context(), true); err != nil {
- t.Fatal(err)
- }
- if err := tx.begin(t.Context(), false); err != nil {
- t.Fatal(err)
- }
- drop := filepath.Join(tx.units, "carlos-edge.service.d")
- if _, err := os.Lstat(drop); !os.IsNotExist(err) {
- t.Fatal("existing edge overrides", err)
- }
- t.Cleanup(func() { os.RemoveAll(drop) })
- // The transaction mechanics run a harmless one-shot; the seed CLI has its
- // own delegated root witness. It must not resume the original sleep unit.
- raw := activationSeedFixtureUnit
- env := filepath.Join(offlineRebootState, "leo.env")
- if err := os.WriteFile(env, []byte("CARLOS_HOST_CLAIM=1\nCARLOS_HOST_CLAIM_ORIGINS=1\n"), 0600); err != nil {
- t.Fatal(err)
- }
- control := filepath.Join(tx.units, claimControlUnit)
- if _, err := os.Lstat(control); !os.IsNotExist(err) {
- t.Fatal("existing control unit", err)
- }
- t.Cleanup(func() { os.Remove(control); offlineRealSystemctl(context.Background(), "daemon-reload") })
- ctl := func(args ...string) (string, error) { b, e := tx.systemctl(t.Context(), args...); return string(b), e }
- if err := installActivationControl(env, tx.units, true, ctl); err != nil {
- t.Fatal(err)
- }
- if _, err := os.Stat(control); !os.IsNotExist(err) {
- t.Fatal("dry run installed control")
- }
- if err := installActivationControl(env, tx.units, false, ctl); err != nil {
- t.Fatal(err)
- }
- source := filepath.Join(offlineRebootState, "credentials.json")
- credentials := []byte(`{"role":"arn:aws:iam::640012953448:role/fixture","region":"eu-west-1","bucket":"activation-fixture","access_key":"fixture","secret_key":"fixture"}`)
- if err := os.WriteFile(source, credentials, 0600); err != nil {
- t.Fatal(err)
- }
- target := filepath.Join(offlineRebootState, "fence", "credentials.json")
- scope := activationFenceScope{Bucket: "activation-fixture", Region: "eu-west-1", Role: "arn:aws:iam::640012953448:role/fixture"}
- if err := installActivationFenceCredentials(source, target, scope, true); err != nil {
- t.Fatal(err)
- }
- if _, err := os.Stat(target); !os.IsNotExist(err) {
- t.Fatal("credential preview wrote")
- }
- if err := installActivationFenceCredentials(source, target, scope, false); err != nil {
- t.Fatal(err)
- }
- if err := os.Mkdir(drop, 0755); err != nil {
- t.Fatal(err)
- }
- late := filepath.Join(drop, "99-operator.conf")
- if err := os.WriteFile(late, []byte("[Service]\nExecStart=\nExecStart=/usr/bin/sleep infinity\n"), 0644); err != nil {
- t.Fatal(err)
- }
- if err := tx.seed(t.Context(), raw, true); err == nil {
- t.Fatal("late command override accepted")
- }
- os.Remove(late)
- os.Remove(drop)
- if err := tx.seed(t.Context(), raw, true); err != nil {
- t.Fatal(err)
- }
- if _, err := os.Lstat(drop); !os.IsNotExist(err) {
- t.Fatal("dry run wrote override")
- }
- if err := tx.seed(t.Context(), raw, false); err != nil {
- t.Fatal(err)
- }
- if err := verifyOfflineRootMigration(t.Context()); err != nil {
- t.Fatal(err)
- }
- if target, err := os.Readlink(filepath.Join(tx.units, "carlos-edge.service")); err != nil || target != "/dev/null" {
- t.Fatal("seed did not restore edge mask", target, err)
- }
- if _, err := os.Lstat(filepath.Join(drop, activationSeedDropIn)); !os.IsNotExist(err) {
- t.Fatal("seed override retained", err)
- }
- if err := tx.restore(t.Context(), true); err != nil {
- t.Fatal(err)
- }
- if err := tx.restore(t.Context(), false); err != nil {
- t.Fatal(err)
- }
-}
-
-func TestRootActivationSequence(t *testing.T) {
- t.Run("console-and-roster", TestRootActivationConsoleRuntime)
- t.Run("leo-offline-tools", TestRootActivationSeedTransaction)
-}
-
-var activationSeedFixtureUnit = []byte(strings.ReplaceAll(strings.ReplaceAll(activationSeedUnit, "/opt/carlos/carlos system files seed", "/usr/bin/true"), "carlos-instance-control.service", ""))
-
func TestRootActivationSeedInterruptions(t *testing.T) {
for _, point := range []string{"journal-seed-prepared", "stage-90-host-floor.conf", "replace-90-host-floor.conf", "seed-guard-replaced", "seed-guard-restored", "seed-override", "stage-carlos-edge.service", "replace-carlos-edge.service", "seed-reload", "seed-started", "seed-stopped", "seed-remasked", "seed-removed", "journal-held"} {
t.Run(point, func(t *testing.T) {
- tx := setupOfflineReal(t)
- defer cleanupOfflineReal(t, tx)
- drop := filepath.Join(tx.units, "carlos-edge.service.d")
- if _, e := os.Lstat(drop); !os.IsNotExist(e) {
- t.Fatal("preexisting override directory", e)
- }
- defer os.RemoveAll(drop)
- if err := os.Mkdir(drop, 0755); err != nil {
- t.Fatal(err)
- }
- if err := os.WriteFile(filepath.Join(drop, "90-host-floor.conf"), []byte(activationSeedGuardOriginal), 0644); err != nil {
- t.Fatal(err)
- }
- if err := os.WriteFile(filepath.Join(drop, "91-runner.conf"), []byte("[Service]\nExecStart=\nExecStart=/usr/bin/sleep infinity\n"), 0644); err != nil {
- t.Fatal(err)
- }
- if e := tx.begin(t.Context(), false); e != nil {
- t.Fatal(e)
- }
- injected := errors.New("interruption")
- hit := false
+ tx := setupActivationSeedCLI(t)
+ defer cleanupActivationSeedCLI(t, tx)
+ stop := errors.New("seed interruption")
+ tx.seedPreflight = activationSeedPreflight
tx.checkpoint = func(name string) error {
- if !hit && name == point {
- hit = true
- return injected
+ if name == point {
+ return stop
}
return nil
}
- if e := tx.seed(t.Context(), activationSeedFixtureUnit, false); !errors.Is(e, injected) || !hit {
- t.Fatal("missing interruption", e)
+ if err := tx.seed(t.Context(), []byte(activationSeedUnit), false); !errors.Is(err, stop) {
+ t.Fatal(point, err)
}
tx.checkpoint = nil
- if target, err := os.Readlink(filepath.Join(tx.units, "carlos-edge.service")); err != nil || target != "/dev/null" {
- t.Fatal("failure left edge unmasked", err)
- }
- if point != "journal-held" {
- if e := tx.restore(t.Context(), false); e == nil {
- t.Fatal("restore accepted unfinished seed")
- }
- }
- if e := tx.seed(t.Context(), activationSeedFixtureUnit, false); e != nil {
- t.Fatal("resume", e)
- }
- if e := tx.restore(t.Context(), false); e != nil {
- t.Fatal("restore", e)
+ if err := tx.seed(t.Context(), []byte(activationSeedUnit), false); err != nil {
+ t.Fatal("resume", point, err)
}
+ assertActivationSeedCLIHeld(t, tx)
+ activationSeedPrepareRestore(t)
+ activationSeedCLI(t, "offline-restore")
})
}
}
-func TestRootRebootActivationSeedPhase1(t *testing.T) {
- tx := setupOfflineReal(t)
- if e := tx.begin(t.Context(), false); e != nil {
- t.Fatal(e)
- }
- drop := filepath.Join(tx.units, "carlos-edge.service.d")
- if _, e := os.Lstat(drop); !os.IsNotExist(e) {
- t.Fatal("preexisting override directory", e)
- }
- injected := errors.New("reboot window")
- tx.checkpoint = func(name string) error {
- if name == "seed-started" {
- return injected
- }
- return nil
- }
- if e := tx.seed(t.Context(), activationSeedFixtureUnit, false); !errors.Is(e, injected) {
- t.Fatal(e)
- }
- boot, e := os.ReadFile("/proc/sys/kernel/random/boot_id")
- if e != nil {
- t.Fatal(e)
- }
- if e = os.WriteFile(offlineRebootState+"/seed-boot", boot, 0600); e != nil {
- t.Fatal(e)
- }
-}
-func TestRootRebootActivationSeedPhase2(t *testing.T) {
- before, e := os.ReadFile(offlineRebootState + "/seed-boot")
- if e != nil {
- t.Fatal(e)
- }
- now, e := os.ReadFile("/proc/sys/kernel/random/boot_id")
- if e != nil || bytes.Equal(before, now) {
- t.Fatal("no actual reboot", e)
- }
- tx := activationOffline{units: "/etc/systemd/system", state: offlineRebootState + "/transaction", systemctl: offlineRealSystemctl, drained: quiesceLegacyRootMutators}
- defer cleanupOfflineReal(t, tx)
- defer os.RemoveAll(filepath.Join(tx.units, "carlos-edge.service.d"))
- unit, e := tx.systemctl(t.Context(), "show", "carlos-edge.service", "--property=LoadState,ActiveState")
- if e != nil || !strings.Contains(string(unit), "LoadState=masked") || !strings.Contains(string(unit), "ActiveState=inactive") {
- t.Fatal("normal edge after reboot", string(unit), e)
- }
- if e = tx.restore(t.Context(), false); e == nil {
- t.Fatal("restore accepted unfinished seed")
- }
- if e = tx.seed(t.Context(), activationSeedFixtureUnit, false); e != nil {
- t.Fatal(e)
- }
- if e = tx.restore(t.Context(), false); e != nil {
- t.Fatal(e)
- }
-}
-
-func TestRootActivationSeedFailure(t *testing.T) {
- tx := setupOfflineReal(t)
- defer cleanupOfflineReal(t, tx)
- drop := filepath.Join(tx.units, "carlos-edge.service.d")
- if _, e := os.Lstat(drop); !os.IsNotExist(e) {
- t.Fatal("existing override directory", e)
- }
- defer os.RemoveAll(drop)
- if e := tx.begin(t.Context(), false); e != nil {
- t.Fatal(e)
- }
- raw := bytes.ReplaceAll(activationSeedFixtureUnit, []byte("/usr/bin/true"), []byte("/usr/bin/false"))
- if e := tx.seed(t.Context(), raw, false); e == nil {
- t.Fatal("failed seed accepted")
+func TestRootActivationSeedProductionFailures(t *testing.T) {
+ tx := setupActivationSeedCLI(t)
+ defer cleanupActivationSeedCLI(t, tx)
+ late := filepath.Join(tx.units, "carlos-edge.service.d/99-operator.conf")
+ activationProductionWrite(t, late, []byte("[Service]\nExecStart=\nExecStart=/usr/bin/true\n"), 0644)
+ if out, err := activationSeedCLICommand(t, "offline-seed", "--dry-run").CombinedOutput(); err == nil {
+ t.Fatal("late override accepted", string(out))
}
- if e := verifyOfflineRootMigration(t.Context()); e != nil {
- t.Fatal("failure did not remask", e)
- }
- result, e := os.ReadFile(filepath.Join(tx.state, "seed-result.txt"))
- if e != nil || !bytes.Contains(result, []byte("ExecMainStatus=1")) {
- t.Fatal("missing failed result", string(result), e)
- }
- if _, e := os.Lstat(filepath.Join(drop, activationSeedDropIn)); !os.IsNotExist(e) {
- t.Fatal("failed seed override retained", e)
+ if err := os.Remove(late); err != nil {
+ t.Fatal(err)
}
- if e := tx.seed(t.Context(), activationSeedFixtureUnit, false); e != nil {
- t.Fatal("retry", e)
+ activationProductionWrite(t, "/data/activation-payload-fail", nil, 0600)
+ if out, err := activationSeedCLICommand(t, "offline-seed").CombinedOutput(); err == nil {
+ t.Fatal("failed real seed unit accepted", string(out))
}
- if e := tx.restore(t.Context(), false); e != nil {
- t.Fatal("restore", e)
+ load, active, err := tx.status(t.Context(), activationEdgeUnit)
+ if err != nil || load != "masked" || (active != "inactive" && active != "failed") {
+ t.Fatal("failed start escaped", load, active, err)
}
-}
-
-func TestRootActivationSeedGuardArmed(t *testing.T) {
- tx := setupOfflineReal(t)
- defer cleanupOfflineReal(t, tx)
- drop := filepath.Join(tx.units, "carlos-edge.service.d")
- if err := os.Mkdir(drop, 0755); err != nil {
+ if err := os.Remove("/data/activation-payload-fail"); err != nil {
t.Fatal(err)
}
- defer os.RemoveAll(drop)
- guard := []byte("[Unit]\nRequires=carlos-system-recover.service\nAfter=carlos-system-recover.service\n[Service]\nExecStart=\nExecStart=/opt/carlos/system/guard/carlos system guard-exec edge\n")
- path := filepath.Join(drop, "90-host-floor.conf")
- if err := os.WriteFile(path, guard, 0644); err != nil {
+ activationSeedCLI(t, "offline-seed")
+ assertActivationSeedCLIHeld(t, tx)
+ cmd := startActivationSeedCLIBlocked(t)
+ if err := cmd.Process.Kill(); err != nil {
t.Fatal(err)
}
- if err := tx.begin(t.Context(), false); err != nil {
+ cmd.Wait()
+ if err := os.Rename("/etc/carlos/host.env", "/etc/carlos/host.env.saved"); err != nil {
t.Fatal(err)
}
- if err := tx.seed(t.Context(), activationSeedFixtureUnit, false); err != nil {
- t.Fatal(err)
+ if out, err := activationSeedCLICommand(t, "offline-seed").CombinedOutput(); err == nil {
+ t.Fatal("preflight succeeded without configuration", string(out))
}
- got, err := os.ReadFile(path)
- if err != nil || !bytes.Equal(got, guard) {
- t.Fatal("guard not restored", err)
+ load, active, err = tx.status(t.Context(), activationEdgeUnit)
+ if err != nil || load != "masked" || (active != "inactive" && active != "failed") {
+ t.Fatal("retry preflight did not remask", load, active, err)
}
- if target, err := os.Readlink(filepath.Join(tx.units, "carlos-edge.service")); err != nil || target != "/dev/null" {
- t.Fatal("edge not remasked", err)
+ if err := os.Rename("/etc/carlos/host.env.saved", "/etc/carlos/host.env"); err != nil {
+ t.Fatal(err)
}
-}
-
-func TestRootActivationSeedRemasksEarlyFailures(t *testing.T) {
- for _, failure := range []string{"reload", "show", "effective", "preflight-resume"} {
- t.Run(failure, func(t *testing.T) {
- tx := setupOfflineReal(t)
- defer cleanupOfflineReal(t, tx)
- drop := filepath.Join(tx.units, "carlos-edge.service.d")
- if err := os.Mkdir(drop, 0755); err != nil {
- t.Fatal(err)
- }
- defer os.RemoveAll(drop)
- if err := os.WriteFile(filepath.Join(drop, "90-host-floor.conf"), []byte(activationSeedGuardOriginal), 0644); err != nil {
- t.Fatal(err)
- }
- if err := tx.begin(t.Context(), false); err != nil {
- t.Fatal(err)
- }
- injected := errors.New("seed boundary failure")
- base := tx.systemctl
- hit := false
- tx.systemctl = func(ctx context.Context, args ...string) ([]byte, error) {
- unmasked := sameActivationFile(filepath.Join(tx.units, "carlos-edge.service"), filepath.Join(tx.state, "carlos-edge.service.original"))
- if !hit && unmasked && (failure == "reload" && args[0] == "daemon-reload" || (failure == "show" || failure == "effective") && args[0] == "show" && strings.Contains(strings.Join(args, " "), "ExecStart,Type")) {
- hit = true
- if failure == "effective" {
- return []byte("ExecStart=wrong"), nil
- }
- return nil, injected
- }
- return base(ctx, args...)
- }
- if failure == "preflight-resume" {
- tx.checkpoint = func(name string) error {
- if name == "seed-reload" {
- hit = true
- return injected
- }
- return nil
- }
- }
- if err := tx.seed(t.Context(), activationSeedFixtureUnit, false); err == nil || !hit {
- t.Fatal("failure not reached", err)
- }
- tx.systemctl = base
- tx.checkpoint = nil
- if failure == "preflight-resume" {
- // Reconstruct the unmasked crash window, then fail the retry preflight.
- units, state, err := tx.dirs(false)
- if err != nil {
- t.Fatal(err)
- }
- r, err := tx.read(state)
- if err != nil {
- t.Fatal(err)
- }
- if err := tx.replace(units, state, r.Items[len(r.Items)-1], true); err != nil {
- t.Fatal(err)
- }
- units.Close()
- state.Close()
- tx.seedPreflight = func(context.Context) error { return injected }
- if err := tx.seed(t.Context(), activationSeedFixtureUnit, false); !errors.Is(err, injected) {
- t.Fatal(err)
- }
- tx.seedPreflight = nil
- }
- load, active, err := tx.status(t.Context(), "carlos-edge.service")
- if err != nil || load != "masked" || active != "inactive" {
- t.Fatal("failure left edge available", load, active, err)
- }
- if err := tx.seed(t.Context(), activationSeedFixtureUnit, false); err != nil {
- t.Fatal("resume", err)
- }
- })
+ if err := os.Remove("/data/activation-payload-wait"); err != nil {
+ t.Fatal(err)
}
+ activationSeedCLI(t, "offline-seed")
+ assertActivationSeedCLIHeld(t, tx)
+ activationSeedPrepareRestore(t)
+ activationSeedCLI(t, "offline-restore")
}
diff --git a/internal/host/systemapply.go b/internal/host/systemapply.go
index 9bf6fb4f9..57fc290f1 100644
--- a/internal/host/systemapply.go
+++ b/internal/host/systemapply.go
@@ -273,7 +273,7 @@ func (a *systemApplier) file(name string) string { return filepath.Join(a.rootDi
// journal is closed, and re-firing OnFailure= over a finished recovery
// would only start it again with nothing left to do.
func (a *systemApplier) Run(ctx context.Context, recover bool) (retErr error) {
- if err := activationAdoptionAllowed(a.etcDir); err != nil {
+ if err := activationTransactionAllowed(a.etcDir, recover); err != nil {
return err
}
defer func() {
diff --git a/internal/provision/testdata/plan14-root-reads-activation-tools.tsv b/internal/provision/testdata/plan14-root-reads-activation-tools.tsv
index 3893d7b83..94bb7bdbe 100644
--- a/internal/provision/testdata/plan14-root-reads-activation-tools.tsv
+++ b/internal/provision/testdata/plan14-root-reads-activation-tools.tsv
@@ -1,7 +1,7 @@
-# Activation-tools runs 1–4: proposals only; loaded after all existing overlays.
+# Activation-tools runs 1–5: proposals only; loaded after all existing overlays.
@coverage . efeeda08e816fc2c7e3aa451b06cce95ab5778af79682290cf98a60d8b9a841a
@coverage internal/claimactivation 625d82dce14bae92f619077ca92a8c9066a9446545dfcc96a6f818e89e20d4c2
-@coverage internal/host d80147d3615919205f195df63aa3a81a0fb3780f354160aa78c76c75a5e85336
+@coverage internal/host cd645b7f97bbb52b333f7fe60419d2d8b8cc41d9237d70b38495c525e8528167
@coverage internal/hostfence 552570510fb62d2a8822a3f24fa76aeb1437c0a9516966dc5ce15ef58e3fb4c6
internal/claimactivation/runner.go Runner.Run authority F,G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=trusted operator phases bind expected signed roster revision, exact deployment and actors; verify each write; dry-run overlay never calls backing Put file:Read@310,file:Read@5986,file:Read@6318 683059aedf27656d4b49c88102089b2bb2f2ce7fcc108ff8545e52a86a0f8385
internal/claimactivation/runner.go checkedStore.Delete authority F,G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=trusted operator phases bind expected signed roster revision, exact deployment and actors; verify each write; dry-run overlay never calls backing Put 6bc9103a30cfd53cea59d611ccacb7dcc0ca0ea620b403497ef543f5bdaa0217
@@ -30,7 +30,7 @@ internal/host/activation_offline_linux.go activationOffline.prepare authority G,
internal/host/activation_offline_linux.go activationOffline.preview authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=explicit root operator command only; protected ancestry, exact inputs, retained mask ownership and durable readback; no automatic claims-off call a092bb6fb96f64e406a266cef59a4516145a7f58935ff8d1bbc42a96a653aab1
internal/host/activation_offline_linux.go activationOffline.read authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=explicit root operator command only; protected ancestry, exact inputs, retained mask ownership and durable readback; no automatic claims-off call 6ffaa998c2441b9a2d7889275b526bace11feb269df6856527b199a40cfa5bec
internal/host/activation_offline_linux.go activationOffline.replace authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=explicit root operator command only; protected ancestry, exact inputs, retained mask ownership and durable readback; no automatic claims-off call 2a64167ac7053a5c73fe64aa9fea7e4f8db3b1409c7e34fa78c1b842328ccdbb
-internal/host/activation_offline_linux.go activationOffline.restore authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=hold edge mask through durable complete; stop adoption triggers before service reconciliation; persistent journal excludes adoption and unit repair through done; restore triggers before complete and edge last; retain all original ownership until done f6b5f2fc9a6053e7ed39db2ac97ae5a2539b0247c7fadaf028d4edfb7f17fddc
+internal/host/activation_offline_linux.go activationOffline.restore authority G,J4 rev3=PROPOSED-activation-tools;identity=root recovery or explicit offline operator;basis=complete permits recovery only; all recorded services and triggers reconciled before durable done; done releases repair before ordinary guarded edge start; pending edge continuation does not reopen transaction; absent transaction preserves existing behavior b41ef5f53c7afe6f1f09fb61f2fd7f617632875f373957b76204d71ea946ce49
internal/host/activation_offline_linux.go activationOffline.save authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=explicit root operator command only; protected ancestry, exact inputs, retained mask ownership and durable readback; no automatic claims-off call 3082a1585eb287687e05756f495aae7de7e11aab6e96bbee920c50b4807ab9b7
internal/host/activation_offline_linux.go activationOffline.status authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=explicit root operator command only; protected ancestry, exact inputs, retained mask ownership and durable readback; no automatic claims-off call 22467426d7d4cdb3d3f41dfab898f5a5ee49c34a58d4489ab53d6f04ab213587
internal/host/activation_offline_linux.go activationOfflineUnits authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=explicit root operator command only; protected ancestry, exact inputs, retained mask ownership and durable readback; no automatic claims-off call 372c65c44288c766e9b2aa0ece8cbc55c95fb1ccae7e9bae80a5bfcda2550b4b
@@ -51,8 +51,8 @@ internal/host/activation_seed_linux.go activationSeedDropIns authority G,J4 rev3
internal/host/activation_seed_linux.go activationSeedPreflight authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=exact guard replacement journaled before unmask; retain original bytes metadata and inode outside search path; remask on returned errors and interrupted retry preflight; seed only across crash/reboot 3d5596c6bb0cb4a9758867c249828915b15c2a5e4822429308e7d71d2b3c3f61
internal/host/activation_seed_linux.go const activationSeedDropIn authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=exact guard replacement journaled before unmask; retain original bytes metadata and inode outside search path; remask on returned errors and interrupted retry preflight; seed only across crash/reboot e21d702f24a9406e5a889041647e2f24185a6d06b7a8a1d0288c77b5359a85f6
internal/host/activation_seed_linux.go const activationSeedUnit authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=exact guard replacement journaled before unmask; retain original bytes metadata and inode outside search path; remask on returned errors and interrupted retry preflight; seed only across crash/reboot 678e56e4a6a23982f3d6bc8c0f9c465ef8fefb9605dfe36354d3d064ebf22714
-@coverage scripts 9285af7eaf41c3b5962d6bd41c2aad94a2811b0cda46b09b9c9cd76f6c2a5aed
-scripts/activation-tools-root-runner.sh external diagnostic G,J4 rev3=PROPOSED-activation-tools;identity=authorized root-runner operator;basis=builds real console and CLI into test-only embedded images; serial designated runner invocation; deletes only owned build artifacts file:external@0 b96384ea9faabd7db0f541bbe1566803e69013d72e12cf2fb12f029271ab9b71
+@coverage scripts 3ce4672ec33aaabdbadf5dd30e28ff6dda9d012bd55936093f57faf9fcabecab
+scripts/activation-tools-root-runner.sh external diagnostic G,J4 rev3=PROPOSED-activation-tools;identity=authorized root-runner operator;basis=builds real console and CLI into test-only embedded images; serial designated runner invocation; deletes only owned build artifacts file:external@0 17400986085d3a406062f45c9669f88c833502f7bbde138f300db9d5e8a86a51
internal/host/activation_seed_linux.go activationOffline.seedGuard authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=exact guard replacement journaled before unmask; retain original bytes metadata and inode outside search path; remask on returned errors and interrupted retry preflight; seed only across crash/reboot 61ad6431a4874f1d01dfafa5a246b816becc7e3946b7c72eebbd02ee52c53c03
internal/host/activation_seed_linux.go activationOffline.seedGuardCheck authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=exact guard replacement journaled before unmask; retain original bytes metadata and inode outside search path; remask on returned errors and interrupted retry preflight; seed only across crash/reboot 1deb9a7d2e0c01aebdcc0e190ddcb084577ff6c5155b571cf1ea63066bc40f28
internal/host/activation_seed_linux.go activationOffline.seedGuardReplace authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=exact guard replacement journaled before unmask; retain original bytes metadata and inode outside search path; remask on returned errors and interrupted retry preflight; seed only across crash/reboot 56cdf1c30d97949c89707cfab1524240c3728c21a87a3ee8bb6e7c570afbe750
@@ -60,9 +60,11 @@ internal/host/activation_seed_linux.go activationOffline.seedRemask authority G,
internal/host/activation_seed_linux.go const activationSeedGuardOriginal authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=exact guard replacement journaled before unmask; retain original bytes metadata and inode outside search path; remask on returned errors and interrupted retry preflight; seed only across crash/reboot de4e081ef3584a30477e20d3d30bcc0d1f3582607be533921c3896d6548440f2
internal/host/activation_seed_linux.go const activationSeedGuardReplacement authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=exact guard replacement journaled before unmask; retain original bytes metadata and inode outside search path; remask on returned errors and interrupted retry preflight; seed only across crash/reboot d5d0cacb84f15b9d5098e05f41a7981852dc53add1c20b946a1a870ef3dc2736
internal/host/activation_seed_linux.go activationSeedReadAt authority G,J4 rev3=PROPOSED-activation-tools;identity=explicit operator CLI uid 0;basis=protected directory descriptor; nofollow nonblocking regular-file read; root ownership and bounded bytes; retained hardlinks permitted only in seed configuration inspection file:Fstat@368,file:Openat@148,file:ReadAll@545 0fecdfd3967cc9db0f5a5f2cb81a8eb4d440ffb145ab9ef604c77f654d74e94e
-internal/host/activation_adoption_linux.go activationAdoptionAllowed authority G,J4 rev3=PROPOSED-activation-tools;identity=root adoption or explicit offline operator;basis=only an explicit protected offline transaction excludes adoption and automatic repair until done; absent transaction preserves existing behavior; complete is durable before edge release; prior authority constraints retained. file:Lstat@139,file:Stat@359 0d69ed444fc4f8f371ce13993031d209e6b872c691134251012d497a33febd66
+internal/host/activation_adoption_linux.go activationAdoptionAllowed authority G,J4 rev3=PROPOSED-activation-tools;identity=root recovery or explicit offline operator;basis=complete permits recovery only; all recorded services and triggers reconciled before durable done; done releases repair before ordinary guarded edge start; pending edge continuation does not reopen transaction; absent transaction preserves existing behavior 9f5c63bae1686a4eced50eed59019280bd1c7beb6e1e9f5d1efbd5994e2011d7
internal/host/activation_adoption_other.go activationAdoptionAllowed authority G,J4 rev3=PROPOSED-activation-tools;identity=root adoption or explicit offline operator;basis=only an explicit protected offline transaction excludes adoption and automatic repair until done; absent transaction preserves existing behavior; complete is durable before edge release; prior authority constraints retained. 9a70a93e601ea57fe2ce1780cc6cf878ea3cde656770fe8c400d346fbdbca361
internal/host/activation_offline_linux.go var activationEdgeUnit authority G,J4 rev3=PROPOSED-activation-tools;identity=root adoption or explicit offline operator;basis=only an explicit protected offline transaction excludes adoption and automatic repair until done; absent transaction preserves existing behavior; complete is durable before edge release; prior authority constraints retained. 42c79dbb523cf08c19f23d2e4f1754902524276b86f58f0d3d5bfb6117d3fcc6
internal/host/agent.go runAdoptUnderRootRelease authority B3,E1,E2,E3 rev3=PROPOSED-activation-tools;identity=root adoption or explicit offline operator;basis=only an explicit protected offline transaction excludes adoption and automatic repair until done; absent transaction preserves existing behavior; complete is durable before edge release; prior authority constraints retained. rev3=PROPOSED-p3;run=16;identity=root adopt/update/control uid 0;basis=no census selects legacy proof/history and update effects; proof observation cannot fetch signed recovery manifests on that path; participating adoption retains root release exclusion and independent generation pins whole:Route@11179,whole:Route@18368,whole:Route@18404,whole:Route@8481 2f755c7ed82fcac622a118d1e4ad0b5d746de2b97e0456e125114e12351cd0c8
internal/host/systemunits.go ensureSystemUnits authority F,J4 rev3=PROPOSED-activation-tools;identity=root adoption or explicit offline operator;basis=only an explicit protected offline transaction excludes adoption and automatic repair until done; absent transaction preserves existing behavior; complete is durable before edge release; prior authority constraints retained. root-input;kept=F;scope=protected-decision-only file:OpenFile@375,file:ReadFile@3018,file:ReadFile@3151,file:ReadFile@4065,file:Stat@2927 25aa3ae39d4ccbb5aa041caceb98fe2512c4a0ccfe1a1d34776750d46eff3a06
-internal/host/systemapply.go systemApplier.Run authority D,F,J4 rev3=PROPOSED-activation-tools;identity=root adoption or explicit offline operator;basis=only an explicit protected offline transaction excludes adoption and automatic repair until done; absent transaction preserves existing behavior; complete is durable before edge release; prior authority constraints retained. root-input;kept=D,F;scope=protected-decision-only PROPOSED E43: root; release-held context and D3 reservation/census verification under release-before-participation exclusion; no-census effects stay on the legacy path. TestPlan14LaunchVerifierToken, TestPlan14LaunchVerifierCallerOrder, TestPlan14LaunchVerifierNoCensus, TestPlan14LaunchVerifierRootRunner. PROPOSED E45: root enforces census-gated release ordering, bounded acquisition and scoped recovery; registry timer primitives preserve supplied identity but producer integration remains pending. PROPOSED E45: root enforces census-gated release ordering, bounded acquisition and scoped recovery; registry timer primitives preserve supplied identity but producer integration remains pending. file:OpenFile@560 1a0ba9c44022eb6e624b38e99cc356bd1f42701ef48c187aa18f80e2d3312884
+internal/host/systemapply.go systemApplier.Run authority D,F,J4 rev3=PROPOSED-activation-tools;identity=root recovery or explicit offline operator;basis=complete permits recovery only; all recorded services and triggers reconciled before durable done; done releases repair before ordinary guarded edge start; pending edge continuation does not reopen transaction; absent transaction preserves existing behavior file:OpenFile@572 c323ba31cd6960119b09163c7eff7f4e7a86b3d78eeffe14baa96e1ced24847c
+internal/host/activation_adoption_linux.go activationTransactionAllowed authority G,J4 rev3=PROPOSED-activation-tools;identity=root recovery or explicit offline operator;basis=complete permits recovery only; all recorded services and triggers reconciled before durable done; done releases repair before ordinary guarded edge start; pending edge continuation does not reopen transaction; absent transaction preserves existing behavior file:Lstat@157,file:Stat@377 2db0bde7302c754e9115cefc4bb211e1ab479383667067f6e1872c944f64a0c3
+internal/host/activation_adoption_other.go activationTransactionAllowed authority G,J4 rev3=PROPOSED-activation-tools;identity=root recovery or explicit offline operator;basis=complete permits recovery only; all recorded services and triggers reconciled before durable done; done releases repair before ordinary guarded edge start; pending edge continuation does not reopen transaction; absent transaction preserves existing behavior 8c451f30ca31ccca7618f0fd6126ed7d31267c74213c0adb6475e6d504c80723
diff --git a/scripts/activation-tools-root-runner.sh b/scripts/activation-tools-root-runner.sh
index d2ffe2b35..9fc51f2ee 100644
--- a/scripts/activation-tools-root-runner.sh
+++ b/scripts/activation-tools-root-runner.sh
@@ -9,12 +9,31 @@ artifacts=()
trap 'rm -f -- "${artifacts[@]}"; rm -rf -- "$scratch"' EXIT
export GOFLAGS=-buildvcs=false
unset GOCACHE
+listener="$scratch/listener.go"
+cat > "$listener" <<'GO'
+//go:build activationloopback
+
+package main
+
+import "os"
+
+func init() {
+ if len(os.Args) == 2 && os.Args[1] == "agent" {
+ os.Args = append(os.Args, "--http", "127.0.0.1:18080", "--https", "127.0.0.1:18443")
+ }
+}
+GO
+python3 - "$worktree" "$listener" "$scratch/overlay.json" <<'PYJSON'
+import json,sys
+with open(sys.argv[3], 'w') as f:
+ json.dump({"Replace": {sys.argv[1]+"/activation_rootrunner_listener.go":sys.argv[2]}}, f)
+PYJSON
for component in console carlos; do
artifact=internal/host/activation-$component-runner.bin
test ! -e "$artifact"
package=.
if [ "$component" = console ]; then package=./cmd/console; fi
- GOOS=linux GOARCH=arm64 CGO_ENABLED=0 nice -n 10 go build -ldflags '-X github.com/carlosframework/platform/internal/version.Build=v1.0.0' -o "$scratch/$component" "$package"
+ GOOS=linux GOARCH=arm64 CGO_ENABLED=0 nice -n 10 go build -overlay "$scratch/overlay.json" -tags activationloopback -ldflags '-X github.com/carlosframework/platform/internal/version.Build=v1.0.0' -o "$scratch/$component" "$package"
python3 - "$scratch/$component" "$artifact" <<'PY'
import shutil,sys
with open(sys.argv[1], 'rb') as src, open(sys.argv[2], 'xb') as dst: