carlos / platform Public

claim Plan 14 Part 1 fix round 4: after a restart effect, main's settle verdict governs

Merge ruling 5: on a no-census box a binding refusal after the effect never
replaces the outcome or restarts again; a real not-up wins and takes D1
rollback; a healthy unit is accepted after one restart; exe permission
denial follows main. Ordinary routes are excluded from proof observation
and minting (no store calls); the record index caches per pass. Ordinary
routes never read the census. PointLive compares resolved targets.
Per-host restart failures exit 0 as on main. Census overlay fix4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paul Campbell pushed by claude agent, operated by paul@keymail.dev 2f401dd622a185ca1cc3494925ac00566082b98d parent 98b1468
17 files changed, +819 −54
  • internal/host/agent.go +5 −3
  • internal/host/legacyrestart.go +1 −1
  • internal/host/plan14_backend_source_test.go +10 −2
  • internal/host/plan14_constructors_test.go +1 −1
  • internal/host/plan14_fix2_index_test.go +3 −3
  • internal/host/plan14_fix3_live_linux_test.go +12 −1
  • internal/host/plan14_fix4_adopt_linux_test.go +99 −0
  • internal/host/plan14_fix4_live_linux_test.go +548 −0
  • internal/host/proofbinding.go +1 −0
  • internal/host/proofnamed.go +5 −20
  • internal/host/proofobserve.go +4 −0
  • internal/host/proofroot.go +3 −2
  • internal/host/rootrestart.go +79 −18
  • internal/provision/plan14_legacy_identity_test.go +20 −0
  • internal/provision/plan14_root_reads_test.go +6 −0
  • internal/provision/testdata/plan14-root-reads-fix4.tsv +15 −0
  • internal/release/versions.go +7 −3

CI passed — run details

CI log
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.012s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	364.985s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.831s
ok  	github.com/carlosframework/platform/cmd/moveprobe	92.503s
ok  	github.com/carlosframework/platform/internal/activator	227.407s
ok  	github.com/carlosframework/platform/internal/agentsock	0.017s
ok  	github.com/carlosframework/platform/internal/alerting	0.004s
ok  	github.com/carlosframework/platform/internal/analytics	0.023s
ok  	github.com/carlosframework/platform/internal/appconfig	2.489s
ok  	github.com/carlosframework/platform/internal/apperr	0.072s
ok  	github.com/carlosframework/platform/internal/applog	0.137s
ok  	github.com/carlosframework/platform/internal/awsinv	0.004s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.004s
ok  	github.com/carlosframework/platform/internal/boxid	0.655s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.009s
ok  	github.com/carlosframework/platform/internal/caps	0.012s
ok  	github.com/carlosframework/platform/internal/certwarm	0.003s
ok  	github.com/carlosframework/platform/internal/cgroup	0.204s
ok  	github.com/carlosframework/platform/internal/changelog	0.309s
ok  	github.com/carlosframework/platform/internal/clamd	0.055s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	372.009s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.338s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.019s
ok  	github.com/carlosframework/platform/internal/dbverify	0.005s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.005s
ok  	github.com/carlosframework/platform/internal/docsite	0.005s
ok  	github.com/carlosframework/platform/internal/domains	2.300s
ok  	github.com/carlosframework/platform/internal/doorkey	0.009s
ok  	github.com/carlosframework/platform/internal/economics	0.006s
ok  	github.com/carlosframework/platform/internal/edge	26.715s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.009s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.149s
ok  	github.com/carlosframework/platform/internal/gate	1.665s
ok  	github.com/carlosframework/platform/internal/genselect	0.019s
ok  	github.com/carlosframework/platform/internal/geo	0.010s
ok  	github.com/carlosframework/platform/internal/geoip	0.045s
ok  	github.com/carlosframework/platform/internal/host	96.519s
ok  	github.com/carlosframework/platform/internal/hostactors	0.531s
ok  	github.com/carlosframework/platform/internal/hostbox	2.768s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.756s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.015s
ok  	github.com/carlosframework/platform/internal/hostexec	5.425s
ok  	github.com/carlosframework/platform/internal/hostfence	0.078s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.406s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.018s
ok  	github.com/carlosframework/platform/internal/hostguard	1.055s
ok  	github.com/carlosframework/platform/internal/hostops	1.635s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.004s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.005s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.693s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.366s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.005s
ok  	github.com/carlosframework/platform/internal/infraview	0.003s
ok  	github.com/carlosframework/platform/internal/instancectl	1.715s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.833s
ok  	github.com/carlosframework/platform/internal/instancefs	0.008s
ok  	github.com/carlosframework/platform/internal/instancekey	0.005s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.009s
ok  	github.com/carlosframework/platform/internal/layout	1.525s
ok  	github.com/carlosframework/platform/internal/lease	9.512s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.007s
ok  	github.com/carlosframework/platform/internal/ledger	0.023s
ok  	github.com/carlosframework/platform/internal/mail	0.303s
ok  	github.com/carlosframework/platform/internal/maintenance	0.102s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.675s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.109s
ok  	github.com/carlosframework/platform/internal/metrics	0.020s
ok  	github.com/carlosframework/platform/internal/moves	21.673s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.926s
ok  	github.com/carlosframework/platform/internal/objstore	0.284s
ok  	github.com/carlosframework/platform/internal/ops	0.226s
ok  	github.com/carlosframework/platform/internal/park	0.331s
ok  	github.com/carlosframework/platform/internal/privatefile	0.005s
ok  	github.com/carlosframework/platform/internal/provision	14.184s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.007s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	4.132s
ok  	github.com/carlosframework/platform/internal/relay	7.255s
ok  	github.com/carlosframework/platform/internal/release	20.648s
ok  	github.com/carlosframework/platform/internal/rollout	3.920s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.005s
ok  	github.com/carlosframework/platform/internal/rootaws	0.025s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.016s
ok  	github.com/carlosframework/platform/internal/rootstate	0.033s
ok  	github.com/carlosframework/platform/internal/s3compat	0.011s
ok  	github.com/carlosframework/platform/internal/safefile	0.005s
ok  	github.com/carlosframework/platform/internal/schedule	2.383s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	10.723s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.656s
ok  	github.com/carlosframework/platform/internal/status	0.059s
ok  	github.com/carlosframework/platform/internal/steering	0.007s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.141s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.005s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.007s
ok  	github.com/carlosframework/platform/internal/unixdial	0.006s
ok  	github.com/carlosframework/platform/internal/updatewords	0.007s
ok  	github.com/carlosframework/platform/internal/usage	0.011s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.607s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: cached, digest verified
  mc: cached, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.024s
=== RUN   TestT4StorageProbeMinio
2026/10/13 16:20:31 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DE23395C3BA7F5, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.022s
minio contract: PASS
diff --git a/internal/host/agent.go b/internal/host/agent.go
index be471d701..3d8fdf552 100644
--- a/internal/host/agent.go
+++ b/internal/host/agent.go
@@ -4339,7 +4339,9 @@ func runAdoptUnderRootRelease(args []string, buildClaims ClaimFactory, ensureUni
// settled: D1's settle passing records the build in the healthy history
// (D3, proofobserve.go). Root is the process that can hash the unit's
// executable.
- settled := recordSettled(systemdUnits{}, procExePath, release.OptDir(), release.RootHealthyDir, time.Now)
+ observer := productionProofObserver(release.OptDir())
+ observer.resolve = restartChecks.resolve
+ settled := observer.recordSettled(release.RootHealthyDir)
restart := rootRestartRouteWith(reg, systemdUnits{}, release.OptDir(), os.Getenv, settled, restartChecks)
// The restart-request half, and the ONE place it dispatches from —
// including for exec-backed routes, whose children belong to the agent:
@@ -4464,7 +4466,7 @@ func runAdoptUnderRootRelease(args []string, buildClaims ClaimFactory, ensureUni
// no-ops on the stopped unit, exits 0, and stamps a build that has never
// run). -stage-only keeps it for the same reason it keeps the converge:
// a relayed stage is exactly the request that performs such a restart.
- passOpts := []release.PassOption{release.WithCapabilities(caps.Ops{}), release.BakeDir(dataDir()), release.WithRestartFailurePropagation(), release.WithRestartAttemptSettled(restartChecks.finish),
+ passOpts := []release.PassOption{release.WithCapabilities(caps.Ops{}), release.BakeDir(dataDir()), release.WithRestartAttemptSettled(restartChecks.finish),
release.WithRunningVersion(restartChecks.runningVersion),
release.WithSelectedRestarts(
rootSelectedRestart(reg, systemdUnits{}, release.OptDir(), os.Getenv, settled, restartChecks, false),
@@ -4489,7 +4491,7 @@ func runAdoptUnderRootRelease(args []string, buildClaims ClaimFactory, ensureUni
// proving box and, on the hourly pass, say when a requirement is off;
// a route the pass changed is looked at again first.
pr := proofRoot{
- obs: productionProofObserver(release.OptDir()),
+ obs: observer,
reg: reg,
store: store,
bakeDir: dataDir(),
diff --git a/internal/host/legacyrestart.go b/internal/host/legacyrestart.go
index be4a5d070..55092d1b3 100644
--- a/internal/host/legacyrestart.go
+++ b/internal/host/legacyrestart.go
@@ -40,7 +40,7 @@ func (c rootRestartChecks) noteLegacy() {
func restartEffectFailed(err error) bool {
var notUp *release.UnitNotUpError
- return errors.As(err, &notUp) && !errors.Is(err, errRestartBinding)
+ return errors.As(err, &notUp)
}
// This is main's stamped-unit path, including its fallback on Show or
diff --git a/internal/host/plan14_backend_source_test.go b/internal/host/plan14_backend_source_test.go
index 473015710..69c8ee102 100644
--- a/internal/host/plan14_backend_source_test.go
+++ b/internal/host/plan14_backend_source_test.go
@@ -8,6 +8,7 @@ import (
"testing"
"time"
+ "github.com/carlosframework/platform/internal/activator"
"github.com/carlosframework/platform/internal/domains"
"github.com/carlosframework/platform/internal/provision"
"github.com/carlosframework/platform/internal/registry"
@@ -74,6 +75,9 @@ func TestPlan14OrdinaryDispatchSourceGap(t *testing.T) {
if err != nil || need {
t.Fatalf("%s ordinary record dispatched: need=%v err=%v", forge, need, err)
}
+ if err := reg.Delete(candidate.Host); err != nil {
+ t.Fatal(err)
+ }
if err := reg.Upsert(candidate); err != nil {
t.Fatal(err)
}
@@ -82,8 +86,12 @@ func TestPlan14OrdinaryDispatchSourceGap(t *testing.T) {
if err := rootRestartRouteWith(reg, u, "", nil, nil, checks)(record.Host); err != nil {
t.Fatal(err)
}
- if len(u.effects) != 0 {
- t.Fatalf("ordinary restart: %v", u.effects)
+ if candidate.Unit == "" {
+ if len(u.effects) != 1 || u.effects[0] != activator.UnitName(record.Host) {
+ t.Fatalf("ordinary restart: %v", u.effects)
+ }
+ } else if len(u.effects) != 0 {
+ t.Fatalf("forged named restart: %v", u.effects)
}
}
diff --git a/internal/host/plan14_constructors_test.go b/internal/host/plan14_constructors_test.go
index 2ee90571a..d2af5b7f9 100644
--- a/internal/host/plan14_constructors_test.go
+++ b/internal/host/plan14_constructors_test.go
@@ -45,7 +45,7 @@ func TestPlan14ProductionProofConstructors(t *testing.T) {
}
}
if call, ok := n.(*ast.CallExpr); ok {
- if fn, ok := call.Fun.(*ast.Ident); ok && fn.Name == "recordSettled" && len(call.Args) == 5 && isRootHealthy(call.Args[3]) {
+ if fn, ok := call.Fun.(*ast.SelectorExpr); ok && fn.Sel.Name == "recordSettled" && len(call.Args) == 1 && isRootHealthy(call.Args[0]) {
settledWriter++
}
}
diff --git a/internal/host/plan14_fix2_index_test.go b/internal/host/plan14_fix2_index_test.go
index 2bb2f39a0..b62e6e309 100644
--- a/internal/host/plan14_fix2_index_test.go
+++ b/internal/host/plan14_fix2_index_test.go
@@ -33,13 +33,13 @@ func (s *fix2IndexStore) Get(ctx context.Context, key string) ([]byte, string, e
}
return s.Store.Get(ctx, key)
}
-func TestPlan14Fix2RecordIndexRetriesAndRefreshesBodies(t *testing.T) {
+func TestPlan14Fix2RecordIndexCachesDiscoveryAndRefreshesBodies(t *testing.T) {
e, rt, rec, source := plan14ProofFixture(t)
- store := &fix2IndexStore{Store: e.store, accountFailure: true}
+ store := &fix2IndexStore{Store: e.store}
source.store = store
source.cacheIndex = true
if _, err := source.resolve(t.Context(), rt); err != nil {
- t.Fatal("partial listing was refused", err)
+ t.Fatal("record discovery failed", err)
}
lists := store.lists
rec.RecordSeq++
diff --git a/internal/host/plan14_fix3_live_linux_test.go b/internal/host/plan14_fix3_live_linux_test.go
index cadca0d4d..76a9049f8 100644
--- a/internal/host/plan14_fix3_live_linux_test.go
+++ b/internal/host/plan14_fix3_live_linux_test.go
@@ -30,6 +30,7 @@ type r2Units struct {
showErr error
tryErr error
tries []string
+ starts []string
}
func (u *r2Units) Show(unit string) (unitProps, error) { return u.props, u.showErr }
@@ -51,6 +52,14 @@ func (u *r2Units) ResetFailed(unit string) error {
return nil
}
+func (u *r2Units) Start(unit string) error {
+ u.starts = append(u.starts, unit)
+ if u.props.ActiveState == "active" {
+ return nil
+ }
+ return u.Restart(unit)
+}
+
type r2Live struct {
e *proofRootEnv
rt registry.Route
@@ -131,7 +140,9 @@ func (f *r2Live) restart() func(string) error {
return rootRestartRouteWith(f.e.root.reg, f.u, f.u.opt, func(string) string { return "5s" }, nil, f.c)
}
func (f *r2Live) pass(t *testing.T) error {
- return release.Pass(t.Context(), f.e.root.reg, f.e.store, f.pub, f.u.opt, f.restart(), release.WithRunningVersion(f.c.runningVersion), release.WithRestartFailurePropagation(), release.WithStartUnit(rollbackUnitRestart(f.e.root.reg, f.u, f.u.opt, func(string) string { return "5s" }, f.c)), release.WithRollbackRestart(rollbackUnitRestart(f.e.root.reg, f.u, f.u.opt, func(string) string { return "5s" }, f.c)))
+ start := f.c
+ start.legacyStart = f.u.Start
+ return release.Pass(t.Context(), f.e.root.reg, f.e.store, f.pub, f.u.opt, f.restart(), release.WithRunningVersion(f.c.runningVersion), release.WithRestartFailurePropagation(), release.WithStartUnit(rollbackUnitRestart(f.e.root.reg, f.u, f.u.opt, func(string) string { return "5s" }, start)), release.WithRollbackRestart(rollbackUnitRestart(f.e.root.reg, f.u, f.u.opt, func(string) string { return "5s" }, f.c)))
}
func (f *r2Live) wantLive(t *testing.T, v string) {
t.Helper()
diff --git a/internal/host/plan14_fix4_adopt_linux_test.go b/internal/host/plan14_fix4_adopt_linux_test.go
new file mode 100644
index 000000000..bdf3bdf2a
--- /dev/null
+++ b/internal/host/plan14_fix4_adopt_linux_test.go
@@ -0,0 +1,99 @@
+package host
+
+import (
+ "encoding/base64"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/activator"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func TestPlan14Fix4AdoptPerHostFailureExitsZero(t *testing.T) {
+ if os.Getenv("PLAN14_FIX4_ADOPT_CHILD") == "1" {
+ args := []string{"-registry", os.Getenv("PLAN14_FIX4_REGISTRY")}
+ if os.Getenv("PLAN14_FIX4_STAGE") == "1" {
+ args = append(args, "-stage-only")
+ }
+ runAdoptWithEnsureSystemUnits(args, nil, func() {})
+ return
+ }
+ if os.Geteuid() == 0 {
+ t.Skip("fixture exercises the local registry as an unprivileged user")
+ }
+ for _, stage := range []bool{false, true} {
+ for _, crash := range []bool{false, true} {
+ t.Run(map[bool]string{false: "hourly", true: "stage"}[stage]+map[bool]string{false: "/restart-error", true: "/rollback-error"}[crash], func(t *testing.T) {
+ e, rt, u, _, pub := plan14AdoptionFixture(t)
+ rt.Unit = activator.UnitName(rt.Host)
+ if err := e.root.reg.Upsert(rt); err != nil {
+ t.Fatal(err)
+ }
+ dir := t.TempDir()
+ if err := os.Mkdir(filepath.Join(dir, "data"), 0755); err != nil {
+ t.Fatal(err)
+ }
+ bucket := filepath.Join(dir, "bucket")
+ store := release.NewDirStore(bucket)
+ keys, err := e.store.List(t.Context(), "")
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, key := range keys {
+ raw, _, err := e.store.Get(t.Context(), key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := store.Put(t.Context(), key, raw, ""); err != nil {
+ t.Fatal(err)
+ }
+ }
+ stub := `#!/bin/sh
+case "$1" in
+ show)
+ if [ -f "$PLAN14_FIX4_EFFECT" ]; then
+ printf 'ActiveState=failed\nSubState=failed\nMainPID=0\n'
+ else
+ printf 'Id=console.service\nActiveState=active\nSubState=running\nMainPID=1\nInvocationID=old\nExecStart={ path=%s ; }\n' "$PLAN14_FIX4_LIVE"
+ fi ;;
+ try-restart)
+ if [ "$PLAN14_FIX4_CRASH" = 1 ]; then : > "$PLAN14_FIX4_EFFECT"; exit 0; fi
+ exit 1 ;;
+ start|restart) exit 1 ;;
+ *) exit 0 ;;
+esac
+`
+ if err := os.WriteFile(filepath.Join(dir, "systemctl"), []byte(stub), 0755); err != nil {
+ t.Fatal(err)
+ }
+ cmd := exec.Command(os.Args[0], "-test.run=^TestPlan14Fix4AdoptPerHostFailureExitsZero$", "-test.v")
+ cmd.Dir = dir
+ bit := func(v bool) string {
+ if v {
+ return "1"
+ }
+ return "0"
+ }
+ cmd.Env = isolatedAdoptOrderEnv(t,
+ "PATH="+dir+":"+os.Getenv("PATH"), "PLAN14_FIX4_ADOPT_CHILD=1", "PLAN14_FIX4_STAGE="+bit(stage), "PLAN14_FIX4_CRASH="+bit(crash),
+ "PLAN14_FIX4_REGISTRY="+e.root.reg.Path(), "PLAN14_FIX4_EFFECT="+filepath.Join(dir, "effect"), "PLAN14_FIX4_LIVE="+release.LiveLink(u.opt, rt.Host),
+ "CARLOS_DATA_DIR="+filepath.Join(dir, "data"), "CARLOS_OPT_DIR="+u.opt, "CARLOS_DEPLOYMENT_DIR="+bucket, "CARLOS_DEPLOYMENT_BUCKET=",
+ "CARLOS_HOST_CLAIM=", "CARLOS_REGION_HA=", "CARLOS_MOVES_PHASE1=", "CARLOS_SECRETS_KEY=", "CARLOS_RELEASE_PUBKEY="+base64.StdEncoding.EncodeToString(pub),
+ )
+ out, err := cmd.CombinedOutput()
+ if err != nil || !strings.Contains(string(out), "update to v2 failed") || !strings.Contains(string(out), "may be serving nothing") {
+ t.Fatalf("per-host failure exit=%v\n%s", err, out)
+ }
+ if crash && !strings.Contains(string(out), "did not come up") {
+ t.Fatal("crash did not take D1 rollback", string(out))
+ }
+ if live, err := release.LiveVersion(u.opt, rt.Host); err != nil || live != "v1" {
+ t.Fatal(live, err)
+ }
+ })
+ }
+ }
+}
diff --git a/internal/host/plan14_fix4_live_linux_test.go b/internal/host/plan14_fix4_live_linux_test.go
new file mode 100644
index 000000000..18ecef2ed
--- /dev/null
+++ b/internal/host/plan14_fix4_live_linux_test.go
@@ -0,0 +1,548 @@
+package host
+
+import (
+ "context"
+ "errors"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/activator"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+ "github.com/carlosframework/platform/internal/rootparticipation"
+)
+
+func (f *r2Live) productionPass(t *testing.T) error {
+ t.Helper()
+ start := f.c
+ start.legacyStart = f.u.Start
+ observer := f.e.root.obs
+ observer.resolve, observer.sys = f.c.resolve, f.u
+ observer.procExe = func(int) string { return f.u.exe }
+ pr := f.e.root
+ pr.obs = observer
+ return pr.around(t.Context(), func() error {
+ restart := rootRestartRouteWith(f.e.root.reg, f.u, f.u.opt, func(string) string { return "5s" }, observer.recordSettled(f.e.bake), f.c)
+ return release.Pass(t.Context(), f.e.root.reg, f.e.store, f.pub, f.u.opt, restart,
+ release.WithRunningVersion(f.c.runningVersion), release.WithRestartAttemptSettled(f.c.finish),
+ release.WithStartUnit(rollbackUnitRestart(f.e.root.reg, f.u, f.u.opt, func(string) string { return "5s" }, start)),
+ release.WithRollbackRestart(rollbackUnitRestart(f.e.root.reg, f.u, f.u.opt, func(string) string { return "5s" }, f.c)))
+ })
+}
+
+func r3RestartCount(u *r2Units) int {
+ n := 0
+ for _, effect := range u.effects {
+ if strings.HasPrefix(effect, "restart ") {
+ n++
+ }
+ }
+ return n
+}
+
+func TestR3StickyBindingErrorMasksCrash(t *testing.T) {
+ if !plan14RootFixture(t) {
+ return
+ }
+ for _, kind := range []string{"none", "list"} {
+ for _, state := range []string{"failed", "auto-restart", "healthy"} {
+ t.Run(kind+"/"+state, func(t *testing.T) {
+ f := r2Fixture(t, kind)
+ original := f.c.host
+ f.c.host = func(ctx context.Context, host string) (proofBinding, error) {
+ if r3RestartCount(f.u) > 0 && strings.HasSuffix(f.u.exe, "/v2/console") {
+ return proofBinding{}, os.ErrPermission
+ }
+ return original(ctx, host)
+ }
+ ready := func(registry.Route) func() error {
+ return func() error {
+ if f.u.exe == release.VersionPath(f.u.opt, "", "v2", f.rt.App) && state != "healthy" {
+ return os.ErrDeadlineExceeded
+ }
+ return nil
+ }
+ }
+ f.c.ready, f.c.legacyReady = ready, ready
+ sleep := f.c.sleep
+ f.c.sleep = func(d time.Duration) {
+ sleep(d)
+ if f.u.exe != release.VersionPath(f.u.opt, "", "v2", f.rt.App) || state == "healthy" {
+ return
+ }
+ f.u.props.MainPID = 0
+ if state == "failed" {
+ f.u.props.ActiveState, f.u.props.SubState = "failed", "failed"
+ } else {
+ f.u.props.ActiveState, f.u.props.SubState = "activating", "auto-restart"
+ }
+ }
+ if err := f.productionPass(t); err != nil {
+ t.Fatal("per-host failure changed main's pass status", err)
+ }
+ want, restarts := "v1", 2
+ if state == "healthy" {
+ want, restarts = "v2", 1
+ }
+ f.wantLive(t, want)
+ f.wantRunning(t, want)
+ if got := r3RestartCount(f.u); got != restarts {
+ t.Fatalf("restarts=%d want=%d: %v", got, restarts, f.u.effects)
+ }
+ })
+ }
+ }
+}
+
+func TestR3FallbackDoubleRestart(t *testing.T) {
+ if !plan14RootFixture(t) {
+ return
+ }
+ for _, rollback := range []bool{false, true} {
+ for _, journalFailure := range []bool{false, true} {
+ t.Run(map[bool]string{false: "restart", true: "rollback"}[rollback]+map[bool]string{false: "/binding", true: "/journal"}[journalFailure], func(t *testing.T) {
+ f := r2Fixture(t, "list")
+ initial, err := f.c.resolve(t.Context(), f.rt)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := release.StageArtifact(f.u.opt, "", "v2", f.rt.App, []byte("new build")); err != nil {
+ t.Fatal(err)
+ }
+ if err := release.PointLive(f.u.opt, f.rt.Host, "", "v2", f.rt.App); err != nil {
+ t.Fatal(err)
+ }
+ if rollback {
+ f.u.crashNew = true
+ }
+ original := f.c.host
+ f.c.host = func(ctx context.Context, host string) (proofBinding, error) {
+ if r3RestartCount(f.u) > 0 {
+ return proofBinding{}, os.ErrPermission
+ }
+ return original(ctx, host)
+ }
+ if journalFailure {
+ f.c.host = original
+ store, err := f.c.attempts()
+ if err != nil {
+ t.Fatal(err)
+ }
+ memory := store.(*rollbackMemory)
+ f.u.afterEffect = func() { memory.failWrite = true }
+ }
+ err = f.restart()(f.rt.Host)
+ if !rollback {
+ if err != nil {
+ t.Fatal(err)
+ }
+ f.wantRunning(t, "v2")
+ if r3RestartCount(f.u) != 1 {
+ t.Fatal(f.u.effects)
+ }
+ return
+ }
+ var notUp *release.UnitNotUpError
+ if !errors.As(err, &notUp) {
+ t.Fatal("missing crash verdict", err)
+ }
+ if err := release.PointLive(f.u.opt, f.rt.Host, "", "v1", f.rt.App); err != nil {
+ t.Fatal(err)
+ }
+ if journalFailure {
+ store, _ := f.c.attempts()
+ store.(*rollbackMemory).failWrite = false
+ }
+ f.c.host = original
+ f.c.rollback = func(ctx context.Context, b proofBinding) (proofBinding, error) {
+ return f.source.rollback(ctx, initial)
+ }
+ f.u.afterEffect = nil
+ if journalFailure {
+ store, _ := f.c.attempts()
+ f.u.afterEffect = func() { store.(*rollbackMemory).failWrite = true }
+ }
+ // Refuse identity after the rollback effect, inside the callback's copy.
+ host := f.c.host
+ f.c.host = func(ctx context.Context, h string) (proofBinding, error) {
+ if r3RestartCount(f.u) >= 2 {
+ return proofBinding{}, os.ErrPermission
+ }
+ return host(ctx, h)
+ }
+ before := r3RestartCount(f.u)
+ if err := rollbackUnitRestart(f.e.root.reg, f.u, f.u.opt, func(string) string { return "5s" }, f.c)(f.rt.Host); err != nil {
+ t.Fatal(err)
+ }
+ if r3RestartCount(f.u)-before != 1 {
+ t.Fatal(f.u.effects)
+ }
+ f.wantLive(t, "v1")
+ f.wantRunning(t, "v1")
+ })
+ }
+ }
+}
+
+type r3ExeUnits struct {
+ *r2Units
+ denied bool
+}
+
+func (u *r3ExeUnits) Exe(pid int) (string, error) {
+ if u.denied {
+ return "", os.ErrPermission
+ }
+ return u.r2Units.Exe(pid)
+}
+
+func TestPlan14Fix4ExeDeniedAfterRestart(t *testing.T) {
+ if !plan14RootFixture(t) {
+ return
+ }
+ f := r2Fixture(t, "list")
+ u := &r3ExeUnits{r2Units: f.u}
+ f.source.sys = u
+ f.u.afterEffect = func() { u.denied = true }
+ if err := release.StageArtifact(f.u.opt, "", "v2", f.rt.App, []byte("new build")); err != nil {
+ t.Fatal(err)
+ }
+ if err := release.PointLive(f.u.opt, f.rt.Host, "", "v2", f.rt.App); err != nil {
+ t.Fatal(err)
+ }
+ if err := rootRestartRouteWith(f.e.root.reg, u, f.u.opt, func(string) string { return "5s" }, nil, f.c)(f.rt.Host); err != nil {
+ t.Fatal(err)
+ }
+ f.wantLive(t, "v2")
+ f.wantRunning(t, "v2")
+ if r3RestartCount(f.u) != 1 {
+ t.Fatal(f.u.effects)
+ }
+}
+
+func TestR3CensusProbeErrorBlocksOrdinary(t *testing.T) {
+ if !plan14RootFixture(t) {
+ return
+ }
+ if err := os.MkdirAll(rootparticipation.Namespace, 0770); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Chmod(rootparticipation.Namespace, 0770); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := legacyRootRestart(); err == nil {
+ t.Fatal("unreadable census fixture did not refuse")
+ }
+ for _, stamped := range []bool{false, true} {
+ t.Run(map[bool]string{false: "unstamped", true: "derived"}[stamped], func(t *testing.T) {
+ f := r2Fixture(t, "none")
+ f.rt.Unit = ""
+ if stamped {
+ f.rt.Unit = activator.UnitName(f.rt.Host)
+ }
+ if err := f.e.root.reg.Delete(f.rt.Host); err != nil {
+ t.Fatal(err)
+ }
+ if err := f.e.root.reg.Upsert(f.rt); err != nil {
+ t.Fatal(err)
+ }
+ f.c.legacy = func() (bool, error) { t.Fatal("ordinary route probed census"); return false, os.ErrPermission }
+ if err := f.productionPass(t); err != nil {
+ t.Fatal(err)
+ }
+ f.wantLive(t, "v2")
+ f.wantRunning(t, "v2")
+ if err := rollbackUnitRestart(f.e.root.reg, f.u, f.u.opt, func(string) string { return "5s" }, f.c)(f.rt.Host); err != nil {
+ t.Fatal(err)
+ }
+ })
+ }
+}
+
+func TestR3PreviousPins(t *testing.T) {
+ if !plan14RootFixture(t) {
+ return
+ }
+ t.Run("relative live link", func(t *testing.T) {
+ f := r2Fixture(t, "list")
+ link := release.LiveLink(f.u.opt, f.rt.Host)
+ relative, err := filepath.Rel(filepath.Dir(link), f.u.exe)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Remove(link); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Symlink(relative, link); err != nil {
+ t.Fatal(err)
+ }
+ if err := f.productionPass(t); err != nil {
+ t.Fatal(err)
+ }
+ f.wantLive(t, "v2")
+ f.wantRunning(t, "v2")
+ prior, err := filepath.EvalSymlinks(filepath.Join(f.u.opt, "previous", f.rt.Host))
+ if err != nil || prior != release.VersionPath(f.u.opt, "", "v1", f.rt.App) {
+ t.Fatal(prior, err)
+ }
+ })
+}
+
+type r3DiscoveryStore struct {
+ release.Store
+ lists, gets int
+ blocked <-chan struct{}
+}
+
+func (s *r3DiscoveryStore) List(ctx context.Context, prefix string) ([]string, error) {
+ s.lists++
+ if s.blocked != nil {
+ <-s.blocked
+ }
+ return nil, os.ErrPermission
+}
+func (s *r3DiscoveryStore) Get(ctx context.Context, key string) ([]byte, string, error) {
+ s.gets++
+ return s.Store.Get(ctx, key)
+}
+
+func TestPlan14Fix4OrdinaryProductionWrapper(t *testing.T) {
+ if !plan14RootFixture(t) {
+ return
+ }
+ for _, kind := range []string{"unit", "exec", "derived", "hibernate"} {
+ for _, stall := range []bool{false, true} {
+ t.Run(kind+map[bool]string{false: "/error", true: "/hang"}[stall], func(t *testing.T) {
+ f := r2Fixture(t, "none")
+ f.rt.Unit = ""
+ f.rt.Backing = registry.BackingUnit
+ f.rt.Socket = "/run/ordinary.sock"
+ f.rt.Addr, f.rt.Health = "", ""
+ if kind == "exec" {
+ f.rt.Backing = "exec"
+ }
+ if kind == "derived" {
+ f.rt.Unit = activator.UnitName(f.rt.Host)
+ }
+ if kind == "hibernate" {
+ f.rt.Hibernate = true
+ }
+ if err := f.e.root.reg.Delete(f.rt.Host); err != nil {
+ t.Fatal(err)
+ }
+ if err := f.e.root.reg.Upsert(f.rt); err != nil {
+ t.Fatal(err)
+ }
+ store := &r3DiscoveryStore{Store: f.e.store}
+ if stall {
+ unblock := make(chan struct{})
+ store.blocked = unblock
+ timer := time.AfterFunc(time.Second, func() { close(unblock) })
+ defer timer.Stop()
+ }
+ f.source.store = store
+ if err := f.productionPass(t); err != nil {
+ t.Fatal(err)
+ }
+ if store.lists != 0 || store.gets != 0 {
+ t.Fatalf("ordinary proof wrapper read records: lists=%d gets=%d", store.lists, store.gets)
+ }
+ f.wantLive(t, "v2")
+ if kind == "hibernate" {
+ if r3RestartCount(f.u) != 0 {
+ t.Fatal("hibernating route restarted", f.u.effects)
+ }
+ } else {
+ f.wantRunning(t, "v2")
+ }
+ })
+ }
+ }
+}
+
+func TestPlan14Fix4IndexFailureCachedForPass(t *testing.T) {
+ if !plan14RootFixture(t) {
+ return
+ }
+ f := r2Fixture(t, "none")
+ store := &r3DiscoveryStore{Store: f.e.store}
+ f.source.store = store
+ other := f.rt
+ other.Host = "second.example.com"
+ if err := f.e.root.reg.Upsert(other); err != nil {
+ t.Fatal(err)
+ }
+ // Observation, adoption, readiness recording and minting share one resolver.
+ if err := f.productionPass(t); err != nil {
+ t.Fatal(err)
+ }
+ if store.lists != 1 {
+ t.Fatalf("index attempts=%d want=1", store.lists)
+ }
+ next := productionProofBindingSource(store, f.u, f.u.opt)
+ if _, err := next.resolve(t.Context(), f.rt); err == nil {
+ t.Fatal("failed discovery accepted")
+ }
+ if store.lists != 2 {
+ t.Fatalf("new pass did not retry: %d", store.lists)
+ }
+}
+
+func TestPlan14Fix4HibernateStartGuard(t *testing.T) {
+ e, rt, u, c, _ := plan14AdoptionFixture(t)
+ rt.Hibernate = true
+ if err := e.root.reg.Upsert(rt); err != nil {
+ t.Fatal(err)
+ }
+ c.legacyStart = func(string) error { t.Fatal("woke hibernating route"); return nil }
+ c.legacy = func() (bool, error) { t.Fatal("hibernating start read census"); return false, os.ErrPermission }
+ if err := rollbackUnitRestart(e.root.reg, u, u.opt, nil, c)(rt.Host); err != nil {
+ t.Fatal(err)
+ }
+ if len(u.effects) != 0 {
+ t.Fatal(u.effects)
+ }
+}
+
+func TestPlan14Fix4NotUpWinsOverBinding(t *testing.T) {
+ if !restartEffectFailed(errors.Join(errRestartBinding, &release.UnitNotUpError{Unit: "console.service"})) {
+ t.Fatal("binding masked D1 failure")
+ }
+}
+
+func TestPlan14Fix4IncidentProductionWiring(t *testing.T) {
+ if !plan14RootFixture(t) {
+ return
+ }
+ for _, kind := range []string{"none", "guard", "list"} {
+ for _, order := range []string{"reset before adoption", "reset after adoption", "never failed"} {
+ t.Run(kind+"/"+order, func(t *testing.T) {
+ f := r2Fixture(t, kind)
+ if err := release.StageArtifact(f.u.opt, "", "v2", f.rt.App, []byte("new build")); err != nil {
+ t.Fatal(err)
+ }
+ if err := release.PointLive(f.u.opt, f.rt.Host, "", "v2", f.rt.App); err != nil {
+ t.Fatal(err)
+ }
+ if err := f.e.root.reg.SetVersions(f.rt.Host, "v2", "v1"); err != nil {
+ t.Fatal(err)
+ }
+ f.u.exe = release.VersionPath(f.u.opt, "", "v2", f.rt.App)
+ f.u.props.ActiveState, f.u.props.SubState, f.u.props.MainPID = "failed", "failed", 0
+ if order == "never failed" {
+ f.u.props.ActiveState, f.u.props.SubState = "inactive", "dead"
+ }
+ if order == "reset before adoption" {
+ if err := f.u.ResetFailed(f.rt.Unit); err != nil {
+ t.Fatal(err)
+ }
+ }
+ before := len(f.u.effects)
+ if kind != "none" {
+ if _, err := f.source.forHost(t.Context(), f.rt.Host); err != nil {
+ t.Fatal("PID-zero binding rejected", err)
+ }
+ }
+ f.promote(t, "v1")
+ if err := f.productionPass(t); err != nil {
+ t.Fatal(err)
+ }
+ f.wantLive(t, "v1")
+ if len(f.u.starts) != 0 {
+ t.Fatal("adoption called start", f.u.starts)
+ }
+ if order != "reset after adoption" && (f.u.props.MainPID != 0 || len(f.u.effects) != before) {
+ t.Fatal("woke inactive unit", f.u.effects)
+ }
+ if order == "reset after adoption" {
+ if err := f.u.ResetFailed(f.rt.Unit); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if err := f.u.Restart(f.rt.Unit); err != nil {
+ t.Fatal(err)
+ }
+ f.wantRunning(t, "v1")
+ })
+ }
+ }
+}
+
+func TestPlan14Fix4RepointFailureStillLogged(t *testing.T) {
+ if !plan14RootFixture(t) {
+ return
+ }
+ f := r2Fixture(t, "list")
+ if err := os.MkdirAll(filepath.Join(f.u.opt, "previous", f.rt.Host), 0755); err != nil {
+ t.Fatal(err)
+ }
+ output, err := os.CreateTemp(t.TempDir(), "pass-output")
+ if err != nil {
+ t.Fatal(err)
+ }
+ original := os.Stdout
+ os.Stdout = output
+ defer func() { os.Stdout = original; output.Close() }()
+ err = f.productionPass(t)
+ os.Stdout = original
+ if err != nil {
+ t.Fatal("repoint failure changed pass status", err)
+ }
+ if err := output.Close(); err != nil {
+ t.Fatal(err)
+ }
+ raw, err := os.ReadFile(output.Name())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !strings.Contains(string(raw), "repoint failed:") {
+ t.Fatal("missing repoint diagnostic", string(raw))
+ }
+ f.wantLive(t, "v1")
+ f.wantRunning(t, "v1")
+ if r3RestartCount(f.u) != 0 {
+ t.Fatal(f.u.effects)
+ }
+}
+
+type r3ResetErrorUnits struct {
+ *r2Units
+ resets int
+}
+
+func (u *r3ResetErrorUnits) ResetFailed(string) error { u.resets++; return os.ErrPermission }
+
+func TestPlan14Fix4RollbackResetFailureMatchesMain(t *testing.T) {
+ if !plan14RootFixture(t) {
+ return
+ }
+ f := r2Fixture(t, "list")
+ u := &r3ResetErrorUnits{r2Units: f.u}
+ f.source.sys = u
+ if err := release.StageArtifact(u.opt, "", "v2", f.rt.App, []byte("new build")); err != nil {
+ t.Fatal(err)
+ }
+ if err := release.PointLive(u.opt, f.rt.Host, "", "v2", f.rt.App); err != nil {
+ t.Fatal(err)
+ }
+ u.crashNew = true
+ err := rootRestartRouteWith(f.e.root.reg, u, u.opt, func(string) string { return "5s" }, nil, f.c)(f.rt.Host)
+ var notUp *release.UnitNotUpError
+ if !errors.As(err, &notUp) {
+ t.Fatal("missing crash verdict", err)
+ }
+ if err := release.PointLive(u.opt, f.rt.Host, "", "v1", f.rt.App); err != nil {
+ t.Fatal(err)
+ }
+ if err := rollbackUnitRestart(f.e.root.reg, u, u.opt, func(string) string { return "5s" }, f.c)(f.rt.Host); err != nil {
+ t.Fatal(err)
+ }
+ if u.resets != 1 || r3RestartCount(f.u) != 2 {
+ t.Fatal("reset failure repeated or prevented restart", u.resets, u.effects)
+ }
+ f.wantLive(t, "v1")
+ f.wantRunning(t, "v1")
+}
diff --git a/internal/host/proofbinding.go b/internal/host/proofbinding.go
index bb87bd0e6..219a9d4c0 100644
--- a/internal/host/proofbinding.go
+++ b/internal/host/proofbinding.go
@@ -49,6 +49,7 @@ type proofBindingResolver func(context.Context, registry.Route) (proofBinding, e
type proofBindingSource struct {
indexMu sync.Mutex
index *provision.RecordIndex
+ indexErr error
cacheIndex bool
checkPointers func(string) error
now func() time.Time
diff --git a/internal/host/proofnamed.go b/internal/host/proofnamed.go
index 1678bb894..487c43940 100644
--- a/internal/host/proofnamed.go
+++ b/internal/host/proofnamed.go
@@ -206,26 +206,11 @@ func (s *proofBindingSource) findRecord(ctx context.Context, host string) (*prov
}
s.indexMu.Lock()
defer s.indexMu.Unlock()
- if s.index == nil {
- var index *provision.RecordIndex
- var err error
- for attempt := 0; attempt < 3; attempt++ {
- index, err = provision.IndexRecords(ctx, s.store)
- if !errors.Is(err, provision.ErrRecordIndexRetry) {
- break
- }
- timer := time.NewTimer(time.Duration(attempt+1) * 25 * time.Millisecond)
- select {
- case <-ctx.Done():
- timer.Stop()
- return nil, "", ctx.Err()
- case <-timer.C:
- }
- }
- if err != nil {
- return nil, "", err
- }
- s.index = index
+ if s.index == nil && s.indexErr == nil {
+ s.index, s.indexErr = provision.IndexRecords(ctx, s.store)
+ }
+ if s.indexErr != nil {
+ return nil, "", s.indexErr
}
return s.index.Find(ctx, s.store, host)
}
diff --git a/internal/host/proofobserve.go b/internal/host/proofobserve.go
index 011f08a76..0b7db3339 100644
--- a/internal/host/proofobserve.go
+++ b/internal/host/proofobserve.go
@@ -20,6 +20,7 @@ import (
"os"
"time"
+ "github.com/carlosframework/platform/internal/activator"
"github.com/carlosframework/platform/internal/registry"
"github.com/carlosframework/platform/internal/release"
)
@@ -149,6 +150,9 @@ func recordSettled(sys unitSystem, procExe func(pid int) string, optDir, bakeDir
func (o proofObserver) recordSettled(history string) func(registry.Route) error {
return func(rt registry.Route) error {
+ if rt.Unit == "" || rt.Unit == activator.UnitName(rt.Host) {
+ return nil
+ }
b, err := o.observe(rt, 0)
if err != nil {
return err
diff --git a/internal/host/proofroot.go b/internal/host/proofroot.go
index 50cb2a6a0..799dcb29d 100644
--- a/internal/host/proofroot.go
+++ b/internal/host/proofroot.go
@@ -36,6 +36,7 @@ import (
"sort"
"time"
+ "github.com/carlosframework/platform/internal/activator"
"github.com/carlosframework/platform/internal/registry"
"github.com/carlosframework/platform/internal/release"
"github.com/carlosframework/platform/internal/safefile"
@@ -104,7 +105,7 @@ func (r proofRoot) recordLive(ctx context.Context) proofObservations {
return obs
}
for _, rt := range routes {
- if !release.Adoptable(rt) {
+ if !release.Adoptable(rt) || rt.Unit == "" || rt.Unit == activator.UnitName(rt.Host) {
continue
}
obs.byHost[rt.Host] = r.observeAndRecord(rt, pol.ProveAfter())
@@ -172,7 +173,7 @@ func (r proofRoot) mintAndLog(ctx context.Context, obs proofObservations) {
type acctApp struct{ acct, app string }
seen := map[acctApp]bool{}
for _, rt := range routes {
- if !release.Adoptable(rt) {
+ if !release.Adoptable(rt) || rt.Unit == "" || rt.Unit == activator.UnitName(rt.Host) {
continue
}
acct := release.RouteAccount(rt)
diff --git a/internal/host/rootrestart.go b/internal/host/rootrestart.go
index 25e5a243e..1f5ff06c6 100644
--- a/internal/host/rootrestart.go
+++ b/internal/host/rootrestart.go
@@ -26,6 +26,8 @@ type rootRestartChecks struct {
legacyLog *sync.Once
legacyReady func(registry.Route) func() error
legacyStart func(string) error
+ legacyRoute *registry.Route
+ effectStarted *bool
targetDigest string
targetFile func(string) (*os.File, error)
dispatch func(context.Context, registry.Route) (bool, error)
@@ -46,13 +48,17 @@ type rootRestartChecks struct {
func productionRootRestartChecks(sys unitSystem, optDir string) rootRestartChecks {
var sourceMu sync.Mutex
var cached *proofBindingSource
+ var sourceErr error
+ var opened bool
source := func(ctx context.Context) (*proofBindingSource, error) {
sourceMu.Lock()
defer sourceMu.Unlock()
- if cached != nil {
- return cached, nil
+ if opened {
+ return cached, sourceErr
}
+ opened = true
s, err := openDeployStore(ctx)
+ sourceErr = err
if err != nil {
return nil, err
}
@@ -133,6 +139,14 @@ func checkedUnitRestart(sys unitSystem, optDir string, getenv func(string) strin
c = checks[0]
}
return func(candidate registry.Route) error {
+ c := c
+ legacy, err := c.allowLegacy()
+ if err != nil {
+ return err
+ }
+ if legacy {
+ c.legacyRoute = &candidate
+ }
if c.attempts == nil {
return os.ErrPermission
}
@@ -227,12 +241,17 @@ func checkedUnitRestart(sys unitSystem, optDir string, getenv func(string) strin
}
err = restartBoundUnit(sys, optDir, getenv, binding, false, settled, c)
if err == nil {
- return settleRollbackAttempt(s, attempt)
+ settleErr := settleRollbackAttempt(s, attempt)
+ if legacy && settleErr != nil {
+ log.Print(settleErr)
+ return nil
+ }
+ return settleErr
}
if errors.Is(err, errUnitStopped) {
return errors.Join(err, settleRollbackAttempt(s, attempt))
}
- if errors.Is(err, errRestartBinding) {
+ if errors.Is(err, errRestartBinding) && !restartEffectFailed(err) {
return errors.Join(err, settleRollbackAttempt(s, attempt))
}
attempt.Failed = true
@@ -272,6 +291,9 @@ func restartBoundUnit(sys unitSystem, optDir string, getenv func(string) string,
effect = func(fn func() error) error { return fn() }
}
if c.verifyOnly {
+ if c.effectStarted != nil {
+ *c.effectStarted = true
+ }
before.InvocationID = ""
} else if err := effect(func() error {
if rollback {
@@ -280,7 +302,13 @@ func restartBoundUnit(sys unitSystem, optDir string, getenv func(string) string,
return errRestartBinding
}
if err := sys.ResetFailed(rt.Unit); err != nil {
- return err
+ if c.legacyRoute == nil {
+ return err
+ }
+ log.Print(err)
+ }
+ if c.effectStarted != nil {
+ *c.effectStarted = true
}
if err := sys.Restart(rt.Unit); err != nil {
return err
@@ -299,6 +327,9 @@ func restartBoundUnit(sys unitSystem, optDir string, getenv func(string) string,
if err != nil || current.Source != binding.Source || current.Version != binding.Version || current.Digest != binding.Digest || current.PID != binding.PID || current.Invocation != binding.Invocation {
return errRestartBinding
}
+ if c.effectStarted != nil {
+ *c.effectStarted = true
+ }
if pointerBoundUnit(before) {
if err := sys.ResetFailed(rt.Unit); err != nil {
return err
@@ -313,6 +344,17 @@ func restartBoundUnit(sys unitSystem, optDir string, getenv func(string) string,
}); err != nil {
return err
}
+ // Without a census, optional binding checks cannot change main's verdict
+ // after the effect. Settle this invocation once, keeping D1's crash result.
+ if c.legacyRoute != nil {
+ err := settleUnit(sys, rt.Unit, before.InvocationID, legacySettleOptions(optDir, getenv, *c.legacyRoute, want, c))
+ if err == nil && settled != nil {
+ if e := effect(func() error { return settled(binding.Route) }); e != nil {
+ log.Print(e)
+ }
+ }
+ return err
+ }
var observed proofBinding
var bindingErr error
checkedReady := func() error {
@@ -345,7 +387,7 @@ func restartBoundUnit(sys unitSystem, optDir string, getenv func(string) string,
WantDir: filepath.Dir(want), Ready: checkedReady, Now: c.now, Sleep: c.sleep, RequireExe: true,
})
if err != nil {
- if bindingErr != nil {
+ if bindingErr != nil && !restartEffectFailed(err) {
return bindingErr
}
return err
@@ -372,19 +414,32 @@ func rollbackUnitRestart(reg *registry.Registry, sys unitSystem, optDir string,
}
return func(host string) (result error) {
+ c := c
+ rt, routeErr := reg.Get(host)
+ if rt != nil && (rt.Unit == "" || (c.legacyStart != nil && rt.Hibernate)) {
+ return nil
+ }
+ if rt != nil && rt.Unit == activator.UnitName(host) {
+ if c.legacyStart != nil {
+ return c.legacyStart(rt.Unit)
+ }
+ return legacyRollbackRestart(sys, optDir, getenv, *rt, c)
+ }
+ effectStarted := false
+ c.effectStarted = &effectStarted
legacy, err := c.allowLegacy()
if err != nil {
return err
}
if legacy {
- rt, err := reg.Get(host)
- if err != nil || rt == nil || rt.Unit == "" {
+ if routeErr != nil || rt == nil {
return nil
}
+ c.legacyRoute = rt
defer func() {
if errors.Is(result, errUnitStopped) {
result = nil
- } else if result != nil && !restartEffectFailed(result) {
+ } else if result != nil && !effectStarted && !restartEffectFailed(result) {
c.noteLegacy()
if c.legacyStart != nil {
result = c.legacyStart(rt.Unit)
@@ -409,7 +464,16 @@ func rollbackUnitRestart(reg *registry.Registry, sys unitSystem, optDir string,
defer unlock()
attempt, binding, err := loadRollback(s, host, c.now())
if attempt.ID != "" {
- defer func() { result = errors.Join(result, settleRollbackAttempt(s, attempt)) }()
+ defer func() {
+ settleErr := settleRollbackAttempt(s, attempt)
+ if legacy && effectStarted {
+ if settleErr != nil {
+ log.Print(settleErr)
+ }
+ return
+ }
+ result = errors.Join(result, settleErr)
+ }()
}
if err != nil {
if !errors.Is(err, os.ErrNotExist) && !attempt.Settled && (attempt.ID == "" || !c.now().After(attempt.Deadline.Add(rollbackMargin))) {
@@ -483,20 +547,17 @@ func rootSelectedRestart(reg *registry.Registry, sys unitSystem, optDir string,
func rootRestartRouteWith(reg *registry.Registry, sys unitSystem, optDir string, getenv func(string) string, settled func(registry.Route) error, checks rootRestartChecks) func(string) error {
restart := checkedUnitRestart(sys, optDir, getenv, settled, checks)
return func(host string) error {
- legacy, err := checks.allowLegacy()
- if err != nil {
- return err
- }
rt, err := reg.Get(host)
- if legacy && (err != nil || rt == nil || rt.Unit == "" || rt.Unit == activator.UnitName(host)) {
+ if err != nil || rt == nil || rt.Unit == "" {
return restartOrdinaryUnit(sys, host)
}
+ if rt.Unit == activator.UnitName(host) {
+ return legacyCheckedUnitRestart(sys, optDir, getenv, *rt, settled, checks)
+ }
+ legacy, err := checks.allowLegacy()
if err != nil {
return err
}
- if rt == nil {
- return os.ErrPermission
- }
if checks.dispatch != nil {
need, err := checks.dispatch(context.Background(), *rt)
if err != nil {
diff --git a/internal/provision/plan14_legacy_identity_test.go b/internal/provision/plan14_legacy_identity_test.go
index 6166ecb63..d0d7caa36 100644
--- a/internal/provision/plan14_legacy_identity_test.go
+++ b/internal/provision/plan14_legacy_identity_test.go
@@ -151,3 +151,23 @@ func TestPlan14Merge3OverlayPreservesLegacySeal(t *testing.T) {
t.Fatal("unapproved legacy fingerprint accepted")
}
}
+
+func TestPlan14Fix4OverlayPreservesLegacySeal(t *testing.T) {
+ root := censusRoot(t)
+ inv := rootReadInventory{Entries: map[string]classifiedRootRead{}, Coverage: map[string]string{}}
+ for i, name := range []string{"plan14-root-reads.tsv", "plan14-root-reads-moves.tsv", "plan14-root-reads-merge.tsv", "plan14-root-reads-fix1.tsv", "plan14-root-reads-fix2.tsv", "plan14-root-reads-merge3.tsv"} {
+ if err := loadRootReadInventory(root+"/internal/provision/testdata/"+name, &inv, i > 0); err != nil {
+ t.Fatal(err)
+ }
+ }
+ seal := rootReadLegacySeal(inv)
+ if err := loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-fix4.tsv", &inv, true); err != nil {
+ t.Fatal(err)
+ }
+ if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 {
+ t.Fatal(errs)
+ }
+ if seal != "65dba959f80cc47e9fd99be145ba8b24bb3b37ffaf77b47d1b9cc3e01da14b47" {
+ t.Fatal("unexpected legacy seal", seal)
+ }
+}
diff --git a/internal/provision/plan14_root_reads_test.go b/internal/provision/plan14_root_reads_test.go
index c76b527ec..42b03aa9d 100644
--- a/internal/provision/plan14_root_reads_test.go
+++ b/internal/provision/plan14_root_reads_test.go
@@ -628,6 +628,12 @@ func TestPlan14RootReadCensus(t *testing.T) {
t.Fatal(err)
}
seal = "65dba959f80cc47e9fd99be145ba8b24bb3b37ffaf77b47d1b9cc3e01da14b47"
+ if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 {
+ t.Fatal(errs)
+ }
+ if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-fix4.tsv"), &inv, true); err != nil {
+ t.Fatal(err)
+ }
errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...)
errs = append(errs, checkRootReadDeclarations(got, inv)...)
if len(errs) > 0 {
diff --git a/internal/provision/testdata/plan14-root-reads-fix4.tsv b/internal/provision/testdata/plan14-root-reads-fix4.tsv
new file mode 100644
index 000000000..497ba5f51
--- /dev/null
+++ b/internal/provision/testdata/plan14-root-reads-fix4.tsv
@@ -0,0 +1,15 @@
+# Plan 14 fix4: merge ruling 5 preserves main settle after one effect; earlier overlays and the legacy freeze remain unchanged.
+internal/host/agent.go runAdoptUnderRootRelease authority B3,C rev3=merge-ruling-5;basis=Share one lazy resolver across restart, health observation and minting; preserve main exit status for per-host failures. whole:Route@10481,whole:Route@17015,whole:Route@17051,whole:Route@7901 1ad4ed7bf005fbe9b4aad663c4549782e03576e4e5940559de9b3c5404b5b0f2
+internal/host/legacyrestart.go restartEffectFailed authority B3 rev3=merge-ruling-5;basis=A real UnitNotUpError retains D1 authority even when joined with an optional binding error. b68bb65884a14fe0cf78a65f56ded4ea41584bba99af9195827d1ea191b340a9
+internal/host/proofnamed.go proofBindingSource.findRecord authority B3 rev3=merge-ruling-5;basis=Cache index success or failure once per pass; selected records and ETags remain freshly read. 4312342c8ee5220e99a61d14c3101df79e4fa9b7a958965d1539046cb2fa8def
+internal/host/proofobserve.go proofObserver.recordSettled authority B3,J1 rev3=merge-ruling-5;basis=Ordinary unstamped and derived routes do not resolve deployment records for post-restart health observation. registry:Account@308,registry:App@333,registry:Host@174,registry:Unit@127,registry:Unit@144,whole:Route@225,whole:Route@318,whole:Route@343 c295a595019279ffb1a569d480efc6dac87c2a5b09f4f11e5a3376944e5e55bd
+internal/host/proofroot.go proofRoot.mintAndLog authority B3,J1,J4 rev3=merge-ruling-5;basis=Exclude ordinary unstamped and derived routes before post-pass binding and proof minting. file:Stat@1633,registry:App@492,registry:App@651,registry:App@774,registry:Host@409,registry:Host@530,registry:Host@706,registry:Unit@362,registry:Unit@379,whole:Route@322,whole:Route@355,whole:Route@467,whole:Route@564,whole:Route@599 31774259fec4b644f3ddf04c0e5cad606e5f22c5be47bfd14c415c16810d71cf
+internal/host/proofroot.go proofRoot.recordLive authority B3,J1,J4 rev3=merge-ruling-5;basis=Exclude ordinary unstamped and derived routes before pre-pass binding and health observation. registry:Host@502,registry:Host@542,registry:Unit@455,registry:Unit@472,whole:Route@415,whole:Route@448,whole:Route@572 4bcb1687933ded4a3e316bc504212382b0fb07c35407806190ae2b17462953a9
+internal/host/rootrestart.go checkedUnitRestart authority B3 rev3=merge-ruling-5;basis=No-census restarts settle the original effect using main readiness; journal write failures cannot undo a healthy build. file:EvalSymlinks@1180,file:EvalSymlinks@1957,registry:Account@1526,registry:Account@2325,registry:App@1558,registry:App@2357,registry:Host@1227,registry:Host@2013,registry:Host@2900,registry:Host@3167,registry:Host@3945,registry:Host@636,registry:Unit@3827,registry:Unit@837,whole:Route@1534,whole:Route@1566,whole:Route@2333,whole:Route@2365,whole:Route@3835,whole:Route@410,whole:Route@425,whole:Route@764,whole:Route@845 c5a8ac351af8bd694ea6c65a12560dccc3049792e3018f02aac044074970754b
+internal/host/rootrestart.go productionRootRestartChecks authority B3 rev3=merge-ruling-5;basis=Cache a failed store opener as well as a successful source for this pass; root binding authority is unchanged. file:Open@1005,whole:Route@1639,whole:Route@1906,whole:Route@838 1b6f077d4912a86de65a611b981052b0a0ac2a81152bfa209720358f745b511e
+internal/host/rootrestart.go restartBoundUnit authority B3 rev3=merge-ruling-5;basis=After an effect, no-census binding uses main settle once; retained-census identity checks stay strict and real NotUp wins. file:EvalSymlinks@635,registry:Account@3053,registry:Account@4552,registry:App@3089,registry:App@4592,registry:Host@1799,registry:Host@2955,registry:Host@3549,registry:Host@4291,registry:Host@682,registry:Unit@1165,registry:Unit@1353,registry:Unit@2172,registry:Unit@2240,registry:Unit@2282,registry:Unit@2581,registry:Unit@3898,registry:Unit@395,whole:Route@1197,whole:Route@198,whole:Route@2535,whole:Route@2650,whole:Route@2765,whole:Route@3059,whole:Route@3095,whole:Route@317,whole:Route@4560,whole:Route@4600,whole:Route@4811 4fb19705e34138b4665405f60e2659b2d1531d8c3ca957086f3bbd7ba15cda50
+internal/host/rootrestart.go rollbackUnitRestart authority B3 rev3=merge-ruling-5;basis=Only pre-effect legacy refusals may retry; ordinary routes skip census reads, Hibernate prevents start, and strict journal recovery survives a missing registry row. file:Read@1990,registry:Hibernate@393,registry:Host@2282,registry:Unit@1078,registry:Unit@2384,registry:Unit@2685,registry:Unit@351,registry:Unit@446,registry:Unit@538,whole:Route@1160,whole:Route@2290,whole:Route@2392,whole:Route@2693,whole:Route@337,whole:Route@433,whole:Route@606,whole:Route@785,whole:Route@822,whole:Route@836 611e1e0cd91a682ee8e1e8cda7ac6479edf167509b2c3a352f28fc71c840b621
+internal/host/rootrestart.go rootRestartRouteWith authority B3 rev3=merge-ruling-5;basis=Ordinary routes bypass census and records; explicit derived stamps retain main checked settle and D1 rollback. registry:Unit@354,registry:Unit@420,whole:Route@1113,whole:Route@1310,whole:Route@341,whole:Route@515,whole:Route@699,whole:Route@822,whole:Route@983 aff504f5ab71c6a5e5f74b96eccb7a96c761daa1d0438ddd233e9d9867578099
+internal/release/versions.go PointLive authority B3,E2 rev3=merge-ruling-5;basis=Compare resolved current and previous paths before retaining the running build; preserve repoint errors and canonical retained pointers. file:EvalSymlinks@1013,file:EvalSymlinks@901,file:Open@1506,file:Readlink@388,file:Stat@147 ab787418ff9aadb632066f25888326984d2167248abd05481221a6e13a8f665d
+@coverage internal/host 6d9e8455aed7ddfae23690f7323f8c19855a8df19fdccc096e2628a568dfa186
+@coverage internal/release 5845490a356a59aa52f0ad081e702c98bf39cdfd8804002600596b1219d010d9
diff --git a/internal/release/versions.go b/internal/release/versions.go
index 829f1739b..25b7886f1 100644
--- a/internal/release/versions.go
+++ b/internal/release/versions.go
@@ -301,13 +301,17 @@ func PointLive(optDir, host, acct, ver, app string, runningVersion ...string) er
return os.ErrPermission
}
running := VersionPath(optDir, acct, runningVersion[0], app)
- if running != old {
- prior, err := os.Readlink(filepath.Join(dir, host))
+ current, err := filepath.EvalSymlinks(link)
+ if err != nil {
+ return err
+ }
+ if running != current {
+ prior, err := filepath.EvalSymlinks(filepath.Join(dir, host))
if err != nil || prior != running {
return os.ErrPermission
}
- old = running
}
+ old = running
}
if err := safefile.MkdirAll(dir, 0755); err != nil {
return err