amadan

carlos / platform Public

Rollback of a failed import wake: discharge a stop once it is known

Seeded pre-land review, P2. A failed wake of an import's row could leave
the rollback stuck, even after its stop was known, until the agent
restarted:

- an aborted legacy start that #292 proves stopped, at once or later
  through its reaper, kept its wake's uncertainty, and rememberStopLocked
  carried it forward;
- a fence raised during the wake's pending build record refused the
  spawn before root was asked, but the wake was already marked uncertain.

So FenceAndStopIfUnfenced kept answering ErrStopUnproven, StopUncertain
stayed true, and the original route was never restored.

The fix, as ruled, and no wider:
- runExecGate marks its refusals launchRefused. doWake records such a
  start as known not started: no process, sidecar or replica, nothing
  uncertain.
- cleanupFailedWake: when #292's whole proof holds for an aborted legacy
  start, that wake's uncertainty is discharged, and the stages it never
  reached are recorded as held. When it fails, the retained start is
  named as the owner of the uncertainty (owedBy).
- reapUnprovenWake: when a retained legacy start's retried proof holds
  and the host's history is uncertain only because of that start, the
  uncertainty is discharged. rememberStopLocked carries that ownership
  only across a wake refused before it did anything, or the same wake;
  any other uncertainty clears it and stays. A legacy retry waits until
  its failed wake has recorded its outcome.

Everything else is main's: a failed proof still keeps the lease and
blocks the rollback, and a generic backend failure stays uncertain.

Tests (internal/activator, with the real Service and provision's real
rollback): an aborted start proven at once, proven later through the
reaper, and a fence during the pending record each restore the original
route with no restart; a failed proof deletes nothing; refusals are not
uncertain; a proof discharges only its own uncertainty. The first three
fail on main's activator code.

Census: doWake, cleanupFailedWake and reapUnprovenWake (accepted legacy
under census rulings 292 and owed) are re-digested in a block marked
PROPOSED-seeded-instances-292fix, pending the Plan 14 controller's
ruling. No class changed. The activator envelope and the seal are
re-pinned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude 041011239229dc911601b50da059353a0991f901 parent 974ce8d
9 files changed, +568 −22
  • internal/activator/activator.go +6 −0
  • internal/activator/containment.go +18 −0
  • internal/activator/genbackend.go +7 −4
  • internal/activator/maintenance.go +40 −2
  • internal/activator/opset.go +19 −1
  • internal/activator/seed_rollback_stop_test.go +436 −0
  • internal/provision/plan14_legacy_identity_test.go +29 −10
  • internal/provision/plan14_root_reads_test.go +5 −2
  • internal/provision/testdata/plan14-root-reads-seeded-instances.tsv +8 −3

CI passed — run details

CI log 
=== gate ===
go: downloading github.com/benbjohnson/litestream v0.5.14
go: downloading github.com/aws/aws-sdk-go-v2 v1.44.0
go: downloading modernc.org/sqlite v1.55.0
go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.324.0
go: downloading github.com/aws/aws-sdk-go-v2/config v1.32.6
go: downloading github.com/aws/aws-sdk-go-v2/service/sns v1.43.0
go: downloading github.com/aws/aws-sdk-go-v2/service/route53 v1.64.1
go: downloading github.com/superfly/ltx v0.5.1
go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.19.6
go: downloading github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0
go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.45.6
go: downloading golang.org/x/crypto v0.52.0
go: downloading github.com/aws/smithy-go v1.28.1
go: downloading golang.org/x/sys v0.46.0
go: downloading github.com/oschwald/maxminddb-golang v1.13.1
go: downloading github.com/aws/aws-sdk-go-v2/service/acm v1.45.0
go: downloading github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.68.0
go: downloading github.com/aws/aws-sdk-go-v2/service/iam v1.58.2
go: downloading github.com/aws/aws-sdk-go-v2/service/resourcegroupstaggingapi v1.37.0
go: downloading github.com/klauspost/compress v1.18.0
go: downloading github.com/pion/logging v0.2.4
go: downloading github.com/pion/stun/v3 v3.0.1
go: downloading github.com/pion/turn/v4 v4.1.4
go: downloading golang.org/x/net v0.55.0
go: downloading github.com/aws/aws-sdk-go-v2/service/sesv2 v1.66.6
go: downloading github.com/aws/aws-sdk-go-v2/service/costexplorer v1.67.5
go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.40
go: downloading github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.38
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.40
go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16
go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4
go: downloading github.com/aws/aws-sdk-go-v2/service/signin v1.0.4
go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.30.8
go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12
go: downloading github.com/pion/randutil v0.1.0
go: downloading github.com/pion/transport/v4 v4.0.1
go: downloading github.com/pion/dtls/v3 v3.0.7
go: downloading github.com/pion/transport/v3 v3.0.8
go: downloading github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31
go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.40
go: downloading github.com/pierrec/lz4/v4 v4.1.22
go: downloading github.com/wlynxg/anet v0.0.5
go: downloading github.com/prometheus/client_golang v1.17.0
go: downloading golang.org/x/sync v0.21.0
go: downloading github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.18
go: downloading github.com/lmittmann/tint v1.1.3
go: downloading github.com/mattn/go-isatty v0.0.20
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/prometheus/common v0.45.0
go: downloading google.golang.org/protobuf v1.36.10
go: downloading github.com/prometheus/client_model v0.5.0
go: downloading github.com/cespare/xxhash/v2 v2.3.0
go: downloading github.com/prometheus/procfs v0.12.0
go: downloading github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0
go: downloading golang.org/x/text v0.37.0
go: downloading modernc.org/libc v1.74.1
go: downloading modernc.org/memory v1.11.0
go: downloading github.com/google/uuid v1.6.0
go: downloading github.com/dustin/go-humanize v1.0.1
go: downloading modernc.org/mathutil v1.7.1
go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	299.198s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	2.847s
ok  	github.com/carlosframework/platform/cmd/moveprobe	93.439s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.720s
ok  	github.com/carlosframework/platform/internal/activator	248.434s
ok  	github.com/carlosframework/platform/internal/agentsock	0.016s
ok  	github.com/carlosframework/platform/internal/alerting	0.003s
ok  	github.com/carlosframework/platform/internal/analytics	0.019s
ok  	github.com/carlosframework/platform/internal/appconfig	2.515s
ok  	github.com/carlosframework/platform/internal/apperr	0.088s
ok  	github.com/carlosframework/platform/internal/applog	0.131s
ok  	github.com/carlosframework/platform/internal/awsinv	0.004s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.004s
ok  	github.com/carlosframework/platform/internal/boxid	0.736s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.008s
ok  	github.com/carlosframework/platform/internal/caps	0.011s
ok  	github.com/carlosframework/platform/internal/certwarm	0.002s
ok  	github.com/carlosframework/platform/internal/cgroup	0.716s
ok  	github.com/carlosframework/platform/internal/changelog	0.307s
ok  	github.com/carlosframework/platform/internal/clamd	0.058s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	393.047s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.189s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.009s
ok  	github.com/carlosframework/platform/internal/dbverify	0.004s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.002s
ok  	github.com/carlosframework/platform/internal/docsite	0.005s
ok  	github.com/carlosframework/platform/internal/domains	2.367s
ok  	github.com/carlosframework/platform/internal/doorkey	0.006s
ok  	github.com/carlosframework/platform/internal/economics	0.004s
ok  	github.com/carlosframework/platform/internal/edge	27.212s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.008s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.132s
ok  	github.com/carlosframework/platform/internal/gate	1.655s
ok  	github.com/carlosframework/platform/internal/genselect	0.015s
ok  	github.com/carlosframework/platform/internal/geo	0.016s
ok  	github.com/carlosframework/platform/internal/geoip	0.030s
ok  	github.com/carlosframework/platform/internal/host	137.244s
ok  	github.com/carlosframework/platform/internal/hostactors	0.471s
ok  	github.com/carlosframework/platform/internal/hostbox	2.275s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.523s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.016s
ok  	github.com/carlosframework/platform/internal/hostexec	5.445s
ok  	github.com/carlosframework/platform/internal/hostfence	0.039s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.997s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.016s
ok  	github.com/carlosframework/platform/internal/hostguard	0.965s
ok  	github.com/carlosframework/platform/internal/hostops	1.616s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.016s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.017s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.464s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.142s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.016s
ok  	github.com/carlosframework/platform/internal/infraview	0.017s
ok  	github.com/carlosframework/platform/internal/instancectl	4.222s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.727s
ok  	github.com/carlosframework/platform/internal/instancefs	0.020s
ok  	github.com/carlosframework/platform/internal/instancekey	0.016s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.020s
ok  	github.com/carlosframework/platform/internal/layout	1.535s
ok  	github.com/carlosframework/platform/internal/lease	9.846s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.006s
ok  	github.com/carlosframework/platform/internal/ledger	0.012s
ok  	github.com/carlosframework/platform/internal/mail	0.159s
ok  	github.com/carlosframework/platform/internal/maintenance	0.082s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.560s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.079s
ok  	github.com/carlosframework/platform/internal/metrics	0.005s
ok  	github.com/carlosframework/platform/internal/moves	21.472s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.775s
ok  	github.com/carlosframework/platform/internal/objstore	0.190s
ok  	github.com/carlosframework/platform/internal/ops	0.193s
ok  	github.com/carlosframework/platform/internal/park	0.221s
ok  	github.com/carlosframework/platform/internal/privatefile	0.004s
ok  	github.com/carlosframework/platform/internal/provision	37.373s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.008s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.138s
ok  	github.com/carlosframework/platform/internal/relay	7.299s
ok  	github.com/carlosframework/platform/internal/release	20.221s
ok  	github.com/carlosframework/platform/internal/rollout	3.818s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.004s
ok  	github.com/carlosframework/platform/internal/rootaws	0.007s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.009s
ok  	github.com/carlosframework/platform/internal/rootstate	0.025s
ok  	github.com/carlosframework/platform/internal/runs	0.581s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.004s
ok  	github.com/carlosframework/platform/internal/schedule	2.497s
ok  	github.com/carlosframework/platform/internal/seeds	0.012s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	11.887s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	4.195s
ok  	github.com/carlosframework/platform/internal/status	0.039s
ok  	github.com/carlosframework/platform/internal/steering	0.006s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.240s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.003s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.004s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.581s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
go: downloading github.com/ncruces/go-strftime v1.0.0
--- shard 1 of 5 ---
cross-compiling darwin/amd64
go: downloading github.com/ncruces/go-strftime v1.0.0
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 46399650 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
go: downloading github.com/ncruces/go-strftime v1.0.0
--- shard 4 of 5 ---
cross-compiling windows/arm64
go: downloading github.com/ncruces/go-strftime v1.0.0
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: cached, digest verified
  mc: cached, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.023s
=== RUN   TestT4StorageProbeMinio
2026/10/15 08:38:11 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DEA727E6EFA366, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.021s
minio contract: PASS
diff --git a/internal/activator/activator.go b/internal/activator/activator.go
index 75f8e5f50..2a0b08d74 100644
--- a/internal/activator/activator.go
+++ b/internal/activator/activator.go
@@ -2524,6 +2524,12 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
if proc != nil || w.genUnproven || w.legacyUncertain {
a.cleanupFailedWake(ctx, rt, w, lease)
} else {
+ // A start its pre-exec gate refused ran nothing: this wake's
+ // stop is known, not uncertain, and its other stages were
+ // never reached (seeded pre-land review, P2).
+ if launchNotStarted(err) {
+ w.stopOutcome.notStarted()
+ }
w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
}
return w, fmt.Errorf("start %s: %w", host, err)
diff --git a/internal/activator/containment.go b/internal/activator/containment.go
index 0426d36ab..e5497e778 100644
--- a/internal/activator/containment.go
+++ b/internal/activator/containment.go
@@ -53,6 +53,24 @@ func (e legacyStartUncertain) Unwrap() error { return e.err }
// markLegacyStartUncertain wraps err as legacyStartUncertain.
func markLegacyStartUncertain(err error) error { return legacyStartUncertain{err: err} }
+// launchRefused marks a start that its pre-exec gate refused
+// (runExecGate): the caller's exec gate, the wake's pending record or the
+// recheck after it answered no, immediately before the exec or before root
+// was asked. Nothing of that launch ran, so a failed wake whose start comes
+// back marked is known not started, not uncertain (seeded pre-land review,
+// P2). Like startUncertain it changes neither the text nor errors.Is.
+type launchRefused struct{ err error }
+
+func (e launchRefused) Error() string { return e.err.Error() }
+func (e launchRefused) Unwrap() error { return e.err }
+
+// launchNotStarted reports whether err is a start its pre-exec gate
+// refused (launchRefused), and nothing marks it as one that may have run.
+func launchNotStarted(err error) bool {
+ var r launchRefused
+ return errors.As(err, &r) && !legacyStartMayRun(err) && !startMayHaveRun(nil, err)
+}
+
// legacyStartMayRun reports whether err is a legacy run unit's start whose
// queued job may still run (legacyStartUncertain).
func legacyStartMayRun(err error) bool {
diff --git a/internal/activator/genbackend.go b/internal/activator/genbackend.go
index 4daef066e..606601454 100644
--- a/internal/activator/genbackend.go
+++ b/internal/activator/genbackend.go
@@ -91,17 +91,20 @@ func withExecGate(ctx context.Context, gate func() error) context.Context {
// the bucket write, and the exec follows the second answer. A spawn with a
// gate also honours its context there: one that has ended starts nothing.
// With neither (every live-link spawn, the flag off) it is a no-op, so those
-// spawns are exactly as they were.
+// spawns are exactly as they were. Every refusal is marked launchRefused:
+// it is asked immediately before the exec, so nothing of the launch ran.
func runExecGate(ctx context.Context) error {
if err := askExecGate(ctx); err != nil {
- return err
+ return launchRefused{err}
}
wrote, err := runPendingRecord(ctx)
if err != nil {
- return err
+ return launchRefused{err}
}
if wrote {
- return askExecGate(ctx)
+ if err := askExecGate(ctx); err != nil {
+ return launchRefused{err}
+ }
}
return nil
}
diff --git a/internal/activator/maintenance.go b/internal/activator/maintenance.go
index 4aa613e15..9c71a5ab3 100644
--- a/internal/activator/maintenance.go
+++ b/internal/activator/maintenance.go
@@ -38,12 +38,27 @@ type stopObservation struct {
sidecarStop, sidecarExited bool
replicaClosed bool
uncertain bool
+ // owedBy is the one retained failed wake whose unproven legacy start
+ // (#292) is this observation's only uncertainty, or nil. Only that
+ // wake's proof discharges it (reapUnprovenWake); any other source of
+ // uncertainty clears it, and then the uncertainty stays.
+ owedBy *unprovenWake
}
func (o *stopObservation) complete() bool {
return o != nil && !o.uncertain && o.mainStop && o.mainExited && o.sidecarStop && o.sidecarExited && o.replicaClosed
}
+// notStarted records a wake whose launch is known not to have started: no
+// process, no sidecar and no replica of it exist, so every stage holds and
+// nothing is uncertain (seeded pre-land review, P2).
+func (o *stopObservation) notStarted() {
+ o.mainStop, o.mainExited = true, true
+ o.sidecarStop, o.sidecarExited = true, true
+ o.replicaClosed = true
+ o.uncertain = false
+}
+
// FenceAndStop installs a local fence before collecting stop evidence. Once this
// call installs its fence, cancellation or failure leaves admission closed.
// ErrFenceNotInstalled means this call could not install or confirm its fence;
@@ -292,6 +307,13 @@ func (a *Service) rememberStopLocked(host string, observation *stopObservation)
if prior := a.stopHistory[host]; prior != nil && prior.uncertain {
retained := *observation
retained.uncertain = true
+ // The prior's uncertainty stays one retained wake's alone only
+ // while nothing else adds to it: a wake refused before it did
+ // anything (the zero observation), or that same wake again.
+ retained.owedBy = nil
+ if *observation == (stopObservation{}) || (observation.uncertain && observation.owedBy == prior.owedBy) {
+ retained.owedBy = prior.owedBy
+ }
observation = &retained
}
a.stopHistory[host] = observation
@@ -346,16 +368,32 @@ func (a *Service) cleanupFailedWake(ctx context.Context, rt registry.Route, w *w
log.Printf("activator: %s: the aborted start's unit is not proven stopped (%v); keeping the lease until it is", rt.Host, err)
w.stopOutcome.mainExited = false
w.stopOutcome.uncertain = true
+ u := &unprovenWake{rt: rt, gen: w.gen, lease: lease, rep: w.rep, build: w.spawnIdentity,
+ layout: w.launchLayout, legacy: true}
+ // This wake's uncertainty is this retained start's, and its
+ // proof discharges it (reapUnprovenWake; seeded pre-land
+ // review, P2).
+ w.stopOutcome.owedBy = u
a.mu.Lock()
if a.unprovenWakes == nil {
a.unprovenWakes = make(map[string]*unprovenWake)
}
- a.unprovenWakes[rt.Host] = &unprovenWake{rt: rt, gen: w.gen, lease: lease, rep: w.rep, build: w.spawnIdentity,
- layout: w.launchLayout, legacy: true}
+ a.unprovenWakes[rt.Host] = u
a.mu.Unlock()
return
}
+ // #292's whole proof held: the start's unit is stopped and its
+ // cgroup empty. That was this wake's only uncertainty; it reached
+ // no sidecar, and no replica unless it holds one, which is closed
+ // below (seeded pre-land review, P2).
w.stopOutcome.mainStop, w.stopOutcome.mainExited = true, true
+ w.stopOutcome.uncertain = false
+ if w.sidecar == nil {
+ w.stopOutcome.sidecarStop, w.stopOutcome.sidecarExited = true, true
+ }
+ if w.rep == nil {
+ w.stopOutcome.replicaClosed = true
+ }
}
if !legacy && (a.gens || w.genUnproven || w.proc != nil) && !a.failedWakeStopped(rt, w.gen, w.proc, w.launchLayout) {
// Its leader did not exit (ErrStillRunning, a failed kill) or its
diff --git a/internal/activator/opset.go b/internal/activator/opset.go
index 0b5344e2d..27c3d9578 100644
--- a/internal/activator/opset.go
+++ b/internal/activator/opset.go
@@ -823,6 +823,12 @@ func (a *Service) reapUnprovenWakesLocked() {
if u.busy {
continue
}
+ // A legacy start's failed wake records its outcome as its entry
+ // goes, in one critical section; its proof waits for that record,
+ // which it may discharge (seeded pre-land review, P2).
+ if u.legacy && a.entries[host] != nil {
+ continue
+ }
u.busy = true
go a.reapUnprovenWake(host, u)
}
@@ -883,7 +889,19 @@ func (a *Service) reapUnprovenWake(host string, u *unprovenWake) {
log.Printf("activator: %s: failed wake's generation %d is proven stopped; lease given back", host, u.gen.N)
}
a.mu.Lock()
- a.rememberStopLocked(host, outcome)
+ if h := a.stopHistory[host]; u.legacy && h != nil && h.uncertain && h.owedBy == u {
+ // The history's only uncertainty was this start's, and #292's
+ // whole proof now holds: it is discharged, and the stages the
+ // failed wake never reached hold (seeded pre-land review, P2).
+ outcome.uncertain = false
+ outcome.sidecarStop, outcome.sidecarExited = true, true
+ if u.rep == nil {
+ outcome.replicaClosed = true
+ }
+ a.stopHistory[host] = outcome
+ } else {
+ a.rememberStopLocked(host, outcome)
+ }
if a.unprovenWakes[host] == u {
delete(a.unprovenWakes, host)
}
diff --git a/internal/activator/seed_rollback_stop_test.go b/internal/activator/seed_rollback_stop_test.go
new file mode 100644
index 000000000..53686535d
--- /dev/null
+++ b/internal/activator/seed_rollback_stop_test.go
@@ -0,0 +1,436 @@
+package activator
+
+import (
+ "context"
+ "errors"
+ "reflect"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/domains"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+ "github.com/carlosframework/platform/internal/seeds"
+)
+
+// The import rollback (seeded-instances spec §5.6) against this package's
+// real Service: a failed wake of the import's row must not leave the
+// rollback stuck once its stop is known (seeded pre-land review, P2).
+//
+// The rollback's fence hook is internal/host's fenceForRollback, which
+// this package cannot import (host imports it). rollbackFence below is
+// that rule, step for step, with the startup sweep taken as done.
+
+type importRollback struct {
+ a *Service
+ b *fakeBackend
+ reg *registry.Registry
+ store *release.DirStore
+ cfg provision.ReconcileConfig
+ rec provision.Record
+ original registry.Route
+
+ mu sync.Mutex
+ removed int // RemoveLocal calls: the import's local database
+ replicas int // DeleteReplica calls: the host's replica
+}
+
+func newImportRollback(t *testing.T) *importRollback {
+ t.Helper()
+ ctx := context.Background()
+ a, b, _, reg := testActivator(t, time.Hour)
+ a.Routes = reg
+ x := &importRollback{a: a, b: b, reg: reg, store: release.NewDirStore(t.TempDir())}
+
+ sqid := domains.Sqid(2)
+ manifest := `{"id":2,"sqid":"` + sqid + `","name":"Shop","created_at":"2026-08-02T09:00:00Z"}`
+ if _, err := x.store.Put(ctx, "accounts/"+sqid+"/account.json", []byte(manifest), ""); err != nil {
+ t.Fatal(err)
+ }
+ if err := provision.PutManifest(ctx, x.store, sqid, "shop", provision.Manifest{Schema: 1, App: "shop", Domain: "shop.example.com"}); err != nil {
+ t.Fatal(err)
+ }
+ id, err := seeds.NewID()
+ if err != nil {
+ t.Fatal(err)
+ }
+ x.rec = provision.Record{Schema: 1, Host: "a.shop.example.com", Slug: "a", App: "shop", Account: sqid,
+ Region: "ie", Channel: "stable", CreatedAt: time.Now(),
+ Seed: &provision.SeedSource{ID: id, Kind: seeds.KindImport, State: seeds.StateReady,
+ SHA256: strings.Repeat("a", 64), Bytes: 10, Replaces: "127.0.0.1:8401",
+ VerifyHash: strings.Repeat("b", 64), By: "t", At: time.Now()}}
+ if err := provision.CreateRecord(ctx, x.store, x.rec); err != nil {
+ t.Fatal(err)
+ }
+ // The legacy row the import replaces: an addr route to the app's own
+ // process, which the rollback must put back exactly.
+ if err := reg.Upsert(registry.Route{Host: x.rec.Host, Kind: "service", Addr: "127.0.0.1:8401", App: "shop",
+ Account: sqid, Region: "ie", Channel: "stable", Status: "ready", Unit: "shop@a.service"}); err != nil {
+ t.Fatal(err)
+ }
+ x.original = *mustRoute(t, reg, x.rec.Host)
+
+ data := release.NewDirStore(t.TempDir())
+ if _, err := data.Put(ctx, seeds.SnapshotKey(x.rec.Host, id), []byte("db"), ""); err != nil {
+ t.Fatal(err)
+ }
+ var gens uint64
+ x.cfg = provision.ReconcileConfig{Region: "ie", DataDir: t.TempDir(),
+ Layout: func(context.Context, release.Store) (string, error) { return release.LayoutModeNested, nil },
+ Seeds: &provision.SeedHooks{
+ Data: data,
+ LegacyOwners: func(context.Context) (*seeds.LegacyOwners, error) {
+ return &seeds.LegacyOwners{V: 1}, nil
+ },
+ EnvBefore: func(string, string) (string, error) { return registry.EnvBeforeAbsent, nil },
+ FenceForRollback: x.rollbackFence,
+ LiftFence: func(host string, gen uint64) { a.LiftFence(host, gen) },
+ RemoveLocal: func(registry.Route) error {
+ x.mu.Lock()
+ x.removed++
+ x.mu.Unlock()
+ return nil
+ },
+ DeleteReplica: func(context.Context, string) error {
+ x.mu.Lock()
+ x.replicas++
+ x.mu.Unlock()
+ return nil
+ },
+ NextFenceGen: func() (uint64, error) {
+ x.mu.Lock()
+ defer x.mu.Unlock()
+ gens++
+ return gens, nil
+ },
+ LocalSeedID: func(string) (string, error) { return id, nil },
+ RetireKey: func(string, string, string) error { return nil },
+ }}
+ x.pass(t) // the swap
+ if row := x.row(t); !registry.IsImportRow(*row, registry.ImportJournal{Host: x.rec.Host, SeedID: id}) {
+ t.Fatalf("the swap did not happen: %+v", row)
+ }
+ return x
+}
+
+// rollbackFence is internal/host's fenceForRollback over this Service.
+func (x *importRollback) rollbackFence(ctx context.Context, host string, gen uint64) (bool, error) {
+ if gen == 0 {
+ return false, ErrFenceGeneration
+ }
+ if x.a.FenceGeneration(host) == gen {
+ return true, nil
+ }
+ fctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+ defer cancel()
+ _, err := x.a.FenceAndStopIfUnfenced(fctx, host, gen)
+ switch {
+ case err == nil:
+ return true, nil
+ case errors.Is(err, ErrFenceHeld):
+ return false, err
+ case errors.Is(err, ErrStopUnproven) && x.a.Quiesced(host) && !x.a.StopUncertain(host):
+ return true, nil
+ }
+ x.a.LiftFence(host, gen)
+ return false, err
+}
+
+func (x *importRollback) pass(t *testing.T) {
+ t.Helper()
+ if err := provision.Reconcile(context.Background(), x.store, x.reg, x.cfg); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func (x *importRollback) row(t *testing.T) *registry.Route {
+ t.Helper()
+ return mustRoute(t, x.reg, x.rec.Host)
+}
+
+func (x *importRollback) abort(t *testing.T) {
+ t.Helper()
+ if _, err := provision.SetRecordSeed(context.Background(), x.store, x.rec.Account, x.rec.App, x.rec.Host,
+ provision.SeedAbort(x.rec.Seed.ID)); err != nil {
+ t.Fatal(err)
+ }
+}
+
+// wake starts a request's wake of the import's row in the background and
+// returns its outcome.
+func (x *importRollback) wake(t *testing.T) <-chan error {
+ t.Helper()
+ rt := *x.row(t)
+ done := make(chan error, 1)
+ go func() {
+ release, err := x.a.beginRequest(context.Background(), rt)
+ if err == nil {
+ release()
+ }
+ done <- err
+ }()
+ return done
+}
+
+// wantRestored: the original route back exactly, the import's local
+// database and replica removed once each, and no fence left.
+func (x *importRollback) wantRestored(t *testing.T) {
+ t.Helper()
+ if got := x.row(t); !reflect.DeepEqual(*got, x.original) {
+ t.Fatalf("row = %+v, want the original %+v", got, x.original)
+ }
+ if j, err := x.reg.ImportJournal(x.rec.Host); err != nil || (j != nil && j.State != registry.JournalCleanupPending) {
+ t.Fatalf("journal = %+v, %v", j, err)
+ }
+ x.mu.Lock()
+ removed := x.removed
+ x.mu.Unlock()
+ if removed != 1 {
+ t.Fatalf("the import's local database was removed %d times, want once", removed)
+ }
+ if x.a.Fenced(x.rec.Host) {
+ t.Fatal("the rollback left its fence")
+ }
+}
+
+// wantBlocked: nothing deleted, the import's row still in place with its
+// gate closed, and the journal still rolling back.
+func (x *importRollback) wantBlocked(t *testing.T) {
+ t.Helper()
+ row := x.row(t)
+ if row.ImportGate != registry.ImportGateClosed {
+ t.Fatalf("row = %+v, want the import's row with its gate closed", row)
+ }
+ if j, err := x.reg.ImportJournal(x.rec.Host); err != nil || j == nil || j.State != registry.JournalRollingBack {
+ t.Fatalf("journal = %+v, %v; want rolling-back", j, err)
+ }
+ x.mu.Lock()
+ removed, replicas := x.removed, x.replicas
+ x.mu.Unlock()
+ if removed != 0 || replicas != 0 {
+ t.Fatalf("deleted under an unproven stop: local %d, replica %d", removed, replicas)
+ }
+}
+
+func (x *importRollback) retained() bool {
+ x.a.mu.Lock()
+ defer x.a.mu.Unlock()
+ _, ok := x.a.unprovenWakes[x.rec.Host]
+ return ok
+}
+
+// reap runs the sweep's retry of retained failed wakes once and waits
+// for it to finish.
+func (x *importRollback) reap(t *testing.T) {
+ t.Helper()
+ x.a.mu.Lock()
+ u := x.a.unprovenWakes[x.rec.Host]
+ x.a.reapUnprovenWakesLocked()
+ x.a.mu.Unlock()
+ waitFor(t, "the reaper's retry to finish", func() bool {
+ x.a.mu.Lock()
+ defer x.a.mu.Unlock()
+ return u == nil || !u.busy || x.a.unprovenWakes[x.rec.Host] != u
+ })
+}
+
+// queueStart makes the next run-unit start one that systemd has queued:
+// it blocks until its context ends and comes back marked as a start root
+// may still carry out (#292).
+func (x *importRollback) queueStart(t *testing.T) {
+ t.Helper()
+ queued := make(chan struct{})
+ x.b.mu.Lock()
+ x.b.startQueued = queued
+ x.b.mu.Unlock()
+ t.Cleanup(func() { close(queued) })
+}
+
+func mustRoute(t *testing.T, reg *registry.Registry, host string) *registry.Route {
+ t.Helper()
+ rt, err := reg.Get(host)
+ if err != nil || rt == nil {
+ t.Fatalf("route %s: %+v, %v", host, rt, err)
+ }
+ return rt
+}
+
+// An import aborted while its verify wake's run-unit start is queued:
+// the rollback's fence cancels the start, #292's proof stops the unit at
+// once, and the same pass restores the original route.
+func TestImportRollbackAfterAnAbortedStartProvenAtOnce(t *testing.T) {
+ x := newImportRollback(t)
+ x.queueStart(t)
+ woke := x.wake(t)
+ waitFor(t, "the verify wake to reach its queued start", func() bool { return x.b.startCount() == 1 })
+
+ x.abort(t)
+ x.pass(t)
+ x.wantRestored(t)
+ if err := <-woke; err == nil {
+ t.Fatal("the aborted wake was served")
+ }
+ x.b.mu.Lock()
+ stops := x.b.legacyStops
+ x.b.mu.Unlock()
+ if stops == 0 {
+ t.Fatal("the queued start's unit was never stopped")
+ }
+}
+
+// The same, with #292's proof failing at first: the rollback waits, and
+// once the sweep's retry proves the unit gone, the next pass restores the
+// original route, with no agent restart in between.
+func TestImportRollbackAfterAnAbortedStartProvenLater(t *testing.T) {
+ x := newImportRollback(t)
+ x.b.mu.Lock()
+ x.b.legacyStopErr = errors.New("stop carlos-instance@a.shop.example.com: instance control socket: connection refused")
+ x.b.mu.Unlock()
+ x.queueStart(t)
+ woke := x.wake(t)
+ waitFor(t, "the verify wake to reach its queued start", func() bool { return x.b.startCount() == 1 })
+
+ x.abort(t)
+ x.pass(t)
+ if err := <-woke; err == nil {
+ t.Fatal("the aborted wake was served")
+ }
+ if !x.retained() {
+ t.Fatal("the unproven start was not retained")
+ }
+ x.wantBlocked(t)
+
+ x.b.mu.Lock()
+ x.b.legacyStopErr = nil
+ x.b.mu.Unlock()
+ x.reap(t)
+ if x.retained() {
+ t.Fatal("the proven start is still retained")
+ }
+ x.pass(t)
+ x.wantRestored(t)
+}
+
+// While #292's proof keeps failing, the rollback deletes nothing, however
+// many passes and retries run.
+func TestImportRollbackKeepsEverythingWhileTheStopIsUnproven(t *testing.T) {
+ x := newImportRollback(t)
+ x.b.mu.Lock()
+ x.b.legacyStopErr = errors.New("stop carlos-instance@a.shop.example.com: unit still active")
+ x.b.mu.Unlock()
+ x.queueStart(t)
+ woke := x.wake(t)
+ waitFor(t, "the verify wake to reach its queued start", func() bool { return x.b.startCount() == 1 })
+
+ x.abort(t)
+ x.pass(t)
+ <-woke
+ for range 3 {
+ x.reap(t)
+ x.pass(t)
+ }
+ if !x.retained() {
+ t.Fatal("an unproven start was let go")
+ }
+ x.wantBlocked(t)
+ if !x.a.StopUncertain(x.rec.Host) {
+ t.Fatal("the unproven stop reads as certain")
+ }
+}
+
+// The rollback's fence lands while the verify wake's build record is
+// being written (amadan #272): the recheck after the write refuses the
+// spawn before root is asked, so nothing started, and the same pass
+// restores the original route.
+func TestImportRollbackFencedDuringThePendingRecord(t *testing.T) {
+ x := newImportRollback(t)
+ x.b.setIdentity("v1")
+ x.abort(t)
+ rolled := make(chan error, 1)
+ var once sync.Once
+ x.a.BeforeExec = func(ctx context.Context, host, build string) error {
+ once.Do(func() {
+ // The rollback's pass runs now, while this write is in flight,
+ // and fences the host before the write returns.
+ go func() { rolled <- provision.Reconcile(context.Background(), x.store, x.reg, x.cfg) }()
+ for deadline := time.Now().Add(5 * time.Second); !x.a.Fenced(host) && time.Now().Before(deadline); {
+ time.Sleep(time.Millisecond)
+ }
+ })
+ return nil
+ }
+ woke := x.wake(t)
+ if err := <-woke; !errors.Is(err, ErrFenced) {
+ t.Fatalf("wake = %v, want ErrFenced", err)
+ }
+ if err := <-rolled; err != nil {
+ t.Fatal(err)
+ }
+ if x.b.startCount() != 0 {
+ t.Fatalf("started %d after the fence", x.b.startCount())
+ }
+ x.wantRestored(t)
+}
+
+// A launch its pre-exec gate refused is known not started; a start that
+// may have run is not, and an unrelated earlier uncertainty is kept.
+func TestALaunchRefusedBeforeExecIsNotUncertain(t *testing.T) {
+ refused := runExecGate(withExecGate(context.Background(), func() error { return ErrFenced }))
+ if !errors.Is(refused, ErrFenced) || !launchNotStarted(refused) {
+ t.Fatalf("a gate refusal = %v, not started %v", refused, launchNotStarted(refused))
+ }
+ if launchNotStarted(markLegacyStartUncertain(refused)) || launchNotStarted(markStartUncertain(refused)) {
+ t.Fatal("a start that may have run reads as not started")
+ }
+ if launchNotStarted(errors.New("start unit: root did not answer")) {
+ t.Fatal("an unmarked failure reads as not started")
+ }
+
+ a, _, _, reg := testActivator(t, time.Hour)
+ defer reg.Close()
+ var known stopObservation
+ known.notStarted()
+ a.mu.Lock()
+ a.rememberStopLocked("fresh.example.com", &known)
+ a.rememberStopLocked("other.example.com", &stopObservation{uncertain: true})
+ a.rememberStopLocked("other.example.com", &known)
+ a.mu.Unlock()
+ if a.StopUncertain("fresh.example.com") {
+ t.Fatal("a launch known not started reads as uncertain")
+ }
+ if !a.StopUncertain("other.example.com") {
+ t.Fatal("a later known stop erased an unrelated uncertainty")
+ }
+}
+
+// A retained start's proof discharges only its own uncertainty: one that
+// another source added to stays.
+func TestTheLegacyProofDischargesOnlyItsOwnUncertainty(t *testing.T) {
+ for _, other := range []bool{false, true} {
+ a, b, _, reg := testActivator(t, time.Hour)
+ host := "own.example.com"
+ u := &unprovenWake{rt: testRoute(host), legacy: true}
+ a.mu.Lock()
+ if other {
+ a.rememberStopLocked(host, &stopObservation{uncertain: true})
+ }
+ a.rememberStopLocked(host, &stopObservation{uncertain: true, owedBy: u})
+ a.rememberStopLocked(host, &stopObservation{}) // a wake refused before it did anything
+ a.unprovenWakes = map[string]*unprovenWake{host: u}
+ a.mu.Unlock()
+ a.reapUnprovenWake(host, u)
+ b.mu.Lock()
+ stops := b.legacyStops
+ b.mu.Unlock()
+ if stops != 1 {
+ t.Fatalf("other %v: %d proofs, want 1", other, stops)
+ }
+ if got := a.StopUncertain(host); got != other {
+ t.Fatalf("other %v: uncertain after the proof = %v, want %v", other, got, other)
+ }
+ reg.Close()
+ }
+}
diff --git a/internal/provision/plan14_legacy_identity_test.go b/internal/provision/plan14_legacy_identity_test.go
index 494efe7e6..cd62c94bc 100644
--- a/internal/provision/plan14_legacy_identity_test.go
+++ b/internal/provision/plan14_legacy_identity_test.go
@@ -479,8 +479,11 @@ func TestPlan14CarlosRunOverlayPreservesLegacySeal(t *testing.T) {
// Census ruling seeded-instances (Plan 14 controller, 2026-10-14). The
// overlay is loaded last, after carlos-run's. This pins what the ruling
-// writes: 98 new frozen fingerprints, 113 rows, every basis naming the
+// writes: 100 new frozen fingerprints, 115 rows, every basis naming the
// ruling, main's authority rows kept, the ruled classes, nothing removed.
+// Three rows are the pre-land fix's PROPOSED-seeded-instances-292fix
+// block, pending a ruling: Service.doWake (moved there from the ruled
+// rows) and #292's cleanupFailedWake and reapUnprovenWake.
func TestPlan14SeededInstancesOverlayPreservesLegacySeal(t *testing.T) {
root := censusRoot(t)
inv := rootReadInventory{Entries: map[string]classifiedRootRead{}, Coverage: map[string]string{}}
@@ -509,33 +512,49 @@ func TestPlan14SeededInstancesOverlayPreservesLegacySeal(t *testing.T) {
t.Fatalf("seeded-instances removed original freeze %s", hash)
}
}
- if len(inv.Legacy) != len(original)+98 {
- t.Fatalf("census ruling seeded-instances permits exactly 98 new frozen declarations, not %d", len(inv.Legacy)-len(original))
+ if len(inv.Legacy) != len(original)+100 {
+ t.Fatalf("the seeded-instances overlay adds exactly 100 frozen fingerprints, not %d", len(inv.Legacy)-len(original))
}
// Census ruling seeded-instances seal: 96 service-user and CLI rows plus
- // activationGuard and activationRule, accepted legacy (J1) as on main.
- const seal = "098cc171097a82e9ec615915a11d95f8d1c5e92372b53faef6becc48bdd75cda"
+ // activationGuard and activationRule, accepted legacy (J1) as on main,
+ // less doWake, which moved to the proposed block; and that block's
+ // three rows.
+ const seal = "680b47f44068ad0971b8de607ab2f5bda86104d6f7f537af9024d6cfe2721dd5"
if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 {
t.Fatal(errs)
}
+ proposed := map[string]bool{
+ "internal/activator/activator.go:Service.doWake": true,
+ "internal/activator/maintenance.go:Service.cleanupFailedWake": true,
+ "internal/activator/opset.go:Service.reapUnprovenWake": true,
+ }
written := 0
for key, e := range inv.Entries {
if e == before[key] {
continue
}
written++
- if !strings.Contains(e.Basis, "rev3=census-ruling-seeded-instances;") {
- t.Fatalf("%s does not name the ruling: %+v", key, e)
+ marker := "census-ruling-seeded-instances"
+ if proposed[key] {
+ // Main's rows, changed by the pre-land fix: their class is
+ // kept, pending the controller's ruling.
+ marker = "PROPOSED-seeded-instances-292fix"
+ if before[key].Class != "legacy" {
+ t.Fatalf("%s was %s on main; the fix's block keeps only legacy rows", key, before[key].Class)
+ }
+ }
+ if !strings.Contains(e.Basis, "rev3="+marker+";") {
+ t.Fatalf("%s does not name %s: %+v", key, marker, e)
}
- if e.Class == "legacy" && (e.Task != "accepted-legacy" || !strings.HasPrefix(e.Basis, "accepted-legacy;rev3=census-ruling-seeded-instances;")) {
+ if e.Class == "legacy" && (e.Task != "accepted-legacy" || !strings.HasPrefix(e.Basis, "accepted-legacy;rev3="+marker+";")) {
t.Fatalf("wrong seeded-instances disposition: %+v", e)
}
if strings.Contains(e.Basis, "NEEDS RULING") {
t.Fatalf("%s still waits for a ruling: %+v", key, e)
}
}
- if written != 113 {
- t.Fatalf("census ruling seeded-instances writes 113 rows, not %d", written)
+ if written != 115 {
+ t.Fatalf("the seeded-instances overlay writes 115 rows, not %d", written)
}
// Main's authority rows keep their class and family. rootJudgeMovesPrev
// is the body of main's rootJudgePrev under a new name and carries its
diff --git a/internal/provision/plan14_root_reads_test.go b/internal/provision/plan14_root_reads_test.go
index 41a9570d0..20a69dd95 100644
--- a/internal/provision/plan14_root_reads_test.go
+++ b/internal/provision/plan14_root_reads_test.go
@@ -727,11 +727,14 @@ func TestPlan14RootReadCensus(t *testing.T) {
// seeds floor's root readers are authority F,J4 and deny-only;
// CollectCensusSeeds and ImportJournalHosts are authority G on the
// condition, recorded in the overlay's header, that they only ever feed
- // a refusal.
+ // a refusal. Its last block, PROPOSED-seeded-instances-292fix, is NOT
+ // ruled on: the pre-land fix's three changed rows of main's units
+ // (doWake, cleanupFailedWake, reapUnprovenWake), pending the Plan 14
+ // controller.
if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-seeded-instances.tsv"), &inv, true); err != nil {
t.Fatal(err)
}
- seal = "098cc171097a82e9ec615915a11d95f8d1c5e92372b53faef6becc48bdd75cda"
+ seal = "680b47f44068ad0971b8de607ab2f5bda86104d6f7f537af9024d6cfe2721dd5"
errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...)
errs = append(errs, checkRootReadDeclarations(got, inv)...)
if len(errs) > 0 {
diff --git a/internal/provision/testdata/plan14-root-reads-seeded-instances.tsv b/internal/provision/testdata/plan14-root-reads-seeded-instances.tsv
index a3d533edd..a85c17cb2 100644
--- a/internal/provision/testdata/plan14-root-reads-seeded-instances.tsv
+++ b/internal/provision/testdata/plan14-root-reads-seeded-instances.tsv
@@ -9,8 +9,6 @@ cli.go runInstances legacy accepted-legacy accepted-legacy;rev3=census-ruling-se
cmd/console/main.go serve legacy accepted-legacy accepted-legacy;rev3=census-ruling-seeded-instances;basis=the console (cmd/console serve under console.service, user carlos-console): main's body with the seed data store opened, the scratch directory named beside CONSOLE_DB and the seeds sweep started beside the other claim workers; one more /data literal, the scratch path's default data-literal@3331,data-literal@4432,data-literal@4669 d251097a8e83707225fcbf8af44a2eeff8dcfdbe1ba52e2b1a9f22397ff7f58d
@legacy cc3892490991455d7f9b17f4de5f4112b71147908e994de4be106c214a9198a1 instances_import.go apiSession.runImport legacy accepted-legacy accepted-legacy;rev3=census-ruling-seeded-instances;basis=the carlos CLI on a person's machine, as that person (no box unit): opens and hashes the file to import, then drives the console's import door; no box state file:Open@97 d56fae6544ca06790d1d4f9c7187facddf8f2722eaaf1dfd677c1cce29233062
instances_import.go apiSession.runImport legacy accepted-legacy accepted-legacy;rev3=census-ruling-seeded-instances;basis=the carlos CLI on a person's machine, as that person (no box unit): opens and hashes the file to import, then drives the console's import door; no box state file:Open@97 d56fae6544ca06790d1d4f9c7187facddf8f2722eaaf1dfd677c1cce29233062
-@legacy acc7afd05702f832c57ed22150fa20a50e3cb603731d97ec9f9cb005c503652f internal/activator/activator.go Service.doWake legacy accepted-legacy accepted-legacy;rev3=census-ruling-seeded-instances;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): main's body with the refreshed row checked against the row the wake began from (wakeRouteChanged, twice) and the seed's first upload recorded after the wake (SeedPlaced); the agent's own lease and fence; no root decision registry:DBPath@1032,registry:DBPath@1048,registry:DBPath@1175,registry:DBPath@1186,registry:DBPath@20878,registry:Host@1014,registry:Host@1166,registry:Host@1500,registry:Host@174,registry:Host@2360,registry:Host@4335,registry:Host@861,registry:Host@936,registry:Replication@20913,registry:Seed@11684,registry:Socket@1061,registry:Socket@1077,registry:Socket@1200,registry:Socket@1211,whole:Route@10535,whole:Route@11785,whole:Route@11856,whole:Route@1395,whole:Route@1400,whole:Route@14240,whole:Route@1454,whole:Route@1460,whole:Route@1526,whole:Route@1534,whole:Route@159,whole:Route@16098,whole:Route@16744,whole:Route@18051,whole:Route@18205,whole:Route@18561,whole:Route@18655,whole:Route@19327,whole:Route@20326,whole:Route@20474,whole:Route@20987,whole:Route@21346,whole:Route@22282,whole:Route@22377,whole:Route@2250,whole:Route@3704,whole:Route@3963,whole:Route@4225,whole:Route@4230,whole:Route@4287,whole:Route@4293,whole:Route@4362,whole:Route@4370,whole:Route@5158,whole:Route@6157,whole:Route@7190,whole:Route@8071,whole:Route@959,whole:Route@9653,whole:Route@9845 9f01cda5d96268f2dc7e56dcb7c5d565e2316d9c5dc7d7dbc463fc365e53f86f
-internal/activator/activator.go Service.doWake legacy accepted-legacy accepted-legacy;rev3=census-ruling-seeded-instances;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): main's body with the refreshed row checked against the row the wake began from (wakeRouteChanged, twice) and the seed's first upload recorded after the wake (SeedPlaced); the agent's own lease and fence; no root decision registry:DBPath@1032,registry:DBPath@1048,registry:DBPath@1175,registry:DBPath@1186,registry:DBPath@20878,registry:Host@1014,registry:Host@1166,registry:Host@1500,registry:Host@174,registry:Host@2360,registry:Host@4335,registry:Host@861,registry:Host@936,registry:Replication@20913,registry:Seed@11684,registry:Socket@1061,registry:Socket@1077,registry:Socket@1200,registry:Socket@1211,whole:Route@10535,whole:Route@11785,whole:Route@11856,whole:Route@1395,whole:Route@1400,whole:Route@14240,whole:Route@1454,whole:Route@1460,whole:Route@1526,whole:Route@1534,whole:Route@159,whole:Route@16098,whole:Route@16744,whole:Route@18051,whole:Route@18205,whole:Route@18561,whole:Route@18655,whole:Route@19327,whole:Route@20326,whole:Route@20474,whole:Route@20987,whole:Route@21346,whole:Route@22282,whole:Route@22377,whole:Route@2250,whole:Route@3704,whole:Route@3963,whole:Route@4225,whole:Route@4230,whole:Route@4287,whole:Route@4293,whole:Route@4362,whole:Route@4370,whole:Route@5158,whole:Route@6157,whole:Route@7190,whole:Route@8071,whole:Route@959,whole:Route@9653,whole:Route@9845 9f01cda5d96268f2dc7e56dcb7c5d565e2316d9c5dc7d7dbc463fc365e53f86f
@legacy ef86ad92b0b3eadd2c0db51cca8e18c799e3e16a50223329ce192b50b0043cac internal/activator/backend_exec.go ExecBackend.Restore legacy accepted-legacy accepted-legacy;rev3=census-ruling-seeded-instances;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): now a dispatcher: it parses the row's seed column and hands a seed to place to seededRestore and every other row to restorePlain, which holds its former body registry:SeedSource@118,whole:Route@227,whole:Route@267 2b5994c3dedaf2550235db47939a2758c3ee837a0a64fe73b9cfa535aed6faa7
internal/activator/backend_exec.go ExecBackend.Restore legacy accepted-legacy accepted-legacy;rev3=census-ruling-seeded-instances;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): now a dispatcher: it parses the row's seed column and hands a seed to place to seededRestore and every other row to restorePlain, which holds its former body registry:SeedSource@118,whole:Route@227,whole:Route@267 2b5994c3dedaf2550235db47939a2758c3ee837a0a64fe73b9cfa535aed6faa7
@legacy 35ec1caef61944ea954c25b4b12d427c6a9fbb9de15e4aba3780e076dbf09e8f internal/activator/backend_exec.go ExecBackend.restorePlain legacy accepted-legacy accepted-legacy;rev3=census-ruling-seeded-instances;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): body of the frozen ExecBackend.Restore, which is now its dispatcher, with the seed marker reconciled before the warm-cache check; same staging fallback and replica restore file:Stat@540,registry:Host@108,whole:Route@128,whole:Route@397,whole:Route@477 a0c39966251e7a2e74b422fb4d53294c3cd63b73d31277d10d7d5a529b4a7cd7
@@ -215,7 +213,7 @@ internal/schedule/runner.go Runner.Tick legacy accepted-legacy accepted-legacy;r
@fields A,Account,Accounts,AddedAt,Addr,Addrs,Alias,Aliases,Alive,App,Archives,Args,At,Attempts,B,Backing,Bucket,BuildState,Callback,CallbackError,CallbackFailed,Canary,Caps,CertExpiresAt,Changed,Channel,CheckedAt,ClaimIncarnation,ClaimOwner,ClaimOwnership,ClaimRowID,ClaimStamp,Class,Contents,CreatedAt,DBPath,DNSCheckedAt,DNSStatus,Deadline,Demand,Digest,DoneBefore,Due,DueAt,Env,EnvBefore,Environment,Error,ErrorKind,ExcludeHosts,ExpectGood,ExpectOp,Expected,F,Failed,FenceGen,Fleet,FleetBox,FleetBoxes,Fleets,From,FromDB,FromOp,FromSock,Gate,GoodDemand,GoodDigest,GoodVersion,Hash,Health,Held,Hibernate,HoldOp,Host,Hosts,ID,Id,IdValue,Idle,IdleSeconds,ImportGate,Journal,Keep,Kind,Label,LastAt,LastError,LastMs,LastStatus,Limit,MaxRun,MaxRunSeconds,Method,Ms,Name,Next,NextAt,NextDue,NextGood,NextValue,Now,Number,Number64,Observed,ObservedValue,Old,On,Op,Original,Owner,Ownership,P,Path,Paths,Platform,Pool,Prefix,Prev,PrevVersion,Provisioned,RecordKey,Region,Registration,Replica,Replicas,Replication,Retirement,Route,Routes,Row,Rows,Seed,SeedID,SeedSource,ServingVersion,Slug,Socket,Spec,Sqid,State,Status,StatusNamed,Steering,StringMap,T,TValue,Tags,Target,Targets,Text,Timer,Timers,Times,To,ToDB,ToOp,ToSock,TokenHash,Tunnel,Unit,UpdateSettings,Ver,Version,W,WakeIncarnation,Wildcards,closed,conn,connect,ctx,d,dec,done,enc,full,input,mu,output,path,remote,sql,wait
@coverage . 1e94e1644297311bfaeec977a479d454eb39c6ab51e7cdd57f2dbc452cd92033
@coverage cmd/console 0554957d7bc31eb4553b6a98c7481738d11bfd4fb809277bf7e0cc723422ad2e
-@coverage internal/activator 12069bbe8bdc62e80b690e3f362b948381324c08dcaf66a95af6aef2798a7a6b
+@coverage internal/activator a7faaf34a27db6a4603c007fd8ab7676fd582a106518144b467ac3c9cae88e35
@coverage internal/console 2e39e3ffb055cf7c230c7d231e7d8adcf0f8543222eac015f610e920f91c217b
@coverage internal/dbdoor 0109cd1cd4f0ca14d57bc6562166424273f770d8261e60b738c61f9a6d080dd9
@coverage internal/domains 3c7159e24bbb66faec61bc32d56019501fb19c072f10eacca9fd920c3cb7c0d6
@@ -232,3 +230,10 @@ internal/schedule/runner.go Runner.Tick legacy accepted-legacy accepted-legacy;r
@coverage internal/release 55c36a3e2a1b32a8736c3f0404a8e6442d747cd5ecbde68a0e8b3364e6fe3324
@coverage internal/schedule 295d28a90ab322dd05781d30bde48bffcf8e8eb49884460c02a1b559bf72259b
@coverage internal/seeds 433f3cc6c1b885bf7b95c4530fc1af78e5d3b64b578a7dee919797afbdc09967
+# PROPOSED-seeded-instances-292fix: pending the Plan 14 controller's ruling. The pre-land fix for the seeded review's P2 (a failed import wake's stop kept uncertain after it is known) changes three units main rows under census ruling 292 and census ruling owed, which census-ruling-seeded-instances does not cover. Each keeps its current class, accepted legacy; nothing is reclassified. Service.doWake's row moves here from the seeded rows above, since its body now carries both.
+@legacy 3011833f100e4a0b582906168dca2f04b635adebf93e8857f2f1d3a9bce36a6e internal/activator/activator.go Service.doWake legacy accepted-legacy accepted-legacy;rev3=PROPOSED-seeded-instances-292fix;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): main's body (census ruling 292's legacy start mark and lease give-back; census ruling owed's pending record) with the seeded change ruled under census-ruling-seeded-instances (the refreshed row checked against the row the wake began from, twice; the seed's first upload recorded after the wake), and now one more line on a failed start: a start its pre-exec gate refused (launchNotStarted: the exec gate, the pending record or its recheck answered no before root was asked) is recorded as known not started (stopObservation.notStarted) instead of uncertain. Read sites unchanged; no new read and no root decision registry:DBPath@1032,registry:DBPath@1048,registry:DBPath@1175,registry:DBPath@1186,registry:DBPath@21121,registry:Host@1014,registry:Host@1166,registry:Host@1500,registry:Host@174,registry:Host@2360,registry:Host@4335,registry:Host@861,registry:Host@936,registry:Replication@21156,registry:Seed@11684,registry:Socket@1061,registry:Socket@1077,registry:Socket@1200,registry:Socket@1211,whole:Route@10535,whole:Route@11785,whole:Route@11856,whole:Route@1395,whole:Route@1400,whole:Route@14240,whole:Route@1454,whole:Route@1460,whole:Route@1526,whole:Route@1534,whole:Route@159,whole:Route@16098,whole:Route@16744,whole:Route@18051,whole:Route@18205,whole:Route@18561,whole:Route@18655,whole:Route@19327,whole:Route@20569,whole:Route@20717,whole:Route@21230,whole:Route@21589,whole:Route@2250,whole:Route@22525,whole:Route@22620,whole:Route@3704,whole:Route@3963,whole:Route@4225,whole:Route@4230,whole:Route@4287,whole:Route@4293,whole:Route@4362,whole:Route@4370,whole:Route@5158,whole:Route@6157,whole:Route@7190,whole:Route@8071,whole:Route@959,whole:Route@9653,whole:Route@9845 f6ac517702861739881d3561b21e759f21deb7038676edd7e411d440205d6471
+internal/activator/activator.go Service.doWake legacy accepted-legacy accepted-legacy;rev3=PROPOSED-seeded-instances-292fix;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): main's body (census ruling 292's legacy start mark and lease give-back; census ruling owed's pending record) with the seeded change ruled under census-ruling-seeded-instances (the refreshed row checked against the row the wake began from, twice; the seed's first upload recorded after the wake), and now one more line on a failed start: a start its pre-exec gate refused (launchNotStarted: the exec gate, the pending record or its recheck answered no before root was asked) is recorded as known not started (stopObservation.notStarted) instead of uncertain. Read sites unchanged; no new read and no root decision registry:DBPath@1032,registry:DBPath@1048,registry:DBPath@1175,registry:DBPath@1186,registry:DBPath@21121,registry:Host@1014,registry:Host@1166,registry:Host@1500,registry:Host@174,registry:Host@2360,registry:Host@4335,registry:Host@861,registry:Host@936,registry:Replication@21156,registry:Seed@11684,registry:Socket@1061,registry:Socket@1077,registry:Socket@1200,registry:Socket@1211,whole:Route@10535,whole:Route@11785,whole:Route@11856,whole:Route@1395,whole:Route@1400,whole:Route@14240,whole:Route@1454,whole:Route@1460,whole:Route@1526,whole:Route@1534,whole:Route@159,whole:Route@16098,whole:Route@16744,whole:Route@18051,whole:Route@18205,whole:Route@18561,whole:Route@18655,whole:Route@19327,whole:Route@20569,whole:Route@20717,whole:Route@21230,whole:Route@21589,whole:Route@2250,whole:Route@22525,whole:Route@22620,whole:Route@3704,whole:Route@3963,whole:Route@4225,whole:Route@4230,whole:Route@4287,whole:Route@4293,whole:Route@4362,whole:Route@4370,whole:Route@5158,whole:Route@6157,whole:Route@7190,whole:Route@8071,whole:Route@959,whole:Route@9653,whole:Route@9845 f6ac517702861739881d3561b21e759f21deb7038676edd7e411d440205d6471
+@legacy ce713e012707529182b59431a9e24c4a1c96b06c6ceeb408aa3ff800d32ff99e internal/activator/maintenance.go Service.cleanupFailedWake legacy accepted-legacy accepted-legacy;rev3=PROPOSED-seeded-instances-292fix;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): census ruling 292's body; when #292's whole proof holds for an aborted legacy start (legacyRunGone: the unit stopped and its cgroup empty) this wake's uncertainty is discharged and the stages it never reached (no sidecar, no replica) are recorded as held; when the proof fails the start is retained exactly as before, now named as the owner of this wake's uncertainty (owedBy). Read sites unchanged; the proof still only decides release-or-keep; no new read and no root decision registry:Host@1824,registry:Host@2337,registry:Host@3288,registry:Host@3628,registry:Host@4240,whole:Route@1691,whole:Route@1930,whole:Route@1934,whole:Route@2970,whole:Route@3653,whole:Route@3657 f59321652db8dc67a25ffabf3589b727b12b25d1136b2b18c48f97d6929d93b0
+internal/activator/maintenance.go Service.cleanupFailedWake legacy accepted-legacy accepted-legacy;rev3=PROPOSED-seeded-instances-292fix;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): census ruling 292's body; when #292's whole proof holds for an aborted legacy start (legacyRunGone: the unit stopped and its cgroup empty) this wake's uncertainty is discharged and the stages it never reached (no sidecar, no replica) are recorded as held; when the proof fails the start is retained exactly as before, now named as the owner of this wake's uncertainty (owedBy). Read sites unchanged; the proof still only decides release-or-keep; no new read and no root decision registry:Host@1824,registry:Host@2337,registry:Host@3288,registry:Host@3628,registry:Host@4240,whole:Route@1691,whole:Route@1930,whole:Route@1934,whole:Route@2970,whole:Route@3653,whole:Route@3657 f59321652db8dc67a25ffabf3589b727b12b25d1136b2b18c48f97d6929d93b0
+@legacy 1cfba7fa3532305b054f166162cde4ba939462488d3a995b07aa10ed47facee8 internal/activator/opset.go Service.reapUnprovenWake legacy accepted-legacy accepted-legacy;rev3=PROPOSED-seeded-instances-292fix;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): census ruling 292's body; when the retried proof holds for a retained legacy start and the host's stop history is uncertain only because of that start (owedBy), the uncertainty is discharged and the stages the failed wake never reached are recorded as held; any other uncertainty is kept through rememberStopLocked as before. Read sites unchanged; no new read and no root decision whole:Route@439,whole:Route@493 7484890de454d363b9e191d28b1c70f8ea6339492169931c2f47e0dfb02db852
+internal/activator/opset.go Service.reapUnprovenWake legacy accepted-legacy accepted-legacy;rev3=PROPOSED-seeded-instances-292fix;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): census ruling 292's body; when the retried proof holds for a retained legacy start and the host's stop history is uncertain only because of that start (owedBy), the uncertainty is discharged and the stages the failed wake never reached are recorded as held; any other uncertainty is kept through rememberStopLocked as before. Read sites unchanged; no new read and no root decision whole:Route@439,whole:Route@493 7484890de454d363b9e191d28b1c70f8ea6339492169931c2f47e0dfb02db852