amadan

carlos / platform Public

fix-shutdown-lease-292

Squashed from fix-shutdown-lease-292 into main.
claude e15ad6be618aeea85b19b5b8cab42df6d75a8af8 parent fc70bc7
23 files changed, +2163 −94
  • README.md +6 −0
  • docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md +64 −20
  • internal/activator/activator.go +230 −44
  • internal/activator/activator_test.go +114 −5
  • internal/activator/backend_exec.go +68 −1
  • internal/activator/backend_legacy_test.go +108 −0
  • internal/activator/backend_unit.go +15 −0
  • internal/activator/containment.go +82 −0
  • internal/activator/maintenance.go +27 −2
  • internal/activator/opset.go +21 −6
  • internal/activator/opset_test.go +27 −12
  • internal/activator/shutdown_lease_test.go +470 −0
  • internal/activator/testdata/swapsim/mutate.sh +1 −1
  • internal/instancectl/legacystart.go +153 −0
  • internal/instancectl/legacystart_live_test.go +215 −0
  • internal/instancectl/legacystart_test.go +114 −0
  • internal/instancectl/server.go +50 −1
  • internal/instancectl/server_test.go +19 −1
  • internal/lease/lease.go +69 −1
  • internal/lease/release_retry_test.go +193 −0
  • internal/provision/plan14_legacy_identity_test.go +77 −0
  • internal/provision/plan14_root_reads_test.go +10 −0
  • internal/provision/testdata/plan14-root-reads-292.tsv +30 −0

CI passed — run details

CI log 
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.012s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	355.219s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.085s
ok  	github.com/carlosframework/platform/cmd/moveprobe	93.309s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.707s
ok  	github.com/carlosframework/platform/internal/activator	238.250s
ok  	github.com/carlosframework/platform/internal/agentsock	0.020s
ok  	github.com/carlosframework/platform/internal/alerting	0.003s
ok  	github.com/carlosframework/platform/internal/analytics	0.019s
ok  	github.com/carlosframework/platform/internal/appconfig	2.406s
ok  	github.com/carlosframework/platform/internal/apperr	0.077s
ok  	github.com/carlosframework/platform/internal/applog	0.131s
ok  	github.com/carlosframework/platform/internal/awsinv	0.005s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.005s
ok  	github.com/carlosframework/platform/internal/boxid	0.919s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.008s
ok  	github.com/carlosframework/platform/internal/caps	0.013s
ok  	github.com/carlosframework/platform/internal/certwarm	0.002s
ok  	github.com/carlosframework/platform/internal/cgroup	0.702s
ok  	github.com/carlosframework/platform/internal/changelog	0.325s
ok  	github.com/carlosframework/platform/internal/clamd	0.054s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	368.330s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.520s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.010s
ok  	github.com/carlosframework/platform/internal/dbverify	0.004s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.003s
ok  	github.com/carlosframework/platform/internal/docsite	0.005s
ok  	github.com/carlosframework/platform/internal/domains	2.215s
ok  	github.com/carlosframework/platform/internal/doorkey	0.006s
ok  	github.com/carlosframework/platform/internal/economics	0.004s
ok  	github.com/carlosframework/platform/internal/edge	26.069s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.008s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.123s
ok  	github.com/carlosframework/platform/internal/gate	1.672s
ok  	github.com/carlosframework/platform/internal/genselect	0.015s
ok  	github.com/carlosframework/platform/internal/geo	0.015s
ok  	github.com/carlosframework/platform/internal/geoip	0.030s
ok  	github.com/carlosframework/platform/internal/host	128.379s
ok  	github.com/carlosframework/platform/internal/hostactors	0.371s
ok  	github.com/carlosframework/platform/internal/hostbox	2.268s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.489s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.010s
ok  	github.com/carlosframework/platform/internal/hostexec	4.873s
ok  	github.com/carlosframework/platform/internal/hostfence	0.032s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.475s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.012s
ok  	github.com/carlosframework/platform/internal/hostguard	1.003s
ok  	github.com/carlosframework/platform/internal/hostops	1.453s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.011s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.009s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.451s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.144s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.011s
ok  	github.com/carlosframework/platform/internal/infraview	0.009s
ok  	github.com/carlosframework/platform/internal/instancectl	3.490s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.718s
ok  	github.com/carlosframework/platform/internal/instancefs	0.016s
ok  	github.com/carlosframework/platform/internal/instancekey	0.012s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.017s
ok  	github.com/carlosframework/platform/internal/layout	1.455s
ok  	github.com/carlosframework/platform/internal/lease	9.813s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.017s
ok  	github.com/carlosframework/platform/internal/ledger	0.010s
ok  	github.com/carlosframework/platform/internal/mail	0.171s
ok  	github.com/carlosframework/platform/internal/maintenance	0.078s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.586s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.071s
ok  	github.com/carlosframework/platform/internal/metrics	0.004s
ok  	github.com/carlosframework/platform/internal/moves	21.102s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.760s
ok  	github.com/carlosframework/platform/internal/objstore	0.180s
ok  	github.com/carlosframework/platform/internal/ops	0.169s
ok  	github.com/carlosframework/platform/internal/park	0.196s
ok  	github.com/carlosframework/platform/internal/privatefile	0.016s
ok  	github.com/carlosframework/platform/internal/provision	13.160s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.009s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	4.048s
ok  	github.com/carlosframework/platform/internal/relay	7.237s
ok  	github.com/carlosframework/platform/internal/release	19.368s
ok  	github.com/carlosframework/platform/internal/rollout	3.755s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.003s
ok  	github.com/carlosframework/platform/internal/rootaws	0.007s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.008s
ok  	github.com/carlosframework/platform/internal/rootstate	0.027s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.004s
ok  	github.com/carlosframework/platform/internal/schedule	2.296s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	9.719s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.943s
ok  	github.com/carlosframework/platform/internal/status	0.039s
ok  	github.com/carlosframework/platform/internal/steering	0.005s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.210s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.002s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.004s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.531s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: cached, digest verified
  mc: cached, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.024s
=== RUN   TestT4StorageProbeMinio
2026/10/14 17:15:04 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DE74C832035733, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.022s
minio contract: PASS
diff --git a/README.md b/README.md
index 492ee8361..941764166 100644
--- a/README.md
+++ b/README.md
@@ -352,6 +352,12 @@ The first release that understands app regions is v0.23.0.
6. Turn the switch on. Re-run step 5 after any period with the switch off,
and whenever a new region becomes eligible.
+**#292: instance-control changes.** The release carrying #292 changes
+`carlos-instance-control`: it orders a host's legacy run starts against
+its stops. So the service must restart on roll. A box whose updater is
+v0.26.0 or later does that itself, so no manual step is needed; an older
+updater needs the hand restart described under "Old updaters" above.
+
### Tenant email senders, plan 2 of 5 (2026-09-21)
This release turns the one implicit per-app email sender into an explicit
diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md
index 996bfe31c..48548fc37 100644
--- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md
+++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md
@@ -18,8 +18,8 @@ plans 1 to 6, in `platform-design-zero-downtime-deploys`.
- **AWS profiles.** Assumed-role profiles need the Bash sandbox off for that
command. Dev is profile `kass`. The flagship is profile `keymail`.
- **Order.** Boxes first, then the console's swap-policy settings, then the
- flag: dev, then `bab/adopt-probe` on the flagship, then the rest of the
- flagship. The deploy docs and release notes carry the migrations rule
+ flag: dev, then one flagship box with a fresh probe app (Gate F), then the
+ rest of the flagship. The deploy docs and release notes carry the migrations rule
before the flag goes on across the fleet. Plan 5 keeps the flag off
outside dev until this plan has merged.
- **The flag.** `CARLOS_SWAP=overlap` in a box's `/etc/carlos/host.env`. Only
@@ -1049,6 +1049,15 @@ first. The unit is `carlos-edge` and it loads `/etc/carlos/host.env`
Restarting `carlos-edge` stops every instance on the box (spec section 10).
Each wakes on its next request.
+To revert a box, remove the `CARLOS_SWAP=` line from `/etc/carlos/host.env`
+and restart `carlos-edge`. Then remove the last generation's leftover `g<n>/`
+directory under each instance's run directory,
+`/run/carlos/instances/<host>/g<n>/`. Dev round 5 (#272) found that the
+flag-off restart leaves it behind with a stale `app.sock`. Nothing uses a
+`g<n>/` directory with the flag off, so it stays until a reboot. List what is
+left with `ls -d /run/carlos/instances/*/g[0-9]*/` after the restart, and
+remove each one.
+
Then run the probe below against a dev app. Create one with:
```bash
@@ -1071,19 +1080,45 @@ dash means the box did not arm (see the dash note above).
## Gate F: the flagship probe box
-Needs Paul's approval, naming the box and its region. The approval also
-names the probe's app actions: flipping `bab/adopt-probe`'s swap policy to
-`hold` and back, and four deploys of it. **Do not start this gate while the
-"Before Gate E: owed" list has an open item that Paul has not waived.**
-
-Find the probe's region with:
+**Status: passed on annie-welding (`ie`) on 2026-10-14 (#272). annie stays
+armed.** The first attempt, on 2026-10-13, was reverted. The retry ran the
+probe against a fresh app, as below. Its arm restart left one tenant's lease
+held for 15 minutes; that is the edge restart, not the flag (#292).
-```bash
-carlos instances list --app adopt-probe --account bab --console https://console.carloku.com
-```
-
-(REGION column; the instance is `probe1.bab.oncarlos.com`.) The flagship
-boxes, profile `keymail`:
+Needs Paul's approval, naming the box and its region. The approval also
+names the probe's app actions: creating a throwaway app in `bab` with one
+instance on that box, flipping its swap policy to `hold` and back, four
+deploys of it, and deleting it afterwards. **Do not start this gate while
+the "Before Gate E: owed" list has an open item that Paul has not waived.**
+
+**Use a fresh app with no program history, not `bab/adopt-probe`.**
+adopt-probe's recorded program is `adoptprobe`, so root refuses a zddprobe
+build for it as the wrong program unless that build is shipped with
+`carlos ship --program-change`, which `carlos deploy` cannot do. That
+refusal, not a slow root pass, is what stopped zdd-1 in the first Gate F:
+root logged `wrong program` and restored the old build, and the watch timed
+out waiting for an install that would never come.
+
+The 2026-10-14 retry set the app up like this:
+
+1. Create the app in `bab` through the console's New app form, with no
+ region. It was `bab/zdd-probe`.
+2. An operator runs `carlos instances enable --app zdd-probe --account bab`,
+ then creates one instance:
+
+ ```bash
+ carlos instances create --app zdd-probe --account bab --host zddprobe1.bab.oncarlos.com --console https://console.carloku.com
+ ```
+
+3. Check it landed on the gate's box: `carlos instances list --app zdd-probe
+ --account bab --console https://console.carloku.com` shows that box's
+ region (the retry's was "Ireland · Dublin"), and only that box's edges
+ file lists the host.
+4. A new host may not resolve yet. While DNS lags, point the load at the
+ box's public IPv4 with `RESOLVE` (see "The probe"):
+ `RESOLVE=zddprobe1.bab.oncarlos.com:443:<box IP>`.
+
+The flagship boxes, profile `keymail`:
| box | instance id | region |
| --- | --- | --- |
@@ -1099,9 +1134,16 @@ Run Gate E's flag command against that one box with `--profile keymail` and
its region. Then run the probe with:
```bash
-APP=adopt-probe ACCOUNT=bab CONSOLE=https://console.carloku.com HOST=probe1.bab.oncarlos.com
+APP=zdd-probe ACCOUNT=bab CONSOLE=https://console.carloku.com HOST=zddprobe1.bab.oncarlos.com
+RESOLVE=zddprobe1.bab.oncarlos.com:443:<box IP> # only while DNS lags
```
+Afterwards, delete the app: `carlos apps delete zdd-probe --account bab
+--console https://console.carloku.com`. On 2026-10-14 the retry then removed
+the instance's leftovers on annie by hand (its instance, builds, links, data
+and `/data/programs/` file) and its bucket object, and checked that no
+`g<n>/` directory was left. An empty `carlos-instances-zddprobe1…slice` stayed.
+
## Gate G: the rest of the flagship
Needs Paul's approval, naming each box before it is done. **Do not start
@@ -1116,8 +1158,8 @@ is Tito's operator's call; do not touch Tito's boxes from here.
## The probe
Spec section 12: zero failed requests under continuous load, and no request
-held longer than a normal response. Run it at Gate E (dev) and Gate F
-(`adopt-probe`).
+held longer than a normal response. Run it at Gate E (dev) and Gate F (a
+fresh app on the flagship box, never `adopt-probe`; see Gate F for why).
### 1. Build the probe app, in scratch
@@ -1142,7 +1184,8 @@ done
### 2. A load phase
Four request loops for the given seconds, cache-busted, logging status and
-time:
+time. With `RESOLVE` set (`<host>:443:<box IP>`), curl skips DNS and goes
+straight to that box:
```bash
load() { # usage: load <phase> <secs>
@@ -1150,7 +1193,7 @@ load() { # usage: load <phase> <secs>
for i in 1 2 3 4; do
( end=$((SECONDS+$2)); n=0
while [ $SECONDS -lt $end ]; do n=$((n+1))
- curl -s -o /dev/null -m 35 -w '%{http_code} %{time_total}\n' "https://$HOST/?zdd=$1-$i-$n" >> "$TMPDIR/zdd-probe/$1/load-$i.log"
+ curl -s ${RESOLVE:+--resolve "$RESOLVE"} -o /dev/null -m 35 -w '%{http_code} %{time_total}\n' "https://$HOST/?zdd=$1-$i-$n" >> "$TMPDIR/zdd-probe/$1/load-$i.log"
done ) &
done
wait
@@ -1199,7 +1242,8 @@ rerun the hold phase once with the policy already set.
rm -rf "$TMPDIR/zdd-probe"
```
-On dev, run `carlos apps delete zdd-probe` once Paul is done with it.
+On dev, run `carlos apps delete zdd-probe` once Paul is done with it. On
+the flagship, delete the Gate F app as Gate F says.
## Names checked against this branch
diff --git a/internal/activator/activator.go b/internal/activator/activator.go
index a3d9866a0..236a41ab7 100644
--- a/internal/activator/activator.go
+++ b/internal/activator/activator.go
@@ -75,6 +75,7 @@ import (
"github.com/carlosframework/platform/internal/hostclaim"
"github.com/carlosframework/platform/internal/hostwrite"
"github.com/carlosframework/platform/internal/instancectl"
+ "github.com/carlosframework/platform/internal/lease"
"github.com/carlosframework/platform/internal/registry"
"github.com/carlosframework/platform/internal/release"
)
@@ -253,6 +254,95 @@ func releaseLease(ctx context.Context, lease Lease) error {
return lease.Release(rctx)
}
+// releaseRetrier is the OPTIONAL half of Lease that a failed Release
+// leaves something to retry with (#292, Astra P3): lease.Held.Release
+// spends its handle even when the DELETE fails, so the same conditional
+// delete is re-issued through RetryRelease. execLease has it.
+type releaseRetrier interface {
+ RetryRelease(ctx context.Context) error
+}
+
+// owedRelease is a lease whose release failed: kept, and retried by
+// finishPendingStops (the sweep, and Shutdown's polling) until it lands or
+// Shutdown's budget ends. busy marks a retry in flight; lastLog paces the
+// failure lines. Both under a.mu.
+type owedRelease struct {
+ host string
+ lease releaseRetrier
+ busy bool
+ lastLog time.Time
+}
+
+// giveBack is releaseLease for a lease the agent holds for host, with a
+// failure kept as an obligation instead of forgotten (#292, Astra P3).
+// It reports whether the release landed now.
+func (a *Service) giveBack(ctx context.Context, host string, lease Lease) bool {
+ err := releaseLease(ctx, lease)
+ if err != nil {
+ a.oweRelease(host, lease, err)
+ }
+ return err == nil
+}
+
+// oweRelease records a failed release of host's lease for retry. A lease
+// that cannot retry is left to its TTL, as before. Never called with a.mu
+// held.
+func (a *Service) oweRelease(host string, lease Lease, err error) {
+ r, ok := lease.(releaseRetrier)
+ if !ok {
+ log.Printf("activator: %s: giving the lease back failed: %v (the lease's TTL frees it)", host, err)
+ return
+ }
+ log.Printf("activator: %s: giving the lease back failed: %v; retrying it", host, err)
+ a.mu.Lock()
+ if a.owedReleases == nil {
+ a.owedReleases = make(map[*owedRelease]struct{})
+ }
+ a.owedReleases[&owedRelease{host: host, lease: r, lastLog: time.Now()}] = struct{}{}
+ a.mu.Unlock()
+}
+
+// retryOwedReleasesLocked starts one retry of every owed release that has
+// none in flight. The caller holds a.mu.
+func (a *Service) retryOwedReleasesLocked() {
+ for o := range a.owedReleases {
+ if o.busy {
+ continue
+ }
+ o.busy = true
+ go a.retryOwedRelease(o)
+ }
+}
+
+// retryOwedRelease is one conditional retry, under the release budget of
+// its own. Landed, or no longer ours to release, the obligation goes.
+func (a *Service) retryOwedRelease(o *owedRelease) {
+ ctx, cancel := context.WithTimeout(context.Background(), leaseReleaseBudget)
+ err := o.lease.RetryRelease(ctx)
+ cancel()
+ // A record past its own expiry is retired even though the delete still
+ // fails (lease.ErrReleaseLapsed): the lease is free to any acquirer by
+ // then, and a persistent refusal must not keep obligations forever.
+ lapsed := errors.Is(err, lease.ErrReleaseLapsed)
+ a.mu.Lock()
+ o.busy = false
+ logIt := err == nil || lapsed || time.Since(o.lastLog) >= 30*time.Second
+ if err == nil || lapsed {
+ delete(a.owedReleases, o)
+ } else if logIt {
+ o.lastLog = time.Now()
+ }
+ a.mu.Unlock()
+ switch {
+ case err == nil:
+ log.Printf("activator: %s: lease given back on retry", o.host)
+ case lapsed:
+ log.Printf("activator: %s: lease release retired, never deleted: %v", o.host, err)
+ case logIt:
+ log.Printf("activator: %s: retrying the lease's release: %v", o.host, err)
+ }
+}
+
// leaseHealth is the OPTIONAL half of Lease: a held lease that can say
// whether it is still held (no failed renewal, not spent). execLease has it
// (lease.Held.Err). A hold's pinned wake asks it at its exec, before the
@@ -1010,6 +1100,10 @@ const configRefusedRetryAfter = 30
var errWakeTimeout = errors.New("timed out waiting for instance wake")
var errStartupStopped = errors.New("activator stopped during startup")
+// errShuttingDown refuses a request once Shutdown has begun (#292): nothing
+// wakes, and nothing is recycled, after the agent has started going away.
+var errShuttingDown = errors.New("activator: the agent is shutting down")
+
// Service is the activator (Eleven's activatorSvc). It structurally satisfies
// the edge package's Activator interface — the edge never imports this
// package: the dependency points activator→edge only (backend_exec.go takes
@@ -1060,7 +1154,16 @@ type Service struct {
// shuttingDown is set by Shutdown before it stops anything, and read by
// the supervised-respawn goroutines: the agent is going away, so nothing
// new may be spawned against a DB whose lease is about to be released.
+ // begin, WakeWorker, wake and the sweep read it too, with swaps on or
+ // off (#292): once it is set nothing wakes, and an exited child is
+ // Shutdown's to stop, never recycled as an unexpected exit.
shuttingDown bool
+ // shutdownCtx is Shutdown's budget, set with shuttingDown: a legacy
+ // proof runs under the sooner of its own budget and this (#292).
+ shutdownCtx context.Context
+ // proofCancels are the legacy proofs running now (legacyRunGone), so
+ // Shutdown can end them with its budget. Under a.mu.
+ proofCancels map[*byte]context.CancelFunc
// convergeBudget is how many build-convergence cycles this sweep may
// still start: reset to maxConvergeCyclesPerSweep at the top of every
@@ -1099,6 +1202,9 @@ type Service struct {
// the host's next wake until the sweep proves it
// (reapUnprovenWakesLocked). Under a.mu.
unprovenWakes map[string]*unprovenWake
+ // owedReleases are lease releases that failed and are retried until
+ // they land (giveBack, #292). Under a.mu.
+ owedReleases map[*owedRelease]struct{}
// outcomeFences are planned-wake outcomes taken but not yet delivered
// to OnGen, per host, made lazily (final review F3): a failed or
// fallen-back first wake's, from before its entry goes until afterWake's
@@ -1343,6 +1449,17 @@ func (a *Service) begin(ctx context.Context, rt registry.Route, stamp bool) (rel
a.mu.Unlock()
return nil, ErrFenced
}
+ // Once Shutdown has begun, a request wakes nothing and recycles
+ // nothing (#292), WakeWorker's rule. Checked on every pass of the
+ // loop, not only at entry: a request queued on a stop or a wake
+ // before the shutdown began would otherwise come round and wake the
+ // host afresh, taking a lease Shutdown never sees. An exited child
+ // found here is left for Shutdown's own stop, not read as the
+ // build's failure.
+ if a.shuttingDown {
+ a.mu.Unlock()
+ return nil, errShuttingDown
+ }
e := a.entries[rt.Host]
if e == nil {
// Asleep → kick off ONE wake; this request and any latecomers all
@@ -1486,7 +1603,9 @@ func (a *Service) BeginSidecarRequest(ctx context.Context, rt registry.Route) er
return ErrFenced
}
e := a.entries[rt.Host]
- if e == nil {
+ if e == nil || a.shuttingDown {
+ // Shutting down: the stop is Shutdown's, and an exited child is
+ // not recycled here (#292).
a.mu.Unlock()
return errSidecarAsleep
}
@@ -1603,7 +1722,10 @@ func (a *Service) wake(e *actEntry) {
// Real time, not a.now: it anchors a first wake's real health deadline
// (spawnPlanned, doClaimWake).
e.wakeStart = time.Now()
- if a.fences[e.host] != 0 || (a.gens && a.shuttingDown) {
+ // A wake that starts after Shutdown began is cancelled before its
+ // lease, whatever CARLOS_SWAP says (#292): with swaps off it used to run
+ // on, and take a lease the exiting agent never gave back.
+ if a.fences[e.host] != 0 || a.shuttingDown {
cancel()
}
a.mu.Unlock()
@@ -1755,7 +1877,7 @@ func (a *Service) wake(e *actEntry) {
// already reachable. The fence and bounce checks below come after
// it, so a stop they begin cancels it.
a.armFirstWatchLocked(e, w)
- if a.fences[e.host] != 0 || (a.gens && a.shuttingDown) {
+ if a.fences[e.host] != 0 || a.shuttingDown {
a.beginStopLocked(e)
} else if e.bounceAfterWake {
// A config delivery or `carlos restart` asked for while this wake
@@ -1871,6 +1993,10 @@ type wokenInstance struct {
// kill was not proven (genStartUnproven). cleanupFailedWake keeps the
// lease until the sweep proves it, whatever a.gens says.
genUnproven bool
+ // legacyUncertain: the spawn was a start of the host's original run
+ // unit that root may still carry out (legacyStartUncertain, #292).
+ // cleanupFailedWake keeps the lease until the unit is proven gone.
+ legacyUncertain bool
// pend is the build a cold wake captured and published as pending at
// step 2b (settleWakeIdentity; ruling R-COLDPUBLISH). wake() drops it in
// the critical section that ends the wake: by then the entry's
@@ -2061,7 +2187,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
if err != nil {
if lease != nil {
w.stopOutcome.uncertain = true
- w.stopOutcome.leaseReleased = releaseLease(ctx, lease) == nil
+ w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
}
var held *errLeaseHeld
if errors.As(err, &held) {
@@ -2072,7 +2198,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
}
if ctx.Err() != nil || !a.wakeAllowed(host) {
- w.stopOutcome.leaseReleased = releaseLease(ctx, lease) == nil
+ w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
return w, ErrFenced
}
// 1.5. Flag off, the host cgroup is proven empty under the lease before
@@ -2108,12 +2234,12 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
log.Printf("activator: %s: %v; leaving the lease to its TTL", host, err)
w.stopOutcome.uncertain = true
} else {
- w.stopOutcome.leaseReleased = releaseLease(ctx, lease) == nil
+ w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
}
return w, fmt.Errorf("restore %s: %w", host, err)
}
if ctx.Err() != nil || !a.wakeAllowed(host) {
- w.stopOutcome.leaseReleased = releaseLease(ctx, lease) == nil
+ w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
return w, ErrFenced
}
@@ -2134,7 +2260,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
if plan.planned() {
next, err := a.checkPlan(ctx, e, rt, plan)
if err != nil {
- w.stopOutcome.leaseReleased = releaseLease(ctx, lease) == nil
+ w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
return w, fmt.Errorf("start %s: %w", host, err)
}
plan = next
@@ -2147,7 +2273,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
// file, so the link is read once more just before the start. It
// narrows a repoint after the wait; it cannot close it.
if v := a.liveVersion(host); v != held {
- w.stopOutcome.leaseReleased = releaseLease(ctx, lease) == nil
+ w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
return w, fmt.Errorf("start %s: %w: %s is on %q, wants %s", host, errStaleLink, host, v, held)
}
}
@@ -2155,7 +2281,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
var idErr error
id, w.pend, idErr = a.settleWakeIdentity(ctx, ib, e, rt, held, heldID)
if idErr != nil {
- w.stopOutcome.leaseReleased = releaseLease(ctx, lease) == nil
+ w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
return w, fmt.Errorf("start %s: %w", host, idErr)
}
if id.StagedPath == "" {
@@ -2193,7 +2319,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
// the same way stop() leaves ErrStillRunning.
log.Printf("activator: %s: %v; leaving the lease to its TTL", host, err)
} else {
- w.stopOutcome.leaseReleased = releaseLease(ctx, lease) == nil
+ w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
}
return w, fmt.Errorf("seed %s: %w", host, err)
}
@@ -2226,7 +2352,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
token string
)
if ctx.Err() != nil || !a.wakeAllowed(host) {
- w.stopOutcome.leaseReleased = releaseLease(ctx, lease) == nil
+ w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
return w, ErrFenced
}
w.stopOutcome.uncertain = true
@@ -2245,7 +2371,7 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
if err != nil {
// w goes back with the error, so wake() drops step 2b's
// pending build (w.pend) and records this stop's outcome.
- w.stopOutcome.leaseReleased = releaseLease(ctx, lease) == nil
+ w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
return w, fmt.Errorf("start %s: %w", host, err)
}
// Published onto the entry NOW, not at the awake transition: the
@@ -2345,10 +2471,14 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
// running with no process handle (plan 2's startUnitAt). Its lease
// is kept, process or not (P2-13).
w.genUnproven = genStartUnproven(err) // swapsim:M9
- if proc != nil || w.genUnproven {
+ // A legacy run unit's start that root may still carry out (a start
+ // job systemd queued, which cancelling the client does not cancel)
+ // is stopped and proven gone before its lease goes (#292, P1).
+ w.legacyUncertain = legacyStartMayRun(err)
+ if proc != nil || w.genUnproven || w.legacyUncertain {
a.cleanupFailedWake(ctx, rt, w, lease)
} else {
- w.stopOutcome.leaseReleased = releaseLease(ctx, lease) == nil
+ w.stopOutcome.leaseReleased = a.giveBack(ctx, host, lease)
}
return w, fmt.Errorf("start %s: %w", host, err)
}
@@ -3967,6 +4097,8 @@ func (a *Service) finishPendingStops() {
// Failed wakes whose generation is not proven stopped hold their lease
// until it is (zero-downtime spec §7.4).
a.reapUnprovenWakesLocked()
+ // Lease releases that failed are retried on the same tick (#292).
+ a.retryOwedReleasesLocked()
a.reapStraySidecarsLocked()
for _, e := range a.entries {
// Members first (zero-downtime spec §7.4): a generation whose stop
@@ -4262,6 +4394,14 @@ func (a *Service) sweepOnce() {
earliest = nil
}
a.mu.Lock()
+ if a.shuttingDown {
+ // Shutdown has begun and is about to stop this sweeper: from here it
+ // owns every entry (#292). A child found exited now is its stop's,
+ // not an unexpected exit to recycle, and nothing is due-woken or
+ // started. Shutdown runs finishPendingStops itself.
+ a.mu.Unlock()
+ return
+ }
// Finish stops that were left pending because proc.Stop returned
// ErrStillRunning (design §6.3, codex R2-5): termination wasn't proven
// at the time, so stop() kept the lease and returned without forgetting
@@ -4853,8 +4993,13 @@ func (a *Service) Shutdown(ctx context.Context) {
// below cover the rest.)
a.mu.Lock()
a.shuttingDown = true
+ // Proofs that start or run from here end with this budget too
+ // (legacyRunGone, #292): none may outlive Shutdown.
+ a.shutdownCtx = ctx
sweepStop, sweepDone := a.sweepStop, a.sweepDone
a.mu.Unlock()
+ // …and those already running end with it too (Astra, round 3).
+ context.AfterFunc(ctx, a.cancelProofs)
// …and cancel the service lifetime, which TEARS DOWN any worker spawn
// already past that check rather than letting the stop below wait it out.
// The wait is right for a Bounce or an idle sleep — no deadline, and
@@ -4889,26 +5034,50 @@ func (a *Service) Shutdown(ctx context.Context) {
case actStopping:
waits = append(waits, e.stopped)
case actWaking:
- // With swaps on, Shutdown covers waking entries too
- // (zero-downtime spec §7.4). The wake is cancelled, and once it
- // resolves, wake() itself stops the entry through the barrier.
- // With swaps off, today's rule holds: a mid-flight wake resolves
- // on its own, its instance is orphaned only if it wins the race
- // with process exit, and the lease TTL still bounds the damage.
- if a.gens { // swapsim:M6
- if e.wakeCancel != nil {
- e.wakeCancel()
- }
- waking = append(waking, e)
+ // Shutdown covers waking entries too (zero-downtime spec §7.4),
+ // whatever CARLOS_SWAP says. The wake is cancelled, and once it
+ // resolves, wake() itself stops the entry (through the barrier
+ // with swaps on), or the failed wake has given its lease back
+ // under a budget of its own (releaseLease).
+ //
+ // Swaps off used to leave a waking entry to resolve on its own,
+ // with the lease's TTL to bound the damage. That is #292: an
+ // edge restart stops every carlos-instance@ unit before the
+ // agent's SIGTERM, the agent recycles one and a request wakes it
+ // again, and that wake's start is a systemctl start job queued
+ // behind the edge restart. The agent exited with the wake still
+ // holding its lease, and the next agent refused the host for the
+ // lease's whole 15 minutes.
+ if e.wakeCancel != nil {
+ e.wakeCancel()
}
+ waking = append(waking, e) // swapsim:M6
}
}
a.mu.Unlock()
+ // One ticker for every wait below, the wake joins included (#292,
+ // Astra P2): a wake that ignores its cancellation can hold a join until
+ // the deadline, and a stop left pending meanwhile must still be
+ // finished, and its lease given back, as soon as its process is proven
+ // gone, not once every wake has joined.
+ poll := time.NewTicker(100 * time.Millisecond)
+ defer poll.Stop()
+ pollPending := func() {
+ a.mu.Lock()
+ a.finishPendingStops()
+ a.mu.Unlock()
+ }
for _, e := range waking {
- select {
- case <-e.ready:
- case <-ctx.Done():
- return
+ for joined := false; !joined; {
+ select {
+ case <-e.ready:
+ joined = true
+ case <-poll.C:
+ pollPending()
+ case <-ctx.Done():
+ a.logUnresolvedAtExit()
+ return
+ }
}
a.mu.Lock()
if a.entries[e.host] == e && e.state == actStopping {
@@ -4916,18 +5085,15 @@ func (a *Service) Shutdown(ctx context.Context) {
}
a.mu.Unlock()
}
- poll := time.NewTicker(100 * time.Millisecond)
- defer poll.Stop()
for _, w := range waits {
for done := false; !done; {
select {
case <-w:
done = true
case <-poll.C:
- a.mu.Lock()
- a.finishPendingStops()
- a.mu.Unlock()
+ pollPending()
case <-ctx.Done():
+ a.logUnresolvedAtExit()
return
}
}
@@ -4946,9 +5112,13 @@ func (a *Service) Shutdown(ctx context.Context) {
// exception to dormancy from the task 2 fix). That flag-off retention
// proves only the leader's exit, not its descendants (final review F6,
// deferred). With neither, the map is empty and this returns at once.
+ //
+ // Lease releases that failed (#292, Astra P3) are retried here the same
+ // way, inside the same budget (retryOwedReleasesLocked, also inside
+ // finishPendingStops), and named at exit if they are still owed.
for {
a.mu.Lock()
- left := len(a.unprovenWakes)
+ left := len(a.unprovenWakes) + len(a.owedReleases)
if left > 0 {
a.finishPendingStops()
}
@@ -4959,19 +5129,35 @@ func (a *Service) Shutdown(ctx context.Context) {
select {
case <-poll.C:
case <-ctx.Done():
- a.mu.Lock()
- var held []string
- for host, u := range a.unprovenWakes {
- held = append(held, fmt.Sprintf("%s (generation %d)", host, u.gen.N))
- }
- a.mu.Unlock()
- slices.Sort(held)
- log.Printf("activator: shutdown: out of time with %d failed wake(s) not proven stopped, their leases still held: %s", len(held), strings.Join(held, ", "))
+ a.logUnresolvedAtExit()
return
}
}
}
+// logUnresolvedAtExit names what Shutdown's budget ran out on: failed
+// wakes not proven stopped, whose leases are kept, and lease releases that
+// failed and were not resolved by a retry. The TTL frees either.
+func (a *Service) logUnresolvedAtExit() {
+ a.mu.Lock()
+ var held, owed []string
+ for host, u := range a.unprovenWakes {
+ held = append(held, fmt.Sprintf("%s (generation %d)", host, u.gen.N))
+ }
+ for o := range a.owedReleases {
+ owed = append(owed, o.host)
+ }
+ a.mu.Unlock()
+ if len(held) > 0 {
+ slices.Sort(held)
+ log.Printf("activator: shutdown: out of time with %d failed wake(s) not proven stopped, their leases still held: %s", len(held), strings.Join(held, ", "))
+ }
+ if len(owed) > 0 {
+ slices.Sort(owed)
+ log.Printf("activator: shutdown: out of time with %d lease release(s) unresolved, held until their TTL: %s", len(owed), strings.Join(owed, ", "))
+ }
+}
+
// ServeHTTP answers one proxied request for a hibernating instance: wake (or
// join a wake), proxy, stamp activity. Lease contention and wake timeouts are
// 503s — "sleeping hosts are slow to wake", never a 5xx cascade of retries.
diff --git a/internal/activator/activator_test.go b/internal/activator/activator_test.go
index 154e307e6..0000ed2e2 100644
--- a/internal/activator/activator_test.go
+++ b/internal/activator/activator_test.go
@@ -34,6 +34,7 @@ import (
"github.com/carlosframework/platform/internal/cgroup"
"github.com/carlosframework/platform/internal/edge"
+ "github.com/carlosframework/platform/internal/lease"
"github.com/carlosframework/platform/internal/provision"
"github.com/carlosframework/platform/internal/registry"
@@ -243,6 +244,8 @@ func (p *fakeProc) exit() {
type fakeLease struct {
rep *fakeRep
b *fakeBackend // records the sync interval the wake threads through
+ // owed: a Release failed and RetryRelease has not yet landed. Under b.mu.
+ owed bool
}
func (l *fakeLease) Replicate(_ context.Context, _ string, syncInterval time.Duration) (Rep, error) {
@@ -272,10 +275,67 @@ func (l *fakeLease) Err() error {
func (l *fakeLease) Release(ctx context.Context) error {
if l.b != nil {
l.b.mu.Lock()
- l.b.releases++
+ defer l.b.mu.Unlock()
l.b.releaseCtxErrs = append(l.b.releaseCtxErrs, ctx.Err())
- l.b.mu.Unlock()
+ if len(l.b.releaseErrs) > 0 {
+ err := l.b.releaseErrs[0]
+ l.b.releaseErrs = l.b.releaseErrs[1:]
+ l.owed = true
+ return err
+ }
+ l.b.releases++
+ l.b.queuedAtRelease = append(l.b.queuedAtRelease, l.b.queuedJobs)
+ }
+ return nil
+}
+
+// RetryRelease is lease.Held's: it discharges what a failed Release left
+// owed, and counts as the release once it lands.
+func (l *fakeLease) RetryRelease(context.Context) error {
+ if l.b == nil {
+ return nil
+ }
+ l.b.mu.Lock()
+ defer l.b.mu.Unlock()
+ l.b.retries++
+ if !l.owed {
+ return nil
+ }
+ if len(l.b.retryErrs) > 0 {
+ err := l.b.retryErrs[0]
+ l.b.retryErrs = l.b.retryErrs[1:]
+ if l.b.leaseLapsed {
+ l.owed = false
+ return fmt.Errorf("%w: %w", lease.ErrReleaseLapsed, err)
+ }
+ return err
+ }
+ l.owed = false
+ l.b.releases++
+ l.b.queuedAtRelease = append(l.b.queuedAtRelease, l.b.queuedJobs)
+ return nil
+}
+
+// LegacyRunGone is ExecBackend's: the legacy run unit's stop, which
+// replaces a queued start job, and its proof.
+func (b *fakeBackend) LegacyRunGone(ctx context.Context, _ registry.Route) error {
+ b.mu.Lock()
+ b.legacyStops++
+ block := b.legacyBlock
+ b.mu.Unlock()
+ if block {
+ <-ctx.Done()
+ b.mu.Lock()
+ b.legacyEnded++
+ b.mu.Unlock()
+ return ctx.Err()
+ }
+ b.mu.Lock()
+ defer b.mu.Unlock()
+ if b.legacyStopErr != nil {
+ return b.legacyStopErr
}
+ b.queuedJobs = 0
return nil
}
@@ -306,8 +366,40 @@ type fakeBackend struct {
seedRoutes []registry.Route
startErr error
startGate chan struct{} // when non-nil, Start() blocks until closed
- reps []*fakeRep
- procs []*fakeProc
+ // startQueued, when non-nil, blocks Start() until it is closed or the
+ // wake's context ends. It models a nested legacy start whose systemctl
+ // start job systemd has queued behind an edge restart (#292). The
+ // instance-control client gives up when its context does, but the
+ // queued job SURVIVES that: a cancelled start counts one in queuedJobs
+ // and answers as ExecBackend does (no process, legacyStartUncertain).
+ // Only LegacyRunGone, the unit's stop, clears it.
+ startQueued chan struct{}
+ queuedJobs int
+ // legacyStops counts LegacyRunGone calls; legacyStopErr is what they
+ // answer (an unproven stop) while set.
+ legacyStops int
+ legacyStopErr error
+ // legacyBlock holds every LegacyRunGone until its context ends (a
+ // stop that never proves); legacyEnded counts those that returned.
+ legacyBlock bool
+ legacyEnded int
+ // startLegacyFail fails every Start as a legacy run unit's start root
+ // may still carry out, at once, outside any shutdown.
+ startLegacyFail bool
+ // leaseLapsed: the owed records have passed their expiry, so a retry
+ // that fails again answers lease.ErrReleaseLapsed, as lease.Held does.
+ leaseLapsed bool
+ // queuedAtRelease is queuedJobs at each lease release that landed: a
+ // release over a still-queued job is the single-writer violation.
+ queuedAtRelease []int
+ // releaseErrs fails the next Release calls, in order (a store that
+ // refuses or times out the DELETE); retryErrs does the same for
+ // RetryRelease. retries counts RetryRelease calls.
+ releaseErrs []error
+ retryErrs []error
+ retries int
+ reps []*fakeRep
+ procs []*fakeProc
// syncIntervals records, per wake, the replication-class tick doWake
// threads into Replicate() — the ladder's seam assertion.
syncIntervals []time.Duration
@@ -470,11 +562,12 @@ func (b *fakeBackend) Seed(_ context.Context, rt registry.Route) error {
return b.seedErr
}
-func (b *fakeBackend) Start(_ context.Context, rt registry.Route) (Proc, string, error) {
+func (b *fakeBackend) Start(ctx context.Context, rt registry.Route) (Proc, string, error) {
b.mu.Lock()
b.starts++
b.order = append(b.order, "start:instance")
gate := b.startGate
+ queued := b.startQueued
fn := b.execVerFn
startErr := b.startErr
proc := &fakeProc{label: "instance", rec: b, stuck: b.procStuck}
@@ -484,9 +577,25 @@ func (b *fakeBackend) Start(_ context.Context, rt registry.Route) (Proc, string,
if startErr != nil {
return nil, "", startErr
}
+ b.mu.Lock()
+ legacyFail := b.startLegacyFail
+ b.mu.Unlock()
+ if legacyFail {
+ return nil, "", markLegacyStartUncertain(markStartUncertain(fmt.Errorf("%s: start unit: root did not answer", rt.Host)))
+ }
if gate != nil {
<-gate
}
+ if queued != nil {
+ select {
+ case <-queued:
+ case <-ctx.Done():
+ b.mu.Lock()
+ b.queuedJobs++
+ b.mu.Unlock()
+ return nil, "", markLegacyStartUncertain(markStartUncertain(fmt.Errorf("%s: start unit: %w", rt.Host, ctx.Err())))
+ }
+ }
ver := ""
if fn != nil {
ver = fn()
diff --git a/internal/activator/backend_exec.go b/internal/activator/backend_exec.go
index 798479ab6..ddaa90770 100644
--- a/internal/activator/backend_exec.go
+++ b/internal/activator/backend_exec.go
@@ -728,6 +728,10 @@ func (l *execLease) Replicate(ctx context.Context, dbPath string, syncInterval t
func (l *execLease) Release(ctx context.Context) error { return l.held.Release(ctx) }
+// RetryRelease is releaseRetrier: the held lease's conditional retry of a
+// release that failed (#292).
+func (l *execLease) RetryRelease(ctx context.Context) error { return l.held.RetryRelease(ctx) }
+
// Err is the held lease's liveness (leaseHealth): a failed renewal, a
// spent handle or a poisoned one.
func (l *execLease) Err() error { return l.held.Err() }
@@ -2417,7 +2421,11 @@ func (b *ExecBackend) spawnUnitFrom(ctx context.Context, rt registry.Route, sock
return fail(markStartUncertain(fmt.Errorf("%s: start unit: %w", rt.Host, err)))
}
if err != nil {
- return fail(markStartUncertain(fmt.Errorf("%s: start unit: %w", rt.Host, err)))
+ // The original run unit (generation 0). Root was asked to start
+ // it, and a start job systemd queued (behind an edge restart, say)
+ // outlives this call's cancellation, so the wake's cleanup stops
+ // the unit and proves it gone before the lease goes (#292).
+ return fail(markLegacyStartUncertain(markStartUncertain(fmt.Errorf("%s: start unit: %w", rt.Host, err))))
}
if inv == "" {
// The server read the invocation after the start and found none:
@@ -2489,6 +2497,65 @@ func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock s
return up, spawnVer, nil
}
+// LegacyRunGone stops rt's original run unit (carlos-instance@, generation
+// 0) and proves it gone (#292, Astra P1): the failed-wake cleanup's proof
+// before an aborted start's lease goes back. The stop goes through
+// instance-control's /stop, which runs `systemctl stop`. systemd keeps one
+// job per unit and a stop job replaces a queued start job, so a successful
+// stop also means that no start job is still queued, even while the edge
+// restart that queued it is in progress. Then the unit must read inactive
+// (or failed) with its cgroup empty or gone, the facts the exit observer
+// trusts (observe, design §6.5). Bounded by ctx, which the caller gives a
+// budget of its own; anything short of the proof is an error, and the
+// caller keeps the lease.
+func (b *ExecBackend) LegacyRunGone(ctx context.Context, rt registry.Route) error {
+ if b.Control == nil {
+ return fmt.Errorf("%s: instance control channel not configured", rt.Host)
+ }
+ unit := unitName(instancectl.RoleRun, rt.Host, 0)
+ if err := b.Control.Stop(ctx, rt.Host, instancectl.RoleRun); err != nil {
+ return fmt.Errorf("stop %s: %w", unit, err)
+ }
+ state := b.UnitState
+ if state == nil {
+ state = systemctlUnitState
+ }
+ events := b.CgroupEvents
+ if events == nil {
+ events = cgroupEventsUnder(b.cgroupRoot())
+ }
+ poll := b.ObservePoll
+ if poll <= 0 {
+ poll = time.Second
+ }
+ slice := instancectl.SliceName(rt.Host)
+ var last error
+ for {
+ active, _, err := state(ctx, unit)
+ switch {
+ case err != nil:
+ last = fmt.Errorf("unit state: %w", err)
+ case active != "inactive" && active != "failed":
+ last = fmt.Errorf("unit is %s", active)
+ default:
+ exists, populated, eerr := events(slice, unit)
+ switch {
+ case eerr != nil:
+ last = fmt.Errorf("cgroup.events: %w", eerr)
+ case exists && populated:
+ last = errors.New("processes remain in its cgroup")
+ default:
+ return nil
+ }
+ }
+ select {
+ case <-ctx.Done():
+ return fmt.Errorf("%s not proven stopped: %v", unit, last)
+ case <-time.After(poll):
+ }
+ }
+}
+
// killFailedGen ends a generation whose start failed or answered no
// invocation. The start may have succeeded with its answer lost, so the
// generation is killed and proven empty (StopGen with kill) on a budget the
diff --git a/internal/activator/backend_legacy_test.go b/internal/activator/backend_legacy_test.go
new file mode 100644
index 000000000..20a98f426
--- /dev/null
+++ b/internal/activator/backend_legacy_test.go
@@ -0,0 +1,108 @@
+package activator
+
+import (
+ "context"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/instancectl"
+)
+
+// queuingControl is root's channel during an edge restart (#292): a start
+// whose job systemd has queued blocks until the client gives up, and the
+// job outlives that. Only a stop of the unit replaces it.
+type queuingControl struct {
+ *fakeControl
+ mu sync.Mutex
+ queued bool
+}
+
+func (q *queuingControl) Start(ctx context.Context, host string, role instancectl.Role, max int64) (string, error) {
+ q.rec("start " + host + " " + string(role))
+ q.mu.Lock()
+ q.queued = true
+ q.mu.Unlock()
+ <-ctx.Done()
+ return "", ctx.Err()
+}
+
+func (q *queuingControl) Stop(ctx context.Context, host string, role instancectl.Role) error {
+ q.mu.Lock()
+ q.queued = false
+ q.mu.Unlock()
+ return q.fakeControl.Stop(ctx, host, role)
+}
+
+func (q *queuingControl) stillQueued() bool {
+ q.mu.Lock()
+ defer q.mu.Unlock()
+ return q.queued
+}
+
+// A legacy start that is cancelled while root's start job is queued comes
+// back marked, so the wake's cleanup knows the job may still run; the
+// proof stops the unit, which clears the job, and only then passes.
+func TestACancelledLegacyStartIsMarkedAndItsStopClearsTheQueuedJob(t *testing.T) {
+ q := &queuingControl{fakeControl: &fakeControl{}}
+ unit := &fakeUnit{}
+ unit.set("inactive", "", false, false)
+ b, rt := nestedBackend(t, &fakeControl{}, unit)
+ b.Control = q
+
+ ctx, cancel := context.WithCancel(context.Background())
+ go func() {
+ for !q.stillQueued() {
+ time.Sleep(5 * time.Millisecond)
+ }
+ cancel()
+ }()
+ p, _, err := b.Start(ctx, rt)
+ if err == nil || p != nil {
+ t.Fatalf("Start = %v, %v; want no process and an error", p, err)
+ }
+ if !legacyStartMayRun(err) {
+ t.Fatalf("a cancelled legacy start is not marked as one whose job may run: %v", err)
+ }
+ if !q.stillQueued() {
+ t.Fatal("the fake lost its queued job on cancellation; it must model a job that survives")
+ }
+
+ pctx, pcancel := context.WithTimeout(context.Background(), 2*time.Second)
+ defer pcancel()
+ if err := b.LegacyRunGone(pctx, rt); err != nil {
+ t.Fatalf("LegacyRunGone: %v", err)
+ }
+ if q.stillQueued() {
+ t.Fatal("the proof passed with the start job still queued")
+ }
+ if !strings.Contains(q.joined(), "stop "+rt.Host+" run") {
+ t.Fatalf("the run unit was not stopped through instance-control; calls:\n%s", q.joined())
+ }
+}
+
+// The proof fails, and so keeps the lease, while the unit is not stopped
+// or its cgroup still holds a process.
+func TestLegacyRunGoneRefusesWhileTheUnitRuns(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ active string
+ populated bool
+ }{
+ {"active", "active", true},
+ {"deactivating", "deactivating", true},
+ {"inactive with processes left", "inactive", true},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ unit := &fakeUnit{}
+ unit.set(tc.active, "inv", true, tc.populated)
+ b, rt := nestedBackend(t, &fakeControl{}, unit)
+ ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)
+ defer cancel()
+ if err := b.LegacyRunGone(ctx, rt); err == nil {
+ t.Fatal("LegacyRunGone proved a unit that still runs")
+ }
+ })
+ }
+}
diff --git a/internal/activator/backend_unit.go b/internal/activator/backend_unit.go
index 8c905f2a4..51daf227d 100644
--- a/internal/activator/backend_unit.go
+++ b/internal/activator/backend_unit.go
@@ -362,6 +362,21 @@ func (b *SplitBackend) KillGen(ctx context.Context, rt registry.Route, n uint64)
return k.KillGen(ctx, rt, n)
}
+// LegacyRunGone is legacyRunProver for the composed backend (#292): an
+// exec row's nested run unit is the exec side's to stop and prove. A
+// unit-backed row never starts a carlos-instance@ unit, so it refuses
+// rather than claiming a proof.
+func (b *SplitBackend) LegacyRunGone(ctx context.Context, rt registry.Route) error {
+ if rt.Backing == registry.BackingUnit {
+ return fmt.Errorf("%s: a unit-backed row has no instance run unit: %w", rt.Host, errNoLegacyProof)
+ }
+ p, ok := b.Exec.(legacyRunProver)
+ if !ok {
+ return fmt.Errorf("%s: %w", rt.Host, errNoLegacyProof)
+ }
+ return p.LegacyRunGone(ctx, rt)
+}
+
// Layout implements Classifier for the composed backend: a unit-backed row
// is always today's flat unit shape (the exec-only layouts do not apply to
// it), and an exec row defers to the exec side's own classification.
diff --git a/internal/activator/containment.go b/internal/activator/containment.go
index 9e907366d..0426d36ab 100644
--- a/internal/activator/containment.go
+++ b/internal/activator/containment.go
@@ -37,6 +37,88 @@ func (e startUncertain) Unwrap() error { return e.err }
// markStartUncertain wraps err as startUncertain.
func markStartUncertain(err error) error { return startUncertain{err: err} }
+// legacyStartUncertain marks a failed start of a nested host's original
+// run unit (generation 0, carlos-instance@) that root was asked for and may
+// still carry out (#292): cancelling the client's /start does not cancel a
+// start job systemd has queued, and that job can launch a writer later.
+// The failed wake's cleanup must stop the unit and prove it gone
+// (legacyRunProver) before the lease goes; unproven, the lease is kept.
+// Like startUncertain it changes neither the text nor errors.Is, so the
+// member's wording is unchanged.
+type legacyStartUncertain struct{ err error }
+
+func (e legacyStartUncertain) Error() string { return e.err.Error() }
+func (e legacyStartUncertain) Unwrap() error { return e.err }
+
+// markLegacyStartUncertain wraps err as legacyStartUncertain.
+func markLegacyStartUncertain(err error) error { return legacyStartUncertain{err: err} }
+
+// legacyStartMayRun reports whether err is a legacy run unit's start whose
+// queued job may still run (legacyStartUncertain).
+func legacyStartMayRun(err error) bool {
+ var l legacyStartUncertain
+ return errors.As(err, &l)
+}
+
+// legacyRunProver is the backend half of that proof: stop the host's
+// original run unit through instance-control and prove it gone (stop
+// answered, unit inactive, its cgroup empty). nil is the proof.
+// ExecBackend has it; SplitBackend passes it to its exec side.
+type legacyRunProver interface {
+ LegacyRunGone(ctx context.Context, rt registry.Route) error
+}
+
+// errNoLegacyProof: the backend cannot stop a legacy run unit, so nothing
+// proves an aborted start gone and its lease is kept.
+var errNoLegacyProof = errors.New("activator: this backend cannot prove a run unit stopped")
+
+// legacyStopBudget bounds one legacy proof: the stop call and the wait for
+// the unit to read stopped, on a context of its own (a cancelled wake's
+// context is often why the start was aborted).
+const legacyStopBudget = instancectl.DeadlineControl
+
+// legacyRunGone runs the backend's proof for rt under legacyStopBudget and
+// never past Shutdown's own budget (Astra, rounds 2 and 3): a proof started
+// once Shutdown has begun runs under its context, and every proof is
+// registered so that one already running when Shutdown began is cancelled
+// the moment that budget ends (cancelProofs). A proof cut short is
+// unproven, so the lease is kept.
+func (a *Service) legacyRunGone(rt registry.Route) error {
+ p, ok := a.backend.(legacyRunProver)
+ if !ok {
+ return errNoLegacyProof
+ }
+ parent := context.Background()
+ token := new(byte)
+ a.mu.Lock()
+ if a.shutdownCtx != nil {
+ parent = a.shutdownCtx
+ }
+ ctx, cancel := context.WithTimeout(parent, legacyStopBudget)
+ if a.proofCancels == nil {
+ a.proofCancels = make(map[*byte]context.CancelFunc)
+ }
+ a.proofCancels[token] = cancel
+ a.mu.Unlock()
+ defer func() {
+ a.mu.Lock()
+ delete(a.proofCancels, token)
+ a.mu.Unlock()
+ cancel()
+ }()
+ return p.LegacyRunGone(ctx, rt)
+}
+
+// cancelProofs cancels every legacy proof still running: Shutdown arms it
+// on its own context, so none outlives the shutdown budget.
+func (a *Service) cancelProofs() {
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ for _, cancel := range a.proofCancels {
+ cancel()
+ }
+}
+
// errStartedThenExited is the one uncertain start whose outcome is known to
// be a crash at the start: root started the unit and it had exited before
// root read its invocation (Astra R3 ruling 206, zdd-readiness Task 6). The
diff --git a/internal/activator/maintenance.go b/internal/activator/maintenance.go
index 533f615de..1b6cafe14 100644
--- a/internal/activator/maintenance.go
+++ b/internal/activator/maintenance.go
@@ -332,7 +332,32 @@ func (a *Service) cleanupFailedWake(ctx context.Context, rt registry.Route, w *w
err := w.proc.Stop(stopCtx)
w.stopOutcome.mainStop, w.stopOutcome.mainExited = err == nil, w.proc.Exited()
}
- if (a.gens || w.genUnproven || w.proc != nil) && !a.failedWakeStopped(rt, w.gen, w.proc, w.launchLayout) {
+ // An aborted or failed start of the host's original run unit that root
+ // may still carry out (#292, Astra P1): cancelling the client's /start
+ // leaves a start job systemd queued, and released over it, the lease
+ // would let that job launch a writer no lease covers. The unit is
+ // stopped (a stop job replaces the queued start) and proven gone on a
+ // budget of its own first. Unproven, the lease is KEPT: retained below
+ // and retried by the reaper, and at worst left to its TTL. A stall is
+ // the safe outcome; two writers are not.
+ legacy := w.proc == nil && w.legacyUncertain && !w.genUnproven
+ if legacy {
+ if err := a.legacyRunGone(rt); err != nil {
+ log.Printf("activator: %s: the aborted start's unit is not proven stopped (%v); keeping the lease until it is", rt.Host, err)
+ w.stopOutcome.mainExited = false
+ w.stopOutcome.uncertain = true
+ a.mu.Lock()
+ if a.unprovenWakes == nil {
+ a.unprovenWakes = make(map[string]*unprovenWake)
+ }
+ a.unprovenWakes[rt.Host] = &unprovenWake{rt: rt, gen: w.gen, lease: lease, rep: w.rep, build: w.spawnIdentity,
+ layout: w.launchLayout, legacy: true}
+ a.mu.Unlock()
+ return
+ }
+ w.stopOutcome.mainStop, w.stopOutcome.mainExited = true, true
+ }
+ if !legacy && (a.gens || w.genUnproven || w.proc != nil) && !a.failedWakeStopped(rt, w.gen, w.proc, w.launchLayout) {
// Its leader did not exit (ErrStillRunning, a failed kill) or its
// child is not proven empty.
w.stopOutcome.mainExited = false
@@ -359,5 +384,5 @@ func (a *Service) cleanupFailedWake(ctx context.Context, rt registry.Route, w *w
w.stopOutcome.leaseReleased = w.stopOutcome.replicaClosed
return
}
- w.stopOutcome.leaseReleased = releaseLease(ctx, lease) == nil
+ w.stopOutcome.leaseReleased = a.giveBack(ctx, rt.Host, lease)
}
diff --git a/internal/activator/opset.go b/internal/activator/opset.go
index a14dc1514..3a502923c 100644
--- a/internal/activator/opset.go
+++ b/internal/activator/opset.go
@@ -749,6 +749,9 @@ type unprovenWake struct {
// (wokenInstance.launchLayout), which every retry of the proof decides
// by (finding 6).
layout Layout
+ // legacy: an aborted start of the host's original run unit whose
+ // queued job may still run (#292, Astra P1). Proven by legacyRunGone.
+ legacy bool
}
// errWakeUnproven refuses a wake while the host's last failed wake is not
@@ -828,10 +831,13 @@ func (a *Service) reapUnprovenWakesLocked() {
//
// A failed close or release is logged and recorded in the stop history, and
// the wake is still forgotten: the generation, which is what retention
-// guards, is proven stopped, and a lease handle is spent by its first
-// release attempt (lease.Held.Release answers ErrSpent ever after), so a
-// retry could never succeed and would refuse the host's wakes forever. The
-// lease's TTL bounds a failed release, as for finishStop.
+// guards, is proven stopped. A failed close is bounded by the lease's TTL,
+// as for finishStop. A failed raw release is kept as an obligation that is
+// retried with the same conditional delete (giveBack, #292).
+//
+// A retained legacy start (#292, Astra P1) has no process and no unit
+// generation: it is proven by stopping its original run unit again
+// (legacyRunGone), never by the generation proof.
func (a *Service) reapUnprovenWake(host string, u *unprovenWake) {
if u.proc != nil && !u.proc.Exited() {
ctx, cancel := context.WithTimeout(context.Background(), actStopTimeout)
@@ -841,7 +847,13 @@ func (a *Service) reapUnprovenWake(host string, u *unprovenWake) {
log.Printf("activator: %s: retry stop of failed wake's generation %d: %v", host, u.gen.N, err)
}
}
- if !a.failedWakeStopped(u.rt, u.gen, u.proc, u.layout) {
+ proven := false
+ if u.legacy {
+ proven = a.legacyRunGone(u.rt) == nil
+ } else {
+ proven = a.failedWakeStopped(u.rt, u.gen, u.proc, u.layout)
+ }
+ if !proven {
a.mu.Lock()
u.busy = false
a.mu.Unlock()
@@ -855,11 +867,14 @@ func (a *Service) reapUnprovenWake(host string, u *unprovenWake) {
outcome.replicaClosed = err == nil
} else if u.lease != nil {
err = releaseLease(ctx, u.lease)
+ if err != nil {
+ a.oweRelease(host, u.lease, err)
+ }
}
cancel()
outcome.leaseReleased = err == nil
if err != nil {
- log.Printf("activator: %s: failed wake's generation %d is proven stopped, but giving its lease back failed: %v (the lease's TTL frees it)", host, u.gen.N, err)
+ log.Printf("activator: %s: failed wake's generation %d is proven stopped, but giving its lease back failed: %v", host, u.gen.N, err)
} else {
log.Printf("activator: %s: failed wake's generation %d is proven stopped; lease given back", host, u.gen.N)
}
diff --git a/internal/activator/opset_test.go b/internal/activator/opset_test.go
index 59b8b5af9..e19d1f95e 100644
--- a/internal/activator/opset_test.go
+++ b/internal/activator/opset_test.go
@@ -594,8 +594,12 @@ func TestAFencedWakeReleasesUnderALiveContext(t *testing.T) {
}
}
-// Flag off: Shutdown keeps today's rule and does not wait for a waking entry.
-func TestShutdownLeavesAWakingEntryWithSwapOff(t *testing.T) {
+// Flag off: Shutdown joins a waking entry exactly as it does with swaps on
+// (#292). It used to leave one to resolve on its own, and the agent then
+// exited with the wake still holding its lease. Here the start ignores its
+// cancelled context, so Shutdown must wait it out, stop what it woke and
+// give the lease back under a live context.
+func TestShutdownJoinsAWakingEntryWithSwapOff(t *testing.T) {
a, b, _, _ := testActivator(t, time.Hour)
gate := make(chan struct{})
b.startGate = gate
@@ -610,18 +614,29 @@ func TestShutdownLeavesAWakingEntryWithSwapOff(t *testing.T) {
go func() { a.Shutdown(context.Background()); close(done) }()
select {
case <-done:
- case <-time.After(time.Second):
- t.Fatal("Shutdown waited for a waking entry with swaps off")
+ t.Fatal("Shutdown returned while a wake was still spawning, with swaps off")
+ case <-time.After(100 * time.Millisecond):
}
close(gate)
- waitFor(t, "the wake to finish", func() bool {
- a.mu.Lock()
- defer a.mu.Unlock()
- e := a.entries[host]
- return e != nil && e.state == actAwake
- })
- a.Bounce(host)
- waitFor(t, "cleanup", func() bool { return a.entryCount() == 0 })
+ select {
+ case <-done:
+ case <-time.After(5 * time.Second):
+ t.Fatal("Shutdown never returned")
+ }
+ if !b.lastProc().isStopped() {
+ t.Fatal("the woken child outlived Shutdown")
+ }
+ if got := b.releaseCount(); got != 1 {
+ t.Fatalf("lease releases = %d, want 1", got)
+ }
+ for _, err := range b.releaseCtxErrors() {
+ if err != nil {
+ t.Fatalf("the lease was released under a dead context (%v)", err)
+ }
+ }
+ if n := a.entryCount(); n != 0 {
+ t.Fatalf("Shutdown left %d entr(ies) behind", n)
+ }
}
// execGenBackend wakes through a real ExecBackend's StartGen (a placed flat
diff --git a/internal/activator/shutdown_lease_test.go b/internal/activator/shutdown_lease_test.go
new file mode 100644
index 000000000..4b7c24297
--- /dev/null
+++ b/internal/activator/shutdown_lease_test.go
@@ -0,0 +1,470 @@
+package activator
+
+import (
+ "context"
+ "errors"
+ "log"
+ "os"
+ "slices"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+)
+
+// Discussion #292, the production sequence with swaps off. `systemctl
+// restart carlos-edge` stops every carlos-instance@ unit first
+// (BindsTo= and After=carlos-edge.service), while the old agent is still
+// running and has not had its SIGTERM. The agent sees the child gone,
+// recycles the entry and gives its lease back, and a request queued on
+// the stop wakes the host again, taking a fresh lease. That wake's start
+// is a systemctl start job systemd queues behind the edge restart, so it
+// is still in flight when the SIGTERM arrives. Shutdown has to abort it
+// and give its lease back before the process exits; otherwise the next
+// agent refuses the host for the lease's whole TTL.
+//
+// Aborting the client's start does not abort the start job systemd queued
+// (Astra, review of 9a8e295c, P1): released over that job, the lease would
+// let the job launch a writer nobody holds a lease for. So the abort stops
+// the unit and proves it gone first, and only then gives the lease back.
+func TestShutdownAbortsAWakeAndReleasesItsLease(t *testing.T) {
+ a, b, reqErr := arrangeQueuedRewake(t, "restart.example.com")
+
+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+ defer cancel()
+ a.Shutdown(ctx)
+ if ctx.Err() != nil {
+ t.Fatal("Shutdown ran out of time instead of aborting the wake")
+ }
+
+ // Every lease this agent took is given back by the time Shutdown
+ // returns: the first wake's through its replica's Close, the
+ // re-wake's through Release, and never over a start job still queued.
+ acq, _, _ := b.counts()
+ if got := b.releaseCount(); got != acq {
+ t.Fatalf("lease releases = %d after Shutdown, want %d (one per acquire)", got, acq)
+ }
+ b.mu.Lock()
+ queuedAt, queuedNow, stops := slices.Clone(b.queuedAtRelease), b.queuedJobs, b.legacyStops
+ b.mu.Unlock()
+ if stops == 0 || queuedNow != 0 {
+ t.Fatalf("the aborted start's queued job was never stopped (stops %d, queued %d)", stops, queuedNow)
+ }
+ for _, q := range queuedAt {
+ if q != 0 {
+ t.Fatalf("a lease was released with %d start job(s) still queued: %v", q, queuedAt)
+ }
+ }
+ for _, err := range b.releaseCtxErrors() {
+ if err != nil {
+ t.Fatalf("a lease was released under a dead context (%v)", err)
+ }
+ }
+ if n := a.entryCount(); n != 0 {
+ t.Fatalf("Shutdown left %d entr(ies) behind", n)
+ }
+ select {
+ case err := <-reqErr:
+ if err == nil {
+ t.Fatal("the request was served by a wake Shutdown aborted")
+ }
+ case <-time.After(5 * time.Second):
+ t.Fatal("the queued request never returned")
+ }
+}
+
+// arrangeQueuedRewake is #292's arrangement: host awake, its unit stopped
+// by systemd, and a request's re-wake inside a start that systemd has
+// queued behind the edge restart. It returns the request's outcome.
+func arrangeQueuedRewake(t *testing.T, host string) (*Service, *fakeBackend, <-chan error) {
+ t.Helper()
+ a, b, _, _ := testActivator(t, time.Hour)
+ arrangeAwake(t, a, testRoute(host))
+
+ // systemd stops the bound unit, well before the agent's SIGTERM.
+ b.lastProc().kill()
+ queued := make(chan struct{})
+ b.mu.Lock()
+ b.startQueued = queued
+ b.mu.Unlock()
+ t.Cleanup(func() { close(queued) })
+
+ reqErr := make(chan error, 1)
+ go func() {
+ release, err := a.beginRequest(context.Background(), testRoute(host))
+ if err == nil {
+ release()
+ }
+ reqErr <- err
+ }()
+ waitFor(t, "the re-wake to reach its queued start", func() bool { return b.startCount() == 2 })
+ if acq, _, _ := b.counts(); acq != 2 {
+ t.Fatalf("acquires = %d, want 2 (the first wake and the re-wake)", acq)
+ }
+ return a, b, reqErr
+}
+
+// P1's other half: an aborted legacy start whose unit cannot be proven
+// stopped KEEPS its lease. A lease left to its TTL is a 15-minute stall; a
+// lease given back over a queued start job is two writers.
+func TestAnAbortedStartNotProvenStoppedKeepsItsLease(t *testing.T) {
+ a, b, _ := arrangeQueuedRewake(t, "unproven.example.com")
+ b.mu.Lock()
+ b.legacyStopErr = errors.New("stop carlos-instance@unproven: instance control socket: connection refused")
+ b.mu.Unlock()
+
+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+ defer cancel()
+ a.Shutdown(ctx)
+
+ acq, _, _ := b.counts()
+ if got := b.releaseCount(); got != acq-1 {
+ t.Fatalf("lease releases = %d, want %d: the first wake's, and NOT the aborted start's", got, acq-1)
+ }
+ b.mu.Lock()
+ stops := b.legacyStops
+ b.mu.Unlock()
+ if stops == 0 {
+ t.Fatal("the aborted start's unit was never asked to stop")
+ }
+ a.mu.Lock()
+ _, retained := a.unprovenWakes["unproven.example.com"]
+ a.mu.Unlock()
+ if !retained {
+ t.Fatal("the unproven start was not retained with its lease")
+ }
+}
+
+// P2: a wake that ignores its cancellation holds Shutdown's join, and an
+// entry whose stop was left pending must still be finished meanwhile:
+// its process exits, and its lease goes back then, not once every wake
+// has joined.
+func TestShutdownFinishesPendingStopsWhileJoiningWakes(t *testing.T) {
+ a, b, _, _ := testActivator(t, time.Hour)
+ stuckHost, wakingHost := "stuck.example.com", "waking.example.com"
+ b.procStuck = true
+ arrangeAwake(t, a, testRoute(stuckHost))
+ stuck := b.lastProc()
+ stuckRep := b.lastRep()
+
+ b.mu.Lock()
+ b.procStuck = false
+ gate := make(chan struct{})
+ b.startGate = gate
+ b.mu.Unlock()
+ go func() {
+ if release, err := a.beginRequest(context.Background(), testRoute(wakingHost)); err == nil {
+ release()
+ }
+ }()
+ waitFor(t, "the wake to reach Start", func() bool { return b.startCount() == 2 })
+
+ ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
+ defer cancel()
+ done := make(chan struct{})
+ go func() { a.Shutdown(ctx); close(done) }()
+ waitFor(t, "the stuck stop to be left pending", func() bool {
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ e := a.entries[stuckHost]
+ return e != nil && e.pendingStop
+ })
+ stuck.exit()
+ waitFor(t, "the pending stop's lease to go back while a wake is still joining", stuckRep.isClosed)
+ select {
+ case <-done:
+ t.Fatal("Shutdown returned while a wake was still spawning")
+ default:
+ }
+ close(gate)
+ <-done
+}
+
+// P3: a failed release is an obligation, not a forgotten handle. A wake
+// that fails gives its lease back, the store refuses the DELETE, and
+// Shutdown retries it (conditionally) inside its own budget.
+func TestShutdownRetriesAFailedLeaseRelease(t *testing.T) {
+ a, b, _, _ := testActivator(t, time.Hour)
+ b.startErr = errors.New("start failed")
+ b.releaseErrs = []error{errors.New("delete lease: context deadline exceeded")}
+ if _, err := a.beginRequest(context.Background(), testRoute("refused.example.com")); err == nil {
+ t.Fatal("the wake succeeded")
+ }
+ if got := b.releaseCount(); got != 0 {
+ t.Fatalf("lease releases = %d before Shutdown, want 0 (the DELETE failed)", got)
+ }
+
+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+ defer cancel()
+ a.Shutdown(ctx)
+ if ctx.Err() != nil {
+ t.Fatal("Shutdown ran out of time")
+ }
+ b.mu.Lock()
+ retries := b.retries
+ b.mu.Unlock()
+ if got := b.releaseCount(); got != 1 || retries == 0 {
+ t.Fatalf("lease releases = %d after %d retries, want the failed release retried and landed", got, retries)
+ }
+}
+
+// P3's other half: a release still unresolved when Shutdown's budget ends
+// is logged by host, not dropped silently.
+func TestShutdownLogsALeaseReleaseItCouldNotResolve(t *testing.T) {
+ logs := lockedLog(t)
+ a, b, _, _ := testActivator(t, time.Hour)
+ b.startErr = errors.New("start failed")
+ b.releaseErrs = []error{errors.New("delete lease: timeout")}
+ for i := 0; i < 1000; i++ {
+ b.retryErrs = append(b.retryErrs, errors.New("delete lease: timeout"))
+ }
+ if _, err := a.beginRequest(context.Background(), testRoute("dark.example.com")); err == nil {
+ t.Fatal("the wake succeeded")
+ }
+ ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
+ defer cancel()
+ a.Shutdown(ctx)
+ if got := b.releaseCount(); got != 0 {
+ t.Fatalf("lease releases = %d, want 0", got)
+ }
+ if out := logs.String(); !strings.Contains(out, "unresolved") || !strings.Contains(out, "dark.example.com") {
+ t.Fatalf("no unresolved-release line naming the host at exit; log:\n%s", out)
+ }
+}
+
+// lockedBuf is a log sink safe against the activator's own goroutines.
+type lockedBuf struct {
+ mu sync.Mutex
+ buf strings.Builder
+}
+
+func (l *lockedBuf) Write(p []byte) (int, error) {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ return l.buf.Write(p)
+}
+
+func (l *lockedBuf) String() string {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ return l.buf.String()
+}
+
+func lockedLog(t *testing.T) *lockedBuf {
+ t.Helper()
+ l := &lockedBuf{}
+ flags := log.Flags()
+ log.SetOutput(l)
+ log.SetFlags(0)
+ t.Cleanup(func() {
+ log.SetOutput(os.Stderr)
+ log.SetFlags(flags)
+ })
+ return l
+}
+
+// Once Shutdown has begun, nothing wakes. A request queued on a stop must
+// not wake the host afresh when that stop finishes after Shutdown began:
+// the wake would take a lease once Shutdown had stopped looking. Startup
+// is closed, as it is on a running agent, and the request is already past
+// the startup gate (queued on a Bounce's stop) when the shutdown begins.
+func TestNoWakeOnceShutdownHasBegun(t *testing.T) {
+ a, b, _, _ := testActivator(t, time.Hour)
+ started := make(chan struct{})
+ close(started)
+ a.Startup = started
+ host := "queued.example.com"
+ arrangeAwake(t, a, testRoute(host))
+ stopGate := make(chan struct{})
+ b.lastProc().setStopGate(stopGate)
+ if !a.Bounce(host) {
+ t.Fatal("Bounce(awake host) = false")
+ }
+ waitFor(t, "the Bounce's stop", func() bool { return b.lastProc().stopEntered() == 1 })
+
+ reqErr := make(chan error, 1)
+ go func() {
+ release, err := a.beginRequest(context.Background(), testRoute(host))
+ if err == nil {
+ release()
+ }
+ reqErr <- err
+ }()
+ // The request is queued on the stop before the shutdown begins.
+ time.Sleep(50 * time.Millisecond)
+
+ done := make(chan struct{})
+ go func() {
+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+ defer cancel()
+ a.Shutdown(ctx)
+ close(done)
+ }()
+ waitFor(t, "Shutdown to begin", func() bool {
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ return a.shuttingDown
+ })
+ close(stopGate)
+ <-done
+
+ select {
+ case err := <-reqErr:
+ if err == nil {
+ t.Fatal("a request woke the host after Shutdown had begun")
+ }
+ case <-time.After(5 * time.Second):
+ t.Fatal("the queued request never returned")
+ }
+ if acq, _, starts := b.counts(); acq != 1 || starts != 1 {
+ t.Fatalf("acquires, starts = %d, %d; want 1, 1 (nothing woken after Shutdown began)", acq, starts)
+ }
+ time.Sleep(50 * time.Millisecond)
+ if n := a.entryCount(); n != 0 {
+ t.Fatalf("%d entr(ies) live after Shutdown", n)
+ }
+}
+
+// An exit seen once Shutdown has begun is the shutdown's, not the
+// build's: neither the sweep nor a request reads it as unexpected or
+// recycles it. Shutdown's own stop takes the entry down.
+func TestAnExitAfterShutdownBeganIsNotUnexpected(t *testing.T) {
+ a, b, _, _ := testActivator(t, time.Hour)
+ host := "exit.example.com"
+ arrangeAwake(t, a, testRoute(host))
+ b.lastProc().kill()
+ // The moment between Shutdown announcing itself and taking its
+ // snapshot of the entries.
+ a.mu.Lock()
+ a.shuttingDown = true
+ a.mu.Unlock()
+
+ a.sweepOnce()
+ if _, err := a.beginRequest(context.Background(), testRoute(host)); err == nil {
+ t.Fatal("a request was served once Shutdown had begun")
+ }
+ a.mu.Lock()
+ e := a.entries[host]
+ state, unexpected := actAwake, false
+ if e != nil {
+ state, unexpected = e.state, e.unexpectedExit
+ }
+ a.mu.Unlock()
+ if e == nil || state != actAwake {
+ t.Fatalf("the exit was recycled after Shutdown began (entry %v, state %v)", e != nil, state)
+ }
+ if unexpected {
+ t.Fatal("an exit after Shutdown began was read as unexpected")
+ }
+ if acq, _, _ := b.counts(); acq != 1 {
+ t.Fatalf("acquires = %d, want 1", acq)
+ }
+
+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+ defer cancel()
+ a.Shutdown(ctx)
+ if n := a.entryCount(); n != 0 {
+ t.Fatalf("Shutdown left %d entr(ies) behind", n)
+ }
+ if got := b.releaseCount(); got != 1 {
+ t.Fatalf("lease releases = %d, want 1", got)
+ }
+}
+
+// Round 2 P2: release obligations expire. Under a persistent DELETE refusal
+// with acquisition still allowed, every failed wake adds one; once their
+// records pass their own expiry the retries retire them (logged) instead
+// of running forever.
+func TestOwedReleasesAreRetiredOnceTheirRecordsExpire(t *testing.T) {
+ logs := lockedLog(t)
+ a, b, _, _ := testActivator(t, time.Hour)
+ b.startErr = errors.New("start failed")
+ hosts := []string{"one.example.com", "two.example.com", "three.example.com"}
+ for range hosts {
+ b.releaseErrs = append(b.releaseErrs, errors.New("delete lease: AccessDenied"))
+ }
+ for i := 0; i < 1000; i++ {
+ b.retryErrs = append(b.retryErrs, errors.New("delete lease: AccessDenied"))
+ }
+ for _, h := range hosts {
+ if _, err := a.beginRequest(context.Background(), testRoute(h)); err == nil {
+ t.Fatalf("%s: the wake succeeded", h)
+ }
+ }
+ owed := func() int {
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ a.finishPendingStops()
+ return len(a.owedReleases)
+ }
+ time.Sleep(50 * time.Millisecond)
+ if n := owed(); n != len(hosts) {
+ t.Fatalf("owed releases = %d before expiry, want %d", n, len(hosts))
+ }
+ b.mu.Lock()
+ b.leaseLapsed = true
+ b.mu.Unlock()
+ waitFor(t, "the expired obligations to be retired", func() bool { return owed() == 0 })
+ out := logs.String()
+ if !strings.Contains(out, "retired") {
+ t.Fatalf("no retirement line; log:\n%s", out)
+ }
+ if strings.Contains(out, "given back on retry") || b.releaseCount() != 0 {
+ t.Fatalf("an expired, never-deleted lease was reported as given back; log:\n%s", out)
+ }
+}
+
+// Astra, lower priority: the legacy proof is bounded by Shutdown's budget
+// as well as its own 30 s, whichever ends first, so no proof outlives
+// Shutdown. Unproven, the lease is still kept.
+func TestTheLegacyProofEndsWithShutdownsBudget(t *testing.T) {
+ a, b, _ := arrangeQueuedRewake(t, "slowproof.example.com")
+ b.mu.Lock()
+ b.legacyBlock = true
+ b.mu.Unlock()
+
+ ctx, cancel := context.WithTimeout(context.Background(), 300*time.Millisecond)
+ defer cancel()
+ a.Shutdown(ctx)
+ waitFor(t, "the proof to end with Shutdown's budget", func() bool {
+ b.mu.Lock()
+ defer b.mu.Unlock()
+ return b.legacyEnded > 0
+ })
+ acq, _, _ := b.counts()
+ if got := b.releaseCount(); got != acq-1 {
+ t.Fatalf("lease releases = %d, want %d: an unproven start keeps its lease", got, acq-1)
+ }
+}
+
+// Astra round 3 P2: a proof already running when Shutdown begins (here a
+// failed start's, from before the shutdown, under its own 30 s budget)
+// ends when Shutdown's budget does. Cut short, it is unproven, and the
+// lease is kept.
+func TestAProofRunningBeforeShutdownEndsWithItsBudget(t *testing.T) {
+ a, b, _, _ := testActivator(t, time.Hour)
+ b.mu.Lock()
+ b.startLegacyFail, b.legacyBlock = true, true
+ b.mu.Unlock()
+ go func() {
+ if release, err := a.beginRequest(context.Background(), testRoute("early.example.com")); err == nil {
+ release()
+ }
+ }()
+ waitFor(t, "the failed start's proof to begin", func() bool {
+ b.mu.Lock()
+ defer b.mu.Unlock()
+ return b.legacyStops == 1
+ })
+ ctx, cancel := context.WithTimeout(context.Background(), 300*time.Millisecond)
+ defer cancel()
+ a.Shutdown(ctx)
+ waitFor(t, "the proof to end with Shutdown's budget", func() bool {
+ b.mu.Lock()
+ defer b.mu.Unlock()
+ return b.legacyEnded > 0
+ })
+ if got := b.releaseCount(); got != 0 {
+ t.Fatalf("lease releases = %d, want 0: a proof cut short keeps the lease", got)
+ }
+}
diff --git a/internal/activator/testdata/swapsim/mutate.sh b/internal/activator/testdata/swapsim/mutate.sh
index 5011d2e10..0e56bbb8e 100755
--- a/internal/activator/testdata/swapsim/mutate.sh
+++ b/internal/activator/testdata/swapsim/mutate.sh
@@ -76,7 +76,7 @@ M2|swap.go|swapsim:M2|TestSwapWitnessAgentRestartMidSwap|s/a\.addMemberLocked\(e
M3|activator.go|swapsim:M3|TestSwapWitnessHoldExecsTheStagedBuild|s/e\.pin != nil/false/
M4|opset.go|swapsim:M4|TestSwapWitnessBarrierJoinsBeforeTheLease,TestSwapWitnessAgentRestartMidSwap|s/proven := a\.stopMember\(e, m\)/proven := true; go a.stopMember(e, m)/
M5|swapadmit.go|swapsim:M5|TestSwapWitnessNoAdmissionOntoAStoppingEntry|s/!waking \&\& e\.state != actAwake/false/
-M6|activator.go|swapsim:M6|TestSwapWitnessShutdownJoinsAWakingEntry|s/if a\.gens \{/if false {/
+M6|activator.go|swapsim:M6|TestSwapWitnessShutdownJoinsAWakingEntry,TestShutdownJoinsAWakingEntryWithSwapOff|s/waking = append\(waking, e\)/_ = waking/
M7|opset.go|swapsim:M7|TestSwapWitnessMemberStopProvesTheGenerationChildEmpty|s/return a\.genTerminated\(rt, m\.gen, m\.proc\)/return true || a.genTerminated(rt, m.gen, m.proc)/
M8|activator.go|swapsim:M8|TestSwapWitnessStopProvesTheCurrentGenerationEmpty|s/if a\.gens \&\& !a\.releasable\(e\)/if false \&\& !a.releasable(e)/
M9|activator.go|swapsim:M9|TestSwapWitnessUnprovenWakeKeepsTheLease|s/w\.genUnproven = genStartUnproven\(err\)/w.genUnproven = false/'
diff --git a/internal/instancectl/legacystart.go b/internal/instancectl/legacystart.go
new file mode 100644
index 000000000..35355f2b6
--- /dev/null
+++ b/internal/instancectl/legacystart.go
@@ -0,0 +1,153 @@
+package instancectl
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "net"
+ "time"
+)
+
+// Legacy run start attempts (#292, Astra round 2 P1, the owner's ruling).
+//
+// The agent proves an aborted start's unit stopped through /stop before it
+// gives the lease back. Cancelling its /start does not join this side's
+// handler, which could still submit the systemd start after that /stop had
+// answered, if it had not yet seen the disconnect; systemd's replacement of
+// a queued start by a stop only covers jobs that already exist. So, for the
+// original run unit (carlos-instance@, generation 0) only, start and stop
+// are ordered per host here:
+//
+// - every /start registers an attempt with a context /stop can cancel,
+// and checks that context immediately before it submits the start;
+// - a /stop fences the host: new starts are refused while it runs, every
+// attempt in flight is cancelled and joined (bounded), and only then is
+// the unit stopped, which replaces any job a joined attempt submitted;
+// - a start whose request was accepted at or before the moment a stop
+// fences the host (the accepted-connection watermark, connSeq) is an
+// earlier request, and is refused, however late its handler registers;
+// one accepted after it runs normally. Serve answers one request per
+// connection, so a connection's accept order is its request's.
+//
+// Once /stop has answered, no start from an earlier request is in flight or
+// can be submitted afterwards. Generation units, seeds, sidecars and the
+// claim verbs are untouched.
+
+// legacyJoinBudget bounds how long a /stop waits for the start attempts it
+// cancelled to return. One that is still running then may yet submit, so
+// the stop answers an error and the agent keeps the lease.
+const legacyJoinBudget = 10 * time.Second
+
+// legacyAttempt is one in-flight legacy run start.
+type legacyAttempt struct {
+ cancel context.CancelFunc
+ done chan struct{}
+}
+
+// connSeqKey carries a connection's accept order in its request contexts.
+type connSeqKey struct{}
+
+// nextConnSeq hands out the next accept-order number (from 1).
+func (s *Server) nextConnSeq() uint64 { return s.connSeq.Add(1) }
+
+// connContext is Serve's ConnContext: the claim verbs' connection value,
+// and the connection's accept order. ConnContext runs on the accept loop,
+// once per connection, in accept order.
+func (s *Server) connContext(ctx context.Context, c net.Conn) context.Context {
+ return context.WithValue(requestConnContext(ctx, c), connSeqKey{}, s.nextConnSeq())
+}
+
+// requestSeq is a request's accept order; 0 when its connection was not
+// stamped (a handler called outside Serve), which no fence refuses.
+func requestSeq(ctx context.Context) uint64 {
+ n, _ := ctx.Value(connSeqKey{}).(uint64)
+ return n
+}
+
+// errLegacyFenced refuses a start while a stop of its host runs, or one
+// whose request came before a stop of its host.
+var errLegacyFenced = errors.New("a stop of this host's run unit has run since this start was asked for")
+
+// beginLegacyStart registers a legacy run start of host for a request of
+// accept order seq. Its context is parent's, cancellable by a /stop of the
+// host; finish must be called when the handler returns.
+func (s *Server) beginLegacyStart(parent context.Context, host string, seq uint64) (context.Context, func(), error) {
+ s.legacyMu.Lock()
+ defer s.legacyMu.Unlock()
+ if s.legacyStopping[host] > 0 || (seq != 0 && seq <= s.legacyBarrier[host]) {
+ return nil, nil, errLegacyFenced
+ }
+ ctx, cancel := context.WithCancel(parent)
+ a := &legacyAttempt{cancel: cancel, done: make(chan struct{})}
+ if s.legacyStarts == nil {
+ s.legacyStarts = make(map[string]map[*legacyAttempt]struct{})
+ }
+ if s.legacyStarts[host] == nil {
+ s.legacyStarts[host] = make(map[*legacyAttempt]struct{})
+ }
+ s.legacyStarts[host][a] = struct{}{}
+ finish := func() {
+ s.legacyMu.Lock()
+ delete(s.legacyStarts[host], a)
+ if len(s.legacyStarts[host]) == 0 {
+ delete(s.legacyStarts, host)
+ }
+ s.legacyMu.Unlock()
+ cancel()
+ close(a.done)
+ }
+ return ctx, finish, nil
+}
+
+// fenceLegacyStarts is a /stop's first step for host: refuse new starts
+// while the stop runs, refuse every start accepted at or before this moment
+// (the accepted-connection watermark, read under legacyMu, so no start can
+// register between the watermark and the snapshot), cancel the attempts in
+// flight and join them, bounded by the join budget and ctx. The fence is
+// per host and in memory only: release lifts the refusal of new starts when
+// the stop is done, and only connections accepted before the watermark
+// stay refused. An error means an attempt did not return in time and may
+// still submit.
+func (s *Server) fenceLegacyStarts(ctx context.Context, host string) (func(), error) {
+ s.legacyMu.Lock()
+ seq := s.connSeq.Load()
+ if s.legacyStopping == nil {
+ s.legacyStopping = make(map[string]int)
+ }
+ if s.legacyBarrier == nil {
+ s.legacyBarrier = make(map[string]uint64)
+ }
+ s.legacyStopping[host]++
+ if seq > s.legacyBarrier[host] {
+ s.legacyBarrier[host] = seq
+ }
+ var inFlight []*legacyAttempt
+ for a := range s.legacyStarts[host] {
+ inFlight = append(inFlight, a)
+ }
+ s.legacyMu.Unlock()
+ release := func() {
+ s.legacyMu.Lock()
+ if s.legacyStopping[host]--; s.legacyStopping[host] <= 0 {
+ delete(s.legacyStopping, host)
+ }
+ s.legacyMu.Unlock()
+ }
+ budget := legacyJoinBudget
+ if s.legacyJoin > 0 {
+ budget = s.legacyJoin
+ }
+ timer := time.NewTimer(budget)
+ defer timer.Stop()
+ for _, a := range inFlight {
+ a.cancel()
+ select {
+ case <-a.done:
+ case <-timer.C:
+ return release, fmt.Errorf("a start of %s is still in flight after %s", host, budget)
+ case <-ctx.Done():
+ return release, fmt.Errorf("a start of %s is still in flight: %w", host, ctx.Err())
+ }
+ }
+ return release, nil
+}
diff --git a/internal/instancectl/legacystart_live_test.go b/internal/instancectl/legacystart_live_test.go
new file mode 100644
index 000000000..617e9a0ff
--- /dev/null
+++ b/internal/instancectl/legacystart_live_test.go
@@ -0,0 +1,215 @@
+package instancectl
+
+import (
+ "context"
+ "io"
+ "net"
+ "net/http"
+ "strings"
+ "testing"
+ "time"
+)
+
+// Astra round 3 P1: connection 1 (keep-alive) is accepted first and later
+// carries the /stop; a delayed /start arrives on connection 2 and has not
+// registered when the stop fences the host. The fence is the accepted
+// connection watermark at that moment, not the stop's own number, so the
+// delayed start is refused when it finally registers.
+func TestADelayedStartAcceptedBeforeTheFenceIsRefusedWhenItRegisters(t *testing.T) {
+ host := "watermark.example.com"
+ s, _ := newServer(t, &fakeCtl{})
+ _ = s.nextConnSeq() // connection 1: will carry the /stop
+ startConn := s.nextConnSeq() // connection 2: the delayed /start
+ release, err := s.fenceLegacyStarts(context.Background(), host)
+ if err != nil {
+ t.Fatalf("fence: %v", err)
+ }
+ release()
+ if _, finish, err := s.beginLegacyStart(context.Background(), host, startConn); err == nil {
+ finish()
+ t.Fatal("a start accepted before the fence registered after it and was admitted")
+ }
+}
+
+// Astra round 3 P2: a client that reuses its connection gets a fresh stamp
+// for every request, because the server answers each on its own connection.
+// A start after a stop, from the same keep-alive client, runs.
+func TestAKeepAliveClientsStartAfterAStopRuns(t *testing.T) {
+ host := "reuse.example.com"
+ ctl := &fakeCtl{answers: map[string]string{"show -p InvocationID --value": "inv1\n"}}
+ _, sock := serveOnSocket(t, ctl)
+ var dials int
+ c := &http.Client{Transport: &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+ dials++
+ return (&net.Dialer{}).DialContext(ctx, "unix", sock)
+ }}}
+ for i, path := range []string{"/start?host=" + host + "&role=run&memory_max=536870912", "/stop?host=" + host + "&role=run", "/start?host=" + host + "&role=run&memory_max=536870912"} {
+ resp, err := c.Post("http://unix"+path, "", nil)
+ if err != nil {
+ t.Fatalf("POST %s: %v", path, err)
+ }
+ body, _ := io.ReadAll(resp.Body)
+ resp.Body.Close()
+ if resp.StatusCode/100 != 2 {
+ t.Fatalf("request %d %s: status %d: %s", i, path, resp.StatusCode, body)
+ }
+ }
+ if dials != 3 {
+ t.Fatalf("dials = %d, want one connection per request", dials)
+ }
+}
+
+// Liveness 1: the fence covers only starts accepted before the stop. A
+// /start accepted after the /stop has returned runs normally, and nothing
+// of a fence outlives the server: a restarted instance-control has none.
+func TestTheFenceCoversOnlyStartsAcceptedBeforeTheStop(t *testing.T) {
+ host := "after.example.com"
+ ctl := &fakeCtl{answers: map[string]string{"show -p InvocationID --value": "inv1\n"}}
+ s, c := newServer(t, ctl)
+ stop := post(t, c, "/stop?host="+host+"&role=run")
+ stop.Body.Close()
+ if stop.StatusCode != http.StatusNoContent {
+ t.Fatalf("/stop status %d", stop.StatusCode)
+ }
+ start := post(t, c, "/start?host="+host+"&role=run&memory_max=536870912")
+ start.Body.Close()
+ if start.StatusCode != http.StatusOK {
+ t.Fatalf("a start accepted after the stop returned: status %d, want 200", start.StatusCode)
+ }
+ s.legacyMu.Lock()
+ stopping := s.legacyStopping[host]
+ s.legacyMu.Unlock()
+ if stopping != 0 {
+ t.Fatalf("the stop's refusal of new starts outlived it (%d)", stopping)
+ }
+ // A restarted service: the fence lived in memory only.
+ _, c2 := newServer(t, &fakeCtl{answers: map[string]string{"show -p InvocationID --value": "inv2\n"}})
+ again := post(t, c2, "/start?host="+host+"&role=run&memory_max=536870912")
+ again.Body.Close()
+ if again.StatusCode != http.StatusOK {
+ t.Fatalf("a restarted instance-control refused a start: status %d", again.StatusCode)
+ }
+}
+
+// heldStart parks host's legacy /start before its tier step until release
+// is closed, and returns once the handler is parked there. The handler does
+// not see its own cancellation (unseenCancel).
+func heldStart(t *testing.T, ctl *fakeCtl, c *http.Client, host string) (release chan struct{}) {
+ t.Helper()
+ entered, gate := make(chan struct{}), make(chan struct{})
+ ctl.mu.Lock()
+ if ctl.before == nil {
+ ctl.before = map[string]func(){}
+ }
+ ctl.before["set-property"] = func() {
+ close(entered)
+ <-gate
+ }
+ ctl.unseenCancel = true
+ ctl.mu.Unlock()
+ postAsync(c, "/start?host="+host+"&role=run&memory_max=536870912")
+ select {
+ case <-entered:
+ case <-time.After(5 * time.Second):
+ t.Fatal("the held start never reached its tier step")
+ }
+ return gate
+}
+
+// Liveness 2: the join is per host. Host A's attempt will not join; /start
+// and /stop of host B answer at once while A's /stop is still joining.
+func TestAStuckJoinDelaysNoOtherHost(t *testing.T) {
+ ctl := &fakeCtl{answers: map[string]string{"show -p InvocationID --value": "inv1\n"}}
+ _, c := newServer(t, ctl)
+ gate := heldStart(t, ctl, c, "stuck.example.com")
+ stopA := postAsync(c, "/stop?host=stuck.example.com&role=run")
+ defer func() { close(gate); <-stopA }()
+ time.Sleep(50 * time.Millisecond) // A's stop is joining
+ for _, path := range []string{"/start?host=free.example.com&role=run&memory_max=536870912", "/stop?host=free.example.com&role=run"} {
+ done := postAsync(c, path)
+ select {
+ case code := <-done:
+ if code/100 != 2 {
+ t.Fatalf("%s: status %d", path, code)
+ }
+ case <-time.After(2 * time.Second):
+ t.Fatalf("%s waited behind another host's stuck join", path)
+ }
+ }
+ select {
+ case <-stopA:
+ t.Fatal("A's stop answered while its attempt was still stuck")
+ default:
+ }
+}
+
+// Liveness 3: a stuck stop answers once, within the bound, and the next
+// /stop of the host does not hang behind the stuck attempt: it answers the
+// same error within the bound, and joins it once it returns.
+func TestAStuckStopAnswersAndTheNextStopDoesNotHang(t *testing.T) {
+ host := "wedged.example.com"
+ ctl := &fakeCtl{answers: map[string]string{"show -p InvocationID --value": "inv1\n"}}
+ _, c := newServer(t, ctl, func(s *Server) { s.legacyJoin = 200 * time.Millisecond })
+ gate := heldStart(t, ctl, c, host)
+ stopOnce := func() (int, string, time.Duration) {
+ begun := time.Now()
+ resp := post(t, c, "/stop?host="+host+"&role=run")
+ body, _ := io.ReadAll(resp.Body)
+ resp.Body.Close()
+ return resp.StatusCode, string(body), time.Since(begun)
+ }
+ for i := 0; i < 2; i++ {
+ code, body, took := stopOnce()
+ if code != http.StatusBadGateway || !strings.Contains(body, "still in flight") {
+ t.Fatalf("stop %d with the attempt stuck: %d %q, want 502 naming the start in flight", i+1, code, body)
+ }
+ if took > 2*time.Second {
+ t.Fatalf("stop %d took %s, past its bound", i+1, took)
+ }
+ }
+ close(gate)
+ time.Sleep(50 * time.Millisecond)
+ if code, body, _ := stopOnce(); code != http.StatusNoContent {
+ t.Fatalf("stop once the attempt returned: %d %q, want 204", code, body)
+ }
+}
+
+// Liveness 4: the other verbs pass through unchanged during a legacy stop.
+// While host A's legacy /stop is joining a stuck attempt, A's sidecar
+// /start and seed /stop answer at once.
+func TestOtherVerbsPassThroughDuringALegacyStop(t *testing.T) {
+ host := "verbs.example.com"
+ ctl := &fakeCtl{answers: map[string]string{"show -p InvocationID --value": "inv1\n"}}
+ _, c := newServer(t, ctl)
+ gate := heldStart(t, ctl, c, host)
+ stop := postAsync(c, "/stop?host="+host+"&role=run")
+ defer func() { close(gate); <-stop }()
+ time.Sleep(50 * time.Millisecond) // the legacy stop is joining
+ for _, path := range []string{"/start?host=" + host + "&role=sidecar&memory_max=536870912", "/stop?host=" + host + "&role=seed"} {
+ done := postAsync(c, path)
+ select {
+ case code := <-done:
+ if code/100 != 2 {
+ t.Fatalf("%s during a legacy stop: status %d", path, code)
+ }
+ case <-time.After(2 * time.Second):
+ t.Fatalf("%s waited behind a legacy stop", path)
+ }
+ }
+}
+
+// postAsync POSTs path from its own goroutine, never touching t, and
+// delivers the status (0 on a transport error).
+func postAsync(c *http.Client, path string) <-chan int {
+ done := make(chan int, 1)
+ go func() {
+ resp, err := c.Post("http://unix"+path, "", nil)
+ if err != nil {
+ done <- 0
+ return
+ }
+ resp.Body.Close()
+ done <- resp.StatusCode
+ }()
+ return done
+}
diff --git a/internal/instancectl/legacystart_test.go b/internal/instancectl/legacystart_test.go
new file mode 100644
index 000000000..ad8b3acf9
--- /dev/null
+++ b/internal/instancectl/legacystart_test.go
@@ -0,0 +1,114 @@
+package instancectl
+
+import (
+ "context"
+ "net/http"
+ "strings"
+ "testing"
+ "time"
+)
+
+// #292, Astra round 2 P1: cancelling the agent's /start does not join
+// root's start handler, which can still submit its systemd start after a
+// /stop for the same host has answered, if it has not yet seen the
+// disconnect. The agent then proves the unit stopped, gives the lease
+// back, and the late job launches a writer no lease covers. Here the
+// handler is held before its submit, past the client's cancellation, and
+// does not see that cancellation (unseenCancel): no start may be submitted
+// once the /stop has returned.
+func TestALegacyStartDelayedPastItsCancellationNeverSubmitsAfterAStop(t *testing.T) {
+ host := "late.example.com"
+ unit := UnitName(RoleRun, host)
+ entered, gate := make(chan struct{}), make(chan struct{})
+ ctl := &fakeCtl{
+ answers: map[string]string{"show -p InvocationID --value": "inv1\n"},
+ unseenCancel: true,
+ before: map[string]func(){"set-property": func() {
+ close(entered)
+ <-gate
+ }},
+ }
+ _, c := newServer(t, ctl)
+
+ startCtx, cancelStart := context.WithCancel(context.Background())
+ startDone := make(chan struct{})
+ go func() {
+ defer close(startDone)
+ req, _ := http.NewRequestWithContext(startCtx, http.MethodPost, "http://unix/start?host="+host+"&role=run&memory_max=536870912", nil)
+ if resp, err := c.Do(req); err == nil {
+ resp.Body.Close()
+ }
+ }()
+ select {
+ case <-entered:
+ case <-time.After(5 * time.Second):
+ t.Fatal("the start handler never reached its tier step")
+ }
+ cancelStart() // the client gives up; the handler has not noticed
+ <-startDone
+
+ stopped := make(chan int, 1)
+ go func() {
+ resp := post(t, c, "/stop?host="+host+"&role=run")
+ resp.Body.Close()
+ stopped <- resp.StatusCode
+ }()
+ // The held handler goes on past the cancellation it has not seen.
+ time.Sleep(100 * time.Millisecond)
+ close(gate)
+ var status int
+ select {
+ case status = <-stopped:
+ case <-time.After(15 * time.Second):
+ t.Fatal("/stop never returned")
+ }
+ if status != http.StatusNoContent {
+ t.Fatalf("/stop status %d, want 204", status)
+ }
+ stopAt := len(strings.Split(ctl.joined(), "\n"))
+
+ // Give the held handler every chance to submit late.
+ deadline := time.Now().Add(time.Second)
+ for time.Now().Before(deadline) {
+ calls := strings.Split(ctl.joined(), "\n")
+ for _, line := range calls[stopAt:] {
+ if line == "start "+unit {
+ t.Fatalf("a start was submitted after /stop returned; calls:\n%s", ctl.joined())
+ }
+ }
+ time.Sleep(20 * time.Millisecond)
+ }
+ calls := ctl.joined()
+ stopIdx := strings.Index(calls, "stop "+unit)
+ if stopIdx < 0 {
+ t.Fatalf("the unit was never stopped; calls:\n%s", calls)
+ }
+ if i := strings.LastIndex(calls, "start "+unit); i > stopIdx {
+ t.Fatalf("a start was submitted after the stop; calls:\n%s", calls)
+ }
+}
+
+// A start whose own request was accepted before a /stop for the host is an
+// earlier request: refused once that stop has run, even if its handler only
+// gets going afterwards.
+func TestALegacyStartFromBeforeAStopIsRefused(t *testing.T) {
+ host := "early.example.com"
+ ctl := &fakeCtl{answers: map[string]string{"show -p InvocationID --value": "inv1\n"}}
+ s, _ := newServer(t, ctl)
+ // Stamp two connections in accept order: the start's first.
+ startSeq := s.nextConnSeq()
+ _ = s.nextConnSeq() // the stop's own connection
+ release, err := s.fenceLegacyStarts(context.Background(), host)
+ if err != nil {
+ t.Fatalf("fence: %v", err)
+ }
+ release()
+ if _, _, err := s.beginLegacyStart(context.Background(), host, startSeq); err == nil {
+ t.Fatal("a start accepted before the stop was admitted after it")
+ }
+ if _, finish, err := s.beginLegacyStart(context.Background(), host, s.nextConnSeq()); err != nil {
+ t.Fatalf("a start accepted after the stop was refused: %v", err)
+ } else {
+ finish()
+ }
+}
diff --git a/internal/instancectl/server.go b/internal/instancectl/server.go
index 5df53ae72..110851e7b 100644
--- a/internal/instancectl/server.go
+++ b/internal/instancectl/server.go
@@ -17,6 +17,7 @@ import (
"strconv"
"strings"
"sync"
+ "sync/atomic"
"time"
"github.com/carlosframework/platform/internal/cgroup"
@@ -116,6 +117,18 @@ type Server struct {
genMu sync.Mutex
genStarting map[string]bool
+ // Legacy run start attempts per host (legacystart.go, #292): the
+ // starts in flight, the stops in progress, and each host's barrier,
+ // the accept order of the latest stop's request. connSeq numbers
+ // connections in accept order.
+ legacyMu sync.Mutex
+ legacyStarts map[string]map[*legacyAttempt]struct{}
+ legacyStopping map[string]int
+ legacyBarrier map[string]uint64
+ connSeq atomic.Uint64
+ // legacyJoin overrides legacyJoinBudget; tests shorten it.
+ legacyJoin time.Duration
+
mu sync.Mutex
report ProbeReport
probed bool
@@ -203,7 +216,11 @@ func runWithClaim(args []string, claim, heldRelease http.Handler, hostApp func(s
// yields vetted connections.
func (s *Server) Serve(ctx context.Context, ln net.Listener) error {
vetted := &vettedListener{Listener: ln, ok: s.peerOK, logf: s.logf}
- srv := &http.Server{Handler: s.Handler(), ReadHeaderTimeout: 5 * time.Second, ConnContext: requestConnContext}
+ srv := &http.Server{Handler: s.Handler(), ReadHeaderTimeout: 5 * time.Second, ConnContext: s.connContext}
+ // One request per connection (#292): a connection's accept order is the
+ // request's (connContext), which the legacy start fence relies on, so a
+ // client may not reuse one. The agent's client never does.
+ srv.SetKeepAlivesEnabled(false)
go func() {
<-ctx.Done()
srv.Close()
@@ -436,6 +453,17 @@ func (s *Server) handleStart(w http.ResponseWriter, r *http.Request) {
return
}
ctx := r.Context()
+ if req.role == RoleRun {
+ // An attempt a /stop of this host can cancel and join, and that a
+ // stop which has already run refuses (legacystart.go, #292).
+ attempt, finish, err := s.beginLegacyStart(ctx, req.host, requestSeq(ctx))
+ if err != nil {
+ http.Error(w, "start: "+err.Error(), http.StatusConflict)
+ return
+ }
+ defer finish()
+ ctx = attempt
+ }
unit := UnitName(req.role, req.host)
// A sidecar naming its generation's staged build, or a run unit naming
// the build its cold wake recorded, is validated before anything is
@@ -470,6 +498,12 @@ func (s *Server) handleStart(w http.ResponseWriter, r *http.Request) {
http.Error(w, "set-property: "+err.Error(), http.StatusBadGateway)
return
}
+ // Never submitted once cancelled (#292): a /stop of this host cancels
+ // the attempt before it joins it, and the client may have gone.
+ if err := ctx.Err(); err != nil {
+ http.Error(w, "start: cancelled before it was submitted: "+err.Error(), http.StatusConflict)
+ return
+ }
if out, err := s.Systemctl(ctx, "start", unit); err != nil {
http.Error(w, "start: "+strings.TrimSpace(out+" "+err.Error()), http.StatusBadGateway)
return
@@ -617,6 +651,21 @@ func (s *Server) handleStop(w http.ResponseWriter, r *http.Request) {
s.stopGen(w, r, req)
return
}
+ if req.role == RoleRun {
+ // Every start of this host's run unit in flight is cancelled and
+ // joined first, and earlier requests' starts are refused from here
+ // on (legacystart.go, #292): once this answers, no start from an
+ // earlier request can still be submitted, and the stop below
+ // replaces any job a joined start had submitted.
+ release, err := s.fenceLegacyStarts(r.Context(), req.host)
+ defer release()
+ if err != nil {
+ // Stopped anyway, but not proven final: a start may follow.
+ _, _ = s.Systemctl(r.Context(), "stop", UnitName(req.role, req.host))
+ http.Error(w, "stop: "+err.Error(), http.StatusBadGateway)
+ return
+ }
+ }
if out, err := s.Systemctl(r.Context(), "stop", UnitName(req.role, req.host)); err != nil {
http.Error(w, "stop: "+strings.TrimSpace(out+" "+err.Error()), http.StatusBadGateway)
return
diff --git a/internal/instancectl/server_test.go b/internal/instancectl/server_test.go
index 5ef76bc3c..d9ec6f9eb 100644
--- a/internal/instancectl/server_test.go
+++ b/internal/instancectl/server_test.go
@@ -28,9 +28,26 @@ type fakeCtl struct {
// simulate a client that disconnects the instant systemctl start
// answers (fix round 1, item 4).
onStart func()
+ // before, keyed by a verb (args[0]), runs once, synchronously, before
+ // that verb's call is recorded: a test holds a handler there.
+ before map[string]func()
+ // unseenCancel models a handler that has not yet seen its client go
+ // (#292): its systemctl calls run although the request's context is
+ // done, the race exec.CommandContext cannot close on its own.
+ unseenCancel bool
}
func (f *fakeCtl) run(ctx context.Context, args ...string) (string, error) {
+ f.mu.Lock()
+ var pre func()
+ if len(args) > 0 && f.before[args[0]] != nil {
+ pre = f.before[args[0]]
+ delete(f.before, args[0])
+ }
+ f.mu.Unlock()
+ if pre != nil {
+ pre()
+ }
f.mu.Lock()
line := strings.Join(args, " ")
f.calls = append(f.calls, line)
@@ -38,6 +55,7 @@ func (f *fakeCtl) run(ctx context.Context, args ...string) (string, error) {
if len(args) > 0 && args[0] == "start" && f.onStart != nil {
hook, f.onStart = f.onStart, nil
}
+ unseen := f.unseenCancel
f.mu.Unlock()
if hook != nil {
hook()
@@ -47,7 +65,7 @@ func (f *fakeCtl) run(ctx context.Context, args ...string) (string, error) {
// must behave the same way, or a test cannot tell a handler that used
// the request's context apart from one that used a detached one (fix
// round 1, item 4).
- if ctx.Err() != nil {
+ if ctx.Err() != nil && !unseen {
return "", ctx.Err()
}
f.mu.Lock()
diff --git a/internal/lease/lease.go b/internal/lease/lease.go
index 61788e219..3da80af54 100644
--- a/internal/lease/lease.go
+++ b/internal/lease/lease.go
@@ -142,6 +142,11 @@ type Held struct {
poison error
replicaClient litestream.ReplicaClient // private local-replica test seam
generationPrefix string
+ // owed is the lease record a failed Release could not delete (#292).
+ // Release spends the handle either way; RetryRelease re-issues the
+ // same conditional delete until it lands or the lease is no longer
+ // ours. Guarded by mu.
+ owed *litestream.Lease
}
// Acquire claims the single-writer lease at s3://bucket/prefix/lock.json
@@ -276,7 +281,70 @@ func (h *Held) Release(ctx context.Context) error {
if lease == nil {
return nil
}
- return h.leaser.ReleaseLease(ctx, lease)
+ err := h.leaser.ReleaseLease(ctx, lease)
+ if err != nil && !releaseSettled(err) {
+ // The handle is spent, but the lease record may still stand: keep
+ // it so RetryRelease can delete it, on the same condition (#292).
+ h.mu.Lock()
+ h.owed = lease
+ h.mu.Unlock()
+ }
+ return err
+}
+
+// releaseSettled reports whether a release error still leaves nothing of
+// this lease to delete: the record is already gone, or its ETag no longer
+// matches (the lease lapsed and is someone else's now).
+func releaseSettled(err error) bool {
+ return errors.Is(err, lss3.ErrLeaseAlreadyReleased) || errors.Is(err, litestream.ErrLeaseNotHeld)
+}
+
+// ReleaseOwed reports whether a failed Release left the lease record
+// standing for RetryRelease.
+func (h *Held) ReleaseOwed() bool {
+ h.mu.Lock()
+ defer h.mu.Unlock()
+ return h.owed != nil
+}
+
+// ErrReleaseLapsed is RetryRelease's answer when the delete failed again
+// but the owed record has passed its own expiry: the lease is free to any
+// acquirer by then, so the obligation is retired rather than retried
+// forever (#292). It wraps the delete's own error.
+var ErrReleaseLapsed = errors.New("lease release retired: the lease record has passed its expiry")
+
+// RetryRelease discharges what a failed Release left owed (#292): the same
+// conditional delete (If-Match on the lease's ETag), so it can never delete
+// a lease someone else has taken since. nil once nothing is owed, including
+// when the record is already gone or is no longer ours. A delete that fails
+// again once the record has passed its own expiry retires the obligation
+// with ErrReleaseLapsed: under a persistent refusal (AccessDenied, say)
+// obligations would otherwise pile up and be retried forever. Callers do
+// not run it concurrently with itself.
+func (h *Held) RetryRelease(ctx context.Context) error {
+ h.mu.Lock()
+ lease := h.owed
+ h.mu.Unlock()
+ if lease == nil {
+ return nil
+ }
+ if err := h.leaser.ReleaseLease(ctx, lease); err != nil && !releaseSettled(err) {
+ if lease.ExpiresAt.IsZero() || time.Now().Before(lease.ExpiresAt) {
+ return err
+ }
+ h.mu.Lock()
+ if h.owed == lease {
+ h.owed = nil
+ }
+ h.mu.Unlock()
+ return fmt.Errorf("%w (expired %s): %w", ErrReleaseLapsed, lease.ExpiresAt.UTC().Format(time.RFC3339), err)
+ }
+ h.mu.Lock()
+ if h.owed == lease {
+ h.owed = nil
+ }
+ h.mu.Unlock()
+ return nil
}
// Replicate trades the held lease for running replication of dbPath to the
diff --git a/internal/lease/release_retry_test.go b/internal/lease/release_retry_test.go
new file mode 100644
index 000000000..235628cc9
--- /dev/null
+++ b/internal/lease/release_retry_test.go
@@ -0,0 +1,193 @@
+package lease
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/aws/aws-sdk-go-v2/aws"
+ "github.com/aws/aws-sdk-go-v2/credentials"
+ awss3 "github.com/aws/aws-sdk-go-v2/service/s3"
+ "github.com/benbjohnson/litestream"
+ lss3 "github.com/benbjohnson/litestream/s3"
+)
+
+// deleteStub answers the lease's DELETE with each status in turn and
+// records the If-Match header it was sent.
+type deleteStub struct {
+ mu sync.Mutex
+ statuses []int
+ ifMatch []string
+}
+
+func (s *deleteStub) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+ if r.Method != http.MethodDelete {
+ http.Error(w, "unexpected "+r.Method, http.StatusMethodNotAllowed)
+ return
+ }
+ s.mu.Lock()
+ s.ifMatch = append(s.ifMatch, r.Header.Get("If-Match"))
+ status := http.StatusNoContent
+ if len(s.statuses) > 0 {
+ status, s.statuses = s.statuses[0], s.statuses[1:]
+ }
+ s.mu.Unlock()
+ switch status {
+ case http.StatusNoContent:
+ w.WriteHeader(status)
+ case http.StatusPreconditionFailed:
+ w.Header().Set("Content-Type", "application/xml")
+ w.WriteHeader(status)
+ _, _ = w.Write([]byte("<Error><Code>PreconditionFailed</Code><Message>no</Message></Error>"))
+ default:
+ w.Header().Set("Content-Type", "application/xml")
+ w.WriteHeader(status)
+ _, _ = w.Write([]byte("<Error><Code>AccessDenied</Code><Message>no</Message></Error>"))
+ }
+}
+
+func (s *deleteStub) sent() []string {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ return append([]string(nil), s.ifMatch...)
+}
+
+// heldAgainst builds a Held whose lease record has ETag etag, with its
+// leaser pointed at srv. The TTL is long, so the renewer never renews
+// inside the test.
+func heldAgainst(t *testing.T, srv *httptest.Server, etag string) *Held {
+ t.Helper()
+ client := awss3.New(awss3.Options{
+ Region: "us-east-1",
+ BaseEndpoint: aws.String(srv.URL),
+ UsePathStyle: true,
+ Credentials: credentials.NewStaticCredentialsProvider("k", "s", ""),
+ RetryMaxAttempts: 1,
+ })
+ leaser := lss3.NewLeaser()
+ leaser.SetClient(client)
+ leaser.Bucket = "b"
+ leaser.Path = "p"
+ leaser.TTL = time.Hour
+ rec := &litestream.Lease{Generation: 1, ETag: etag, Owner: "test", ExpiresAt: time.Now().Add(time.Hour)}
+ return &Held{leaser: leaser, rn: startRenewer(leaser, rec, time.Now(), marginForTTL(time.Hour))}
+}
+
+// A failed release leaves a conditional obligation that RetryRelease
+// discharges with the same ETag (#292): Release spends the handle, so
+// without it the lease stays held for its whole TTL.
+func TestRetryReleaseDischargesAFailedRelease(t *testing.T) {
+ stub := &deleteStub{statuses: []int{http.StatusForbidden, http.StatusNoContent}}
+ srv := httptest.NewServer(stub)
+ defer srv.Close()
+ h := heldAgainst(t, srv, `"e1"`)
+ ctx := context.Background()
+
+ if err := h.Release(ctx); err == nil {
+ t.Fatal("Release succeeded against a refusing store")
+ }
+ if !h.ReleaseOwed() {
+ t.Fatal("a failed release left no obligation")
+ }
+ if err := h.RetryRelease(ctx); err != nil {
+ t.Fatalf("RetryRelease: %v", err)
+ }
+ if h.ReleaseOwed() {
+ t.Fatal("the obligation outlived a successful retry")
+ }
+ if err := h.RetryRelease(ctx); err != nil {
+ t.Fatalf("RetryRelease with nothing owed: %v", err)
+ }
+ got := stub.sent()
+ if len(got) != 2 || got[0] != `"e1"` || got[1] != `"e1"` {
+ t.Fatalf("If-Match sent = %q, want the lease's ETag on both deletes, and no third", got)
+ }
+}
+
+// A retry whose condition fails (the lease expired and someone else holds
+// it now) has nothing of ours left to release: the obligation is resolved,
+// and nothing else's lease is deleted.
+func TestRetryReleaseResolvesWhenTheLeaseIsNoLongerOurs(t *testing.T) {
+ stub := &deleteStub{statuses: []int{http.StatusForbidden, http.StatusPreconditionFailed}}
+ srv := httptest.NewServer(stub)
+ defer srv.Close()
+ h := heldAgainst(t, srv, `"e1"`)
+ ctx := context.Background()
+ _ = h.Release(ctx)
+ if err := h.RetryRelease(ctx); err != nil {
+ t.Fatalf("RetryRelease after a failed condition: %v", err)
+ }
+ if h.ReleaseOwed() {
+ t.Fatal("a lease that is no longer ours is still owed")
+ }
+}
+
+// A successful release owes nothing.
+func TestASuccessfulReleaseOwesNothing(t *testing.T) {
+ stub := &deleteStub{}
+ srv := httptest.NewServer(stub)
+ defer srv.Close()
+ h := heldAgainst(t, srv, `"e1"`)
+ if err := h.Release(context.Background()); err != nil {
+ t.Fatalf("Release: %v", err)
+ }
+ if h.ReleaseOwed() {
+ t.Fatal("a successful release left an obligation")
+ }
+}
+
+// Round 2 P2: under a persistent DELETE failure with acquisition still
+// allowed, obligations must not pile up forever. Each failed release stays
+// owed until its own record's expiry; past it, a retry that still fails
+// retires the obligation (ErrReleaseLapsed) rather than retrying forever.
+// Two acquisitions in a row, both failing to release, across that expiry.
+func TestAnOwedReleaseIsRetiredOnceItsRecordHasExpired(t *testing.T) {
+ stub := &deleteStub{statuses: []int{
+ http.StatusForbidden, http.StatusForbidden, http.StatusForbidden, http.StatusForbidden,
+ http.StatusForbidden, http.StatusForbidden, http.StatusForbidden, http.StatusForbidden,
+ }}
+ srv := httptest.NewServer(stub)
+ defer srv.Close()
+ ctx := context.Background()
+ expires := time.Now().Add(300 * time.Millisecond)
+ var held []*Held
+ for i := 0; i < 2; i++ {
+ h := heldExpiring(t, srv, fmt.Sprintf(`"e%d"`, i), expires)
+ if err := h.Release(ctx); err == nil {
+ t.Fatal("Release succeeded against a refusing store")
+ }
+ held = append(held, h)
+ }
+ for _, h := range held {
+ if err := h.RetryRelease(ctx); err == nil || errors.Is(err, ErrReleaseLapsed) {
+ t.Fatalf("a retry before the record's expiry = %v, want the store's refusal", err)
+ }
+ if !h.ReleaseOwed() {
+ t.Fatal("an unexpired record stopped being owed")
+ }
+ }
+ time.Sleep(time.Until(expires) + 50*time.Millisecond)
+ for _, h := range held {
+ if err := h.RetryRelease(ctx); !errors.Is(err, ErrReleaseLapsed) {
+ t.Fatalf("a failed retry past the record's expiry = %v, want ErrReleaseLapsed", err)
+ }
+ if h.ReleaseOwed() {
+ t.Fatal("an expired record is still owed")
+ }
+ }
+}
+
+// heldExpiring is heldAgainst with the record's own expiry set.
+func heldExpiring(t *testing.T, srv *httptest.Server, etag string, expires time.Time) *Held {
+ t.Helper()
+ h := heldAgainst(t, srv, etag)
+ h.rn.mu.Lock()
+ h.rn.lease.ExpiresAt = expires
+ h.rn.mu.Unlock()
+ return h
+}
diff --git a/internal/provision/plan14_legacy_identity_test.go b/internal/provision/plan14_legacy_identity_test.go
index 532c38815..fe0ec9bb3 100644
--- a/internal/provision/plan14_legacy_identity_test.go
+++ b/internal/provision/plan14_legacy_identity_test.go
@@ -317,3 +317,80 @@ func TestPlan14Zddr4OverlayPreservesLegacySeal(t *testing.T) {
t.Fatal("unapproved legacy fingerprint accepted")
}
}
+
+// Census ruling 292 (amadan #292, platform-1e, extended): the edge-restart
+// lease overlay, loaded after reserved285, the last overlay before it.
+func TestPlan14Overlay292PreservesLegacySeal(t *testing.T) {
+ root := censusRoot(t)
+ inv := rootReadInventory{Entries: map[string]classifiedRootRead{}, Coverage: map[string]string{}}
+ for i, name := range []string{"plan14-root-reads.tsv", "plan14-root-reads-moves.tsv", "plan14-root-reads-merge.tsv", "plan14-root-reads-fix1.tsv", "plan14-root-reads-fix2.tsv", "plan14-root-reads-merge3.tsv", "plan14-root-reads-fix4.tsv", "plan14-root-reads-fix5.tsv", "plan14-root-reads-fix6.tsv", "plan14-root-reads-zddr3.tsv", "plan14-root-reads-hotfix281.tsv", "plan14-root-reads-zddr4.tsv", "plan14-root-reads-reserved285.tsv"} {
+ if err := loadRootReadInventory(root+"/internal/provision/testdata/"+name, &inv, i > 0); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if errs := checkRootReadLegacy(inv, "e736a5e2652cc546db02c5a385ae7da9fa8a4313d3064f6b431642431974d65d"); len(errs) != 0 {
+ t.Fatal(errs)
+ }
+ original := make(map[string]bool, len(inv.Legacy))
+ for hash := range inv.Legacy {
+ original[hash] = true
+ }
+ if err := loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-292.tsv", &inv, true); err != nil {
+ t.Fatal(err)
+ }
+ for hash := range original {
+ if !inv.Legacy[hash] {
+ t.Fatalf("292 removed original freeze %s", hash)
+ }
+ }
+ if len(inv.Legacy) != len(original)+12 {
+ t.Fatal("census ruling 292 permits exactly twelve new frozen declarations")
+ }
+ // Census ruling 292 seal: eight changed activator rows (begin,
+ // BeginSidecarRequest, doWake, sweepOnce, wake, spawnUnitFrom,
+ // cleanupFailedWake, reapUnprovenWake), three new proof units
+ // (ExecBackend.LegacyRunGone, SplitBackend.LegacyRunGone,
+ // legacyRunGone) and instance-control's Server declaration.
+ const seal = "3b2506dd2ab3986d0df0ceda7f596e5a46ea4d1dfbc17b1d0fd8d5f176e8038d"
+ if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 {
+ t.Fatal(errs)
+ }
+ agent := []string{
+ "internal/activator/activator.go:Service.BeginSidecarRequest",
+ "internal/activator/activator.go:Service.begin",
+ "internal/activator/activator.go:Service.doWake",
+ "internal/activator/activator.go:Service.sweepOnce",
+ "internal/activator/activator.go:Service.wake",
+ "internal/activator/backend_exec.go:ExecBackend.spawnUnitFrom",
+ "internal/activator/backend_exec.go:ExecBackend.LegacyRunGone",
+ "internal/activator/backend_unit.go:SplitBackend.LegacyRunGone",
+ "internal/activator/containment.go:Service.legacyRunGone",
+ "internal/activator/maintenance.go:Service.cleanupFailedWake",
+ "internal/activator/opset.go:Service.reapUnprovenWake",
+ }
+ const boundary = "Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record."
+ const rootClause = "Boundary: root's legacy start/stop serialisation per host: a stop fences, cancels and joins that host's in-flight legacy starts before its systemd stop; stop-only direction, no new start, path or read."
+ check := func(key, rev3, suffix string) {
+ t.Helper()
+ e, ok := inv.Entries[key]
+ if !ok || e.Class != "legacy" || e.Task != "accepted-legacy" || !strings.HasPrefix(e.Basis, "accepted-legacy;rev3="+rev3+";") || !strings.HasSuffix(e.Basis, suffix) {
+ t.Fatalf("wrong 292 disposition for %s: %+v", key, e)
+ }
+ if f, ok := inv.Frozen[key]; !ok || rootReadLegacyFingerprint(f) != rootReadLegacyFingerprint(e) {
+ t.Fatalf("%s is not frozen as classified", key)
+ }
+ }
+ for _, key := range agent {
+ check(key, "census-ruling-292", boundary)
+ }
+ check("internal/instancectl/server.go:type Server", "census-ruling-292,J2,J4", rootClause)
+ for _, key := range []string{"internal/activator/activator.go:Service.begin", "internal/activator/activator.go:Service.wake"} {
+ if !strings.Contains(inv.Entries[key].Basis, "claim wakes are cancelled at shutdown with swaps off, matching the fence path; stop-only, and the lease is released.") {
+ t.Fatalf("%s lacks the claims note", key)
+ }
+ }
+ inv.Legacy[strings.Repeat("0", 64)] = true
+ if errs := checkRootReadLegacy(inv, seal); len(errs) == 0 {
+ t.Fatal("unapproved legacy fingerprint accepted")
+ }
+}
diff --git a/internal/provision/plan14_root_reads_test.go b/internal/provision/plan14_root_reads_test.go
index d1c2bb971..0d0fd0ab5 100644
--- a/internal/provision/plan14_root_reads_test.go
+++ b/internal/provision/plan14_root_reads_test.go
@@ -674,6 +674,16 @@ func TestPlan14RootReadCensus(t *testing.T) {
if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-reserved285.tsv"), &inv, true); err != nil {
t.Fatal(err)
}
+ if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-292.tsv"), &inv, true); err != nil {
+ t.Fatal(err)
+ }
+ // Census ruling 292 (amadan #292, platform-1e, extended): the agent's
+ // shutdown gates, the legacy run unit's termination proof, the retried
+ // conditional lease release, and root's per-host ordering of legacy run
+ // starts against stops (type Server, J2,J4). Eight changed activator
+ // rows, three new proof units and the reclassified instance-control
+ // Server declaration add twelve frozen declarations.
+ seal = "3b2506dd2ab3986d0df0ceda7f596e5a46ea4d1dfbc17b1d0fd8d5f176e8038d"
errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...)
errs = append(errs, checkRootReadDeclarations(got, inv)...)
if len(errs) > 0 {
diff --git a/internal/provision/testdata/plan14-root-reads-292.tsv b/internal/provision/testdata/plan14-root-reads-292.tsv
new file mode 100644
index 000000000..95aa8f35a
--- /dev/null
+++ b/internal/provision/testdata/plan14-root-reads-292.tsv
@@ -0,0 +1,30 @@
+# Plan 14 census ruling 292 (amadan #292, the edge-restart lease strand; Astra rounds 1 to 3): the agent's shutdown gates, the legacy run unit's termination proof before an aborted start's lease goes, the retried conditional lease release, and root's per-host ordering of legacy run starts against stops. Approved by platform-1e as census-ruling-292, extended for the proof units and the instance-control rows.
+# Agent-side boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. Root-side boundary: root's legacy start/stop serialisation per host: a stop fences, cancels and joins that host's in-flight legacy starts before its systemd stop; stop-only direction, no new start, path or read. If any agent-side unit here ever runs as root, or any unit reads beyond that boundary, it stops being legacy and needs the Plan 14 owner's ruling. Earlier overlays are immutable.
+@legacy da7f39bcd686e64850d4ecf625bc938a416c621b77f5d1103782de0c99e03e56 internal/activator/activator.go Service.BeginSidecarRequest legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; once Shutdown has begun, a sidecar-marked request is answered as asleep and an exited child is left to Shutdown's own stop, never recycled as an unexpected exit. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Host@259,registry:Host@333 93734e46bc8257b10ed6cb3c3af4146165a92a83d50ce1b44e0d030356d39b6b
+internal/activator/activator.go Service.BeginSidecarRequest legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; once Shutdown has begun, a sidecar-marked request is answered as asleep and an exited child is left to Shutdown's own stop, never recycled as an unexpected exit. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Host@259,registry:Host@333 93734e46bc8257b10ed6cb3c3af4146165a92a83d50ce1b44e0d030356d39b6b
+@legacy 9ab43c1b1c04a27a3efa4a2f24b8030c4cfdcbd9e2a0c225d6e0eebc02011927 internal/activator/activator.go Service.begin legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; once Shutdown has begun, a request is refused (errShuttingDown) on every pass of its loop, before it can wake or recycle anything. Claims: claim wakes are cancelled at shutdown with swaps off, matching the fence path; stop-only, and the lease is released. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Host@1702,registry:Host@1781,registry:Host@326,registry:Host@910,whole:Route@1711,whole:Route@1718 2f4449d5665d0e84c17a6618e2aa4cd6394808b169c7f05a3f27f634f626b010
+internal/activator/activator.go Service.begin legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; once Shutdown has begun, a request is refused (errShuttingDown) on every pass of its loop, before it can wake or recycle anything. Claims: claim wakes are cancelled at shutdown with swaps off, matching the fence path; stop-only, and the lease is released. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Host@1702,registry:Host@1781,registry:Host@326,registry:Host@910,whole:Route@1711,whole:Route@1718 2f4449d5665d0e84c17a6618e2aa4cd6394808b169c7f05a3f27f634f626b010
+@legacy b6bfb87b76a780fe09bccdec6637427719cc2cf891a8ddd946c2802415d6de00 internal/activator/activator.go Service.doWake legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; a failed start of the legacy run unit is marked for the termination proof before its lease goes, and every bail-out gives the lease back through giveBack, which keeps a failed release for retry. RetryRelease only repeats the same conditional delete against the same ETag, never an unconditional one. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:DBPath@1032,registry:DBPath@1048,registry:DBPath@1175,registry:DBPath@1186,registry:DBPath@20307,registry:Host@1014,registry:Host@1166,registry:Host@1277,registry:Host@174,registry:Host@2137,registry:Host@3858,registry:Host@861,registry:Host@936,registry:Replication@20342,registry:Seed@11113,registry:Socket@1061,registry:Socket@1077,registry:Socket@1200,registry:Socket@1211,whole:Route@10018,whole:Route@11214,whole:Route@11285,whole:Route@1231,whole:Route@1237,whole:Route@1303,whole:Route@1311,whole:Route@13669,whole:Route@15527,whole:Route@159,whole:Route@16173,whole:Route@17480,whole:Route@17634,whole:Route@17990,whole:Route@18084,whole:Route@18756,whole:Route@19755,whole:Route@19903,whole:Route@2027,whole:Route@20416,whole:Route@21314,whole:Route@21409,whole:Route@3481,whole:Route@3740,whole:Route@3810,whole:Route@3816,whole:Route@3885,whole:Route@3893,whole:Route@4681,whole:Route@5680,whole:Route@6713,whole:Route@7594,whole:Route@9176,whole:Route@9368,whole:Route@959 7e6b169869d79b18322cc09d5a7cdb48eb207e9de12d55363ae1295120d5a712
+internal/activator/activator.go Service.doWake legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; a failed start of the legacy run unit is marked for the termination proof before its lease goes, and every bail-out gives the lease back through giveBack, which keeps a failed release for retry. RetryRelease only repeats the same conditional delete against the same ETag, never an unconditional one. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:DBPath@1032,registry:DBPath@1048,registry:DBPath@1175,registry:DBPath@1186,registry:DBPath@20307,registry:Host@1014,registry:Host@1166,registry:Host@1277,registry:Host@174,registry:Host@2137,registry:Host@3858,registry:Host@861,registry:Host@936,registry:Replication@20342,registry:Seed@11113,registry:Socket@1061,registry:Socket@1077,registry:Socket@1200,registry:Socket@1211,whole:Route@10018,whole:Route@11214,whole:Route@11285,whole:Route@1231,whole:Route@1237,whole:Route@1303,whole:Route@1311,whole:Route@13669,whole:Route@15527,whole:Route@159,whole:Route@16173,whole:Route@17480,whole:Route@17634,whole:Route@17990,whole:Route@18084,whole:Route@18756,whole:Route@19755,whole:Route@19903,whole:Route@2027,whole:Route@20416,whole:Route@21314,whole:Route@21409,whole:Route@3481,whole:Route@3740,whole:Route@3810,whole:Route@3816,whole:Route@3885,whole:Route@3893,whole:Route@4681,whole:Route@5680,whole:Route@6713,whole:Route@7594,whole:Route@9176,whole:Route@9368,whole:Route@959 7e6b169869d79b18322cc09d5a7cdb48eb207e9de12d55363ae1295120d5a712
+@legacy 4f1feca518693fadcefc38fae71c707ad244308ae62f88db3f6af44f538ee9c1 internal/activator/activator.go Service.sweepOnce legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; once Shutdown has begun, the sweep returns before it recycles, due-wakes or starts anything. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Hibernate@12146,registry:Hibernate@21304,registry:Hibernate@5109,registry:Hibernate@6767,registry:Hibernate@9147,registry:Host@21373,registry:Host@21431,registry:Host@21824,registry:Host@22555,registry:Host@22637,registry:Host@23702,registry:Host@23881,registry:Host@23930,registry:NextDue@22510,registry:Status@21320,whole:Route@12148,whole:Route@14068,whole:Route@15778,whole:Route@15968,whole:Route@16261,whole:Route@16433,whole:Route@16580,whole:Route@20791,whole:Route@22224,whole:Route@22564,whole:Route@22571,whole:Route@23517,whole:Route@23914,whole:Route@5111,whole:Route@6769,whole:Route@9149 55664900e468c0ef1b3bef3bee3699a7341622ebbd94634da6820d0cc4a9d579
+internal/activator/activator.go Service.sweepOnce legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; once Shutdown has begun, the sweep returns before it recycles, due-wakes or starts anything. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Hibernate@12146,registry:Hibernate@21304,registry:Hibernate@5109,registry:Hibernate@6767,registry:Hibernate@9147,registry:Host@21373,registry:Host@21431,registry:Host@21824,registry:Host@22555,registry:Host@22637,registry:Host@23702,registry:Host@23881,registry:Host@23930,registry:NextDue@22510,registry:Status@21320,whole:Route@12148,whole:Route@14068,whole:Route@15778,whole:Route@15968,whole:Route@16261,whole:Route@16433,whole:Route@16580,whole:Route@20791,whole:Route@22224,whole:Route@22564,whole:Route@22571,whole:Route@23517,whole:Route@23914,whole:Route@5111,whole:Route@6769,whole:Route@9149 55664900e468c0ef1b3bef3bee3699a7341622ebbd94634da6820d0cc4a9d579
+@legacy eba2b8a74d4615e23626ac96c87ef8b192de7f42c0a430f978adafeec871db6f internal/activator/activator.go Service.wake legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; a wake that starts after Shutdown began is cancelled before its lease, and one that finishes is stopped, with swaps on or off. Claims: claim wakes are cancelled at shutdown with swaps off, matching the fence path; stop-only, and the lease is released. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Socket@6807,registry:Socket@6968,whole:Route@2091,whole:Route@2148,whole:Route@5344,whole:Route@6760,whole:Route@6809,whole:Route@6970,whole:Route@8209,whole:Route@8230 25e8f2be15ec434d2bc414e0d144a3f77bd433a81b08f5a0be5be212df24071c
+internal/activator/activator.go Service.wake legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; a wake that starts after Shutdown began is cancelled before its lease, and one that finishes is stopped, with swaps on or off. Claims: claim wakes are cancelled at shutdown with swaps off, matching the fence path; stop-only, and the lease is released. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Socket@6807,registry:Socket@6968,whole:Route@2091,whole:Route@2148,whole:Route@5344,whole:Route@6760,whole:Route@6809,whole:Route@6970,whole:Route@8209,whole:Route@8230 25e8f2be15ec434d2bc414e0d144a3f77bd433a81b08f5a0be5be212df24071c
+@legacy aa5ac6f18801ee88c600b00d7f964d5d5c68f4c1c9a745066dabd5fd9eb04c79 internal/activator/backend_exec.go ExecBackend.spawnUnitFrom legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; what changed: a failed start of the original run unit (generation 0) that root may still carry out is marked legacyStartUncertain, its text unchanged, so the failed wake's cleanup proves the unit stopped before the lease goes. It does not widen what is spawned or how. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Host@1071,registry:Host@1473,registry:Host@1576,registry:Host@1994,registry:Host@2474,registry:Host@3400,registry:Host@3986,registry:Host@4256,registry:Host@4476,registry:Host@4661,registry:Host@5056,registry:Host@5337,registry:Host@5399,registry:Host@5552,registry:Host@5591,registry:Host@5870,registry:Host@5964,whole:Route@1167,whole:Route@1402,whole:Route@194,whole:Route@2678,whole:Route@4309,whole:Route@5490,whole:Route@5649,whole:Route@758,whole:Route@856 036895b3ad78d31d9b788f85ce11736ed21adadfa3d6a04683de5cef1b3fe963
+internal/activator/backend_exec.go ExecBackend.spawnUnitFrom legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; what changed: a failed start of the original run unit (generation 0) that root may still carry out is marked legacyStartUncertain, its text unchanged, so the failed wake's cleanup proves the unit stopped before the lease goes. It does not widen what is spawned or how. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Host@1071,registry:Host@1473,registry:Host@1576,registry:Host@1994,registry:Host@2474,registry:Host@3400,registry:Host@3986,registry:Host@4256,registry:Host@4476,registry:Host@4661,registry:Host@5056,registry:Host@5337,registry:Host@5399,registry:Host@5552,registry:Host@5591,registry:Host@5870,registry:Host@5964,whole:Route@1167,whole:Route@1402,whole:Route@194,whole:Route@2678,whole:Route@4309,whole:Route@5490,whole:Route@5649,whole:Route@758,whole:Route@856 036895b3ad78d31d9b788f85ce11736ed21adadfa3d6a04683de5cef1b3fe963
+@legacy 64b4b59426a6ca65d46cd603776efb3b340f5353d7b3479a3c934c85c644b5a4 internal/activator/backend_exec.go ExecBackend.LegacyRunGone legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: new unit; stops the host's original run unit through instance-control's existing /stop and proves it gone from systemctl show state and cgroup.events, as the exit observer reads. The termination proof only decides release-or-keep, and never starts, wakes or admits anything; unknown or error means keep. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Host@174,registry:Host@225,registry:Host@268,registry:Host@623 59ee0ef2591b7596fdeedd65786dac0b671e54140055247c5caef3a576347f3d
+internal/activator/backend_exec.go ExecBackend.LegacyRunGone legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: new unit; stops the host's original run unit through instance-control's existing /stop and proves it gone from systemctl show state and cgroup.events, as the exit observer reads. The termination proof only decides release-or-keep, and never starts, wakes or admits anything; unknown or error means keep. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Host@174,registry:Host@225,registry:Host@268,registry:Host@623 59ee0ef2591b7596fdeedd65786dac0b671e54140055247c5caef3a576347f3d
+@legacy 28136608ac1b1e67bf0844caebd2011456b6bf560a984e925a026cf6ef403f47 internal/activator/backend_unit.go SplitBackend.LegacyRunGone legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: new unit; forwards an exec row to the exec half's LegacyRunGone and refuses a unit-backed row, reading Backing as SplitBackend.Layout does. The termination proof only decides release-or-keep, and never starts, wakes or admits anything; unknown or error means keep. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. file:Exec@240,registry:Backing@89,registry:Host@200,registry:Host@305,whole:Route@364 13f8a08cb85de111beae3e48dfb48fe64a2269f7d6fbb0539d7e68dc674f16aa
+internal/activator/backend_unit.go SplitBackend.LegacyRunGone legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: new unit; forwards an exec row to the exec half's LegacyRunGone and refuses a unit-backed row, reading Backing as SplitBackend.Layout does. The termination proof only decides release-or-keep, and never starts, wakes or admits anything; unknown or error means keep. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. file:Exec@240,registry:Backing@89,registry:Host@200,registry:Host@305,whole:Route@364 13f8a08cb85de111beae3e48dfb48fe64a2269f7d6fbb0539d7e68dc674f16aa
+@legacy 7549e1150abccb1e9f62229dbc25c435adb23bf1a08e013273b9a68c06e9462b internal/activator/containment.go Service.legacyRunGone legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: new unit; runs the backend's termination proof for the route under its own budget, never past Shutdown's. The termination proof only decides release-or-keep, and never starts, wakes or admits anything; unknown or error means keep. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. whole:Route@573 680bd2d7f55e99e5aef3b3ec89a00465811b87dc2a68ad8e4b3364443156f43f
+internal/activator/containment.go Service.legacyRunGone legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: new unit; runs the backend's termination proof for the route under its own budget, never past Shutdown's. The termination proof only decides release-or-keep, and never starts, wakes or admits anything; unknown or error means keep. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. whole:Route@573 680bd2d7f55e99e5aef3b3ec89a00465811b87dc2a68ad8e4b3364443156f43f
+@legacy 0ea3b4516b2fa8a7b311af9bc14a061666175ad7b05942c32f76d4982c2ba3af internal/activator/maintenance.go Service.cleanupFailedWake legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; an aborted legacy run unit start is proven stopped before its lease goes, else retained with its lease; a failed release is kept for retry. The termination proof only decides release-or-keep, and never starts, wakes or admits anything; unknown or error means keep. RetryRelease only repeats the same conditional delete against the same ETag, never an unconditional one. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Host@1824,registry:Host@2031,registry:Host@2666,registry:Host@3006,registry:Host@3618,whole:Route@1691,whole:Route@2056,whole:Route@2060,whole:Route@2348,whole:Route@3031,whole:Route@3035 533ae97a1d2801faf9b0ef74eba10c268a7bac158e289aa296880fddf757c3eb
+internal/activator/maintenance.go Service.cleanupFailedWake legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; an aborted legacy run unit start is proven stopped before its lease goes, else retained with its lease; a failed release is kept for retry. The termination proof only decides release-or-keep, and never starts, wakes or admits anything; unknown or error means keep. RetryRelease only repeats the same conditional delete against the same ETag, never an unconditional one. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. registry:Host@1824,registry:Host@2031,registry:Host@2666,registry:Host@3006,registry:Host@3618,whole:Route@1691,whole:Route@2056,whole:Route@2060,whole:Route@2348,whole:Route@3031,whole:Route@3035 533ae97a1d2801faf9b0ef74eba10c268a7bac158e289aa296880fddf757c3eb
+@legacy 0a1e28f9d86865b0e22a7e0fc5562e93dfce43c985abd065f76e12d1576d0f2d internal/activator/opset.go Service.reapUnprovenWake legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; a retained legacy start is retried through the same termination proof, and a failed raw release is kept for retry. The termination proof only decides release-or-keep, and never starts, wakes or admits anything; unknown or error means keep. RetryRelease only repeats the same conditional delete against the same ETag, never an unconditional one. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. whole:Route@439,whole:Route@493 8543c417e6a98553b8500975d41679d8257495d6c077116aab2ada1d50aba647
+internal/activator/opset.go Service.reapUnprovenWake legacy accepted-legacy accepted-legacy;rev3=census-ruling-292;basis=Census ruling 292: changed legacy row; a retained legacy start is retried through the same termination proof, and a failed raw release is kept for retry. The termination proof only decides release-or-keep, and never starts, wakes or admits anything; unknown or error means keep. RetryRelease only repeats the same conditional delete against the same ETag, never an unconditional one. Boundary: agent-side shutdown gates running as carlos; they refuse or stop new work once shutdown begins, and a cancelled wake releases its lease; no new read and no root decision; plus the legacy run unit's termination proof (systemctl show state and cgroup.events, as the exit observer reads), and a retried conditional lease delete of the same record. whole:Route@439,whole:Route@493 8543c417e6a98553b8500975d41679d8257495d6c077116aab2ada1d50aba647
+@legacy 80ed90a720baa7c64ac4413dff85f60f86bb78848f29c96ab448573a8cc13873 internal/instancectl/server.go type Server legacy accepted-legacy accepted-legacy;rev3=census-ruling-292,J2,J4;basis=Census ruling 292: changed row, root-run (was diagnostic H3); the declaration gains the per-host legacy start attempts, stop counts, accept-order barriers, the connection counter and a test join budget, and still executes no decision. Boundary: root's legacy start/stop serialisation per host: a stop fences, cancels and joins that host's in-flight legacy starts before its systemd stop; stop-only direction, no new start, path or read. file:Statfs@2708 3d2ba16fd871c32390edd658501aca37d335e881dbeb5d09b050d3f013d3f734
+internal/instancectl/server.go type Server legacy accepted-legacy accepted-legacy;rev3=census-ruling-292,J2,J4;basis=Census ruling 292: changed row, root-run (was diagnostic H3); the declaration gains the per-host legacy start attempts, stop counts, accept-order barriers, the connection counter and a test join budget, and still executes no decision. Boundary: root's legacy start/stop serialisation per host: a stop fences, cancels and joins that host's in-flight legacy starts before its systemd stop; stop-only direction, no new start, path or read. file:Statfs@2708 3d2ba16fd871c32390edd658501aca37d335e881dbeb5d09b050d3f013d3f734
+# Envelopes, census ruling 292: each changed root source envelope below is accepted only because the units inside it are the ones listed above. internal/lease adds Held.ReleaseOwed and Held.RetryRelease: RetryRelease only repeats the same conditional delete against the same ETag, never an unconditional one. internal/instancectl adds legacystart.go (root, J2,J4): root's legacy start/stop serialisation per host: a stop fences, cancels and joins that host's in-flight legacy starts before its systemd stop; stop-only direction, no new start, path or read.
+@coverage internal/activator 88538744825ca92fbb551105a60cc823d9e86b9ee6ecaedff9365f2f68a87710
+@coverage internal/lease 7ca044ef2fac40a860e9e80c83834929227fbefe32d55b1e61a60893ae129cb4
+@coverage internal/instancectl a4317dea0de2980ce53afe3c0cfc0f0b17bdefda683073674e63e21cd980fc73