carlos / platform Public

Wake latency pass 2b: one-round-trip lease, build-state write on its cached copy, config layers read together (census-ruling-wake-latency-3)

Squashed from wake-latency-3 into main.

claude 8 hours ago 9c535162fb703c8971ebe7c87336a18e9ebf60cb parent a5f0509

15 files changed, +651 −20

docs/superpowers/notes/census-proposal-wake-latency-3.md +13 −0
internal/appconfig/environments.go +23 −3
internal/appconfig/instances.go +17 −2
internal/appconfig/loadconcurrent_test.go +83 −0
internal/host/buildstate.go +87 −9
internal/host/buildstate_cas_test.go +126 −0
internal/host/buildstate_test.go +5 −2
internal/lease/createlease.go +78 −0
internal/lease/endpoint_test.go +5 −2
internal/lease/fastacquire_test.go +174 −0
internal/lease/lease.go +4 −1
internal/provision/plan9_b8_census_test.go +1 −1
internal/provision/testdata/plan14-root-reads-chain.tsv +1 −0
internal/provision/testdata/plan14-root-reads-wake-latency-3.tsv +6 −0
internal/provision/wake_latency_3_census_test.go +28 −0

CI failed — run details

CI log

=== gate ===
.amadan/test-gate.sh
=== launchctx (0 s)
ok  	amadan.net/carlos/platform.git/launchctx	0.012s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
=== vet (18 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (172 s)
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.769s
ok  	github.com/carlosframework/platform/cmd/moveprobe	12.104s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.785s
2026/10/24 22:53:51 activator: hook.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: nohook.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: ladder0.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: ladder1.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: ladder2.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: ladder3.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: fresh.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: restored.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: noseed.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: badseed.example.com: seed for badseed.example.com: activator: seed unproven; processes remain: processes remain after a successful seed: still running; leaving the lease to its TTL
2026/10/24 22:53:51 activator: unproven.example.com: unproven.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone; leaving the lease to its TTL
2026/10/24 22:53:51 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/24 22:53:51 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof681284357/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof681284357/004/next-wake.example.com.db
2026/10/24 22:53:51 activator: next-wake.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof681284357/004/next-wake.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 22:53:51 activator: next-wake.example.com: seeding next-wake.example.com: exit status 1: activator: seed unproven; processes remain (the seed's processes remain after stopping it; its database was left in place: cgroup: strict containment unsupported); leaving the lease to its TTL
2026/10/24 22:53:51 activator: next-wake.example.com: next-wake.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup: strict containment unsupported; leaving the lease to its TTL
2026/10/24 22:53:51 activator: next-wake.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 22:53:51 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/24 22:53:51 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof681284357/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof681284357/004/next-wake.example.com.db
2026/10/24 22:53:51 activator: seeded next-wake.example.com
2026/10/24 22:53:51 activator: next-wake.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: instance one.example.com exited unexpectedly — recycling
2026/10/24 22:53:51 activator: one.example.com hibernated
2026/10/24 22:53:51 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: LEASE HELD for held.example.com by "old-incarnation" — a previous incarnation hasn't released; refusing to wake (retry after 7s)
2026/10/24 22:53:51 activator: hib.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: tick.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: visitor.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: asleep.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: fenced.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: imminent.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: overdue.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: two.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: two.example.com hibernated
2026/10/24 22:53:51 activator: one.example.com hibernated
2026/10/24 22:53:51 activator: b.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:51 activator: b.example.com hibernated
2026/10/24 22:53:51 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:53:51 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:53:52 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
--- FAIL: TestWorkerReadyEndsSettleWindowEarly (0.01s)
    activator_test.go:3689: the agent socket would REFUSE this child's ready call: it presents "", WorkerToken says ""
2026/10/24 22:53:52 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:53:52 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:53:52 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:53:52 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:53:52 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 22:53:52 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:53:52 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:53:52 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 22:53:52 activator: bot.x.internal hibernated
2026/10/24 22:53:52 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendStartMergesConfigEnvUnderOwnEnv3932276387/002/merge-precedence.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/24 22:53:53 activator: purged 1 cached response(s) for deploy.example.com on spawn
2026/10/24 22:53:53 activator: restored restored.example.com from s3://test-bucket/restored.example.com
2026/10/24 22:53:53 activator: damaged.example.com restored CORRUPT from s3://test-bucket/damaged.example.com: integrity_check: file is not a database (26) — quarantined at /tmp/TestExecBackendRestoreCorruptReplicaFailsAndQuarantines2481969400/002/damaged.example.com.db.corrupt, instance NOT started
2026/10/24 22:53:53 activator: restored damaged.example.com from s3://test-bucket/damaged.example.com
2026/10/24 22:53:53 activator: no replica for brand-new.example.com in s3://test-bucket/brand-new.example.com — booting fresh
2026/10/24 22:53:53 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1926127616/002/seeded.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/24 22:53:53 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1926127616/001/live/seeded.example.com seed --fixtures --db /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1926127616/002/seeded.example.com.db
2026/10/24 22:53:53 activator: seeded seeded.example.com
2026/10/24 22:53:53 activator: seeding custom-db.example.com: /tmp/TestExecBackendSeedHonoursRouteDBPath3597313170/001/live/custom-db.example.com seed --db /tmp/TestExecBackendSeedHonoursRouteDBPath3597313170/004/elsewhere.db
2026/10/24 22:53:53 activator: seeded custom-db.example.com
2026/10/24 22:53:53 activator: seeding badseed.example.com: /tmp/TestExecBackendSeedNonZeroExitFails3512604343/001/live/badseed.example.com seed --db /tmp/TestExecBackendSeedNonZeroExitFails3512604343/002/badseed.example.com.db
schema is corrupt
2026/10/24 22:53:53 activator: badseed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 22:53:53 activator: seeding hangs.example.com: /tmp/TestExecBackendSeedIsTimeBounded2385633404/001/live/hangs.example.com seed --db /tmp/TestExecBackendSeedIsTimeBounded2385633404/002/hangs.example.com.db
2026/10/24 22:53:53 activator: hangs.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 22:53:53 activator: seeding crashed.example.com: /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB792866347/001/live/crashed.example.com seed --db /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB792866347/002/crashed.example.com.db
died mid-schema
2026/10/24 22:53:53 activator: crashed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 22:53:53 activator: seeding existing-db.example.com: /tmp/TestExecBackendSeedFailureKeepsPreexistingDB317262822/001/live/existing-db.example.com seed --db /tmp/TestExecBackendSeedFailureKeepsPreexistingDB317262822/002/existing-db.example.com.db
nope
2026/10/24 22:53:53 activator: existing-db.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 22:53:53 activator: seeding leaky-seed.example.com: /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild2056794354/001/live/leaky-seed.example.com seed --db /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild2056794354/002/leaky-seed.example.com.db
2026/10/24 22:53:53 activator: seeded leaky-seed.example.com (it exited 0 but left a child holding its output pipe open for more than 200ms)
2026/10/24 22:53:54 activator: a.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:53:54 activator: b.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:54 activator: sidecar for a.example.com was down — restarted (backoff now 2s)
2026/10/24 22:53:54 activator: h.example.com: route refreshed: db "/tmp/h.example.com.db" -> "/data/instances/h.example.com/h.example.com.db" socket "/tmp/nope.sock" -> "/run/carlos/instances/h.example.com/app.sock"
2026/10/24 22:53:54 activator: h.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:54 activator: stale.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:53:54 activator: stop stale.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/24 22:53:54 activator: stale.example.com hibernated
2026/10/24 22:53:54 activator: stale.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:53:54 activator: leftover.example.com: the host cgroup holds processes not proven gone: a leftover still populates the host cgroup; leaving the lease to its TTL
2026/10/24 22:53:54 activator: leftover.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:54 activator: old-kernel.example.com: this kernel has no cgroup.kill, so the host cgroup cannot be killed; its leader's exit is the only proof there is
2026/10/24 22:53:54 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/24 22:53:54 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/24 22:53:54 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/24 22:53:54 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/24 22:53:54 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/24 22:53:54 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/24 22:53:54 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/24 22:53:54 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/24 22:53:54 activator: retry.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 22:53:54 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/24 22:53:54 activator: retry.example.com: strict group not closed: cgroup: strict containment refused
2026/10/24 22:53:55 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/24 22:53:55 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 22:53:55 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 22:53:55 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 22:53:55 activator: stuck.example.com: strict group still not cleaned up after 2 retries; spawns of this host stay refused until the agent restarts
2026/10/24 22:53:55 activator: owned.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 22:53:56 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/24 22:53:56 activator: owned.example.com: strict group not closed: cgroup: strict containment refused
2026/10/24 22:53:56 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/24 22:53:56 activator: race.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 22:53:56 activator: transition.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 22:53:56 activator: transition2.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 22:53:57 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/24 22:53:57 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/24 22:53:57 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:57 activator: kass.example.com hibernated
2026/10/24 22:53:57 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:57 activator: kass.example.com idle 11s — hibernating
2026/10/24 22:53:57 activator: kass.example.com hibernated
2026/10/24 22:53:57 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:57 activator: one.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/24 22:53:57 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:53:57 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:53:57 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:53:57 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:53:57 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:53:57 activator: bot.x.internal: v2 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:53:57 activator: worker bot.x.internal NOT respawned (attempt (release)): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/24 22:53:57 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 22:53:57 activator: link.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:57 activator: link.example.com hibernated
2026/10/24 22:53:57 activator: link.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:57 activator: plain-link.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:57 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:53:57 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:01 activator: tenant.test: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:01 activator: tenant.test not started: the executable it hashed (/tmp/rl1399617104/app) is build "v1", but the start names build "v2" (the live link moved between the two reads); the next wake reads it again
2026/10/24 22:54:01 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:01 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 22:54:01 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 22:54:02 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: replica h.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/24 22:54:02 activator: replica h.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/24 22:54:02 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 22:54:02 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: why.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: why.example.com hibernated
2026/10/24 22:54:02 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: c.example.com hibernated
2026/10/24 22:54:02 activator: h.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: h.example.com hibernated
2026/10/24 22:54:02 activator: h.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: h.example.com: run run_a is not proven stopped: cgroup.events: permission denied
2026/10/24 22:54:02 activator: stop h.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 22:54:02 activator: h.example.com hibernated
2026/10/24 22:54:02 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 22:54:02 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 22:54:02 activator: seeding unproven-seed.example.com: /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga1499684685/002/live/unproven-seed.example.com seed --db /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga1499684685/003/unproven-seed.example.com.db
2026/10/24 22:54:02 activator: unproven-seed.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga1499684685/003/unproven-seed.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 22:54:02 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3406351606/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3406351606/003/obligation.example.com.db
2026/10/24 22:54:02 activator: obligation.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3406351606/003/obligation.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 22:54:02 activator: obligation.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 22:54:02 activator: no replica for obligation.example.com in s3://test-bucket/obligation.example.com — booting fresh
2026/10/24 22:54:02 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3406351606/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3406351606/003/obligation.example.com.db
2026/10/24 22:54:02 activator: seeded obligation.example.com
2026/10/24 22:54:02 activator: seeding helper-seed.example.com: /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt1521443115/002/live/helper-seed.example.com seed --db /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt1521443115/003/helper-seed.example.com.db
2026/10/24 22:54:02 activator: seeded helper-seed.example.com
2026/10/24 22:54:02 activator: seeding helper-unproven.example.com: /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven3919958306/002/live/helper-unproven.example.com seed --db /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven3919958306/003/helper-unproven.example.com.db
2026/10/24 22:54:02 activator: helper-unproven.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven3919958306/003/helper-unproven.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 22:54:02 activator: seeding write-ahead.example.com: /tmp/TestASeedRecordsItsObligationBeforeItRuns1886758730/002/live/write-ahead.example.com seed --db /tmp/TestASeedRecordsItsObligationBeforeItRuns1886758730/003/write-ahead.example.com.db
2026/10/24 22:54:02 activator: seeded write-ahead.example.com
2026/10/24 22:54:02 activator: write-ahead.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 22:54:02 activator: no replica for write-ahead.example.com in s3://test-bucket/write-ahead.example.com — booting fresh
2026/10/24 22:54:02 activator: seeding no-room.example.com: /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing3357992152/002/live/no-room.example.com seed --db /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing3357992152/003/no-room.example.com.db
2026/10/24 22:54:02 activator: seeding stuck-wal.example.com: /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation2126259409/002/live/stuck-wal.example.com seed --db /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation2126259409/003/stuck-wal.example.com.db
2026/10/24 22:54:02 activator: seeding preexisting.example.com: /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept4089856099/002/live/preexisting.example.com seed --db /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept4089856099/003/preexisting.example.com.db
2026/10/24 22:54:02 activator: preexisting.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept4089856099/003/preexisting.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 22:54:02 activator: preexisting.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/24 22:54:02 activator: old-kernel-seed.example.com: this kernel has no cgroup.kill; the seed's leader's exit is the only proof there is
2026/10/24 22:54:02 activator: seeding failed-role-seed.example.com: /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB3492010494/002/live/failed-role-seed.example.com seed --db /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB3492010494/003/failed-role-seed.example.com.db
2026/10/24 22:54:02 activator: seeding discharge-fails.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai1626333994/002/live/discharge-fails.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai1626333994/003/discharge-fails.example.com.db
2026/10/24 22:54:02 activator: discharge-fails.example.com: the seed's obligation could not be discharged and its processes are not proven gone; /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai1626333994/003/discharge-fails.example.com.db.seed-unproven stays
2026/10/24 22:54:02 activator: seeding discharge-fails-proven.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur2083901755/002/live/discharge-fails-proven.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur2083901755/003/discharge-fails-proven.example.com.db
2026/10/24 22:54:02 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_B,_l4138021395/003/g/56/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:02 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall3049928940/003/g/57/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:02 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall3049928940/003/g/58/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:02 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: swap.example.com hibernated
2026/10/24 22:54:02 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatholds_pinne2237336648/007/g/60/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:02 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 22:54:02 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:02 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatoverlap_succ4264849386/007/g/62/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:03 activator: h.example.com: started carlos-instance-gen@h.example.com:64.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:03 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: swap.example.com hibernated
2026/10/24 22:54:03 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 22:54:03 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_super2431640071/007/g/71/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:03 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: swap.example.com hibernated
2026/10/24 22:54:03 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartholds_pinned_build,_lea335269827/007/g/73/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:03 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 22:54:03 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_a_fai890514265/007/g/75/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:03 activator: fence.example hibernated
2026/10/24 22:54:03 activator: fence.example hibernated
2026/10/24 22:54:03 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink389079006/003/g/c.x.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink389079006/003/g/c.x.sock: no such file or directory)
2026/10/24 22:54:03 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink389079006/003/g/c.xx.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink389079006/003/g/c.xx.sock: no such file or directory)
2026/10/24 22:54:03 activator: publish generations to /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile1342536114/002/not-a-dir/zdd/generations.json: open directory /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile1342536114/002/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 22:54:03 activator: cur.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: off.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: plain.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: ident.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: ident.example.com hibernated
2026/10/24 22:54:03 activator: ident.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: instance ident.example.com exited unexpectedly — recycling
2026/10/24 22:54:03 activator: ident.example.com hibernated
2026/10/24 22:54:03 activator: ident.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: due.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: due.example.com hibernated
2026/10/24 22:54:03 activator: due.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: due.example.com hibernated
2026/10/24 22:54:03 activator: ret.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: ret.example.com hibernated
2026/10/24 22:54:03 activator: ret.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:03 activator: ret.example.com hibernated
2026/10/24 22:54:03 activator: bot.x.internal: woke in 20ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 20ms, replicate 0s
2026/10/24 22:54:03 activator: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.example.com: a generation socket path would exceed 107 bytes; generation 9 keeps the canonical socket
2026/10/24 22:54:03 activator: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.example.com: a generation socket path would exceed 107 bytes; generation 9 keeps the canonical socket
2026/10/24 22:54:04 activator: one.example.com: generation 1089 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:04 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:04 activator: one.example.com hibernated
2026/10/24 22:54:04 activator: purged 1 cached response(s) for flat.example.com on spawn
2026/10/24 22:54:04 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile1490922792/001/versions/vA/app seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile1490922792/002/seedexe.example.com.db
2026/10/24 22:54:04 activator: seeded seedexe.example.com
2026/10/24 22:54:04 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile1490922792/001/live/seedexe.example.com seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile1490922792/002/seedexe.example.com.db
2026/10/24 22:54:04 activator: seeded seedexe.example.com
2026/10/24 22:54:04 activator: seeding h.example.com: /tmp/TestNestedSeedExeNamesTheRecordedFile666349526/001/live/h.example.com seed --db /tmp/nb3377739799/data/instances/h.example.com/h.example.com.db
2026/10/24 22:54:04 activator: seeded h.example.com
2026/10/24 22:54:04 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing1566803766/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/24 22:54:04 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing1566803766/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/24 22:54:04 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing1566803766/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/24 22:54:04 activator: seeding h.example.com: /tmp/TestADeclinedNestedSeedRunsThePinnedFile2304967000/001/live/h.example.com seed --db /tmp/nb1930763007/data/instances/h.example.com/h.example.com.db
2026/10/24 22:54:04 activator: seeded h.example.com
2026/10/24 22:54:04 activator: seeding tier.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed2361979466/002/versions/vA/app seed --db /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed2361979466/003/tier.example.com.db
2026/10/24 22:54:04 activator: tier.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed2361979466/003/tier.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 22:54:04 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=1073741824 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:04 activator: h.example.com: carlos-instance-gen@h.example.com:6.service launched in 0s, healthy 0s later
2026/10/24 22:54:04 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=1073741824 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:04 activator: h.example.com: carlos-instance-gen@h.example.com:6.service launched in 0s, healthy 0s later
2026/10/24 22:54:04 activator: seeding h.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTiernested_seed2627029867/001/live/h.example.com seed --db /tmp/nb1651362546/data/instances/h.example.com/h.example.com.db
2026/10/24 22:54:04 activator: seeded h.example.com
2026/10/24 22:54:04 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsthe_configured_k867720472/001/shop.example.com.env: the platform mints none while it is
2026/10/24 22:54:04 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsan_own_key_pres2049212102/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/24 22:54:04 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestFirstStartWaitsForTheConfiguredKey633661320/002/shop.example.com.env: the platform mints none while it is
2026/10/24 22:54:04 activator: control listener for shop.example.com: listen unix /tmp/TestFirstStartWaitsForTheConfiguredKey633661320/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:04 activator: CARLOS_INSTANCE_KEY in /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptinto_the_own_file366322513/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/24 22:54:04 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptdelivered_in_the_co1156728768/001/shop.example.com.env: the platform mints none while it is
2026/10/24 22:54:04 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration1389029747/001/shop.example.com.env: the platform mints none while it is
2026/10/24 22:54:04 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration1389029747/002/shop.example.com.env: the platform mints none while it is
2026/10/24 22:54:04 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyLeavesAFileWithTwoKeysAlone3353795265/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/24 22:54:04 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistse1017988048/002/shop.example.com.env: the platform mints none while it is
2026/10/24 22:54:04 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistsn1577521239/002/shop.example.com.env: the platform mints none while it is
2026/10/24 22:54:04 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance1819216696/001/live/seeded.example.com seed --db /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance1819216696/002/seeded.example.com.db
2026/10/24 22:54:04 activator: seeded seeded.example.com
2026/10/24 22:54:04 activator: restored sleepy.example.com from s3://test-bucket/sleepy.example.com
hello
world
2026/10/24 22:54:05 activator: fence.example hibernated
2026/10/24 22:54:05 activator: fence.example hibernated
2026/10/24 22:54:05 activator: fence.example hibernated
2026/10/24 22:54:05 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:05 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/24 22:54:05 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/24 22:54:05 activator: barrier.example.com hibernated
2026/10/24 22:54:05 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:05 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:05 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/24 22:54:05 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/24 22:54:05 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/24 22:54:05 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/24 22:54:05 activator: barrier.example.com hibernated
2026/10/24 22:54:05 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:05 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:05 activator: barrier.example.com: generation 12's child is not proven empty: g12 still holds a process
2026/10/24 22:54:05 activator: generation 12 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/24 22:54:05 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:05 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/24 22:54:05 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 22:54:05 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/24 22:54:05 activator: barrier.example.com hibernated
2026/10/24 22:54:05 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:05 activator: stop generation 11: cgroup.kill: the child is not empty
2026/10/24 22:54:05 activator: generation 11 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/24 22:54:06 activator: placed.example.com: kill generation 1097 after a failed health wait: g still holds a process
2026/10/24 22:54:06 activator: placed.example.com: generation 1097's child is not proven empty: g still holds a process
2026/10/24 22:54:06 activator: placed.example.com: failed wake's generation 1097 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:06 activator: placed.example.com: wake failed and its generation is not proven stopped: start placed.example.com: instance not healthy within 300ms: Get "http://carlos/": dial unix /tmp/g4122951659/g/1097/placed.example.com.sock: connect: no such file or directory; activator: generation start failed and its termination is unproven: kill generation 1097 after a failed health wait: g still holds a process
2026/10/24 22:54:06 activator: placed.example.com: failed wake's generation 1097 is proven stopped; lease given back
2026/10/24 22:54:07 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:07 activator: unproven.example.com: failed wake's generation 1100 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:07 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 22:54:07 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:07 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:07 activator: unproven.example.com: failed wake's generation 1100 is proven stopped; lease given back
2026/10/24 22:54:07 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:07 activator: unproven.example.com: failed wake's generation 1101 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:07 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 22:54:07 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:07 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:07 activator: unproven.example.com: failed wake's generation 1101 is proven stopped; lease given back
2026/10/24 22:54:07 activator: unproven.example.com: generation 1102's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:07 activator: unproven.example.com: failed wake's generation 1102 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:07 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 22:54:08 activator: unhealthy-f6.example.com: failed wake's generation 1103 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 22:54:08 activator: unhealthy-f6.example.com: failed wake's generation 1103 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:08 activator: unhealthy-f6.example.com: failed wake's generation 1103 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 22:54:08 activator: unhealthy-f6.example.com: failed wake's generation 1103 is proven stopped; lease given back
2026/10/24 22:54:08 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/24 22:54:08 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/24 22:54:08 activator: one.example.com: generation 1105 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:08 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:09 activator: instance one.example.com exited unexpectedly — recycling
2026/10/24 22:54:09 activator: one.example.com hibernated
2026/10/24 22:54:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:09 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:09 activator: bot.x.internal hibernated
2026/10/24 22:54:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:09 activator: worker bot.x.internal respawned (attempt (restart)), now running v2
2026/10/24 22:54:09 activator: bot.x.internal hibernated
2026/10/24 22:54:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:09 activator: worker bot.x.internal respawned (attempt (config)), now running v2
2026/10/24 22:54:09 activator: bot.x.internal hibernated
2026/10/24 22:54:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:09 activator: bot.x.internal hibernated
2026/10/24 22:54:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:09 activator: bot.x.internal hibernated
2026/10/24 22:54:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:09 activator: bot.x.internal hibernated
2026/10/24 22:54:09 activator: moved.example.com: promoting 2 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/24 22:54:09 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:09 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:09 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:09 activator: x.example hibernated
2026/10/24 22:54:10 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/24 22:54:10 activator: x.example hibernated
2026/10/24 22:54:10 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/24 22:54:10 activator: x.example hibernated
2026/10/24 22:54:10 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:10 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:10 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:10 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/24 22:54:10 activator: x.example hibernated
2026/10/24 22:54:10 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:10 activator: x.example hibernated
2026/10/24 22:54:10 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:10 activator: x.example hibernated
2026/10/24 22:54:10 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:10 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:10 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:10 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:10 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:10 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:10 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:10 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:10 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/24 22:54:10 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_keyless_replica_is_refu4258477102/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:10 activator: replica shop-au.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/24 22:54:10 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_is_176997687/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:10 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_pre-existing_env_key_th2557200807/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:10 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_dif3985950533/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:10 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_with_no_1683616022/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:10 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY cannot be checked against its home's key (no entry for it in /etc/carlos/replica-home-keys; O8)
2026/10/24 22:54:10 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_home_updated,_this_re2738591624/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:10 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY is not the one the bucket's configuration now holds: a key change is in flight (ruling 27)
2026/10/24 22:54:10 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_bucket_cannot_be_read2901855379/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:10 activator: replica shop-au.example.com not started: the bucket's configuration could not be read: bucket unreachable
2026/10/24 22:54:10 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketno_fresh_reader_refused794697549/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:10 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY cannot be checked against the bucket's configuration
2026/10/24 22:54:10 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketresidual_a_change_only_o650904830/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:10 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_config-only_homes_keyl3198562612/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:10 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_home_row_mints_and_its_800362699/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:10 activator: seeding shop-au.example.com: /tmp/TestReplicaSeedIsGatedthe_homes_key3645919221/001/live/shop-au.example.com seed --db /tmp/TestReplicaSeedIsGatedthe_homes_key3645919221/002/shop-au.example.com.db
2026/10/24 22:54:10 activator: seeded shop-au.example.com
2026/10/24 22:54:10 activator: replica shop-au.example.com not started: where its CARLOS_INSTANCE_KEY came from is unknown
2026/10/24 22:54:10 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaGateDecidesOnTheLaunchesOwnRead2408532505/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:10 provision: a.shop.example.com: swapped for import sd_3inklpgap6fzt5fbo7njwamwv4 (the original route is journaled)
2026/10/24 22:54:10 provision: a.shop.example.com: import sd_3inklpgap6fzt5fbo7njwamwv4 rolled back to its original route
2026/10/24 22:54:10 provision: a.shop.example.com: import sd_3inklpgap6fzt5fbo7njwamwv4 is cleaned up and its journal row retired
2026/10/24 22:54:10 activator: h.example.com: placed seed sd_7jb4apqpgyxi27ye2typirh4g4 (8192 bytes)
2026/10/24 22:54:10 activator: h.example.com: placed seed sd_7pchxlh3pyqbxjgwi2bnrn42ju (8192 bytes)
2026/10/24 22:54:10 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/nb1673785350/data/instances/h.example.com/h.example.com.env: the platform mints none while it is
2026/10/24 22:54:10 activator: h.example.com: placed seed sd_7h2csjqqglouv5a7htwiyjoz24 (8192 bytes)
2026/10/24 22:54:10 activator: CARLOS_INSTANCE_KEY in /tmp/nb3821518388/data/instances/h.example.com/h.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/24 22:54:10 activator: h.example.com: placed seed sd_3j2iciuhtqhtp5fqdk6zxxawya (8192 bytes)
2026/10/24 22:54:10 activator: h.example.com: placed seed sd_ojnmnfv6vxqskhkvqmaicfyw54 (8192 bytes)
2026/10/24 22:54:10 activator: h.example.com: placed seed sd_wfe7c7yshegsr4zgnsa52kpoy4 (8192 bytes)
2026/10/24 22:54:10 activator: h.example.com: keeping seed sd_wfe7c7yshegsr4zgnsa52kpoy4 placed by an earlier wake
2026/10/24 22:54:10 activator: h.example.com: placed seed sd_xv22pwzgjer25ohbnm7v4swvdy (8192 bytes)
2026/10/24 22:54:10 activator: h.example.com: placed seed sd_lvm6sm5uzqb4kn7h6dx7cmsu4m (8192 bytes)
2026/10/24 22:54:10 activator: h.example.com: placed seed sd_3j3ziz6rp4u2nyh5ld42p6f27q (8192 bytes)
2026/10/24 22:54:10 activator: h.example.com: seed sd_m5hcjvh7mv4khdaw2c7zwvrpcu: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/24 22:54:10 activator: h.example.com: seed sd_ni7da3a7ahubouksdodqdyu47i: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/24 22:54:10 activator: h.example.com: seed sd_g2won2atptbzcr6my3m3tuyku4: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/24 22:54:10 activator: h.example.com: seed sd_nfxyfcdv5qgxwovsbqonprt6ci: h.example.com could not start from its seed: the snapshot could not be read. The next wake tries again.: bucket down
2026/10/24 22:54:10 activator: h.example.com: seed sd_5sk3z5mcdxw2hxgeqt74ci5jja: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/24 22:54:10 activator: h.example.com: seed sd_kru4fel4lcnrebljdekyk3ssza: h.example.com could not start from its seed: the snapshot is not a database that passes an integrity check.: integrity_check: file is not a database (26)
2026/10/24 22:54:10 activator: h.example.com: seed sd_2qk2c75rn4hru4ifomahtvvoqi: h.example.com could not start from its seed: claim control is on for this box.
2026/10/24 22:54:10 activator: h.example.com: seed sd_2qk2c75rn4hru4ifomahtvvoqi: h.example.com could not start from its seed: this box does not run instances nested.
2026/10/24 22:54:10 activator: h.example.com: seed sd_xjvk3rmceospundxood7jqle3q: seed fallback for h.example.com: cannot tell whether it has a replica of its own: listing failed
2026/10/24 22:54:10 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 22:54:10 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 22:54:10 activator: h.example.com: seed sd_5voebprthnqe7z7xxgkqosadee: seed fallback for h.example.com: found no replica of its own to restore
2026/10/24 22:54:10 activator: h.example.com: promoting 1 staged object(s) from moves/h.example.com/ (residence move landing)
2026/10/24 22:54:10 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 22:54:10 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 22:54:10 activator: h.example.com: seed sd_kbejdi747k65jo7pocpcqqpcha: seed fallback for h.example.com: found no replica of its own to restore
2026/10/24 22:54:10 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 22:54:10 activator: h.example.com: seed sd_6wxd6zbd52dqycocdcyrmiwxku: seed fallback for h.example.com: found no replica of its own to restore
2026/10/24 22:54:10 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 22:54:10 activator: h.example.com: placed seed sd_cre62xjxojr6sjrvaec6yqh4jm (8192 bytes)
2026/10/24 22:54:10 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 22:54:10 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 22:54:10 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 22:54:10 activator: h.example.com: seed sd_4cpq57onj7lyemngcx7vv23kdu: h.example.com could not start from its seed: the snapshot could not be read. The next wake tries again.: connection reset
2026/10/24 22:54:10 activator: h.example.com: seed sd_v57tl72odnstsk7i3m4bg3zpvy: h.example.com could not start from its seed: it could not be put in place. The next wake tries again.: openat2 /tmp/nb835657012/data/instances/h.example.com/.seed-sd_v57tl72odnstsk7i3m4bg3zpvy.tmp: permission denied
2026/10/24 22:54:10 activator: h.example.com: seed sd_znywxiptwjvmfsqc34xewkjexa: h.example.com could not start from its seed: the snapshot is not a database that passes an integrity check.: open: unable to open database file (14)
2026/10/24 22:54:10 activator: h.example.com: seed sd_5id3bwd4pvaxti64bjbkz7jfsi: h.example.com could not start from its seed: it could not be put in place. The next wake tries again.: instancefs: path contains a symlink: /tmp/nb1989588491/data/instances/h.example.com/.seed-sd_5id3bwd4pvaxti64bjbkz7jfsi.tmp
2026/10/24 22:54:10 activator: h.example.com: placed seed sd_iis5n6m2oup7kwxuqoitdcque4 (8192 bytes)
2026/10/24 22:54:10 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 22:54:10 activator: h.example.com: seed sd_xvzy2nkahcqyadslxmfufqkmye not recorded as placed yet: record unreachable
2026/10/24 22:54:10 activator: h.example.com: /tmp/nb3842609028/data/instances/h.example.com/h.example.com.db held seed sd_yzpknmom4odl5vw7nvnjgjm2xm's marker, not this route's (seed "sd_jvwhduph4bmwygjktfporrdw64"): purged before the restore
2026/10/24 22:54:10 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 22:54:10 activator: h.example.com: /tmp/nb2074633413/data/instances/h.example.com/h.example.com.db held seed sd_h5fr2aqduy7n5m7m4h7fu5cyga's marker, not this route's (seed "sd_pntpohaugdhji55kexbjufsoei"): purged before the restore
2026/10/24 22:54:10 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 22:54:10 activator: h.example.com: /tmp/nb1694754884/data/instances/h.example.com/h.example.com.db held seed sd_dtd5wxo7eavgamx6zbolxeshfa's marker, not this route's (seed ""): purged before the restore
2026/10/24 22:54:10 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 22:54:10 activator: h.example.com: /tmp/nb651402143/data/instances/h.example.com/h.example.com.db held seed sd_gyokhb24dqhvhv7rglyloljcjm's marker, not this route's (seed "sd_srfc6v5arwfqbfbdwfxsmt3fly"): purged before the restore
2026/10/24 22:54:10 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 22:54:10 activator: h.example.com: /tmp/nb3304042670/data/instances/h.example.com/h.example.com.db held seed sd_fsufg5ut33ve5paoc44wqiofeq's marker, not this route's (seed ""): purged before the restore
2026/10/24 22:54:10 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 22:54:10 activator: h.example.com: seed sd_astgmg5afoygz4b7g75dt5e7n4: h.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup.events: unreadable
2026/10/24 22:54:10 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 22:54:10 activator: h.example.com: placed seed sd_astgmg5afoygz4b7g75dt5e7n4 (8192 bytes)
2026/10/24 22:54:10 activator: h.example.com: placed seed sd_dqyfaitd5e3inetxxwmszxyffa (8192 bytes)
2026/10/24 22:54:10 activator: h.example.com: /tmp/nb2924432140/data/instances/h.example.com/h.example.com.db held seed an aborted import's data's marker, not this route's (seed ""): purged before the restore
2026/10/24 22:54:10 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 22:54:10 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 22:54:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:10 activator: swap.example.com hibernated
2026/10/24 22:54:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:10 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/24 22:54:10 activator: swap.example.com hibernated
2026/10/24 22:54:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:10 activator: LEASE LOST for swap.example.com (renewal failed) — standing down
2026/10/24 22:54:10 activator: swap.example.com hibernated
2026/10/24 22:54:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:10 activator: swap.example.com hibernated
2026/10/24 22:54:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:10 activator: swap.example.com is no longer hibernate-managed — releasing
2026/10/24 22:54:10 activator: swap.example.com hibernated
2026/10/24 22:54:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:10 activator: swap.example.com hibernated
2026/10/24 22:54:11 activator: exit.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:11 activator: exit.example.com hibernated
2026/10/24 22:54:11 activator: wake shop.example.com: activator stopped during startup
2026/10/24 22:54:11 activator: wake shop.example.com: activator stopped during startup
2026/10/24 22:54:11 activator: shop.example.com: route refreshed: db "" -> "" socket "/run/carlos/shop.example.com.sock" -> "/run/carlos/instances/shop.example.com/app.sock"
2026/10/24 22:54:11 activator: shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:11 activator: missed-join.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:11 activator: stop missed-join.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/24 22:54:11 activator: missed-join.example.com hibernated
2026/10/24 22:54:11 activator: nested-stuck.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:11 activator: stop nested-stuck.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/24 22:54:11 activator: nested-stuck.example.com: the sidecar a restart brought back for a stopping entry is not proven gone (activator: stop unproven; processes remain); keeping the lease until it has exited
2026/10/24 22:54:12 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 22:54:12 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 22:54:12 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 22:54:12 activator: nested-stuck.example.com hibernated
2026/10/24 22:54:12 activator: checks.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:12 activator: owned.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:12 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:12 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 22:54:12 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 22:54:12 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:12 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 22:54:12 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/24 22:54:12 activator: bot.x.internal hibernated
2026/10/24 22:54:12 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:12 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir: not a directory; the sweep retries
2026/10/24 22:54:12 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir: not a directory
2026/10/24 22:54:12 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir: not a directory; the sweep retries
2026/10/24 22:54:12 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/24 22:54:12 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir: not a directory; the sweep retries
2026/10/24 22:54:12 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir: not a directory; the sweep retries
2026/10/24 22:54:12 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir: not a directory
2026/10/24 22:54:12 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir: not a directory; the sweep retries
2026/10/24 22:54:12 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_2096029690/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/24 22:54:12 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:12 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:12 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:12 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:12 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:12 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 22:54:13 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:13 activator: worker bot.x.internal: the attempt's respawn did not finish within 150ms of its caller leaving; reporting its build as possibly executed
2026/10/24 22:54:13 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com hibernated
2026/10/24 22:54:13 activator: swap.example.com: generation 1149 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com hibernated
2026/10/24 22:54:13 activator: swap.example.com: generation 1151 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com hibernated
2026/10/24 22:54:13 activator: swap.example.com: generation 1153 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com hibernated
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com hibernated
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com hibernated
2026/10/24 22:54:13 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com hibernated
2026/10/24 22:54:13 activator: swap.example.com: generation 1158 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com hibernated
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com hibernated
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com hibernated
2026/10/24 22:54:13 activator: swap.example.com: generation 1163 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com hibernated
2026/10/24 22:54:13 activator: swap.example.com: generation 1166 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com hibernated
2026/10/24 22:54:13 activator: swap.example.com: generation 1168 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing2383710780/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing2383710780/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 22:54:13 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing2383710780/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing2383710780/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: h.example.com: started carlos-instance-gen@h.example.com:7.service memory.max=1073741824 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:13 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: processes remain
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com hibernated
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:13 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:14 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:14 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:21 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: instance control /stop: 502: stop: processes remain
2026/10/24 22:54:21 activator: seeding h.example.com: /tmp/TestSeedUnitCleansUpOnlyItsOwnDBOnFailure1339915596/001/live/h.example.com seed --fast --db /tmp/nb2459315712/data/instances/h.example.com/h.example.com.db
2026/10/24 22:54:21 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:21 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 22:54:21 activator: seeding h.example.com: /tmp/TestSeedUnitFailureStopsBeforeCleanupAndKeepsTheDBWhenProcessesR2175950916/001/live/h.example.com seed --fast --db /tmp/nb1497147834/data/instances/h.example.com/h.example.com.db
2026/10/24 22:54:21 activator: seeding h.example.com: /tmp/TestANestedSeedRecordsItsObligationBeforeItRuns3352619645/001/live/h.example.com seed --db /tmp/nb3886382859/data/instances/h.example.com/h.example.com.db
2026/10/24 22:54:21 activator: seeded h.example.com
2026/10/24 22:54:21 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI3267707266/001/live/h.example.com seed --db /tmp/nb3451614120/data/instances/h.example.com/h.example.com.db
2026/10/24 22:54:21 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 22:54:21 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 22:54:21 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI816950269/001/live/h.example.com seed --db /tmp/nb203308606/data/instances/h.example.com/h.example.com.db
2026/10/24 22:54:21 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 22:54:21 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 22:54:21 activator: fifo-obligation.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/24 22:54:21 activator: one.example.com: generation 1892 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: one.example.com: generation 1894 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: one.example.com: first wake of v2 had 30s of its health window left at the exec, under the 1h0m0s floor; booting the good build v1 without trying it
2026/10/24 22:54:22 activator: one.example.com: generation 1897 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: one.example.com: generation 1898 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: one.example.com: generation 1899 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: one.example.com hibernated
2026/10/24 22:54:22 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: one.example.com: generation 1901 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: tenant.test not started: it is held on v1, and the launch hashed "v2" (the live link moved after the check); the next wake checks again
2026/10/24 22:54:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: one.example.com: generation 1906 of v0: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:22 activator: one.example.com: woke in 4ms: checks 4ms, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: one.example.com: generation 1907 of v0: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: one.example.com not woken: it is held on v1, and this wake would run "v2" (the live link moved after the check); the next wake checks again
2026/10/24 22:54:22 activator: one.example.com not woken: it is held on v1, and this wake would run "" (the live link moved after the check); the next wake checks again
2026/10/24 22:54:22 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:22 activator: swap.example.com hibernated
2026/10/24 22:54:22 activator: swap.example.com: generation 1909 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:22 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:24 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:24 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:24 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:25 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 22:54:25 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 22:54:25 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:25 activator: seeding h.example.com: /tmp/TestANestedUnversionedLinkWithNoHookWakesAsToday945130736/001/live/h.example.com seed --db /tmp/nb2569711848/data/instances/h.example.com/h.example.com.db
2026/10/24 22:54:25 activator: seeded h.example.com
2026/10/24 22:54:25 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:25 activator: h.example.com: carlos-instance-gen@h.example.com:1916.service launched in 0s, healthy 0s later
2026/10/24 22:54:25 activator: h.example.com: woke in 1ms: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 22:54:25 activator: u.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 22:54:25 activator: e.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:25 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:25 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:25 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:25 activator: swap.example.com hibernated
2026/10/24 22:54:25 activator: swap.example.com: generation 1922 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:25 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 22:54:25 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:25 activator: one.example.com: generation 1925 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:25 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:25 activator: one.example.com: generation 1927 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:25 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:25 activator: one.example.com: generation 1930's child is not proven empty: activator: stop unproven; processes remain
2026/10/24 22:54:25 activator: one.example.com: failed wake's generation 1930 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:25 activator: one.example.com: generation 1931 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:25 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:26 activator: one.example.com: generation 1932 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:26 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:26 activator: one.example.com hibernated
2026/10/24 22:54:26 activator: one.example.com: generation 1933 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:26 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:27 activator: one.example.com: first wake of v2 had 400ms of its health window left at the exec, under the 20s floor; booting the good build v1 without trying it
2026/10/24 22:54:27 activator: one.example.com: generation 1935 of v1: record 0s, start 250ms, healthy 0s later
2026/10/24 22:54:27 activator: one.example.com: woke in 250ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 250ms, replicate 0s
2026/10/24 22:54:27 activator: two.example.com: generation 1936 of v1: record 0s, start 251ms, healthy 0s later
2026/10/24 22:54:27 activator: two.example.com: woke in 251ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 251ms, replicate 0s
2026/10/24 22:54:27 activator: x.example: hold's pinned wake refused or abandoned: the claims admission refused before the exec: hostbox: unresolved lifecycle contract: isolated root launcher cannot carry claim reservations
2026/10/24 22:54:27 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat230369409/003/g/1937/real.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:29 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat230369409/003/g/1938/real.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:29 activator: real.example.com: generation 1938 of v1: record 0s, start 1ms, healthy 0s later
2026/10/24 22:54:29 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat230369409/003/g/1938/real.example.com.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat230369409/003/g/1938/real.example.com.sock: no such file or directory)
2026/10/24 22:54:29 activator: real.example.com: woke in 2.001s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 2.001s, replicate 0s
2026/10/24 22:54:29 activator: real.example.com hibernated
2026/10/24 22:54:31 activator: real.example.com: generation 1940 of v1: record 0s, start 1ms, healthy 0s later
2026/10/24 22:54:31 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign3222354150/004/hand-set.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign3222354150/004/hand-set.sock: no such file or directory)
2026/10/24 22:54:31 activator: real.example.com: woke in 2.002s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 2.002s, replicate 0s
2026/10/24 22:54:31 activator: real.example.com hibernated
2026/10/24 22:54:31 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:31 activator: bot.x.internal: worker generation 0's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 22:54:31 activator: bot.x.internal: worker generation 1942's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 22:54:31 activator: bot.x.internal: failed wake's generation 1942 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:32 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:32 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing1474368770/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing1474368770/002/zdd: not a directory; the sweep retries
2026/10/24 22:54:32 activator: worker bot.x.internal not spawned: the build it would exec (v2 /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing1474368770/003/v2) could not be published first: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing1474368770/002/zdd: not a directory
2026/10/24 22:54:32 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing1474368770/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing1474368770/002/zdd: not a directory; the sweep retries
2026/10/24 22:54:32 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing1474368770/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing1474368770/002/zdd: not a directory; the sweep retries
2026/10/24 22:54:32 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn1501310774/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn1501310774/002/not-a-dir: not a directory; the sweep retries
2026/10/24 22:54:32 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn1501310774/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn1501310774/002/not-a-dir: not a directory; the sweep retries
2026/10/24 22:54:32 activator: one.example.com: generation 1945 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:32 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:32 activator: instance one.example.com exited unexpectedly — recycling
2026/10/24 22:54:32 activator: one.example.com hibernated
2026/10/24 22:54:32 activator: one.example.com: generation 1946 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:32 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:32 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:32 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:32 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:32 activator: one.example.com hibernated
2026/10/24 22:54:34 activator: one.example.com: first wake of v2 had 500ms of its health window left at the exec, under the 1s floor; booting the good build v1 without trying it
2026/10/24 22:54:34 activator: one.example.com: generation 1951 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:34 activator: one.example.com: woke in 1.501s: checks 0s, lease 0s, restore 0s, identity 1.5s, spawn 0s, replicate 0s
2026/10/24 22:54:36 activator: one.example.com: generation 1952 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:36 activator: one.example.com: woke in 1.5s: checks 0s, lease 0s, restore 0s, identity 1.5s, spawn 0s, replicate 0s
2026/10/24 22:54:36 activator: one.example.com: generation 1954 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:36 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:36 activator: bot.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:36 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/24 22:54:36 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/24 22:54:36 activator: worker bot.example.com respawned (exited), now running 
2026/10/24 22:54:36 activator: bot.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:36 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/24 22:54:36 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/24 22:54:36 activator: worker bot.example.com respawned (exited), now running 
2026/10/24 22:54:36 activator: bot.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:36 activator: bot.example.com: worker generation 1957's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 22:54:36 activator: stop bot.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 22:54:36 activator: bot.example.com hibernated
2026/10/24 22:54:36 activator: bot3.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:36 activator: bot3.example.com: worker generation 1958's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 22:54:36 activator: worker bot3.example.com NOT respawned (test: a restart with no old handle): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/24 22:54:36 activator: bot4.example.com: worker generation 1960's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 22:54:36 activator: bot4.example.com: failed wake's generation 1960 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:36 activator: bot4.example.com: failed wake's generation 1960 is proven stopped; lease given back
2026/10/24 22:54:36 activator: bot2.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:36 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/24 22:54:36 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/24 22:54:36 activator: bot2.example.com: worker generation 1961's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 22:54:36 activator: worker bot2.example.com NOT respawned (exited): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/24 22:54:36 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/24 22:54:36 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/24 22:54:36 activator: worker bot2.example.com respawned (exited), now running 
2026/10/24 22:54:36 activator: bot.example.com: woke in 21ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 1ms, settle 20ms, replicate 0s
2026/10/24 22:54:36 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/24 22:54:36 activator: worker bot.example.com respawned (exited), now running v1
2026/10/24 22:54:36 activator: bot.example.com hibernated
2026/10/24 22:54:36 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:37 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:37 activator: bot10.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:37 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/24 22:54:37 activator: worker bot10.example.com: the shed respawn child's exit is not proven (activator: stop unproven; processes remain); the entry keeps it and its build until the sweep proves it
2026/10/24 22:54:37 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/24 22:54:37 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/24 22:54:37 activator: bot10.example.com: the shed respawn child has exited
2026/10/24 22:54:37 activator: bot10.example.com hibernated
2026/10/24 22:54:37 activator: bot10.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:37 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/24 22:54:37 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/24 22:54:37 activator: bot10.example.com respawn did not finish before its stop; stopping the rest, the lease is kept until it has
2026/10/24 22:54:37 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/24 22:54:37 activator: bot10.example.com hibernated
2026/10/24 22:54:37 activator: bot8.example.com: worker generation 1964's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 22:54:37 activator: bot8.example.com: failed wake's generation 1964 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:37 activator: bot8.example.com: failed wake's generation 1964 is proven stopped; lease given back
2026/10/24 22:54:37 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:37 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:37 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:37 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:37 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/24 22:54:37 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:37 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:37 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/24 22:54:38 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/24 22:54:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:38 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:38 activator: bot.x.internal hibernated
2026/10/24 22:54:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:38 activator: worker bot.x.internal respawned (attempt (restart)), now running v1
2026/10/24 22:54:38 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 22:54:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:38 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:38 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:39 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 22:54:39 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 22:54:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:39 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:39 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 22:54:39 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 22:54:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:39 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 22:54:39 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 22:54:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:39 activator: bot.x.internal: worker generation 1967's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 22:54:39 activator: worker bot.x.internal NOT respawned (attempt (release)): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/24 22:54:39 activator: bot.x.internal: worker generation 1967's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 22:54:39 activator: worker bot.x.internal NOT respawned (restoring v1): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/24 22:54:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:39 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: lease lost; abandoning it
2026/10/24 22:54:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:39 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir: not a directory; the sweep retries
2026/10/24 22:54:39 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir: not a directory
2026/10/24 22:54:39 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir: not a directory; the sweep retries
2026/10/24 22:54:39 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/24 22:54:39 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir: not a directory; the sweep retries
2026/10/24 22:54:39 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir: not a directory; the sweep retries
2026/10/24 22:54:39 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir: not a directory
2026/10/24 22:54:39 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir: not a directory; the sweep retries
2026/10/24 22:54:39 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2008726913/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/24 22:54:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:39 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:39 activator: bot.x.internal hibernated
2026/10/24 22:54:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:39 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:39 activator: worker bot.x.internal NOT respawned (falling back to v1): swap: lease lost; abandoning it
2026/10/24 22:54:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:39 activator: WORKER bot.x.internal failed to respawn (attempt (restart)): exec: text file busy — the sweep's crash restart takes it from here
2026/10/24 22:54:39 activator: WORKER bot.x.internal failed to respawn (restoring v1): exec: text file busy — the sweep's crash restart takes it from here
2026/10/24 22:54:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:39 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:39 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/24 22:54:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:40 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 22:54:40 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/24 22:54:40 activator: bot.x.internal hibernated
2026/10/24 22:54:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:40 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 22:54:40 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 22:54:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:40 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 22:54:40 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 22:54:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:40 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:40 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/24 22:54:40 activator: bot.x.internal hibernated
2026/10/24 22:54:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:40 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:40 activator: LEASE LOST for bot.x.internal (renewal failed) — standing down
2026/10/24 22:54:40 activator: bot.x.internal hibernated
2026/10/24 22:54:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:41 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:41 activator: bot.x.internal hibernated
2026/10/24 22:54:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:41 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 22:54:41 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/24 22:54:41 activator: bot.x.internal hibernated
2026/10/24 22:54:41 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:41 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 22:54:41 activator: h.example.com: unit restarted outside the activator (invocation inv2, ours inv1); stopping it
2026/10/24 22:54:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: two.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd: not a directory; the sweep retries
2026/10/24 22:54:42 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd: not a directory; the sweep retries
2026/10/24 22:54:42 activator: worker first.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd: not a directory
2026/10/24 22:54:42 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd: not a directory; the sweep retries
2026/10/24 22:54:42 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd: not a directory; the sweep retries
2026/10/24 22:54:42 activator: one.example.com: generation 1981 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 22:54:42 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:42 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:42 activator: replica shop.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: one.example.com: generation 1986 of v3: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:42 activator: SIDECAR restart for one.example.com failed: swap: lease lost (next attempt in 2s)
2026/10/24 22:54:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:42 activator: one.example.com idle 1m31s — hibernating
2026/10/24 22:54:42 activator: one.example.com hibernated
2026/10/24 22:54:42 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 22:54:42 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:42 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 22:54:42 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:42 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 22:54:42 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:42 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:42 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:42 activator: bot.example.com: woke in 21ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 1ms, settle 20ms, replicate 0s
2026/10/24 22:54:42 activator: bot.example.com hibernated
2026/10/24 22:54:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: bot1.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:42 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd: not a directory; the sweep retries
2026/10/24 22:54:42 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd: not a directory
2026/10/24 22:54:42 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd: not a directory; the sweep retries
2026/10/24 22:54:42 activator: WORKER bot.x.internal failed to respawn (exited): worker build not published before its exec: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished3207181966/002/data/zdd: not a directory — the sweep's crash restart takes it from here
2026/10/24 22:54:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:42 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:42 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 22:54:42 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 22:54:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 22:54:42 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 22:54:42 activator: empty.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:42 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 22:54:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:42 activator: WORKER bot.x.internal failed to respawn (exited): fork/exec: resource temporarily unavailable — the sweep's crash restart takes it from here
2026/10/24 22:54:42 activator: bot.shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: bot2.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: odd.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 22:54:42 activator: old.shop.example.com: route refreshed: db "/data/instances/old/old.shop.example.com.db" -> "" socket "/run/carlos/instances/old/app.sock" -> ""
2026/10/24 22:54:42 activator: wake old.shop.example.com: old.shop.example.com: the route changed before the wake: the activator no longer manages it
2026/10/24 22:54:42 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 22:54:42 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 22:54:42 activator: worker bot1.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:42 activator: worker bot1.x.internal exited; it serves v1 but live/bot1.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/24 22:54:42 activator: worker bot2.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:42 activator: worker bot2.x.internal exited; live/bot2.x.internal names v2, but this sweep's converge budget is spent, so it recycles its captured build v1 and converges on a later sweep
2026/10/24 22:54:42 activator: worker bot1.x.internal respawned (converging onto a new build), now running v2
2026/10/24 22:54:42 activator: worker bot2.x.internal respawned (exited), now running v1
2026/10/24 22:54:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:42 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 22:54:42 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 22:54:42 activator: worker bot2.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:42 activator: worker bot2.x.internal respawned (converging onto a new build), now running v2
2026/10/24 22:54:42 activator: swap.example.com hibernated
2026/10/24 22:54:42 activator: swap.example.com hibernated
2026/10/24 22:54:42 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 22:54:42 activator: worker bot.x.internal hit its max_run; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/24 22:54:42 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 22:54:42 activator: shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 22:54:42 activator: worker bot.x.internal respawned (hit its max_run), now running v2
2026/10/24 22:54:42 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:42 activator: one.example.com: generation 1995 of v3: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: one.example.com: generation 1997 of v3: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/24 22:54:42 activator: two.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/24 22:54:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: control listener for shop.example.com: listen unix /tmp/TestARefusedLaunchLeavesNoBuildStateRecord222381056/002/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com hibernated
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/24 22:54:42 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/24 22:54:42 activator: one.example.com hibernated
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com hibernated
2026/10/24 22:54:42 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com hibernated
2026/10/24 22:54:42 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 22:54:42 activator: swap.example.com hibernated
2026/10/24 22:54:42 activator: LEASE HELD for swap.example.com by "box-b" — a previous incarnation hasn't released; refusing to wake (retry after 1m0s)
2026/10/24 22:54:42 activator: swap.example.com: hold's pinned wake refused or abandoned: instance lease held by "box-b" (retry in 1m0s)
2026/10/24 22:54:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/24 22:54:42 activator: swap.example.com hibernated
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com hibernated
2026/10/24 22:54:42 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: socket dir: permission denied
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com hibernated
2026/10/24 22:54:42 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2029.service exited before it was observed
2026/10/24 22:54:42 activator: swap.example.com hibernated
2026/10/24 22:54:42 activator: swap.example.com: generation 2030 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: nested-costop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:42 activator: stop sidecar nested-costop.example.com: activator: stop unproven; processes remain
2026/10/24 22:54:42 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 22:54:42 activator: slowproof.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: instance slowproof.example.com exited unexpectedly — recycling
2026/10/24 22:54:42 activator: slowproof.example.com hibernated
2026/10/24 22:54:42 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 22:54:42 activator: queued.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 22:54:42 activator: nested-costop.example.com hibernated
2026/10/24 22:54:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:42 activator: swap.example.com hibernated
2026/10/24 22:54:42 activator: queued.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: refused.example.com: giving the lease back failed: delete lease: context deadline exceeded; retrying it
2026/10/24 22:54:43 activator: refused.example.com: lease given back on retry
2026/10/24 22:54:43 activator: stuck.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: failed wake's generation 2042 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:43 activator: swap.example.com: failed wake's generation 2042 is proven stopped; lease given back
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: bot9.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:43 activator: worker bot9.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/24 22:54:43 activator: bot9.example.com: worker generation 2050's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 22:54:43 activator: generation 2050 of bot9.example.com did not prove its exit; keeping it in the operation set
2026/10/24 22:54:43 activator: LEASE LOST for bot9.example.com (renewal failed) — standing down
2026/10/24 22:54:43 activator: bot9.example.com hibernated
2026/10/24 22:54:43 activator: seeding shop.example.com: /tmp/TestARefusedLaunchLeavesNoBuildStateRecordseed_refused_beside_a3481138664/002/versions/v1/app seed --db /tmp/TestARefusedLaunchLeavesNoBuildStateRecordseed_refused_beside_a3481138664/003/shop.example.com.db
2026/10/24 22:54:43 activator: bot6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:43 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/24 22:54:43 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: instance unproven.example.com exited unexpectedly — recycling
2026/10/24 22:54:43 activator: unproven.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: unproven.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@unproven: instance control socket: connection refused); keeping the lease until it is
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: the backend ran a different generation from the hold's choice: generation 2066 of v9 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_849873663/004/versions/v2/app"; the hold chose v2 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_849873663/004/versions/v2/app"
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: hold's pinned wake refused or abandoned: restore swap.example.com: replica unreadable
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:43 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:43 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: start g: activator: generation start failed and its termination is unproven
2026/10/24 22:54:43 activator: SIDECAR for late-sidecar.example failed to start: activator: maintenance fence active — retaining returned process
2026/10/24 22:54:43 activator: late-sidecar.example: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:43 activator: late-sidecar.example hibernated
2026/10/24 22:54:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:43 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:43 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:43 activator: stuck.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: h.example.com: kill carlos-instance-gen@h.example.com:2074.service after a failed start: stop timed out
2026/10/24 22:54:43 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2074.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/24 22:54:43 activator: early.example.com: the aborted start's unit is not proven stopped (context canceled); keeping the lease until it is
2026/10/24 22:54:43 activator: shutdown: out of time with 1 failed wake(s) not proven stopped, their leases still held: early.example.com (generation 2032)
2026/10/24 22:54:43 activator: slowproof.example.com: the aborted start's unit is not proven stopped (context deadline exceeded); keeping the lease until it is
2026/10/24 22:54:43 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:43 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:43 activator: own.example.com: failed wake's generation 0 is proven stopped; lease given back
2026/10/24 22:54:43 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:43 activator: worker bot.x.internal exited; live/bot.x.internal names v2, but the release scheduler owns build changes, so it recycles its captured build v1
2026/10/24 22:54:43 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:43 provision: a.shop.example.com: swapped for import sd_cu6noxhwesarsiwgf5gqwiwuoe (the original route is journaled)
2026/10/24 22:54:43 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:43 activator: worker bot.x.internal exited; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/24 22:54:43 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 22:54:43 provision: a.shop.example.com: import sd_cu6noxhwesarsiwgf5gqwiwuoe rolled back to its original route
2026/10/24 22:54:43 provision: a.shop.example.com: import sd_cu6noxhwesarsiwgf5gqwiwuoe is cleaned up and its journal row retired
2026/10/24 22:54:43 provision: a.shop.example.com: swapped for import sd_5azhss7itemmygjorxhxwbjysy (the original route is journaled)
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: a.shop.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@a.shop.example.com: instance control socket: connection refused); keeping the lease until it is
2026/10/24 22:54:43 provision: a.shop.example.com: import sd_5azhss7itemmygjorxhxwbjysy rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 22:54:43 activator: a.shop.example.com: failed wake's generation 2075 is proven stopped; lease given back
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 provision: a.shop.example.com: import sd_5azhss7itemmygjorxhxwbjysy rolled back to its original route
2026/10/24 22:54:43 provision: a.shop.example.com: import sd_5azhss7itemmygjorxhxwbjysy is cleaned up and its journal row retired
2026/10/24 22:54:43 provision: a.shop.example.com: swapped for import sd_miovgtnawq2tp3ofprkrh33v5q (the original route is journaled)
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: h.example.com: kill carlos-instance-gen@h.example.com:2081.service after a failed start: stop timed out
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: generation 2086's child is not proven empty: the child still holds a process
2026/10/24 22:54:43 activator: swap.example.com: failed wake's generation 2086 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:43 activator: seeding shop.example.com: /tmp/TestARefusedLaunchLeavesNoBuildStateRecorda_seed_that_ran_is_re1471792662/001/versions/v1/app seed --db /tmp/TestARefusedLaunchLeavesNoBuildStateRecorda_seed_that_ran_is_re1471792662/002/shop.example.com.db
2026/10/24 22:54:43 activator: shop.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com: failed wake's generation 2086 is proven stopped; lease given back
2026/10/24 22:54:43 activator: a.shop.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@a.shop.example.com: unit still active); keeping the lease until it is
2026/10/24 22:54:43 provision: a.shop.example.com: import sd_miovgtnawq2tp3ofprkrh33v5q rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 22:54:43 provision: a.shop.example.com: swapped for import sd_pzmqqi2ow65mjvtbjtd5vrevfa (the original route is journaled)
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: bot7.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:43 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 provision: a.shop.example.com: import sd_miovgtnawq2tp3ofprkrh33v5q rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: staging: staged digest mismatch: have be0041e1a437dbea5b67dfe58bad04a0fc30d5036d469429f3534f39ec53d8b5, want c3bae8fb3a8bf395783d7f7f6ffcd4aadc797c95ce24b7e8e2a98af6b63d8f6d
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: a stop-evidence operation holds the host
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 provision: a.shop.example.com: import sd_pzmqqi2ow65mjvtbjtd5vrevfa rolled back to its original route
2026/10/24 22:54:43 provision: a.shop.example.com: import sd_pzmqqi2ow65mjvtbjtd5vrevfa is cleaned up and its journal row retired
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/24 22:54:43 provision: a.shop.example.com: import sd_miovgtnawq2tp3ofprkrh33v5q rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/24 22:54:43 activator: own.example.com: failed wake's generation 0 is proven stopped; lease given back
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: generation 2109 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 provision: a.shop.example.com: import sd_miovgtnawq2tp3ofprkrh33v5q rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 22:54:43 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:43 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: shutdown: out of time with 1 failed wake(s) not proven stopped, their leases still held: unproven.example.com (generation 2053)
2026/10/24 22:54:43 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:43 activator: x.example hibernated
2026/10/24 22:54:43 provision: a.shop.example.com: swapped for import sd_7grudkzwhvw2jut4fbs4fbqf2y (the original route is journaled)
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: a.shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: a.shop.example.com hibernated
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 provision: a.shop.example.com: import sd_7grudkzwhvw2jut4fbs4fbqf2y rolled back to its original route
2026/10/24 22:54:43 provision: a.shop.example.com: import sd_7grudkzwhvw2jut4fbs4fbqf2y is cleaned up and its journal row retired
2026/10/24 22:54:43 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:43 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: x.example hibernated
2026/10/24 22:54:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:43 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/24 22:54:43 activator: one.example.com idle 1h0m0s — hibernating
2026/10/24 22:54:43 activator: one.example.com hibernated
2026/10/24 22:54:43 activator: one.example.com idle 1h0m0s — hibernating
2026/10/24 22:54:43 activator: one.example.com hibernated
2026/10/24 22:54:43 activator: bot6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:43 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/24 22:54:43 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:43 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/24 22:54:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: swap.example.com hibernated
2026/10/24 22:54:43 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: h.example.com: kill carlos-instance-gen@h.example.com:2131.service after a failed start: stop timed out
2026/10/24 22:54:43 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: the unit failed to start; activator: generation start failed and its termination is unproven: stop timed out
2026/10/24 22:54:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:43 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:43 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:43 activator: bot.x.internal hibernated
2026/10/24 22:54:43 activator: restart.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:43 activator: instance restart.example.com exited unexpectedly — recycling
2026/10/24 22:54:43 activator: restart.example.com hibernated
2026/10/24 22:54:44 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:44 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:44 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/24 22:54:44 activator: one.example.com hibernated
2026/10/24 22:54:44 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:44 activator: x.example hibernated
2026/10/24 22:54:44 activator: exited-flat.example.com: failed wake's generation 2138 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 22:54:44 activator: exited-flat.example.com: failed wake's generation 2138 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:44 activator: f6.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 22:54:44 activator: forked.example.com: failed wake's generation 2139 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:44 activator: forked.example.com: failed wake's generation 2139 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 22:54:44 activator: forked.example.com: failed wake's generation 2139 is proven stopped; lease given back
2026/10/24 22:54:44 activator: fenced.example.com: failed wake's generation 2140 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:44 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:44 activator: fenced.example.com: failed wake's generation 2140 is proven stopped; lease given back
2026/10/24 22:54:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: swap.example.com hibernated
2026/10/24 22:54:44 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/24 22:54:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: swap.example.com hibernated
2026/10/24 22:54:44 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: cgroup: the host's parent cgroup holds a process directly: holds 1 process(es)
2026/10/24 22:54:44 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:44 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/24 22:54:44 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 22:54:44 activator: x.example hibernated
2026/10/24 22:54:44 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/24 22:54:44 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 22:54:44 activator: barrier.example.com hibernated
2026/10/24 22:54:44 activator: barrier.example.com hibernated
2026/10/24 22:54:44 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:44 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/24 22:54:44 activator: barrier.example.com hibernated
2026/10/24 22:54:44 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:44 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: swap.example.com hibernated
2026/10/24 22:54:44 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/24 22:54:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:44 activator: fence.example hibernated
2026/10/24 22:54:44 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 22:54:44 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 22:54:44 activator: bot.x.internal hibernated
2026/10/24 22:54:44 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: bot7.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:44 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/24 22:54:44 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/24 22:54:44 activator: one.example.com hibernated
2026/10/24 22:54:44 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:44 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:44 activator: stop fence.example: stop refused
2026/10/24 22:54:44 activator: fence.example hibernated
2026/10/24 22:54:44 activator: exited-foreign.example.com: failed wake's generation 2163 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 22:54:44 activator: exited-foreign.example.com: failed wake's generation 2163 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:44 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:44 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:44 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:44 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 22:54:44 activator: bot.x.internal hibernated
2026/10/24 22:54:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:44 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:44 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:44 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:54:44 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:44 activator: fenced.example.com: failed wake's generation 2167 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:44 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:44 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 22:54:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: swap.example.com hibernated
2026/10/24 22:54:44 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/24 22:54:44 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:44 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 22:54:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: fenced.example.com: failed wake's generation 2167 is proven stopped; lease given back
2026/10/24 22:54:44 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:44 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:44 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 22:54:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:44 activator: x.example hibernated
2026/10/24 22:54:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 22:54:44 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:44 activator: bot5.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:44 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/24 22:54:44 activator: unproven.example.com: generation 2179's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:44 activator: unproven.example.com: failed wake's generation 2179 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:44 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 22:54:44 activator: SIDECAR for partial-sidecar.example failed to start: sidecar started then failed — retaining returned process
2026/10/24 22:54:44 activator: partial-sidecar.example: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:44 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: unproven.example.com: generation 2179's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:44 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: partial-sidecar.example hibernated
2026/10/24 22:54:44 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:44 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 22:54:44 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/24 22:54:44 activator: barrier.example.com hibernated
2026/10/24 22:54:44 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:44 activator: kass.example.com idle 11s — hibernating
2026/10/24 22:54:44 activator: kass.example.com hibernated
2026/10/24 22:54:44 activator: unproven.example.com: failed wake's generation 2179 is proven stopped; lease given back
2026/10/24 22:54:44 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:44 activator: h.example.com: processes remain after failed; not releasing
2026/10/24 22:54:44 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:44 activator: bot6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:44 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/24 22:54:44 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: unproven.example.com hibernated
2026/10/24 22:54:44 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:44 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 22:54:44 activator: bot.x.internal hibernated
2026/10/24 22:54:44 activator: fence.example hibernated
2026/10/24 22:54:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:44 activator: f6.example.com: failed wake's generation 0 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 22:54:44 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:44 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/24 22:54:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:44 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:44 activator: SIDECAR restart for one.example.com failed: the build state could not be written, so this build was not started: bucket refused the build state (next attempt in 2s)
2026/10/24 22:54:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:44 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:44 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:45 activator: fence.example hibernated
2026/10/24 22:54:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: swap.example.com hibernated
2026/10/24 22:54:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a cgroup that took cgroup.kill could not be made fresh before a spawn: carlos-exec@h.example.com: device or resource busy
2026/10/24 22:54:45 activator: fence.example hibernated
2026/10/24 22:54:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:45 activator: x.example hibernated
2026/10/24 22:54:45 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 22:54:45 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:45 activator: seeding h.example.com: /tmp/TestPlan14GenerationBackendAuxiliaryBoundaryclaim=falseseedpi2660563025/001/live/h.example.com seed --db /tmp/nb693457163/data/instances/h.example.com/h.example.com.db
2026/10/24 22:54:45 activator: seeded h.example.com
2026/10/24 22:54:45 activator: seeding h.example.com: /tmp/TestPlan14GenerationBackendAuxiliaryBoundaryclaim=falseseedpi3960357665/001/live/h.example.com seed --db /tmp/nb3104991625/data/instances/h.example.com/h.example.com.db
2026/10/24 22:54:45 activator: seeded h.example.com
2026/10/24 22:54:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:45 activator: bot5.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:45 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/24 22:54:45 activator: SIDECAR restart for partial-sidecar.example failed: sidecar started then failed — retaining returned process
2026/10/24 22:54:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:45 activator: h.example.com: unit restarted outside the activator (invocation someone-else, ours inv7); stopping it
2026/10/24 22:54:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:45 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/24 22:54:45 activator: barrier.example.com hibernated
2026/10/24 22:54:45 activator: partial-sidecar.example hibernated
2026/10/24 22:54:45 activator: unproven.example.com: generation 2206's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:45 activator: unproven.example.com: failed wake's generation 2206 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:45 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: h: start unit: instance control: generation termination unproven: the unit exited before it was observed
2026/10/24 22:54:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: bot7.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:45 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/24 22:54:45 activator: unproven.example.com: generation 2206's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:45 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:45 activator: bot.x.internal hibernated
2026/10/24 22:54:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:45 activator: unproven.example.com: failed wake's generation 2206 is proven stopped; lease given back
2026/10/24 22:54:45 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:45 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:45 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 22:54:45 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: unproven.example.com hibernated
2026/10/24 22:54:45 activator: one.example.com: generation 2210 of v3: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:45 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:45 activator: SIDECAR for one.example.com failed to start: the build state could not be written, so this build was not started: bucket refused the build state — serving without it (the sweep retries)
2026/10/24 22:54:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/24 22:54:45 activator: x.example hibernated
2026/10/24 22:54:45 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 22:54:45 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:45 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:45 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 22:54:45 activator: SIDECAR for swap.example.com is not proven gone for its replacement; tracking it (the sweep re-proves it)
2026/10/24 22:54:45 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/24 22:54:45 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 22:54:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 22:54:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 22:54:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:45 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 22:54:45 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: swap.example.com hibernated
2026/10/24 22:54:45 activator: swap.example.com: a discarded SIDECAR is not proven gone (activator: stop unproven; processes remain); keeping it tracked
2026/10/24 22:54:45 activator: swap.example.com hibernated
2026/10/24 22:54:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: flip.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:45 activator: bot.x.internal hibernated
2026/10/24 22:54:45 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 22:54:45 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:45 activator: flip.example.com is no longer hibernate-managed — releasing
2026/10/24 22:54:45 activator: flip.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/24 22:54:45 activator: flip.example.com hibernated
2026/10/24 22:54:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:45 activator: stuck.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: one.example.com hibernated
2026/10/24 22:54:45 activator: f6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: proven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: f6.example.com idle 2m0s — hibernating
2026/10/24 22:54:45 activator: stop f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 22:54:45 activator: stop f6.example.com: its host cgroup is not proven empty; keeping the lease until it is
2026/10/24 22:54:45 activator: sidecar for swap.example.com was down — restarted (backoff now 4s)
2026/10/24 22:54:45 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 22:54:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: stuck.example.com idle 2m0s — hibernating
2026/10/24 22:54:45 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 22:54:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:45 activator: proven.example.com idle 2m0s — hibernating
2026/10/24 22:54:45 activator: proven.example.com hibernated
2026/10/24 22:54:45 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/24 22:54:45 activator: barrier.example.com hibernated
before
2026/10/24 22:54:45 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:45 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:54:45 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 22:54:45 activator: f6.example.com hibernated
2026/10/24 22:54:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:45 activator: bot6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:45 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/24 22:54:45 activator: worker bot.x.internal exited — restarting (backoff now 4s)
2026/10/24 22:54:45 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:54:45 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 22:54:45 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 22:54:45 activator: bot.x.internal hibernated
2026/10/24 22:54:45 activator: unproven.example.com: failed wake's generation 2232 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:45 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 22:54:45 activator: stop fence.example: earlier process may still run
2026/10/24 22:54:45 activator: fence.example hibernated
2026/10/24 22:54:45 activator: fence.example hibernated
2026/10/24 22:54:45 activator: unproven.example.com: retry stop of failed wake's generation 2232: cgroup.kill: the child is not empty
2026/10/24 22:54:45 activator: pending-f6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 22:54:45 activator: bot.x.internal hibernated
2026/10/24 22:54:45 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 22:54:45 activator: pending-f6.example.com idle 2m0s — hibernating
2026/10/24 22:54:45 activator: stop pending-f6.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 22:54:45 activator: worker bot.x.internal exited — restarting (backoff now 8s)
2026/10/24 22:54:45 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:54:45 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 22:54:45 activator: unproven.example.com: generation 2232's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:45 activator: stop pending-f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 22:54:45 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/24 22:54:45 activator: x.example hibernated
2026/10/24 22:54:45 activator: unproven.example.com: failed wake's generation 2232 is proven stopped; lease given back
2026/10/24 22:54:45 activator: pending-f6.example.com hibernated
2026/10/24 22:54:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:45 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: unproven.example.com hibernated
2026/10/24 22:54:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: swap.example.com hibernated
2026/10/24 22:54:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a generation's limits could not be applied: h.example.com: the placement shim is not available
2026/10/24 22:54:45 activator: undelegated-wake.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 22:54:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:45 activator: h.example.com: started carlos-instance-gen@h.example.com:2242.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:45 activator: one.example.com: generation 2242 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:45 activator: one.example.com: woke in 1ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 1ms, replicate 0s
2026/10/24 22:54:45 activator: nested-stop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: bot.x.internal idle 11s — hibernating
2026/10/24 22:54:45 activator: bot.x.internal hibernated
2026/10/24 22:54:45 activator: nested-stop.example.com idle 2m0s — hibernating
2026/10/24 22:54:45 activator: nested-stop.example.com hibernated
2026/10/24 22:54:45 activator: stop fence.example: stop refused
2026/10/24 22:54:45 activator: fence.example hibernated
2026/10/24 22:54:45 activator: one.example.com hibernated
2026/10/24 22:54:45 activator: seeding seedpin.example.com: /tmp/TestADeclinedFlatSeedRunsThePinnedFile2422549732/001/versions/v1/app seed --db /tmp/TestADeclinedFlatSeedRunsThePinnedFile2422549732/002/seedpin.example.com.db
2026/10/24 22:54:45 activator: seeded seedpin.example.com
2026/10/24 22:54:45 activator: fence.example hibernated
2026/10/24 22:54:45 activator: bot.x.internal hibernated
2026/10/24 22:54:45 activator: bot.x.internal hibernated
2026/10/24 22:54:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:45 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 22:54:45 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: barrier.example.com hibernated
2026/10/24 22:54:46 activator: shutdown-pending.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: shutdown-pending.example.com idle 2m0s — hibernating
2026/10/24 22:54:46 activator: stop shutdown-pending.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 22:54:46 activator: undelegated.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: undelegated.example.com idle 2m0s — hibernating
2026/10/24 22:54:46 activator: undelegated.example.com: stopped with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 22:54:46 activator: undelegated.example.com hibernated
2026/10/24 22:54:46 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 22:54:46 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 22:54:46 activator: one.example.com: generation 2252 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:46 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 22:54:46 activator: bot7.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:46 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/24 22:54:46 activator: bot.x.internal hibernated
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:46 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 22:54:46 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:46 activator: tickheld.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal idle 11s — hibernating
2026/10/24 22:54:46 activator: bot.x.internal hibernated
2026/10/24 22:54:46 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 22:54:46 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: unproven.example.com: generation 2258's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:46 activator: unproven.example.com: failed wake's generation 2258 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:46 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 22:54:46 activator: tickheld.bab.oncarlos.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 22:54:46 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 22:54:46 activator: tickheld.bab.oncarlos.com hibernated
2026/10/24 22:54:46 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 22:54:46 activator: unproven.example.com: generation 2258's child is not proven empty: the generation's child still holds a process
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: no-host-proof.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: unproven.example.com: failed wake's generation 2258 is proven stopped; lease given back
2026/10/24 22:54:46 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: unproven.example.com hibernated
2026/10/24 22:54:46 activator: shutdown-pending.example.com hibernated
2026/10/24 22:54:46 activator: untimed.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: stop fence.example: private backend detail
2026/10/24 22:54:46 activator: fence.example hibernated
2026/10/24 22:54:46 activator: h.example.com: kill carlos-instance-gen@h.example.com:2262.service after a failed start: stop timed out
2026/10/24 22:54:46 activator: one.example.com: wake failed and its generation is not proven stopped: start one.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2262.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/24 22:54:46 activator: untimed.bab.oncarlos.com idle 16m0s — hibernating
2026/10/24 22:54:46 activator: untimed.bab.oncarlos.com hibernated
2026/10/24 22:54:46 activator: f6.example.com: failed wake's generation 4 ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 22:54:46 activator: faroff.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: ledgerpay.bde.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal idle 11s — hibernating
2026/10/24 22:54:46 activator: bot.x.internal hibernated
2026/10/24 22:54:46 activator: faroff.bab.oncarlos.com idle 16m0s — hibernating
2026/10/24 22:54:46 activator: faroff.bab.oncarlos.com hibernated
2026/10/24 22:54:46 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 22:54:46 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:46 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: one.example.com idle 1m0s — hibernating
2026/10/24 22:54:46 activator: shop.example.com: seed placement not recorded yet: record unreachable
2026/10/24 22:54:46 activator: shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:46 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 22:54:46 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: swap.example.com hibernated
2026/10/24 22:54:46 activator: swap.example.com: generation 2269 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:46 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:46 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 22:54:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: barrier.example.com hibernated
2026/10/24 22:54:46 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: processes remained; the sweep tries again
2026/10/24 22:54:46 activator: one.example.com hibernated
2026/10/24 22:54:46 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:46 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 22:54:46 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: one.example.com hibernated
2026/10/24 22:54:46 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: LEASE LOST for one.example.com (lease renewal failed) — standing down
2026/10/24 22:54:46 activator: one.example.com hibernated
2026/10/24 22:54:46 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com idle 1h0m0s — hibernating
2026/10/24 22:54:46 activator: c.example.com hibernated
2026/10/24 22:54:46 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 22:54:46 activator: one.example.com hibernated
2026/10/24 22:54:46 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com hibernated
2026/10/24 22:54:46 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:46 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 22:54:46 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 22:54:46 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 22:54:46 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 22:54:46 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 22:54:46 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 22:54:46 activator: no-host-proof.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 22:54:46 activator: bot.x.internal hibernated
2026/10/24 22:54:46 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 22:54:46 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 22:54:46 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: one.example.com serving v1 but v2 is adopted — cycling onto the new build
2026/10/24 22:54:46 activator: one.example.com hibernated
2026/10/24 22:54:46 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com hibernated
2026/10/24 22:54:46 activator: c.example.com hibernated
2026/10/24 22:54:46 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:46 activator: WORKER bot.x.internal failed to respawn (converging onto a new build): exec: live symlink gone — the sweep's crash restart takes it from here
2026/10/24 22:54:46 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:46 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:54:46 activator: worker bot.x.internal respawned (exited), now running v2
2026/10/24 22:54:46 activator: stop sidecar fence.example: private backend detail
2026/10/24 22:54:46 activator: fence.example hibernated
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:46 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:54:46 activator: worker bot.x.internal NOT respawned (exited): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/24 22:54:46 activator: bot.x.internal hibernated
2026/10/24 22:54:46 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:46 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:54:46 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:46 activator: worker bot.x.internal NOT respawned (converging onto a new build): instance config unavailable on this box: cannot open sealed config — standing down and releasing its lease
2026/10/24 22:54:46 activator: bot.x.internal hibernated
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: barrier.example.com hibernated
2026/10/24 22:54:46 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:46 activator: h.example.com: carlos-instance-gen@h.example.com:1.service launched in 0s, healthy 0s later
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: always-on sidecar bot.x.internal is not running — starting it
2026/10/24 22:54:46 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:46 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:46 activator: h.example.com: carlos-instance-gen@h.example.com:6.service launched in 0s, healthy 0s later
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com idle 1h0m0s — hibernating
2026/10/24 22:54:46 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:46 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 22:54:46 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:54:46 activator: worker bot.x.internal respawned (hit its max_run), now running 
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com hibernated
2026/10/24 22:54:46 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:46 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:54:46 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 22:54:46 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:46 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:54:46 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 22:54:46 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 22:54:46 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:46 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: swap.example.com hibernated
2026/10/24 22:54:46 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/24 22:54:46 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:46 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 22:54:46 activator: bot-a.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 22:54:46 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/24 22:54:46 activator: bot-b.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com hibernated
2026/10/24 22:54:46 activator: no-host-proof.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 8s)
2026/10/24 22:54:46 activator: worker bot-a.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:46 activator: worker bot-a.x.internal respawned (converging onto a new build), now running v2
2026/10/24 22:54:46 activator: h.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: sidecar for swap.example.com was down — restarted (backoff now 16s)
2026/10/24 22:54:46 activator: h.example.com idle 2m0s — hibernating
2026/10/24 22:54:46 activator: stop h.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 22:54:46 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:46 activator: c.example.com hibernated
2026/10/24 22:54:47 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:47 activator: h.example.com hibernated
2026/10/24 22:54:47 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/24 22:54:47 activator: bot.x.internal hibernated
2026/10/24 22:54:47 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 22:54:47 activator: stop c.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 22:54:47 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: c.example.com hibernated
2026/10/24 22:54:47 activator: waking.example: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: waking.example hibernated
2026/10/24 22:54:47 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 22:54:47 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: final checkpoint fence.example: private backend detail
2026/10/24 22:54:47 activator: fence.example hibernated
2026/10/24 22:54:47 activator: c.example.com idle 1h0m0s — hibernating
2026/10/24 22:54:47 activator: c.example.com hibernated
2026/10/24 22:54:47 activator: worker bot-b.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:47 activator: worker bot-b.x.internal respawned (converging onto a new build), now running v2
2026/10/24 22:54:47 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:47 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/24 22:54:47 activator: one.example.com hibernated
2026/10/24 22:54:47 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 22:54:47 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:47 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 22:54:47 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:54:47 activator: worker bot.x.internal NOT respawned (hit its max_run): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/24 22:54:47 activator: bot.x.internal hibernated
2026/10/24 22:54:47 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: f6-layout.example.com: failed wake's generation 2312 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 22:54:47 activator: f6-layout.example.com: failed wake's generation 2312 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:47 activator: c.example.com hibernated
2026/10/24 22:54:47 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: v1 not started: its build state could not be written: bucket: 503
2026/10/24 22:54:47 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:47 activator: h.example.com: carlos-instance-gen@h.example.com:8.service launched in 0s, healthy 0s later
2026/10/24 22:54:47 activator: one.example.com idle 11s — hibernating
2026/10/24 22:54:47 activator: one.example.com hibernated
2026/10/24 22:54:47 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: barrier.example.com hibernated
2026/10/24 22:54:47 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:47 activator: h.example.com: carlos-instance-gen@h.example.com:9.service launched in 0s, healthy 0s later
2026/10/24 22:54:47 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: c.example.com idle 1h0m0s — hibernating
2026/10/24 22:54:47 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 22:54:47 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 22:54:47 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com hibernated
2026/10/24 22:54:47 activator: swap.example.com: generation 2318 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: instance control: 502 mid-roll; the sweep tries again
2026/10/24 22:54:47 activator: c.example.com hibernated
2026/10/24 22:54:47 activator: sidecar staged for awake tenant.test — adopting
2026/10/24 22:54:47 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: sidecar staged for awake tenant.test — adopting
2026/10/24 22:54:47 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/24 22:54:47 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 22:54:47 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 22:54:47 activator: c.example.com hibernated
2026/10/24 22:54:47 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:47 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:47 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:47 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:47 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 22:54:47 activator: one.example.com hibernated
2026/10/24 22:54:47 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:47 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:47 activator: worker bot.x.internal NOT respawned (exited): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/24 22:54:47 activator: instance c.example.com exited unexpectedly — recycling
2026/10/24 22:54:47 activator: c.example.com hibernated
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: fence.example hibernated
2026/10/24 22:54:47 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:47 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 22:54:47 activator: one.example.com: generation 2332 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:47 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:47 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:47 activator: h.example.com: carlos-instance-gen@h.example.com:6.service launched in 0s, healthy 0s later
2026/10/24 22:54:47 activator: instance bots.example.com exited unexpectedly — recycling
2026/10/24 22:54:47 activator: bots.example.com hibernated
2026/10/24 22:54:47 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:47 activator: f6-layout.example.com: failed wake's generation 2336 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 22:54:47 activator: f6-layout.example.com: failed wake's generation 2336 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: bot.x.internal hit its max_run of 1m0s — stopping
2026/10/24 22:54:47 activator: bot.x.internal hibernated
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com hibernated
2026/10/24 22:54:47 activator: one.example.com: generation 2340 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:47 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:47 activator: bot.x.internal idle 31s — hibernating
2026/10/24 22:54:47 activator: bot.x.internal hibernated
2026/10/24 22:54:47 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/24 22:54:47 activator: barrier.example.com hibernated
2026/10/24 22:54:47 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:47 activator: one.example.com idle 11s — hibernating
2026/10/24 22:54:47 activator: one.example.com hibernated
2026/10/24 22:54:47 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com hibernated
2026/10/24 22:54:47 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com hibernated
2026/10/24 22:54:47 activator: swap.example.com: generation 2351 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: shop.example.com: generation 4's child is not proven empty: kill gen: timeout
2026/10/24 22:54:47 activator: plain.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: instance control: 502 mid-roll; the sweep tries again
2026/10/24 22:54:47 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:47 activator: fence.example hibernated
2026/10/24 22:54:47 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: squatted.example.com: child exited during the health wait (external process serving its socket?) — not normalizing the socket mode
2026/10/24 22:54:47 activator: LEASE LOST for c.example.com (lost) — standing down
2026/10/24 22:54:47 activator: c.example.com hibernated
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:47 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: one.example.com hibernated
2026/10/24 22:54:47 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2380343162/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2380343162/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 22:54:47 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2380343162/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2380343162/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 22:54:47 activator: f6-layout.example.com: failed wake's generation 2364 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 22:54:47 activator: f6-layout.example.com: failed wake's generation 2364 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:47 activator: one.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:47 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/24 22:54:47 activator: barrier.example.com hibernated
2026/10/24 22:54:47 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:47 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:54:47 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:47 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:47 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com hibernated
2026/10/24 22:54:47 activator: swap.example.com: generation 2372 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:47 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 22:54:47 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:47 activator: swap.example.com hibernated
2026/10/24 22:54:47 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:47 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/24 22:54:48 activator: shop.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 22:54:48 activator: own.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: own.example.com idle 1m0s — hibernating
2026/10/24 22:54:48 activator: own.example.com hibernated
2026/10/24 22:54:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: one.example.com idle 11s — hibernating
2026/10/24 22:54:48 activator: one.example.com hibernated
2026/10/24 22:54:48 activator: always-on sidecar bot00.x.internal is not running — starting it
2026/10/24 22:54:48 activator: always-on sidecar bot01.x.internal is not running — starting it
2026/10/24 22:54:48 activator: always-on sidecar bot02.x.internal is not running — starting it
2026/10/24 22:54:48 activator: always-on sidecar bot03.x.internal is not running — starting it
2026/10/24 22:54:48 activator: always-on sidecar bot04.x.internal is not running — starting it
2026/10/24 22:54:48 activator: always-on sidecar bot05.x.internal is not running — starting it
2026/10/24 22:54:48 activator: always-on sidecar bot06.x.internal is not running — starting it
2026/10/24 22:54:48 activator: always-on sidecar bot07.x.internal is not running — starting it
2026/10/24 22:54:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: bot04.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot02.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot05.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot00.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot06.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot07.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot03.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot01.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: always-on sidecar bot08.x.internal is not running — starting it
2026/10/24 22:54:48 activator: always-on sidecar bot09.x.internal is not running — starting it
2026/10/24 22:54:48 activator: always-on sidecar bot10.x.internal is not running — starting it
2026/10/24 22:54:48 activator: bot09.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot08.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot10.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 22:54:48 activator: h.example.com: carlos-instance-gen@h.example.com:9.service launched in 0s, healthy 0s later
2026/10/24 22:54:48 activator: fence.example hibernated
2026/10/24 22:54:48 activator: one.example.com: generation 2381 of v3: record 0s, start 20ms, healthy 0s later
2026/10/24 22:54:48 activator: one.example.com: woke in 20ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 20ms, replicate 0s
2026/10/24 22:54:48 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: stop fence.example: activator: stop evidence unavailable
2026/10/24 22:54:48 activator: fence.example hibernated
2026/10/24 22:54:48 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: c.example.com serving v1 but v2 is adopted — cycling onto the new build
2026/10/24 22:54:48 activator: c.example.com hibernated
2026/10/24 22:54:48 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor1232541844/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor1232541844/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 22:54:48 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor1232541844/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor1232541844/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 22:54:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: f6-layout.example.com: failed wake's generation 2389 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 22:54:48 activator: f6-layout.example.com: failed wake's generation 2389 is not proven stopped; keeping the lease until it is
2026/10/24 22:54:48 activator: one.example.com: live/one.example.com names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/24 22:54:48 activator: SIDECAR for botless.example.com failed to start: exec: live symlink gone — serving without it (the sweep retries)
2026/10/24 22:54:48 activator: botless.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:48 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/24 22:54:48 activator: SIDECAR restart for botless.example.com failed: exec: live symlink gone (next attempt in 2s)
2026/10/24 22:54:48 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/24 22:54:48 activator: one.example.com hibernated
2026/10/24 22:54:48 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: swap.example.com hibernated
2026/10/24 22:54:48 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: barrier.example.com hibernated
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: sidecar for botless.example.com was down — restarted (backoff now 4s)
2026/10/24 22:54:48 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:48 activator: bot.x.internal idle 11s — hibernating
2026/10/24 22:54:48 activator: bot.x.internal hibernated
2026/10/24 22:54:48 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:48 activator: one.example.com: generation 2395 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 22:54:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot.x.internal idle 1m31s — hibernating
2026/10/24 22:54:48 activator: bot.x.internal hibernated
2026/10/24 22:54:48 activator: one.example.com hibernated
2026/10/24 22:54:48 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:48 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/24 22:54:48 activator: one.example.com idle 5m1s — hibernating
2026/10/24 22:54:48 activator: one.example.com hibernated
2026/10/24 22:54:48 activator: sidecar for bots.example.com was down — restarted (backoff now 8s)
2026/10/24 22:54:48 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: mid.example.com has a scheduled delivery due — waking it
2026/10/24 22:54:48 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:48 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished601052077/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished601052077/001/zdd: not a directory; the sweep retries
2026/10/24 22:54:48 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished601052077/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished601052077/001/zdd: not a directory; the sweep retries
2026/10/24 22:54:48 activator: final checkpoint fence.example: activator: stop evidence unavailable
2026/10/24 22:54:48 activator: fence.example hibernated
2026/10/24 22:54:48 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawngated_build_is_the_ex1193141457/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:48 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/24 22:54:48 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawna_held_build_starts_n952526697/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:48 activator: mid.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: mid.example.com is no longer hibernate-managed — releasing (wake finished first)
2026/10/24 22:54:48 activator: mid.example.com hibernated
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake878727660/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake878727660/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 22:54:48 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake878727660/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake878727660/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 22:54:48 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:48 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it stays down until a release attempt or a wake starts it
2026/10/24 22:54:48 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: c.example.com hibernated
2026/10/24 22:54:48 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:48 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:48 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:48 activator: reborn.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: reborn.example.com idle 11s — hibernating
2026/10/24 22:54:48 activator: reborn.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/24 22:54:48 activator: reborn.example.com hibernated
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: barrier.example.com hibernated
2026/10/24 22:54:48 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/24 22:54:48 activator: reborn.example.com has a scheduled delivery due — waking it
2026/10/24 22:54:48 activator: reborn.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawnno_gate_is_todays_sp2300469360/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 22:54:48 activator: deaf.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: LEASE LOST for bot.x.internal (lease renewal failed) — standing down
2026/10/24 22:54:48 activator: bot.x.internal hibernated
2026/10/24 22:54:48 activator: deaf.example.com idle 11s — hibernating
2026/10/24 22:54:48 activator: next-due probe deaf.example.com: dial unix: no such file — recording none (it sleeps until a request)
2026/10/24 22:54:48 activator: deaf.example.com hibernated
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:48 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: one.example.com hibernated
2026/10/24 22:54:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: one.example.com hibernated
2026/10/24 22:54:48 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:48 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:48 activator: bot.x.internal idle 1m1s — hibernating
2026/10/24 22:54:48 activator: bot.x.internal hibernated
2026/10/24 22:54:48 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: bots.example.com idle 11s — hibernating
2026/10/24 22:54:48 activator: bots.example.com hibernated
2026/10/24 22:54:48 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:48 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it recycles its captured build v1
2026/10/24 22:54:48 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:48 activator: one.example.com: sidecar starting on the instance's own build v1 (/opt/carlos/versions/v1/app): this box will not start v2, the build its live link names
2026/10/24 22:54:48 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/24 22:54:48 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:48 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: fence.example hibernated
2026/10/24 22:54:48 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:48 activator: worker bot.x.internal serving v1 but live symlink is now v3 — respawning onto the new build
2026/10/24 22:54:48 activator: worker bot.x.internal respawned (converging onto a new build), now running v3
2026/10/24 22:54:48 activator: bots.example.com idle 11s — hibernating
2026/10/24 22:54:48 activator: bots.example.com hibernated
2026/10/24 22:54:48 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 22:54:48 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 22:54:48 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 22:54:48 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake2690966611/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake2690966611/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 22:54:48 activator: x.example: hold's pinned wake refused or abandoned: x.example: refusing to wake: staged build not published before its exec
2026/10/24 22:54:48 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake2690966611/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake2690966611/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 22:54:48 activator: due.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: late.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: due.example.com idle 11s — hibernating
2026/10/24 22:54:48 activator: due.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/24 22:54:48 activator: due.example.com hibernated
2026/10/24 22:54:48 activator: late.example.com idle 11s — hibernating
2026/10/24 22:54:48 activator: late.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/24 22:54:48 activator: late.example.com hibernated
2026/10/24 22:54:48 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: c.example.com is no longer hibernate-managed — releasing
2026/10/24 22:54:48 activator: c.example.com hibernated
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: one.example.com: generation 2418 of v2: record 0s, start 18ms, healthy 0s later
2026/10/24 22:54:48 activator: one.example.com: woke in 20ms: checks 2ms, lease 0s, restore 0s, identity 0s, spawn 19ms, replicate 0s
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: due.example.com has a scheduled delivery due — waking it
2026/10/24 22:54:48 activator: due.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: late.example.com has a scheduled delivery due — waking it
2026/10/24 22:54:48 activator: late.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: one.example.com idle 11s — hibernating
2026/10/24 22:54:48 activator: one.example.com hibernated
2026/10/24 22:54:48 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 22:54:48 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: the running child keeps running
2026/10/24 22:54:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: two.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 22:54:48 activator: due.example.com idle 11s — hibernating
2026/10/24 22:54:48 activator: due.example.com hibernated
2026/10/24 22:54:48 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 22:54:48 activator: one.example.com hibernated
2026/10/24 22:54:48 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 22:54:48 activator: worker bot.x.internal reported idle but is always-on (hibernate:false) — ignoring
2026/10/24 22:54:48 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 22:54:48 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 22:54:48 activator: bot.x.internal hibernated
2026/10/24 22:54:48 activator: two.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 22:54:48 activator: two.example.com hibernated
2026/10/24 22:54:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:48 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, nor the instance's own build v1: a failure record holds this build: v1
2026/10/24 22:54:48 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/24 22:54:48 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:48 activator: c.example.com hibernated
2026/10/24 22:54:48 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:49 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 22:54:49 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, and the instance's own build v1 cannot be named: staged copy unreadable
2026/10/24 22:54:49 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/24 22:54:49 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:49 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 22:54:49 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
FAIL
FAIL	github.com/carlosframework/platform/internal/activator	58.479s
ok  	github.com/carlosframework/platform/internal/agentsock	0.047s
ok  	github.com/carlosframework/platform/internal/alerting	0.021s
ok  	github.com/carlosframework/platform/internal/analytics	0.049s
ok  	github.com/carlosframework/platform/internal/appconfig	2.902s
ok  	github.com/carlosframework/platform/internal/apperr	0.213s
ok  	github.com/carlosframework/platform/internal/applog	0.373s
ok  	github.com/carlosframework/platform/internal/assetcache	0.220s
ok  	github.com/carlosframework/platform/internal/awsinv	0.005s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.008s
ok  	github.com/carlosframework/platform/internal/boxid	0.989s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.018s
ok  	github.com/carlosframework/platform/internal/caps	0.028s
ok  	github.com/carlosframework/platform/internal/certwarm	0.004s
ok  	github.com/carlosframework/platform/internal/cgroup	0.811s
ok  	github.com/carlosframework/platform/internal/changelog	0.627s
ok  	github.com/carlosframework/platform/internal/claimactivation	1.323s
ok  	github.com/carlosframework/platform/internal/clamd	0.075s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	8.229s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.032s
ok  	github.com/carlosframework/platform/internal/dbquery	0.021s
ok  	github.com/carlosframework/platform/internal/dbreads	2.156s
?   	github.com/carlosframework/platform/internal/dbreads/dbreadstest	[no test files]
ok  	github.com/carlosframework/platform/internal/dbverify	0.013s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.020s
ok  	github.com/carlosframework/platform/internal/docsite	0.033s
ok  	github.com/carlosframework/platform/internal/domains	2.913s
ok  	github.com/carlosframework/platform/internal/doorkey	0.027s
ok  	github.com/carlosframework/platform/internal/economics	0.011s
ok  	github.com/carlosframework/platform/internal/edge	45.695s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.008s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.133s
ok  	github.com/carlosframework/platform/internal/gate	1.655s
ok  	github.com/carlosframework/platform/internal/genselect	0.015s
ok  	github.com/carlosframework/platform/internal/geo	0.014s
ok  	github.com/carlosframework/platform/internal/geoip	0.029s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.153s
ok  	github.com/carlosframework/platform/internal/host	143.225s
ok  	github.com/carlosframework/platform/internal/hostactors	5.994s
ok  	github.com/carlosframework/platform/internal/hostactors/awsmembership	0.010s
ok  	github.com/carlosframework/platform/internal/hostbox	3.158s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.758s
ok  	github.com/carlosframework/platform/internal/hostenv	0.006s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.006s
ok  	github.com/carlosframework/platform/internal/hostexec	5.297s
ok  	github.com/carlosframework/platform/internal/hostfence	0.050s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.338s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.020s
ok  	github.com/carlosframework/platform/internal/hostguard	1.031s
ok  	github.com/carlosframework/platform/internal/hostops	1.914s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.003s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.003s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.644s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.214s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.004s
ok  	github.com/carlosframework/platform/internal/infraview	0.003s
ok  	github.com/carlosframework/platform/internal/instancectl	5.237s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.851s
ok  	github.com/carlosframework/platform/internal/instancefs	0.006s
ok  	github.com/carlosframework/platform/internal/instancekey	0.003s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.017s
ok  	github.com/carlosframework/platform/internal/layout	1.763s
ok  	github.com/carlosframework/platform/internal/lbhealth	0.007s
ok  	github.com/carlosframework/platform/internal/lease	10.241s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.583s
ok  	github.com/carlosframework/platform/internal/ledger	0.018s
ok  	github.com/carlosframework/platform/internal/mail	0.221s
ok  	github.com/carlosframework/platform/internal/maintenance	0.077s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.533s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.083s
ok  	github.com/carlosframework/platform/internal/metrics	0.006s
ok  	github.com/carlosframework/platform/internal/moves	22.313s
ok  	github.com/carlosframework/platform/internal/moves/archive	1.066s
ok  	github.com/carlosframework/platform/internal/objstore	0.338s
ok  	github.com/carlosframework/platform/internal/ops	0.244s
ok  	github.com/carlosframework/platform/internal/park	0.320s
ok  	github.com/carlosframework/platform/internal/plans	0.009s
ok  	github.com/carlosframework/platform/internal/privatefile	0.006s
ok  	github.com/carlosframework/platform/internal/provision	23.742s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.010s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.592s
ok  	github.com/carlosframework/platform/internal/relay	7.240s
ok  	github.com/carlosframework/platform/internal/release	25.664s
--- FAIL: TestAMoveOntoTheInstalledGoodBuildWaitsForASignedRollback (5.03s)
    --- FAIL: TestAMoveOntoTheInstalledGoodBuildWaitsForASignedRollback/worker (5.01s)
        holdran_test.go:796: timed out waiting for the move onto v1
FAIL
FAIL	github.com/carlosframework/platform/internal/rollout	8.643s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.004s
ok  	github.com/carlosframework/platform/internal/rootaws	0.010s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.066s
ok  	github.com/carlosframework/platform/internal/rootstate	0.027s
ok  	github.com/carlosframework/platform/internal/runs	0.590s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.005s
ok  	github.com/carlosframework/platform/internal/schedule	2.695s
ok  	github.com/carlosframework/platform/internal/seeds	0.010s
ok  	github.com/carlosframework/platform/internal/sim/engines	2.722s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	10.484s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.984s
ok  	github.com/carlosframework/platform/internal/status	0.077s
ok  	github.com/carlosframework/platform/internal/steering	0.012s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.286s
ok  	github.com/carlosframework/platform/internal/svcrole	0.006s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.005s
ok  	github.com/carlosframework/platform/internal/unixdial	0.004s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.006s
ok  	github.com/carlosframework/platform/internal/version	0.003s
ok  	github.com/carlosframework/platform/internal/vet	1.696s
FAIL
=== internal_console@0 (84 s)
ok  	github.com/carlosframework/platform/internal/console	82.441s
=== internal_console@1 (40 s)
ok  	github.com/carlosframework/platform/internal/console	39.167s
=== internal_console@2 (43 s)
ok  	github.com/carlosframework/platform/internal/console	41.991s
=== internal_console@3 (46 s)
ok  	github.com/carlosframework/platform/internal/console	44.932s
=== internal_console@4 (42 s)
ok  	github.com/carlosframework/platform/internal/console	40.790s
=== internal_console@5 (38 s)
ok  	github.com/carlosframework/platform/internal/console	37.188s
=== internal_console@6 (44 s)
ok  	github.com/carlosframework/platform/internal/console	42.687s
=== internal_console@7 (46 s)
ok  	github.com/carlosframework/platform/internal/console	44.945s
=== root@0 (113 s)
ok  	github.com/carlosframework/platform	111.490s
=== root@1 (97 s)
ok  	github.com/carlosframework/platform	95.558s
=== root@2 (78 s)
ok  	github.com/carlosframework/platform	77.081s

FAILED: rest
make: *** [Makefile:19: test] Error 1

amadan: step gate failed: exit status 2

amadan: job failed at step gate — its output is above, not at the tail of this log

diff --git a/docs/superpowers/notes/census-proposal-wake-latency-3.md b/docs/superpowers/notes/census-proposal-wake-latency-3.md

new file mode 100644

index 000000000..b65a2c40e

--- /dev/null

+++ b/docs/superpowers/notes/census-proposal-wake-latency-3.md

@@ -0,0 +1,13 @@

+# Census approval: wake-latency-3

+Ruling `census-ruling-wake-latency-3`. Every change below keeps the disposition it already has, so it needs an approval, not a decision.

+|---|---|---|---|

+| `internal/host/buildstate.go` `buildStates.write` | body | unchanged (1) | its first try goes on the copy this agent last wrote at that key, conditioned on that copy's etag (one round trip); a conflict, an error, or a merge that refuses or has nothing to write on the copy is decided on a fresh read as before; no read added (the fresh read is the one it always made) |

+Envelopes re-pinned: `internal/appconfig`, `internal/host`, `internal/lease`.

+Seal: `86570d1326baad6d866b4066b6bf65218434b201fb1f2c05ebc6e3c79f9c5a98` → `86570d1326baad6d866b4066b6bf65218434b201fb1f2c05ebc6e3c79f9c5a98`.

+Overlay: `internal/provision/testdata/plan14-root-reads-wake-latency-3.tsv`, appended to `internal/provision/testdata/plan14-root-reads-chain.tsv`.

diff --git a/internal/appconfig/environments.go b/internal/appconfig/environments.go

index d7da25498..1ddc56e13 100644

--- a/internal/appconfig/environments.go

+++ b/internal/appconfig/environments.go

@@ -16,6 +16,7 @@ import (

"regexp"

"sort"

"strings"

+ "sync"

"github.com/carlosframework/platform/internal/release"

)

@@ -182,7 +183,27 @@ func loadMergedETags(ctx context.Context, s release.Store, acct, app, envName st

if !release.ValidApp(app) {

return nil, "", "", fmt.Errorf("config: invalid app %q", app)

}

+ // The environment is read beside the default bundle, not after it: two

+ // independent objects, and a wake's config admission waits on both

+ // before its lease (wake latency, pass 2). The answers are taken in the

+ // order the reads used to run, so every error is the one it was.

+ var (

+ env *Manifest

+ envETag string

+ envFound bool

+ envErr error

+ envRead = envName != "" && ValidEnvironmentName(envName)

+ wg sync.WaitGroup

+ )

+ if envRead {

+ wg.Add(1)

+ go func() {

+ defer wg.Done()

+ env, envETag, envFound, envErr = getIfExistsAtETag(ctx, s, EnvironmentKey(acct, app, envName))

+ }()

+ }

def, defETag, defFound, err := getIfExistsAtETag(ctx, s, Key(acct, app))

+ wg.Wait()

if err != nil {

return nil, "", "", err

}

@@ -192,11 +213,10 @@ func loadMergedETags(ctx context.Context, s release.Store, acct, app, envName st

}

return def, defETag, "", nil

}

- if !ValidEnvironmentName(envName) {

+ if !envRead {

return nil, "", "", fmt.Errorf("config: invalid environment name %q", envName)

}

- env, envETag, envFound, err := getIfExistsAtETag(ctx, s, EnvironmentKey(acct, app, envName))

- if err != nil {

+ if err := envErr; err != nil {

return nil, "", "", err

}

if !envFound {

diff --git a/internal/appconfig/instances.go b/internal/appconfig/instances.go

index 022ee8ebd..1cc6eaf87 100644

--- a/internal/appconfig/instances.go

+++ b/internal/appconfig/instances.go

@@ -29,6 +29,7 @@ import (

"fmt"

"regexp"

"strings"

+ "sync"

"github.com/carlosframework/platform/internal/release"

)

@@ -223,13 +224,27 @@ func LoadMergedHostVersions(ctx context.Context, s release.Store, acct, app, env

if !ValidInstanceHost(host) {

return nil, MergedVersions{}, fmt.Errorf("config: invalid instance host %q", host)

}

+ // The host's overlay is read beside the app's layers (loadMergedETags),

+ // not after them; the answers are taken in the old order.

+ var (

+ inst *Manifest

+ etag string

+ found bool

+ instErr error

+ wg sync.WaitGroup

+ )

+ wg.Add(1)

+ go func() {

+ defer wg.Done()

+ inst, etag, found, instErr = LoadInstanceConfig(ctx, s, acct, app, host)

+ }()

base, defETag, envETag, err := loadMergedETags(ctx, s, acct, app, envName)

+ wg.Wait()

if err != nil {

return nil, MergedVersions{}, err

}

v := MergedVersions{Default: defETag, Environment: envETag}

- inst, etag, found, err := LoadInstanceConfig(ctx, s, acct, app, host)

- if err != nil {

+ if err := instErr; err != nil {

return nil, MergedVersions{}, err

}

if !found {

diff --git a/internal/appconfig/loadconcurrent_test.go b/internal/appconfig/loadconcurrent_test.go

new file mode 100644

index 000000000..978512187

--- /dev/null

+++ b/internal/appconfig/loadconcurrent_test.go

@@ -0,0 +1,83 @@

+package appconfig

+import (

+ "context"

+ "encoding/json"

+ "sync"

+ "testing"

+ "time"

+ "github.com/carlosframework/platform/internal/release"

+// gatedStore holds every Get until want of them are in flight at once, or

+// a deadline passes. A loader that reads its layers one after another

+// waits out the deadline on each; one that reads them together does not.

+type gatedStore struct {

+ release.Store

+ want int

+ mu sync.Mutex

+ in int

+ open chan struct{}

+func (g *gatedStore) Get(ctx context.Context, key string) ([]byte, string, error) {

+ g.mu.Lock()

+ g.in++

+ if g.in == g.want {

+ close(g.open)

+ }

+ g.mu.Unlock()

+ select {

+ case <-g.open:

+ case <-time.After(300 * time.Millisecond):

+ }

+ return g.Store.Get(ctx, key)

+func putManifest(t *testing.T, s release.Store, key string, m *Manifest) {

+ t.Helper()

+ raw, err := json.Marshal(m)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if _, err := s.Put(context.Background(), key, raw, ""); err != nil {

+ t.Fatal(err)

+ }

+// A wake's config admission reads three layers (the app's default bundle,

+// its environment, the host's overlay). They are independent objects, so

+// they are read together: one round trip, not three, before the lease.

+func TestLoadMergedHostVersionsReadsItsLayersTogether(t *testing.T) {

+ dir := release.NewDirStore(t.TempDir())

+ putManifest(t, dir, Key("bab", "shop"), &Manifest{Schema: 1, App: "shop", Vars: map[string]Var{"A": {Kind: "env", Value: "default"}, "B": {Kind: "env", Value: "default"}}})

+ putManifest(t, dir, EnvironmentKey("bab", "shop", "prod"), &Manifest{Schema: 1, App: "shop", Vars: map[string]Var{"B": {Kind: "env", Value: "env"}, "C": {Kind: "env", Value: "env"}}})

+ putManifest(t, dir, InstanceConfigKey("bab", "shop", "one.example.com"), &Manifest{Schema: 1, App: "shop", Vars: map[string]Var{"C": {Kind: "env", Value: "host"}}})

+ g := &gatedStore{Store: dir, want: 3, open: make(chan struct{})}

+ start := time.Now()

+ m, v, err := LoadMergedHostVersions(context.Background(), g, "bab", "shop", "prod", "one.example.com")

+ if err != nil {

+ t.Fatal(err)

+ }

+ if took := time.Since(start); took > 250*time.Millisecond {

+ t.Errorf("took %s: the three layers were not read together", took)

+ }

+ if m.Vars["A"].Value != "default" || m.Vars["B"].Value != "env" || m.Vars["C"].Value != "host" {

+ t.Errorf("merge = %+v, want default < environment < host", m.Vars)

+ }

+ if v.Default == "" || v.Environment == "" || v.Overlay == "" {

+ t.Errorf("versions = %+v, want each layer's etag", v)

+ }

+// Reading together changes no answer: an environment the route is bound to

+// but nobody created is still the error, whatever the other reads found.

+func TestLoadMergedHostVersionsStillRefusesAMissingEnvironment(t *testing.T) {

+ dir := release.NewDirStore(t.TempDir())

+ putManifest(t, dir, Key("bab", "shop"), &Manifest{Schema: 1, App: "shop", Vars: map[string]Var{}})

+ putManifest(t, dir, InstanceConfigKey("bab", "shop", "one.example.com"), &Manifest{Schema: 1, App: "shop", Vars: map[string]Var{}})

+ if _, _, err := LoadMergedHostVersions(context.Background(), dir, "bab", "shop", "prod", "one.example.com"); err == nil {

+ t.Fatal("a missing environment was accepted")

+ }

diff --git a/internal/host/buildstate.go b/internal/host/buildstate.go

index eab6ea3ea..a6676123f 100644

--- a/internal/host/buildstate.go

+++ b/internal/host/buildstate.go

@@ -59,6 +59,9 @@ type buildStates struct {

// resume is the first host the last refreshAll did not reach; the next

// one starts there (ruling R-REFRESH-ROTATE). Guarded by mu.

resume string

+ // written is each object write last put, by bucket and key, for the

+ // next write's conditional first try (writtenCopy). Guarded by mu.

+ written map[string]writtenCopy

}

// newBuildStates is nil, nil on a box with no replica bucket: it runs no

@@ -133,16 +136,23 @@ func cachedStores(regions map[string]string, open func(ctx context.Context, buck

// holder's Location in the claim bucket (an unreadable claim is an error,

// never the hibernate prefix); otherwise the hibernate bucket and prefix.

func (b *buildStates) locate(ctx context.Context, host string) (release.Store, string, error) {

+ _, s, prefix, err := b.locateIn(ctx, host)

+ return s, prefix, err

+// locateIn is locate with the bucket it chose, which keys write's copy of

+// the object it last wrote.

+func (b *buildStates) locateIn(ctx context.Context, host string) (string, release.Store, string, error) {

bucket, prefix := b.hibBucket, ""

if l, ok := activator.BuildStateLocationFrom(ctx); ok {

bucket, prefix = l.Bucket, l.Prefix

} else if b.claims != nil {

obs, err := b.claims(ctx, host)

if err != nil {

- return nil, "", fmt.Errorf("read claim: %w", err)

+ return "", nil, "", fmt.Errorf("read claim: %w", err)

}

if obs.Absent || obs.Claim == nil || obs.Claim.Holder == nil || obs.Claim.Holder.Location == "" {

- return nil, "", fmt.Errorf("claim of %s has no holder location", host)

+ return "", nil, "", fmt.Errorf("claim of %s has no holder location", host)

}

bucket, prefix = b.claimBucket, obs.Claim.Holder.Location

} else {

@@ -151,9 +161,9 @@ func (b *buildStates) locate(ctx context.Context, host string) (release.Store, s

prefix = strings.TrimSuffix(prefix, "/")

s, err := b.storeFor(ctx, bucket)

if err != nil {

- return nil, "", err

+ return "", nil, "", err

}

- return s, prefix, nil

+ return bucket, s, prefix, nil

}

func (b *buildStates) hostLock(host string) *sync.Mutex {

@@ -452,17 +462,38 @@ func (b *buildStates) write(ctx context.Context, host string, merge func(cur *re

m := b.hostLock(host)

m.Lock()

defer m.Unlock()

- store, prefix, err := b.locate(ctx, host)

+ bucket, store, prefix, err := b.locateIn(ctx, host)

if err != nil {

return release.BuildState{}, err

}

key := release.BuildStateKey(prefix)

- for try := 0; try < updateTries; try++ {

- cur, etag, err := getState(ctx, store, key, host)

- if err != nil {

+ // The first try goes on the copy this agent last wrote, when it has

+ // one: a wake's record is then one conditional write, not a read and a

+ // write. The condition is that copy's etag, so the write lands only on

+ // the very object the merge saw, and the merge decides exactly what it

+ // would on a fresh read. Anything else (the object moved on, the merge

+ // refused or had nothing to write on a copy that may be stale) is

+ // decided again on a fresh read, as before.

+ cached, hasCopy := b.lastWritten(bucket, key)

+ tries := updateTries

+ if hasCopy {

+ tries++ // the copy's try is extra: the fresh reads keep all theirs

+ }

+ for try := 0; try < tries; try++ {

+ var (

+ cur *release.BuildState

+ etag string

+ )

+ fromCopy := hasCopy && try == 0

+ if fromCopy {

+ cur, etag = cached.state(), cached.etag

+ } else if cur, etag, err = getState(ctx, store, key, host); err != nil {

return release.BuildState{}, err

}

next, put, err := merge(cur)

+ if fromCopy && (err != nil || !put) {

+ continue

+ }

if err != nil {

return release.BuildState{}, err

}

@@ -481,11 +512,15 @@ func (b *buildStates) write(ctx context.Context, host string, merge func(cur *re

if b.beforePut != nil {

b.beforePut(ctx, store, key)

}

- if _, err := store.Put(ctx, key, raw, etag); errors.Is(err, release.ErrCASConflict) {

+ newEtag, err := store.Put(ctx, key, raw, etag)

+ if errors.Is(err, release.ErrCASConflict) {

+ b.forget(bucket, key)

continue

} else if err != nil {

+ b.forget(bucket, key)

return release.BuildState{}, err

}

+ b.remember(bucket, key, raw, newEtag)

// A process about to run against the database has (or is about to

// have) a replica; any other writer says.

hasRep := true

@@ -503,6 +538,49 @@ func (b *buildStates) write(ctx context.Context, host string, merge func(cur *re

return release.BuildState{}, fmt.Errorf("build state of %s: %d tries lost the race: %w", host, updateTries, release.ErrCASConflict)

}

+// writtenCopy is the object write last put at a key, and the etag that

+// write returned: the state the next write's conditional first try goes on.

+type writtenCopy struct {

+ raw []byte

+ etag string

+// state decodes a fresh copy each time, so a merge can never change the

+// cached bytes.

+func (c writtenCopy) state() *release.BuildState {

+ var s release.BuildState

+ if json.Unmarshal(c.raw, &s) != nil {

+ return nil

+ }

+ return &s

+func (b *buildStates) lastWritten(bucket, key string) (writtenCopy, bool) {

+ b.mu.Lock()

+ defer b.mu.Unlock()

+ c, ok := b.written[bucket+"\x00"+key]

+ return c, ok && c.etag != "" && c.state() != nil

+func (b *buildStates) remember(bucket, key string, raw []byte, etag string) {

+ b.mu.Lock()

+ defer b.mu.Unlock()

+ if etag == "" {

+ delete(b.written, bucket+"\x00"+key)

+ return

+ }

+ if b.written == nil {

+ b.written = map[string]writtenCopy{}

+ }

+ b.written[bucket+"\x00"+key] = writtenCopy{raw: append([]byte(nil), raw...), etag: etag}

+func (b *buildStates) forget(bucket, key string) {

+ b.mu.Lock()

+ defer b.mu.Unlock()

+ delete(b.written, bucket+"\x00"+key)

// beforeExec is Service.BeforeExec (settings spec §4.2): before build runs

// against host's database, under its lease, the object names build as

// last_executed, carries root's resolution from the consumed file, and

diff --git a/internal/host/buildstate_cas_test.go b/internal/host/buildstate_cas_test.go

new file mode 100644

index 000000000..63961f410

--- /dev/null

+++ b/internal/host/buildstate_cas_test.go

@@ -0,0 +1,126 @@

+package host

+import (

+ "context"

+ "errors"

+ "testing"

+ "github.com/carlosframework/platform/internal/release"

+// bsGetCounter counts reads, so a test can see a write that needed none.

+type bsGetCounter struct {

+ release.Store

+ gets int

+func (c *bsGetCounter) Get(ctx context.Context, key string) ([]byte, string, error) {

+ c.gets++

+ return c.Store.Get(ctx, key)

+func countingBuildStates(t *testing.T) (*buildStates, *bsGetCounter) {

+ t.Helper()

+ b, s, _ := testBuildStates(t)

+ cs := &bsGetCounter{Store: s}

+ b.storeFor = storesOf(map[string]release.Store{"hib": cs})

+ return b, cs

+// The state this agent last wrote is the one a conditional write can go

+// on: a wake's record is then one round trip, not a read and a write.

+func TestBuildStateWritesOnItsOwnLastCopyWithoutARead(t *testing.T) {

+ b, cs := countingBuildStates(t)

+ ctx := context.Background()

+ if err := put(t, b, ctx, "a.example.com", release.BuildState{LastExecuted: "v1", Completed: 1}); err != nil {

+ t.Fatal(err)

+ }

+ cs.gets = 0

+ got, err := b.update(ctx, "a.example.com", bump)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if cs.gets != 0 {

+ t.Errorf("the write read the object %d time(s); its own last copy was current", cs.gets)

+ }

+ if got.Completed != 2 || got.LastExecuted != "v1" {

+ t.Errorf("state = %+v, want the cached copy bumped", got)

+ }

+ cur, _, err := getState(ctx, cs.Store, "inst/a.example.com/build-state.json", "a.example.com")

+ if err != nil || cur == nil || cur.Completed != 2 {

+ t.Fatalf("object = %+v, %v", cur, err)

+ }

+// Another writer moved the object on: the conditional write fails, and the

+// write reads it and merges on what is really there.

+func TestBuildStateCopyThatMovedOnIsReadAgain(t *testing.T) {

+ b, cs := countingBuildStates(t)

+ ctx := context.Background()

+ if err := put(t, b, ctx, "a.example.com", release.BuildState{LastExecuted: "v1", Completed: 1}); err != nil {

+ t.Fatal(err)

+ }

+ key := "inst/a.example.com/build-state.json"

+ if _, err := cs.Store.Put(ctx, key, []byte(`{"schema":1,"host":"a.example.com","last_executed":"v9","completed":5}`), mustEtag(t, cs.Store, key)); err != nil {

+ t.Fatal(err)

+ }

+ cs.gets = 0

+ got, err := b.update(ctx, "a.example.com", bump)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if cs.gets == 0 {

+ t.Error("the write did not read an object that had moved on")

+ }

+ if got.Completed != 6 || got.LastExecuted != "v9" {

+ t.Errorf("state = %+v, want the other writer's state bumped", got)

+ }

+// A merge that refuses on the cached copy is asked again on a fresh read:

+// the copy may be stale, and a refusal on stale state would be wrong.

+func TestBuildStateRefusalOnTheCachedCopyIsCheckedAgainstAFreshRead(t *testing.T) {

+ b, cs := countingBuildStates(t)

+ ctx := context.Background()

+ if err := put(t, b, ctx, "a.example.com", release.BuildState{LastExecuted: "v1", Completed: 1}); err != nil {

+ t.Fatal(err)

+ }

+ key := "inst/a.example.com/build-state.json"

+ if _, err := cs.Store.Put(ctx, key, []byte(`{"schema":1,"host":"a.example.com","last_executed":"v1","completed":2}`), mustEtag(t, cs.Store, key)); err != nil {

+ t.Fatal(err)

+ }

+ errStale := errors.New("stale")

+ got, err := b.write(ctx, "a.example.com", func(cur *release.BuildState) (release.BuildState, bool, error) {

+ if cur != nil && cur.Completed == 1 {

+ return release.BuildState{}, false, errStale

+ }

+ return *cur, true, nil

+ }, nil)

+ if err != nil {

+ t.Fatalf("write = %v; the refusal came from the cached copy, not the object", err)

+ }

+ if got.Completed != 2 {

+ t.Errorf("state = %+v", got)

+ }

+// An object that no longer parses is never written over (R-BS-CORRUPT),

+// cached copy or not: the conditional write fails and the read refuses.

+func TestBuildStateCachedCopyNeverWritesOverACorruptObject(t *testing.T) {

+ b, cs := countingBuildStates(t)

+ ctx := context.Background()

+ if err := put(t, b, ctx, "a.example.com", release.BuildState{LastExecuted: "v1"}); err != nil {

+ t.Fatal(err)

+ }

+ key := "inst/a.example.com/build-state.json"

+ if _, err := cs.Store.Put(ctx, key, []byte("{"), mustEtag(t, cs.Store, key)); err != nil {

+ t.Fatal(err)

+ }

+ if _, err := b.update(ctx, "a.example.com", bump); err == nil {

+ t.Fatal("wrote over an object that does not parse")

+ }

+ raw, _, err := cs.Store.Get(ctx, key)

+ if err != nil || string(raw) != "{" {

+ t.Fatalf("object = %q, %v", raw, err)

+ }

diff --git a/internal/host/buildstate_test.go b/internal/host/buildstate_test.go

index e4d0aa2ef..0be283fcf 100644

--- a/internal/host/buildstate_test.go

+++ b/internal/host/buildstate_test.go

@@ -599,8 +599,11 @@ func TestAPersistentConflictIsAnError(t *testing.T) {

if _, err := b.update(ctx, "a.example.com", bump); !errors.Is(err, release.ErrCASConflict) {

t.Fatalf("err = %v; want a CAS conflict", err)

}

- if n != updateTries {

- t.Fatalf("tried %d times, want %d", n, updateTries)

+ // The put above left this agent a copy of the object, so the first try

+ // goes on that copy (buildStates.write) and every fresh read still gets

+ // its own: updateTries of them after it.

+ if n != updateTries+1 {

+ t.Fatalf("tried %d times, want %d (the copy's try and %d fresh ones)", n, updateTries+1, updateTries)

}

if after := copyOf(t, reg); after.Column() != before.Column() {

t.Fatalf("a lost update changed the copy: %+v -> %+v", before, after)

diff --git a/internal/lease/createlease.go b/internal/lease/createlease.go

new file mode 100644

index 000000000..71aa2a1d4

--- /dev/null

+++ b/internal/lease/createlease.go

@@ -0,0 +1,78 @@

+package lease

+import (

+ "bytes"

+ "context"

+ "encoding/json"

+ "errors"

+ "fmt"

+ "time"

+ "github.com/aws/aws-sdk-go-v2/aws"

+ awss3 "github.com/aws/aws-sdk-go-v2/service/s3"

+ "github.com/aws/smithy-go"

+ "github.com/benbjohnson/litestream"

+ lss3 "github.com/benbjohnson/litestream/s3"

+// errLockPresent is createLease's answer when a lock object exists, live or

+// expired: litestream's AcquireLease decides it, as it always has.

+var errLockPresent = errors.New("lease: a lock is present")

+// createLease is litestream's AcquireLease for the common case, the lock

+// absent (a release deletes it), in one round trip instead of two. It

+// writes what AcquireLease writes for an absent lock: generation 1, the

+// leaser's owner, now plus its TTL, the same JSON and content type, and the

+// same create-only condition (If-None-Match: *). So the lock it leaves, and

+// what a reader of it decides, are AcquireLease's. A 412 means a lock is

+// there, and errLockPresent hands the decision back to AcquireLease (the

+// takeover of an expired lock, the refusal of a live one). Any other error

+// is the acquire's, as AcquireLease's own write failing would be.

+func createLease(ctx context.Context, client *awss3.Client, l *lss3.Leaser) (*litestream.Lease, error) {

+ lease := &litestream.Lease{Generation: 1, ExpiresAt: time.Now().Add(l.TTL), Owner: l.Owner}

+ data, err := json.Marshal(lease)

+ if err != nil {

+ return nil, fmt.Errorf("marshal lock file: %w", err)

+ }

+ key := lss3.DefaultLeasePath

+ if l.Path != "" {

+ key = l.Path + "/" + lss3.DefaultLeasePath

+ }

+ out, err := client.PutObject(ctx, &awss3.PutObjectInput{

+ Bucket: aws.String(l.Bucket),

+ Key: aws.String(key),

+ Body: bytes.NewReader(data),

+ ContentType: aws.String("application/json"),

+ IfNoneMatch: aws.String("*"),

+ })

+ if err != nil {

+ if preconditionFailed(err) {

+ return nil, errLockPresent

+ }

+ return nil, fmt.Errorf("put lock file: %w", err)

+ }

+ if out.ETag != nil {

+ lease.ETag = *out.ETag

+ }

+ return lease, nil

+// preconditionFailed is litestream's isPreconditionFailed (unexported

+// there), plus the 409 S3 answers when two conditional writes of one key

+// race: either way a lock is, or is about to be, present.

+func preconditionFailed(err error) bool {

+ var apiErr smithy.APIError

+ if errors.As(err, &apiErr) {

+ switch apiErr.ErrorCode() {

+ case "PreconditionFailed", "412", "ConditionalRequestConflict":

+ return true

+ }

+ var opErr *smithy.OperationError

+ if errors.As(err, &opErr) {

+ if h, ok := opErr.Err.(interface{ HTTPStatusCode() int }); ok {

+ return h.HTTPStatusCode() == 412 || h.HTTPStatusCode() == 409

+ }

+ return false

diff --git a/internal/lease/endpoint_test.go b/internal/lease/endpoint_test.go

index dd341a561..294dad169 100644

--- a/internal/lease/endpoint_test.go

+++ b/internal/lease/endpoint_test.go

@@ -132,8 +132,11 @@ func TestAcquire_ReachesEndpointOverride(t *testing.T) {

sawPut = true

}

- if !sawGet || !sawPut {

- t.Fatalf("lease traffic did not reach the override: got %v, want a GET and a PUT of %s", got, wantKey)

+ // An absent lock is one create-only PUT (createLease); a GET comes

+ // only when a lock is present. Either way the PUT must arrive here.

+ _ = sawGet

+ if !sawPut {

+ t.Fatalf("lease traffic did not reach the override: got %v, want a PUT of %s", got, wantKey)

}

// The PUT body is the lock itself — proof the acquire really happened

diff --git a/internal/lease/fastacquire_test.go b/internal/lease/fastacquire_test.go

new file mode 100644

index 000000000..cc7fe5578

--- /dev/null

+++ b/internal/lease/fastacquire_test.go

@@ -0,0 +1,174 @@

+package lease

+import (

+ "context"

+ "encoding/json"

+ "errors"

+ "fmt"

+ "io"

+ "net/http"

+ "net/http/httptest"

+ "strings"

+ "sync"

+ "testing"

+ "time"

+ "github.com/benbjohnson/litestream"

+// casS3 is one key's worth of S3 with conditional writes: If-None-Match: *

+// refuses an existing object, If-Match refuses another etag, with 412.

+type casS3 struct {

+ mu sync.Mutex

+ reqs []string

+ body []byte

+ etag string

+ n int

+ puts []http.Header

+func newCasS3(t *testing.T) (*casS3, string) {

+ t.Helper()

+ f := &casS3{}

+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

+ f.mu.Lock()

+ defer f.mu.Unlock()

+ f.reqs = append(f.reqs, r.Method)

+ fail := func() {

+ w.Header().Set("Content-Type", "application/xml")

+ w.WriteHeader(http.StatusPreconditionFailed)

+ _, _ = w.Write([]byte(`<?xml version="1.0" encoding="UTF-8"?><Error><Code>PreconditionFailed</Code><Message>precondition</Message></Error>`))

+ }

+ switch r.Method {

+ case http.MethodGet:

+ if f.body == nil {

+ w.Header().Set("Content-Type", "application/xml")

+ w.WriteHeader(http.StatusNotFound)

+ _, _ = w.Write([]byte(`<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>no such key</Message></Error>`))

+ return

+ }

+ w.Header().Set("ETag", f.etag)

+ _, _ = w.Write(f.body)

+ case http.MethodPut:

+ f.puts = append(f.puts, r.Header.Clone())

+ if r.Header.Get("If-None-Match") == "*" && f.body != nil {

+ fail()

+ return

+ }

+ if m := r.Header.Get("If-Match"); m != "" && strings.Trim(m, `"`) != strings.Trim(f.etag, `"`) {

+ fail()

+ return

+ }

+ b, _ := io.ReadAll(r.Body)

+ f.n++

+ f.body, f.etag = b, fmt.Sprintf(`"etag-%d"`, f.n)

+ w.Header().Set("ETag", f.etag)

+ case http.MethodDelete:

+ if m := r.Header.Get("If-Match"); m != "" && strings.Trim(m, `"`) != strings.Trim(f.etag, `"`) {

+ fail()

+ return

+ }

+ f.body, f.etag = nil, ""

+ w.WriteHeader(http.StatusNoContent)

+ }

+ }))

+ t.Cleanup(srv.Close)

+ return f, srv.URL

+func (f *casS3) set(t *testing.T, l litestream.Lease) {

+ t.Helper()

+ raw, err := json.Marshal(l)

+ if err != nil {

+ t.Fatal(err)

+ }

+ f.mu.Lock()

+ defer f.mu.Unlock()

+ f.n++

+ f.body, f.etag = raw, fmt.Sprintf(`"etag-%d"`, f.n)

+func (f *casS3) methods() []string {

+ f.mu.Lock()

+ defer f.mu.Unlock()

+ return append([]string(nil), f.reqs...)

+func (f *casS3) lease(t *testing.T) litestream.Lease {

+ t.Helper()

+ f.mu.Lock()

+ defer f.mu.Unlock()

+ var l litestream.Lease

+ if err := json.Unmarshal(f.body, &l); err != nil {

+ t.Fatalf("lock %q: %v", f.body, err)

+ }

+ return l

+func casEnv(t *testing.T) (*casS3, context.Context) {

+ t.Helper()

+ f, url := newCasS3(t)

+ fakeCreds(t)

+ t.Setenv("CARLOS_S3_ENDPOINT", url)

+ return f, context.Background()

+// A released lease deletes its lock, so the next wake finds none: it

+// writes the lease create-only in one round trip, the same lease

+// litestream writes for an absent lock (generation 1, this owner).

+func TestAcquireAnAbsentLockIsOneCreateOnlyWrite(t *testing.T) {

+ f, ctx := casEnv(t)

+ held, err := Acquire(ctx, "b", "inst/x.example.com", "eu-west-1")

+ if err != nil {

+ t.Fatal(err)

+ }

+ if got := f.methods(); len(got) != 1 || got[0] != http.MethodPut {

+ t.Fatalf("requests = %v, want one PUT", got)

+ }

+ if h := f.puts[0]; h.Get("If-None-Match") != "*" || h.Get("If-Match") != "" {

+ t.Fatalf("the write was not create-only: %v", h)

+ }

+ l := f.lease(t)

+ if l.Generation != 1 || l.Owner == "" || time.Until(l.ExpiresAt) < LeaseTTL-time.Minute {

+ t.Fatalf("lease = %+v, want generation 1, an owner and a full TTL", l)

+ }

+ // The handle carries the etag its write returned: release is

+ // conditional on it, so it deletes this lock and no other.

+ if err := held.Release(ctx); err != nil {

+ t.Fatal(err)

+ }

+ if got := f.methods(); got[len(got)-1] != http.MethodDelete {

+ t.Fatalf("requests = %v, want the release's DELETE", got)

+ }

+// An expired lock is taken over as litestream takes it over: read, then a

+// conditional write of the next generation.

+func TestAcquireAnExpiredLockFallsBackToTheTakeover(t *testing.T) {

+ f, ctx := casEnv(t)

+ f.set(t, litestream.Lease{Generation: 4, ExpiresAt: time.Now().Add(-time.Minute), Owner: "gone:1"})

+ held, err := Acquire(ctx, "b", "inst/x.example.com", "eu-west-1")

+ if err != nil {

+ t.Fatal(err)

+ }

+ defer held.Release(ctx)

+ if l := f.lease(t); l.Generation != 5 || l.Owner == "gone:1" {

+ t.Fatalf("lease = %+v, want generation 5 under this owner", l)

+ }

+// A live lock is someone else's: refused with its holder, as before, and

+// left exactly as it was.

+func TestAcquireALiveLockIsRefused(t *testing.T) {

+ f, ctx := casEnv(t)

+ f.set(t, litestream.Lease{Generation: 2, ExpiresAt: time.Now().Add(10 * time.Minute), Owner: "other:7"})

+ before := f.lease(t)

+ _, err := Acquire(ctx, "b", "inst/x.example.com", "eu-west-1")

+ var held *HeldError

+ if !errors.As(err, &held) || held.Owner != "other:7" {

+ t.Fatalf("Acquire = %v, want a HeldError naming other:7", err)

+ }

+ if after := f.lease(t); after != before {

+ t.Fatalf("lock changed: %+v -> %+v", before, after)

+ }

diff --git a/internal/lease/lease.go b/internal/lease/lease.go

index 11c14c5de..b926b7e78 100644

--- a/internal/lease/lease.go

+++ b/internal/lease/lease.go

@@ -190,7 +190,10 @@ func AcquireTTL(ctx context.Context, bucket, prefix, region string, ttl time.Dur

leaser.TTL = ttl

started := time.Now()

- lease, err := leaser.AcquireLease(ctx)

+ lease, err := createLease(ctx, client, leaser)

+ if errors.Is(err, errLockPresent) {

+ lease, err = leaser.AcquireLease(ctx)

+ }

if err != nil {

// A *litestream.LeaseExistsError means someone else holds it —

// re-surfaced as *HeldError so callers stay litestream-free.

diff --git a/internal/provision/plan9_b8_census_test.go b/internal/provision/plan9_b8_census_test.go

index a16efb787..7a4016832 100644

--- a/internal/provision/plan9_b8_census_test.go

+++ b/internal/provision/plan9_b8_census_test.go

@@ -366,7 +366,7 @@ func TestPlan9B8RecordWriteCensus(t *testing.T) {

if len(got) < 100 {

t.Fatalf("census found only %d units under %s; the walk is broken", len(got), root)

}

- inventory := serviceRunsWriteCensusInventory()

+ inventory := wakeLatency3WriteCensusInventory()

problems := append(censusDiff(inventory, got), censusPolicy(inventory)...)

if len(problems) > 0 {

t.Fatalf("record-write census rejected %d site(s); update the explicit inventory only after reviewing the source:\n %s", len(problems), strings.Join(problems, "\n "))

diff --git a/internal/provision/testdata/plan14-root-reads-chain.tsv b/internal/provision/testdata/plan14-root-reads-chain.tsv

index af74d718e..671a7bc0d 100644

--- a/internal/provision/testdata/plan14-root-reads-chain.tsv

+++ b/internal/provision/testdata/plan14-root-reads-chain.tsv

@@ -9,3 +9,4 @@ plan14-root-reads-sec-mail-user-boundary.tsv 68ca9635da386f34ef483c790af19e95412

plan14-root-reads-edge-asset-cache-2.tsv de15b849fcb93b76caca55be924cde41512f8c470f40b4ebdd77ec65ad7e7069

plan14-root-reads-pro-6214-service-runs-decisions.tsv ec2d8daaec385b34b3be9c8c6430427c082ac36cd40150e1c6f520e20e3da72d

plan14-root-reads-pro-6214-service-runs.tsv 86570d1326baad6d866b4066b6bf65218434b201fb1f2c05ebc6e3c79f9c5a98

+plan14-root-reads-wake-latency-3.tsv 86570d1326baad6d866b4066b6bf65218434b201fb1f2c05ebc6e3c79f9c5a98

diff --git a/internal/provision/testdata/plan14-root-reads-wake-latency-3.tsv b/internal/provision/testdata/plan14-root-reads-wake-latency-3.tsv

new file mode 100644

index 000000000..c9a8aa3f9

--- /dev/null

+++ b/internal/provision/testdata/plan14-root-reads-wake-latency-3.tsv

@@ -0,0 +1,6 @@

+# Census overlay wake-latency-3, written by census-propose (#369). census-ruling-wake-latency-3, approved by the census owner before the merge; the approval card is docs/superpowers/notes/census-proposal-wake-latency-3.md.

+# 1 body, 0 sites, 3 envelope: each changed unit re-frozen at its existing disposition with its claim appended.

+internal/host/buildstate.go buildStates.write authority E3,J1,J4 root-input;rev3=census-ruling-wake-latency-3;basis=its first try goes on the copy this agent last wrote at that key, conditioned on that copy's etag (one round trip); a conflict, an error, or a merge that refuses or has nothing to write on the copy is decided on a fresh read as before; no read added (the fresh read is the one it always made) file:ReadAt@2527 0c6620f6c436457c18142203cee157e4d2104c79304d1cfd1f159bd4fea40797

+@coverage internal/appconfig d5a8d65a8dce2372d40a0c32c73e57e677adba6a39a2b7f41fdc6bc4a86ad452

+@coverage internal/host 95ca64c575909995d8a9dd5b5a2f437f96b36017a804e2bf0b8f4b221e2ec660

+@coverage internal/lease 33f2ca8b19741ca152122dbf93f561d7b70f534206ea3cd7ee3c30123c1c27e5

diff --git a/internal/provision/wake_latency_3_census_test.go b/internal/provision/wake_latency_3_census_test.go

new file mode 100644

index 000000000..381314290

--- /dev/null

+++ b/internal/provision/wake_latency_3_census_test.go

@@ -0,0 +1,28 @@

+package provision

+// Record-write census entries for wake-latency-3, layered on

+// serviceRunsWriteCensusInventory. Reviewed by the branch author; the census

+// owner approves the card (census-ruling-wake-latency-3,

+// docs/superpowers/notes/census-proposal-wake-latency-3.md).

+// - buildStates.write's first try goes on the copy this agent last wrote,

+// conditioned on its etag. Its one Put, the build-state object, is the

+// same Put with the same condition rule (the etag the merge's state came

+// with); its class is unchanged: nonrecord.

+func wakeLatency3WriteCensusInventory() []censusEntry {

+ entries := append([]censusEntry(nil), serviceRunsWriteCensusInventory()...)

+ replacements := []censusEntry{

+ {"internal/host/buildstate.go", "buildStates.write", "nonrecord", "sel:Put", "0c6620f6c436457c18142203cee157e4d2104c79304d1cfd1f159bd4fea40797"},

+ }

+ for _, r := range replacements {

+ found := false

+ for i, e := range entries {

+ if e.File == r.File && e.Unit == r.Unit {

+ entries[i], found = r, true

+ }

+ if !found {

+ entries = append(entries, r)

+ }

+ return entries