amadan
CI failed 2m 25s
- Branch
design-edge-request-protection- Commit
e044dff1claude
- Runner
ci-aws~paul@keymail.dev/ip-172-31-31-147- Reported by
ci-aws~paul@keymail.dev/ip-172-31-31-147- Queued
- Started
- Waited for a runner
- 10s
- Last change
- Attempt
- 1 of 2
Steps run in order, one after another.
-
gate failed 2m 24s
.amadan/test-gate.sh === launchctx (0 s) ok amadan.net/carlos/platform.git/launchctx 0.017s ? amadan.net/carlos/platform.git/launchctx/controlname [no test files] ? amadan.net/carlos/platform.git/launchctx/wireid [no test files] === vet (1 s) === gofmt (0 s) gofmt -l .: clean === rest (143 s) ? github.com/carlosframework/platform/cmd/changelog [no test files] ok github.com/carlosframework/platform/cmd/console 3.671s ok github.com/carlosframework/platform/cmd/moveprobe 11.985s ok github.com/carlosframework/platform/cmd/zddprobe 0.746s 2026/10/24 21:30:48 activator: hook.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: nohook.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: ladder0.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: ladder1.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: ladder2.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: ladder3.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: fresh.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: restored.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: noseed.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: badseed.example.com: seed for badseed.example.com: activator: seed unproven; processes remain: processes remain after a successful seed: still running; leaving the lease to its TTL 2026/10/24 21:30:48 activator: unproven.example.com: unproven.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone; leaving the lease to its TTL 2026/10/24 21:30:48 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh 2026/10/24 21:30:48 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3396839126/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3396839126/004/next-wake.example.com.db 2026/10/24 21:30:48 activator: next-wake.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3396839126/004/next-wake.example.com.db.seed-unproven refuses it as warm until they are 2026/10/24 21:30:48 activator: next-wake.example.com: seeding next-wake.example.com: exit status 1: activator: seed unproven; processes remain (the seed's processes remain after stopping it; its database was left in place: cgroup: strict containment unsupported); leaving the lease to its TTL 2026/10/24 21:30:48 activator: next-wake.example.com: next-wake.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup: strict containment unsupported; leaving the lease to its TTL 2026/10/24 21:30:48 activator: next-wake.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again 2026/10/24 21:30:48 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh 2026/10/24 21:30:48 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3396839126/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3396839126/004/next-wake.example.com.db 2026/10/24 21:30:48 activator: seeded next-wake.example.com 2026/10/24 21:30:48 activator: next-wake.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: instance one.example.com exited unexpectedly — recycling 2026/10/24 21:30:48 activator: one.example.com hibernated 2026/10/24 21:30:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: LEASE HELD for held.example.com by "old-incarnation" — a previous incarnation hasn't released; refusing to wake (retry after 7s) 2026/10/24 21:30:48 activator: hib.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: tick.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: visitor.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: asleep.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:48 activator: fenced.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:49 activator: imminent.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:49 activator: overdue.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:49 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:49 activator: two.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:49 activator: two.example.com hibernated 2026/10/24 21:30:49 activator: one.example.com hibernated 2026/10/24 21:30:49 activator: b.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:49 activator: b.example.com hibernated 2026/10/24 21:30:49 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:30:49 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:30:49 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:30:49 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:30:49 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:30:49 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:30:49 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:30:49 activator: worker bot.x.internal respawned (exited), now running 2026/10/24 21:30:49 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:30:49 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:30:49 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it 2026/10/24 21:30:49 activator: bot.x.internal hibernated 2026/10/24 21:30:50 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendStartMergesConfigEnvUnderOwnEnv4120487044/002/merge-precedence.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted 2026/10/24 21:30:50 activator: purged 1 cached response(s) for deploy.example.com on spawn 2026/10/24 21:30:50 activator: restored restored.example.com from s3://test-bucket/restored.example.com 2026/10/24 21:30:50 activator: damaged.example.com restored CORRUPT from s3://test-bucket/damaged.example.com: integrity_check: file is not a database (26) — quarantined at /tmp/TestExecBackendRestoreCorruptReplicaFailsAndQuarantines3017527669/002/damaged.example.com.db.corrupt, instance NOT started 2026/10/24 21:30:50 activator: restored damaged.example.com from s3://test-bucket/damaged.example.com 2026/10/24 21:30:50 activator: no replica for brand-new.example.com in s3://test-bucket/brand-new.example.com — booting fresh 2026/10/24 21:30:50 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1385449360/002/seeded.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted 2026/10/24 21:30:50 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1385449360/001/live/seeded.example.com seed --fixtures --db /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1385449360/002/seeded.example.com.db 2026/10/24 21:30:50 activator: seeded seeded.example.com 2026/10/24 21:30:50 activator: seeding custom-db.example.com: /tmp/TestExecBackendSeedHonoursRouteDBPath2083819304/001/live/custom-db.example.com seed --db /tmp/TestExecBackendSeedHonoursRouteDBPath2083819304/004/elsewhere.db 2026/10/24 21:30:50 activator: seeded custom-db.example.com 2026/10/24 21:30:50 activator: seeding badseed.example.com: /tmp/TestExecBackendSeedNonZeroExitFails723266046/001/live/badseed.example.com seed --db /tmp/TestExecBackendSeedNonZeroExitFails723266046/002/badseed.example.com.db schema is corrupt 2026/10/24 21:30:50 activator: badseed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is 2026/10/24 21:30:50 activator: seeding hangs.example.com: /tmp/TestExecBackendSeedIsTimeBounded1721172399/001/live/hangs.example.com seed --db /tmp/TestExecBackendSeedIsTimeBounded1721172399/002/hangs.example.com.db 2026/10/24 21:30:50 activator: hangs.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is 2026/10/24 21:30:50 activator: seeding crashed.example.com: /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB2777798474/001/live/crashed.example.com seed --db /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB2777798474/002/crashed.example.com.db died mid-schema 2026/10/24 21:30:50 activator: crashed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is 2026/10/24 21:30:50 activator: seeding existing-db.example.com: /tmp/TestExecBackendSeedFailureKeepsPreexistingDB4271391541/001/live/existing-db.example.com seed --db /tmp/TestExecBackendSeedFailureKeepsPreexistingDB4271391541/002/existing-db.example.com.db nope 2026/10/24 21:30:50 activator: existing-db.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is 2026/10/24 21:30:50 activator: seeding leaky-seed.example.com: /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild3176716889/001/live/leaky-seed.example.com seed --db /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild3176716889/002/leaky-seed.example.com.db 2026/10/24 21:30:50 activator: seeded leaky-seed.example.com (it exited 0 but left a child holding its output pipe open for more than 200ms) 2026/10/24 21:30:51 activator: a.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:30:51 activator: b.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:51 activator: sidecar for a.example.com was down — restarted (backoff now 2s) 2026/10/24 21:30:51 activator: h.example.com: route refreshed: db "/tmp/h.example.com.db" -> "/data/instances/h.example.com/h.example.com.db" socket "/tmp/nope.sock" -> "/run/carlos/instances/h.example.com/app.sock" 2026/10/24 21:30:51 activator: h.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:51 activator: stale.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:30:51 activator: stop stale.example.com: a sidecar start is still in flight; keeping the lease until it returns 2026/10/24 21:30:51 activator: stale.example.com hibernated 2026/10/24 21:30:51 activator: stale.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:30:51 activator: leftover.example.com: the host cgroup holds processes not proven gone: a leftover still populates the host cgroup; leaving the lease to its TTL 2026/10/24 21:30:51 activator: leftover.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:51 activator: old-kernel.example.com: this kernel has no cgroup.kill, so the host cgroup cannot be killed; its leader's exit is the only proof there is 2026/10/24 21:30:51 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh 2026/10/24 21:30:51 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing) 2026/10/24 21:30:51 activator: restored moved.example.com from s3://test-bucket/moved.example.com 2026/10/24 21:30:51 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing) 2026/10/24 21:30:51 activator: restored moved.example.com from s3://test-bucket/moved.example.com 2026/10/24 21:30:51 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh 2026/10/24 21:30:51 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing) 2026/10/24 21:30:51 activator: restored moved.example.com from s3://test-bucket/moved.example.com 2026/10/24 21:30:51 activator: retry.example.com: strict group did not drain: cgroup: strict containment requires a live deadline 2026/10/24 21:30:52 activator: retry.example.com: strict group cleaned up on retry 1 2026/10/24 21:30:52 activator: retry.example.com: strict group not closed: cgroup: strict containment refused 2026/10/24 21:30:52 activator: retry.example.com: strict group cleaned up on retry 1 2026/10/24 21:30:52 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline 2026/10/24 21:30:52 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline 2026/10/24 21:30:52 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline 2026/10/24 21:30:52 activator: stuck.example.com: strict group still not cleaned up after 2 retries; spawns of this host stay refused until the agent restarts 2026/10/24 21:30:52 activator: owned.example.com: strict group did not drain: cgroup: strict containment requires a live deadline 2026/10/24 21:30:53 activator: owned.example.com: strict group cleaned up on retry 1 2026/10/24 21:30:53 activator: owned.example.com: strict group not closed: cgroup: strict containment refused 2026/10/24 21:30:53 activator: owned.example.com: strict group cleaned up on retry 1 2026/10/24 21:30:54 activator: race.example.com: strict group did not drain: cgroup: strict containment requires a live deadline 2026/10/24 21:30:54 activator: transition.example.com: strict group did not drain: cgroup: strict containment requires a live deadline 2026/10/24 21:30:54 activator: transition2.example.com: strict group did not drain: cgroup: strict containment requires a live deadline 2026/10/24 21:30:54 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh 2026/10/24 21:30:54 activator: restored moved.example.com from s3://test-bucket/moved.example.com 2026/10/24 21:30:54 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:54 activator: kass.example.com hibernated 2026/10/24 21:30:54 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:54 activator: kass.example.com idle 11s — hibernating 2026/10/24 21:30:54 activator: kass.example.com hibernated 2026/10/24 21:30:54 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:54 activator: one.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build 2026/10/24 21:30:54 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:30:54 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:30:54 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:30:54 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:30:55 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:30:55 activator: bot.x.internal: v2 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:30:55 activator: worker bot.x.internal NOT respawned (attempt (release)): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state 2026/10/24 21:30:55 activator: worker bot.x.internal respawned (restoring v1), now running v1 2026/10/24 21:30:55 activator: link.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:55 activator: link.example.com hibernated 2026/10/24 21:30:55 activator: link.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:55 activator: plain-link.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:55 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:55 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:59 activator: tenant.test: v1 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:30:59 activator: tenant.test not started: the executable it hashed (/tmp/rl4078734386/app) is build "v1", but the start names build "v2" (the live link moved between the two reads); the next wake reads it again 2026/10/24 21:30:59 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:30:59 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later 2026/10/24 21:30:59 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a 2026/10/24 21:30:59 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:59 activator: replica h.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own 2026/10/24 21:30:59 activator: replica h.example.com not started: CARLOS_INSTANCE_KEY is not its home's key 2026/10/24 21:30:59 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a 2026/10/24 21:30:59 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:59 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:59 activator: why.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:59 activator: why.example.com hibernated 2026/10/24 21:30:59 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:59 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:59 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:59 activator: c.example.com hibernated 2026/10/24 21:30:59 activator: h.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:59 activator: h.example.com hibernated 2026/10/24 21:30:59 activator: h.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:30:59 activator: h.example.com: run run_a is not proven stopped: cgroup.events: permission denied 2026/10/24 21:30:59 activator: stop h.example.com: a run is not proven stopped; keeping replication and the lease until it is 2026/10/24 21:30:59 activator: h.example.com hibernated 2026/10/24 21:30:59 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a 2026/10/24 21:30:59 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a 2026/10/24 21:30:59 activator: seeding unproven-seed.example.com: /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga2534324464/002/live/unproven-seed.example.com seed --db /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga2534324464/003/unproven-seed.example.com.db 2026/10/24 21:30:59 activator: unproven-seed.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga2534324464/003/unproven-seed.example.com.db.seed-unproven refuses it as warm until they are 2026/10/24 21:30:59 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm107864036/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm107864036/003/obligation.example.com.db 2026/10/24 21:30:59 activator: obligation.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm107864036/003/obligation.example.com.db.seed-unproven refuses it as warm until they are 2026/10/24 21:30:59 activator: obligation.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again 2026/10/24 21:30:59 activator: no replica for obligation.example.com in s3://test-bucket/obligation.example.com — booting fresh 2026/10/24 21:30:59 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm107864036/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm107864036/003/obligation.example.com.db 2026/10/24 21:30:59 activator: seeded obligation.example.com 2026/10/24 21:30:59 activator: seeding helper-seed.example.com: /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt3265900727/002/live/helper-seed.example.com seed --db /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt3265900727/003/helper-seed.example.com.db 2026/10/24 21:30:59 activator: seeded helper-seed.example.com 2026/10/24 21:30:59 activator: seeding helper-unproven.example.com: /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven2271479888/002/live/helper-unproven.example.com seed --db /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven2271479888/003/helper-unproven.example.com.db 2026/10/24 21:30:59 activator: helper-unproven.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven2271479888/003/helper-unproven.example.com.db.seed-unproven refuses it as warm until they are 2026/10/24 21:30:59 activator: seeding write-ahead.example.com: /tmp/TestASeedRecordsItsObligationBeforeItRuns811764108/002/live/write-ahead.example.com seed --db /tmp/TestASeedRecordsItsObligationBeforeItRuns811764108/003/write-ahead.example.com.db 2026/10/24 21:30:59 activator: seeded write-ahead.example.com 2026/10/24 21:30:59 activator: write-ahead.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again 2026/10/24 21:30:59 activator: no replica for write-ahead.example.com in s3://test-bucket/write-ahead.example.com — booting fresh 2026/10/24 21:30:59 activator: seeding no-room.example.com: /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing2815485007/002/live/no-room.example.com seed --db /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing2815485007/003/no-room.example.com.db 2026/10/24 21:30:59 activator: seeding stuck-wal.example.com: /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation3119787773/002/live/stuck-wal.example.com seed --db /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation3119787773/003/stuck-wal.example.com.db 2026/10/24 21:30:59 activator: seeding preexisting.example.com: /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2242273890/002/live/preexisting.example.com seed --db /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2242273890/003/preexisting.example.com.db 2026/10/24 21:30:59 activator: preexisting.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2242273890/003/preexisting.example.com.db.seed-unproven refuses it as warm until they are 2026/10/24 21:30:59 activator: preexisting.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept 2026/10/24 21:30:59 activator: old-kernel-seed.example.com: this kernel has no cgroup.kill; the seed's leader's exit is the only proof there is 2026/10/24 21:30:59 activator: seeding failed-role-seed.example.com: /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB3339933585/002/live/failed-role-seed.example.com seed --db /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB3339933585/003/failed-role-seed.example.com.db 2026/10/24 21:30:59 activator: seeding discharge-fails.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai2120713116/002/live/discharge-fails.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai2120713116/003/discharge-fails.example.com.db 2026/10/24 21:30:59 activator: discharge-fails.example.com: the seed's obligation could not be discharged and its processes are not proven gone; /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai2120713116/003/discharge-fails.example.com.db.seed-unproven stays 2026/10/24 21:30:59 activator: seeding discharge-fails-proven.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur942013482/002/live/discharge-fails-proven.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur942013482/003/discharge-fails-proven.example.com.db 2026/10/24 21:30:59 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:00 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:00 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_B,_l3474248913/003/g/56/launch.example.com.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall932405319/003/g/57/launch.example.com.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall932405319/003/g/58/launch.example.com.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:00 activator: swap.example.com hibernated 2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatholds_pinne1107576778/007/g/60/launch.example.com.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:00 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed 2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatoverlap_succ3444221334/007/g/62/launch.example.com.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:00 activator: h.example.com: started carlos-instance-gen@h.example.com:64.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:00 activator: swap.example.com hibernated 2026/10/24 21:31:00 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed 2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_super1523654838/007/g/71/launch.example.com.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:00 activator: swap.example.com hibernated 2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartholds_pinned_build,_lea636946253/007/g/73/launch.example.com.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:00 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed 2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_a_fai1239971314/007/g/75/launch.example.com.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:00 activator: fence.example hibernated 2026/10/24 21:31:00 activator: fence.example hibernated 2026/10/24 21:31:00 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink496994850/003/g/c.x.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink496994850/003/g/c.x.sock: no such file or directory) 2026/10/24 21:31:00 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink496994850/003/g/c.xx.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink496994850/003/g/c.xx.sock: no such file or directory) 2026/10/24 21:31:00 activator: publish generations to /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile2542515473/002/not-a-dir/zdd/generations.json: open directory /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile2542515473/002/not-a-dir/zdd: not a directory; the sweep retries 2026/10/24 21:31:00 activator: cur.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:00 activator: off.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:00 activator: plain.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:01 activator: ident.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:01 activator: ident.example.com hibernated 2026/10/24 21:31:01 activator: ident.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:01 activator: instance ident.example.com exited unexpectedly — recycling 2026/10/24 21:31:01 activator: ident.example.com hibernated 2026/10/24 21:31:01 activator: ident.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:01 activator: due.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:01 activator: due.example.com hibernated 2026/10/24 21:31:01 activator: due.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:01 activator: due.example.com hibernated 2026/10/24 21:31:01 activator: ret.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:01 activator: ret.example.com hibernated 2026/10/24 21:31:01 activator: ret.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:01 activator: ret.example.com hibernated 2026/10/24 21:31:01 activator: bot.x.internal: woke in 21ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 21ms, replicate 0s 2026/10/24 21:31:01 activator: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.example.com: a generation socket path would exceed 107 bytes; generation 9 keeps the canonical socket 2026/10/24 21:31:01 activator: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.example.com: a generation socket path would exceed 107 bytes; generation 9 keeps the canonical socket 2026/10/24 21:31:01 activator: one.example.com: generation 1089 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:01 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:01 activator: one.example.com hibernated 2026/10/24 21:31:01 activator: purged 1 cached response(s) for flat.example.com on spawn 2026/10/24 21:31:01 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile2191065555/001/versions/vA/app seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile2191065555/002/seedexe.example.com.db 2026/10/24 21:31:01 activator: seeded seedexe.example.com 2026/10/24 21:31:01 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile2191065555/001/live/seedexe.example.com seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile2191065555/002/seedexe.example.com.db 2026/10/24 21:31:01 activator: seeded seedexe.example.com 2026/10/24 21:31:01 activator: seeding h.example.com: /tmp/TestNestedSeedExeNamesTheRecordedFile3853841281/001/live/h.example.com seed --db /tmp/nb3164354641/data/instances/h.example.com/h.example.com.db 2026/10/24 21:31:01 activator: seeded h.example.com 2026/10/24 21:31:01 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing720671539/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build 2026/10/24 21:31:01 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing720671539/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build 2026/10/24 21:31:01 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing720671539/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build 2026/10/24 21:31:01 activator: seeding h.example.com: /tmp/TestADeclinedNestedSeedRunsThePinnedFile4182352202/001/live/h.example.com seed --db /tmp/nb3516696583/data/instances/h.example.com/h.example.com.db 2026/10/24 21:31:01 activator: seeded h.example.com 2026/10/24 21:31:01 activator: seeding tier.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed140444053/002/versions/vA/app seed --db /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed140444053/003/tier.example.com.db 2026/10/24 21:31:01 activator: tier.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed140444053/003/tier.example.com.db.seed-unproven refuses it as warm until they are 2026/10/24 21:31:01 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=1073741824 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:01 activator: h.example.com: carlos-instance-gen@h.example.com:6.service launched in 0s, healthy 0s later 2026/10/24 21:31:01 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=1073741824 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:01 activator: h.example.com: carlos-instance-gen@h.example.com:6.service launched in 0s, healthy 0s later 2026/10/24 21:31:01 activator: seeding h.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTiernested_seed1602490405/001/live/h.example.com seed --db /tmp/nb4289507177/data/instances/h.example.com/h.example.com.db 2026/10/24 21:31:01 activator: seeded h.example.com 2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsthe_configured_k2961139940/001/shop.example.com.env: the platform mints none while it is 2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsan_own_key_pres3096036078/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it 2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestFirstStartWaitsForTheConfiguredKey3119809557/002/shop.example.com.env: the platform mints none while it is 2026/10/24 21:31:01 activator: control listener for shop.example.com: listen unix /tmp/TestFirstStartWaitsForTheConfiguredKey3119809557/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY in /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptinto_the_own_file1158803400/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it 2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptdelivered_in_the_co2679738147/001/shop.example.com.env: the platform mints none while it is 2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration3660167693/001/shop.example.com.env: the platform mints none while it is 2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration3660167693/002/shop.example.com.env: the platform mints none while it is 2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyLeavesAFileWithTwoKeysAlone131738540/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it 2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistse119992415/002/shop.example.com.env: the platform mints none while it is 2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistsn787051167/002/shop.example.com.env: the platform mints none while it is 2026/10/24 21:31:02 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance3368214908/001/live/seeded.example.com seed --db /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance3368214908/002/seeded.example.com.db 2026/10/24 21:31:02 activator: seeded seeded.example.com 2026/10/24 21:31:02 activator: restored sleepy.example.com from s3://test-bucket/sleepy.example.com hello world 2026/10/24 21:31:02 activator: fence.example hibernated 2026/10/24 21:31:02 activator: fence.example hibernated 2026/10/24 21:31:02 activator: fence.example hibernated 2026/10/24 21:31:02 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:02 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set 2026/10/24 21:31:02 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is 2026/10/24 21:31:02 activator: barrier.example.com hibernated 2026/10/24 21:31:02 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:02 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:02 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process 2026/10/24 21:31:02 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set 2026/10/24 21:31:02 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is 2026/10/24 21:31:02 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process 2026/10/24 21:31:02 activator: barrier.example.com hibernated 2026/10/24 21:31:03 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:03 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:03 activator: barrier.example.com: generation 12's child is not proven empty: g12 still holds a process 2026/10/24 21:31:03 activator: generation 12 of barrier.example.com did not prove its exit; keeping it in the operation set 2026/10/24 21:31:03 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:03 activator: stop barrier.example.com: cgroup.kill: the child is not empty 2026/10/24 21:31:03 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is 2026/10/24 21:31:03 activator: stop barrier.example.com: cgroup.kill: the child is not empty 2026/10/24 21:31:03 activator: barrier.example.com hibernated 2026/10/24 21:31:03 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:03 activator: stop generation 11: cgroup.kill: the child is not empty 2026/10/24 21:31:03 activator: generation 11 of barrier.example.com did not prove its exit; keeping it in the operation set 2026/10/24 21:31:03 activator: placed.example.com: kill generation 1097 after a failed health wait: g still holds a process 2026/10/24 21:31:03 activator: placed.example.com: generation 1097's child is not proven empty: g still holds a process 2026/10/24 21:31:03 activator: placed.example.com: failed wake's generation 1097 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:03 activator: placed.example.com: wake failed and its generation is not proven stopped: start placed.example.com: instance not healthy within 300ms: Get "http://carlos/": dial unix /tmp/g235445025/g/1097/placed.example.com.sock: connect: no such file or directory; activator: generation start failed and its termination is unproven: kill generation 1097 after a failed health wait: g still holds a process 2026/10/24 21:31:03 activator: placed.example.com: failed wake's generation 1097 is proven stopped; lease given back 2026/10/24 21:31:04 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:04 activator: unproven.example.com: failed wake's generation 1100 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:04 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out 2026/10/24 21:31:04 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:04 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:04 activator: unproven.example.com: failed wake's generation 1100 is proven stopped; lease given back 2026/10/24 21:31:04 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:04 activator: unproven.example.com: failed wake's generation 1101 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:04 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out 2026/10/24 21:31:04 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:04 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:04 activator: unproven.example.com: failed wake's generation 1101 is proven stopped; lease given back 2026/10/24 21:31:04 activator: unproven.example.com: generation 1102's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:04 activator: unproven.example.com: failed wake's generation 1102 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:04 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out 2026/10/24 21:31:05 activator: unhealthy-f6.example.com: failed wake's generation 1103 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process 2026/10/24 21:31:05 activator: unhealthy-f6.example.com: failed wake's generation 1103 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:05 activator: unhealthy-f6.example.com: failed wake's generation 1103 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process 2026/10/24 21:31:05 activator: unhealthy-f6.example.com: failed wake's generation 1103 is proven stopped; lease given back 2026/10/24 21:31:05 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is 2026/10/24 21:31:06 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is 2026/10/24 21:31:06 activator: one.example.com: generation 1105 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:06 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:06 activator: instance one.example.com exited unexpectedly — recycling 2026/10/24 21:31:06 activator: one.example.com hibernated 2026/10/24 21:31:06 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:06 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:06 activator: bot.x.internal hibernated 2026/10/24 21:31:06 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:06 activator: worker bot.x.internal respawned (attempt (restart)), now running v2 2026/10/24 21:31:06 activator: bot.x.internal hibernated 2026/10/24 21:31:06 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:06 activator: worker bot.x.internal respawned (attempt (config)), now running v2 2026/10/24 21:31:06 activator: bot.x.internal hibernated 2026/10/24 21:31:06 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:06 activator: bot.x.internal hibernated 2026/10/24 21:31:07 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:07 activator: bot.x.internal hibernated 2026/10/24 21:31:07 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:07 activator: bot.x.internal hibernated 2026/10/24 21:31:07 activator: moved.example.com: promoting 2 staged object(s) from moves/moved.example.com/ (residence move landing) 2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:07 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:07 activator: x.example hibernated 2026/10/24 21:31:07 activator: x.example recovery probe healthy, but its fence changed; stopping it 2026/10/24 21:31:07 activator: x.example hibernated 2026/10/24 21:31:07 activator: x.example recovery probe healthy, but its fence changed; stopping it 2026/10/24 21:31:07 activator: x.example hibernated 2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:07 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:07 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it 2026/10/24 21:31:07 activator: x.example hibernated 2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:07 activator: x.example hibernated 2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:07 activator: x.example hibernated 2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:07 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:07 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:07 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:07 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:07 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY is not its home's key 2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_keyless_replica_is_refu3416413544/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:07 activator: replica shop-au.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own 2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_is_3056958043/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_pre-existing_env_key_th3118349134/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_dif260765861/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_with_no_2293327036/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:07 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY cannot be checked against its home's key (no entry for it in /etc/carlos/replica-home-keys; O8) 2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_home_updated,_this_re2090866327/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:07 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY is not the one the bucket's configuration now holds: a key change is in flight (ruling 27) 2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_bucket_cannot_be_read3779415383/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:07 activator: replica shop-au.example.com not started: the bucket's configuration could not be read: bucket unreachable 2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketno_fresh_reader_refused3496770306/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:07 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY cannot be checked against the bucket's configuration 2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketresidual_a_change_only_o3632784341/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_config-only_homes_keyl2988459519/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_home_row_mints_and_its_3751163128/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:07 activator: seeding shop-au.example.com: /tmp/TestReplicaSeedIsGatedthe_homes_key691337841/001/live/shop-au.example.com seed --db /tmp/TestReplicaSeedIsGatedthe_homes_key691337841/002/shop-au.example.com.db 2026/10/24 21:31:07 activator: seeded shop-au.example.com 2026/10/24 21:31:07 activator: replica shop-au.example.com not started: where its CARLOS_INSTANCE_KEY came from is unknown 2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaGateDecidesOnTheLaunchesOwnRead2022993743/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:07 provision: a.shop.example.com: swapped for import sd_jbbbgrfjpmfzyrn7gw7zlbkkpi (the original route is journaled) 2026/10/24 21:31:07 provision: a.shop.example.com: import sd_jbbbgrfjpmfzyrn7gw7zlbkkpi rolled back to its original route 2026/10/24 21:31:07 provision: a.shop.example.com: import sd_jbbbgrfjpmfzyrn7gw7zlbkkpi is cleaned up and its journal row retired 2026/10/24 21:31:07 activator: h.example.com: placed seed sd_nhimk5l5np5rlhmoc7a6zaf6em (8192 bytes) 2026/10/24 21:31:07 activator: h.example.com: placed seed sd_c75qt54d3bxvxncbfyeaxfnifm (8192 bytes) 2026/10/24 21:31:07 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/nb1894519496/data/instances/h.example.com/h.example.com.env: the platform mints none while it is 2026/10/24 21:31:07 activator: h.example.com: placed seed sd_4vlamgbfrtwbqdxl53h6wxgu6q (8192 bytes) 2026/10/24 21:31:07 activator: CARLOS_INSTANCE_KEY in /tmp/nb4227882448/data/instances/h.example.com/h.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it 2026/10/24 21:31:07 activator: h.example.com: placed seed sd_xnm6duxahoayddv2y4h774eykq (8192 bytes) 2026/10/24 21:31:07 activator: h.example.com: placed seed sd_sstbch3dbqvvoevuwp73eubhou (8192 bytes) 2026/10/24 21:31:07 activator: h.example.com: placed seed sd_6ievog7mnpmcijxetpinkt5p3y (8192 bytes) 2026/10/24 21:31:07 activator: h.example.com: keeping seed sd_6ievog7mnpmcijxetpinkt5p3y placed by an earlier wake 2026/10/24 21:31:07 activator: h.example.com: placed seed sd_6i637fmcmiiehzfpeajyb2l6ba (8192 bytes) 2026/10/24 21:31:07 activator: h.example.com: placed seed sd_kkgwqgr6bmtaejeph5gvo2pqqm (8192 bytes) 2026/10/24 21:31:07 activator: h.example.com: placed seed sd_dcntnsool5i3ramvg5ejy4iwim (8192 bytes) 2026/10/24 21:31:07 activator: h.example.com: seed sd_2ogh5s6vprzywec3e7ejorqmmi: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record. 2026/10/24 21:31:07 activator: h.example.com: seed sd_uw4a4qckfmqxby4d2jmtatsfx4: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record. 2026/10/24 21:31:07 activator: h.example.com: seed sd_kkfguefoqd3fzxleqlaxivz3wa: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record. 2026/10/24 21:31:07 activator: h.example.com: seed sd_do2mxmtu5zpslvckyexuftjqyu: h.example.com could not start from its seed: the snapshot could not be read. The next wake tries again.: bucket down 2026/10/24 21:31:07 activator: h.example.com: seed sd_gx3nvnnzjpu2dqldvedjseol7y: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record. 2026/10/24 21:31:07 activator: h.example.com: seed sd_ma6yahryw2ysw476sswmajmtji: h.example.com could not start from its seed: the snapshot is not a database that passes an integrity check.: integrity_check: file is not a database (26) 2026/10/24 21:31:07 activator: h.example.com: seed sd_a25nshjsgfu652mgcczonkhare: h.example.com could not start from its seed: claim control is on for this box. 2026/10/24 21:31:07 activator: h.example.com: seed sd_a25nshjsgfu652mgcczonkhare: h.example.com could not start from its seed: this box does not run instances nested. 2026/10/24 21:31:07 activator: h.example.com: seed sd_a3hkzcbftmtqspccaobrcfoogi: seed fallback for h.example.com: cannot tell whether it has a replica of its own: listing failed 2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com 2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh 2026/10/24 21:31:07 activator: h.example.com: seed sd_ltoqhue45x6tjho35637xo3dom: seed fallback for h.example.com: found no replica of its own to restore 2026/10/24 21:31:07 activator: h.example.com: promoting 1 staged object(s) from moves/h.example.com/ (residence move landing) 2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com 2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh 2026/10/24 21:31:07 activator: h.example.com: seed sd_dtz75toogkn7takslvhoxh7kju: seed fallback for h.example.com: found no replica of its own to restore 2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh 2026/10/24 21:31:07 activator: h.example.com: seed sd_7kzzo4vxgl5c5nbns3qfetjzxq: seed fallback for h.example.com: found no replica of its own to restore 2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com 2026/10/24 21:31:07 activator: h.example.com: placed seed sd_dhgk36dlk2ciuh2ut5wewqwzru (8192 bytes) 2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh 2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh 2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh 2026/10/24 21:31:07 activator: h.example.com: seed sd_nt42ymoturbqpzzmuor2fbh7am: h.example.com could not start from its seed: the snapshot could not be read. The next wake tries again.: connection reset 2026/10/24 21:31:07 activator: h.example.com: seed sd_ttpqc2kq4ef7rxnpcc76ycjmai: h.example.com could not start from its seed: it could not be put in place. The next wake tries again.: openat2 /tmp/nb729134131/data/instances/h.example.com/.seed-sd_ttpqc2kq4ef7rxnpcc76ycjmai.tmp: permission denied 2026/10/24 21:31:07 activator: h.example.com: seed sd_xlyfsxjt55erubtg7do3ujz7du: h.example.com could not start from its seed: the snapshot is not a database that passes an integrity check.: open: unable to open database file (14) 2026/10/24 21:31:07 activator: h.example.com: seed sd_d5pdi7cg5qapqjtjoc6iu4jft4: h.example.com could not start from its seed: it could not be put in place. The next wake tries again.: instancefs: path contains a symlink: /tmp/nb2040639698/data/instances/h.example.com/.seed-sd_d5pdi7cg5qapqjtjoc6iu4jft4.tmp 2026/10/24 21:31:07 activator: h.example.com: placed seed sd_pxg3vbxijikmq32ucykw75d3u4 (8192 bytes) 2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com 2026/10/24 21:31:07 activator: h.example.com: seed sd_zhwy6r2xei2z6ngaeaovvauy2a not recorded as placed yet: record unreachable 2026/10/24 21:31:07 activator: h.example.com: /tmp/nb770880587/data/instances/h.example.com/h.example.com.db held seed sd_65fjnz7xkzdxubplhb2apwaoja's marker, not this route's (seed "sd_oqlteiucz4zmptcgy7v6naguoi"): purged before the restore 2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com 2026/10/24 21:31:07 activator: h.example.com: /tmp/nb672842420/data/instances/h.example.com/h.example.com.db held seed sd_fi44qykjns4xfohystlhiyui2a's marker, not this route's (seed "sd_ch5sjo7st5d4hbvhjt7yax6wba"): purged before the restore 2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com 2026/10/24 21:31:07 activator: h.example.com: /tmp/nb3911238823/data/instances/h.example.com/h.example.com.db held seed sd_cq2g7qy2s77ecme2dbgdrhbnaa's marker, not this route's (seed ""): purged before the restore 2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh 2026/10/24 21:31:07 activator: h.example.com: /tmp/nb1568073016/data/instances/h.example.com/h.example.com.db held seed sd_s6vm2fdmjdxi2spygcgcgt5v2u's marker, not this route's (seed "sd_z3362cgid63rgj3k3x2vk3leri"): purged before the restore 2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com 2026/10/24 21:31:07 activator: h.example.com: /tmp/nb45767515/data/instances/h.example.com/h.example.com.db held seed sd_m7ecmb2xdibtgjq44524yrunmq's marker, not this route's (seed ""): purged before the restore 2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh 2026/10/24 21:31:07 activator: h.example.com: seed sd_5qoqxvowbahmnwjrmakqjw47uu: h.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup.events: unreadable 2026/10/24 21:31:07 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again 2026/10/24 21:31:07 activator: h.example.com: placed seed sd_5qoqxvowbahmnwjrmakqjw47uu (8192 bytes) 2026/10/24 21:31:07 activator: h.example.com: placed seed sd_6zyyj4h7aschtqmemdlatv5ply (8192 bytes) 2026/10/24 21:31:07 activator: h.example.com: /tmp/nb1464908442/data/instances/h.example.com/h.example.com.db held seed an aborted import's data's marker, not this route's (seed ""): purged before the restore 2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh 2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh 2026/10/24 21:31:07 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:07 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:07 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:07 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:07 activator: swap.example.com hibernated 2026/10/24 21:31:07 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:07 activator: swap.example.com idle 2h0m0s — hibernating 2026/10/24 21:31:07 activator: swap.example.com hibernated 2026/10/24 21:31:07 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:07 activator: LEASE LOST for swap.example.com (renewal failed) — standing down 2026/10/24 21:31:07 activator: swap.example.com hibernated 2026/10/24 21:31:08 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:08 activator: swap.example.com hibernated 2026/10/24 21:31:08 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:08 activator: swap.example.com is no longer hibernate-managed — releasing 2026/10/24 21:31:08 activator: swap.example.com hibernated 2026/10/24 21:31:08 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:08 activator: swap.example.com hibernated 2026/10/24 21:31:08 activator: exit.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:08 activator: exit.example.com hibernated 2026/10/24 21:31:08 activator: wake shop.example.com: activator stopped during startup 2026/10/24 21:31:08 activator: wake shop.example.com: activator stopped during startup 2026/10/24 21:31:08 activator: shop.example.com: route refreshed: db "" -> "" socket "/run/carlos/shop.example.com.sock" -> "/run/carlos/instances/shop.example.com/app.sock" 2026/10/24 21:31:08 activator: shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:08 activator: missed-join.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:08 activator: stop missed-join.example.com: a sidecar start is still in flight; keeping the lease until it returns 2026/10/24 21:31:09 activator: missed-join.example.com hibernated 2026/10/24 21:31:09 activator: nested-stuck.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:09 activator: stop nested-stuck.example.com: a sidecar start is still in flight; keeping the lease until it returns 2026/10/24 21:31:09 activator: nested-stuck.example.com: the sidecar a restart brought back for a stopping entry is not proven gone (activator: stop unproven; processes remain); keeping the lease until it has exited 2026/10/24 21:31:09 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have 2026/10/24 21:31:09 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have 2026/10/24 21:31:09 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have 2026/10/24 21:31:09 activator: nested-stuck.example.com hibernated 2026/10/24 21:31:09 activator: checks.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:09 activator: owned.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:09 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it 2026/10/24 21:31:09 activator: worker bot.x.internal respawned (restoring v1), now running v1 2026/10/24 21:31:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:09 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it 2026/10/24 21:31:09 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops 2026/10/24 21:31:09 activator: bot.x.internal hibernated 2026/10/24 21:31:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:09 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory; the sweep retries 2026/10/24 21:31:09 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory 2026/10/24 21:31:09 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory; the sweep retries 2026/10/24 21:31:09 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory — the sweep's crash restart takes it from here 2026/10/24 21:31:09 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory; the sweep retries 2026/10/24 21:31:09 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory; the sweep retries 2026/10/24 21:31:09 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory 2026/10/24 21:31:09 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory; the sweep retries 2026/10/24 21:31:09 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory — the sweep's crash restart takes it from here 2026/10/24 21:31:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:09 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:09 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:10 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it 2026/10/24 21:31:10 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:10 activator: worker bot.x.internal: the attempt's respawn did not finish within 150ms of its caller leaving; reporting its build as possibly executed 2026/10/24 21:31:10 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com hibernated 2026/10/24 21:31:10 activator: swap.example.com: generation 1149 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com hibernated 2026/10/24 21:31:10 activator: swap.example.com: generation 1151 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com hibernated 2026/10/24 21:31:10 activator: swap.example.com: generation 1153 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com hibernated 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com hibernated 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com hibernated 2026/10/24 21:31:10 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com hibernated 2026/10/24 21:31:10 activator: swap.example.com: generation 1158 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com hibernated 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com hibernated 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com hibernated 2026/10/24 21:31:10 activator: swap.example.com: generation 1163 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com hibernated 2026/10/24 21:31:10 activator: swap.example.com: generation 1166 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com hibernated 2026/10/24 21:31:10 activator: swap.example.com: generation 1168 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3965323956/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3965323956/005/not-a-dir/zdd: not a directory; the sweep retries 2026/10/24 21:31:10 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3965323956/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3965323956/005/not-a-dir/zdd: not a directory; the sweep retries 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: h.example.com: started carlos-instance-gen@h.example.com:7.service memory.max=1073741824 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:10 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: processes remain 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:10 activator: swap.example.com hibernated 2026/10/24 21:31:11 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:11 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:11 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:11 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:11 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:11 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:17 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: instance control /stop: 502: stop: processes remain 2026/10/24 21:31:17 activator: seeding h.example.com: /tmp/TestSeedUnitCleansUpOnlyItsOwnDBOnFailure1032246445/001/live/h.example.com seed --fast --db /tmp/nb3813122750/data/instances/h.example.com/h.example.com.db 2026/10/24 21:31:17 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:17 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later 2026/10/24 21:31:17 activator: seeding h.example.com: /tmp/TestSeedUnitFailureStopsBeforeCleanupAndKeepsTheDBWhenProcessesR3465610916/001/live/h.example.com seed --fast --db /tmp/nb342890965/data/instances/h.example.com/h.example.com.db 2026/10/24 21:31:18 activator: seeding h.example.com: /tmp/TestANestedSeedRecordsItsObligationBeforeItRuns1402265559/001/live/h.example.com seed --db /tmp/nb497377878/data/instances/h.example.com/h.example.com.db 2026/10/24 21:31:18 activator: seeded h.example.com 2026/10/24 21:31:18 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI4254598315/001/live/h.example.com seed --db /tmp/nb607401055/data/instances/h.example.com/h.example.com.db 2026/10/24 21:31:18 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again 2026/10/24 21:31:18 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh 2026/10/24 21:31:18 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI931097750/001/live/h.example.com seed --db /tmp/nb2916150891/data/instances/h.example.com/h.example.com.db 2026/10/24 21:31:18 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again 2026/10/24 21:31:18 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh 2026/10/24 21:31:18 activator: fifo-obligation.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept 2026/10/24 21:31:18 activator: one.example.com: generation 1902 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:18 activator: one.example.com: generation 1904 of v1: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:18 activator: one.example.com: first wake of v2 had 30s of its health window left at the exec, under the 1h0m0s floor; booting the good build v1 without trying it 2026/10/24 21:31:18 activator: one.example.com: generation 1907 of v1: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:18 activator: one.example.com: generation 1908 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:18 activator: one.example.com: generation 1909 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:18 activator: one.example.com hibernated 2026/10/24 21:31:18 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:18 activator: one.example.com: generation 1911 of v1: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:18 activator: tenant.test not started: it is held on v1, and the launch hashed "v2" (the live link moved after the check); the next wake checks again 2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:19 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:19 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:19 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:19 activator: one.example.com: generation 1916 of v0: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:19 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:19 activator: one.example.com: generation 1917 of v0: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:19 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:19 activator: one.example.com not woken: it is held on v1, and this wake would run "v2" (the live link moved after the check); the next wake checks again 2026/10/24 21:31:19 activator: one.example.com not woken: it is held on v1, and this wake would run "" (the live link moved after the check); the next wake checks again 2026/10/24 21:31:19 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:19 activator: swap.example.com hibernated 2026/10/24 21:31:19 activator: swap.example.com: generation 1919 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:19 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:20 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:20 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:20 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s 2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s 2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:21 activator: seeding h.example.com: /tmp/TestANestedUnversionedLinkWithNoHookWakesAsToday716706617/001/live/h.example.com seed --db /tmp/nb77297083/data/instances/h.example.com/h.example.com.db 2026/10/24 21:31:21 activator: seeded h.example.com 2026/10/24 21:31:21 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:21 activator: h.example.com: carlos-instance-gen@h.example.com:1926.service launched in 0s, healthy 0s later 2026/10/24 21:31:21 activator: h.example.com: woke in 1ms: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s 2026/10/24 21:31:21 activator: u.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s 2026/10/24 21:31:21 activator: e.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:21 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:21 activator: swap.example.com hibernated 2026/10/24 21:31:21 activator: swap.example.com: generation 1932 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:21 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s 2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:21 activator: one.example.com: generation 1935 of v1: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:21 activator: one.example.com: generation 1937 of v1: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:21 activator: one.example.com: generation 1940's child is not proven empty: activator: stop unproven; processes remain 2026/10/24 21:31:21 activator: one.example.com: failed wake's generation 1940 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:22 activator: one.example.com: generation 1941 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:22 activator: one.example.com: generation 1942 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:22 activator: one.example.com hibernated 2026/10/24 21:31:22 activator: one.example.com: generation 1943 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:23 activator: one.example.com: first wake of v2 had 400ms of its health window left at the exec, under the 20s floor; booting the good build v1 without trying it 2026/10/24 21:31:23 activator: one.example.com: generation 1945 of v1: record 0s, start 251ms, healthy 0s later 2026/10/24 21:31:23 activator: one.example.com: woke in 251ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 251ms, replicate 0s 2026/10/24 21:31:23 activator: two.example.com: generation 1946 of v1: record 0s, start 250ms, healthy 0s later 2026/10/24 21:31:23 activator: two.example.com: woke in 250ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 250ms, replicate 0s 2026/10/24 21:31:23 activator: x.example: hold's pinned wake refused or abandoned: the claims admission refused before the exec: hostbox: unresolved lifecycle contract: isolated root launcher cannot carry claim reservations 2026/10/24 21:31:23 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1436732107/003/g/1947/real.example.com.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:25 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1436732107/003/g/1948/real.example.com.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:25 activator: real.example.com: generation 1948 of v1: record 0s, start 1ms, healthy 0s later 2026/10/24 21:31:25 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1436732107/003/g/1948/real.example.com.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1436732107/003/g/1948/real.example.com.sock: no such file or directory) 2026/10/24 21:31:25 activator: real.example.com: woke in 2.001s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 2.001s, replicate 0s 2026/10/24 21:31:25 activator: real.example.com hibernated 2026/10/24 21:31:27 activator: real.example.com: generation 1950 of v1: record 0s, start 1ms, healthy 0s later 2026/10/24 21:31:27 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign3884098207/004/hand-set.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign3884098207/004/hand-set.sock: no such file or directory) 2026/10/24 21:31:27 activator: real.example.com: woke in 2.002s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 2.002s, replicate 0s 2026/10/24 21:31:27 activator: real.example.com hibernated 2026/10/24 21:31:27 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:27 activator: bot.x.internal: worker generation 0's host cgroup is not proven empty: the host cgroup still holds a process 2026/10/24 21:31:27 activator: bot.x.internal: worker generation 1952's host cgroup is not proven empty: the host cgroup still holds a process 2026/10/24 21:31:27 activator: bot.x.internal: failed wake's generation 1952 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:28 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:28 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd: not a directory; the sweep retries 2026/10/24 21:31:28 activator: worker bot.x.internal not spawned: the build it would exec (v2 /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/003/v2) could not be published first: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd: not a directory 2026/10/24 21:31:28 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd: not a directory; the sweep retries 2026/10/24 21:31:28 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd: not a directory; the sweep retries 2026/10/24 21:31:28 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn2205933732/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn2205933732/002/not-a-dir: not a directory; the sweep retries 2026/10/24 21:31:28 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn2205933732/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn2205933732/002/not-a-dir: not a directory; the sweep retries 2026/10/24 21:31:28 activator: one.example.com: generation 1955 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:28 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:28 activator: instance one.example.com exited unexpectedly — recycling 2026/10/24 21:31:28 activator: one.example.com hibernated 2026/10/24 21:31:28 activator: one.example.com: generation 1956 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:28 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:28 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:28 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:28 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:28 activator: one.example.com hibernated 2026/10/24 21:31:30 activator: one.example.com: first wake of v2 had 499ms of its health window left at the exec, under the 1s floor; booting the good build v1 without trying it 2026/10/24 21:31:30 activator: one.example.com: generation 1961 of v1: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:30 activator: one.example.com: woke in 1.501s: checks 0s, lease 0s, restore 0s, identity 1.501s, spawn 0s, replicate 0s 2026/10/24 21:31:32 activator: one.example.com: generation 1962 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:32 activator: one.example.com: woke in 1.5s: checks 0s, lease 0s, restore 0s, identity 1.5s, spawn 0s, replicate 0s 2026/10/24 21:31:32 activator: one.example.com: generation 1964 of v1: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:32 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:32 activator: bot.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:32 activator: worker bot.example.com exited — restarting (backoff now 2s) 2026/10/24 21:31:32 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com 2026/10/24 21:31:32 activator: worker bot.example.com respawned (exited), now running 2026/10/24 21:31:32 activator: bot.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:32 activator: worker bot.example.com exited — restarting (backoff now 2s) 2026/10/24 21:31:32 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com 2026/10/24 21:31:32 activator: worker bot.example.com respawned (exited), now running 2026/10/24 21:31:32 activator: bot.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:32 activator: bot.example.com: worker generation 1967's host cgroup is not proven empty: the host cgroup still holds a process 2026/10/24 21:31:32 activator: stop bot.example.com: not every process of the entry is proven gone; keeping the lease until it is 2026/10/24 21:31:32 activator: bot.example.com hibernated 2026/10/24 21:31:32 activator: bot3.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:32 activator: bot3.example.com: worker generation 1968's host cgroup is not proven empty: the host cgroup still holds a process 2026/10/24 21:31:32 activator: worker bot3.example.com NOT respawned (test: a restart with no old handle): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries 2026/10/24 21:31:32 activator: bot4.example.com: worker generation 1970's host cgroup is not proven empty: the host cgroup still holds a process 2026/10/24 21:31:32 activator: bot4.example.com: failed wake's generation 1970 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:32 activator: bot4.example.com: failed wake's generation 1970 is proven stopped; lease given back 2026/10/24 21:31:32 activator: bot2.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:32 activator: worker bot2.example.com exited — restarting (backoff now 2s) 2026/10/24 21:31:32 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com 2026/10/24 21:31:32 activator: bot2.example.com: worker generation 1971's host cgroup is not proven empty: the host cgroup still holds a process 2026/10/24 21:31:32 activator: worker bot2.example.com NOT respawned (exited): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries 2026/10/24 21:31:32 activator: worker bot2.example.com exited — restarting (backoff now 2s) 2026/10/24 21:31:32 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com 2026/10/24 21:31:32 activator: worker bot2.example.com respawned (exited), now running 2026/10/24 21:31:32 activator: bot.example.com: woke in 22ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 1ms, settle 21ms, replicate 0s 2026/10/24 21:31:32 activator: worker bot.example.com exited — restarting (backoff now 2s) 2026/10/24 21:31:32 activator: worker bot.example.com respawned (exited), now running v1 2026/10/24 21:31:32 activator: bot.example.com hibernated 2026/10/24 21:31:32 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:33 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:33 activator: bot10.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:33 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it 2026/10/24 21:31:33 activator: worker bot10.example.com: the shed respawn child's exit is not proven (activator: stop unproven; processes remain); the entry keeps it and its build until the sweep proves it 2026/10/24 21:31:33 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down 2026/10/24 21:31:33 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is 2026/10/24 21:31:33 activator: bot10.example.com: the shed respawn child has exited 2026/10/24 21:31:33 activator: bot10.example.com hibernated 2026/10/24 21:31:33 activator: bot10.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:33 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it 2026/10/24 21:31:33 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down 2026/10/24 21:31:33 activator: bot10.example.com respawn did not finish before its stop; stopping the rest, the lease is kept until it has 2026/10/24 21:31:33 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is 2026/10/24 21:31:33 activator: bot10.example.com hibernated 2026/10/24 21:31:33 activator: bot8.example.com: worker generation 1974's host cgroup is not proven empty: the host cgroup still holds a process 2026/10/24 21:31:33 activator: bot8.example.com: failed wake's generation 1974 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:33 activator: bot8.example.com: failed wake's generation 1974 is proven stopped; lease given back 2026/10/24 21:31:33 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:33 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:33 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:33 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:33 activator: worker bot.x.internal respawned (falling back to v1), now running v1 2026/10/24 21:31:33 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:33 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:33 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:33 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle 2026/10/24 21:31:34 activator: worker bot.x.internal respawned (falling back to v1), now running v1 2026/10/24 21:31:34 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:34 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:34 activator: bot.x.internal hibernated 2026/10/24 21:31:34 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:34 activator: worker bot.x.internal respawned (attempt (restart)), now running v1 2026/10/24 21:31:34 activator: worker bot.x.internal respawned (restoring v1), now running v1 2026/10/24 21:31:34 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:34 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:34 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:34 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:34 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it 2026/10/24 21:31:35 activator: worker bot.x.internal respawned (restoring v1), now running v1 2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:35 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it 2026/10/24 21:31:35 activator: worker bot.x.internal respawned (restoring v1), now running v1 2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it 2026/10/24 21:31:35 activator: worker bot.x.internal respawned (restoring v1), now running v1 2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:35 activator: bot.x.internal: worker generation 1977's host cgroup is not proven empty: the host cgroup still holds a process 2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (attempt (release)): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries 2026/10/24 21:31:35 activator: bot.x.internal: worker generation 1977's host cgroup is not proven empty: the host cgroup still holds a process 2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (restoring v1): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries 2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: lease lost; abandoning it 2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:35 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory; the sweep retries 2026/10/24 21:31:35 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory 2026/10/24 21:31:35 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory; the sweep retries 2026/10/24 21:31:35 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory — the sweep's crash restart takes it from here 2026/10/24 21:31:35 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory; the sweep retries 2026/10/24 21:31:35 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory; the sweep retries 2026/10/24 21:31:35 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory 2026/10/24 21:31:35 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory; the sweep retries 2026/10/24 21:31:35 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory — the sweep's crash restart takes it from here 2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:35 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:35 activator: bot.x.internal hibernated 2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:35 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (falling back to v1): swap: lease lost; abandoning it 2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:35 activator: WORKER bot.x.internal failed to respawn (attempt (restart)): exec: text file busy — the sweep's crash restart takes it from here 2026/10/24 21:31:35 activator: WORKER bot.x.internal failed to respawn (restoring v1): exec: text file busy — the sweep's crash restart takes it from here 2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:35 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:36 activator: worker bot.x.internal respawned (falling back to v1), now running v1 2026/10/24 21:31:36 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:36 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:36 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it 2026/10/24 21:31:36 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops 2026/10/24 21:31:36 activator: bot.x.internal hibernated 2026/10/24 21:31:36 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:36 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it 2026/10/24 21:31:36 activator: worker bot.x.internal respawned (restoring v1), now running v1 2026/10/24 21:31:36 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:36 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it 2026/10/24 21:31:36 activator: worker bot.x.internal respawned (restoring v1), now running v1 2026/10/24 21:31:36 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:36 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:36 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it 2026/10/24 21:31:36 activator: bot.x.internal hibernated 2026/10/24 21:31:36 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:36 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:36 activator: LEASE LOST for bot.x.internal (renewal failed) — standing down 2026/10/24 21:31:36 activator: bot.x.internal hibernated 2026/10/24 21:31:37 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:37 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:37 activator: bot.x.internal hibernated 2026/10/24 21:31:37 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:37 activator: worker bot.x.internal respawned (attempt (release)), now running v2 2026/10/24 21:31:37 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it 2026/10/24 21:31:37 activator: bot.x.internal hibernated 2026/10/24 21:31:37 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawngated_build_is_the_ex831226331/003/live.example.com.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:37 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawna_held_build_starts_n2970356973/003/live.example.com.sock.control: bind: invalid argument (continuing without it) --- FAIL: TestExecBackendStartSidecarGen (0.01s) --- FAIL: TestExecBackendStartSidecarGen/foreign (0.00s) swap_sidecar_test.go:669: fork/exec /tmp/TestExecBackendStartSidecarGenforeign4131554283/001/versions/v2/app: text file busy 2026/10/24 21:31:38 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawnno_gate_is_todays_sp3847055878/003/live.example.com.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:38 activator: shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: stop fence.example: earlier process may still run 2026/10/24 21:31:38 activator: fence.example hibernated 2026/10/24 21:31:38 activator: fence.example hibernated 2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory; the sweep retries 2026/10/24 21:31:38 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory; the sweep retries 2026/10/24 21:31:38 activator: worker first.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory 2026/10/24 21:31:38 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory; the sweep retries 2026/10/24 21:31:38 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory; the sweep retries 2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: swap.example.com hibernated 2026/10/24 21:31:38 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:1988.service exited before it was observed 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:38 activator: one.example.com idle 1m31s — hibernating 2026/10/24 21:31:38 activator: one.example.com hibernated 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here 2026/10/24 21:31:38 provision: a.shop.example.com: swapped for import sd_vrw6ytbwk7j3x4r3isnojpwgvm (the original route is journaled) 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: one.example.com idle 1h0m0s — hibernating 2026/10/24 21:31:38 activator: one.example.com hibernated 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (hit its max_run), now running v1 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:38 activator: bot6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: bot1.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: bot.example.com: woke in 21ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 1ms, settle 20ms, replicate 0s 2026/10/24 21:31:38 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2 2026/10/24 21:31:38 activator: bot.example.com hibernated 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory; the sweep retries 2026/10/24 21:31:38 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory 2026/10/24 21:31:38 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory; the sweep retries 2026/10/24 21:31:38 activator: WORKER bot.x.internal failed to respawn (exited): worker build not published before its exec: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory — the sweep's crash restart takes it from here 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: WORKER bot.x.internal failed to respawn (exited): fork/exec: resource temporarily unavailable — the sweep's crash restart takes it from here 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: bot9.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: worker bot9.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it 2026/10/24 21:31:38 activator: bot9.example.com: worker generation 1994's host cgroup is not proven empty: the host cgroup still holds a process 2026/10/24 21:31:38 activator: generation 1994 of bot9.example.com did not prove its exit; keeping it in the operation set 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:38 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (hit its max_run), now running v1 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:38 activator: LEASE LOST for bot9.example.com (renewal failed) — standing down 2026/10/24 21:31:38 activator: bot9.example.com hibernated 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here 2026/10/24 21:31:38 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2 2026/10/24 21:31:38 activator: bot2.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot.x.internal exited; live/bot.x.internal names v2, but the release scheduler owns build changes, so it recycles its captured build v1 2026/10/24 21:31:38 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (hit its max_run), now running v1 2026/10/24 21:31:38 activator: a.shop.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@a.shop.example.com: unit still active); keeping the lease until it is 2026/10/24 21:31:38 provision: a.shop.example.com: import sd_vrw6ytbwk7j3x4r3isnojpwgvm rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable) 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:38 activator: worker bot1.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot1.x.internal exited; it serves v1 but live/bot1.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling 2026/10/24 21:31:38 activator: worker bot2.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot2.x.internal exited; live/bot2.x.internal names v2, but this sweep's converge budget is spent, so it recycles its captured build v1 and converges on a later sweep 2026/10/24 21:31:38 activator: worker bot1.x.internal respawned (converging onto a new build), now running v2 2026/10/24 21:31:38 activator: worker bot2.x.internal respawned (exited), now running v1 2026/10/24 21:31:38 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2 2026/10/24 21:31:38 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (hit its max_run), now running v1 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot.x.internal exited; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2 2026/10/24 21:31:38 activator: worker bot2.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:38 activator: worker bot2.x.internal respawned (converging onto a new build), now running v2 2026/10/24 21:31:38 provision: a.shop.example.com: import sd_vrw6ytbwk7j3x4r3isnojpwgvm rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable) 2026/10/24 21:31:38 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child 2026/10/24 21:31:38 activator: worker bot.x.internal hit its max_run; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2 2026/10/24 21:31:38 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (hit its max_run), now running v2 2026/10/24 21:31:38 provision: a.shop.example.com: import sd_vrw6ytbwk7j3x4r3isnojpwgvm rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable) 2026/10/24 21:31:38 activator: nested-costop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:38 activator: stop sidecar nested-costop.example.com: activator: stop unproven; processes remain 2026/10/24 21:31:38 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have 2026/10/24 21:31:38 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have 2026/10/24 21:31:38 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have 2026/10/24 21:31:38 provision: a.shop.example.com: import sd_vrw6ytbwk7j3x4r3isnojpwgvm rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable) 2026/10/24 21:31:38 activator: nested-costop.example.com hibernated 2026/10/24 21:31:38 activator: restart.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: instance restart.example.com exited unexpectedly — recycling 2026/10/24 21:31:38 activator: restart.example.com hibernated 2026/10/24 21:31:38 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: swap.example.com hibernated 2026/10/24 21:31:38 activator: h.example.com: kill carlos-instance-gen@h.example.com:2002.service after a failed start: stop timed out 2026/10/24 21:31:38 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2002.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out 2026/10/24 21:31:38 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: h.example.com: kill carlos-instance-gen@h.example.com:2004.service after a failed start: stop timed out 2026/10/24 21:31:38 activator: SIDECAR for late-sidecar.example failed to start: activator: maintenance fence active — retaining returned process 2026/10/24 21:31:38 activator: late-sidecar.example: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:38 activator: late-sidecar.example hibernated 2026/10/24 21:31:38 activator: one.example.com is no longer hibernate-managed — releasing 2026/10/24 21:31:38 activator: one.example.com hibernated 2026/10/24 21:31:38 activator: replica shop.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own 2026/10/24 21:31:38 activator: slowproof.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: instance slowproof.example.com exited unexpectedly — recycling 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: slowproof.example.com hibernated 2026/10/24 21:31:38 activator: own.example.com: failed wake's generation 0 is proven stopped; lease given back 2026/10/24 21:31:38 activator: bot7.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 provision: a.shop.example.com: swapped for import sd_smee6gqicg2fasqnkhupnie6xm (the original route is journaled) 2026/10/24 21:31:38 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:38 activator: bot5.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it 2026/10/24 21:31:38 activator: queued.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: instance unproven.example.com exited unexpectedly — recycling 2026/10/24 21:31:38 activator: unproven.example.com hibernated 2026/10/24 21:31:38 activator: empty.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:38 activator: one.example.com: sidecar starting on the instance's own build v1 (/opt/carlos/versions/v1/app): this box will not start v2, the build its live link names 2026/10/24 21:31:38 activator: sidecar for one.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:38 activator: bot.x.internal serving v1 but live symlink is now v2 — cycling onto the new build 2026/10/24 21:31:38 activator: bot.x.internal hibernated 2026/10/24 21:31:38 activator: early.example.com: the aborted start's unit is not proven stopped (context canceled); keeping the lease until it is 2026/10/24 21:31:38 activator: shutdown: out of time with 1 failed wake(s) not proven stopped, their leases still held: early.example.com (generation 1998) 2026/10/24 21:31:38 activator: refused.example.com: giving the lease back failed: delete lease: context deadline exceeded; retrying it 2026/10/24 21:31:38 activator: refused.example.com: lease given back on retry 2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: one.example.com hibernated 2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: one.example.com hibernated 2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: one.example.com hibernated 2026/10/24 21:31:38 activator: unproven.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@unproven: instance control socket: connection refused); keeping the lease until it is 2026/10/24 21:31:38 activator: odd.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: stuck.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2 2026/10/24 21:31:38 provision: a.shop.example.com: import sd_smee6gqicg2fasqnkhupnie6xm rolled back to its original route 2026/10/24 21:31:38 provision: a.shop.example.com: import sd_smee6gqicg2fasqnkhupnie6xm is cleaned up and its journal row retired 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:38 activator: one.example.com hibernated 2026/10/24 21:31:38 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven 2026/10/24 21:31:38 activator: bot.x.internal hibernated 2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:38 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: queued.example.com hibernated 2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: one.example.com serving v1 but v2 is adopted — cycling onto the new build 2026/10/24 21:31:39 activator: one.example.com hibernated 2026/10/24 21:31:39 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a 2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: stuck.example.com hibernated 2026/10/24 21:31:39 provision: a.shop.example.com: swapped for import sd_2qytelynd4qpmqvjgyhcp7l3ly (the original route is journaled) 2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:39 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:31:39 activator: worker bot.x.internal respawned (exited), now running 2026/10/24 21:31:39 activator: a.shop.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@a.shop.example.com: instance control socket: connection refused); keeping the lease until it is 2026/10/24 21:31:39 provision: a.shop.example.com: import sd_2qytelynd4qpmqvjgyhcp7l3ly rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable) 2026/10/24 21:31:39 activator: a.shop.example.com: failed wake's generation 2028 is proven stopped; lease given back 2026/10/24 21:31:39 provision: a.shop.example.com: import sd_2qytelynd4qpmqvjgyhcp7l3ly rolled back to its original route 2026/10/24 21:31:39 provision: a.shop.example.com: import sd_2qytelynd4qpmqvjgyhcp7l3ly is cleaned up and its journal row retired 2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: swap.example.com hibernated 2026/10/24 21:31:39 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start 2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: slowproof.example.com: the aborted start's unit is not proven stopped (context deadline exceeded); keeping the lease until it is 2026/10/24 21:31:39 provision: a.shop.example.com: swapped for import sd_3vuhuk6ovoheqq4chzis5unm4q (the original route is journaled) 2026/10/24 21:31:39 activator: control listener for shop.example.com: listen unix /tmp/TestARefusedLaunchLeavesNoBuildStateRecord3912167717/002/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it) 2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: own.example.com: failed wake's generation 0 is proven stopped; lease given back 2026/10/24 21:31:39 provision: a.shop.example.com: import sd_3vuhuk6ovoheqq4chzis5unm4q rolled back to its original route 2026/10/24 21:31:39 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:39 activator: worker bot.x.internal NOT respawned (converging onto a new build): instance config unavailable on this box: cannot open sealed config — standing down and releasing its lease 2026/10/24 21:31:39 activator: bot.x.internal hibernated 2026/10/24 21:31:39 provision: a.shop.example.com: import sd_3vuhuk6ovoheqq4chzis5unm4q is cleaned up and its journal row retired 2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:39 activator: bot5.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it 2026/10/24 21:31:39 activator: bot6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it 2026/10/24 21:31:39 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, nor the instance's own build v1: a failure record holds this build: v1 2026/10/24 21:31:39 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s) 2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:39 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:31:39 activator: worker bot.x.internal NOT respawned (exited): instance config unavailable on this box: no key — standing down and releasing its lease 2026/10/24 21:31:39 activator: bot.x.internal hibernated 2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: one.example.com is no longer hibernate-managed — releasing 2026/10/24 21:31:39 activator: one.example.com hibernated 2026/10/24 21:31:39 provision: a.shop.example.com: swapped for import sd_sofvdtjd7r4megtdbwje6n42ay (the original route is journaled) 2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:39 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle 2026/10/24 21:31:39 activator: a.shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: a.shop.example.com hibernated 2026/10/24 21:31:39 activator: shutdown: out of time with 1 failed wake(s) not proven stopped, their leases still held: unproven.example.com (generation 2017) 2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:39 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:39 activator: bot.x.internal hibernated 2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 provision: a.shop.example.com: import sd_sofvdtjd7r4megtdbwje6n42ay rolled back to its original route 2026/10/24 21:31:39 provision: a.shop.example.com: import sd_sofvdtjd7r4megtdbwje6n42ay is cleaned up and its journal row retired 2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: processes remained; the sweep tries again 2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: c.example.com hibernated 2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:39 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: c.example.com hibernated 2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: c.example.com hibernated 2026/10/24 21:31:39 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:39 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:31:39 activator: worker bot.x.internal NOT respawned (exited): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state 2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: swap.example.com hibernated 2026/10/24 21:31:39 activator: h.example.com: kill carlos-instance-gen@h.example.com:2053.service after a failed start: stop timed out 2026/10/24 21:31:39 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: the unit failed to start; activator: generation start failed and its termination is unproven: stop timed out 2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: swap.example.com hibernated 2026/10/24 21:31:39 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start 2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: c.example.com hibernated 2026/10/24 21:31:39 activator: one.example.com: generation 2057 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:39 activator: one.example.com: generation 2060 of v1: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: seeding shop.example.com: /tmp/TestARefusedLaunchLeavesNoBuildStateRecordseed_refused_beside_a3244348707/002/versions/v1/app seed --db /tmp/TestARefusedLaunchLeavesNoBuildStateRecordseed_refused_beside_a3244348707/003/shop.example.com.db 2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: c.example.com hibernated 2026/10/24 21:31:39 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:39 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:39 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:39 activator: x.example hibernated 2026/10/24 21:31:39 activator: bot7.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it 2026/10/24 21:31:39 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, and the instance's own build v1 cannot be named: staged copy unreadable 2026/10/24 21:31:39 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s) 2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: swap.example.com hibernated 2026/10/24 21:31:39 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: cgroup: the host's parent cgroup holds a process directly: holds 1 process(es) 2026/10/24 21:31:39 activator: c.example.com idle 1h0m0s — hibernating 2026/10/24 21:31:39 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child 2026/10/24 21:31:39 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:31:39 activator: worker bot.x.internal NOT respawned (hit its max_run): instance config unavailable on this box: no key — standing down and releasing its lease 2026/10/24 21:31:39 activator: bot.x.internal hibernated 2026/10/24 21:31:39 activator: c.example.com hibernated 2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:39 activator: c.example.com: run run_a is not proven stopped: processes remained 2026/10/24 21:31:39 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is 2026/10/24 21:31:39 activator: bot-a.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: c.example.com: run run_a is not proven stopped: processes remained 2026/10/24 21:31:39 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is 2026/10/24 21:31:39 activator: bot-b.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: SIDECAR for partial-sidecar.example failed to start: sidecar started then failed — retaining returned process 2026/10/24 21:31:39 activator: partial-sidecar.example: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:39 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child 2026/10/24 21:31:39 activator: worker bot.x.internal respawned (hit its max_run), now running v1 2026/10/24 21:31:39 activator: bot.x.internal hibernated 2026/10/24 21:31:39 activator: partial-sidecar.example hibernated 2026/10/24 21:31:39 activator: worker bot-b.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:39 activator: worker bot-b.x.internal respawned (converging onto a new build), now running v2 2026/10/24 21:31:39 activator: c.example.com: run run_a is not proven stopped: processes remained 2026/10/24 21:31:39 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is 2026/10/24 21:31:39 activator: c.example.com: run run_a is not proven stopped: processes remained 2026/10/24 21:31:39 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is 2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: c.example.com hibernated 2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: instance c.example.com exited unexpectedly — recycling 2026/10/24 21:31:40 activator: c.example.com hibernated 2026/10/24 21:31:40 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:40 activator: worker bot.x.internal respawned (converging onto a new build), now running v2 2026/10/24 21:31:40 activator: one.example.com: generation 2077 of v3: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:31:40 activator: worker bot-a.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:40 activator: worker bot-a.x.internal respawned (converging onto a new build), now running v2 2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: swap.example.com hibernated 2026/10/24 21:31:40 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch 2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: swap.example.com hibernated 2026/10/24 21:31:40 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch 2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: c.example.com: run run_a is not proven stopped: processes remained 2026/10/24 21:31:40 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is 2026/10/24 21:31:40 activator: one.example.com: v2 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:31:40 activator: c.example.com hibernated 2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:40 activator: seeding shop.example.com: /tmp/TestARefusedLaunchLeavesNoBuildStateRecorda_seed_that_ran_is_re2455287812/001/versions/v1/app seed --db /tmp/TestARefusedLaunchLeavesNoBuildStateRecorda_seed_that_ran_is_re2455287812/002/shop.example.com.db 2026/10/24 21:31:40 activator: shop.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is 2026/10/24 21:31:40 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: x.example hibernated 2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: c.example.com idle 1h0m0s — hibernating 2026/10/24 21:31:40 activator: c.example.com hibernated 2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:40 activator: bot6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:40 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it 2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: x.example hibernated 2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:40 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:40 activator: one.example.com idle 1h0m0s — hibernating 2026/10/24 21:31:40 activator: one.example.com hibernated 2026/10/24 21:31:40 activator: SIDECAR restart for partial-sidecar.example failed: sidecar started then failed — retaining returned process 2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:40 activator: partial-sidecar.example hibernated 2026/10/24 21:31:40 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:40 activator: worker bot.x.internal respawned (converging onto a new build), now running v2 2026/10/24 21:31:40 activator: bot.x.internal hibernated 2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: c.example.com: run run_a is not proven stopped: processes remained 2026/10/24 21:31:40 activator: stop c.example.com: not every process of the entry is proven gone; keeping the lease until it is 2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:40 activator: one.example.com: generation 2105 of v1: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:40 activator: c.example.com: run run_a is not proven stopped: processes remained 2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: sidecar for one.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:40 activator: LEASE LOST for c.example.com (lost) — standing down 2026/10/24 21:31:40 activator: c.example.com hibernated 2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:40 activator: c.example.com: run run_a is not proven stopped: processes remained 2026/10/24 21:31:40 activator: sidecar staged for awake tenant.test — adopting 2026/10/24 21:31:40 activator: sidecar for tenant.test was down — restarted (backoff now 2s) 2026/10/24 21:31:40 activator: c.example.com: run run_a is not proven stopped: processes remained 2026/10/24 21:31:40 activator: c.example.com hibernated 2026/10/24 21:31:40 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:40 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:40 activator: sidecar staged for awake tenant.test — adopting 2026/10/24 21:31:40 activator: sidecar for tenant.test was down — restarted (backoff now 2s) 2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: swap.example.com hibernated 2026/10/24 21:31:40 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a cgroup that took cgroup.kill could not be made fresh before a spawn: carlos-exec@h.example.com: device or resource busy 2026/10/24 21:31:40 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:40 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:40 activator: two.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:40 activator: x.example hibernated 2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: c.example.com idle 1h0m0s — hibernating 2026/10/24 21:31:40 activator: c.example.com: run run_a is not proven stopped: processes remained 2026/10/24 21:31:40 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is 2026/10/24 21:31:40 activator: bot7.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:40 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it 2026/10/24 21:31:40 activator: c.example.com hibernated 2026/10/24 21:31:40 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it 2026/10/24 21:31:40 activator: bot.x.internal hibernated 2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:40 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:40 activator: undelegated-wake.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: SIDECAR restart for one.example.com failed: swap: lease lost (next attempt in 2s) 2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:40 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:40 activator: bot.x.internal hibernated 2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:40 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:40 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it 2026/10/24 21:31:40 activator: bot.x.internal hibernated 2026/10/24 21:31:40 activator: one.example.com: generation 2122 of v3: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: plain.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: one.example.com: generation 2124 of v3: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: c.example.com serving v1 but v2 is adopted — cycling onto the new build 2026/10/24 21:31:40 activator: c.example.com hibernated 2026/10/24 21:31:40 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:31:40 activator: always-on sidecar bot.x.internal is not running — starting it 2026/10/24 21:31:40 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it 2026/10/24 21:31:40 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:40 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it 2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:41 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:41 activator: kass.example.com idle 11s — hibernating 2026/10/24 21:31:41 activator: kass.example.com hibernated 2026/10/24 21:31:41 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:41 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:31:41 activator: SIDECAR restart for one.example.com failed: the build state could not be written, so this build was not started: bucket refused the build state (next attempt in 2s) 2026/10/24 21:31:41 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: c.example.com idle 1h0m0s — hibernating 2026/10/24 21:31:41 activator: c.example.com hibernated 2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:41 activator: always-on sidecar bot00.x.internal is not running — starting it 2026/10/24 21:31:41 activator: always-on sidecar bot01.x.internal is not running — starting it 2026/10/24 21:31:41 activator: always-on sidecar bot02.x.internal is not running — starting it 2026/10/24 21:31:41 activator: always-on sidecar bot03.x.internal is not running — starting it 2026/10/24 21:31:41 activator: always-on sidecar bot04.x.internal is not running — starting it 2026/10/24 21:31:41 activator: always-on sidecar bot05.x.internal is not running — starting it 2026/10/24 21:31:41 activator: always-on sidecar bot06.x.internal is not running — starting it 2026/10/24 21:31:41 activator: always-on sidecar bot07.x.internal is not running — starting it 2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:41 activator: x.example hibernated 2026/10/24 21:31:41 activator: two.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build 2026/10/24 21:31:41 activator: bot07.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: bot02.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: bot04.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: bot03.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: bot05.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: bot06.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: bot01.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: bot00.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: always-on sidecar bot08.x.internal is not running — starting it 2026/10/24 21:31:41 activator: always-on sidecar bot09.x.internal is not running — starting it 2026/10/24 21:31:41 activator: always-on sidecar bot10.x.internal is not running — starting it 2026/10/24 21:31:41 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:41 activator: bot09.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: bot08.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: bot10.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: bot6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it 2026/10/24 21:31:41 activator: sidecar for one.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: swap.example.com: v1 not started: its build state could not be written: bucket: 503 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: no-host-proof.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: sidecar staged for awake swap.example.com — adopting 2026/10/24 21:31:41 activator: sidecar for swap.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child 2026/10/24 21:31:41 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:31:41 activator: worker bot.x.internal respawned (hit its max_run), now running 2026/10/24 21:31:41 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child 2026/10/24 21:31:41 activator: bot.x.internal hibernated 2026/10/24 21:31:41 activator: bot.x.internal hibernated 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: instance control: 502 mid-roll; the sweep tries again 2026/10/24 21:31:41 activator: bot.x.internal hibernated 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: swap.example.com hibernated 2026/10/24 21:31:41 activator: swap.example.com: generation 2148 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running 2026/10/24 21:31:41 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: c.example.com hibernated 2026/10/24 21:31:41 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:41 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it 2026/10/24 21:31:41 activator: x.example hibernated 2026/10/24 21:31:41 activator: one.example.com: generation 2151 of v3: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:41 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:31:41 activator: SIDECAR for one.example.com failed to start: the build state could not be written, so this build was not started: bucket refused the build state — serving without it (the sweep retries) 2026/10/24 21:31:41 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:41 activator: shutdown-pending.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: stuck.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: shutdown-pending.example.com idle 2m0s — hibernating 2026/10/24 21:31:41 activator: stop shutdown-pending.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: swap.example.com hibernated 2026/10/24 21:31:41 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a generation's limits could not be applied: h.example.com: the placement shim is not available 2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:41 activator: stuck.example.com idle 2m0s — hibernating 2026/10/24 21:31:41 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven 2026/10/24 21:31:41 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:41 activator: x.example hibernated 2026/10/24 21:31:41 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it 2026/10/24 21:31:41 activator: bot.x.internal hibernated 2026/10/24 21:31:41 activator: fence.example hibernated 2026/10/24 21:31:41 activator: stop fence.example: stop refused 2026/10/24 21:31:41 activator: fence.example hibernated 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: swap.example.com hibernated 2026/10/24 21:31:41 activator: bot7.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it 2026/10/24 21:31:41 activator: no-host-proof.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: squatted.example.com: child exited during the health wait (external process serving its socket?) — not normalizing the socket mode before 2026/10/24 21:31:41 activator: h.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: h.example.com idle 2m0s — hibernating 2026/10/24 21:31:41 activator: stop h.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven 2026/10/24 21:31:41 activator: shutdown-pending.example.com hibernated 2026/10/24 21:31:41 activator: h.example.com hibernated 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: swap.example.com hibernated 2026/10/24 21:31:41 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:31:41 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: the build state could not be written, so this build was not started: bucket refused the build state 2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:41 activator: bot.x.internal hibernated 2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running 2026/10/24 21:31:41 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: c.example.com is no longer hibernate-managed — releasing 2026/10/24 21:31:41 activator: c.example.com hibernated 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: instance control: 502 mid-roll; the sweep tries again 2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running 2026/10/24 21:31:41 activator: stop fence.example: stop refused 2026/10/24 21:31:41 activator: fence.example hibernated 2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:41 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it 2026/10/24 21:31:41 activator: x.example hibernated 2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running 2026/10/24 21:31:41 activator: seeding h.example.com: /tmp/TestPlan14GenerationBackendAuxiliaryBoundaryclaim=falseseedpi2576486630/001/live/h.example.com seed --db /tmp/nb2326995521/data/instances/h.example.com/h.example.com.db 2026/10/24 21:31:41 activator: seeded h.example.com 2026/10/24 21:31:41 activator: seeding h.example.com: /tmp/TestPlan14GenerationBackendAuxiliaryBoundaryclaim=falseseedpi785541684/001/live/h.example.com seed --db /tmp/nb2470485756/data/instances/h.example.com/h.example.com.db 2026/10/24 21:31:41 activator: seeded h.example.com 2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running 2026/10/24 21:31:41 activator: shop.example.com: seed placement not recorded yet: record unreachable 2026/10/24 21:31:41 activator: shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: worker bot.x.internal reported idle but is always-on (hibernate:false) — ignoring 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 4s) 2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running 2026/10/24 21:31:41 activator: x.example source opened on the recovery probe's session; serving 2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: fence.example hibernated 2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 8s) 2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running 2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:41 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:42 activator: fence.example hibernated 2026/10/24 21:31:42 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build 2026/10/24 21:31:42 activator: WORKER bot.x.internal failed to respawn (converging onto a new build): exec: live symlink gone — the sweep's crash restart takes it from here 2026/10/24 21:31:42 activator: bot.x.internal has a scheduled delivery due — waking it 2026/10/24 21:31:42 activator: bot.x.internal has a scheduled delivery due — waking it 2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:42 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal 2026/10/24 21:31:42 activator: worker bot.x.internal respawned (exited), now running v2 2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:42 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later 2026/10/24 21:31:42 activator: bot.x.internal has a scheduled delivery due — waking it 2026/10/24 21:31:42 activator: no-host-proof.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: bot.x.internal idle 11s — hibernating 2026/10/24 21:31:42 activator: bot.x.internal hibernated 2026/10/24 21:31:42 activator: bot.x.internal has a scheduled delivery due — waking it 2026/10/24 21:31:42 activator: bot.x.internal has a scheduled delivery due — waking it 2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:42 activator: one.example.com: generation 2179 of v1: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:42 activator: bot.x.internal idle 11s — hibernating 2026/10/24 21:31:42 activator: bot.x.internal hibernated 2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: bot.x.internal has a scheduled delivery due — waking it 2026/10/24 21:31:42 activator: fenced.example.com: failed wake's generation 2181 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:42 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: c.example.com hibernated 2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:42 activator: f6-layout.example.com: failed wake's generation 2184 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process 2026/10/24 21:31:42 activator: f6-layout.example.com: failed wake's generation 2184 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:42 activator: fenced.example.com: failed wake's generation 2181 is proven stopped; lease given back 2026/10/24 21:31:42 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:42 activator: worker bot.x.internal respawned (converging onto a new build), now running v2 2026/10/24 21:31:42 activator: nested-stop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: nested-stop.example.com idle 2m0s — hibernating 2026/10/24 21:31:42 activator: nested-stop.example.com hibernated 2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: untimed.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: f6.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process 2026/10/24 21:31:42 activator: exited-flat.example.com: failed wake's generation 2191 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process 2026/10/24 21:31:42 activator: exited-flat.example.com: failed wake's generation 2191 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:42 activator: untimed.bab.oncarlos.com idle 16m0s — hibernating 2026/10/24 21:31:42 activator: untimed.bab.oncarlos.com hibernated 2026/10/24 21:31:42 activator: forked.example.com: failed wake's generation 2192 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:42 activator: stop fence.example: private backend detail 2026/10/24 21:31:42 activator: fence.example hibernated 2026/10/24 21:31:42 activator: forked.example.com: failed wake's generation 2192 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process 2026/10/24 21:31:42 activator: pending-f6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: pending-f6.example.com idle 2m0s — hibernating 2026/10/24 21:31:42 activator: stop pending-f6.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven 2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: ledgerpay.bde.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: forked.example.com: failed wake's generation 2192 is proven stopped; lease given back 2026/10/24 21:31:42 activator: stop pending-f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process 2026/10/24 21:31:42 activator: seeding seedpin.example.com: /tmp/TestADeclinedFlatSeedRunsThePinnedFile2559396634/001/versions/v1/app seed --db /tmp/TestADeclinedFlatSeedRunsThePinnedFile2559396634/002/seedpin.example.com.db 2026/10/24 21:31:42 activator: seeded seedpin.example.com 2026/10/24 21:31:42 activator: pending-f6.example.com hibernated 2026/10/24 21:31:42 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state 2026/10/24 21:31:42 activator: fence.example hibernated 2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:42 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:42 activator: h.example.com: carlos-instance-gen@h.example.com:8.service launched in 0s, healthy 0s later 2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:42 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it stays down until a release attempt or a wake starts it 2026/10/24 21:31:42 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:42 activator: h.example.com: carlos-instance-gen@h.example.com:9.service launched in 0s, healthy 0s later 2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:42 activator: f6-layout.example.com: failed wake's generation 2202 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process 2026/10/24 21:31:42 activator: f6-layout.example.com: failed wake's generation 2202 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:42 activator: bots.example.com idle 11s — hibernating 2026/10/24 21:31:42 activator: bots.example.com hibernated 2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:42 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:42 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:42 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: sidecar for bots.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:42 activator: fenced.example.com: failed wake's generation 2203 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:42 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:42 activator: bot.shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: swap.example.com hibernated 2026/10/24 21:31:42 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt 2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:42 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it recycles its captured build v1 2026/10/24 21:31:42 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:42 activator: swap.example.com hibernated 2026/10/24 21:31:42 activator: exited-foreign.example.com: failed wake's generation 2208 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process 2026/10/24 21:31:42 activator: exited-foreign.example.com: failed wake's generation 2208 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:42 activator: sidecar for bots.example.com was down — restarted (backoff now 4s) 2026/10/24 21:31:42 activator: fenced.example.com: failed wake's generation 2203 is proven stopped; lease given back 2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:42 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:42 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:42 activator: h.example.com: carlos-instance-gen@h.example.com:1.service launched in 0s, healthy 0s later 2026/10/24 21:31:42 activator: sidecar for bots.example.com was down — restarted (backoff now 8s) 2026/10/24 21:31:42 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:42 activator: h.example.com: carlos-instance-gen@h.example.com:6.service launched in 0s, healthy 0s later 2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s) 2026/10/24 21:31:42 activator: worker bot.x.internal respawned (exited), now running v1 2026/10/24 21:31:42 activator: stop sidecar fence.example: private backend detail 2026/10/24 21:31:42 activator: fence.example hibernated 2026/10/24 21:31:42 activator: sidecar for bots.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: sidecar for bots.example.com was down — restarted (backoff now 4s) 2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:42 activator: h.example.com: unit restarted outside the activator (invocation someone-else, ours inv7); stopping it 2026/10/24 21:31:42 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:42 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later 2026/10/24 21:31:42 activator: h.example.com: unit restarted outside the activator (invocation inv2, ours inv1); stopping it 2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: h.example.com: started carlos-instance-gen@h.example.com:2217.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:43 activator: one.example.com: generation 2217 of v1: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:43 activator: one.example.com: woke in 1ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 1ms, replicate 0s 2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:43 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:43 activator: one.example.com hibernated 2026/10/24 21:31:43 activator: bots.example.com idle 11s — hibernating 2026/10/24 21:31:43 activator: bots.example.com hibernated 2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:43 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:43 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later 2026/10/24 21:31:43 activator: h.example.com: processes remain after failed; not releasing 2026/10/24 21:31:43 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: one.example.com: live/one.example.com names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1 2026/10/24 21:31:43 activator: f6-layout.example.com: failed wake's generation 2221 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process 2026/10/24 21:31:43 activator: f6-layout.example.com: failed wake's generation 2221 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:43 activator: worker bot.x.internal serving v1 but live symlink is now v3 — respawning onto the new build 2026/10/24 21:31:43 activator: worker bot.x.internal respawned (converging onto a new build), now running v3 2026/10/24 21:31:43 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:43 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later 2026/10/24 21:31:43 activator: SIDECAR for botless.example.com failed to start: exec: live symlink gone — serving without it (the sweep retries) 2026/10/24 21:31:43 activator: botless.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:43 activator: bot.x.internal has a scheduled delivery due — waking it 2026/10/24 21:31:43 activator: SIDECAR restart for botless.example.com failed: exec: live symlink gone (next attempt in 2s) 2026/10/24 21:31:43 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build 2026/10/24 21:31:43 activator: one.example.com hibernated 2026/10/24 21:31:43 activator: waking.example: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: waking.example hibernated 2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com hibernated 2026/10/24 21:31:43 activator: swap.example.com: generation 2225 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child 2026/10/24 21:31:43 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: the running child keeps running 2026/10/24 21:31:43 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child 2026/10/24 21:31:43 activator: sidecar for botless.example.com was down — restarted (backoff now 4s) 2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:43 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: one.example.com is no longer hibernate-managed — releasing 2026/10/24 21:31:43 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child 2026/10/24 21:31:43 activator: one.example.com hibernated 2026/10/24 21:31:43 activator: f6.example.com: failed wake's generation 0 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: reborn.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: reborn.example.com idle 11s — hibernating 2026/10/24 21:31:43 activator: reborn.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it 2026/10/24 21:31:43 activator: reborn.example.com hibernated 2026/10/24 21:31:43 activator: swap.example.com hibernated 2026/10/24 21:31:43 activator: final checkpoint fence.example: private backend detail 2026/10/24 21:31:43 activator: fence.example hibernated 2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: reborn.example.com has a scheduled delivery due — waking it 2026/10/24 21:31:43 activator: reborn.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: sidecar staged for awake swap.example.com — adopting 2026/10/24 21:31:43 activator: sidecar for swap.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: sidecar staged for awake swap.example.com — adopting 2026/10/24 21:31:43 activator: sidecar for swap.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:43 activator: shop.example.com: generation 4's child is not proven empty: kill gen: timeout 2026/10/24 21:31:43 activator: old.shop.example.com: route refreshed: db "/data/instances/old/old.shop.example.com.db" -> "" socket "/run/carlos/instances/old/app.sock" -> "" 2026/10/24 21:31:43 activator: wake old.shop.example.com: old.shop.example.com: the route changed before the wake: the activator no longer manages it 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: h.example.com: kill carlos-instance-gen@h.example.com:2241.service after a failed start: stop timed out 2026/10/24 21:31:43 activator: one.example.com: wake failed and its generation is not proven stopped: start one.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2241.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out 2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:43 activator: LEASE LOST for bot.x.internal (lease renewal failed) — standing down 2026/10/24 21:31:43 activator: bot.x.internal hibernated 2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:43 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:43 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later 2026/10/24 21:31:43 activator: f6-layout.example.com: failed wake's generation 2242 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process 2026/10/24 21:31:43 activator: f6-layout.example.com: failed wake's generation 2242 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: sidecar staged for awake swap.example.com — adopting 2026/10/24 21:31:43 activator: sidecar for swap.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com hibernated 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries) 2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:43 activator: one.example.com: generation 2249 of v1: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:43 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated 2026/10/24 21:31:43 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s) 2026/10/24 21:31:43 activator: one.example.com hibernated 2026/10/24 21:31:43 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1 2026/10/24 21:31:43 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 8s) 2026/10/24 21:31:43 activator: sidecar for swap.example.com was down — restarted (backoff now 16s) 2026/10/24 21:31:43 activator: one.example.com: generation 2252 of v2: record 0s, start 20ms, healthy 0s later 2026/10/24 21:31:43 activator: one.example.com: woke in 20ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 20ms, replicate 0s 2026/10/24 21:31:43 activator: fence.example hibernated 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: proven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: proven.example.com idle 2m0s — hibernating 2026/10/24 21:31:43 activator: proven.example.com hibernated 2026/10/24 21:31:43 activator: swap.example.com hibernated 2026/10/24 21:31:43 activator: undelegated.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: sidecar staged for awake swap.example.com — adopting 2026/10/24 21:31:43 activator: sidecar for swap.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:43 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:43 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later 2026/10/24 21:31:43 activator: undelegated.example.com idle 2m0s — hibernating 2026/10/24 21:31:43 activator: undelegated.example.com: stopped with no delegated cgroup; its leader's exit is the only proof there is 2026/10/24 21:31:43 activator: undelegated.example.com hibernated 2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated 2026/10/24 21:31:43 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is 2026/10/24 21:31:43 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:43 activator: h.example.com: carlos-instance-gen@h.example.com:6.service launched in 0s, healthy 0s later 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: one.example.com: generation 2262 of v3: record 0s, start 20ms, healthy 0s later 2026/10/24 21:31:43 activator: one.example.com: woke in 20ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 20ms, replicate 0s 2026/10/24 21:31:43 activator: sidecar staged for awake swap.example.com — adopting 2026/10/24 21:31:43 activator: sidecar for swap.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated 2026/10/24 21:31:43 activator: shop.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process 2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated 2026/10/24 21:31:43 activator: SIDECAR for swap.example.com is not proven gone for its replacement; tracking it (the sweep re-proves it) 2026/10/24 21:31:43 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s) 2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated 2026/10/24 21:31:43 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com hibernated 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:43 activator: swap.example.com hibernated 2026/10/24 21:31:43 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2486839030/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2486839030/003/not-a-dir/zdd: not a directory; the sweep retries 2026/10/24 21:31:43 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2486839030/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2486839030/003/not-a-dir/zdd: not a directory; the sweep retries 2026/10/24 21:31:43 activator: instance bots.example.com exited unexpectedly — recycling 2026/10/24 21:31:43 activator: bots.example.com hibernated 2026/10/24 21:31:43 activator: swap.example.com: a discarded SIDECAR is not proven gone (activator: stop unproven; processes remain); keeping it tracked 2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:43 activator: h.example.com: carlos-instance-gen@h.example.com:9.service launched in 0s, healthy 0s later 2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated 2026/10/24 21:31:44 activator: swap.example.com hibernated 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com hibernated 2026/10/24 21:31:44 activator: swap.example.com: generation 2276 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: sidecar staged for awake swap.example.com — adopting 2026/10/24 21:31:44 activator: sidecar for swap.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:44 activator: late.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: deaf.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: late.example.com idle 11s — hibernating 2026/10/24 21:31:44 activator: fence.example hibernated 2026/10/24 21:31:44 activator: late.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it 2026/10/24 21:31:44 activator: late.example.com hibernated 2026/10/24 21:31:44 activator: deaf.example.com idle 11s — hibernating 2026/10/24 21:31:44 activator: next-due probe deaf.example.com: dial unix: no such file — recording none (it sleeps until a request) 2026/10/24 21:31:44 activator: deaf.example.com hibernated 2026/10/24 21:31:44 activator: due.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: own.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:44 activator: due.example.com idle 11s — hibernating 2026/10/24 21:31:44 activator: due.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it 2026/10/24 21:31:44 activator: due.example.com hibernated 2026/10/24 21:31:44 activator: late.example.com has a scheduled delivery due — waking it 2026/10/24 21:31:44 activator: late.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: own.example.com idle 1m0s — hibernating 2026/10/24 21:31:44 activator: own.example.com hibernated 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: sidecar staged for awake swap.example.com — adopting 2026/10/24 21:31:44 activator: sidecar for swap.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:44 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s) 2026/10/24 21:31:44 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later 2026/10/24 21:31:44 activator: bot.x.internal hit its max_run of 1m0s — stopping 2026/10/24 21:31:44 activator: bot.x.internal hibernated 2026/10/24 21:31:44 activator: swap.example.com hibernated 2026/10/24 21:31:44 activator: due.example.com has a scheduled delivery due — waking it 2026/10/24 21:31:44 activator: due.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:44 activator: bot.x.internal hibernated 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build 2026/10/24 21:31:44 activator: one.example.com hibernated 2026/10/24 21:31:44 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished4121104141/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished4121104141/001/zdd: not a directory; the sweep retries 2026/10/24 21:31:44 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished4121104141/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished4121104141/001/zdd: not a directory; the sweep retries 2026/10/24 21:31:44 activator: due.example.com idle 11s — hibernating 2026/10/24 21:31:44 activator: due.example.com hibernated 2026/10/24 21:31:44 activator: one.example.com idle 11s — hibernating 2026/10/24 21:31:44 activator: one.example.com hibernated 2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor3121912923/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor3121912923/005/not-a-dir/zdd: not a directory; the sweep retries 2026/10/24 21:31:44 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor3121912923/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor3121912923/005/not-a-dir/zdd: not a directory; the sweep retries 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com hibernated 2026/10/24 21:31:44 activator: swap.example.com: generation 2303's child is not proven empty: the child still holds a process 2026/10/24 21:31:44 activator: swap.example.com: failed wake's generation 2303 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: one.example.com hibernated 2026/10/24 21:31:44 activator: swap.example.com: failed wake's generation 2303 is proven stopped; lease given back 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com hibernated 2026/10/24 21:31:44 activator: swap.example.com: generation 2307 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com hibernated 2026/10/24 21:31:44 activator: swap.example.com: failed wake's generation 2309 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:44 activator: faroff.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: failed wake's generation 2309 is proven stopped; lease given back 2026/10/24 21:31:44 activator: faroff.bab.oncarlos.com idle 16m0s — hibernating 2026/10/24 21:31:44 activator: faroff.bab.oncarlos.com hibernated 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com hibernated 2026/10/24 21:31:44 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt 2026/10/24 21:31:44 activator: stop fence.example: activator: stop evidence unavailable 2026/10/24 21:31:44 activator: fence.example hibernated 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com hibernated 2026/10/24 21:31:44 activator: LEASE HELD for swap.example.com by "box-b" — a previous incarnation hasn't released; refusing to wake (retry after 1m0s) 2026/10/24 21:31:44 activator: swap.example.com: hold's pinned wake refused or abandoned: instance lease held by "box-b" (retry in 1m0s) 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: flip.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: one.example.com idle 1m0s — hibernating 2026/10/24 21:31:44 activator: unproven.example.com: generation 2321's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:44 activator: unproven.example.com: failed wake's generation 2321 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:44 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out 2026/10/24 21:31:44 activator: flip.example.com is no longer hibernate-managed — releasing 2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: flip.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it 2026/10/24 21:31:44 activator: flip.example.com hibernated 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: unproven.example.com: generation 2321's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:44 activator: mid.example.com has a scheduled delivery due — waking it 2026/10/24 21:31:44 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build 2026/10/24 21:31:44 activator: one.example.com hibernated 2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake543479007/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake543479007/005/not-a-dir/zdd: not a directory; the sweep retries 2026/10/24 21:31:44 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake543479007/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake543479007/005/not-a-dir/zdd: not a directory; the sweep retries 2026/10/24 21:31:44 activator: f6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: one.example.com hibernated 2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: unproven.example.com: failed wake's generation 2321 is proven stopped; lease given back 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: LEASE LOST for one.example.com (lease renewal failed) — standing down 2026/10/24 21:31:44 activator: one.example.com hibernated 2026/10/24 21:31:44 activator: f6.example.com idle 2m0s — hibernating 2026/10/24 21:31:44 activator: stop f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process 2026/10/24 21:31:44 activator: stop f6.example.com: its host cgroup is not proven empty; keeping the lease until it is 2026/10/24 21:31:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:44 activator: mid.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: mid.example.com is no longer hibernate-managed — releasing (wake finished first) 2026/10/24 21:31:44 activator: mid.example.com hibernated 2026/10/24 21:31:44 activator: bot.x.internal idle 11s — hibernating 2026/10/24 21:31:44 activator: bot.x.internal hibernated 2026/10/24 21:31:44 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: unproven.example.com hibernated 2026/10/24 21:31:44 activator: swap.example.com idle 2h0m0s — hibernating 2026/10/24 21:31:44 activator: swap.example.com hibernated 2026/10/24 21:31:44 activator: f6.example.com hibernated 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: two.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is 2026/10/24 21:31:44 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build 2026/10/24 21:31:44 activator: one.example.com hibernated 2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: barrier.example.com hibernated 2026/10/24 21:31:45 activator: two.example.com serving v1 but live symlink is now v2 — cycling onto the new build 2026/10/24 21:31:45 activator: two.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build 2026/10/24 21:31:45 activator: one.example.com hibernated 2026/10/24 21:31:45 activator: f6.example.com: failed wake's generation 4 ran with no delegated cgroup; its leader's exit is the only proof there is 2026/10/24 21:31:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: the backend ran a different generation from the hold's choice: generation 2350 of v9 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_2061185657/004/versions/v2/app"; the hold chose v2 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_2061185657/004/versions/v2/app" 2026/10/24 21:31:45 activator: one.example.com idle 11s — hibernating 2026/10/24 21:31:45 activator: one.example.com hibernated 2026/10/24 21:31:45 activator: final checkpoint fence.example: activator: stop evidence unavailable 2026/10/24 21:31:45 activator: fence.example hibernated 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: restore swap.example.com: replica unreadable 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: socket dir: permission denied 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: sidecar staged for awake swap.example.com — adopting 2026/10/24 21:31:45 activator: sidecar for swap.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed 2026/10/24 21:31:45 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1542582095/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1542582095/003/not-a-dir/zdd: not a directory; the sweep retries 2026/10/24 21:31:45 activator: x.example: hold's pinned wake refused or abandoned: x.example: refusing to wake: staged build not published before its exec 2026/10/24 21:31:45 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1542582095/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1542582095/003/not-a-dir/zdd: not a directory; the sweep retries 2026/10/24 21:31:45 activator: sidecar for swap.example.com was down — restarted (backoff now 4s) 2026/10/24 21:31:45 activator: tickheld.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: unproven.example.com: generation 2362's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:45 activator: unproven.example.com: failed wake's generation 2362 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:45 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: h: start unit: instance control: generation termination unproven: the unit exited before it was observed 2026/10/24 21:31:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries) 2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: unproven.example.com: generation 2362's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:45 activator: barrier.example.com idle 2h0m0s — hibernating 2026/10/24 21:31:45 activator: barrier.example.com hibernated 2026/10/24 21:31:45 activator: tickheld.bab.oncarlos.com serving v1 but live symlink is now v2 — cycling onto the new build 2026/10/24 21:31:45 activator: tickheld.bab.oncarlos.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: bot.x.internal idle 31s — hibernating 2026/10/24 21:31:45 activator: bot.x.internal hibernated 2026/10/24 21:31:45 activator: one.example.com idle 11s — hibernating 2026/10/24 21:31:45 activator: one.example.com hibernated 2026/10/24 21:31:45 activator: unproven.example.com: failed wake's generation 2362 is proven stopped; lease given back 2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: sidecar for swap.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:45 activator: bot.x.internal idle 1m31s — hibernating 2026/10/24 21:31:45 activator: bot.x.internal hibernated 2026/10/24 21:31:45 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: unproven.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: sidecar staged for awake swap.example.com — adopting 2026/10/24 21:31:45 activator: sidecar for swap.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: one.example.com idle 5m1s — hibernating 2026/10/24 21:31:45 activator: one.example.com hibernated 2026/10/24 21:31:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process 2026/10/24 21:31:45 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is 2026/10/24 21:31:45 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process 2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: fence.example hibernated 2026/10/24 21:31:45 activator: barrier.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: staging: staged digest mismatch: have be0041e1a437dbea5b67dfe58bad04a0fc30d5036d469429f3534f39ec53d8b5, want c3bae8fb3a8bf395783d7f7f6ffcd4aadc797c95ce24b7e8e2a98af6b63d8f6d 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: start g: activator: generation start failed and its termination is unproven 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed 2026/10/24 21:31:45 activator: bot.x.internal has a scheduled delivery due — waking it 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: sidecar staged for awake swap.example.com — adopting 2026/10/24 21:31:45 activator: sidecar for swap.example.com was down — restarted (backoff now 2s) 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:45 activator: fence.example hibernated 2026/10/24 21:31:45 activator: bot.x.internal idle 11s — hibernating 2026/10/24 21:31:45 activator: bot.x.internal hibernated 2026/10/24 21:31:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries) 2026/10/24 21:31:45 activator: fence.example hibernated 2026/10/24 21:31:45 activator: one.example.com idle 11s — hibernating 2026/10/24 21:31:45 activator: one.example.com hibernated 2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: unproven.example.com: failed wake's generation 2398 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:45 activator: barrier.example.com idle 2h0m0s — hibernating 2026/10/24 21:31:45 activator: barrier.example.com hibernated 2026/10/24 21:31:45 activator: unproven.example.com: retry stop of failed wake's generation 2398: cgroup.kill: the child is not empty 2026/10/24 21:31:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: bot.x.internal idle 1m1s — hibernating 2026/10/24 21:31:45 activator: bot.x.internal hibernated 2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: unproven.example.com: generation 2398's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:45 activator: unproven.example.com: failed wake's generation 2398 is proven stopped; lease given back 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: unproven.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: a stop-evidence operation holds the host 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: generation 2411 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s 2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down 2026/10/24 21:31:45 activator: barrier.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries) 2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: swap.example.com hibernated 2026/10/24 21:31:45 activator: swap.example.com: generation 2418 of v2: record 0s, start 0s, healthy 0s later 2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: unproven.example.com: generation 2420's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:45 activator: unproven.example.com: failed wake's generation 2420 is not proven stopped; keeping the lease until it is 2026/10/24 21:31:45 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out 2026/10/24 21:31:45 activator: unproven.example.com: generation 2420's child is not proven empty: the generation's child still holds a process 2026/10/24 21:31:45 activator: unproven.example.com: failed wake's generation 2420 is proven stopped; lease given back 2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down 2026/10/24 21:31:45 activator: barrier.example.com hibernated 2026/10/24 21:31:45 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: unproven.example.com hibernated 2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: barrier.example.com hibernated 2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: barrier.example.com hibernated 2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:46 activator: x.example recovery probe healthy; held until the source opens 2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:46 activator: barrier.example.com hibernated 2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:46 activator: barrier.example.com hibernated 2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:46 activator: barrier.example.com is no longer hibernate-managed — releasing 2026/10/24 21:31:46 activator: barrier.example.com hibernated 2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:46 activator: barrier.example.com is no longer hibernate-managed — releasing 2026/10/24 21:31:46 activator: barrier.example.com hibernated 2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:46 activator: barrier.example.com hibernated 2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s 2026/10/24 21:31:46 activator: barrier.example.com hibernated FAIL FAIL github.com/carlosframework/platform/internal/activator 57.972s ok github.com/carlosframework/platform/internal/agentsock 0.023s ok github.com/carlosframework/platform/internal/alerting 0.003s ok github.com/carlosframework/platform/internal/analytics 0.034s ok github.com/carlosframework/platform/internal/appconfig 3.034s ok github.com/carlosframework/platform/internal/apperr 0.164s ok github.com/carlosframework/platform/internal/applog 0.291s ok github.com/carlosframework/platform/internal/awsinv 0.009s ok github.com/carlosframework/platform/internal/boxchoice 0.008s ok github.com/carlosframework/platform/internal/boxid 0.919s ? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files] ? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files] ok github.com/carlosframework/platform/internal/boxvolumes 0.008s ok github.com/carlosframework/platform/internal/caps 0.019s ok github.com/carlosframework/platform/internal/certwarm 0.004s ok github.com/carlosframework/platform/internal/cgroup 0.728s ok github.com/carlosframework/platform/internal/changelog 0.637s ok github.com/carlosframework/platform/internal/claimactivation 1.167s ok github.com/carlosframework/platform/internal/clamd 0.055s ? github.com/carlosframework/platform/internal/cliflag [no test files] ? github.com/carlosframework/platform/internal/controlname [no test files] ? github.com/carlosframework/platform/internal/controlsock [no test files] ok github.com/carlosframework/platform/internal/dbdoor 8.035s ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.017s ok github.com/carlosframework/platform/internal/dbquery 0.010s ok github.com/carlosframework/platform/internal/dbreads 2.096s ? github.com/carlosframework/platform/internal/dbreads/dbreadstest [no test files] ok github.com/carlosframework/platform/internal/dbverify 0.006s ok github.com/carlosframework/platform/internal/dnsrec 0.002s ok github.com/carlosframework/platform/internal/docsite 0.019s ok github.com/carlosframework/platform/internal/domains 2.872s ok github.com/carlosframework/platform/internal/doorkey 0.007s ok github.com/carlosframework/platform/internal/economics 0.005s ok github.com/carlosframework/platform/internal/edge 45.260s ? github.com/carlosframework/platform/internal/edgeproof [no test files] ok github.com/carlosframework/platform/internal/edgesfile 0.010s ok github.com/carlosframework/platform/internal/fleetapi 0.209s ok github.com/carlosframework/platform/internal/gate 1.822s ok github.com/carlosframework/platform/internal/genselect 0.033s ok github.com/carlosframework/platform/internal/geo 0.021s ok github.com/carlosframework/platform/internal/geoip 0.042s ok github.com/carlosframework/platform/internal/historyreadback 0.285s ok github.com/carlosframework/platform/internal/host 140.016s ok github.com/carlosframework/platform/internal/hostactors 6.608s ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.008s ok github.com/carlosframework/platform/internal/hostbox 3.145s ok github.com/carlosframework/platform/internal/hostclaim 0.759s ok github.com/carlosframework/platform/internal/hostenv 0.009s ok github.com/carlosframework/platform/internal/hostevidence 0.006s ok github.com/carlosframework/platform/internal/hostexec 5.383s ok github.com/carlosframework/platform/internal/hostfence 0.046s ok github.com/carlosframework/platform/internal/hostfiles 10.229s ok github.com/carlosframework/platform/internal/hostfloor 0.017s ok github.com/carlosframework/platform/internal/hostguard 1.077s ok github.com/carlosframework/platform/internal/hostops 2.217s ? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files] ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.007s ok github.com/carlosframework/platform/internal/hostquarantine 0.004s ok github.com/carlosframework/platform/internal/hostrelocate 0.584s ok github.com/carlosframework/platform/internal/hostresolve 0.177s ok github.com/carlosframework/platform/internal/hostwrite 0.007s ok github.com/carlosframework/platform/internal/infraview 0.005s ok github.com/carlosframework/platform/internal/instancectl 5.250s ok github.com/carlosframework/platform/internal/instanceexec 2.795s ok github.com/carlosframework/platform/internal/instancefs 0.011s ok github.com/carlosframework/platform/internal/instancekey 0.007s ok github.com/carlosframework/platform/internal/instanceupdates 0.008s ok github.com/carlosframework/platform/internal/layout 1.968s ok github.com/carlosframework/platform/internal/lbhealth 0.005s ok github.com/carlosframework/platform/internal/lease 10.096s ok github.com/carlosframework/platform/internal/lease/leasetest 0.537s ok github.com/carlosframework/platform/internal/ledger 0.019s ok github.com/carlosframework/platform/internal/mail 0.181s ok github.com/carlosframework/platform/internal/maintenance 0.082s ok github.com/carlosframework/platform/internal/maintenanceexec 5.566s ok github.com/carlosframework/platform/internal/maintenanceguard 0.082s ok github.com/carlosframework/platform/internal/metrics 0.006s ok github.com/carlosframework/platform/internal/moves 21.355s ok github.com/carlosframework/platform/internal/moves/archive 1.046s ok github.com/carlosframework/platform/internal/objstore 0.276s ok github.com/carlosframework/platform/internal/ops 0.179s ok github.com/carlosframework/platform/internal/park 0.230s ok github.com/carlosframework/platform/internal/plans 0.006s ok github.com/carlosframework/platform/internal/privatefile 0.005s ok github.com/carlosframework/platform/internal/provision 14.123s ok github.com/carlosframework/platform/internal/publicrelease 0.016s ? github.com/carlosframework/platform/internal/regionname [no test files] ok github.com/carlosframework/platform/internal/registry 5.646s ok github.com/carlosframework/platform/internal/relay 7.223s ok github.com/carlosframework/platform/internal/release 24.543s ok github.com/carlosframework/platform/internal/rollout 3.703s ok github.com/carlosframework/platform/internal/rolloutwords 0.004s ok github.com/carlosframework/platform/internal/rootaws 0.009s ok github.com/carlosframework/platform/internal/rootparticipation 0.060s ok github.com/carlosframework/platform/internal/rootstate 0.030s ok github.com/carlosframework/platform/internal/runs 0.538s ok github.com/carlosframework/platform/internal/s3compat 0.005s ok github.com/carlosframework/platform/internal/safefile 0.006s ok github.com/carlosframework/platform/internal/schedule 2.760s ok github.com/carlosframework/platform/internal/seeds 0.014s ok github.com/carlosframework/platform/internal/sim/engines 2.714s ok github.com/carlosframework/platform/internal/sim/hostlife 9.984s ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.097s ok github.com/carlosframework/platform/internal/status 0.074s ok github.com/carlosframework/platform/internal/steering 0.006s ok github.com/carlosframework/platform/internal/stopevidence 0.258s ok github.com/carlosframework/platform/internal/svcrole 0.002s ok github.com/carlosframework/platform/internal/telemetry 0.004s ok github.com/carlosframework/platform/internal/termfence 0.005s ok github.com/carlosframework/platform/internal/tunnelclient 0.003s ok github.com/carlosframework/platform/internal/unixdial 0.003s ok github.com/carlosframework/platform/internal/updatewords 0.004s ok github.com/carlosframework/platform/internal/usage 0.004s ok github.com/carlosframework/platform/internal/version 0.002s ok github.com/carlosframework/platform/internal/vet 1.692s FAIL === internal_console@0 (79 s) ok github.com/carlosframework/platform/internal/console 77.341s === internal_console@1 (43 s) ok github.com/carlosframework/platform/internal/console 41.397s === internal_console@2 (41 s) ok github.com/carlosframework/platform/internal/console 39.236s === internal_console@3 (41 s) ok github.com/carlosframework/platform/internal/console 38.764s === internal_console@4 (41 s) ok github.com/carlosframework/platform/internal/console 39.632s === internal_console@5 (41 s) ok github.com/carlosframework/platform/internal/console 38.693s === internal_console@6 (36 s) ok github.com/carlosframework/platform/internal/console 33.923s === internal_console@7 (45 s) ok github.com/carlosframework/platform/internal/console 42.988s === root@0 (109 s) ok github.com/carlosframework/platform 106.914s === root@1 (84 s) ok github.com/carlosframework/platform 82.232s === root@2 (91 s) ok github.com/carlosframework/platform 88.947s FAILED: rest make: *** [Makefile:19: test] Error 1 amadan: step gate failed: exit status 2 -
cross not run
No output.
-
minio-contract not run
No output.
Whole run log
=== gate ===
.amadan/test-gate.sh
=== launchctx (0 s)
ok amadan.net/carlos/platform.git/launchctx 0.017s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (1 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (143 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.671s
ok github.com/carlosframework/platform/cmd/moveprobe 11.985s
ok github.com/carlosframework/platform/cmd/zddprobe 0.746s
2026/10/24 21:30:48 activator: hook.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: nohook.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: ladder0.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: ladder1.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: ladder2.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: ladder3.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: fresh.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: restored.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: noseed.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: badseed.example.com: seed for badseed.example.com: activator: seed unproven; processes remain: processes remain after a successful seed: still running; leaving the lease to its TTL
2026/10/24 21:30:48 activator: unproven.example.com: unproven.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone; leaving the lease to its TTL
2026/10/24 21:30:48 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/24 21:30:48 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3396839126/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3396839126/004/next-wake.example.com.db
2026/10/24 21:30:48 activator: next-wake.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3396839126/004/next-wake.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 21:30:48 activator: next-wake.example.com: seeding next-wake.example.com: exit status 1: activator: seed unproven; processes remain (the seed's processes remain after stopping it; its database was left in place: cgroup: strict containment unsupported); leaving the lease to its TTL
2026/10/24 21:30:48 activator: next-wake.example.com: next-wake.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup: strict containment unsupported; leaving the lease to its TTL
2026/10/24 21:30:48 activator: next-wake.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 21:30:48 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/24 21:30:48 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3396839126/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3396839126/004/next-wake.example.com.db
2026/10/24 21:30:48 activator: seeded next-wake.example.com
2026/10/24 21:30:48 activator: next-wake.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: instance one.example.com exited unexpectedly — recycling
2026/10/24 21:30:48 activator: one.example.com hibernated
2026/10/24 21:30:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: LEASE HELD for held.example.com by "old-incarnation" — a previous incarnation hasn't released; refusing to wake (retry after 7s)
2026/10/24 21:30:48 activator: hib.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: tick.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: visitor.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: asleep.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:48 activator: fenced.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:49 activator: imminent.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:49 activator: overdue.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:49 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:49 activator: two.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:49 activator: two.example.com hibernated
2026/10/24 21:30:49 activator: one.example.com hibernated
2026/10/24 21:30:49 activator: b.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:49 activator: b.example.com hibernated
2026/10/24 21:30:49 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:30:49 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:30:49 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:30:49 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:30:49 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:30:49 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:30:49 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:30:49 activator: worker bot.x.internal respawned (exited), now running
2026/10/24 21:30:49 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:30:49 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:30:49 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 21:30:49 activator: bot.x.internal hibernated
2026/10/24 21:30:50 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendStartMergesConfigEnvUnderOwnEnv4120487044/002/merge-precedence.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/24 21:30:50 activator: purged 1 cached response(s) for deploy.example.com on spawn
2026/10/24 21:30:50 activator: restored restored.example.com from s3://test-bucket/restored.example.com
2026/10/24 21:30:50 activator: damaged.example.com restored CORRUPT from s3://test-bucket/damaged.example.com: integrity_check: file is not a database (26) — quarantined at /tmp/TestExecBackendRestoreCorruptReplicaFailsAndQuarantines3017527669/002/damaged.example.com.db.corrupt, instance NOT started
2026/10/24 21:30:50 activator: restored damaged.example.com from s3://test-bucket/damaged.example.com
2026/10/24 21:30:50 activator: no replica for brand-new.example.com in s3://test-bucket/brand-new.example.com — booting fresh
2026/10/24 21:30:50 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1385449360/002/seeded.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/24 21:30:50 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1385449360/001/live/seeded.example.com seed --fixtures --db /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1385449360/002/seeded.example.com.db
2026/10/24 21:30:50 activator: seeded seeded.example.com
2026/10/24 21:30:50 activator: seeding custom-db.example.com: /tmp/TestExecBackendSeedHonoursRouteDBPath2083819304/001/live/custom-db.example.com seed --db /tmp/TestExecBackendSeedHonoursRouteDBPath2083819304/004/elsewhere.db
2026/10/24 21:30:50 activator: seeded custom-db.example.com
2026/10/24 21:30:50 activator: seeding badseed.example.com: /tmp/TestExecBackendSeedNonZeroExitFails723266046/001/live/badseed.example.com seed --db /tmp/TestExecBackendSeedNonZeroExitFails723266046/002/badseed.example.com.db
schema is corrupt
2026/10/24 21:30:50 activator: badseed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 21:30:50 activator: seeding hangs.example.com: /tmp/TestExecBackendSeedIsTimeBounded1721172399/001/live/hangs.example.com seed --db /tmp/TestExecBackendSeedIsTimeBounded1721172399/002/hangs.example.com.db
2026/10/24 21:30:50 activator: hangs.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 21:30:50 activator: seeding crashed.example.com: /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB2777798474/001/live/crashed.example.com seed --db /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB2777798474/002/crashed.example.com.db
died mid-schema
2026/10/24 21:30:50 activator: crashed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 21:30:50 activator: seeding existing-db.example.com: /tmp/TestExecBackendSeedFailureKeepsPreexistingDB4271391541/001/live/existing-db.example.com seed --db /tmp/TestExecBackendSeedFailureKeepsPreexistingDB4271391541/002/existing-db.example.com.db
nope
2026/10/24 21:30:50 activator: existing-db.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 21:30:50 activator: seeding leaky-seed.example.com: /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild3176716889/001/live/leaky-seed.example.com seed --db /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild3176716889/002/leaky-seed.example.com.db
2026/10/24 21:30:50 activator: seeded leaky-seed.example.com (it exited 0 but left a child holding its output pipe open for more than 200ms)
2026/10/24 21:30:51 activator: a.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:30:51 activator: b.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:51 activator: sidecar for a.example.com was down — restarted (backoff now 2s)
2026/10/24 21:30:51 activator: h.example.com: route refreshed: db "/tmp/h.example.com.db" -> "/data/instances/h.example.com/h.example.com.db" socket "/tmp/nope.sock" -> "/run/carlos/instances/h.example.com/app.sock"
2026/10/24 21:30:51 activator: h.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:51 activator: stale.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:30:51 activator: stop stale.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/24 21:30:51 activator: stale.example.com hibernated
2026/10/24 21:30:51 activator: stale.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:30:51 activator: leftover.example.com: the host cgroup holds processes not proven gone: a leftover still populates the host cgroup; leaving the lease to its TTL
2026/10/24 21:30:51 activator: leftover.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:51 activator: old-kernel.example.com: this kernel has no cgroup.kill, so the host cgroup cannot be killed; its leader's exit is the only proof there is
2026/10/24 21:30:51 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/24 21:30:51 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/24 21:30:51 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/24 21:30:51 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/24 21:30:51 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/24 21:30:51 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/24 21:30:51 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/24 21:30:51 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/24 21:30:51 activator: retry.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 21:30:52 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/24 21:30:52 activator: retry.example.com: strict group not closed: cgroup: strict containment refused
2026/10/24 21:30:52 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/24 21:30:52 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 21:30:52 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 21:30:52 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 21:30:52 activator: stuck.example.com: strict group still not cleaned up after 2 retries; spawns of this host stay refused until the agent restarts
2026/10/24 21:30:52 activator: owned.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 21:30:53 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/24 21:30:53 activator: owned.example.com: strict group not closed: cgroup: strict containment refused
2026/10/24 21:30:53 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/24 21:30:54 activator: race.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 21:30:54 activator: transition.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 21:30:54 activator: transition2.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 21:30:54 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/24 21:30:54 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/24 21:30:54 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:54 activator: kass.example.com hibernated
2026/10/24 21:30:54 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:54 activator: kass.example.com idle 11s — hibernating
2026/10/24 21:30:54 activator: kass.example.com hibernated
2026/10/24 21:30:54 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:54 activator: one.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/24 21:30:54 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:30:54 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:30:54 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:30:54 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:30:55 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:30:55 activator: bot.x.internal: v2 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:30:55 activator: worker bot.x.internal NOT respawned (attempt (release)): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/24 21:30:55 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 21:30:55 activator: link.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:55 activator: link.example.com hibernated
2026/10/24 21:30:55 activator: link.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:55 activator: plain-link.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:55 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:55 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:59 activator: tenant.test: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:30:59 activator: tenant.test not started: the executable it hashed (/tmp/rl4078734386/app) is build "v1", but the start names build "v2" (the live link moved between the two reads); the next wake reads it again
2026/10/24 21:30:59 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:30:59 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 21:30:59 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 21:30:59 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:59 activator: replica h.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/24 21:30:59 activator: replica h.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/24 21:30:59 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 21:30:59 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:59 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:59 activator: why.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:59 activator: why.example.com hibernated
2026/10/24 21:30:59 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:59 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:59 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:59 activator: c.example.com hibernated
2026/10/24 21:30:59 activator: h.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:59 activator: h.example.com hibernated
2026/10/24 21:30:59 activator: h.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:30:59 activator: h.example.com: run run_a is not proven stopped: cgroup.events: permission denied
2026/10/24 21:30:59 activator: stop h.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 21:30:59 activator: h.example.com hibernated
2026/10/24 21:30:59 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 21:30:59 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 21:30:59 activator: seeding unproven-seed.example.com: /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga2534324464/002/live/unproven-seed.example.com seed --db /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga2534324464/003/unproven-seed.example.com.db
2026/10/24 21:30:59 activator: unproven-seed.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga2534324464/003/unproven-seed.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 21:30:59 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm107864036/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm107864036/003/obligation.example.com.db
2026/10/24 21:30:59 activator: obligation.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm107864036/003/obligation.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 21:30:59 activator: obligation.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 21:30:59 activator: no replica for obligation.example.com in s3://test-bucket/obligation.example.com — booting fresh
2026/10/24 21:30:59 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm107864036/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm107864036/003/obligation.example.com.db
2026/10/24 21:30:59 activator: seeded obligation.example.com
2026/10/24 21:30:59 activator: seeding helper-seed.example.com: /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt3265900727/002/live/helper-seed.example.com seed --db /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt3265900727/003/helper-seed.example.com.db
2026/10/24 21:30:59 activator: seeded helper-seed.example.com
2026/10/24 21:30:59 activator: seeding helper-unproven.example.com: /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven2271479888/002/live/helper-unproven.example.com seed --db /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven2271479888/003/helper-unproven.example.com.db
2026/10/24 21:30:59 activator: helper-unproven.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven2271479888/003/helper-unproven.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 21:30:59 activator: seeding write-ahead.example.com: /tmp/TestASeedRecordsItsObligationBeforeItRuns811764108/002/live/write-ahead.example.com seed --db /tmp/TestASeedRecordsItsObligationBeforeItRuns811764108/003/write-ahead.example.com.db
2026/10/24 21:30:59 activator: seeded write-ahead.example.com
2026/10/24 21:30:59 activator: write-ahead.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 21:30:59 activator: no replica for write-ahead.example.com in s3://test-bucket/write-ahead.example.com — booting fresh
2026/10/24 21:30:59 activator: seeding no-room.example.com: /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing2815485007/002/live/no-room.example.com seed --db /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing2815485007/003/no-room.example.com.db
2026/10/24 21:30:59 activator: seeding stuck-wal.example.com: /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation3119787773/002/live/stuck-wal.example.com seed --db /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation3119787773/003/stuck-wal.example.com.db
2026/10/24 21:30:59 activator: seeding preexisting.example.com: /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2242273890/002/live/preexisting.example.com seed --db /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2242273890/003/preexisting.example.com.db
2026/10/24 21:30:59 activator: preexisting.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2242273890/003/preexisting.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 21:30:59 activator: preexisting.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/24 21:30:59 activator: old-kernel-seed.example.com: this kernel has no cgroup.kill; the seed's leader's exit is the only proof there is
2026/10/24 21:30:59 activator: seeding failed-role-seed.example.com: /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB3339933585/002/live/failed-role-seed.example.com seed --db /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB3339933585/003/failed-role-seed.example.com.db
2026/10/24 21:30:59 activator: seeding discharge-fails.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai2120713116/002/live/discharge-fails.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai2120713116/003/discharge-fails.example.com.db
2026/10/24 21:30:59 activator: discharge-fails.example.com: the seed's obligation could not be discharged and its processes are not proven gone; /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai2120713116/003/discharge-fails.example.com.db.seed-unproven stays
2026/10/24 21:30:59 activator: seeding discharge-fails-proven.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur942013482/002/live/discharge-fails-proven.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur942013482/003/discharge-fails-proven.example.com.db
2026/10/24 21:30:59 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:00 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:00 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_B,_l3474248913/003/g/56/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall932405319/003/g/57/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall932405319/003/g/58/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:00 activator: swap.example.com hibernated
2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatholds_pinne1107576778/007/g/60/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:00 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatoverlap_succ3444221334/007/g/62/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:00 activator: h.example.com: started carlos-instance-gen@h.example.com:64.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:00 activator: swap.example.com hibernated
2026/10/24 21:31:00 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_super1523654838/007/g/71/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:00 activator: swap.example.com hibernated
2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartholds_pinned_build,_lea636946253/007/g/73/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:00 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 21:31:00 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:00 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_a_fai1239971314/007/g/75/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:00 activator: fence.example hibernated
2026/10/24 21:31:00 activator: fence.example hibernated
2026/10/24 21:31:00 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink496994850/003/g/c.x.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink496994850/003/g/c.x.sock: no such file or directory)
2026/10/24 21:31:00 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink496994850/003/g/c.xx.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink496994850/003/g/c.xx.sock: no such file or directory)
2026/10/24 21:31:00 activator: publish generations to /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile2542515473/002/not-a-dir/zdd/generations.json: open directory /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile2542515473/002/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 21:31:00 activator: cur.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:00 activator: off.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:00 activator: plain.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:01 activator: ident.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:01 activator: ident.example.com hibernated
2026/10/24 21:31:01 activator: ident.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:01 activator: instance ident.example.com exited unexpectedly — recycling
2026/10/24 21:31:01 activator: ident.example.com hibernated
2026/10/24 21:31:01 activator: ident.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:01 activator: due.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:01 activator: due.example.com hibernated
2026/10/24 21:31:01 activator: due.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:01 activator: due.example.com hibernated
2026/10/24 21:31:01 activator: ret.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:01 activator: ret.example.com hibernated
2026/10/24 21:31:01 activator: ret.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:01 activator: ret.example.com hibernated
2026/10/24 21:31:01 activator: bot.x.internal: woke in 21ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 21ms, replicate 0s
2026/10/24 21:31:01 activator: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.example.com: a generation socket path would exceed 107 bytes; generation 9 keeps the canonical socket
2026/10/24 21:31:01 activator: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.example.com: a generation socket path would exceed 107 bytes; generation 9 keeps the canonical socket
2026/10/24 21:31:01 activator: one.example.com: generation 1089 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:01 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:01 activator: one.example.com hibernated
2026/10/24 21:31:01 activator: purged 1 cached response(s) for flat.example.com on spawn
2026/10/24 21:31:01 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile2191065555/001/versions/vA/app seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile2191065555/002/seedexe.example.com.db
2026/10/24 21:31:01 activator: seeded seedexe.example.com
2026/10/24 21:31:01 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile2191065555/001/live/seedexe.example.com seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile2191065555/002/seedexe.example.com.db
2026/10/24 21:31:01 activator: seeded seedexe.example.com
2026/10/24 21:31:01 activator: seeding h.example.com: /tmp/TestNestedSeedExeNamesTheRecordedFile3853841281/001/live/h.example.com seed --db /tmp/nb3164354641/data/instances/h.example.com/h.example.com.db
2026/10/24 21:31:01 activator: seeded h.example.com
2026/10/24 21:31:01 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing720671539/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/24 21:31:01 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing720671539/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/24 21:31:01 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing720671539/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/24 21:31:01 activator: seeding h.example.com: /tmp/TestADeclinedNestedSeedRunsThePinnedFile4182352202/001/live/h.example.com seed --db /tmp/nb3516696583/data/instances/h.example.com/h.example.com.db
2026/10/24 21:31:01 activator: seeded h.example.com
2026/10/24 21:31:01 activator: seeding tier.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed140444053/002/versions/vA/app seed --db /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed140444053/003/tier.example.com.db
2026/10/24 21:31:01 activator: tier.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed140444053/003/tier.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 21:31:01 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=1073741824 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:01 activator: h.example.com: carlos-instance-gen@h.example.com:6.service launched in 0s, healthy 0s later
2026/10/24 21:31:01 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=1073741824 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:01 activator: h.example.com: carlos-instance-gen@h.example.com:6.service launched in 0s, healthy 0s later
2026/10/24 21:31:01 activator: seeding h.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTiernested_seed1602490405/001/live/h.example.com seed --db /tmp/nb4289507177/data/instances/h.example.com/h.example.com.db
2026/10/24 21:31:01 activator: seeded h.example.com
2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsthe_configured_k2961139940/001/shop.example.com.env: the platform mints none while it is
2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsan_own_key_pres3096036078/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestFirstStartWaitsForTheConfiguredKey3119809557/002/shop.example.com.env: the platform mints none while it is
2026/10/24 21:31:01 activator: control listener for shop.example.com: listen unix /tmp/TestFirstStartWaitsForTheConfiguredKey3119809557/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY in /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptinto_the_own_file1158803400/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptdelivered_in_the_co2679738147/001/shop.example.com.env: the platform mints none while it is
2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration3660167693/001/shop.example.com.env: the platform mints none while it is
2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration3660167693/002/shop.example.com.env: the platform mints none while it is
2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyLeavesAFileWithTwoKeysAlone131738540/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistse119992415/002/shop.example.com.env: the platform mints none while it is
2026/10/24 21:31:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistsn787051167/002/shop.example.com.env: the platform mints none while it is
2026/10/24 21:31:02 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance3368214908/001/live/seeded.example.com seed --db /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance3368214908/002/seeded.example.com.db
2026/10/24 21:31:02 activator: seeded seeded.example.com
2026/10/24 21:31:02 activator: restored sleepy.example.com from s3://test-bucket/sleepy.example.com
hello
world
2026/10/24 21:31:02 activator: fence.example hibernated
2026/10/24 21:31:02 activator: fence.example hibernated
2026/10/24 21:31:02 activator: fence.example hibernated
2026/10/24 21:31:02 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:02 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/24 21:31:02 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/24 21:31:02 activator: barrier.example.com hibernated
2026/10/24 21:31:02 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:02 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:02 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/24 21:31:02 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/24 21:31:02 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/24 21:31:02 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/24 21:31:02 activator: barrier.example.com hibernated
2026/10/24 21:31:03 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:03 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:03 activator: barrier.example.com: generation 12's child is not proven empty: g12 still holds a process
2026/10/24 21:31:03 activator: generation 12 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/24 21:31:03 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:03 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/24 21:31:03 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 21:31:03 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/24 21:31:03 activator: barrier.example.com hibernated
2026/10/24 21:31:03 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:03 activator: stop generation 11: cgroup.kill: the child is not empty
2026/10/24 21:31:03 activator: generation 11 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/24 21:31:03 activator: placed.example.com: kill generation 1097 after a failed health wait: g still holds a process
2026/10/24 21:31:03 activator: placed.example.com: generation 1097's child is not proven empty: g still holds a process
2026/10/24 21:31:03 activator: placed.example.com: failed wake's generation 1097 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:03 activator: placed.example.com: wake failed and its generation is not proven stopped: start placed.example.com: instance not healthy within 300ms: Get "http://carlos/": dial unix /tmp/g235445025/g/1097/placed.example.com.sock: connect: no such file or directory; activator: generation start failed and its termination is unproven: kill generation 1097 after a failed health wait: g still holds a process
2026/10/24 21:31:03 activator: placed.example.com: failed wake's generation 1097 is proven stopped; lease given back
2026/10/24 21:31:04 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:04 activator: unproven.example.com: failed wake's generation 1100 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:04 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 21:31:04 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:04 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:04 activator: unproven.example.com: failed wake's generation 1100 is proven stopped; lease given back
2026/10/24 21:31:04 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:04 activator: unproven.example.com: failed wake's generation 1101 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:04 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 21:31:04 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:04 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:04 activator: unproven.example.com: failed wake's generation 1101 is proven stopped; lease given back
2026/10/24 21:31:04 activator: unproven.example.com: generation 1102's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:04 activator: unproven.example.com: failed wake's generation 1102 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:04 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 21:31:05 activator: unhealthy-f6.example.com: failed wake's generation 1103 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 21:31:05 activator: unhealthy-f6.example.com: failed wake's generation 1103 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:05 activator: unhealthy-f6.example.com: failed wake's generation 1103 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 21:31:05 activator: unhealthy-f6.example.com: failed wake's generation 1103 is proven stopped; lease given back
2026/10/24 21:31:05 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/24 21:31:06 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/24 21:31:06 activator: one.example.com: generation 1105 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:06 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:06 activator: instance one.example.com exited unexpectedly — recycling
2026/10/24 21:31:06 activator: one.example.com hibernated
2026/10/24 21:31:06 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:06 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:06 activator: bot.x.internal hibernated
2026/10/24 21:31:06 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:06 activator: worker bot.x.internal respawned (attempt (restart)), now running v2
2026/10/24 21:31:06 activator: bot.x.internal hibernated
2026/10/24 21:31:06 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:06 activator: worker bot.x.internal respawned (attempt (config)), now running v2
2026/10/24 21:31:06 activator: bot.x.internal hibernated
2026/10/24 21:31:06 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:06 activator: bot.x.internal hibernated
2026/10/24 21:31:07 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:07 activator: bot.x.internal hibernated
2026/10/24 21:31:07 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:07 activator: bot.x.internal hibernated
2026/10/24 21:31:07 activator: moved.example.com: promoting 2 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:07 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:07 activator: x.example hibernated
2026/10/24 21:31:07 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/24 21:31:07 activator: x.example hibernated
2026/10/24 21:31:07 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/24 21:31:07 activator: x.example hibernated
2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:07 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:07 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/24 21:31:07 activator: x.example hibernated
2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:07 activator: x.example hibernated
2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:07 activator: x.example hibernated
2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:07 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:07 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:07 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:07 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:07 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:07 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_keyless_replica_is_refu3416413544/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:07 activator: replica shop-au.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_is_3056958043/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_pre-existing_env_key_th3118349134/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_dif260765861/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_with_no_2293327036/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:07 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY cannot be checked against its home's key (no entry for it in /etc/carlos/replica-home-keys; O8)
2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_home_updated,_this_re2090866327/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:07 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY is not the one the bucket's configuration now holds: a key change is in flight (ruling 27)
2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_bucket_cannot_be_read3779415383/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:07 activator: replica shop-au.example.com not started: the bucket's configuration could not be read: bucket unreachable
2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketno_fresh_reader_refused3496770306/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:07 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY cannot be checked against the bucket's configuration
2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketresidual_a_change_only_o3632784341/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_config-only_homes_keyl2988459519/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_home_row_mints_and_its_3751163128/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:07 activator: seeding shop-au.example.com: /tmp/TestReplicaSeedIsGatedthe_homes_key691337841/001/live/shop-au.example.com seed --db /tmp/TestReplicaSeedIsGatedthe_homes_key691337841/002/shop-au.example.com.db
2026/10/24 21:31:07 activator: seeded shop-au.example.com
2026/10/24 21:31:07 activator: replica shop-au.example.com not started: where its CARLOS_INSTANCE_KEY came from is unknown
2026/10/24 21:31:07 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaGateDecidesOnTheLaunchesOwnRead2022993743/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:07 provision: a.shop.example.com: swapped for import sd_jbbbgrfjpmfzyrn7gw7zlbkkpi (the original route is journaled)
2026/10/24 21:31:07 provision: a.shop.example.com: import sd_jbbbgrfjpmfzyrn7gw7zlbkkpi rolled back to its original route
2026/10/24 21:31:07 provision: a.shop.example.com: import sd_jbbbgrfjpmfzyrn7gw7zlbkkpi is cleaned up and its journal row retired
2026/10/24 21:31:07 activator: h.example.com: placed seed sd_nhimk5l5np5rlhmoc7a6zaf6em (8192 bytes)
2026/10/24 21:31:07 activator: h.example.com: placed seed sd_c75qt54d3bxvxncbfyeaxfnifm (8192 bytes)
2026/10/24 21:31:07 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/nb1894519496/data/instances/h.example.com/h.example.com.env: the platform mints none while it is
2026/10/24 21:31:07 activator: h.example.com: placed seed sd_4vlamgbfrtwbqdxl53h6wxgu6q (8192 bytes)
2026/10/24 21:31:07 activator: CARLOS_INSTANCE_KEY in /tmp/nb4227882448/data/instances/h.example.com/h.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/24 21:31:07 activator: h.example.com: placed seed sd_xnm6duxahoayddv2y4h774eykq (8192 bytes)
2026/10/24 21:31:07 activator: h.example.com: placed seed sd_sstbch3dbqvvoevuwp73eubhou (8192 bytes)
2026/10/24 21:31:07 activator: h.example.com: placed seed sd_6ievog7mnpmcijxetpinkt5p3y (8192 bytes)
2026/10/24 21:31:07 activator: h.example.com: keeping seed sd_6ievog7mnpmcijxetpinkt5p3y placed by an earlier wake
2026/10/24 21:31:07 activator: h.example.com: placed seed sd_6i637fmcmiiehzfpeajyb2l6ba (8192 bytes)
2026/10/24 21:31:07 activator: h.example.com: placed seed sd_kkgwqgr6bmtaejeph5gvo2pqqm (8192 bytes)
2026/10/24 21:31:07 activator: h.example.com: placed seed sd_dcntnsool5i3ramvg5ejy4iwim (8192 bytes)
2026/10/24 21:31:07 activator: h.example.com: seed sd_2ogh5s6vprzywec3e7ejorqmmi: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/24 21:31:07 activator: h.example.com: seed sd_uw4a4qckfmqxby4d2jmtatsfx4: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/24 21:31:07 activator: h.example.com: seed sd_kkfguefoqd3fzxleqlaxivz3wa: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/24 21:31:07 activator: h.example.com: seed sd_do2mxmtu5zpslvckyexuftjqyu: h.example.com could not start from its seed: the snapshot could not be read. The next wake tries again.: bucket down
2026/10/24 21:31:07 activator: h.example.com: seed sd_gx3nvnnzjpu2dqldvedjseol7y: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/24 21:31:07 activator: h.example.com: seed sd_ma6yahryw2ysw476sswmajmtji: h.example.com could not start from its seed: the snapshot is not a database that passes an integrity check.: integrity_check: file is not a database (26)
2026/10/24 21:31:07 activator: h.example.com: seed sd_a25nshjsgfu652mgcczonkhare: h.example.com could not start from its seed: claim control is on for this box.
2026/10/24 21:31:07 activator: h.example.com: seed sd_a25nshjsgfu652mgcczonkhare: h.example.com could not start from its seed: this box does not run instances nested.
2026/10/24 21:31:07 activator: h.example.com: seed sd_a3hkzcbftmtqspccaobrcfoogi: seed fallback for h.example.com: cannot tell whether it has a replica of its own: listing failed
2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 21:31:07 activator: h.example.com: seed sd_ltoqhue45x6tjho35637xo3dom: seed fallback for h.example.com: found no replica of its own to restore
2026/10/24 21:31:07 activator: h.example.com: promoting 1 staged object(s) from moves/h.example.com/ (residence move landing)
2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 21:31:07 activator: h.example.com: seed sd_dtz75toogkn7takslvhoxh7kju: seed fallback for h.example.com: found no replica of its own to restore
2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 21:31:07 activator: h.example.com: seed sd_7kzzo4vxgl5c5nbns3qfetjzxq: seed fallback for h.example.com: found no replica of its own to restore
2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 21:31:07 activator: h.example.com: placed seed sd_dhgk36dlk2ciuh2ut5wewqwzru (8192 bytes)
2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 21:31:07 activator: h.example.com: seed sd_nt42ymoturbqpzzmuor2fbh7am: h.example.com could not start from its seed: the snapshot could not be read. The next wake tries again.: connection reset
2026/10/24 21:31:07 activator: h.example.com: seed sd_ttpqc2kq4ef7rxnpcc76ycjmai: h.example.com could not start from its seed: it could not be put in place. The next wake tries again.: openat2 /tmp/nb729134131/data/instances/h.example.com/.seed-sd_ttpqc2kq4ef7rxnpcc76ycjmai.tmp: permission denied
2026/10/24 21:31:07 activator: h.example.com: seed sd_xlyfsxjt55erubtg7do3ujz7du: h.example.com could not start from its seed: the snapshot is not a database that passes an integrity check.: open: unable to open database file (14)
2026/10/24 21:31:07 activator: h.example.com: seed sd_d5pdi7cg5qapqjtjoc6iu4jft4: h.example.com could not start from its seed: it could not be put in place. The next wake tries again.: instancefs: path contains a symlink: /tmp/nb2040639698/data/instances/h.example.com/.seed-sd_d5pdi7cg5qapqjtjoc6iu4jft4.tmp
2026/10/24 21:31:07 activator: h.example.com: placed seed sd_pxg3vbxijikmq32ucykw75d3u4 (8192 bytes)
2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 21:31:07 activator: h.example.com: seed sd_zhwy6r2xei2z6ngaeaovvauy2a not recorded as placed yet: record unreachable
2026/10/24 21:31:07 activator: h.example.com: /tmp/nb770880587/data/instances/h.example.com/h.example.com.db held seed sd_65fjnz7xkzdxubplhb2apwaoja's marker, not this route's (seed "sd_oqlteiucz4zmptcgy7v6naguoi"): purged before the restore
2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 21:31:07 activator: h.example.com: /tmp/nb672842420/data/instances/h.example.com/h.example.com.db held seed sd_fi44qykjns4xfohystlhiyui2a's marker, not this route's (seed "sd_ch5sjo7st5d4hbvhjt7yax6wba"): purged before the restore
2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 21:31:07 activator: h.example.com: /tmp/nb3911238823/data/instances/h.example.com/h.example.com.db held seed sd_cq2g7qy2s77ecme2dbgdrhbnaa's marker, not this route's (seed ""): purged before the restore
2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 21:31:07 activator: h.example.com: /tmp/nb1568073016/data/instances/h.example.com/h.example.com.db held seed sd_s6vm2fdmjdxi2spygcgcgt5v2u's marker, not this route's (seed "sd_z3362cgid63rgj3k3x2vk3leri"): purged before the restore
2026/10/24 21:31:07 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 21:31:07 activator: h.example.com: /tmp/nb45767515/data/instances/h.example.com/h.example.com.db held seed sd_m7ecmb2xdibtgjq44524yrunmq's marker, not this route's (seed ""): purged before the restore
2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 21:31:07 activator: h.example.com: seed sd_5qoqxvowbahmnwjrmakqjw47uu: h.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup.events: unreadable
2026/10/24 21:31:07 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 21:31:07 activator: h.example.com: placed seed sd_5qoqxvowbahmnwjrmakqjw47uu (8192 bytes)
2026/10/24 21:31:07 activator: h.example.com: placed seed sd_6zyyj4h7aschtqmemdlatv5ply (8192 bytes)
2026/10/24 21:31:07 activator: h.example.com: /tmp/nb1464908442/data/instances/h.example.com/h.example.com.db held seed an aborted import's data's marker, not this route's (seed ""): purged before the restore
2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 21:31:07 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 21:31:07 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:07 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:07 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:07 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:07 activator: swap.example.com hibernated
2026/10/24 21:31:07 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:07 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/24 21:31:07 activator: swap.example.com hibernated
2026/10/24 21:31:07 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:07 activator: LEASE LOST for swap.example.com (renewal failed) — standing down
2026/10/24 21:31:07 activator: swap.example.com hibernated
2026/10/24 21:31:08 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:08 activator: swap.example.com hibernated
2026/10/24 21:31:08 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:08 activator: swap.example.com is no longer hibernate-managed — releasing
2026/10/24 21:31:08 activator: swap.example.com hibernated
2026/10/24 21:31:08 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:08 activator: swap.example.com hibernated
2026/10/24 21:31:08 activator: exit.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:08 activator: exit.example.com hibernated
2026/10/24 21:31:08 activator: wake shop.example.com: activator stopped during startup
2026/10/24 21:31:08 activator: wake shop.example.com: activator stopped during startup
2026/10/24 21:31:08 activator: shop.example.com: route refreshed: db "" -> "" socket "/run/carlos/shop.example.com.sock" -> "/run/carlos/instances/shop.example.com/app.sock"
2026/10/24 21:31:08 activator: shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:08 activator: missed-join.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:08 activator: stop missed-join.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/24 21:31:09 activator: missed-join.example.com hibernated
2026/10/24 21:31:09 activator: nested-stuck.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:09 activator: stop nested-stuck.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/24 21:31:09 activator: nested-stuck.example.com: the sidecar a restart brought back for a stopping entry is not proven gone (activator: stop unproven; processes remain); keeping the lease until it has exited
2026/10/24 21:31:09 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 21:31:09 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 21:31:09 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 21:31:09 activator: nested-stuck.example.com hibernated
2026/10/24 21:31:09 activator: checks.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:09 activator: owned.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:09 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 21:31:09 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 21:31:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:09 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 21:31:09 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/24 21:31:09 activator: bot.x.internal hibernated
2026/10/24 21:31:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:09 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory; the sweep retries
2026/10/24 21:31:09 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory
2026/10/24 21:31:09 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory; the sweep retries
2026/10/24 21:31:09 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/24 21:31:09 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory; the sweep retries
2026/10/24 21:31:09 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory; the sweep retries
2026/10/24 21:31:09 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory
2026/10/24 21:31:09 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory; the sweep retries
2026/10/24 21:31:09 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3190023497/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/24 21:31:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:09 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:09 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:09 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:10 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 21:31:10 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:10 activator: worker bot.x.internal: the attempt's respawn did not finish within 150ms of its caller leaving; reporting its build as possibly executed
2026/10/24 21:31:10 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com hibernated
2026/10/24 21:31:10 activator: swap.example.com: generation 1149 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com hibernated
2026/10/24 21:31:10 activator: swap.example.com: generation 1151 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com hibernated
2026/10/24 21:31:10 activator: swap.example.com: generation 1153 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com hibernated
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com hibernated
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com hibernated
2026/10/24 21:31:10 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com hibernated
2026/10/24 21:31:10 activator: swap.example.com: generation 1158 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com hibernated
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com hibernated
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com hibernated
2026/10/24 21:31:10 activator: swap.example.com: generation 1163 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com hibernated
2026/10/24 21:31:10 activator: swap.example.com: generation 1166 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com hibernated
2026/10/24 21:31:10 activator: swap.example.com: generation 1168 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3965323956/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3965323956/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 21:31:10 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3965323956/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3965323956/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: h.example.com: started carlos-instance-gen@h.example.com:7.service memory.max=1073741824 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:10 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: processes remain
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:10 activator: swap.example.com hibernated
2026/10/24 21:31:11 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:11 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:11 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:11 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:11 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:11 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:17 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: instance control /stop: 502: stop: processes remain
2026/10/24 21:31:17 activator: seeding h.example.com: /tmp/TestSeedUnitCleansUpOnlyItsOwnDBOnFailure1032246445/001/live/h.example.com seed --fast --db /tmp/nb3813122750/data/instances/h.example.com/h.example.com.db
2026/10/24 21:31:17 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:17 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 21:31:17 activator: seeding h.example.com: /tmp/TestSeedUnitFailureStopsBeforeCleanupAndKeepsTheDBWhenProcessesR3465610916/001/live/h.example.com seed --fast --db /tmp/nb342890965/data/instances/h.example.com/h.example.com.db
2026/10/24 21:31:18 activator: seeding h.example.com: /tmp/TestANestedSeedRecordsItsObligationBeforeItRuns1402265559/001/live/h.example.com seed --db /tmp/nb497377878/data/instances/h.example.com/h.example.com.db
2026/10/24 21:31:18 activator: seeded h.example.com
2026/10/24 21:31:18 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI4254598315/001/live/h.example.com seed --db /tmp/nb607401055/data/instances/h.example.com/h.example.com.db
2026/10/24 21:31:18 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 21:31:18 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 21:31:18 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI931097750/001/live/h.example.com seed --db /tmp/nb2916150891/data/instances/h.example.com/h.example.com.db
2026/10/24 21:31:18 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 21:31:18 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 21:31:18 activator: fifo-obligation.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/24 21:31:18 activator: one.example.com: generation 1902 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:18 activator: one.example.com: generation 1904 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:18 activator: one.example.com: first wake of v2 had 30s of its health window left at the exec, under the 1h0m0s floor; booting the good build v1 without trying it
2026/10/24 21:31:18 activator: one.example.com: generation 1907 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:18 activator: one.example.com: generation 1908 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:18 activator: one.example.com: generation 1909 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:18 activator: one.example.com hibernated
2026/10/24 21:31:18 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:18 activator: one.example.com: generation 1911 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:18 activator: tenant.test not started: it is held on v1, and the launch hashed "v2" (the live link moved after the check); the next wake checks again
2026/10/24 21:31:18 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:19 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:19 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:19 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:19 activator: one.example.com: generation 1916 of v0: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:19 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:19 activator: one.example.com: generation 1917 of v0: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:19 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:19 activator: one.example.com not woken: it is held on v1, and this wake would run "v2" (the live link moved after the check); the next wake checks again
2026/10/24 21:31:19 activator: one.example.com not woken: it is held on v1, and this wake would run "" (the live link moved after the check); the next wake checks again
2026/10/24 21:31:19 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:19 activator: swap.example.com hibernated
2026/10/24 21:31:19 activator: swap.example.com: generation 1919 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:19 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:20 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:20 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:20 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:21 activator: seeding h.example.com: /tmp/TestANestedUnversionedLinkWithNoHookWakesAsToday716706617/001/live/h.example.com seed --db /tmp/nb77297083/data/instances/h.example.com/h.example.com.db
2026/10/24 21:31:21 activator: seeded h.example.com
2026/10/24 21:31:21 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:21 activator: h.example.com: carlos-instance-gen@h.example.com:1926.service launched in 0s, healthy 0s later
2026/10/24 21:31:21 activator: h.example.com: woke in 1ms: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 21:31:21 activator: u.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 21:31:21 activator: e.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:21 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:21 activator: swap.example.com hibernated
2026/10/24 21:31:21 activator: swap.example.com: generation 1932 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:21 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:21 activator: one.example.com: generation 1935 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:21 activator: one.example.com: generation 1937 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:21 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:21 activator: one.example.com: generation 1940's child is not proven empty: activator: stop unproven; processes remain
2026/10/24 21:31:21 activator: one.example.com: failed wake's generation 1940 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:22 activator: one.example.com: generation 1941 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:22 activator: one.example.com: generation 1942 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:22 activator: one.example.com hibernated
2026/10/24 21:31:22 activator: one.example.com: generation 1943 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:22 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:23 activator: one.example.com: first wake of v2 had 400ms of its health window left at the exec, under the 20s floor; booting the good build v1 without trying it
2026/10/24 21:31:23 activator: one.example.com: generation 1945 of v1: record 0s, start 251ms, healthy 0s later
2026/10/24 21:31:23 activator: one.example.com: woke in 251ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 251ms, replicate 0s
2026/10/24 21:31:23 activator: two.example.com: generation 1946 of v1: record 0s, start 250ms, healthy 0s later
2026/10/24 21:31:23 activator: two.example.com: woke in 250ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 250ms, replicate 0s
2026/10/24 21:31:23 activator: x.example: hold's pinned wake refused or abandoned: the claims admission refused before the exec: hostbox: unresolved lifecycle contract: isolated root launcher cannot carry claim reservations
2026/10/24 21:31:23 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1436732107/003/g/1947/real.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:25 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1436732107/003/g/1948/real.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:25 activator: real.example.com: generation 1948 of v1: record 0s, start 1ms, healthy 0s later
2026/10/24 21:31:25 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1436732107/003/g/1948/real.example.com.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1436732107/003/g/1948/real.example.com.sock: no such file or directory)
2026/10/24 21:31:25 activator: real.example.com: woke in 2.001s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 2.001s, replicate 0s
2026/10/24 21:31:25 activator: real.example.com hibernated
2026/10/24 21:31:27 activator: real.example.com: generation 1950 of v1: record 0s, start 1ms, healthy 0s later
2026/10/24 21:31:27 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign3884098207/004/hand-set.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign3884098207/004/hand-set.sock: no such file or directory)
2026/10/24 21:31:27 activator: real.example.com: woke in 2.002s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 2.002s, replicate 0s
2026/10/24 21:31:27 activator: real.example.com hibernated
2026/10/24 21:31:27 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:27 activator: bot.x.internal: worker generation 0's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 21:31:27 activator: bot.x.internal: worker generation 1952's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 21:31:27 activator: bot.x.internal: failed wake's generation 1952 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:28 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:28 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd: not a directory; the sweep retries
2026/10/24 21:31:28 activator: worker bot.x.internal not spawned: the build it would exec (v2 /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/003/v2) could not be published first: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd: not a directory
2026/10/24 21:31:28 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd: not a directory; the sweep retries
2026/10/24 21:31:28 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3567091529/002/zdd: not a directory; the sweep retries
2026/10/24 21:31:28 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn2205933732/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn2205933732/002/not-a-dir: not a directory; the sweep retries
2026/10/24 21:31:28 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn2205933732/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn2205933732/002/not-a-dir: not a directory; the sweep retries
2026/10/24 21:31:28 activator: one.example.com: generation 1955 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:28 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:28 activator: instance one.example.com exited unexpectedly — recycling
2026/10/24 21:31:28 activator: one.example.com hibernated
2026/10/24 21:31:28 activator: one.example.com: generation 1956 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:28 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:28 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:28 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:28 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:28 activator: one.example.com hibernated
2026/10/24 21:31:30 activator: one.example.com: first wake of v2 had 499ms of its health window left at the exec, under the 1s floor; booting the good build v1 without trying it
2026/10/24 21:31:30 activator: one.example.com: generation 1961 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:30 activator: one.example.com: woke in 1.501s: checks 0s, lease 0s, restore 0s, identity 1.501s, spawn 0s, replicate 0s
2026/10/24 21:31:32 activator: one.example.com: generation 1962 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:32 activator: one.example.com: woke in 1.5s: checks 0s, lease 0s, restore 0s, identity 1.5s, spawn 0s, replicate 0s
2026/10/24 21:31:32 activator: one.example.com: generation 1964 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:32 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:32 activator: bot.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:32 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/24 21:31:32 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/24 21:31:32 activator: worker bot.example.com respawned (exited), now running
2026/10/24 21:31:32 activator: bot.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:32 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/24 21:31:32 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/24 21:31:32 activator: worker bot.example.com respawned (exited), now running
2026/10/24 21:31:32 activator: bot.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:32 activator: bot.example.com: worker generation 1967's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 21:31:32 activator: stop bot.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 21:31:32 activator: bot.example.com hibernated
2026/10/24 21:31:32 activator: bot3.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:32 activator: bot3.example.com: worker generation 1968's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 21:31:32 activator: worker bot3.example.com NOT respawned (test: a restart with no old handle): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/24 21:31:32 activator: bot4.example.com: worker generation 1970's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 21:31:32 activator: bot4.example.com: failed wake's generation 1970 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:32 activator: bot4.example.com: failed wake's generation 1970 is proven stopped; lease given back
2026/10/24 21:31:32 activator: bot2.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:32 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/24 21:31:32 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/24 21:31:32 activator: bot2.example.com: worker generation 1971's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 21:31:32 activator: worker bot2.example.com NOT respawned (exited): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/24 21:31:32 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/24 21:31:32 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/24 21:31:32 activator: worker bot2.example.com respawned (exited), now running
2026/10/24 21:31:32 activator: bot.example.com: woke in 22ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 1ms, settle 21ms, replicate 0s
2026/10/24 21:31:32 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/24 21:31:32 activator: worker bot.example.com respawned (exited), now running v1
2026/10/24 21:31:32 activator: bot.example.com hibernated
2026/10/24 21:31:32 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:33 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:33 activator: bot10.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:33 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/24 21:31:33 activator: worker bot10.example.com: the shed respawn child's exit is not proven (activator: stop unproven; processes remain); the entry keeps it and its build until the sweep proves it
2026/10/24 21:31:33 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/24 21:31:33 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/24 21:31:33 activator: bot10.example.com: the shed respawn child has exited
2026/10/24 21:31:33 activator: bot10.example.com hibernated
2026/10/24 21:31:33 activator: bot10.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:33 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/24 21:31:33 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/24 21:31:33 activator: bot10.example.com respawn did not finish before its stop; stopping the rest, the lease is kept until it has
2026/10/24 21:31:33 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/24 21:31:33 activator: bot10.example.com hibernated
2026/10/24 21:31:33 activator: bot8.example.com: worker generation 1974's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 21:31:33 activator: bot8.example.com: failed wake's generation 1974 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:33 activator: bot8.example.com: failed wake's generation 1974 is proven stopped; lease given back
2026/10/24 21:31:33 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:33 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:33 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:33 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:33 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/24 21:31:33 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:33 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:33 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:33 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/24 21:31:34 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/24 21:31:34 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:34 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:34 activator: bot.x.internal hibernated
2026/10/24 21:31:34 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:34 activator: worker bot.x.internal respawned (attempt (restart)), now running v1
2026/10/24 21:31:34 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 21:31:34 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:34 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:34 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:34 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:34 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 21:31:35 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:35 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 21:31:35 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 21:31:35 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:35 activator: bot.x.internal: worker generation 1977's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (attempt (release)): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/24 21:31:35 activator: bot.x.internal: worker generation 1977's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (restoring v1): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: lease lost; abandoning it
2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:35 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory; the sweep retries
2026/10/24 21:31:35 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory
2026/10/24 21:31:35 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory; the sweep retries
2026/10/24 21:31:35 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/24 21:31:35 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory; the sweep retries
2026/10/24 21:31:35 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory; the sweep retries
2026/10/24 21:31:35 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory
2026/10/24 21:31:35 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory; the sweep retries
2026/10/24 21:31:35 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2977783892/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:35 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:35 activator: bot.x.internal hibernated
2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:35 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:35 activator: worker bot.x.internal NOT respawned (falling back to v1): swap: lease lost; abandoning it
2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:35 activator: WORKER bot.x.internal failed to respawn (attempt (restart)): exec: text file busy — the sweep's crash restart takes it from here
2026/10/24 21:31:35 activator: WORKER bot.x.internal failed to respawn (restoring v1): exec: text file busy — the sweep's crash restart takes it from here
2026/10/24 21:31:35 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:35 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:36 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/24 21:31:36 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:36 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:36 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 21:31:36 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/24 21:31:36 activator: bot.x.internal hibernated
2026/10/24 21:31:36 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:36 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 21:31:36 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 21:31:36 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:36 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 21:31:36 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 21:31:36 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:36 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:36 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/24 21:31:36 activator: bot.x.internal hibernated
2026/10/24 21:31:36 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:36 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:36 activator: LEASE LOST for bot.x.internal (renewal failed) — standing down
2026/10/24 21:31:36 activator: bot.x.internal hibernated
2026/10/24 21:31:37 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:37 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:37 activator: bot.x.internal hibernated
2026/10/24 21:31:37 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:37 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 21:31:37 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/24 21:31:37 activator: bot.x.internal hibernated
2026/10/24 21:31:37 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawngated_build_is_the_ex831226331/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:37 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawna_held_build_starts_n2970356973/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
--- FAIL: TestExecBackendStartSidecarGen (0.01s)
--- FAIL: TestExecBackendStartSidecarGen/foreign (0.00s)
swap_sidecar_test.go:669: fork/exec /tmp/TestExecBackendStartSidecarGenforeign4131554283/001/versions/v2/app: text file busy
2026/10/24 21:31:38 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawnno_gate_is_todays_sp3847055878/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:38 activator: shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: stop fence.example: earlier process may still run
2026/10/24 21:31:38 activator: fence.example hibernated
2026/10/24 21:31:38 activator: fence.example hibernated
2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory; the sweep retries
2026/10/24 21:31:38 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory; the sweep retries
2026/10/24 21:31:38 activator: worker first.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory
2026/10/24 21:31:38 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory; the sweep retries
2026/10/24 21:31:38 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory; the sweep retries
2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: swap.example.com hibernated
2026/10/24 21:31:38 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:1988.service exited before it was observed
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:38 activator: one.example.com idle 1m31s — hibernating
2026/10/24 21:31:38 activator: one.example.com hibernated
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/24 21:31:38 provision: a.shop.example.com: swapped for import sd_vrw6ytbwk7j3x4r3isnojpwgvm (the original route is journaled)
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: one.example.com idle 1h0m0s — hibernating
2026/10/24 21:31:38 activator: one.example.com hibernated
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:38 activator: bot6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: bot1.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: bot.example.com: woke in 21ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 1ms, settle 20ms, replicate 0s
2026/10/24 21:31:38 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 21:31:38 activator: bot.example.com hibernated
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory; the sweep retries
2026/10/24 21:31:38 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory
2026/10/24 21:31:38 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory; the sweep retries
2026/10/24 21:31:38 activator: WORKER bot.x.internal failed to respawn (exited): worker build not published before its exec: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished2783360517/002/data/zdd: not a directory — the sweep's crash restart takes it from here
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: WORKER bot.x.internal failed to respawn (exited): fork/exec: resource temporarily unavailable — the sweep's crash restart takes it from here
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: bot9.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: worker bot9.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/24 21:31:38 activator: bot9.example.com: worker generation 1994's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 21:31:38 activator: generation 1994 of bot9.example.com did not prove its exit; keeping it in the operation set
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:38 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:38 activator: LEASE LOST for bot9.example.com (renewal failed) — standing down
2026/10/24 21:31:38 activator: bot9.example.com hibernated
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/24 21:31:38 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 21:31:38 activator: bot2.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot.x.internal exited; live/bot.x.internal names v2, but the release scheduler owns build changes, so it recycles its captured build v1
2026/10/24 21:31:38 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 21:31:38 activator: a.shop.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@a.shop.example.com: unit still active); keeping the lease until it is
2026/10/24 21:31:38 provision: a.shop.example.com: import sd_vrw6ytbwk7j3x4r3isnojpwgvm rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:38 activator: worker bot1.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot1.x.internal exited; it serves v1 but live/bot1.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/24 21:31:38 activator: worker bot2.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot2.x.internal exited; live/bot2.x.internal names v2, but this sweep's converge budget is spent, so it recycles its captured build v1 and converges on a later sweep
2026/10/24 21:31:38 activator: worker bot1.x.internal respawned (converging onto a new build), now running v2
2026/10/24 21:31:38 activator: worker bot2.x.internal respawned (exited), now running v1
2026/10/24 21:31:38 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 21:31:38 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot.x.internal exited; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 21:31:38 activator: worker bot2.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:38 activator: worker bot2.x.internal respawned (converging onto a new build), now running v2
2026/10/24 21:31:38 provision: a.shop.example.com: import sd_vrw6ytbwk7j3x4r3isnojpwgvm rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 21:31:38 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 21:31:38 activator: worker bot.x.internal hit its max_run; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 21:31:38 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (hit its max_run), now running v2
2026/10/24 21:31:38 provision: a.shop.example.com: import sd_vrw6ytbwk7j3x4r3isnojpwgvm rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 21:31:38 activator: nested-costop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:38 activator: stop sidecar nested-costop.example.com: activator: stop unproven; processes remain
2026/10/24 21:31:38 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 21:31:38 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 21:31:38 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 21:31:38 provision: a.shop.example.com: import sd_vrw6ytbwk7j3x4r3isnojpwgvm rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 21:31:38 activator: nested-costop.example.com hibernated
2026/10/24 21:31:38 activator: restart.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: instance restart.example.com exited unexpectedly — recycling
2026/10/24 21:31:38 activator: restart.example.com hibernated
2026/10/24 21:31:38 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: swap.example.com hibernated
2026/10/24 21:31:38 activator: h.example.com: kill carlos-instance-gen@h.example.com:2002.service after a failed start: stop timed out
2026/10/24 21:31:38 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2002.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/24 21:31:38 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: h.example.com: kill carlos-instance-gen@h.example.com:2004.service after a failed start: stop timed out
2026/10/24 21:31:38 activator: SIDECAR for late-sidecar.example failed to start: activator: maintenance fence active — retaining returned process
2026/10/24 21:31:38 activator: late-sidecar.example: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:38 activator: late-sidecar.example hibernated
2026/10/24 21:31:38 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/24 21:31:38 activator: one.example.com hibernated
2026/10/24 21:31:38 activator: replica shop.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/24 21:31:38 activator: slowproof.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: instance slowproof.example.com exited unexpectedly — recycling
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: slowproof.example.com hibernated
2026/10/24 21:31:38 activator: own.example.com: failed wake's generation 0 is proven stopped; lease given back
2026/10/24 21:31:38 activator: bot7.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 provision: a.shop.example.com: swapped for import sd_smee6gqicg2fasqnkhupnie6xm (the original route is journaled)
2026/10/24 21:31:38 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:38 activator: bot5.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/24 21:31:38 activator: queued.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: instance unproven.example.com exited unexpectedly — recycling
2026/10/24 21:31:38 activator: unproven.example.com hibernated
2026/10/24 21:31:38 activator: empty.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:38 activator: one.example.com: sidecar starting on the instance's own build v1 (/opt/carlos/versions/v1/app): this box will not start v2, the build its live link names
2026/10/24 21:31:38 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:38 activator: bot.x.internal serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 21:31:38 activator: bot.x.internal hibernated
2026/10/24 21:31:38 activator: early.example.com: the aborted start's unit is not proven stopped (context canceled); keeping the lease until it is
2026/10/24 21:31:38 activator: shutdown: out of time with 1 failed wake(s) not proven stopped, their leases still held: early.example.com (generation 1998)
2026/10/24 21:31:38 activator: refused.example.com: giving the lease back failed: delete lease: context deadline exceeded; retrying it
2026/10/24 21:31:38 activator: refused.example.com: lease given back on retry
2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: one.example.com hibernated
2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: one.example.com hibernated
2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: one.example.com hibernated
2026/10/24 21:31:38 activator: unproven.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@unproven: instance control socket: connection refused); keeping the lease until it is
2026/10/24 21:31:38 activator: odd.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: stuck.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 21:31:38 provision: a.shop.example.com: import sd_smee6gqicg2fasqnkhupnie6xm rolled back to its original route
2026/10/24 21:31:38 provision: a.shop.example.com: import sd_smee6gqicg2fasqnkhupnie6xm is cleaned up and its journal row retired
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:38 activator: one.example.com hibernated
2026/10/24 21:31:38 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 21:31:38 activator: bot.x.internal hibernated
2026/10/24 21:31:38 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:38 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: queued.example.com hibernated
2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: one.example.com serving v1 but v2 is adopted — cycling onto the new build
2026/10/24 21:31:39 activator: one.example.com hibernated
2026/10/24 21:31:39 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: stuck.example.com hibernated
2026/10/24 21:31:39 provision: a.shop.example.com: swapped for import sd_2qytelynd4qpmqvjgyhcp7l3ly (the original route is journaled)
2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:39 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:31:39 activator: worker bot.x.internal respawned (exited), now running
2026/10/24 21:31:39 activator: a.shop.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@a.shop.example.com: instance control socket: connection refused); keeping the lease until it is
2026/10/24 21:31:39 provision: a.shop.example.com: import sd_2qytelynd4qpmqvjgyhcp7l3ly rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 21:31:39 activator: a.shop.example.com: failed wake's generation 2028 is proven stopped; lease given back
2026/10/24 21:31:39 provision: a.shop.example.com: import sd_2qytelynd4qpmqvjgyhcp7l3ly rolled back to its original route
2026/10/24 21:31:39 provision: a.shop.example.com: import sd_2qytelynd4qpmqvjgyhcp7l3ly is cleaned up and its journal row retired
2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: swap.example.com hibernated
2026/10/24 21:31:39 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: slowproof.example.com: the aborted start's unit is not proven stopped (context deadline exceeded); keeping the lease until it is
2026/10/24 21:31:39 provision: a.shop.example.com: swapped for import sd_3vuhuk6ovoheqq4chzis5unm4q (the original route is journaled)
2026/10/24 21:31:39 activator: control listener for shop.example.com: listen unix /tmp/TestARefusedLaunchLeavesNoBuildStateRecord3912167717/002/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: own.example.com: failed wake's generation 0 is proven stopped; lease given back
2026/10/24 21:31:39 provision: a.shop.example.com: import sd_3vuhuk6ovoheqq4chzis5unm4q rolled back to its original route
2026/10/24 21:31:39 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:39 activator: worker bot.x.internal NOT respawned (converging onto a new build): instance config unavailable on this box: cannot open sealed config — standing down and releasing its lease
2026/10/24 21:31:39 activator: bot.x.internal hibernated
2026/10/24 21:31:39 provision: a.shop.example.com: import sd_3vuhuk6ovoheqq4chzis5unm4q is cleaned up and its journal row retired
2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:39 activator: bot5.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/24 21:31:39 activator: bot6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/24 21:31:39 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, nor the instance's own build v1: a failure record holds this build: v1
2026/10/24 21:31:39 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:39 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:31:39 activator: worker bot.x.internal NOT respawned (exited): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/24 21:31:39 activator: bot.x.internal hibernated
2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/24 21:31:39 activator: one.example.com hibernated
2026/10/24 21:31:39 provision: a.shop.example.com: swapped for import sd_sofvdtjd7r4megtdbwje6n42ay (the original route is journaled)
2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:39 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/24 21:31:39 activator: a.shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: a.shop.example.com hibernated
2026/10/24 21:31:39 activator: shutdown: out of time with 1 failed wake(s) not proven stopped, their leases still held: unproven.example.com (generation 2017)
2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:39 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:39 activator: bot.x.internal hibernated
2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 provision: a.shop.example.com: import sd_sofvdtjd7r4megtdbwje6n42ay rolled back to its original route
2026/10/24 21:31:39 provision: a.shop.example.com: import sd_sofvdtjd7r4megtdbwje6n42ay is cleaned up and its journal row retired
2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: processes remained; the sweep tries again
2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: c.example.com hibernated
2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:39 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: c.example.com hibernated
2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: c.example.com hibernated
2026/10/24 21:31:39 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:39 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:31:39 activator: worker bot.x.internal NOT respawned (exited): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: swap.example.com hibernated
2026/10/24 21:31:39 activator: h.example.com: kill carlos-instance-gen@h.example.com:2053.service after a failed start: stop timed out
2026/10/24 21:31:39 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: the unit failed to start; activator: generation start failed and its termination is unproven: stop timed out
2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: swap.example.com hibernated
2026/10/24 21:31:39 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: c.example.com hibernated
2026/10/24 21:31:39 activator: one.example.com: generation 2057 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:39 activator: one.example.com: generation 2060 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: seeding shop.example.com: /tmp/TestARefusedLaunchLeavesNoBuildStateRecordseed_refused_beside_a3244348707/002/versions/v1/app seed --db /tmp/TestARefusedLaunchLeavesNoBuildStateRecordseed_refused_beside_a3244348707/003/shop.example.com.db
2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: c.example.com hibernated
2026/10/24 21:31:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:39 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:39 activator: x.example hibernated
2026/10/24 21:31:39 activator: bot7.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/24 21:31:39 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, and the instance's own build v1 cannot be named: staged copy unreadable
2026/10/24 21:31:39 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: swap.example.com hibernated
2026/10/24 21:31:39 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: cgroup: the host's parent cgroup holds a process directly: holds 1 process(es)
2026/10/24 21:31:39 activator: c.example.com idle 1h0m0s — hibernating
2026/10/24 21:31:39 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 21:31:39 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:31:39 activator: worker bot.x.internal NOT respawned (hit its max_run): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/24 21:31:39 activator: bot.x.internal hibernated
2026/10/24 21:31:39 activator: c.example.com hibernated
2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:39 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 21:31:39 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 21:31:39 activator: bot-a.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 21:31:39 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 21:31:39 activator: bot-b.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: SIDECAR for partial-sidecar.example failed to start: sidecar started then failed — retaining returned process
2026/10/24 21:31:39 activator: partial-sidecar.example: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:39 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:39 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 21:31:39 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 21:31:39 activator: bot.x.internal hibernated
2026/10/24 21:31:39 activator: partial-sidecar.example hibernated
2026/10/24 21:31:39 activator: worker bot-b.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:39 activator: worker bot-b.x.internal respawned (converging onto a new build), now running v2
2026/10/24 21:31:39 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 21:31:39 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 21:31:39 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 21:31:39 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 21:31:39 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: c.example.com hibernated
2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: instance c.example.com exited unexpectedly — recycling
2026/10/24 21:31:40 activator: c.example.com hibernated
2026/10/24 21:31:40 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:40 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 21:31:40 activator: one.example.com: generation 2077 of v3: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:31:40 activator: worker bot-a.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:40 activator: worker bot-a.x.internal respawned (converging onto a new build), now running v2
2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: swap.example.com hibernated
2026/10/24 21:31:40 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: swap.example.com hibernated
2026/10/24 21:31:40 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 21:31:40 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 21:31:40 activator: one.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:31:40 activator: c.example.com hibernated
2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:40 activator: seeding shop.example.com: /tmp/TestARefusedLaunchLeavesNoBuildStateRecorda_seed_that_ran_is_re2455287812/001/versions/v1/app seed --db /tmp/TestARefusedLaunchLeavesNoBuildStateRecorda_seed_that_ran_is_re2455287812/002/shop.example.com.db
2026/10/24 21:31:40 activator: shop.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 21:31:40 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: x.example hibernated
2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: c.example.com idle 1h0m0s — hibernating
2026/10/24 21:31:40 activator: c.example.com hibernated
2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:40 activator: bot6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:40 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: x.example hibernated
2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:40 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:40 activator: one.example.com idle 1h0m0s — hibernating
2026/10/24 21:31:40 activator: one.example.com hibernated
2026/10/24 21:31:40 activator: SIDECAR restart for partial-sidecar.example failed: sidecar started then failed — retaining returned process
2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:40 activator: partial-sidecar.example hibernated
2026/10/24 21:31:40 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:40 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 21:31:40 activator: bot.x.internal hibernated
2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 21:31:40 activator: stop c.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:40 activator: one.example.com: generation 2105 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:40 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:40 activator: LEASE LOST for c.example.com (lost) — standing down
2026/10/24 21:31:40 activator: c.example.com hibernated
2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:40 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 21:31:40 activator: sidecar staged for awake tenant.test — adopting
2026/10/24 21:31:40 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/24 21:31:40 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 21:31:40 activator: c.example.com hibernated
2026/10/24 21:31:40 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:40 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:40 activator: sidecar staged for awake tenant.test — adopting
2026/10/24 21:31:40 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:40 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: swap.example.com hibernated
2026/10/24 21:31:40 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a cgroup that took cgroup.kill could not be made fresh before a spawn: carlos-exec@h.example.com: device or resource busy
2026/10/24 21:31:40 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:40 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:40 activator: two.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:40 activator: x.example hibernated
2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: c.example.com idle 1h0m0s — hibernating
2026/10/24 21:31:40 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 21:31:40 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 21:31:40 activator: bot7.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:40 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/24 21:31:40 activator: c.example.com hibernated
2026/10/24 21:31:40 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 21:31:40 activator: bot.x.internal hibernated
2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:40 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:40 activator: undelegated-wake.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: SIDECAR restart for one.example.com failed: swap: lease lost (next attempt in 2s)
2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:40 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:40 activator: bot.x.internal hibernated
2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:40 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:40 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 21:31:40 activator: bot.x.internal hibernated
2026/10/24 21:31:40 activator: one.example.com: generation 2122 of v3: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: plain.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: one.example.com: generation 2124 of v3: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:40 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, seed 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:31:40 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: c.example.com serving v1 but v2 is adopted — cycling onto the new build
2026/10/24 21:31:40 activator: c.example.com hibernated
2026/10/24 21:31:40 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:31:40 activator: always-on sidecar bot.x.internal is not running — starting it
2026/10/24 21:31:40 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 21:31:40 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:40 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:40 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 21:31:40 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:41 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:41 activator: kass.example.com idle 11s — hibernating
2026/10/24 21:31:41 activator: kass.example.com hibernated
2026/10/24 21:31:41 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:41 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:31:41 activator: SIDECAR restart for one.example.com failed: the build state could not be written, so this build was not started: bucket refused the build state (next attempt in 2s)
2026/10/24 21:31:41 activator: kass.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: c.example.com idle 1h0m0s — hibernating
2026/10/24 21:31:41 activator: c.example.com hibernated
2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:41 activator: always-on sidecar bot00.x.internal is not running — starting it
2026/10/24 21:31:41 activator: always-on sidecar bot01.x.internal is not running — starting it
2026/10/24 21:31:41 activator: always-on sidecar bot02.x.internal is not running — starting it
2026/10/24 21:31:41 activator: always-on sidecar bot03.x.internal is not running — starting it
2026/10/24 21:31:41 activator: always-on sidecar bot04.x.internal is not running — starting it
2026/10/24 21:31:41 activator: always-on sidecar bot05.x.internal is not running — starting it
2026/10/24 21:31:41 activator: always-on sidecar bot06.x.internal is not running — starting it
2026/10/24 21:31:41 activator: always-on sidecar bot07.x.internal is not running — starting it
2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:41 activator: x.example hibernated
2026/10/24 21:31:41 activator: two.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/24 21:31:41 activator: bot07.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: bot02.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: bot04.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: bot03.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: bot05.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: bot06.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: bot01.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: bot00.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: always-on sidecar bot08.x.internal is not running — starting it
2026/10/24 21:31:41 activator: always-on sidecar bot09.x.internal is not running — starting it
2026/10/24 21:31:41 activator: always-on sidecar bot10.x.internal is not running — starting it
2026/10/24 21:31:41 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:41 activator: bot09.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: bot08.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: bot10.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: bot6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/24 21:31:41 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: swap.example.com: v1 not started: its build state could not be written: bucket: 503
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: no-host-proof.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 21:31:41 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 21:31:41 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:31:41 activator: worker bot.x.internal respawned (hit its max_run), now running
2026/10/24 21:31:41 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 21:31:41 activator: bot.x.internal hibernated
2026/10/24 21:31:41 activator: bot.x.internal hibernated
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: instance control: 502 mid-roll; the sweep tries again
2026/10/24 21:31:41 activator: bot.x.internal hibernated
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: swap.example.com hibernated
2026/10/24 21:31:41 activator: swap.example.com: generation 2148 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running
2026/10/24 21:31:41 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: c.example.com hibernated
2026/10/24 21:31:41 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:41 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/24 21:31:41 activator: x.example hibernated
2026/10/24 21:31:41 activator: one.example.com: generation 2151 of v3: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:41 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:31:41 activator: SIDECAR for one.example.com failed to start: the build state could not be written, so this build was not started: bucket refused the build state — serving without it (the sweep retries)
2026/10/24 21:31:41 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:41 activator: shutdown-pending.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: stuck.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: shutdown-pending.example.com idle 2m0s — hibernating
2026/10/24 21:31:41 activator: stop shutdown-pending.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: swap.example.com hibernated
2026/10/24 21:31:41 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a generation's limits could not be applied: h.example.com: the placement shim is not available
2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:41 activator: stuck.example.com idle 2m0s — hibernating
2026/10/24 21:31:41 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 21:31:41 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:41 activator: x.example hibernated
2026/10/24 21:31:41 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/24 21:31:41 activator: bot.x.internal hibernated
2026/10/24 21:31:41 activator: fence.example hibernated
2026/10/24 21:31:41 activator: stop fence.example: stop refused
2026/10/24 21:31:41 activator: fence.example hibernated
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: swap.example.com hibernated
2026/10/24 21:31:41 activator: bot7.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/24 21:31:41 activator: no-host-proof.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: squatted.example.com: child exited during the health wait (external process serving its socket?) — not normalizing the socket mode
before
2026/10/24 21:31:41 activator: h.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: h.example.com idle 2m0s — hibernating
2026/10/24 21:31:41 activator: stop h.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 21:31:41 activator: shutdown-pending.example.com hibernated
2026/10/24 21:31:41 activator: h.example.com hibernated
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: swap.example.com hibernated
2026/10/24 21:31:41 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:31:41 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:41 activator: bot.x.internal hibernated
2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running
2026/10/24 21:31:41 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: c.example.com is no longer hibernate-managed — releasing
2026/10/24 21:31:41 activator: c.example.com hibernated
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: instance control: 502 mid-roll; the sweep tries again
2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running
2026/10/24 21:31:41 activator: stop fence.example: stop refused
2026/10/24 21:31:41 activator: fence.example hibernated
2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:41 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/24 21:31:41 activator: x.example hibernated
2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running
2026/10/24 21:31:41 activator: seeding h.example.com: /tmp/TestPlan14GenerationBackendAuxiliaryBoundaryclaim=falseseedpi2576486630/001/live/h.example.com seed --db /tmp/nb2326995521/data/instances/h.example.com/h.example.com.db
2026/10/24 21:31:41 activator: seeded h.example.com
2026/10/24 21:31:41 activator: seeding h.example.com: /tmp/TestPlan14GenerationBackendAuxiliaryBoundaryclaim=falseseedpi785541684/001/live/h.example.com seed --db /tmp/nb2470485756/data/instances/h.example.com/h.example.com.db
2026/10/24 21:31:41 activator: seeded h.example.com
2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running
2026/10/24 21:31:41 activator: shop.example.com: seed placement not recorded yet: record unreachable
2026/10/24 21:31:41 activator: shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: worker bot.x.internal reported idle but is always-on (hibernate:false) — ignoring
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 4s)
2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running
2026/10/24 21:31:41 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: fence.example hibernated
2026/10/24 21:31:41 activator: worker bot.x.internal exited — restarting (backoff now 8s)
2026/10/24 21:31:41 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:31:41 activator: worker bot.x.internal respawned (exited), now running
2026/10/24 21:31:41 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:41 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:41 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:42 activator: fence.example hibernated
2026/10/24 21:31:42 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 21:31:42 activator: WORKER bot.x.internal failed to respawn (converging onto a new build): exec: live symlink gone — the sweep's crash restart takes it from here
2026/10/24 21:31:42 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 21:31:42 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:42 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 21:31:42 activator: worker bot.x.internal respawned (exited), now running v2
2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:42 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 21:31:42 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 21:31:42 activator: no-host-proof.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: bot.x.internal idle 11s — hibernating
2026/10/24 21:31:42 activator: bot.x.internal hibernated
2026/10/24 21:31:42 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 21:31:42 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:42 activator: one.example.com: generation 2179 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:42 activator: bot.x.internal idle 11s — hibernating
2026/10/24 21:31:42 activator: bot.x.internal hibernated
2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 21:31:42 activator: fenced.example.com: failed wake's generation 2181 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:42 activator: c.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: c.example.com hibernated
2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:42 activator: f6-layout.example.com: failed wake's generation 2184 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 21:31:42 activator: f6-layout.example.com: failed wake's generation 2184 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:42 activator: fenced.example.com: failed wake's generation 2181 is proven stopped; lease given back
2026/10/24 21:31:42 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:42 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 21:31:42 activator: nested-stop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: nested-stop.example.com idle 2m0s — hibernating
2026/10/24 21:31:42 activator: nested-stop.example.com hibernated
2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: untimed.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: f6.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 21:31:42 activator: exited-flat.example.com: failed wake's generation 2191 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 21:31:42 activator: exited-flat.example.com: failed wake's generation 2191 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:42 activator: untimed.bab.oncarlos.com idle 16m0s — hibernating
2026/10/24 21:31:42 activator: untimed.bab.oncarlos.com hibernated
2026/10/24 21:31:42 activator: forked.example.com: failed wake's generation 2192 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:42 activator: stop fence.example: private backend detail
2026/10/24 21:31:42 activator: fence.example hibernated
2026/10/24 21:31:42 activator: forked.example.com: failed wake's generation 2192 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 21:31:42 activator: pending-f6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: pending-f6.example.com idle 2m0s — hibernating
2026/10/24 21:31:42 activator: stop pending-f6.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: ledgerpay.bde.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: forked.example.com: failed wake's generation 2192 is proven stopped; lease given back
2026/10/24 21:31:42 activator: stop pending-f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 21:31:42 activator: seeding seedpin.example.com: /tmp/TestADeclinedFlatSeedRunsThePinnedFile2559396634/001/versions/v1/app seed --db /tmp/TestADeclinedFlatSeedRunsThePinnedFile2559396634/002/seedpin.example.com.db
2026/10/24 21:31:42 activator: seeded seedpin.example.com
2026/10/24 21:31:42 activator: pending-f6.example.com hibernated
2026/10/24 21:31:42 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 21:31:42 activator: fence.example hibernated
2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:42 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:42 activator: h.example.com: carlos-instance-gen@h.example.com:8.service launched in 0s, healthy 0s later
2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:42 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it stays down until a release attempt or a wake starts it
2026/10/24 21:31:42 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:42 activator: h.example.com: carlos-instance-gen@h.example.com:9.service launched in 0s, healthy 0s later
2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:42 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:42 activator: f6-layout.example.com: failed wake's generation 2202 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 21:31:42 activator: f6-layout.example.com: failed wake's generation 2202 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:42 activator: bots.example.com idle 11s — hibernating
2026/10/24 21:31:42 activator: bots.example.com hibernated
2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:42 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:42 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:42 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:42 activator: fenced.example.com: failed wake's generation 2203 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:42 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:42 activator: bot.shop.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: swap.example.com hibernated
2026/10/24 21:31:42 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:42 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it recycles its captured build v1
2026/10/24 21:31:42 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:42 activator: swap.example.com hibernated
2026/10/24 21:31:42 activator: exited-foreign.example.com: failed wake's generation 2208 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 21:31:42 activator: exited-foreign.example.com: failed wake's generation 2208 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:42 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/24 21:31:42 activator: fenced.example.com: failed wake's generation 2203 is proven stopped; lease given back
2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:42 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:42 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:42 activator: h.example.com: carlos-instance-gen@h.example.com:1.service launched in 0s, healthy 0s later
2026/10/24 21:31:42 activator: sidecar for bots.example.com was down — restarted (backoff now 8s)
2026/10/24 21:31:42 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:42 activator: h.example.com: carlos-instance-gen@h.example.com:6.service launched in 0s, healthy 0s later
2026/10/24 21:31:42 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 21:31:42 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 21:31:42 activator: stop sidecar fence.example: private backend detail
2026/10/24 21:31:42 activator: fence.example hibernated
2026/10/24 21:31:42 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:42 activator: h.example.com: unit restarted outside the activator (invocation someone-else, ours inv7); stopping it
2026/10/24 21:31:42 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:42 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 21:31:42 activator: h.example.com: unit restarted outside the activator (invocation inv2, ours inv1); stopping it
2026/10/24 21:31:42 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: h.example.com: started carlos-instance-gen@h.example.com:2217.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:43 activator: one.example.com: generation 2217 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:43 activator: one.example.com: woke in 1ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 1ms, replicate 0s
2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:43 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:43 activator: one.example.com hibernated
2026/10/24 21:31:43 activator: bots.example.com idle 11s — hibernating
2026/10/24 21:31:43 activator: bots.example.com hibernated
2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:43 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:43 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 21:31:43 activator: h.example.com: processes remain after failed; not releasing
2026/10/24 21:31:43 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: one.example.com: live/one.example.com names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/24 21:31:43 activator: f6-layout.example.com: failed wake's generation 2221 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 21:31:43 activator: f6-layout.example.com: failed wake's generation 2221 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:43 activator: worker bot.x.internal serving v1 but live symlink is now v3 — respawning onto the new build
2026/10/24 21:31:43 activator: worker bot.x.internal respawned (converging onto a new build), now running v3
2026/10/24 21:31:43 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:43 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 21:31:43 activator: SIDECAR for botless.example.com failed to start: exec: live symlink gone — serving without it (the sweep retries)
2026/10/24 21:31:43 activator: botless.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:43 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 21:31:43 activator: SIDECAR restart for botless.example.com failed: exec: live symlink gone (next attempt in 2s)
2026/10/24 21:31:43 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/24 21:31:43 activator: one.example.com hibernated
2026/10/24 21:31:43 activator: waking.example: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: waking.example hibernated
2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com hibernated
2026/10/24 21:31:43 activator: swap.example.com: generation 2225 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 21:31:43 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: the running child keeps running
2026/10/24 21:31:43 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 21:31:43 activator: sidecar for botless.example.com was down — restarted (backoff now 4s)
2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:43 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/24 21:31:43 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 21:31:43 activator: one.example.com hibernated
2026/10/24 21:31:43 activator: f6.example.com: failed wake's generation 0 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: reborn.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: reborn.example.com idle 11s — hibernating
2026/10/24 21:31:43 activator: reborn.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/24 21:31:43 activator: reborn.example.com hibernated
2026/10/24 21:31:43 activator: swap.example.com hibernated
2026/10/24 21:31:43 activator: final checkpoint fence.example: private backend detail
2026/10/24 21:31:43 activator: fence.example hibernated
2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: reborn.example.com has a scheduled delivery due — waking it
2026/10/24 21:31:43 activator: reborn.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 21:31:43 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 21:31:43 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:43 activator: shop.example.com: generation 4's child is not proven empty: kill gen: timeout
2026/10/24 21:31:43 activator: old.shop.example.com: route refreshed: db "/data/instances/old/old.shop.example.com.db" -> "" socket "/run/carlos/instances/old/app.sock" -> ""
2026/10/24 21:31:43 activator: wake old.shop.example.com: old.shop.example.com: the route changed before the wake: the activator no longer manages it
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: h.example.com: kill carlos-instance-gen@h.example.com:2241.service after a failed start: stop timed out
2026/10/24 21:31:43 activator: one.example.com: wake failed and its generation is not proven stopped: start one.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2241.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:43 activator: LEASE LOST for bot.x.internal (lease renewal failed) — standing down
2026/10/24 21:31:43 activator: bot.x.internal hibernated
2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:43 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:43 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 21:31:43 activator: f6-layout.example.com: failed wake's generation 2242 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 21:31:43 activator: f6-layout.example.com: failed wake's generation 2242 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 21:31:43 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com hibernated
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/24 21:31:43 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:43 activator: one.example.com: generation 2249 of v1: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:43 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 21:31:43 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/24 21:31:43 activator: one.example.com hibernated
2026/10/24 21:31:43 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/24 21:31:43 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 8s)
2026/10/24 21:31:43 activator: sidecar for swap.example.com was down — restarted (backoff now 16s)
2026/10/24 21:31:43 activator: one.example.com: generation 2252 of v2: record 0s, start 20ms, healthy 0s later
2026/10/24 21:31:43 activator: one.example.com: woke in 20ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 20ms, replicate 0s
2026/10/24 21:31:43 activator: fence.example hibernated
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: proven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: proven.example.com idle 2m0s — hibernating
2026/10/24 21:31:43 activator: proven.example.com hibernated
2026/10/24 21:31:43 activator: swap.example.com hibernated
2026/10/24 21:31:43 activator: undelegated.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 21:31:43 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:43 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:43 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 21:31:43 activator: undelegated.example.com idle 2m0s — hibernating
2026/10/24 21:31:43 activator: undelegated.example.com: stopped with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 21:31:43 activator: undelegated.example.com hibernated
2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 21:31:43 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 21:31:43 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:43 activator: h.example.com: carlos-instance-gen@h.example.com:6.service launched in 0s, healthy 0s later
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: one.example.com: generation 2262 of v3: record 0s, start 20ms, healthy 0s later
2026/10/24 21:31:43 activator: one.example.com: woke in 20ms: checks 0s, lease 0s, restore 0s, identity 0s, spawn 20ms, replicate 0s
2026/10/24 21:31:43 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 21:31:43 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 21:31:43 activator: shop.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 21:31:43 activator: SIDECAR for swap.example.com is not proven gone for its replacement; tracking it (the sweep re-proves it)
2026/10/24 21:31:43 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 21:31:43 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com hibernated
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: bots.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:43 activator: swap.example.com hibernated
2026/10/24 21:31:43 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2486839030/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2486839030/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 21:31:43 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2486839030/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2486839030/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 21:31:43 activator: instance bots.example.com exited unexpectedly — recycling
2026/10/24 21:31:43 activator: bots.example.com hibernated
2026/10/24 21:31:43 activator: swap.example.com: a discarded SIDECAR is not proven gone (activator: stop unproven; processes remain); keeping it tracked
2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:43 activator: h.example.com: carlos-instance-gen@h.example.com:9.service launched in 0s, healthy 0s later
2026/10/24 21:31:43 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:43 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 21:31:44 activator: swap.example.com hibernated
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com hibernated
2026/10/24 21:31:44 activator: swap.example.com: generation 2276 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 21:31:44 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:44 activator: late.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: deaf.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: late.example.com idle 11s — hibernating
2026/10/24 21:31:44 activator: fence.example hibernated
2026/10/24 21:31:44 activator: late.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/24 21:31:44 activator: late.example.com hibernated
2026/10/24 21:31:44 activator: deaf.example.com idle 11s — hibernating
2026/10/24 21:31:44 activator: next-due probe deaf.example.com: dial unix: no such file — recording none (it sleeps until a request)
2026/10/24 21:31:44 activator: deaf.example.com hibernated
2026/10/24 21:31:44 activator: due.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: own.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:44 activator: due.example.com idle 11s — hibernating
2026/10/24 21:31:44 activator: due.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/24 21:31:44 activator: due.example.com hibernated
2026/10/24 21:31:44 activator: late.example.com has a scheduled delivery due — waking it
2026/10/24 21:31:44 activator: late.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: own.example.com idle 1m0s — hibernating
2026/10/24 21:31:44 activator: own.example.com hibernated
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 21:31:44 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:44 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912 (build 0s, prepare (dirs 0s, admin token 0s, instance key 0s, env 0s, control socket 0s, key gate 0s, logs 0s) 0s, launch file 0s, gate 0s, root start 0s)
2026/10/24 21:31:44 activator: h.example.com: carlos-instance@h.example.com.service launched in 0s, healthy 0s later
2026/10/24 21:31:44 activator: bot.x.internal hit its max_run of 1m0s — stopping
2026/10/24 21:31:44 activator: bot.x.internal hibernated
2026/10/24 21:31:44 activator: swap.example.com hibernated
2026/10/24 21:31:44 activator: due.example.com has a scheduled delivery due — waking it
2026/10/24 21:31:44 activator: due.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:44 activator: bot.x.internal hibernated
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/24 21:31:44 activator: one.example.com hibernated
2026/10/24 21:31:44 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished4121104141/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished4121104141/001/zdd: not a directory; the sweep retries
2026/10/24 21:31:44 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished4121104141/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished4121104141/001/zdd: not a directory; the sweep retries
2026/10/24 21:31:44 activator: due.example.com idle 11s — hibernating
2026/10/24 21:31:44 activator: due.example.com hibernated
2026/10/24 21:31:44 activator: one.example.com idle 11s — hibernating
2026/10/24 21:31:44 activator: one.example.com hibernated
2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor3121912923/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor3121912923/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 21:31:44 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor3121912923/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor3121912923/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com hibernated
2026/10/24 21:31:44 activator: swap.example.com: generation 2303's child is not proven empty: the child still holds a process
2026/10/24 21:31:44 activator: swap.example.com: failed wake's generation 2303 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: one.example.com hibernated
2026/10/24 21:31:44 activator: swap.example.com: failed wake's generation 2303 is proven stopped; lease given back
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com hibernated
2026/10/24 21:31:44 activator: swap.example.com: generation 2307 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com hibernated
2026/10/24 21:31:44 activator: swap.example.com: failed wake's generation 2309 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:44 activator: faroff.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: failed wake's generation 2309 is proven stopped; lease given back
2026/10/24 21:31:44 activator: faroff.bab.oncarlos.com idle 16m0s — hibernating
2026/10/24 21:31:44 activator: faroff.bab.oncarlos.com hibernated
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com hibernated
2026/10/24 21:31:44 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/24 21:31:44 activator: stop fence.example: activator: stop evidence unavailable
2026/10/24 21:31:44 activator: fence.example hibernated
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com hibernated
2026/10/24 21:31:44 activator: LEASE HELD for swap.example.com by "box-b" — a previous incarnation hasn't released; refusing to wake (retry after 1m0s)
2026/10/24 21:31:44 activator: swap.example.com: hold's pinned wake refused or abandoned: instance lease held by "box-b" (retry in 1m0s)
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: flip.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: one.example.com idle 1m0s — hibernating
2026/10/24 21:31:44 activator: unproven.example.com: generation 2321's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:44 activator: unproven.example.com: failed wake's generation 2321 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:44 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 21:31:44 activator: flip.example.com is no longer hibernate-managed — releasing
2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: flip.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/24 21:31:44 activator: flip.example.com hibernated
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: unproven.example.com: generation 2321's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:44 activator: mid.example.com has a scheduled delivery due — waking it
2026/10/24 21:31:44 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 21:31:44 activator: one.example.com hibernated
2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake543479007/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake543479007/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 21:31:44 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake543479007/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake543479007/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 21:31:44 activator: f6.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: one.example.com hibernated
2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: unproven.example.com: failed wake's generation 2321 is proven stopped; lease given back
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: LEASE LOST for one.example.com (lease renewal failed) — standing down
2026/10/24 21:31:44 activator: one.example.com hibernated
2026/10/24 21:31:44 activator: f6.example.com idle 2m0s — hibernating
2026/10/24 21:31:44 activator: stop f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 21:31:44 activator: stop f6.example.com: its host cgroup is not proven empty; keeping the lease until it is
2026/10/24 21:31:44 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:44 activator: mid.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: mid.example.com is no longer hibernate-managed — releasing (wake finished first)
2026/10/24 21:31:44 activator: mid.example.com hibernated
2026/10/24 21:31:44 activator: bot.x.internal idle 11s — hibernating
2026/10/24 21:31:44 activator: bot.x.internal hibernated
2026/10/24 21:31:44 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: unproven.example.com hibernated
2026/10/24 21:31:44 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/24 21:31:44 activator: swap.example.com hibernated
2026/10/24 21:31:44 activator: f6.example.com hibernated
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: two.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 21:31:44 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 21:31:44 activator: one.example.com hibernated
2026/10/24 21:31:44 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:44 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: barrier.example.com hibernated
2026/10/24 21:31:45 activator: two.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 21:31:45 activator: two.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 21:31:45 activator: one.example.com hibernated
2026/10/24 21:31:45 activator: f6.example.com: failed wake's generation 4 ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 21:31:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: the backend ran a different generation from the hold's choice: generation 2350 of v9 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_2061185657/004/versions/v2/app"; the hold chose v2 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_2061185657/004/versions/v2/app"
2026/10/24 21:31:45 activator: one.example.com idle 11s — hibernating
2026/10/24 21:31:45 activator: one.example.com hibernated
2026/10/24 21:31:45 activator: final checkpoint fence.example: activator: stop evidence unavailable
2026/10/24 21:31:45 activator: fence.example hibernated
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: restore swap.example.com: replica unreadable
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: socket dir: permission denied
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 21:31:45 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 21:31:45 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1542582095/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1542582095/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 21:31:45 activator: x.example: hold's pinned wake refused or abandoned: x.example: refusing to wake: staged build not published before its exec
2026/10/24 21:31:45 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1542582095/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1542582095/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 21:31:45 activator: sidecar for swap.example.com was down — restarted (backoff now 4s)
2026/10/24 21:31:45 activator: tickheld.bab.oncarlos.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: unproven.example.com: generation 2362's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:45 activator: unproven.example.com: failed wake's generation 2362 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:45 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: h: start unit: instance control: generation termination unproven: the unit exited before it was observed
2026/10/24 21:31:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: unproven.example.com: generation 2362's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:45 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/24 21:31:45 activator: barrier.example.com hibernated
2026/10/24 21:31:45 activator: tickheld.bab.oncarlos.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 21:31:45 activator: tickheld.bab.oncarlos.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: bot.x.internal idle 31s — hibernating
2026/10/24 21:31:45 activator: bot.x.internal hibernated
2026/10/24 21:31:45 activator: one.example.com idle 11s — hibernating
2026/10/24 21:31:45 activator: one.example.com hibernated
2026/10/24 21:31:45 activator: unproven.example.com: failed wake's generation 2362 is proven stopped; lease given back
2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:45 activator: bot.x.internal idle 1m31s — hibernating
2026/10/24 21:31:45 activator: bot.x.internal hibernated
2026/10/24 21:31:45 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: unproven.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 21:31:45 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: one.example.com idle 5m1s — hibernating
2026/10/24 21:31:45 activator: one.example.com hibernated
2026/10/24 21:31:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/24 21:31:45 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 21:31:45 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: fence.example hibernated
2026/10/24 21:31:45 activator: barrier.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: staging: staged digest mismatch: have be0041e1a437dbea5b67dfe58bad04a0fc30d5036d469429f3534f39ec53d8b5, want c3bae8fb3a8bf395783d7f7f6ffcd4aadc797c95ce24b7e8e2a98af6b63d8f6d
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: start g: activator: generation start failed and its termination is unproven
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 21:31:45 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 21:31:45 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:45 activator: fence.example hibernated
2026/10/24 21:31:45 activator: bot.x.internal idle 11s — hibernating
2026/10/24 21:31:45 activator: bot.x.internal hibernated
2026/10/24 21:31:45 activator: one.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/24 21:31:45 activator: fence.example hibernated
2026/10/24 21:31:45 activator: one.example.com idle 11s — hibernating
2026/10/24 21:31:45 activator: one.example.com hibernated
2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: unproven.example.com: failed wake's generation 2398 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:45 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/24 21:31:45 activator: barrier.example.com hibernated
2026/10/24 21:31:45 activator: unproven.example.com: retry stop of failed wake's generation 2398: cgroup.kill: the child is not empty
2026/10/24 21:31:45 activator: bot.x.internal: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, settle 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: bot.x.internal idle 1m1s — hibernating
2026/10/24 21:31:45 activator: bot.x.internal hibernated
2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: unproven.example.com: generation 2398's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:45 activator: unproven.example.com: failed wake's generation 2398 is proven stopped; lease given back
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: unproven.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: a stop-evidence operation holds the host
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: generation 2411 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s, sidecar 0s
2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/24 21:31:45 activator: barrier.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: swap.example.com hibernated
2026/10/24 21:31:45 activator: swap.example.com: generation 2418 of v2: record 0s, start 0s, healthy 0s later
2026/10/24 21:31:45 activator: swap.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: unproven.example.com: generation 2420's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:45 activator: unproven.example.com: failed wake's generation 2420 is not proven stopped; keeping the lease until it is
2026/10/24 21:31:45 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 21:31:45 activator: unproven.example.com: generation 2420's child is not proven empty: the generation's child still holds a process
2026/10/24 21:31:45 activator: unproven.example.com: failed wake's generation 2420 is proven stopped; lease given back
2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/24 21:31:45 activator: barrier.example.com hibernated
2026/10/24 21:31:45 activator: unproven.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: unproven.example.com hibernated
2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: barrier.example.com hibernated
2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: barrier.example.com hibernated
2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:45 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:45 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:46 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:46 activator: barrier.example.com hibernated
2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:46 activator: barrier.example.com hibernated
2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:46 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/24 21:31:46 activator: barrier.example.com hibernated
2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:46 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/24 21:31:46 activator: barrier.example.com hibernated
2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:46 activator: barrier.example.com hibernated
2026/10/24 21:31:46 activator: barrier.example.com: woke in 0s: checks 0s, lease 0s, restore 0s, identity 0s, spawn 0s, replicate 0s
2026/10/24 21:31:46 activator: barrier.example.com hibernated
FAIL
FAIL github.com/carlosframework/platform/internal/activator 57.972s
ok github.com/carlosframework/platform/internal/agentsock 0.023s
ok github.com/carlosframework/platform/internal/alerting 0.003s
ok github.com/carlosframework/platform/internal/analytics 0.034s
ok github.com/carlosframework/platform/internal/appconfig 3.034s
ok github.com/carlosframework/platform/internal/apperr 0.164s
ok github.com/carlosframework/platform/internal/applog 0.291s
ok github.com/carlosframework/platform/internal/awsinv 0.009s
ok github.com/carlosframework/platform/internal/boxchoice 0.008s
ok github.com/carlosframework/platform/internal/boxid 0.919s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.008s
ok github.com/carlosframework/platform/internal/caps 0.019s
ok github.com/carlosframework/platform/internal/certwarm 0.004s
ok github.com/carlosframework/platform/internal/cgroup 0.728s
ok github.com/carlosframework/platform/internal/changelog 0.637s
ok github.com/carlosframework/platform/internal/claimactivation 1.167s
ok github.com/carlosframework/platform/internal/clamd 0.055s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 8.035s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.017s
ok github.com/carlosframework/platform/internal/dbquery 0.010s
ok github.com/carlosframework/platform/internal/dbreads 2.096s
? github.com/carlosframework/platform/internal/dbreads/dbreadstest [no test files]
ok github.com/carlosframework/platform/internal/dbverify 0.006s
ok github.com/carlosframework/platform/internal/dnsrec 0.002s
ok github.com/carlosframework/platform/internal/docsite 0.019s
ok github.com/carlosframework/platform/internal/domains 2.872s
ok github.com/carlosframework/platform/internal/doorkey 0.007s
ok github.com/carlosframework/platform/internal/economics 0.005s
ok github.com/carlosframework/platform/internal/edge 45.260s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.010s
ok github.com/carlosframework/platform/internal/fleetapi 0.209s
ok github.com/carlosframework/platform/internal/gate 1.822s
ok github.com/carlosframework/platform/internal/genselect 0.033s
ok github.com/carlosframework/platform/internal/geo 0.021s
ok github.com/carlosframework/platform/internal/geoip 0.042s
ok github.com/carlosframework/platform/internal/historyreadback 0.285s
ok github.com/carlosframework/platform/internal/host 140.016s
ok github.com/carlosframework/platform/internal/hostactors 6.608s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.008s
ok github.com/carlosframework/platform/internal/hostbox 3.145s
ok github.com/carlosframework/platform/internal/hostclaim 0.759s
ok github.com/carlosframework/platform/internal/hostenv 0.009s
ok github.com/carlosframework/platform/internal/hostevidence 0.006s
ok github.com/carlosframework/platform/internal/hostexec 5.383s
ok github.com/carlosframework/platform/internal/hostfence 0.046s
ok github.com/carlosframework/platform/internal/hostfiles 10.229s
ok github.com/carlosframework/platform/internal/hostfloor 0.017s
ok github.com/carlosframework/platform/internal/hostguard 1.077s
ok github.com/carlosframework/platform/internal/hostops 2.217s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.007s
ok github.com/carlosframework/platform/internal/hostquarantine 0.004s
ok github.com/carlosframework/platform/internal/hostrelocate 0.584s
ok github.com/carlosframework/platform/internal/hostresolve 0.177s
ok github.com/carlosframework/platform/internal/hostwrite 0.007s
ok github.com/carlosframework/platform/internal/infraview 0.005s
ok github.com/carlosframework/platform/internal/instancectl 5.250s
ok github.com/carlosframework/platform/internal/instanceexec 2.795s
ok github.com/carlosframework/platform/internal/instancefs 0.011s
ok github.com/carlosframework/platform/internal/instancekey 0.007s
ok github.com/carlosframework/platform/internal/instanceupdates 0.008s
ok github.com/carlosframework/platform/internal/layout 1.968s
ok github.com/carlosframework/platform/internal/lbhealth 0.005s
ok github.com/carlosframework/platform/internal/lease 10.096s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.537s
ok github.com/carlosframework/platform/internal/ledger 0.019s
ok github.com/carlosframework/platform/internal/mail 0.181s
ok github.com/carlosframework/platform/internal/maintenance 0.082s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.566s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.082s
ok github.com/carlosframework/platform/internal/metrics 0.006s
ok github.com/carlosframework/platform/internal/moves 21.355s
ok github.com/carlosframework/platform/internal/moves/archive 1.046s
ok github.com/carlosframework/platform/internal/objstore 0.276s
ok github.com/carlosframework/platform/internal/ops 0.179s
ok github.com/carlosframework/platform/internal/park 0.230s
ok github.com/carlosframework/platform/internal/plans 0.006s
ok github.com/carlosframework/platform/internal/privatefile 0.005s
ok github.com/carlosframework/platform/internal/provision 14.123s
ok github.com/carlosframework/platform/internal/publicrelease 0.016s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.646s
ok github.com/carlosframework/platform/internal/relay 7.223s
ok github.com/carlosframework/platform/internal/release 24.543s
ok github.com/carlosframework/platform/internal/rollout 3.703s
ok github.com/carlosframework/platform/internal/rolloutwords 0.004s
ok github.com/carlosframework/platform/internal/rootaws 0.009s
ok github.com/carlosframework/platform/internal/rootparticipation 0.060s
ok github.com/carlosframework/platform/internal/rootstate 0.030s
ok github.com/carlosframework/platform/internal/runs 0.538s
ok github.com/carlosframework/platform/internal/s3compat 0.005s
ok github.com/carlosframework/platform/internal/safefile 0.006s
ok github.com/carlosframework/platform/internal/schedule 2.760s
ok github.com/carlosframework/platform/internal/seeds 0.014s
ok github.com/carlosframework/platform/internal/sim/engines 2.714s
ok github.com/carlosframework/platform/internal/sim/hostlife 9.984s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.097s
ok github.com/carlosframework/platform/internal/status 0.074s
ok github.com/carlosframework/platform/internal/steering 0.006s
ok github.com/carlosframework/platform/internal/stopevidence 0.258s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.005s
ok github.com/carlosframework/platform/internal/tunnelclient 0.003s
ok github.com/carlosframework/platform/internal/unixdial 0.003s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.004s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.692s
FAIL
=== internal_console@0 (79 s)
ok github.com/carlosframework/platform/internal/console 77.341s
=== internal_console@1 (43 s)
ok github.com/carlosframework/platform/internal/console 41.397s
=== internal_console@2 (41 s)
ok github.com/carlosframework/platform/internal/console 39.236s
=== internal_console@3 (41 s)
ok github.com/carlosframework/platform/internal/console 38.764s
=== internal_console@4 (41 s)
ok github.com/carlosframework/platform/internal/console 39.632s
=== internal_console@5 (41 s)
ok github.com/carlosframework/platform/internal/console 38.693s
=== internal_console@6 (36 s)
ok github.com/carlosframework/platform/internal/console 33.923s
=== internal_console@7 (45 s)
ok github.com/carlosframework/platform/internal/console 42.988s
=== root@0 (109 s)
ok github.com/carlosframework/platform 106.914s
=== root@1 (84 s)
ok github.com/carlosframework/platform 82.232s
=== root@2 (91 s)
ok github.com/carlosframework/platform 88.947s
FAILED: rest
make: *** [Makefile:19: test] Error 1
amadan: step gate failed: exit status 2
amadan: job failed at step gate — its output is above, not at the tail of this log
Watching for changes…