carlos / platform Public

CI failed 2m 23s

Branch: design-self-serve-offline-moves
Commit: f1681c63 claude

Runner: ci-aws~paul@keymail.dev/ip-172-31-21-247
Reported by: ci-aws~paul@keymail.dev/ip-172-31-21-247
Queued: 3 days ago
Started: 3 days ago
Last change: 3 days ago
Attempt: 1 of 2

Steps run in order, one after another.

gate failed 2m 22s

.amadan/test-gate.sh
=== launchctx (0 s)
ok  	amadan.net/carlos/platform.git/launchctx	0.013s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
=== vet (1 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (142 s)
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.644s
ok  	github.com/carlosframework/platform/cmd/moveprobe	11.852s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.750s
2026/10/22 08:31:01 activator: badseed.example.com: seed for badseed.example.com: activator: seed unproven; processes remain: processes remain after a successful seed: still running; leaving the lease to its TTL
2026/10/22 08:31:01 activator: unproven.example.com: unproven.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone; leaving the lease to its TTL
2026/10/22 08:31:01 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/22 08:31:01 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof4147963793/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof4147963793/004/next-wake.example.com.db
2026/10/22 08:31:01 activator: next-wake.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof4147963793/004/next-wake.example.com.db.seed-unproven refuses it as warm until they are
2026/10/22 08:31:01 activator: next-wake.example.com: seeding next-wake.example.com: exit status 1: activator: seed unproven; processes remain (the seed's processes remain after stopping it; its database was left in place: cgroup: strict containment unsupported); leaving the lease to its TTL
2026/10/22 08:31:01 activator: next-wake.example.com: next-wake.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup: strict containment unsupported; leaving the lease to its TTL
2026/10/22 08:31:01 activator: next-wake.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/22 08:31:01 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/22 08:31:01 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof4147963793/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof4147963793/004/next-wake.example.com.db
2026/10/22 08:31:01 activator: seeded next-wake.example.com
2026/10/22 08:31:01 activator: instance one.example.com exited unexpectedly — recycling
2026/10/22 08:31:01 activator: one.example.com hibernated
2026/10/22 08:31:02 activator: LEASE HELD for held.example.com by "old-incarnation" — a previous incarnation hasn't released; refusing to wake (retry after 7s)
2026/10/22 08:31:02 activator: two.example.com hibernated
2026/10/22 08:31:02 activator: one.example.com hibernated
2026/10/22 08:31:02 activator: b.example.com hibernated
2026/10/22 08:31:02 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:02 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:02 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:02 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:02 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/22 08:31:02 activator: bot.x.internal hibernated
2026/10/22 08:31:03 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendStartMergesConfigEnvUnderOwnEnv3044231293/002/merge-precedence.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/22 08:31:03 activator: purged 1 cached response(s) for deploy.example.com on spawn
2026/10/22 08:31:03 activator: restored restored.example.com from s3://test-bucket/restored.example.com
2026/10/22 08:31:03 activator: damaged.example.com restored CORRUPT from s3://test-bucket/damaged.example.com: integrity_check: file is not a database (26) — quarantined at /tmp/TestExecBackendRestoreCorruptReplicaFailsAndQuarantines1047758299/002/damaged.example.com.db.corrupt, instance NOT started
2026/10/22 08:31:03 activator: restored damaged.example.com from s3://test-bucket/damaged.example.com
2026/10/22 08:31:03 activator: no replica for brand-new.example.com in s3://test-bucket/brand-new.example.com — booting fresh
2026/10/22 08:31:03 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1055553186/002/seeded.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/22 08:31:03 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1055553186/001/live/seeded.example.com seed --fixtures --db /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1055553186/002/seeded.example.com.db
2026/10/22 08:31:03 activator: seeded seeded.example.com
2026/10/22 08:31:03 activator: seeding custom-db.example.com: /tmp/TestExecBackendSeedHonoursRouteDBPath2429712151/001/live/custom-db.example.com seed --db /tmp/TestExecBackendSeedHonoursRouteDBPath2429712151/004/elsewhere.db
2026/10/22 08:31:03 activator: seeded custom-db.example.com
2026/10/22 08:31:03 activator: seeding badseed.example.com: /tmp/TestExecBackendSeedNonZeroExitFails2537431860/001/live/badseed.example.com seed --db /tmp/TestExecBackendSeedNonZeroExitFails2537431860/002/badseed.example.com.db
schema is corrupt
2026/10/22 08:31:03 activator: badseed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:31:03 activator: seeding hangs.example.com: /tmp/TestExecBackendSeedIsTimeBounded3684001735/001/live/hangs.example.com seed --db /tmp/TestExecBackendSeedIsTimeBounded3684001735/002/hangs.example.com.db
2026/10/22 08:31:03 activator: hangs.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:31:03 activator: seeding crashed.example.com: /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB1910316173/001/live/crashed.example.com seed --db /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB1910316173/002/crashed.example.com.db
died mid-schema
2026/10/22 08:31:03 activator: crashed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:31:03 activator: seeding existing-db.example.com: /tmp/TestExecBackendSeedFailureKeepsPreexistingDB853910656/001/live/existing-db.example.com seed --db /tmp/TestExecBackendSeedFailureKeepsPreexistingDB853910656/002/existing-db.example.com.db
nope
2026/10/22 08:31:03 activator: existing-db.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:31:03 activator: seeding leaky-seed.example.com: /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild1580889491/001/live/leaky-seed.example.com seed --db /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild1580889491/002/leaky-seed.example.com.db
2026/10/22 08:31:04 activator: seeded leaky-seed.example.com (it exited 0 but left a child holding its output pipe open for more than 200ms)
2026/10/22 08:31:04 activator: sidecar for a.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:04 activator: h.example.com: route refreshed: db "/tmp/h.example.com.db" -> "/data/instances/h.example.com/h.example.com.db" socket "/tmp/nope.sock" -> "/run/carlos/instances/h.example.com/app.sock"
2026/10/22 08:31:04 activator: stop stale.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/22 08:31:05 activator: stale.example.com hibernated
2026/10/22 08:31:05 activator: leftover.example.com: the host cgroup holds processes not proven gone: a leftover still populates the host cgroup; leaving the lease to its TTL
2026/10/22 08:31:05 activator: old-kernel.example.com: this kernel has no cgroup.kill, so the host cgroup cannot be killed; its leader's exit is the only proof there is
2026/10/22 08:31:05 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/22 08:31:05 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/22 08:31:05 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/22 08:31:05 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/22 08:31:05 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/22 08:31:05 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/22 08:31:05 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/22 08:31:05 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/22 08:31:05 activator: retry.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:05 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/22 08:31:05 activator: retry.example.com: strict group not closed: cgroup: strict containment refused
2026/10/22 08:31:05 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/22 08:31:06 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:06 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:06 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:06 activator: stuck.example.com: strict group still not cleaned up after 2 retries; spawns of this host stay refused until the agent restarts
2026/10/22 08:31:06 activator: owned.example.com: strict group not closed: cgroup: strict containment refused
2026/10/22 08:31:06 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/22 08:31:06 activator: owned.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:07 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/22 08:31:07 activator: race.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:07 activator: transition.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:07 activator: transition2.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:07 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/22 08:31:07 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/22 08:31:07 activator: kass.example.com hibernated
2026/10/22 08:31:07 activator: kass.example.com idle 11s — hibernating
2026/10/22 08:31:07 activator: kass.example.com hibernated
2026/10/22 08:31:08 activator: one.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/22 08:31:08 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:08 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:08 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:08 activator: bot.x.internal: v2 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:08 activator: worker bot.x.internal NOT respawned (attempt (release)): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/22 08:31:08 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:08 activator: link.example.com hibernated
2026/10/22 08:31:12 activator: tenant.test: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:12 activator: tenant.test not started: the executable it hashed (/tmp/rl182650569/app) is build "v1", but the start names build "v2" (the live link moved between the two reads); the next wake reads it again
2026/10/22 08:31:12 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:12 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/22 08:31:12 activator: replica h.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/22 08:31:12 activator: replica h.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/22 08:31:12 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/22 08:31:12 activator: why.example.com hibernated
2026/10/22 08:31:12 activator: c.example.com hibernated
2026/10/22 08:31:13 activator: h.example.com hibernated
2026/10/22 08:31:13 activator: h.example.com: run run_a is not proven stopped: cgroup.events: permission denied
2026/10/22 08:31:13 activator: stop h.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:13 activator: h.example.com hibernated
2026/10/22 08:31:13 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/22 08:31:13 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/22 08:31:13 activator: seeding unproven-seed.example.com: /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3520681911/002/live/unproven-seed.example.com seed --db /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3520681911/003/unproven-seed.example.com.db
2026/10/22 08:31:13 activator: unproven-seed.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3520681911/003/unproven-seed.example.com.db.seed-unproven refuses it as warm until they are
2026/10/22 08:31:13 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3278024083/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3278024083/003/obligation.example.com.db
2026/10/22 08:31:13 activator: obligation.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3278024083/003/obligation.example.com.db.seed-unproven refuses it as warm until they are
2026/10/22 08:31:13 activator: obligation.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/22 08:31:13 activator: no replica for obligation.example.com in s3://test-bucket/obligation.example.com — booting fresh
2026/10/22 08:31:13 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3278024083/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3278024083/003/obligation.example.com.db
2026/10/22 08:31:13 activator: seeded obligation.example.com
2026/10/22 08:31:13 activator: seeding helper-seed.example.com: /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt877537303/002/live/helper-seed.example.com seed --db /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt877537303/003/helper-seed.example.com.db
2026/10/22 08:31:13 activator: seeded helper-seed.example.com
2026/10/22 08:31:13 activator: seeding helper-unproven.example.com: /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven2894699476/002/live/helper-unproven.example.com seed --db /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven2894699476/003/helper-unproven.example.com.db
2026/10/22 08:31:13 activator: helper-unproven.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven2894699476/003/helper-unproven.example.com.db.seed-unproven refuses it as warm until they are
2026/10/22 08:31:13 activator: seeding write-ahead.example.com: /tmp/TestASeedRecordsItsObligationBeforeItRuns1348733180/002/live/write-ahead.example.com seed --db /tmp/TestASeedRecordsItsObligationBeforeItRuns1348733180/003/write-ahead.example.com.db
2026/10/22 08:31:13 activator: seeded write-ahead.example.com
2026/10/22 08:31:13 activator: write-ahead.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/22 08:31:13 activator: no replica for write-ahead.example.com in s3://test-bucket/write-ahead.example.com — booting fresh
2026/10/22 08:31:13 activator: seeding no-room.example.com: /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing1851767913/002/live/no-room.example.com seed --db /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing1851767913/003/no-room.example.com.db
2026/10/22 08:31:13 activator: seeding stuck-wal.example.com: /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation1929804846/002/live/stuck-wal.example.com seed --db /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation1929804846/003/stuck-wal.example.com.db
2026/10/22 08:31:13 activator: seeding preexisting.example.com: /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2856154708/002/live/preexisting.example.com seed --db /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2856154708/003/preexisting.example.com.db
2026/10/22 08:31:13 activator: preexisting.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2856154708/003/preexisting.example.com.db.seed-unproven refuses it as warm until they are
2026/10/22 08:31:13 activator: preexisting.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/22 08:31:13 activator: old-kernel-seed.example.com: this kernel has no cgroup.kill; the seed's leader's exit is the only proof there is
2026/10/22 08:31:13 activator: seeding failed-role-seed.example.com: /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB2647495277/002/live/failed-role-seed.example.com seed --db /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB2647495277/003/failed-role-seed.example.com.db
2026/10/22 08:31:13 activator: seeding discharge-fails.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai3392541116/002/live/discharge-fails.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai3392541116/003/discharge-fails.example.com.db
2026/10/22 08:31:13 activator: discharge-fails.example.com: the seed's obligation could not be discharged and its processes are not proven gone; /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai3392541116/003/discharge-fails.example.com.db.seed-unproven stays
2026/10/22 08:31:13 activator: seeding discharge-fails-proven.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur3883688648/002/live/discharge-fails-proven.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur3883688648/003/discharge-fails-proven.example.com.db
2026/10/22 08:31:13 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_B,_l1535638335/003/g/56/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:13 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall2376683386/003/g/57/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:13 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall2376683386/003/g/58/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:13 activator: swap.example.com hibernated
2026/10/22 08:31:13 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatholds_pinne658209731/007/g/60/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:13 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/22 08:31:13 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatoverlap_succ1990650775/007/g/62/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:13 activator: h.example.com: started carlos-instance-gen@h.example.com:64.service memory.max=536870912
2026/10/22 08:31:13 activator: swap.example.com hibernated
2026/10/22 08:31:13 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/22 08:31:14 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_super3663884204/007/g/71/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:14 activator: swap.example.com hibernated
2026/10/22 08:31:14 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartholds_pinned_build,_lea3541125021/007/g/73/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:14 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/22 08:31:14 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_a_fai3267201203/007/g/75/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:14 activator: fence.example hibernated
2026/10/22 08:31:14 activator: fence.example hibernated
2026/10/22 08:31:14 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink214055692/003/g/c.x.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink214055692/003/g/c.x.sock: no such file or directory)
2026/10/22 08:31:14 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink214055692/003/g/c.xx.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink214055692/003/g/c.xx.sock: no such file or directory)
2026/10/22 08:31:14 activator: publish generations to /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile3766591572/002/not-a-dir/zdd/generations.json: open directory /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile3766591572/002/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:14 activator: ident.example.com hibernated
2026/10/22 08:31:14 activator: instance ident.example.com exited unexpectedly — recycling
2026/10/22 08:31:14 activator: ident.example.com hibernated
2026/10/22 08:31:14 activator: due.example.com hibernated
2026/10/22 08:31:14 activator: due.example.com hibernated
2026/10/22 08:31:14 activator: ret.example.com hibernated
2026/10/22 08:31:14 activator: ret.example.com hibernated
2026/10/22 08:31:14 activator: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.example.com: a generation socket path would exceed 107 bytes; generation 9 keeps the canonical socket
2026/10/22 08:31:15 activator: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.example.com: a generation socket path would exceed 107 bytes; generation 9 keeps the canonical socket
2026/10/22 08:31:15 activator: one.example.com hibernated
2026/10/22 08:31:15 activator: purged 1 cached response(s) for flat.example.com on spawn
2026/10/22 08:31:16 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile3396821082/001/versions/vA/app seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile3396821082/002/seedexe.example.com.db
2026/10/22 08:31:16 activator: seeded seedexe.example.com
2026/10/22 08:31:16 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile3396821082/001/live/seedexe.example.com seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile3396821082/002/seedexe.example.com.db
2026/10/22 08:31:16 activator: seeded seedexe.example.com
2026/10/22 08:31:16 activator: seeding h.example.com: /tmp/TestNestedSeedExeNamesTheRecordedFile2170313703/001/live/h.example.com seed --db /tmp/nb2993019360/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:16 activator: seeded h.example.com
2026/10/22 08:31:16 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3930596972/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/22 08:31:16 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3930596972/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/22 08:31:16 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3930596972/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/22 08:31:16 activator: seeding h.example.com: /tmp/TestADeclinedNestedSeedRunsThePinnedFile557346830/001/live/h.example.com seed --db /tmp/nb3854069580/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:16 activator: seeded h.example.com
2026/10/22 08:31:16 activator: seeding tier.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed4274476984/002/versions/vA/app seed --db /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed4274476984/003/tier.example.com.db
2026/10/22 08:31:16 activator: tier.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed4274476984/003/tier.example.com.db.seed-unproven refuses it as warm until they are
2026/10/22 08:31:16 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=1073741824
2026/10/22 08:31:16 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=1073741824
2026/10/22 08:31:16 activator: seeding h.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTiernested_seed1779931139/001/live/h.example.com seed --db /tmp/nb1074425907/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:16 activator: seeded h.example.com
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsthe_configured_k3412559592/001/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsan_own_key_pres3032516552/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestFirstStartWaitsForTheConfiguredKey2445548724/002/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: control listener for shop.example.com: listen unix /tmp/TestFirstStartWaitsForTheConfiguredKey2445548724/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY in /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptinto_the_own_file4050879215/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptdelivered_in_the_co414423756/001/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration2802714138/001/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration2802714138/002/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyLeavesAFileWithTwoKeysAlone3860004338/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistse4060251211/002/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistsn335831397/002/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance2318189224/001/live/seeded.example.com seed --db /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance2318189224/002/seeded.example.com.db
2026/10/22 08:31:16 activator: seeded seeded.example.com
2026/10/22 08:31:16 activator: restored sleepy.example.com from s3://test-bucket/sleepy.example.com
hello
world
2026/10/22 08:31:17 activator: fence.example hibernated
2026/10/22 08:31:17 activator: fence.example hibernated
2026/10/22 08:31:17 activator: fence.example hibernated
2026/10/22 08:31:17 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/22 08:31:17 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/22 08:31:17 activator: barrier.example.com hibernated
2026/10/22 08:31:17 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/22 08:31:17 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/22 08:31:17 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/22 08:31:17 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/22 08:31:17 activator: barrier.example.com hibernated
2026/10/22 08:31:17 activator: barrier.example.com: generation 12's child is not proven empty: g12 still holds a process
2026/10/22 08:31:17 activator: generation 12 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/22 08:31:17 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/22 08:31:17 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:17 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/22 08:31:17 activator: barrier.example.com hibernated
2026/10/22 08:31:17 activator: stop generation 11: cgroup.kill: the child is not empty
2026/10/22 08:31:17 activator: generation 11 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/22 08:31:18 activator: placed.example.com: kill generation 1097 after a failed health wait: g still holds a process
2026/10/22 08:31:18 activator: placed.example.com: generation 1097's child is not proven empty: g still holds a process
2026/10/22 08:31:18 activator: placed.example.com: failed wake's generation 1097 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:18 activator: placed.example.com: wake failed and its generation is not proven stopped: start placed.example.com: instance not healthy within 300ms: Get "http://carlos/": dial unix /tmp/g3228977635/g/1097/placed.example.com.sock: connect: no such file or directory; activator: generation start failed and its termination is unproven: kill generation 1097 after a failed health wait: g still holds a process
2026/10/22 08:31:18 activator: placed.example.com: failed wake's generation 1097 is proven stopped; lease given back
2026/10/22 08:31:18 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:18 activator: unproven.example.com: failed wake's generation 1100 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:18 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/22 08:31:18 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:18 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:19 activator: unproven.example.com: failed wake's generation 1100 is proven stopped; lease given back
2026/10/22 08:31:19 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:19 activator: unproven.example.com: failed wake's generation 1101 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:19 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/22 08:31:19 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:19 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:19 activator: unproven.example.com: failed wake's generation 1101 is proven stopped; lease given back
2026/10/22 08:31:19 activator: unproven.example.com: generation 1102's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:19 activator: unproven.example.com: failed wake's generation 1102 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:19 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/22 08:31:19 activator: unproven.example.com: generation 1102's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:20 activator: unhealthy-f6.example.com: failed wake's generation 1103 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:20 activator: unhealthy-f6.example.com: failed wake's generation 1103 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:20 activator: unhealthy-f6.example.com: failed wake's generation 1103 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:20 activator: unhealthy-f6.example.com: failed wake's generation 1103 is proven stopped; lease given back
2026/10/22 08:31:20 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/22 08:31:20 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/22 08:31:21 activator: instance one.example.com exited unexpectedly — recycling
2026/10/22 08:31:21 activator: one.example.com hibernated
2026/10/22 08:31:21 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:21 activator: bot.x.internal hibernated
2026/10/22 08:31:21 activator: worker bot.x.internal respawned (attempt (restart)), now running v2
2026/10/22 08:31:21 activator: bot.x.internal hibernated
2026/10/22 08:31:21 activator: worker bot.x.internal respawned (attempt (config)), now running v2
2026/10/22 08:31:21 activator: bot.x.internal hibernated
2026/10/22 08:31:21 activator: bot.x.internal hibernated
2026/10/22 08:31:21 activator: bot.x.internal hibernated
2026/10/22 08:31:21 activator: bot.x.internal hibernated
2026/10/22 08:31:21 activator: moved.example.com: promoting 2 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/22 08:31:21 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:21 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:21 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:21 activator: x.example hibernated
2026/10/22 08:31:21 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/22 08:31:21 activator: x.example hibernated
2026/10/22 08:31:21 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/22 08:31:21 activator: x.example hibernated
2026/10/22 08:31:21 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:21 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:21 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:21 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/22 08:31:21 activator: x.example hibernated
2026/10/22 08:31:22 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:22 activator: x.example hibernated
2026/10/22 08:31:22 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:22 activator: x.example hibernated
2026/10/22 08:31:22 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:22 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:22 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:22 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:22 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:22 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:22 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:22 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_keyless_replica_is_refu1590799808/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_is_1803079236/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_pre-existing_env_key_th527459339/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_dif3628283046/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_with_no_999976661/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY cannot be checked against its home's key (no entry for it in /etc/carlos/replica-home-keys; O8)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_home_updated,_this_re1781334059/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY is not the one the bucket's configuration now holds: a key change is in flight (ruling 27)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_bucket_cannot_be_read4026021051/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: the bucket's configuration could not be read: bucket unreachable
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketno_fresh_reader_refused2733906244/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY cannot be checked against the bucket's configuration
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketresidual_a_change_only_o3209305564/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_config-only_homes_keyl355447216/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_home_row_mints_and_its_1387110742/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: seeding shop-au.example.com: /tmp/TestReplicaSeedIsGatedthe_homes_key1454341284/001/live/shop-au.example.com seed --db /tmp/TestReplicaSeedIsGatedthe_homes_key1454341284/002/shop-au.example.com.db
2026/10/22 08:31:22 activator: seeded shop-au.example.com
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: where its CARLOS_INSTANCE_KEY came from is unknown
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaGateDecidesOnTheLaunchesOwnRead1102268350/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 provision: a.shop.example.com: swapped for import sd_rc5xz7vxkj7s7hv7hevkflwuwy (the original route is journaled)
2026/10/22 08:31:22 provision: a.shop.example.com: import sd_rc5xz7vxkj7s7hv7hevkflwuwy rolled back to its original route
2026/10/22 08:31:22 provision: a.shop.example.com: import sd_rc5xz7vxkj7s7hv7hevkflwuwy is cleaned up and its journal row retired
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_zy2iqqyep5t4u54p5qxzowf74m (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_lpuhntwik4krxujelbdooiui4q (8192 bytes)
2026/10/22 08:31:22 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/nb3086044622/data/instances/h.example.com/h.example.com.env: the platform mints none while it is
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_yvojngjpvievk7yeafnrho2jbi (8192 bytes)
2026/10/22 08:31:22 activator: CARLOS_INSTANCE_KEY in /tmp/nb605777362/data/instances/h.example.com/h.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_kx7zxdenqrtdvo2tefvmrn46p4 (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_smaylqdvv2iufpnbtj6rtjbqre (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_tofpfe7pfirkqqej33va26cgue (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: keeping seed sd_tofpfe7pfirkqqej33va26cgue placed by an earlier wake
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_ifi5jaihjkpqac3kahew3x2n4u (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_o4zdy6cutydv6nseojz6rxavja (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_pfsvewhzcsvcdvhysgtvq6b2qu (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: seed sd_3nsdb6xblybbbn42k2lld3tefu: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/22 08:31:22 activator: h.example.com: seed sd_ulm3vzvz5xarbqyaoisgnmmcfm: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/22 08:31:22 activator: h.example.com: seed sd_wfyhw3sxbcqw66eat37lbupwsm: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/22 08:31:22 activator: h.example.com: seed sd_fkc63qv5r2ccoijy2i6jhwgu3e: h.example.com could not start from its seed: the snapshot could not be read. The next wake tries again.: bucket down
2026/10/22 08:31:22 activator: h.example.com: seed sd_243ise37pckxlzppmpiaik62vy: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/22 08:31:22 activator: h.example.com: seed sd_5y4kyu2zfetwfxd745fwblocee: h.example.com could not start from its seed: the snapshot is not a database that passes an integrity check.: integrity_check: file is not a database (26)
2026/10/22 08:31:22 activator: h.example.com: seed sd_cuk25uerbhlugefkb5stclfaca: h.example.com could not start from its seed: claim control is on for this box.
2026/10/22 08:31:22 activator: h.example.com: seed sd_cuk25uerbhlugefkb5stclfaca: h.example.com could not start from its seed: this box does not run instances nested.
2026/10/22 08:31:22 activator: h.example.com: seed sd_ezuf6wbq3uiwr34t2rh7chi45a: seed fallback for h.example.com: cannot tell whether it has a replica of its own: listing failed
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: h.example.com: seed sd_egbc3otufnfcqvzbtfz2lcxytm: seed fallback for h.example.com: found no replica of its own to restore
2026/10/22 08:31:22 activator: h.example.com: promoting 1 staged object(s) from moves/h.example.com/ (residence move landing)
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: h.example.com: seed sd_tqvwbu7a4ezadpt3yqqvni2beq: seed fallback for h.example.com: found no replica of its own to restore
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: h.example.com: seed sd_ywbevcmxoviriwdxz3kjlrpu6q: seed fallback for h.example.com: found no replica of its own to restore
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_vlk46ezauetfktqvyeixxgdhem (8192 bytes)
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: h.example.com: seed sd_llubg3dkwanzk3qacutb6cirqa: h.example.com could not start from its seed: the snapshot could not be read. The next wake tries again.: connection reset
2026/10/22 08:31:22 activator: h.example.com: seed sd_5epgqre5qv6vy7vsutqpqmnpce: h.example.com could not start from its seed: it could not be put in place. The next wake tries again.: openat2 /tmp/nb3208243919/data/instances/h.example.com/.seed-sd_5epgqre5qv6vy7vsutqpqmnpce.tmp: permission denied
2026/10/22 08:31:22 activator: h.example.com: seed sd_kr2ptbbg2qyhmhf5ylpnnt2xk4: h.example.com could not start from its seed: the snapshot is not a database that passes an integrity check.: open: unable to open database file (14)
2026/10/22 08:31:22 activator: h.example.com: seed sd_l64lsch7s2lad7r63kpfqjcvpu: h.example.com could not start from its seed: it could not be put in place. The next wake tries again.: instancefs: path contains a symlink: /tmp/nb2235448171/data/instances/h.example.com/.seed-sd_l64lsch7s2lad7r63kpfqjcvpu.tmp
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_sckuz2ncca7afqsixezqib4g6e (8192 bytes)
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: h.example.com: seed sd_6rgi2m55bidltd554nzpd3wqj4 not recorded as placed yet: record unreachable
2026/10/22 08:31:22 activator: h.example.com: /tmp/nb2781168839/data/instances/h.example.com/h.example.com.db held seed sd_hmj3bm6r7yjocvtkfc6n5rex6m's marker, not this route's (seed "sd_6cuk6sgndtemiianj2fzpnayg4"): purged before the restore
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: h.example.com: /tmp/nb3527812220/data/instances/h.example.com/h.example.com.db held seed sd_vpclpsndt5vcyfrpbkntlpnffq's marker, not this route's (seed "sd_hrkclcg4q3qlludgyx6jwga3pa"): purged before the restore
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: h.example.com: /tmp/nb1243637768/data/instances/h.example.com/h.example.com.db held seed sd_qz2n4eijtnlonbsgfh3kqxem2u's marker, not this route's (seed ""): purged before the restore
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: h.example.com: /tmp/nb645690783/data/instances/h.example.com/h.example.com.db held seed sd_p6qz3ujoh22rkvt7patsp55pn4's marker, not this route's (seed "sd_impecdpcqqjetx46gykiq34nnu"): purged before the restore
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: h.example.com: /tmp/nb2048621708/data/instances/h.example.com/h.example.com.db held seed sd_d2engyzwqh6lzleirzimfwa3ly's marker, not this route's (seed ""): purged before the restore
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: h.example.com: seed sd_adwxwmu2fwtydrixrg5msvb6ry: h.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup.events: unreadable
2026/10/22 08:31:22 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_adwxwmu2fwtydrixrg5msvb6ry (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_55ookp4oacpldff2weglnv63pm (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: /tmp/nb3259886175/data/instances/h.example.com/h.example.com.db held seed an aborted import's data's marker, not this route's (seed ""): purged before the restore
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: swap.example.com hibernated
2026/10/22 08:31:22 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/22 08:31:22 activator: swap.example.com hibernated
2026/10/22 08:31:22 activator: LEASE LOST for swap.example.com (renewal failed) — standing down
2026/10/22 08:31:22 activator: swap.example.com hibernated
2026/10/22 08:31:22 activator: swap.example.com hibernated
2026/10/22 08:31:22 activator: swap.example.com is no longer hibernate-managed — releasing
2026/10/22 08:31:22 activator: swap.example.com hibernated
2026/10/22 08:31:22 activator: swap.example.com hibernated
2026/10/22 08:31:23 activator: exit.example.com hibernated
2026/10/22 08:31:23 activator: wake shop.example.com: activator stopped during startup
2026/10/22 08:31:23 activator: wake shop.example.com: activator stopped during startup
2026/10/22 08:31:23 activator: shop.example.com: route refreshed: db "" -> "" socket "/run/carlos/shop.example.com.sock" -> "/run/carlos/instances/shop.example.com/app.sock"
2026/10/22 08:31:23 activator: stop missed-join.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/22 08:31:23 activator: missed-join.example.com hibernated
2026/10/22 08:31:23 activator: stop nested-stuck.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/22 08:31:23 activator: nested-stuck.example.com: the sidecar a restart brought back for a stopping entry is not proven gone (activator: stop unproven; processes remain); keeping the lease until it has exited
2026/10/22 08:31:24 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/22 08:31:24 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/22 08:31:24 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/22 08:31:24 activator: nested-stuck.example.com hibernated
2026/10/22 08:31:24 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:24 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:24 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:24 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/22 08:31:24 activator: bot.x.internal hibernated
2026/10/22 08:31:24 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:24 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory
2026/10/22 08:31:24 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:24 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/22 08:31:24 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:24 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:24 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory
2026/10/22 08:31:24 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:24 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/22 08:31:24 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:24 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:24 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:25 activator: worker bot.x.internal: the attempt's respawn did not finish within 150ms of its caller leaving; reporting its build as possibly executed
2026/10/22 08:31:25 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing2796713498/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing2796713498/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:25 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing2796713498/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing2796713498/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:25 activator: h.example.com: started carlos-instance-gen@h.example.com:7.service memory.max=1073741824
2026/10/22 08:31:25 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: processes remain
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:32 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: instance control /stop: 502: stop: processes remain
2026/10/22 08:31:32 activator: seeding h.example.com: /tmp/TestSeedUnitCleansUpOnlyItsOwnDBOnFailure3200155845/001/live/h.example.com seed --fast --db /tmp/nb1935035825/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:32 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:32 activator: seeding h.example.com: /tmp/TestSeedUnitFailureStopsBeforeCleanupAndKeepsTheDBWhenProcessesR779426154/001/live/h.example.com seed --fast --db /tmp/nb2108990651/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:33 activator: seeding h.example.com: /tmp/TestANestedSeedRecordsItsObligationBeforeItRuns1138658861/001/live/h.example.com seed --db /tmp/nb3456414439/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:33 activator: seeded h.example.com
2026/10/22 08:31:33 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI122081684/001/live/h.example.com seed --db /tmp/nb767264685/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:33 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/22 08:31:33 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:33 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI4277421212/001/live/h.example.com seed --db /tmp/nb1116236300/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:33 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/22 08:31:33 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:33 activator: fifo-obligation.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/22 08:31:33 activator: one.example.com: first wake of v2 had 30s of its health window left at the exec, under the 1h0m0s floor; booting the good build v1 without trying it
2026/10/22 08:31:33 activator: one.example.com hibernated
2026/10/22 08:31:33 activator: tenant.test not started: it is held on v1, and the launch hashed "v2" (the live link moved after the check); the next wake checks again
2026/10/22 08:31:34 activator: one.example.com not woken: it is held on v1, and this wake would run "v2" (the live link moved after the check); the next wake checks again
2026/10/22 08:31:34 activator: one.example.com not woken: it is held on v1, and this wake would run "" (the live link moved after the check); the next wake checks again
2026/10/22 08:31:34 activator: swap.example.com hibernated
2026/10/22 08:31:36 activator: seeding h.example.com: /tmp/TestANestedUnversionedLinkWithNoHookWakesAsToday3198820901/001/live/h.example.com seed --db /tmp/nb2232874831/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:36 activator: seeded h.example.com
2026/10/22 08:31:36 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:37 activator: swap.example.com hibernated
2026/10/22 08:31:37 activator: one.example.com: generation 1936's child is not proven empty: activator: stop unproven; processes remain
2026/10/22 08:31:37 activator: one.example.com: failed wake's generation 1936 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:37 activator: one.example.com hibernated
2026/10/22 08:31:38 activator: one.example.com: first wake of v2 had 400ms of its health window left at the exec, under the 20s floor; booting the good build v1 without trying it
2026/10/22 08:31:38 activator: x.example: hold's pinned wake refused or abandoned: the claims admission refused before the exec: hostbox: unresolved lifecycle contract: isolated root launcher cannot carry claim reservations
2026/10/22 08:31:38 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866311826/003/g/1943/real.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:40 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866311826/003/g/1944/real.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:40 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866311826/003/g/1944/real.example.com.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866311826/003/g/1944/real.example.com.sock: no such file or directory)
2026/10/22 08:31:40 activator: real.example.com hibernated
2026/10/22 08:31:42 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign2459861641/004/hand-set.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign2459861641/004/hand-set.sock: no such file or directory)
2026/10/22 08:31:42 activator: real.example.com hibernated
2026/10/22 08:31:42 activator: bot.x.internal: worker generation 0's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:42 activator: bot.x.internal: worker generation 1948's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:42 activator: bot.x.internal: failed wake's generation 1948 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:43 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd: not a directory; the sweep retries
2026/10/22 08:31:43 activator: worker bot.x.internal not spawned: the build it would exec (v2 /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/003/v2) could not be published first: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd: not a directory
2026/10/22 08:31:43 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd: not a directory; the sweep retries
2026/10/22 08:31:43 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd: not a directory; the sweep retries
2026/10/22 08:31:43 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn4010990313/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn4010990313/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:43 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn4010990313/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn4010990313/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:43 activator: instance one.example.com exited unexpectedly — recycling
2026/10/22 08:31:43 activator: one.example.com hibernated
2026/10/22 08:31:44 activator: one.example.com hibernated
2026/10/22 08:31:45 activator: one.example.com: first wake of v2 had 499ms of its health window left at the exec, under the 1s floor; booting the good build v1 without trying it
2026/10/22 08:31:47 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/22 08:31:47 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/22 08:31:47 activator: worker bot.example.com respawned (exited), now running 
2026/10/22 08:31:47 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/22 08:31:47 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/22 08:31:47 activator: worker bot.example.com respawned (exited), now running 
2026/10/22 08:31:47 activator: bot.example.com: worker generation 1963's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:47 activator: stop bot.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:47 activator: bot.example.com hibernated
2026/10/22 08:31:47 activator: bot3.example.com: worker generation 1964's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:47 activator: worker bot3.example.com NOT respawned (test: a restart with no old handle): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/22 08:31:47 activator: bot4.example.com: worker generation 1966's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:47 activator: bot4.example.com: failed wake's generation 1966 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:47 activator: bot4.example.com: failed wake's generation 1966 is proven stopped; lease given back
2026/10/22 08:31:47 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/22 08:31:47 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/22 08:31:47 activator: bot2.example.com: worker generation 1967's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:47 activator: worker bot2.example.com NOT respawned (exited): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/22 08:31:47 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/22 08:31:47 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/22 08:31:47 activator: worker bot2.example.com respawned (exited), now running 
2026/10/22 08:31:47 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/22 08:31:47 activator: worker bot.example.com respawned (exited), now running v1
2026/10/22 08:31:48 activator: bot.example.com hibernated
2026/10/22 08:31:48 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/22 08:31:48 activator: worker bot10.example.com: the shed respawn child's exit is not proven (activator: stop unproven; processes remain); the entry keeps it and its build until the sweep proves it
2026/10/22 08:31:48 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/22 08:31:48 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/22 08:31:48 activator: bot10.example.com: the shed respawn child has exited
2026/10/22 08:31:48 activator: bot10.example.com hibernated
2026/10/22 08:31:48 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/22 08:31:48 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/22 08:31:48 activator: bot10.example.com respawn did not finish before its stop; stopping the rest, the lease is kept until it has
2026/10/22 08:31:48 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/22 08:31:48 activator: bot10.example.com hibernated
2026/10/22 08:31:48 activator: bot8.example.com: worker generation 1970's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:48 activator: bot8.example.com: failed wake's generation 1970 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:48 activator: bot8.example.com: failed wake's generation 1970 is proven stopped; lease given back
2026/10/22 08:31:48 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:48 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:48 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/22 08:31:48 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:49 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:49 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/22 08:31:49 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/22 08:31:49 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:49 activator: bot.x.internal hibernated
2026/10/22 08:31:49 activator: worker bot.x.internal respawned (attempt (restart)), now running v1
2026/10/22 08:31:49 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:49 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:49 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:50 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:50 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:50 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:50 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:50 activator: bot.x.internal: worker generation 1973's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (attempt (release)): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/22 08:31:50 activator: bot.x.internal: worker generation 1973's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (restoring v1): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: lease lost; abandoning it
2026/10/22 08:31:50 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:50 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory
2026/10/22 08:31:50 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:50 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/22 08:31:50 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:50 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:50 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory
2026/10/22 08:31:50 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:50 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/22 08:31:50 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:50 activator: bot.x.internal hibernated
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: lease lost; abandoning it
--- FAIL: TestAWorkerFallbackWithoutOwnershipSpawnsNothing (0.03s)
    workerswap_test.go:653: ReplaceWorker = {Outcome:abandoned Gen:{N:0 Socket: Control: Build: StagedPath: Digest: Demand: Tier:{MemoryMax:0} Worker:false} Reason:re-checked at the spawn: swap: lease lost Drain:0s Mode: Executed:true CleanExit:false}, want failed with nothing serving
2026/10/22 08:31:50 activator: WORKER bot.x.internal failed to respawn (attempt (restart)): exec: text file busy — the sweep's crash restart takes it from here
2026/10/22 08:31:50 activator: WORKER bot.x.internal failed to respawn (restoring v1): exec: text file busy — the sweep's crash restart takes it from here
2026/10/22 08:31:50 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:51 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/22 08:31:51 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:51 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/22 08:31:51 activator: bot.x.internal hibernated
2026/10/22 08:31:51 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:51 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:51 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:51 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:51 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:51 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/22 08:31:51 activator: bot.x.internal hibernated
2026/10/22 08:31:52 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:52 activator: LEASE LOST for bot.x.internal (renewal failed) — standing down
2026/10/22 08:31:52 activator: bot.x.internal hibernated
2026/10/22 08:31:52 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:52 activator: bot.x.internal hibernated
2026/10/22 08:31:52 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:52 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/22 08:31:52 activator: bot.x.internal hibernated
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:53 activator: squatted.example.com: child exited during the health wait (external process serving its socket?) — not normalizing the socket mode
2026/10/22 08:31:53 activator: SIDECAR for late-sidecar.example failed to start: activator: maintenance fence active — retaining returned process
2026/10/22 08:31:53 activator: late-sidecar.example hibernated
2026/10/22 08:31:53 activator: own.example.com: failed wake's generation 0 is proven stopped; lease given back
2026/10/22 08:31:53 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:53 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:53 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/22 08:31:53 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/22 08:31:53 activator: swap.example.com hibernated
2026/10/22 08:31:53 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:53 activator: x.example hibernated
2026/10/22 08:31:53 activator: fence.example hibernated
2026/10/22 08:31:53 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it stays down until a release attempt or a wake starts it
2026/10/22 08:31:53 activator: worker bot9.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/22 08:31:53 activator: bot9.example.com: worker generation 2003's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:53 activator: generation 2003 of bot9.example.com did not prove its exit; keeping it in the operation set
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: swap.example.com hibernated
2026/10/22 08:31:53 activator: LEASE LOST for bot9.example.com (renewal failed) — standing down
2026/10/22 08:31:53 activator: bot9.example.com hibernated
2026/10/22 08:31:53 activator: swap.example.com hibernated
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 activator: worker bot1.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot1.x.internal exited; it serves v1 but live/bot1.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/22 08:31:53 activator: worker bot2.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot2.x.internal exited; live/bot2.x.internal names v2, but this sweep's converge budget is spent, so it recycles its captured build v1 and converges on a later sweep
2026/10/22 08:31:53 activator: worker bot2.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 activator: worker bot1.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:53 provision: a.shop.example.com: swapped for import sd_sq5h2qnaeezevsozmcprcqyoee (the original route is journaled)
2026/10/22 08:31:53 activator: seeding seedpin.example.com: /tmp/TestADeclinedFlatSeedRunsThePinnedFile2094590256/001/versions/v1/app seed --db /tmp/TestADeclinedFlatSeedRunsThePinnedFile2094590256/002/seedpin.example.com.db
2026/10/22 08:31:53 activator: seeded seedpin.example.com
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 activator: worker bot2.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:53 activator: worker bot2.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:53 provision: a.shop.example.com: import sd_sq5h2qnaeezevsozmcprcqyoee rolled back to its original route
2026/10/22 08:31:53 provision: a.shop.example.com: import sd_sq5h2qnaeezevsozmcprcqyoee is cleaned up and its journal row retired
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:53 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912
2026/10/22 08:31:53 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawngated_build_is_the_ex3434526572/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:53 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawna_held_build_starts_n4044055006/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:53 activator: swap.example.com hibernated
2026/10/22 08:31:53 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawnno_gate_is_todays_sp1289300416/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:53 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/22 08:31:53 activator: own.example.com: failed wake's generation 0 is proven stopped; lease given back
2026/10/22 08:31:53 activator: shop.example.com: generation 4's child is not proven empty: kill gen: timeout
2026/10/22 08:31:53 activator: x.example hibernated
2026/10/22 08:31:53 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:53 activator: stop fence.example: earlier process may still run
2026/10/22 08:31:53 activator: fence.example hibernated
2026/10/22 08:31:53 activator: fence.example hibernated
2026/10/22 08:31:53 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/22 08:31:53 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 activator: swap.example.com hibernated
2026/10/22 08:31:53 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 provision: a.shop.example.com: swapped for import sd_7napxduijdznkofzktf7oxjdau (the original route is journaled)
2026/10/22 08:31:53 activator: a.shop.example.com hibernated
2026/10/22 08:31:54 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:54 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it recycles its captured build v1
2026/10/22 08:31:54 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:54 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:54 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: the running child keeps running
2026/10/22 08:31:54 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:54 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:54 provision: a.shop.example.com: import sd_7napxduijdznkofzktf7oxjdau rolled back to its original route
2026/10/22 08:31:54 provision: a.shop.example.com: import sd_7napxduijdznkofzktf7oxjdau is cleaned up and its journal row retired
2026/10/22 08:31:54 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:54 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:54 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:54 activator: worker bot.x.internal serving v1 but live symlink is now v3 — respawning onto the new build
2026/10/22 08:31:54 activator: worker bot.x.internal respawned (converging onto a new build), now running v3
2026/10/22 08:31:54 activator: shop.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: swap.example.com: failed wake's generation 2039 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:54 activator: swap.example.com: failed wake's generation 2039 is proven stopped; lease given back
2026/10/22 08:31:54 activator: waking.example hibernated
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: swap.example.com: generation 2042's child is not proven empty: the child still holds a process
2026/10/22 08:31:54 activator: swap.example.com: failed wake's generation 2042 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:54 activator: swap.example.com: failed wake's generation 2042 is proven stopped; lease given back
2026/10/22 08:31:54 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:54 activator: x.example hibernated
2026/10/22 08:31:54 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:54 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:54 provision: a.shop.example.com: swapped for import sd_imywmiphqs52l7np4t3e5vmr5i (the original route is journaled)
before
2026/10/22 08:31:54 activator: one.example.com: sidecar starting on the instance's own build v1 (/opt/carlos/versions/v1/app): this box will not start v2, the build its live link names
2026/10/22 08:31:54 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/22 08:31:54 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/22 08:31:54 activator: a.shop.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@a.shop.example.com: unit still active); keeping the lease until it is
2026/10/22 08:31:54 provision: a.shop.example.com: import sd_imywmiphqs52l7np4t3e5vmr5i rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/22 08:31:54 activator: one.example.com hibernated
2026/10/22 08:31:54 activator: one.example.com hibernated
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance-gen@h.example.com:2062.service memory.max=536870912
2026/10/22 08:31:54 provision: a.shop.example.com: import sd_imywmiphqs52l7np4t3e5vmr5i rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/22 08:31:54 activator: stop fence.example: stop refused
2026/10/22 08:31:54 activator: fence.example hibernated
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/22 08:31:54 provision: a.shop.example.com: import sd_imywmiphqs52l7np4t3e5vmr5i rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: LEASE HELD for swap.example.com by "box-b" — a previous incarnation hasn't released; refusing to wake (retry after 1m0s)
2026/10/22 08:31:54 activator: swap.example.com: hold's pinned wake refused or abandoned: instance lease held by "box-b" (retry in 1m0s)
2026/10/22 08:31:54 activator: one.example.com hibernated
2026/10/22 08:31:54 activator: one.example.com hibernated
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:54 provision: a.shop.example.com: import sd_imywmiphqs52l7np4t3e5vmr5i rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: h.example.com: processes remain after failed; not releasing
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:54 activator: one.example.com hibernated
2026/10/22 08:31:54 activator: fence.example hibernated
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:54 activator: h.example.com: unit restarted outside the activator (invocation inv2, ours inv1); stopping it
2026/10/22 08:31:54 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:54 activator: x.example hibernated
2026/10/22 08:31:54 activator: one.example.com serving v1 but v2 is adopted — cycling onto the new build
2026/10/22 08:31:54 activator: one.example.com hibernated
2026/10/22 08:31:54 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:54 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: the backend ran a different generation from the hold's choice: generation 2080 of v9 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_1261890475/004/versions/v2/app"; the hold chose v2 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_1261890475/004/versions/v2/app"
2026/10/22 08:31:54 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:54 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:54 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:54 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, nor the instance's own build v1: a failure record holds this build: v1
2026/10/22 08:31:54 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/22 08:31:54 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:54 activator: worker bot.x.internal NOT respawned (converging onto a new build): instance config unavailable on this box: cannot open sealed config — standing down and releasing its lease
2026/10/22 08:31:54 activator: bot.x.internal hibernated
2026/10/22 08:31:54 activator: h.example.com: unit restarted outside the activator (invocation someone-else, ours inv7); stopping it
2026/10/22 08:31:54 activator: old.shop.example.com: route refreshed: db "/data/instances/old/old.shop.example.com.db" -> "" socket "/run/carlos/instances/old/app.sock" -> ""
2026/10/22 08:31:54 activator: wake old.shop.example.com: old.shop.example.com: the route changed before the wake: the activator no longer manages it
2026/10/22 08:31:54 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:54 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:54 activator: worker bot.x.internal NOT respawned (exited): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/22 08:31:54 activator: bot.x.internal hibernated
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: swap.example.com: hold's pinned wake refused or abandoned: restore swap.example.com: replica unreadable
2026/10/22 08:31:54 activator: h.example.com: kill carlos-instance-gen@h.example.com:2091.service after a failed start: stop timed out
2026/10/22 08:31:54 activator: one.example.com: wake failed and its generation is not proven stopped: start one.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2091.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/22 08:31:54 activator: sidecar staged for awake tenant.test — adopting
2026/10/22 08:31:54 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:54 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/22 08:31:54 activator: stop fence.example: private backend detail
2026/10/22 08:31:54 activator: fence.example hibernated
2026/10/22 08:31:54 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:54 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/22 08:31:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:55 activator: sidecar staged for awake tenant.test — adopting
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/22 08:31:55 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/22 08:31:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/22 08:31:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:55 activator: worker bot.x.internal hit its max_run; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/22 08:31:55 activator: x.example hibernated
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (hit its max_run), now running v2
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: socket dir: permission denied
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/22 08:31:55 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, and the instance's own build v1 cannot be named: staged copy unreadable
2026/10/22 08:31:55 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:55 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: staging: staged digest mismatch: have be0041e1a437dbea5b67dfe58bad04a0fc30d5036d469429f3534f39ec53d8b5, want c3bae8fb3a8bf395783d7f7f6ffcd4aadc797c95ce24b7e8e2a98af6b63d8f6d
2026/10/22 08:31:55 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/22 08:31:55 activator: x.example hibernated
2026/10/22 08:31:55 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/22 08:31:55 activator: one.example.com hibernated
2026/10/22 08:31:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:55 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:55 activator: worker bot.x.internal NOT respawned (hit its max_run): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/22 08:31:55 activator: bot.x.internal hibernated
2026/10/22 08:31:55 activator: early.example.com: the aborted start's unit is not proven stopped (context canceled); keeping the lease until it is
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:55 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/22 08:31:55 activator: stop sidecar fence.example: private backend detail
2026/10/22 08:31:55 activator: fence.example hibernated
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:55 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory; the sweep retries
2026/10/22 08:31:55 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory; the sweep retries
2026/10/22 08:31:55 activator: worker first.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory
2026/10/22 08:31:55 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory; the sweep retries
2026/10/22 08:31:55 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory; the sweep retries
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory; the sweep retries
2026/10/22 08:31:55 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory
2026/10/22 08:31:55 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory; the sweep retries
2026/10/22 08:31:55 activator: WORKER bot.x.internal failed to respawn (exited): worker build not published before its exec: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory — the sweep's crash restart takes it from here
2026/10/22 08:31:55 activator: bot.example.com hibernated
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/22 08:31:55 activator: x.example hibernated
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: start g: activator: generation start failed and its termination is unproven
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2130.service exited before it was observed
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:55 activator: worker bot.x.internal NOT respawned (exited): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: x.example hibernated
2026/10/22 08:31:55 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: WORKER bot.x.internal failed to respawn (exited): fork/exec: resource temporarily unavailable — the sweep's crash restart takes it from here
2026/10/22 08:31:55 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:55 activator: final checkpoint fence.example: private backend detail
2026/10/22 08:31:55 activator: fence.example hibernated
2026/10/22 08:31:55 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/22 08:31:55 activator: replica shop.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:55 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: proven.example.com idle 2m0s — hibernating
2026/10/22 08:31:55 activator: proven.example.com hibernated
2026/10/22 08:31:55 activator: bot.x.internal hibernated
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: h.example.com: kill carlos-instance-gen@h.example.com:2156.service after a failed start: stop timed out
2026/10/22 08:31:55 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2156.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/22 08:31:56 activator: swap.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/22 08:31:56 activator: h.example.com: kill carlos-instance-gen@h.example.com:2160.service after a failed start: stop timed out
2026/10/22 08:31:56 activator: kass.example.com idle 11s — hibernating
2026/10/22 08:31:56 activator: kass.example.com hibernated
2026/10/22 08:31:56 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:56 activator: pending-f6.example.com idle 2m0s — hibernating
2026/10/22 08:31:56 activator: stop pending-f6.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/22 08:31:56 activator: stop sidecar nested-costop.example.com: activator: stop unproven; processes remain
2026/10/22 08:31:56 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/22 08:31:56 activator: stop pending-f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:56 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/22 08:31:56 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/22 08:31:56 activator: pending-f6.example.com hibernated
2026/10/22 08:31:56 activator: nested-stop.example.com idle 2m0s — hibernating
2026/10/22 08:31:56 activator: nested-stop.example.com hibernated
2026/10/22 08:31:56 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/22 08:31:56 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:56 activator: nested-costop.example.com hibernated
2026/10/22 08:31:56 activator: one.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:56 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:56 activator: fence.example hibernated
2026/10/22 08:31:56 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/22 08:31:56 activator: control listener for shop.example.com: listen unix /tmp/TestARefusedLaunchLeavesNoBuildStateRecord2940391471/002/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:56 activator: undelegated.example.com idle 2m0s — hibernating
2026/10/22 08:31:56 activator: undelegated.example.com: stopped with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:31:56 activator: undelegated.example.com hibernated
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/22 08:31:56 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:56 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:56 activator: swap.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com: a discarded SIDECAR is not proven gone (activator: stop unproven; processes remain); keeping it tracked
2026/10/22 08:31:56 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:56 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:56 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:56 activator: swap.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: a stop-evidence operation holds the host
2026/10/22 08:31:56 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:56 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: shutdown-pending.example.com idle 2m0s — hibernating
2026/10/22 08:31:56 activator: stop shutdown-pending.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/22 08:31:56 activator: f6.example.com idle 2m0s — hibernating
2026/10/22 08:31:56 activator: stop f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:56 activator: stop f6.example.com: its host cgroup is not proven empty; keeping the lease until it is
2026/10/22 08:31:56 activator: f6.example.com hibernated
2026/10/22 08:31:56 activator: instance c.example.com exited unexpectedly — recycling
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:56 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:56 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/22 08:31:56 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:56 activator: fence.example hibernated
2026/10/22 08:31:56 activator: c.example.com idle 1h0m0s — hibernating
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:56 activator: seeding shop.example.com: /tmp/TestARefusedLaunchLeavesNoBuildStateRecordseed_refused_beside_a3688975537/002/versions/v1/app seed --db /tmp/TestARefusedLaunchLeavesNoBuildStateRecordseed_refused_beside_a3688975537/003/shop.example.com.db
2026/10/22 08:31:56 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:56 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:56 provision: a.shop.example.com: swapped for import sd_l5m3iojwjops4rxkvhv37tqkr4 (the original route is journaled)
2026/10/22 08:31:56 activator: shutdown-pending.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:56 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 8s)
2026/10/22 08:31:56 activator: swap.example.com hibernated
2026/10/22 08:31:56 activator: h.example.com: kill carlos-instance-gen@h.example.com:2215.service after a failed start: stop timed out
2026/10/22 08:31:56 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: the unit failed to start; activator: generation start failed and its termination is unproven: stop timed out
2026/10/22 08:31:56 provision: a.shop.example.com: import sd_l5m3iojwjops4rxkvhv37tqkr4 rolled back to its original route
2026/10/22 08:31:56 activator: sidecar for swap.example.com was down — restarted (backoff now 16s)
2026/10/22 08:31:56 provision: a.shop.example.com: import sd_l5m3iojwjops4rxkvhv37tqkr4 is cleaned up and its journal row retired
2026/10/22 08:31:56 provision: a.shop.example.com: swapped for import sd_23ogtfusi7pnum4eap2bxjv5gu (the original route is journaled)
2026/10/22 08:31:56 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:56 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:56 activator: a.shop.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@a.shop.example.com: instance control socket: connection refused); keeping the lease until it is
2026/10/22 08:31:56 provision: a.shop.example.com: import sd_23ogtfusi7pnum4eap2bxjv5gu rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/22 08:31:56 activator: swap.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/22 08:31:56 activator: a.shop.example.com: failed wake's generation 2216 is proven stopped; lease given back
2026/10/22 08:31:56 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:56 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:56 activator: swap.example.com: v1 not started: its build state could not be written: bucket: 503
2026/10/22 08:31:56 activator: c.example.com idle 1h0m0s — hibernating
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: SIDECAR restart for one.example.com failed: swap: lease lost (next attempt in 2s)
2026/10/22 08:31:56 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:56 activator: worker bot.x.internal exited; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/22 08:31:56 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:56 provision: a.shop.example.com: import sd_23ogtfusi7pnum4eap2bxjv5gu rolled back to its original route
2026/10/22 08:31:56 provision: a.shop.example.com: import sd_23ogtfusi7pnum4eap2bxjv5gu is cleaned up and its journal row retired
2026/10/22 08:31:57 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:57 activator: stop c.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: cgroup: the host's parent cgroup holds a process directly: holds 1 process(es)
2026/10/22 08:31:57 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:57 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:57 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:57 activator: LEASE LOST for c.example.com (lost) — standing down
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:57 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:57 activator: stop fence.example: activator: stop evidence unavailable
2026/10/22 08:31:57 activator: fence.example hibernated
2026/10/22 08:31:57 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: instance control: 502 mid-roll; the sweep tries again
2026/10/22 08:31:57 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:57 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:57 activator: c.example.com idle 1h0m0s — hibernating
2026/10/22 08:31:57 activator: seeding shop.example.com: /tmp/TestARefusedLaunchLeavesNoBuildStateRecorda_seed_that_ran_is_re435509183/001/versions/v1/app seed --db /tmp/TestARefusedLaunchLeavesNoBuildStateRecorda_seed_that_ran_is_re435509183/002/shop.example.com.db
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:57 activator: shop.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:31:57 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: processes remained; the sweep tries again
2026/10/22 08:31:57 activator: one.example.com hibernated
2026/10/22 08:31:57 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:57 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/22 08:31:57 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: shop.example.com: seed placement not recorded yet: record unreachable
2026/10/22 08:31:57 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: one.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:57 activator: SIDECAR restart for one.example.com failed: the build state could not be written, so this build was not started: bucket refused the build state (next attempt in 2s)
2026/10/22 08:31:57 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake617005226/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake617005226/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:57 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake617005226/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake617005226/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:57 activator: stop fence.example: stop refused
2026/10/22 08:31:57 activator: fence.example hibernated
2026/10/22 08:31:57 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:57 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:57 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:57 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:57 activator: final checkpoint fence.example: activator: stop evidence unavailable
2026/10/22 08:31:57 activator: fence.example hibernated
2026/10/22 08:31:57 activator: c.example.com serving v1 but v2 is adopted — cycling onto the new build
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:57 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: c.example.com idle 1h0m0s — hibernating
2026/10/22 08:31:57 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: instance control: 502 mid-roll; the sweep tries again
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:57 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 4s)
2026/10/22 08:31:57 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor2190742349/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor2190742349/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:57 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor2190742349/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor2190742349/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:57 activator: fence.example hibernated
2026/10/22 08:31:57 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:57 activator: SIDECAR for one.example.com failed to start: the build state could not be written, so this build was not started: bucket refused the build state — serving without it (the sweep retries)
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a cgroup that took cgroup.kill could not be made fresh before a spawn: carlos-exec@h.example.com: device or resource busy
2026/10/22 08:31:57 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:57 activator: fence.example hibernated
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:58 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:58 activator: one.example.com idle 1h0m0s — hibernating
2026/10/22 08:31:58 activator: one.example.com hibernated
2026/10/22 08:31:58 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:58 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:58 activator: seeding h.example.com: /tmp/TestPlan14GenerationBackendAuxiliaryBoundaryclaim=falseseedpi2333166748/001/live/h.example.com seed --db /tmp/nb1806560079/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:58 activator: seeded h.example.com
2026/10/22 08:31:58 activator: seeding h.example.com: /tmp/TestPlan14GenerationBackendAuxiliaryBoundaryclaim=falseseedpi3858699653/001/live/h.example.com seed --db /tmp/nb671201033/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:58 activator: seeded h.example.com
2026/10/22 08:31:58 activator: one.example.com idle 1h0m0s — hibernating
2026/10/22 08:31:58 activator: one.example.com hibernated
2026/10/22 08:31:58 activator: h.example.com idle 2m0s — hibernating
2026/10/22 08:31:58 activator: stop h.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/22 08:31:58 activator: fence.example hibernated
2026/10/22 08:31:58 activator: h.example.com hibernated
2026/10/22 08:31:58 activator: stuck.example.com idle 2m0s — hibernating
2026/10/22 08:31:58 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/22 08:31:58 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished1518301518/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished1518301518/001/zdd: not a directory; the sweep retries
2026/10/22 08:31:58 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished1518301518/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished1518301518/001/zdd: not a directory; the sweep retries
2026/10/22 08:31:58 activator: fence.example hibernated
2026/10/22 08:31:58 activator: f6.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake822514821/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake822514821/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake822514821/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake822514821/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:58 activator: exited-flat.example.com: failed wake's generation 2305 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:58 activator: exited-flat.example.com: failed wake's generation 2305 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/22 08:31:58 activator: one.example.com hibernated
2026/10/22 08:31:58 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:58 activator: fenced.example.com: failed wake's generation 2311 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:58 activator: unproven.example.com: generation 2313's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:58 activator: unproven.example.com: failed wake's generation 2313 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/22 08:31:58 activator: unproven.example.com: generation 2313's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:58 activator: fenced.example.com: failed wake's generation 2311 is proven stopped; lease given back
2026/10/22 08:31:58 activator: c.example.com is no longer hibernate-managed — releasing
2026/10/22 08:31:58 activator: c.example.com hibernated
2026/10/22 08:31:58 activator: unproven.example.com: failed wake's generation 2313 is proven stopped; lease given back
2026/10/22 08:31:58 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:58 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:58 activator: bot.x.internal hibernated
2026/10/22 08:31:58 activator: unproven.example.com hibernated
2026/10/22 08:31:58 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:58 activator: swap.example.com hibernated
2026/10/22 08:31:58 activator: forked.example.com: failed wake's generation 2318 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: forked.example.com: failed wake's generation 2318 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:58 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/22 08:31:58 activator: barrier.example.com hibernated
2026/10/22 08:31:58 activator: forked.example.com: failed wake's generation 2318 is proven stopped; lease given back
2026/10/22 08:31:58 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:58 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:58 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/22 08:31:58 activator: one.example.com hibernated
2026/10/22 08:31:58 activator: exited-foreign.example.com: failed wake's generation 2326 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:58 activator: exited-foreign.example.com: failed wake's generation 2326 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/22 08:31:58 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:58 activator: instance unproven.example.com exited unexpectedly — recycling
2026/10/22 08:31:58 activator: unproven.example.com hibernated
2026/10/22 08:31:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1746421889/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1746421889/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:58 activator: x.example: hold's pinned wake refused or abandoned: x.example: refusing to wake: staged build not published before its exec
2026/10/22 08:31:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1746421889/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1746421889/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:58 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:58 activator: unproven.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@unproven: instance control socket: connection refused); keeping the lease until it is
2026/10/22 08:31:58 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/22 08:31:58 activator: instance slowproof.example.com exited unexpectedly — recycling
2026/10/22 08:31:58 activator: slowproof.example.com hibernated
2026/10/22 08:31:58 activator: barrier.example.com hibernated
2026/10/22 08:31:58 activator: swap.example.com hibernated
2026/10/22 08:31:58 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a generation's limits could not be applied: h.example.com: the placement shim is not available
2026/10/22 08:31:58 activator: barrier.example.com hibernated
2026/10/22 08:31:58 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:58 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:58 activator: f6-layout.example.com: failed wake's generation 2337 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:58 activator: f6-layout.example.com: failed wake's generation 2337 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: refused.example.com: giving the lease back failed: delete lease: context deadline exceeded; retrying it
2026/10/22 08:31:58 activator: refused.example.com: lease given back on retry
2026/10/22 08:31:58 activator: c.example.com hibernated
2026/10/22 08:31:58 activator: fenced.example.com: failed wake's generation 2341 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:58 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/22 08:31:58 activator: bot.x.internal hibernated
2026/10/22 08:31:58 activator: unproven.example.com: generation 2343's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:58 activator: unproven.example.com: failed wake's generation 2343 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: h: start unit: instance control: generation termination unproven: the unit exited before it was observed
2026/10/22 08:31:58 activator: fenced.example.com: failed wake's generation 2341 is proven stopped; lease given back
2026/10/22 08:31:58 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:58 activator: unproven.example.com: generation 2343's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:58 activator: unproven.example.com: failed wake's generation 2343 is proven stopped; lease given back
2026/10/22 08:31:58 activator: unproven.example.com hibernated
2026/10/22 08:31:58 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/22 08:31:58 activator: instance restart.example.com exited unexpectedly — recycling
2026/10/22 08:31:58 activator: restart.example.com hibernated
2026/10/22 08:31:59 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/22 08:31:59 activator: barrier.example.com hibernated
2026/10/22 08:31:59 activator: slowproof.example.com: the aborted start's unit is not proven stopped (context deadline exceeded); keeping the lease until it is
2026/10/22 08:31:59 activator: shutdown: out of time with 1 failed wake(s) not proven stopped, their leases still held: slowproof.example.com (generation 2333)
2026/10/22 08:31:59 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:59 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:59 activator: bot.x.internal hit its max_run of 1m0s — stopping
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:59 activator: SIDECAR for swap.example.com is not proven gone for its replacement; tracking it (the sweep re-proves it)
2026/10/22 08:31:59 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/22 08:31:59 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:59 activator: f6.example.com: failed wake's generation 0 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:59 activator: stuck.example.com hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal reported idle but is always-on (hibernate:false) — ignoring
2026/10/22 08:31:59 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:59 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:59 activator: WORKER bot.x.internal failed to respawn (converging onto a new build): exec: live symlink gone — the sweep's crash restart takes it from here
2026/10/22 08:31:59 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:59 activator: one.example.com: live/one.example.com names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/22 08:31:59 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running v2
2026/10/22 08:31:59 activator: queued.example.com hibernated
2026/10/22 08:31:59 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:59 activator: f6-layout.example.com: failed wake's generation 2361 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:59 activator: f6-layout.example.com: failed wake's generation 2361 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:59 activator: shutdown: out of time with 1 failed wake(s) not proven stopped, their leases still held: unproven.example.com (generation 2330)
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; live/bot.x.internal names v2, but the release scheduler owns build changes, so it recycles its captured build v1
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:59 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/22 08:31:59 activator: swap.example.com hibernated
2026/10/22 08:31:59 activator: one.example.com hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:59 activator: fence.example hibernated
2026/10/22 08:31:59 activator: bot.x.internal serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: untimed.bab.oncarlos.com idle 16m0s — hibernating
2026/10/22 08:31:59 activator: untimed.bab.oncarlos.com hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: unproven.example.com: failed wake's generation 2363 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:59 activator: unproven.example.com: retry stop of failed wake's generation 2363: cgroup.kill: the child is not empty
2026/10/22 08:31:59 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/22 08:31:59 activator: barrier.example.com hibernated
2026/10/22 08:31:59 activator: LEASE LOST for one.example.com (lease renewal failed) — standing down
2026/10/22 08:31:59 activator: one.example.com hibernated
2026/10/22 08:31:59 activator: unproven.example.com: generation 2363's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:59 activator: mid.example.com has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: unproven.example.com: failed wake's generation 2363 is proven stopped; lease given back
2026/10/22 08:31:59 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/22 08:31:59 activator: one.example.com hibernated
2026/10/22 08:31:59 activator: flip.example.com is no longer hibernate-managed — releasing
2026/10/22 08:31:59 activator: flip.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/22 08:31:59 activator: flip.example.com hibernated
2026/10/22 08:31:59 activator: unproven.example.com hibernated
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:59 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:59 activator: mid.example.com is no longer hibernate-managed — releasing (wake finished first)
2026/10/22 08:31:59 activator: mid.example.com hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:59 activator: f6-layout.example.com: failed wake's generation 2375 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:59 activator: f6-layout.example.com: failed wake's generation 2375 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:59 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:59 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:59 activator: worker bot-a.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:59 activator: worker bot-a.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 4s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: bot.x.internal idle 11s — hibernating
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 8s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: bot.x.internal idle 11s — hibernating
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: bot.x.internal idle 31s — hibernating
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: tickheld.bab.oncarlos.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/22 08:31:59 activator: tickheld.bab.oncarlos.com hibernated
2026/10/22 08:31:59 activator: worker bot-b.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:59 activator: worker bot-b.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:59 activator: bot.x.internal idle 11s — hibernating
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/22 08:31:59 activator: barrier.example.com hibernated
2026/10/22 08:31:59 activator: always-on sidecar bot.x.internal is not running — starting it
2026/10/22 08:31:59 activator: bot.x.internal idle 11s — hibernating
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: unproven.example.com: generation 2380's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:59 activator: unproven.example.com: failed wake's generation 2380 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:59 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/22 08:31:59 activator: swap.example.com hibernated
2026/10/22 08:31:59 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/22 08:31:59 activator: unproven.example.com: generation 2380's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:59 activator: f6-layout.example.com: failed wake's generation 2382 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:59 activator: f6-layout.example.com: failed wake's generation 2382 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:59 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:59 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (hit its max_run), now running 
2026/10/22 08:31:59 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:59 activator: unproven.example.com: failed wake's generation 2380 is proven stopped; lease given back
2026/10/22 08:32:00 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:32:00 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:32:00 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:32:00 activator: unproven.example.com hibernated
2026/10/22 08:32:00 activator: LEASE LOST for bot.x.internal (lease renewal failed) — standing down
2026/10/22 08:32:00 activator: bot.x.internal hibernated
2026/10/22 08:32:00 activator: one.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: one.example.com hibernated
2026/10/22 08:32:00 activator: always-on sidecar bot00.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot01.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot02.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot03.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot04.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot05.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot06.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot07.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot08.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot09.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot10.x.internal is not running — starting it
2026/10/22 08:32:00 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/22 08:32:00 activator: one.example.com hibernated
2026/10/22 08:32:00 activator: two.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/22 08:32:00 activator: two.example.com hibernated
2026/10/22 08:32:00 activator: own.example.com idle 1m0s — hibernating
2026/10/22 08:32:00 activator: own.example.com hibernated
2026/10/22 08:32:00 activator: bots.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: bots.example.com hibernated
2026/10/22 08:32:00 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/22 08:32:00 activator: one.example.com hibernated
2026/10/22 08:32:00 activator: instance bots.example.com exited unexpectedly — recycling
2026/10/22 08:32:00 activator: bots.example.com hibernated
2026/10/22 08:32:00 activator: one.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: one.example.com hibernated
2026/10/22 08:32:00 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:32:00 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:32:00 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:32:00 activator: bot.x.internal hibernated
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: swap.example.com hibernated
2026/10/22 08:32:00 activator: deaf.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: next-due probe deaf.example.com: dial unix: no such file — recording none (it sleeps until a request)
2026/10/22 08:32:00 activator: deaf.example.com hibernated
2026/10/22 08:32:00 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:32:00 activator: faroff.bab.oncarlos.com idle 16m0s — hibernating
2026/10/22 08:32:00 activator: faroff.bab.oncarlos.com hibernated
2026/10/22 08:32:00 activator: bots.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: bots.example.com hibernated
2026/10/22 08:32:00 activator: SIDECAR for botless.example.com failed to start: exec: live symlink gone — serving without it (the sweep retries)
2026/10/22 08:32:00 activator: reborn.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: reborn.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/22 08:32:00 activator: reborn.example.com hibernated
2026/10/22 08:32:00 activator: due.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: due.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/22 08:32:00 activator: due.example.com hibernated
2026/10/22 08:32:00 activator: late.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: late.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/22 08:32:00 activator: late.example.com hibernated
2026/10/22 08:32:00 activator: SIDECAR restart for botless.example.com failed: exec: live symlink gone (next attempt in 2s)
2026/10/22 08:32:00 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:32:00 activator: reborn.example.com has a scheduled delivery due — waking it
2026/10/22 08:32:00 activator: late.example.com has a scheduled delivery due — waking it
2026/10/22 08:32:00 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:32:00 activator: due.example.com has a scheduled delivery due — waking it
2026/10/22 08:32:00 activator: sidecar for botless.example.com was down — restarted (backoff now 4s)
2026/10/22 08:32:00 activator: SIDECAR for partial-sidecar.example failed to start: sidecar started then failed — retaining returned process
2026/10/22 08:32:00 activator: due.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: due.example.com hibernated
2026/10/22 08:32:00 activator: partial-sidecar.example hibernated
2026/10/22 08:32:00 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/22 08:32:00 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:32:00 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/22 08:32:00 activator: bot.x.internal hibernated
2026/10/22 08:32:00 activator: bot.x.internal hibernated
2026/10/22 08:32:00 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/22 08:32:00 activator: one.example.com idle 1m0s — hibernating
2026/10/22 08:32:00 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/22 08:32:00 activator: one.example.com hibernated
2026/10/22 08:32:00 activator: sidecar for bots.example.com was down — restarted (backoff now 8s)
2026/10/22 08:32:00 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:32:00 activator: f6.example.com: failed wake's generation 4 ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:32:00 activator: swap.example.com hibernated
2026/10/22 08:32:00 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:32:00 activator: bot.x.internal hibernated
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:32:00 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/22 08:32:00 activator: one.example.com hibernated
2026/10/22 08:32:00 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/22 08:32:00 activator: SIDECAR restart for partial-sidecar.example failed: sidecar started then failed — retaining returned process
2026/10/22 08:32:00 activator: two.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/22 08:32:00 activator: partial-sidecar.example hibernated
2026/10/22 08:32:00 activator: swap.example.com hibernated
2026/10/22 08:32:00 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/22 08:32:00 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:01 activator: bot.x.internal hibernated
2026/10/22 08:32:01 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/22 08:32:01 activator: bot.x.internal hibernated
2026/10/22 08:32:01 activator: bot.x.internal hibernated
FAIL
FAIL	github.com/carlosframework/platform/internal/activator	59.684s
ok  	github.com/carlosframework/platform/internal/agentsock	0.034s
ok  	github.com/carlosframework/platform/internal/alerting	0.010s
ok  	github.com/carlosframework/platform/internal/analytics	0.029s
ok  	github.com/carlosframework/platform/internal/appconfig	2.878s
ok  	github.com/carlosframework/platform/internal/apperr	0.133s
ok  	github.com/carlosframework/platform/internal/applog	0.352s
ok  	github.com/carlosframework/platform/internal/awsinv	0.009s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.008s
ok  	github.com/carlosframework/platform/internal/boxid	0.856s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.014s
ok  	github.com/carlosframework/platform/internal/caps	0.019s
ok  	github.com/carlosframework/platform/internal/certwarm	0.003s
ok  	github.com/carlosframework/platform/internal/cgroup	0.757s
ok  	github.com/carlosframework/platform/internal/changelog	0.488s
ok  	github.com/carlosframework/platform/internal/claimactivation	1.206s
ok  	github.com/carlosframework/platform/internal/clamd	0.057s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	5.654s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.011s
ok  	github.com/carlosframework/platform/internal/dbverify	0.006s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.005s
ok  	github.com/carlosframework/platform/internal/docsite	0.005s
ok  	github.com/carlosframework/platform/internal/domains	2.867s
ok  	github.com/carlosframework/platform/internal/doorkey	0.013s
ok  	github.com/carlosframework/platform/internal/economics	0.007s
ok  	github.com/carlosframework/platform/internal/edge	45.219s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.009s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.192s
ok  	github.com/carlosframework/platform/internal/gate	1.933s
ok  	github.com/carlosframework/platform/internal/genselect	0.024s
ok  	github.com/carlosframework/platform/internal/geo	0.029s
ok  	github.com/carlosframework/platform/internal/geoip	0.033s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.282s
ok  	github.com/carlosframework/platform/internal/host	138.962s
ok  	github.com/carlosframework/platform/internal/hostactors	6.918s
ok  	github.com/carlosframework/platform/internal/hostactors/awsmembership	0.011s
ok  	github.com/carlosframework/platform/internal/hostbox	3.167s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.836s
ok  	github.com/carlosframework/platform/internal/hostenv	0.007s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.007s
ok  	github.com/carlosframework/platform/internal/hostexec	5.534s
ok  	github.com/carlosframework/platform/internal/hostfence	0.068s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.278s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.028s
ok  	github.com/carlosframework/platform/internal/hostguard	1.030s
ok  	github.com/carlosframework/platform/internal/hostops	2.260s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.002s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.005s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.575s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.230s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.003s
ok  	github.com/carlosframework/platform/internal/infraview	0.004s
ok  	github.com/carlosframework/platform/internal/instancectl	5.250s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.776s
ok  	github.com/carlosframework/platform/internal/instancefs	0.008s
ok  	github.com/carlosframework/platform/internal/instancekey	0.003s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.007s
ok  	github.com/carlosframework/platform/internal/layout	1.943s
ok  	github.com/carlosframework/platform/internal/lbhealth	0.008s
ok  	github.com/carlosframework/platform/internal/lease	10.252s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.508s
ok  	github.com/carlosframework/platform/internal/ledger	0.014s
ok  	github.com/carlosframework/platform/internal/mail	0.220s
ok  	github.com/carlosframework/platform/internal/maintenance	0.083s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.590s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.089s
ok  	github.com/carlosframework/platform/internal/metrics	0.005s
ok  	github.com/carlosframework/platform/internal/moves	21.401s
ok  	github.com/carlosframework/platform/internal/moves/archive	1.079s
ok  	github.com/carlosframework/platform/internal/objstore	0.253s
ok  	github.com/carlosframework/platform/internal/ops	0.214s
ok  	github.com/carlosframework/platform/internal/park	0.281s
ok  	github.com/carlosframework/platform/internal/plans	0.004s
ok  	github.com/carlosframework/platform/internal/privatefile	0.004s
ok  	github.com/carlosframework/platform/internal/provision	13.312s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.009s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.675s
ok  	github.com/carlosframework/platform/internal/relay	7.233s
ok  	github.com/carlosframework/platform/internal/release	24.209s
ok  	github.com/carlosframework/platform/internal/rollout	3.802s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.005s
ok  	github.com/carlosframework/platform/internal/rootaws	0.008s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.058s
ok  	github.com/carlosframework/platform/internal/rootstate	0.027s
ok  	github.com/carlosframework/platform/internal/runs	0.563s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.007s
ok  	github.com/carlosframework/platform/internal/schedule	2.692s
ok  	github.com/carlosframework/platform/internal/seeds	0.011s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	10.615s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	4.029s
ok  	github.com/carlosframework/platform/internal/status	0.080s
ok  	github.com/carlosframework/platform/internal/steering	0.005s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.170s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.005s
ok  	github.com/carlosframework/platform/internal/unixdial	0.002s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.007s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.650s
FAIL
=== internal_console@0 (77 s)
ok  	github.com/carlosframework/platform/internal/console	74.955s
=== internal_console@1 (37 s)
ok  	github.com/carlosframework/platform/internal/console	34.690s
=== internal_console@2 (42 s)
ok  	github.com/carlosframework/platform/internal/console	39.465s
=== internal_console@3 (37 s)
ok  	github.com/carlosframework/platform/internal/console	35.049s
=== internal_console@4 (37 s)
ok  	github.com/carlosframework/platform/internal/console	34.729s
=== internal_console@5 (35 s)
ok  	github.com/carlosframework/platform/internal/console	32.560s
=== internal_console@6 (36 s)
ok  	github.com/carlosframework/platform/internal/console	34.445s
=== internal_console@7 (43 s)
ok  	github.com/carlosframework/platform/internal/console	40.652s
=== root@0 (32 s)
ok  	github.com/carlosframework/platform	30.000s
=== root@1 (44 s)
ok  	github.com/carlosframework/platform	41.895s
=== root@2 (41 s)
ok  	github.com/carlosframework/platform	38.945s

FAILED: rest
make: *** [Makefile:19: test] Error 1

amadan: step gate failed: exit status 2

cross not run

No output.
minio-contract not run

No output.

Whole run log

=== gate ===
.amadan/test-gate.sh
=== launchctx (0 s)
ok  	amadan.net/carlos/platform.git/launchctx	0.013s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
=== vet (1 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (142 s)
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.644s
ok  	github.com/carlosframework/platform/cmd/moveprobe	11.852s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.750s
2026/10/22 08:31:01 activator: badseed.example.com: seed for badseed.example.com: activator: seed unproven; processes remain: processes remain after a successful seed: still running; leaving the lease to its TTL
2026/10/22 08:31:01 activator: unproven.example.com: unproven.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone; leaving the lease to its TTL
2026/10/22 08:31:01 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/22 08:31:01 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof4147963793/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof4147963793/004/next-wake.example.com.db
2026/10/22 08:31:01 activator: next-wake.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof4147963793/004/next-wake.example.com.db.seed-unproven refuses it as warm until they are
2026/10/22 08:31:01 activator: next-wake.example.com: seeding next-wake.example.com: exit status 1: activator: seed unproven; processes remain (the seed's processes remain after stopping it; its database was left in place: cgroup: strict containment unsupported); leaving the lease to its TTL
2026/10/22 08:31:01 activator: next-wake.example.com: next-wake.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup: strict containment unsupported; leaving the lease to its TTL
2026/10/22 08:31:01 activator: next-wake.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/22 08:31:01 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/22 08:31:01 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof4147963793/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof4147963793/004/next-wake.example.com.db
2026/10/22 08:31:01 activator: seeded next-wake.example.com
2026/10/22 08:31:01 activator: instance one.example.com exited unexpectedly — recycling
2026/10/22 08:31:01 activator: one.example.com hibernated
2026/10/22 08:31:02 activator: LEASE HELD for held.example.com by "old-incarnation" — a previous incarnation hasn't released; refusing to wake (retry after 7s)
2026/10/22 08:31:02 activator: two.example.com hibernated
2026/10/22 08:31:02 activator: one.example.com hibernated
2026/10/22 08:31:02 activator: b.example.com hibernated
2026/10/22 08:31:02 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:02 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:02 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:02 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:02 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/22 08:31:02 activator: bot.x.internal hibernated
2026/10/22 08:31:03 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendStartMergesConfigEnvUnderOwnEnv3044231293/002/merge-precedence.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/22 08:31:03 activator: purged 1 cached response(s) for deploy.example.com on spawn
2026/10/22 08:31:03 activator: restored restored.example.com from s3://test-bucket/restored.example.com
2026/10/22 08:31:03 activator: damaged.example.com restored CORRUPT from s3://test-bucket/damaged.example.com: integrity_check: file is not a database (26) — quarantined at /tmp/TestExecBackendRestoreCorruptReplicaFailsAndQuarantines1047758299/002/damaged.example.com.db.corrupt, instance NOT started
2026/10/22 08:31:03 activator: restored damaged.example.com from s3://test-bucket/damaged.example.com
2026/10/22 08:31:03 activator: no replica for brand-new.example.com in s3://test-bucket/brand-new.example.com — booting fresh
2026/10/22 08:31:03 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1055553186/002/seeded.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/22 08:31:03 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1055553186/001/live/seeded.example.com seed --fixtures --db /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath1055553186/002/seeded.example.com.db
2026/10/22 08:31:03 activator: seeded seeded.example.com
2026/10/22 08:31:03 activator: seeding custom-db.example.com: /tmp/TestExecBackendSeedHonoursRouteDBPath2429712151/001/live/custom-db.example.com seed --db /tmp/TestExecBackendSeedHonoursRouteDBPath2429712151/004/elsewhere.db
2026/10/22 08:31:03 activator: seeded custom-db.example.com
2026/10/22 08:31:03 activator: seeding badseed.example.com: /tmp/TestExecBackendSeedNonZeroExitFails2537431860/001/live/badseed.example.com seed --db /tmp/TestExecBackendSeedNonZeroExitFails2537431860/002/badseed.example.com.db
schema is corrupt
2026/10/22 08:31:03 activator: badseed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:31:03 activator: seeding hangs.example.com: /tmp/TestExecBackendSeedIsTimeBounded3684001735/001/live/hangs.example.com seed --db /tmp/TestExecBackendSeedIsTimeBounded3684001735/002/hangs.example.com.db
2026/10/22 08:31:03 activator: hangs.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:31:03 activator: seeding crashed.example.com: /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB1910316173/001/live/crashed.example.com seed --db /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB1910316173/002/crashed.example.com.db
died mid-schema
2026/10/22 08:31:03 activator: crashed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:31:03 activator: seeding existing-db.example.com: /tmp/TestExecBackendSeedFailureKeepsPreexistingDB853910656/001/live/existing-db.example.com seed --db /tmp/TestExecBackendSeedFailureKeepsPreexistingDB853910656/002/existing-db.example.com.db
nope
2026/10/22 08:31:03 activator: existing-db.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:31:03 activator: seeding leaky-seed.example.com: /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild1580889491/001/live/leaky-seed.example.com seed --db /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild1580889491/002/leaky-seed.example.com.db
2026/10/22 08:31:04 activator: seeded leaky-seed.example.com (it exited 0 but left a child holding its output pipe open for more than 200ms)
2026/10/22 08:31:04 activator: sidecar for a.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:04 activator: h.example.com: route refreshed: db "/tmp/h.example.com.db" -> "/data/instances/h.example.com/h.example.com.db" socket "/tmp/nope.sock" -> "/run/carlos/instances/h.example.com/app.sock"
2026/10/22 08:31:04 activator: stop stale.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/22 08:31:05 activator: stale.example.com hibernated
2026/10/22 08:31:05 activator: leftover.example.com: the host cgroup holds processes not proven gone: a leftover still populates the host cgroup; leaving the lease to its TTL
2026/10/22 08:31:05 activator: old-kernel.example.com: this kernel has no cgroup.kill, so the host cgroup cannot be killed; its leader's exit is the only proof there is
2026/10/22 08:31:05 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/22 08:31:05 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/22 08:31:05 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/22 08:31:05 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/22 08:31:05 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/22 08:31:05 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/22 08:31:05 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/22 08:31:05 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/22 08:31:05 activator: retry.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:05 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/22 08:31:05 activator: retry.example.com: strict group not closed: cgroup: strict containment refused
2026/10/22 08:31:05 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/22 08:31:06 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:06 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:06 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:06 activator: stuck.example.com: strict group still not cleaned up after 2 retries; spawns of this host stay refused until the agent restarts
2026/10/22 08:31:06 activator: owned.example.com: strict group not closed: cgroup: strict containment refused
2026/10/22 08:31:06 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/22 08:31:06 activator: owned.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:07 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/22 08:31:07 activator: race.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:07 activator: transition.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:07 activator: transition2.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/22 08:31:07 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/22 08:31:07 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/22 08:31:07 activator: kass.example.com hibernated
2026/10/22 08:31:07 activator: kass.example.com idle 11s — hibernating
2026/10/22 08:31:07 activator: kass.example.com hibernated
2026/10/22 08:31:08 activator: one.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/22 08:31:08 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:08 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:08 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:08 activator: bot.x.internal: v2 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:08 activator: worker bot.x.internal NOT respawned (attempt (release)): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/22 08:31:08 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:08 activator: link.example.com hibernated
2026/10/22 08:31:12 activator: tenant.test: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:12 activator: tenant.test not started: the executable it hashed (/tmp/rl182650569/app) is build "v1", but the start names build "v2" (the live link moved between the two reads); the next wake reads it again
2026/10/22 08:31:12 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:12 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/22 08:31:12 activator: replica h.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/22 08:31:12 activator: replica h.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/22 08:31:12 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/22 08:31:12 activator: why.example.com hibernated
2026/10/22 08:31:12 activator: c.example.com hibernated
2026/10/22 08:31:13 activator: h.example.com hibernated
2026/10/22 08:31:13 activator: h.example.com: run run_a is not proven stopped: cgroup.events: permission denied
2026/10/22 08:31:13 activator: stop h.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:13 activator: h.example.com hibernated
2026/10/22 08:31:13 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/22 08:31:13 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/22 08:31:13 activator: seeding unproven-seed.example.com: /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3520681911/002/live/unproven-seed.example.com seed --db /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3520681911/003/unproven-seed.example.com.db
2026/10/22 08:31:13 activator: unproven-seed.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3520681911/003/unproven-seed.example.com.db.seed-unproven refuses it as warm until they are
2026/10/22 08:31:13 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3278024083/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3278024083/003/obligation.example.com.db
2026/10/22 08:31:13 activator: obligation.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3278024083/003/obligation.example.com.db.seed-unproven refuses it as warm until they are
2026/10/22 08:31:13 activator: obligation.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/22 08:31:13 activator: no replica for obligation.example.com in s3://test-bucket/obligation.example.com — booting fresh
2026/10/22 08:31:13 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3278024083/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm3278024083/003/obligation.example.com.db
2026/10/22 08:31:13 activator: seeded obligation.example.com
2026/10/22 08:31:13 activator: seeding helper-seed.example.com: /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt877537303/002/live/helper-seed.example.com seed --db /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt877537303/003/helper-seed.example.com.db
2026/10/22 08:31:13 activator: seeded helper-seed.example.com
2026/10/22 08:31:13 activator: seeding helper-unproven.example.com: /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven2894699476/002/live/helper-unproven.example.com seed --db /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven2894699476/003/helper-unproven.example.com.db
2026/10/22 08:31:13 activator: helper-unproven.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven2894699476/003/helper-unproven.example.com.db.seed-unproven refuses it as warm until they are
2026/10/22 08:31:13 activator: seeding write-ahead.example.com: /tmp/TestASeedRecordsItsObligationBeforeItRuns1348733180/002/live/write-ahead.example.com seed --db /tmp/TestASeedRecordsItsObligationBeforeItRuns1348733180/003/write-ahead.example.com.db
2026/10/22 08:31:13 activator: seeded write-ahead.example.com
2026/10/22 08:31:13 activator: write-ahead.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/22 08:31:13 activator: no replica for write-ahead.example.com in s3://test-bucket/write-ahead.example.com — booting fresh
2026/10/22 08:31:13 activator: seeding no-room.example.com: /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing1851767913/002/live/no-room.example.com seed --db /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing1851767913/003/no-room.example.com.db
2026/10/22 08:31:13 activator: seeding stuck-wal.example.com: /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation1929804846/002/live/stuck-wal.example.com seed --db /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation1929804846/003/stuck-wal.example.com.db
2026/10/22 08:31:13 activator: seeding preexisting.example.com: /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2856154708/002/live/preexisting.example.com seed --db /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2856154708/003/preexisting.example.com.db
2026/10/22 08:31:13 activator: preexisting.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2856154708/003/preexisting.example.com.db.seed-unproven refuses it as warm until they are
2026/10/22 08:31:13 activator: preexisting.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/22 08:31:13 activator: old-kernel-seed.example.com: this kernel has no cgroup.kill; the seed's leader's exit is the only proof there is
2026/10/22 08:31:13 activator: seeding failed-role-seed.example.com: /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB2647495277/002/live/failed-role-seed.example.com seed --db /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB2647495277/003/failed-role-seed.example.com.db
2026/10/22 08:31:13 activator: seeding discharge-fails.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai3392541116/002/live/discharge-fails.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai3392541116/003/discharge-fails.example.com.db
2026/10/22 08:31:13 activator: discharge-fails.example.com: the seed's obligation could not be discharged and its processes are not proven gone; /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai3392541116/003/discharge-fails.example.com.db.seed-unproven stays
2026/10/22 08:31:13 activator: seeding discharge-fails-proven.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur3883688648/002/live/discharge-fails-proven.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur3883688648/003/discharge-fails-proven.example.com.db
2026/10/22 08:31:13 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_B,_l1535638335/003/g/56/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:13 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall2376683386/003/g/57/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:13 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall2376683386/003/g/58/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:13 activator: swap.example.com hibernated
2026/10/22 08:31:13 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatholds_pinne658209731/007/g/60/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:13 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/22 08:31:13 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatoverlap_succ1990650775/007/g/62/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:13 activator: h.example.com: started carlos-instance-gen@h.example.com:64.service memory.max=536870912
2026/10/22 08:31:13 activator: swap.example.com hibernated
2026/10/22 08:31:13 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/22 08:31:14 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_super3663884204/007/g/71/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:14 activator: swap.example.com hibernated
2026/10/22 08:31:14 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartholds_pinned_build,_lea3541125021/007/g/73/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:14 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/22 08:31:14 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_a_fai3267201203/007/g/75/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:14 activator: fence.example hibernated
2026/10/22 08:31:14 activator: fence.example hibernated
2026/10/22 08:31:14 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink214055692/003/g/c.x.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink214055692/003/g/c.x.sock: no such file or directory)
2026/10/22 08:31:14 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink214055692/003/g/c.xx.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink214055692/003/g/c.xx.sock: no such file or directory)
2026/10/22 08:31:14 activator: publish generations to /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile3766591572/002/not-a-dir/zdd/generations.json: open directory /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile3766591572/002/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:14 activator: ident.example.com hibernated
2026/10/22 08:31:14 activator: instance ident.example.com exited unexpectedly — recycling
2026/10/22 08:31:14 activator: ident.example.com hibernated
2026/10/22 08:31:14 activator: due.example.com hibernated
2026/10/22 08:31:14 activator: due.example.com hibernated
2026/10/22 08:31:14 activator: ret.example.com hibernated
2026/10/22 08:31:14 activator: ret.example.com hibernated
2026/10/22 08:31:14 activator: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.example.com: a generation socket path would exceed 107 bytes; generation 9 keeps the canonical socket
2026/10/22 08:31:15 activator: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.example.com: a generation socket path would exceed 107 bytes; generation 9 keeps the canonical socket
2026/10/22 08:31:15 activator: one.example.com hibernated
2026/10/22 08:31:15 activator: purged 1 cached response(s) for flat.example.com on spawn
2026/10/22 08:31:16 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile3396821082/001/versions/vA/app seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile3396821082/002/seedexe.example.com.db
2026/10/22 08:31:16 activator: seeded seedexe.example.com
2026/10/22 08:31:16 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile3396821082/001/live/seedexe.example.com seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile3396821082/002/seedexe.example.com.db
2026/10/22 08:31:16 activator: seeded seedexe.example.com
2026/10/22 08:31:16 activator: seeding h.example.com: /tmp/TestNestedSeedExeNamesTheRecordedFile2170313703/001/live/h.example.com seed --db /tmp/nb2993019360/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:16 activator: seeded h.example.com
2026/10/22 08:31:16 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3930596972/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/22 08:31:16 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3930596972/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/22 08:31:16 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3930596972/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/22 08:31:16 activator: seeding h.example.com: /tmp/TestADeclinedNestedSeedRunsThePinnedFile557346830/001/live/h.example.com seed --db /tmp/nb3854069580/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:16 activator: seeded h.example.com
2026/10/22 08:31:16 activator: seeding tier.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed4274476984/002/versions/vA/app seed --db /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed4274476984/003/tier.example.com.db
2026/10/22 08:31:16 activator: tier.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed4274476984/003/tier.example.com.db.seed-unproven refuses it as warm until they are
2026/10/22 08:31:16 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=1073741824
2026/10/22 08:31:16 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=1073741824
2026/10/22 08:31:16 activator: seeding h.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTiernested_seed1779931139/001/live/h.example.com seed --db /tmp/nb1074425907/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:16 activator: seeded h.example.com
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsthe_configured_k3412559592/001/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsan_own_key_pres3032516552/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestFirstStartWaitsForTheConfiguredKey2445548724/002/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: control listener for shop.example.com: listen unix /tmp/TestFirstStartWaitsForTheConfiguredKey2445548724/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY in /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptinto_the_own_file4050879215/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptdelivered_in_the_co414423756/001/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration2802714138/001/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration2802714138/002/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyLeavesAFileWithTwoKeysAlone3860004338/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistse4060251211/002/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistsn335831397/002/shop.example.com.env: the platform mints none while it is
2026/10/22 08:31:16 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance2318189224/001/live/seeded.example.com seed --db /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance2318189224/002/seeded.example.com.db
2026/10/22 08:31:16 activator: seeded seeded.example.com
2026/10/22 08:31:16 activator: restored sleepy.example.com from s3://test-bucket/sleepy.example.com
hello
world
2026/10/22 08:31:17 activator: fence.example hibernated
2026/10/22 08:31:17 activator: fence.example hibernated
2026/10/22 08:31:17 activator: fence.example hibernated
2026/10/22 08:31:17 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/22 08:31:17 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/22 08:31:17 activator: barrier.example.com hibernated
2026/10/22 08:31:17 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/22 08:31:17 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/22 08:31:17 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/22 08:31:17 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/22 08:31:17 activator: barrier.example.com hibernated
2026/10/22 08:31:17 activator: barrier.example.com: generation 12's child is not proven empty: g12 still holds a process
2026/10/22 08:31:17 activator: generation 12 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/22 08:31:17 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/22 08:31:17 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:17 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/22 08:31:17 activator: barrier.example.com hibernated
2026/10/22 08:31:17 activator: stop generation 11: cgroup.kill: the child is not empty
2026/10/22 08:31:17 activator: generation 11 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/22 08:31:18 activator: placed.example.com: kill generation 1097 after a failed health wait: g still holds a process
2026/10/22 08:31:18 activator: placed.example.com: generation 1097's child is not proven empty: g still holds a process
2026/10/22 08:31:18 activator: placed.example.com: failed wake's generation 1097 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:18 activator: placed.example.com: wake failed and its generation is not proven stopped: start placed.example.com: instance not healthy within 300ms: Get "http://carlos/": dial unix /tmp/g3228977635/g/1097/placed.example.com.sock: connect: no such file or directory; activator: generation start failed and its termination is unproven: kill generation 1097 after a failed health wait: g still holds a process
2026/10/22 08:31:18 activator: placed.example.com: failed wake's generation 1097 is proven stopped; lease given back
2026/10/22 08:31:18 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:18 activator: unproven.example.com: failed wake's generation 1100 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:18 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/22 08:31:18 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:18 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:19 activator: unproven.example.com: failed wake's generation 1100 is proven stopped; lease given back
2026/10/22 08:31:19 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:19 activator: unproven.example.com: failed wake's generation 1101 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:19 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/22 08:31:19 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:19 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:19 activator: unproven.example.com: failed wake's generation 1101 is proven stopped; lease given back
2026/10/22 08:31:19 activator: unproven.example.com: generation 1102's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:19 activator: unproven.example.com: failed wake's generation 1102 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:19 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/22 08:31:19 activator: unproven.example.com: generation 1102's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:20 activator: unhealthy-f6.example.com: failed wake's generation 1103 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:20 activator: unhealthy-f6.example.com: failed wake's generation 1103 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:20 activator: unhealthy-f6.example.com: failed wake's generation 1103 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:20 activator: unhealthy-f6.example.com: failed wake's generation 1103 is proven stopped; lease given back
2026/10/22 08:31:20 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/22 08:31:20 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/22 08:31:21 activator: instance one.example.com exited unexpectedly — recycling
2026/10/22 08:31:21 activator: one.example.com hibernated
2026/10/22 08:31:21 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:21 activator: bot.x.internal hibernated
2026/10/22 08:31:21 activator: worker bot.x.internal respawned (attempt (restart)), now running v2
2026/10/22 08:31:21 activator: bot.x.internal hibernated
2026/10/22 08:31:21 activator: worker bot.x.internal respawned (attempt (config)), now running v2
2026/10/22 08:31:21 activator: bot.x.internal hibernated
2026/10/22 08:31:21 activator: bot.x.internal hibernated
2026/10/22 08:31:21 activator: bot.x.internal hibernated
2026/10/22 08:31:21 activator: bot.x.internal hibernated
2026/10/22 08:31:21 activator: moved.example.com: promoting 2 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/22 08:31:21 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:21 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:21 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:21 activator: x.example hibernated
2026/10/22 08:31:21 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/22 08:31:21 activator: x.example hibernated
2026/10/22 08:31:21 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/22 08:31:21 activator: x.example hibernated
2026/10/22 08:31:21 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:21 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:21 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:21 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/22 08:31:21 activator: x.example hibernated
2026/10/22 08:31:22 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:22 activator: x.example hibernated
2026/10/22 08:31:22 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:22 activator: x.example hibernated
2026/10/22 08:31:22 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:22 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:22 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:22 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:22 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:22 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:22 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:22 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_keyless_replica_is_refu1590799808/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_is_1803079236/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_pre-existing_env_key_th527459339/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_dif3628283046/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_with_no_999976661/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY cannot be checked against its home's key (no entry for it in /etc/carlos/replica-home-keys; O8)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_home_updated,_this_re1781334059/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY is not the one the bucket's configuration now holds: a key change is in flight (ruling 27)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_bucket_cannot_be_read4026021051/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: the bucket's configuration could not be read: bucket unreachable
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketno_fresh_reader_refused2733906244/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY cannot be checked against the bucket's configuration
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketresidual_a_change_only_o3209305564/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_config-only_homes_keyl355447216/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_home_row_mints_and_its_1387110742/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 activator: seeding shop-au.example.com: /tmp/TestReplicaSeedIsGatedthe_homes_key1454341284/001/live/shop-au.example.com seed --db /tmp/TestReplicaSeedIsGatedthe_homes_key1454341284/002/shop-au.example.com.db
2026/10/22 08:31:22 activator: seeded shop-au.example.com
2026/10/22 08:31:22 activator: replica shop-au.example.com not started: where its CARLOS_INSTANCE_KEY came from is unknown
2026/10/22 08:31:22 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaGateDecidesOnTheLaunchesOwnRead1102268350/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:22 provision: a.shop.example.com: swapped for import sd_rc5xz7vxkj7s7hv7hevkflwuwy (the original route is journaled)
2026/10/22 08:31:22 provision: a.shop.example.com: import sd_rc5xz7vxkj7s7hv7hevkflwuwy rolled back to its original route
2026/10/22 08:31:22 provision: a.shop.example.com: import sd_rc5xz7vxkj7s7hv7hevkflwuwy is cleaned up and its journal row retired
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_zy2iqqyep5t4u54p5qxzowf74m (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_lpuhntwik4krxujelbdooiui4q (8192 bytes)
2026/10/22 08:31:22 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/nb3086044622/data/instances/h.example.com/h.example.com.env: the platform mints none while it is
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_yvojngjpvievk7yeafnrho2jbi (8192 bytes)
2026/10/22 08:31:22 activator: CARLOS_INSTANCE_KEY in /tmp/nb605777362/data/instances/h.example.com/h.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_kx7zxdenqrtdvo2tefvmrn46p4 (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_smaylqdvv2iufpnbtj6rtjbqre (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_tofpfe7pfirkqqej33va26cgue (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: keeping seed sd_tofpfe7pfirkqqej33va26cgue placed by an earlier wake
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_ifi5jaihjkpqac3kahew3x2n4u (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_o4zdy6cutydv6nseojz6rxavja (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_pfsvewhzcsvcdvhysgtvq6b2qu (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: seed sd_3nsdb6xblybbbn42k2lld3tefu: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/22 08:31:22 activator: h.example.com: seed sd_ulm3vzvz5xarbqyaoisgnmmcfm: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/22 08:31:22 activator: h.example.com: seed sd_wfyhw3sxbcqw66eat37lbupwsm: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/22 08:31:22 activator: h.example.com: seed sd_fkc63qv5r2ccoijy2i6jhwgu3e: h.example.com could not start from its seed: the snapshot could not be read. The next wake tries again.: bucket down
2026/10/22 08:31:22 activator: h.example.com: seed sd_243ise37pckxlzppmpiaik62vy: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/22 08:31:22 activator: h.example.com: seed sd_5y4kyu2zfetwfxd745fwblocee: h.example.com could not start from its seed: the snapshot is not a database that passes an integrity check.: integrity_check: file is not a database (26)
2026/10/22 08:31:22 activator: h.example.com: seed sd_cuk25uerbhlugefkb5stclfaca: h.example.com could not start from its seed: claim control is on for this box.
2026/10/22 08:31:22 activator: h.example.com: seed sd_cuk25uerbhlugefkb5stclfaca: h.example.com could not start from its seed: this box does not run instances nested.
2026/10/22 08:31:22 activator: h.example.com: seed sd_ezuf6wbq3uiwr34t2rh7chi45a: seed fallback for h.example.com: cannot tell whether it has a replica of its own: listing failed
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: h.example.com: seed sd_egbc3otufnfcqvzbtfz2lcxytm: seed fallback for h.example.com: found no replica of its own to restore
2026/10/22 08:31:22 activator: h.example.com: promoting 1 staged object(s) from moves/h.example.com/ (residence move landing)
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: h.example.com: seed sd_tqvwbu7a4ezadpt3yqqvni2beq: seed fallback for h.example.com: found no replica of its own to restore
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: h.example.com: seed sd_ywbevcmxoviriwdxz3kjlrpu6q: seed fallback for h.example.com: found no replica of its own to restore
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_vlk46ezauetfktqvyeixxgdhem (8192 bytes)
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: h.example.com: seed sd_llubg3dkwanzk3qacutb6cirqa: h.example.com could not start from its seed: the snapshot could not be read. The next wake tries again.: connection reset
2026/10/22 08:31:22 activator: h.example.com: seed sd_5epgqre5qv6vy7vsutqpqmnpce: h.example.com could not start from its seed: it could not be put in place. The next wake tries again.: openat2 /tmp/nb3208243919/data/instances/h.example.com/.seed-sd_5epgqre5qv6vy7vsutqpqmnpce.tmp: permission denied
2026/10/22 08:31:22 activator: h.example.com: seed sd_kr2ptbbg2qyhmhf5ylpnnt2xk4: h.example.com could not start from its seed: the snapshot is not a database that passes an integrity check.: open: unable to open database file (14)
2026/10/22 08:31:22 activator: h.example.com: seed sd_l64lsch7s2lad7r63kpfqjcvpu: h.example.com could not start from its seed: it could not be put in place. The next wake tries again.: instancefs: path contains a symlink: /tmp/nb2235448171/data/instances/h.example.com/.seed-sd_l64lsch7s2lad7r63kpfqjcvpu.tmp
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_sckuz2ncca7afqsixezqib4g6e (8192 bytes)
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: h.example.com: seed sd_6rgi2m55bidltd554nzpd3wqj4 not recorded as placed yet: record unreachable
2026/10/22 08:31:22 activator: h.example.com: /tmp/nb2781168839/data/instances/h.example.com/h.example.com.db held seed sd_hmj3bm6r7yjocvtkfc6n5rex6m's marker, not this route's (seed "sd_6cuk6sgndtemiianj2fzpnayg4"): purged before the restore
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: h.example.com: /tmp/nb3527812220/data/instances/h.example.com/h.example.com.db held seed sd_vpclpsndt5vcyfrpbkntlpnffq's marker, not this route's (seed "sd_hrkclcg4q3qlludgyx6jwga3pa"): purged before the restore
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: h.example.com: /tmp/nb1243637768/data/instances/h.example.com/h.example.com.db held seed sd_qz2n4eijtnlonbsgfh3kqxem2u's marker, not this route's (seed ""): purged before the restore
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: h.example.com: /tmp/nb645690783/data/instances/h.example.com/h.example.com.db held seed sd_p6qz3ujoh22rkvt7patsp55pn4's marker, not this route's (seed "sd_impecdpcqqjetx46gykiq34nnu"): purged before the restore
2026/10/22 08:31:22 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/22 08:31:22 activator: h.example.com: /tmp/nb2048621708/data/instances/h.example.com/h.example.com.db held seed sd_d2engyzwqh6lzleirzimfwa3ly's marker, not this route's (seed ""): purged before the restore
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: h.example.com: seed sd_adwxwmu2fwtydrixrg5msvb6ry: h.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup.events: unreadable
2026/10/22 08:31:22 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_adwxwmu2fwtydrixrg5msvb6ry (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: placed seed sd_55ookp4oacpldff2weglnv63pm (8192 bytes)
2026/10/22 08:31:22 activator: h.example.com: /tmp/nb3259886175/data/instances/h.example.com/h.example.com.db held seed an aborted import's data's marker, not this route's (seed ""): purged before the restore
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:22 activator: swap.example.com hibernated
2026/10/22 08:31:22 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/22 08:31:22 activator: swap.example.com hibernated
2026/10/22 08:31:22 activator: LEASE LOST for swap.example.com (renewal failed) — standing down
2026/10/22 08:31:22 activator: swap.example.com hibernated
2026/10/22 08:31:22 activator: swap.example.com hibernated
2026/10/22 08:31:22 activator: swap.example.com is no longer hibernate-managed — releasing
2026/10/22 08:31:22 activator: swap.example.com hibernated
2026/10/22 08:31:22 activator: swap.example.com hibernated
2026/10/22 08:31:23 activator: exit.example.com hibernated
2026/10/22 08:31:23 activator: wake shop.example.com: activator stopped during startup
2026/10/22 08:31:23 activator: wake shop.example.com: activator stopped during startup
2026/10/22 08:31:23 activator: shop.example.com: route refreshed: db "" -> "" socket "/run/carlos/shop.example.com.sock" -> "/run/carlos/instances/shop.example.com/app.sock"
2026/10/22 08:31:23 activator: stop missed-join.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/22 08:31:23 activator: missed-join.example.com hibernated
2026/10/22 08:31:23 activator: stop nested-stuck.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/22 08:31:23 activator: nested-stuck.example.com: the sidecar a restart brought back for a stopping entry is not proven gone (activator: stop unproven; processes remain); keeping the lease until it has exited
2026/10/22 08:31:24 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/22 08:31:24 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/22 08:31:24 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/22 08:31:24 activator: nested-stuck.example.com hibernated
2026/10/22 08:31:24 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:24 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:24 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:24 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/22 08:31:24 activator: bot.x.internal hibernated
2026/10/22 08:31:24 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:24 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory
2026/10/22 08:31:24 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:24 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/22 08:31:24 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:24 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:24 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory
2026/10/22 08:31:24 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:24 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_1626276145/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/22 08:31:24 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:24 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:24 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:25 activator: worker bot.x.internal: the attempt's respawn did not finish within 150ms of its caller leaving; reporting its build as possibly executed
2026/10/22 08:31:25 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:25 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing2796713498/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing2796713498/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:25 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing2796713498/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing2796713498/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:25 activator: h.example.com: started carlos-instance-gen@h.example.com:7.service memory.max=1073741824
2026/10/22 08:31:25 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: processes remain
2026/10/22 08:31:25 activator: swap.example.com hibernated
2026/10/22 08:31:32 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: instance control /stop: 502: stop: processes remain
2026/10/22 08:31:32 activator: seeding h.example.com: /tmp/TestSeedUnitCleansUpOnlyItsOwnDBOnFailure3200155845/001/live/h.example.com seed --fast --db /tmp/nb1935035825/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:32 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:32 activator: seeding h.example.com: /tmp/TestSeedUnitFailureStopsBeforeCleanupAndKeepsTheDBWhenProcessesR779426154/001/live/h.example.com seed --fast --db /tmp/nb2108990651/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:33 activator: seeding h.example.com: /tmp/TestANestedSeedRecordsItsObligationBeforeItRuns1138658861/001/live/h.example.com seed --db /tmp/nb3456414439/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:33 activator: seeded h.example.com
2026/10/22 08:31:33 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI122081684/001/live/h.example.com seed --db /tmp/nb767264685/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:33 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/22 08:31:33 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:33 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI4277421212/001/live/h.example.com seed --db /tmp/nb1116236300/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:33 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/22 08:31:33 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/22 08:31:33 activator: fifo-obligation.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/22 08:31:33 activator: one.example.com: first wake of v2 had 30s of its health window left at the exec, under the 1h0m0s floor; booting the good build v1 without trying it
2026/10/22 08:31:33 activator: one.example.com hibernated
2026/10/22 08:31:33 activator: tenant.test not started: it is held on v1, and the launch hashed "v2" (the live link moved after the check); the next wake checks again
2026/10/22 08:31:34 activator: one.example.com not woken: it is held on v1, and this wake would run "v2" (the live link moved after the check); the next wake checks again
2026/10/22 08:31:34 activator: one.example.com not woken: it is held on v1, and this wake would run "" (the live link moved after the check); the next wake checks again
2026/10/22 08:31:34 activator: swap.example.com hibernated
2026/10/22 08:31:36 activator: seeding h.example.com: /tmp/TestANestedUnversionedLinkWithNoHookWakesAsToday3198820901/001/live/h.example.com seed --db /tmp/nb2232874831/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:36 activator: seeded h.example.com
2026/10/22 08:31:36 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:37 activator: swap.example.com hibernated
2026/10/22 08:31:37 activator: one.example.com: generation 1936's child is not proven empty: activator: stop unproven; processes remain
2026/10/22 08:31:37 activator: one.example.com: failed wake's generation 1936 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:37 activator: one.example.com hibernated
2026/10/22 08:31:38 activator: one.example.com: first wake of v2 had 400ms of its health window left at the exec, under the 20s floor; booting the good build v1 without trying it
2026/10/22 08:31:38 activator: x.example: hold's pinned wake refused or abandoned: the claims admission refused before the exec: hostbox: unresolved lifecycle contract: isolated root launcher cannot carry claim reservations
2026/10/22 08:31:38 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866311826/003/g/1943/real.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:40 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866311826/003/g/1944/real.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:40 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866311826/003/g/1944/real.example.com.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866311826/003/g/1944/real.example.com.sock: no such file or directory)
2026/10/22 08:31:40 activator: real.example.com hibernated
2026/10/22 08:31:42 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign2459861641/004/hand-set.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign2459861641/004/hand-set.sock: no such file or directory)
2026/10/22 08:31:42 activator: real.example.com hibernated
2026/10/22 08:31:42 activator: bot.x.internal: worker generation 0's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:42 activator: bot.x.internal: worker generation 1948's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:42 activator: bot.x.internal: failed wake's generation 1948 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:43 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd: not a directory; the sweep retries
2026/10/22 08:31:43 activator: worker bot.x.internal not spawned: the build it would exec (v2 /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/003/v2) could not be published first: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd: not a directory
2026/10/22 08:31:43 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd: not a directory; the sweep retries
2026/10/22 08:31:43 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3352883400/002/zdd: not a directory; the sweep retries
2026/10/22 08:31:43 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn4010990313/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn4010990313/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:43 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn4010990313/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn4010990313/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:43 activator: instance one.example.com exited unexpectedly — recycling
2026/10/22 08:31:43 activator: one.example.com hibernated
2026/10/22 08:31:44 activator: one.example.com hibernated
2026/10/22 08:31:45 activator: one.example.com: first wake of v2 had 499ms of its health window left at the exec, under the 1s floor; booting the good build v1 without trying it
2026/10/22 08:31:47 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/22 08:31:47 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/22 08:31:47 activator: worker bot.example.com respawned (exited), now running 
2026/10/22 08:31:47 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/22 08:31:47 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/22 08:31:47 activator: worker bot.example.com respawned (exited), now running 
2026/10/22 08:31:47 activator: bot.example.com: worker generation 1963's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:47 activator: stop bot.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:47 activator: bot.example.com hibernated
2026/10/22 08:31:47 activator: bot3.example.com: worker generation 1964's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:47 activator: worker bot3.example.com NOT respawned (test: a restart with no old handle): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/22 08:31:47 activator: bot4.example.com: worker generation 1966's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:47 activator: bot4.example.com: failed wake's generation 1966 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:47 activator: bot4.example.com: failed wake's generation 1966 is proven stopped; lease given back
2026/10/22 08:31:47 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/22 08:31:47 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/22 08:31:47 activator: bot2.example.com: worker generation 1967's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:47 activator: worker bot2.example.com NOT respawned (exited): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/22 08:31:47 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/22 08:31:47 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/22 08:31:47 activator: worker bot2.example.com respawned (exited), now running 
2026/10/22 08:31:47 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/22 08:31:47 activator: worker bot.example.com respawned (exited), now running v1
2026/10/22 08:31:48 activator: bot.example.com hibernated
2026/10/22 08:31:48 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/22 08:31:48 activator: worker bot10.example.com: the shed respawn child's exit is not proven (activator: stop unproven; processes remain); the entry keeps it and its build until the sweep proves it
2026/10/22 08:31:48 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/22 08:31:48 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/22 08:31:48 activator: bot10.example.com: the shed respawn child has exited
2026/10/22 08:31:48 activator: bot10.example.com hibernated
2026/10/22 08:31:48 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/22 08:31:48 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/22 08:31:48 activator: bot10.example.com respawn did not finish before its stop; stopping the rest, the lease is kept until it has
2026/10/22 08:31:48 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/22 08:31:48 activator: bot10.example.com hibernated
2026/10/22 08:31:48 activator: bot8.example.com: worker generation 1970's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:48 activator: bot8.example.com: failed wake's generation 1970 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:48 activator: bot8.example.com: failed wake's generation 1970 is proven stopped; lease given back
2026/10/22 08:31:48 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:48 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:48 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/22 08:31:48 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:49 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:49 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/22 08:31:49 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/22 08:31:49 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:49 activator: bot.x.internal hibernated
2026/10/22 08:31:49 activator: worker bot.x.internal respawned (attempt (restart)), now running v1
2026/10/22 08:31:49 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:49 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:49 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:50 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:50 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:50 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:50 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:50 activator: bot.x.internal: worker generation 1973's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (attempt (release)): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/22 08:31:50 activator: bot.x.internal: worker generation 1973's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (restoring v1): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: lease lost; abandoning it
2026/10/22 08:31:50 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:50 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory
2026/10/22 08:31:50 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:50 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/22 08:31:50 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:50 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:50 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory
2026/10/22 08:31:50 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory; the sweep retries
2026/10/22 08:31:50 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3562843535/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/22 08:31:50 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:50 activator: bot.x.internal hibernated
2026/10/22 08:31:50 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: lease lost; abandoning it
--- FAIL: TestAWorkerFallbackWithoutOwnershipSpawnsNothing (0.03s)
    workerswap_test.go:653: ReplaceWorker = {Outcome:abandoned Gen:{N:0 Socket: Control: Build: StagedPath: Digest: Demand: Tier:{MemoryMax:0} Worker:false} Reason:re-checked at the spawn: swap: lease lost Drain:0s Mode: Executed:true CleanExit:false}, want failed with nothing serving
2026/10/22 08:31:50 activator: WORKER bot.x.internal failed to respawn (attempt (restart)): exec: text file busy — the sweep's crash restart takes it from here
2026/10/22 08:31:50 activator: WORKER bot.x.internal failed to respawn (restoring v1): exec: text file busy — the sweep's crash restart takes it from here
2026/10/22 08:31:50 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:51 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/22 08:31:51 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:51 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/22 08:31:51 activator: bot.x.internal hibernated
2026/10/22 08:31:51 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:51 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:51 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/22 08:31:51 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/22 08:31:51 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:51 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/22 08:31:51 activator: bot.x.internal hibernated
2026/10/22 08:31:52 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:52 activator: LEASE LOST for bot.x.internal (renewal failed) — standing down
2026/10/22 08:31:52 activator: bot.x.internal hibernated
2026/10/22 08:31:52 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:52 activator: bot.x.internal hibernated
2026/10/22 08:31:52 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/22 08:31:52 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/22 08:31:52 activator: bot.x.internal hibernated
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:53 activator: squatted.example.com: child exited during the health wait (external process serving its socket?) — not normalizing the socket mode
2026/10/22 08:31:53 activator: SIDECAR for late-sidecar.example failed to start: activator: maintenance fence active — retaining returned process
2026/10/22 08:31:53 activator: late-sidecar.example hibernated
2026/10/22 08:31:53 activator: own.example.com: failed wake's generation 0 is proven stopped; lease given back
2026/10/22 08:31:53 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:53 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:53 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/22 08:31:53 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/22 08:31:53 activator: swap.example.com hibernated
2026/10/22 08:31:53 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:53 activator: x.example hibernated
2026/10/22 08:31:53 activator: fence.example hibernated
2026/10/22 08:31:53 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it stays down until a release attempt or a wake starts it
2026/10/22 08:31:53 activator: worker bot9.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/22 08:31:53 activator: bot9.example.com: worker generation 2003's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:53 activator: generation 2003 of bot9.example.com did not prove its exit; keeping it in the operation set
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: swap.example.com hibernated
2026/10/22 08:31:53 activator: LEASE LOST for bot9.example.com (renewal failed) — standing down
2026/10/22 08:31:53 activator: bot9.example.com hibernated
2026/10/22 08:31:53 activator: swap.example.com hibernated
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 activator: worker bot1.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot1.x.internal exited; it serves v1 but live/bot1.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/22 08:31:53 activator: worker bot2.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot2.x.internal exited; live/bot2.x.internal names v2, but this sweep's converge budget is spent, so it recycles its captured build v1 and converges on a later sweep
2026/10/22 08:31:53 activator: worker bot2.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 activator: worker bot1.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:53 provision: a.shop.example.com: swapped for import sd_sq5h2qnaeezevsozmcprcqyoee (the original route is journaled)
2026/10/22 08:31:53 activator: seeding seedpin.example.com: /tmp/TestADeclinedFlatSeedRunsThePinnedFile2094590256/001/versions/v1/app seed --db /tmp/TestADeclinedFlatSeedRunsThePinnedFile2094590256/002/seedpin.example.com.db
2026/10/22 08:31:53 activator: seeded seedpin.example.com
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 activator: worker bot2.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:53 activator: worker bot2.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:53 provision: a.shop.example.com: import sd_sq5h2qnaeezevsozmcprcqyoee rolled back to its original route
2026/10/22 08:31:53 provision: a.shop.example.com: import sd_sq5h2qnaeezevsozmcprcqyoee is cleaned up and its journal row retired
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:53 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:53 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912
2026/10/22 08:31:53 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawngated_build_is_the_ex3434526572/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:53 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawna_held_build_starts_n4044055006/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:53 activator: swap.example.com hibernated
2026/10/22 08:31:53 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawnno_gate_is_todays_sp1289300416/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:53 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/22 08:31:53 activator: own.example.com: failed wake's generation 0 is proven stopped; lease given back
2026/10/22 08:31:53 activator: shop.example.com: generation 4's child is not proven empty: kill gen: timeout
2026/10/22 08:31:53 activator: x.example hibernated
2026/10/22 08:31:53 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:53 activator: stop fence.example: earlier process may still run
2026/10/22 08:31:53 activator: fence.example hibernated
2026/10/22 08:31:53 activator: fence.example hibernated
2026/10/22 08:31:53 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/22 08:31:53 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 activator: swap.example.com hibernated
2026/10/22 08:31:53 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/22 08:31:53 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:53 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:53 provision: a.shop.example.com: swapped for import sd_7napxduijdznkofzktf7oxjdau (the original route is journaled)
2026/10/22 08:31:53 activator: a.shop.example.com hibernated
2026/10/22 08:31:54 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:54 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it recycles its captured build v1
2026/10/22 08:31:54 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:54 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:54 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: the running child keeps running
2026/10/22 08:31:54 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:54 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:54 provision: a.shop.example.com: import sd_7napxduijdznkofzktf7oxjdau rolled back to its original route
2026/10/22 08:31:54 provision: a.shop.example.com: import sd_7napxduijdznkofzktf7oxjdau is cleaned up and its journal row retired
2026/10/22 08:31:54 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:54 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:54 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:54 activator: worker bot.x.internal serving v1 but live symlink is now v3 — respawning onto the new build
2026/10/22 08:31:54 activator: worker bot.x.internal respawned (converging onto a new build), now running v3
2026/10/22 08:31:54 activator: shop.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: swap.example.com: failed wake's generation 2039 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:54 activator: swap.example.com: failed wake's generation 2039 is proven stopped; lease given back
2026/10/22 08:31:54 activator: waking.example hibernated
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: swap.example.com: generation 2042's child is not proven empty: the child still holds a process
2026/10/22 08:31:54 activator: swap.example.com: failed wake's generation 2042 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:54 activator: swap.example.com: failed wake's generation 2042 is proven stopped; lease given back
2026/10/22 08:31:54 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:54 activator: x.example hibernated
2026/10/22 08:31:54 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:54 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:54 provision: a.shop.example.com: swapped for import sd_imywmiphqs52l7np4t3e5vmr5i (the original route is journaled)
before
2026/10/22 08:31:54 activator: one.example.com: sidecar starting on the instance's own build v1 (/opt/carlos/versions/v1/app): this box will not start v2, the build its live link names
2026/10/22 08:31:54 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/22 08:31:54 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/22 08:31:54 activator: a.shop.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@a.shop.example.com: unit still active); keeping the lease until it is
2026/10/22 08:31:54 provision: a.shop.example.com: import sd_imywmiphqs52l7np4t3e5vmr5i rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/22 08:31:54 activator: one.example.com hibernated
2026/10/22 08:31:54 activator: one.example.com hibernated
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance-gen@h.example.com:2062.service memory.max=536870912
2026/10/22 08:31:54 provision: a.shop.example.com: import sd_imywmiphqs52l7np4t3e5vmr5i rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/22 08:31:54 activator: stop fence.example: stop refused
2026/10/22 08:31:54 activator: fence.example hibernated
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/22 08:31:54 provision: a.shop.example.com: import sd_imywmiphqs52l7np4t3e5vmr5i rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: LEASE HELD for swap.example.com by "box-b" — a previous incarnation hasn't released; refusing to wake (retry after 1m0s)
2026/10/22 08:31:54 activator: swap.example.com: hold's pinned wake refused or abandoned: instance lease held by "box-b" (retry in 1m0s)
2026/10/22 08:31:54 activator: one.example.com hibernated
2026/10/22 08:31:54 activator: one.example.com hibernated
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:54 provision: a.shop.example.com: import sd_imywmiphqs52l7np4t3e5vmr5i rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: h.example.com: processes remain after failed; not releasing
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:54 activator: one.example.com hibernated
2026/10/22 08:31:54 activator: fence.example hibernated
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:54 activator: h.example.com: unit restarted outside the activator (invocation inv2, ours inv1); stopping it
2026/10/22 08:31:54 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:54 activator: x.example hibernated
2026/10/22 08:31:54 activator: one.example.com serving v1 but v2 is adopted — cycling onto the new build
2026/10/22 08:31:54 activator: one.example.com hibernated
2026/10/22 08:31:54 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:54 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: the backend ran a different generation from the hold's choice: generation 2080 of v9 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_1261890475/004/versions/v2/app"; the hold chose v2 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_1261890475/004/versions/v2/app"
2026/10/22 08:31:54 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:54 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:54 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:54 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, nor the instance's own build v1: a failure record holds this build: v1
2026/10/22 08:31:54 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/22 08:31:54 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:54 activator: worker bot.x.internal NOT respawned (converging onto a new build): instance config unavailable on this box: cannot open sealed config — standing down and releasing its lease
2026/10/22 08:31:54 activator: bot.x.internal hibernated
2026/10/22 08:31:54 activator: h.example.com: unit restarted outside the activator (invocation someone-else, ours inv7); stopping it
2026/10/22 08:31:54 activator: old.shop.example.com: route refreshed: db "/data/instances/old/old.shop.example.com.db" -> "" socket "/run/carlos/instances/old/app.sock" -> ""
2026/10/22 08:31:54 activator: wake old.shop.example.com: old.shop.example.com: the route changed before the wake: the activator no longer manages it
2026/10/22 08:31:54 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:54 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:54 activator: worker bot.x.internal NOT respawned (exited): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/22 08:31:54 activator: bot.x.internal hibernated
2026/10/22 08:31:54 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: swap.example.com: hold's pinned wake refused or abandoned: restore swap.example.com: replica unreadable
2026/10/22 08:31:54 activator: h.example.com: kill carlos-instance-gen@h.example.com:2091.service after a failed start: stop timed out
2026/10/22 08:31:54 activator: one.example.com: wake failed and its generation is not proven stopped: start one.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2091.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/22 08:31:54 activator: sidecar staged for awake tenant.test — adopting
2026/10/22 08:31:54 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/22 08:31:54 activator: swap.example.com hibernated
2026/10/22 08:31:54 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:54 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/22 08:31:54 activator: stop fence.example: private backend detail
2026/10/22 08:31:54 activator: fence.example hibernated
2026/10/22 08:31:54 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:54 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/22 08:31:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:55 activator: sidecar staged for awake tenant.test — adopting
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/22 08:31:55 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/22 08:31:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/22 08:31:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:55 activator: worker bot.x.internal hit its max_run; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/22 08:31:55 activator: x.example hibernated
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (hit its max_run), now running v2
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: socket dir: permission denied
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/22 08:31:55 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, and the instance's own build v1 cannot be named: staged copy unreadable
2026/10/22 08:31:55 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:55 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: staging: staged digest mismatch: have be0041e1a437dbea5b67dfe58bad04a0fc30d5036d469429f3534f39ec53d8b5, want c3bae8fb3a8bf395783d7f7f6ffcd4aadc797c95ce24b7e8e2a98af6b63d8f6d
2026/10/22 08:31:55 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/22 08:31:55 activator: x.example hibernated
2026/10/22 08:31:55 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/22 08:31:55 activator: one.example.com hibernated
2026/10/22 08:31:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:55 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:55 activator: worker bot.x.internal NOT respawned (hit its max_run): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/22 08:31:55 activator: bot.x.internal hibernated
2026/10/22 08:31:55 activator: early.example.com: the aborted start's unit is not proven stopped (context canceled); keeping the lease until it is
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:55 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/22 08:31:55 activator: stop sidecar fence.example: private backend detail
2026/10/22 08:31:55 activator: fence.example hibernated
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:55 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory; the sweep retries
2026/10/22 08:31:55 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory; the sweep retries
2026/10/22 08:31:55 activator: worker first.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory
2026/10/22 08:31:55 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory; the sweep retries
2026/10/22 08:31:55 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory; the sweep retries
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory; the sweep retries
2026/10/22 08:31:55 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory
2026/10/22 08:31:55 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory; the sweep retries
2026/10/22 08:31:55 activator: WORKER bot.x.internal failed to respawn (exited): worker build not published before its exec: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished822540964/002/data/zdd: not a directory — the sweep's crash restart takes it from here
2026/10/22 08:31:55 activator: bot.example.com hibernated
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/22 08:31:55 activator: x.example hibernated
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: start g: activator: generation start failed and its termination is unproven
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2130.service exited before it was observed
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:55 activator: worker bot.x.internal NOT respawned (exited): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: x.example hibernated
2026/10/22 08:31:55 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: WORKER bot.x.internal failed to respawn (exited): fork/exec: resource temporarily unavailable — the sweep's crash restart takes it from here
2026/10/22 08:31:55 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:55 activator: final checkpoint fence.example: private backend detail
2026/10/22 08:31:55 activator: fence.example hibernated
2026/10/22 08:31:55 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/22 08:31:55 activator: replica shop.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/22 08:31:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:55 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:55 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:55 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:55 activator: proven.example.com idle 2m0s — hibernating
2026/10/22 08:31:55 activator: proven.example.com hibernated
2026/10/22 08:31:55 activator: bot.x.internal hibernated
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: swap.example.com hibernated
2026/10/22 08:31:55 activator: h.example.com: kill carlos-instance-gen@h.example.com:2156.service after a failed start: stop timed out
2026/10/22 08:31:55 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2156.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/22 08:31:56 activator: swap.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/22 08:31:56 activator: h.example.com: kill carlos-instance-gen@h.example.com:2160.service after a failed start: stop timed out
2026/10/22 08:31:56 activator: kass.example.com idle 11s — hibernating
2026/10/22 08:31:56 activator: kass.example.com hibernated
2026/10/22 08:31:56 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:56 activator: pending-f6.example.com idle 2m0s — hibernating
2026/10/22 08:31:56 activator: stop pending-f6.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/22 08:31:56 activator: stop sidecar nested-costop.example.com: activator: stop unproven; processes remain
2026/10/22 08:31:56 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/22 08:31:56 activator: stop pending-f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:56 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/22 08:31:56 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/22 08:31:56 activator: pending-f6.example.com hibernated
2026/10/22 08:31:56 activator: nested-stop.example.com idle 2m0s — hibernating
2026/10/22 08:31:56 activator: nested-stop.example.com hibernated
2026/10/22 08:31:56 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/22 08:31:56 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:56 activator: nested-costop.example.com hibernated
2026/10/22 08:31:56 activator: one.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:56 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:56 activator: fence.example hibernated
2026/10/22 08:31:56 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/22 08:31:56 activator: control listener for shop.example.com: listen unix /tmp/TestARefusedLaunchLeavesNoBuildStateRecord2940391471/002/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/22 08:31:56 activator: undelegated.example.com idle 2m0s — hibernating
2026/10/22 08:31:56 activator: undelegated.example.com: stopped with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:31:56 activator: undelegated.example.com hibernated
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/22 08:31:56 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:56 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:56 activator: swap.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com: a discarded SIDECAR is not proven gone (activator: stop unproven; processes remain); keeping it tracked
2026/10/22 08:31:56 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:56 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:56 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:56 activator: swap.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: a stop-evidence operation holds the host
2026/10/22 08:31:56 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:56 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: shutdown-pending.example.com idle 2m0s — hibernating
2026/10/22 08:31:56 activator: stop shutdown-pending.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/22 08:31:56 activator: f6.example.com idle 2m0s — hibernating
2026/10/22 08:31:56 activator: stop f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/22 08:31:56 activator: stop f6.example.com: its host cgroup is not proven empty; keeping the lease until it is
2026/10/22 08:31:56 activator: f6.example.com hibernated
2026/10/22 08:31:56 activator: instance c.example.com exited unexpectedly — recycling
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:56 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:56 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/22 08:31:56 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:56 activator: fence.example hibernated
2026/10/22 08:31:56 activator: c.example.com idle 1h0m0s — hibernating
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:56 activator: seeding shop.example.com: /tmp/TestARefusedLaunchLeavesNoBuildStateRecordseed_refused_beside_a3688975537/002/versions/v1/app seed --db /tmp/TestARefusedLaunchLeavesNoBuildStateRecordseed_refused_beside_a3688975537/003/shop.example.com.db
2026/10/22 08:31:56 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:56 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:56 provision: a.shop.example.com: swapped for import sd_l5m3iojwjops4rxkvhv37tqkr4 (the original route is journaled)
2026/10/22 08:31:56 activator: shutdown-pending.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:56 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 8s)
2026/10/22 08:31:56 activator: swap.example.com hibernated
2026/10/22 08:31:56 activator: h.example.com: kill carlos-instance-gen@h.example.com:2215.service after a failed start: stop timed out
2026/10/22 08:31:56 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: the unit failed to start; activator: generation start failed and its termination is unproven: stop timed out
2026/10/22 08:31:56 provision: a.shop.example.com: import sd_l5m3iojwjops4rxkvhv37tqkr4 rolled back to its original route
2026/10/22 08:31:56 activator: sidecar for swap.example.com was down — restarted (backoff now 16s)
2026/10/22 08:31:56 provision: a.shop.example.com: import sd_l5m3iojwjops4rxkvhv37tqkr4 is cleaned up and its journal row retired
2026/10/22 08:31:56 provision: a.shop.example.com: swapped for import sd_23ogtfusi7pnum4eap2bxjv5gu (the original route is journaled)
2026/10/22 08:31:56 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:56 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:56 activator: a.shop.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@a.shop.example.com: instance control socket: connection refused); keeping the lease until it is
2026/10/22 08:31:56 provision: a.shop.example.com: import sd_23ogtfusi7pnum4eap2bxjv5gu rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/22 08:31:56 activator: swap.example.com hibernated
2026/10/22 08:31:56 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/22 08:31:56 activator: a.shop.example.com: failed wake's generation 2216 is proven stopped; lease given back
2026/10/22 08:31:56 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:56 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:56 activator: swap.example.com: v1 not started: its build state could not be written: bucket: 503
2026/10/22 08:31:56 activator: c.example.com idle 1h0m0s — hibernating
2026/10/22 08:31:56 activator: c.example.com hibernated
2026/10/22 08:31:56 activator: SIDECAR restart for one.example.com failed: swap: lease lost (next attempt in 2s)
2026/10/22 08:31:56 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:56 activator: worker bot.x.internal exited; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/22 08:31:56 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:56 provision: a.shop.example.com: import sd_23ogtfusi7pnum4eap2bxjv5gu rolled back to its original route
2026/10/22 08:31:56 provision: a.shop.example.com: import sd_23ogtfusi7pnum4eap2bxjv5gu is cleaned up and its journal row retired
2026/10/22 08:31:57 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:57 activator: stop c.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: cgroup: the host's parent cgroup holds a process directly: holds 1 process(es)
2026/10/22 08:31:57 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:57 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:57 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:57 activator: LEASE LOST for c.example.com (lost) — standing down
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:57 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:57 activator: stop fence.example: activator: stop evidence unavailable
2026/10/22 08:31:57 activator: fence.example hibernated
2026/10/22 08:31:57 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: instance control: 502 mid-roll; the sweep tries again
2026/10/22 08:31:57 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:57 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:57 activator: c.example.com idle 1h0m0s — hibernating
2026/10/22 08:31:57 activator: seeding shop.example.com: /tmp/TestARefusedLaunchLeavesNoBuildStateRecorda_seed_that_ran_is_re435509183/001/versions/v1/app seed --db /tmp/TestARefusedLaunchLeavesNoBuildStateRecorda_seed_that_ran_is_re435509183/002/shop.example.com.db
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:57 activator: shop.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:31:57 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: processes remained; the sweep tries again
2026/10/22 08:31:57 activator: one.example.com hibernated
2026/10/22 08:31:57 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:57 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/22 08:31:57 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: shop.example.com: seed placement not recorded yet: record unreachable
2026/10/22 08:31:57 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: one.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:57 activator: SIDECAR restart for one.example.com failed: the build state could not be written, so this build was not started: bucket refused the build state (next attempt in 2s)
2026/10/22 08:31:57 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake617005226/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake617005226/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:57 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake617005226/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake617005226/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:57 activator: stop fence.example: stop refused
2026/10/22 08:31:57 activator: fence.example hibernated
2026/10/22 08:31:57 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:57 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:57 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:57 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:57 activator: final checkpoint fence.example: activator: stop evidence unavailable
2026/10/22 08:31:57 activator: fence.example hibernated
2026/10/22 08:31:57 activator: c.example.com serving v1 but v2 is adopted — cycling onto the new build
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:57 activator: x.example source opened on the recovery probe's session; serving
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: c.example.com idle 1h0m0s — hibernating
2026/10/22 08:31:57 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: instance control: 502 mid-roll; the sweep tries again
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/22 08:31:57 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 4s)
2026/10/22 08:31:57 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:57 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor2190742349/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor2190742349/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:57 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor2190742349/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor2190742349/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:57 activator: fence.example hibernated
2026/10/22 08:31:57 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:57 activator: SIDECAR for one.example.com failed to start: the build state could not be written, so this build was not started: bucket refused the build state — serving without it (the sweep retries)
2026/10/22 08:31:57 activator: swap.example.com hibernated
2026/10/22 08:31:57 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a cgroup that took cgroup.kill could not be made fresh before a spawn: carlos-exec@h.example.com: device or resource busy
2026/10/22 08:31:57 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:57 activator: fence.example hibernated
2026/10/22 08:31:57 activator: c.example.com hibernated
2026/10/22 08:31:58 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:58 activator: one.example.com idle 1h0m0s — hibernating
2026/10/22 08:31:58 activator: one.example.com hibernated
2026/10/22 08:31:58 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:58 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:58 activator: seeding h.example.com: /tmp/TestPlan14GenerationBackendAuxiliaryBoundaryclaim=falseseedpi2333166748/001/live/h.example.com seed --db /tmp/nb1806560079/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:58 activator: seeded h.example.com
2026/10/22 08:31:58 activator: seeding h.example.com: /tmp/TestPlan14GenerationBackendAuxiliaryBoundaryclaim=falseseedpi3858699653/001/live/h.example.com seed --db /tmp/nb671201033/data/instances/h.example.com/h.example.com.db
2026/10/22 08:31:58 activator: seeded h.example.com
2026/10/22 08:31:58 activator: one.example.com idle 1h0m0s — hibernating
2026/10/22 08:31:58 activator: one.example.com hibernated
2026/10/22 08:31:58 activator: h.example.com idle 2m0s — hibernating
2026/10/22 08:31:58 activator: stop h.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/22 08:31:58 activator: fence.example hibernated
2026/10/22 08:31:58 activator: h.example.com hibernated
2026/10/22 08:31:58 activator: stuck.example.com idle 2m0s — hibernating
2026/10/22 08:31:58 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/22 08:31:58 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished1518301518/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished1518301518/001/zdd: not a directory; the sweep retries
2026/10/22 08:31:58 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished1518301518/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished1518301518/001/zdd: not a directory; the sweep retries
2026/10/22 08:31:58 activator: fence.example hibernated
2026/10/22 08:31:58 activator: f6.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake822514821/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake822514821/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake822514821/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake822514821/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:58 activator: exited-flat.example.com: failed wake's generation 2305 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:58 activator: exited-flat.example.com: failed wake's generation 2305 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/22 08:31:58 activator: one.example.com hibernated
2026/10/22 08:31:58 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:58 activator: fenced.example.com: failed wake's generation 2311 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:58 activator: unproven.example.com: generation 2313's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:58 activator: unproven.example.com: failed wake's generation 2313 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/22 08:31:58 activator: unproven.example.com: generation 2313's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:58 activator: fenced.example.com: failed wake's generation 2311 is proven stopped; lease given back
2026/10/22 08:31:58 activator: c.example.com is no longer hibernate-managed — releasing
2026/10/22 08:31:58 activator: c.example.com hibernated
2026/10/22 08:31:58 activator: unproven.example.com: failed wake's generation 2313 is proven stopped; lease given back
2026/10/22 08:31:58 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:58 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:58 activator: bot.x.internal hibernated
2026/10/22 08:31:58 activator: unproven.example.com hibernated
2026/10/22 08:31:58 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:58 activator: swap.example.com hibernated
2026/10/22 08:31:58 activator: forked.example.com: failed wake's generation 2318 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: forked.example.com: failed wake's generation 2318 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:58 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/22 08:31:58 activator: barrier.example.com hibernated
2026/10/22 08:31:58 activator: forked.example.com: failed wake's generation 2318 is proven stopped; lease given back
2026/10/22 08:31:58 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:58 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:58 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/22 08:31:58 activator: one.example.com hibernated
2026/10/22 08:31:58 activator: exited-foreign.example.com: failed wake's generation 2326 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:58 activator: exited-foreign.example.com: failed wake's generation 2326 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/22 08:31:58 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:58 activator: instance unproven.example.com exited unexpectedly — recycling
2026/10/22 08:31:58 activator: unproven.example.com hibernated
2026/10/22 08:31:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1746421889/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1746421889/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:58 activator: x.example: hold's pinned wake refused or abandoned: x.example: refusing to wake: staged build not published before its exec
2026/10/22 08:31:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1746421889/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1746421889/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/22 08:31:58 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:58 activator: unproven.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@unproven: instance control socket: connection refused); keeping the lease until it is
2026/10/22 08:31:58 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/22 08:31:58 activator: instance slowproof.example.com exited unexpectedly — recycling
2026/10/22 08:31:58 activator: slowproof.example.com hibernated
2026/10/22 08:31:58 activator: barrier.example.com hibernated
2026/10/22 08:31:58 activator: swap.example.com hibernated
2026/10/22 08:31:58 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a generation's limits could not be applied: h.example.com: the placement shim is not available
2026/10/22 08:31:58 activator: barrier.example.com hibernated
2026/10/22 08:31:58 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/22 08:31:58 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:58 activator: f6-layout.example.com: failed wake's generation 2337 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:58 activator: f6-layout.example.com: failed wake's generation 2337 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: refused.example.com: giving the lease back failed: delete lease: context deadline exceeded; retrying it
2026/10/22 08:31:58 activator: refused.example.com: lease given back on retry
2026/10/22 08:31:58 activator: c.example.com hibernated
2026/10/22 08:31:58 activator: fenced.example.com: failed wake's generation 2341 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:58 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/22 08:31:58 activator: bot.x.internal hibernated
2026/10/22 08:31:58 activator: unproven.example.com: generation 2343's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:58 activator: unproven.example.com: failed wake's generation 2343 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:58 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: h: start unit: instance control: generation termination unproven: the unit exited before it was observed
2026/10/22 08:31:58 activator: fenced.example.com: failed wake's generation 2341 is proven stopped; lease given back
2026/10/22 08:31:58 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/22 08:31:58 activator: unproven.example.com: generation 2343's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:58 activator: unproven.example.com: failed wake's generation 2343 is proven stopped; lease given back
2026/10/22 08:31:58 activator: unproven.example.com hibernated
2026/10/22 08:31:58 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/22 08:31:58 activator: instance restart.example.com exited unexpectedly — recycling
2026/10/22 08:31:58 activator: restart.example.com hibernated
2026/10/22 08:31:59 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/22 08:31:59 activator: barrier.example.com hibernated
2026/10/22 08:31:59 activator: slowproof.example.com: the aborted start's unit is not proven stopped (context deadline exceeded); keeping the lease until it is
2026/10/22 08:31:59 activator: shutdown: out of time with 1 failed wake(s) not proven stopped, their leases still held: slowproof.example.com (generation 2333)
2026/10/22 08:31:59 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:59 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:59 activator: bot.x.internal hit its max_run of 1m0s — stopping
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:59 activator: SIDECAR for swap.example.com is not proven gone for its replacement; tracking it (the sweep re-proves it)
2026/10/22 08:31:59 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/22 08:31:59 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:59 activator: f6.example.com: failed wake's generation 0 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:59 activator: stuck.example.com hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal reported idle but is always-on (hibernate:false) — ignoring
2026/10/22 08:31:59 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/22 08:31:59 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:59 activator: WORKER bot.x.internal failed to respawn (converging onto a new build): exec: live symlink gone — the sweep's crash restart takes it from here
2026/10/22 08:31:59 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:59 activator: one.example.com: live/one.example.com names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/22 08:31:59 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running v2
2026/10/22 08:31:59 activator: queued.example.com hibernated
2026/10/22 08:31:59 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/22 08:31:59 activator: f6-layout.example.com: failed wake's generation 2361 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:59 activator: f6-layout.example.com: failed wake's generation 2361 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:59 activator: shutdown: out of time with 1 failed wake(s) not proven stopped, their leases still held: unproven.example.com (generation 2330)
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; live/bot.x.internal names v2, but the release scheduler owns build changes, so it recycles its captured build v1
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/22 08:31:59 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/22 08:31:59 activator: swap.example.com hibernated
2026/10/22 08:31:59 activator: one.example.com hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:59 activator: fence.example hibernated
2026/10/22 08:31:59 activator: bot.x.internal serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: untimed.bab.oncarlos.com idle 16m0s — hibernating
2026/10/22 08:31:59 activator: untimed.bab.oncarlos.com hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: unproven.example.com: failed wake's generation 2363 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:59 activator: unproven.example.com: retry stop of failed wake's generation 2363: cgroup.kill: the child is not empty
2026/10/22 08:31:59 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/22 08:31:59 activator: barrier.example.com hibernated
2026/10/22 08:31:59 activator: LEASE LOST for one.example.com (lease renewal failed) — standing down
2026/10/22 08:31:59 activator: one.example.com hibernated
2026/10/22 08:31:59 activator: unproven.example.com: generation 2363's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:59 activator: mid.example.com has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: unproven.example.com: failed wake's generation 2363 is proven stopped; lease given back
2026/10/22 08:31:59 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/22 08:31:59 activator: one.example.com hibernated
2026/10/22 08:31:59 activator: flip.example.com is no longer hibernate-managed — releasing
2026/10/22 08:31:59 activator: flip.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/22 08:31:59 activator: flip.example.com hibernated
2026/10/22 08:31:59 activator: unproven.example.com hibernated
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:31:59 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:59 activator: mid.example.com is no longer hibernate-managed — releasing (wake finished first)
2026/10/22 08:31:59 activator: mid.example.com hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:59 activator: f6-layout.example.com: failed wake's generation 2375 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:59 activator: f6-layout.example.com: failed wake's generation 2375 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:59 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:59 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:59 activator: worker bot-a.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:59 activator: worker bot-a.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 4s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: bot.x.internal idle 11s — hibernating
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 8s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: bot.x.internal idle 11s — hibernating
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: bot.x.internal idle 31s — hibernating
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/22 08:31:59 activator: tickheld.bab.oncarlos.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/22 08:31:59 activator: tickheld.bab.oncarlos.com hibernated
2026/10/22 08:31:59 activator: worker bot-b.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:31:59 activator: worker bot-b.x.internal respawned (converging onto a new build), now running v2
2026/10/22 08:31:59 activator: bot.x.internal idle 11s — hibernating
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/22 08:31:59 activator: barrier.example.com hibernated
2026/10/22 08:31:59 activator: always-on sidecar bot.x.internal is not running — starting it
2026/10/22 08:31:59 activator: bot.x.internal idle 11s — hibernating
2026/10/22 08:31:59 activator: bot.x.internal hibernated
2026/10/22 08:31:59 activator: unproven.example.com: generation 2380's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:59 activator: unproven.example.com: failed wake's generation 2380 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:59 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/22 08:31:59 activator: swap.example.com hibernated
2026/10/22 08:31:59 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/22 08:31:59 activator: unproven.example.com: generation 2380's child is not proven empty: the generation's child still holds a process
2026/10/22 08:31:59 activator: f6-layout.example.com: failed wake's generation 2382 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/22 08:31:59 activator: f6-layout.example.com: failed wake's generation 2382 is not proven stopped; keeping the lease until it is
2026/10/22 08:31:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:31:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:31:59 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:31:59 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:31:59 activator: worker bot.x.internal respawned (hit its max_run), now running 
2026/10/22 08:31:59 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:31:59 activator: unproven.example.com: failed wake's generation 2380 is proven stopped; lease given back
2026/10/22 08:32:00 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/22 08:32:00 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/22 08:32:00 activator: worker bot.x.internal respawned (exited), now running 
2026/10/22 08:32:00 activator: unproven.example.com hibernated
2026/10/22 08:32:00 activator: LEASE LOST for bot.x.internal (lease renewal failed) — standing down
2026/10/22 08:32:00 activator: bot.x.internal hibernated
2026/10/22 08:32:00 activator: one.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: one.example.com hibernated
2026/10/22 08:32:00 activator: always-on sidecar bot00.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot01.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot02.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot03.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot04.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot05.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot06.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot07.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot08.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot09.x.internal is not running — starting it
2026/10/22 08:32:00 activator: always-on sidecar bot10.x.internal is not running — starting it
2026/10/22 08:32:00 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/22 08:32:00 activator: one.example.com hibernated
2026/10/22 08:32:00 activator: two.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/22 08:32:00 activator: two.example.com hibernated
2026/10/22 08:32:00 activator: own.example.com idle 1m0s — hibernating
2026/10/22 08:32:00 activator: own.example.com hibernated
2026/10/22 08:32:00 activator: bots.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: bots.example.com hibernated
2026/10/22 08:32:00 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/22 08:32:00 activator: one.example.com hibernated
2026/10/22 08:32:00 activator: instance bots.example.com exited unexpectedly — recycling
2026/10/22 08:32:00 activator: bots.example.com hibernated
2026/10/22 08:32:00 activator: one.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: one.example.com hibernated
2026/10/22 08:32:00 activator: sidecar staged for awake swap.example.com — adopting
2026/10/22 08:32:00 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/22 08:32:00 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/22 08:32:00 activator: bot.x.internal hibernated
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: swap.example.com hibernated
2026/10/22 08:32:00 activator: deaf.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: next-due probe deaf.example.com: dial unix: no such file — recording none (it sleeps until a request)
2026/10/22 08:32:00 activator: deaf.example.com hibernated
2026/10/22 08:32:00 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:32:00 activator: faroff.bab.oncarlos.com idle 16m0s — hibernating
2026/10/22 08:32:00 activator: faroff.bab.oncarlos.com hibernated
2026/10/22 08:32:00 activator: bots.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: bots.example.com hibernated
2026/10/22 08:32:00 activator: SIDECAR for botless.example.com failed to start: exec: live symlink gone — serving without it (the sweep retries)
2026/10/22 08:32:00 activator: reborn.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: reborn.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/22 08:32:00 activator: reborn.example.com hibernated
2026/10/22 08:32:00 activator: due.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: due.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/22 08:32:00 activator: due.example.com hibernated
2026/10/22 08:32:00 activator: late.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: late.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/22 08:32:00 activator: late.example.com hibernated
2026/10/22 08:32:00 activator: SIDECAR restart for botless.example.com failed: exec: live symlink gone (next attempt in 2s)
2026/10/22 08:32:00 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:32:00 activator: reborn.example.com has a scheduled delivery due — waking it
2026/10/22 08:32:00 activator: late.example.com has a scheduled delivery due — waking it
2026/10/22 08:32:00 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:32:00 activator: due.example.com has a scheduled delivery due — waking it
2026/10/22 08:32:00 activator: sidecar for botless.example.com was down — restarted (backoff now 4s)
2026/10/22 08:32:00 activator: SIDECAR for partial-sidecar.example failed to start: sidecar started then failed — retaining returned process
2026/10/22 08:32:00 activator: due.example.com idle 11s — hibernating
2026/10/22 08:32:00 activator: due.example.com hibernated
2026/10/22 08:32:00 activator: partial-sidecar.example hibernated
2026/10/22 08:32:00 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/22 08:32:00 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:32:00 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/22 08:32:00 activator: bot.x.internal hibernated
2026/10/22 08:32:00 activator: bot.x.internal hibernated
2026/10/22 08:32:00 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/22 08:32:00 activator: one.example.com idle 1m0s — hibernating
2026/10/22 08:32:00 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/22 08:32:00 activator: one.example.com hibernated
2026/10/22 08:32:00 activator: sidecar for bots.example.com was down — restarted (backoff now 8s)
2026/10/22 08:32:00 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:32:00 activator: f6.example.com: failed wake's generation 4 ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/22 08:32:00 activator: swap.example.com hibernated
2026/10/22 08:32:00 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/22 08:32:00 activator: bot.x.internal hibernated
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:32:00 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/22 08:32:00 activator: one.example.com hibernated
2026/10/22 08:32:00 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/22 08:32:00 activator: SIDECAR restart for partial-sidecar.example failed: sidecar started then failed — retaining returned process
2026/10/22 08:32:00 activator: two.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/22 08:32:00 activator: partial-sidecar.example hibernated
2026/10/22 08:32:00 activator: swap.example.com hibernated
2026/10/22 08:32:00 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/22 08:32:00 activator: x.example recovery probe healthy; held until the source opens
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:00 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/22 08:32:00 activator: barrier.example.com hibernated
2026/10/22 08:32:01 activator: bot.x.internal hibernated
2026/10/22 08:32:01 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/22 08:32:01 activator: bot.x.internal hibernated
2026/10/22 08:32:01 activator: bot.x.internal hibernated
FAIL
FAIL	github.com/carlosframework/platform/internal/activator	59.684s
ok  	github.com/carlosframework/platform/internal/agentsock	0.034s
ok  	github.com/carlosframework/platform/internal/alerting	0.010s
ok  	github.com/carlosframework/platform/internal/analytics	0.029s
ok  	github.com/carlosframework/platform/internal/appconfig	2.878s
ok  	github.com/carlosframework/platform/internal/apperr	0.133s
ok  	github.com/carlosframework/platform/internal/applog	0.352s
ok  	github.com/carlosframework/platform/internal/awsinv	0.009s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.008s
ok  	github.com/carlosframework/platform/internal/boxid	0.856s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.014s
ok  	github.com/carlosframework/platform/internal/caps	0.019s
ok  	github.com/carlosframework/platform/internal/certwarm	0.003s
ok  	github.com/carlosframework/platform/internal/cgroup	0.757s
ok  	github.com/carlosframework/platform/internal/changelog	0.488s
ok  	github.com/carlosframework/platform/internal/claimactivation	1.206s
ok  	github.com/carlosframework/platform/internal/clamd	0.057s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	5.654s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.011s
ok  	github.com/carlosframework/platform/internal/dbverify	0.006s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.005s
ok  	github.com/carlosframework/platform/internal/docsite	0.005s
ok  	github.com/carlosframework/platform/internal/domains	2.867s
ok  	github.com/carlosframework/platform/internal/doorkey	0.013s
ok  	github.com/carlosframework/platform/internal/economics	0.007s
ok  	github.com/carlosframework/platform/internal/edge	45.219s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.009s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.192s
ok  	github.com/carlosframework/platform/internal/gate	1.933s
ok  	github.com/carlosframework/platform/internal/genselect	0.024s
ok  	github.com/carlosframework/platform/internal/geo	0.029s
ok  	github.com/carlosframework/platform/internal/geoip	0.033s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.282s
ok  	github.com/carlosframework/platform/internal/host	138.962s
ok  	github.com/carlosframework/platform/internal/hostactors	6.918s
ok  	github.com/carlosframework/platform/internal/hostactors/awsmembership	0.011s
ok  	github.com/carlosframework/platform/internal/hostbox	3.167s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.836s
ok  	github.com/carlosframework/platform/internal/hostenv	0.007s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.007s
ok  	github.com/carlosframework/platform/internal/hostexec	5.534s
ok  	github.com/carlosframework/platform/internal/hostfence	0.068s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.278s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.028s
ok  	github.com/carlosframework/platform/internal/hostguard	1.030s
ok  	github.com/carlosframework/platform/internal/hostops	2.260s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.002s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.005s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.575s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.230s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.003s
ok  	github.com/carlosframework/platform/internal/infraview	0.004s
ok  	github.com/carlosframework/platform/internal/instancectl	5.250s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.776s
ok  	github.com/carlosframework/platform/internal/instancefs	0.008s
ok  	github.com/carlosframework/platform/internal/instancekey	0.003s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.007s
ok  	github.com/carlosframework/platform/internal/layout	1.943s
ok  	github.com/carlosframework/platform/internal/lbhealth	0.008s
ok  	github.com/carlosframework/platform/internal/lease	10.252s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.508s
ok  	github.com/carlosframework/platform/internal/ledger	0.014s
ok  	github.com/carlosframework/platform/internal/mail	0.220s
ok  	github.com/carlosframework/platform/internal/maintenance	0.083s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.590s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.089s
ok  	github.com/carlosframework/platform/internal/metrics	0.005s
ok  	github.com/carlosframework/platform/internal/moves	21.401s
ok  	github.com/carlosframework/platform/internal/moves/archive	1.079s
ok  	github.com/carlosframework/platform/internal/objstore	0.253s
ok  	github.com/carlosframework/platform/internal/ops	0.214s
ok  	github.com/carlosframework/platform/internal/park	0.281s
ok  	github.com/carlosframework/platform/internal/plans	0.004s
ok  	github.com/carlosframework/platform/internal/privatefile	0.004s
ok  	github.com/carlosframework/platform/internal/provision	13.312s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.009s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.675s
ok  	github.com/carlosframework/platform/internal/relay	7.233s
ok  	github.com/carlosframework/platform/internal/release	24.209s
ok  	github.com/carlosframework/platform/internal/rollout	3.802s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.005s
ok  	github.com/carlosframework/platform/internal/rootaws	0.008s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.058s
ok  	github.com/carlosframework/platform/internal/rootstate	0.027s
ok  	github.com/carlosframework/platform/internal/runs	0.563s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.007s
ok  	github.com/carlosframework/platform/internal/schedule	2.692s
ok  	github.com/carlosframework/platform/internal/seeds	0.011s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	10.615s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	4.029s
ok  	github.com/carlosframework/platform/internal/status	0.080s
ok  	github.com/carlosframework/platform/internal/steering	0.005s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.170s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.005s
ok  	github.com/carlosframework/platform/internal/unixdial	0.002s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.007s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.650s
FAIL
=== internal_console@0 (77 s)
ok  	github.com/carlosframework/platform/internal/console	74.955s
=== internal_console@1 (37 s)
ok  	github.com/carlosframework/platform/internal/console	34.690s
=== internal_console@2 (42 s)
ok  	github.com/carlosframework/platform/internal/console	39.465s
=== internal_console@3 (37 s)
ok  	github.com/carlosframework/platform/internal/console	35.049s
=== internal_console@4 (37 s)
ok  	github.com/carlosframework/platform/internal/console	34.729s
=== internal_console@5 (35 s)
ok  	github.com/carlosframework/platform/internal/console	32.560s
=== internal_console@6 (36 s)
ok  	github.com/carlosframework/platform/internal/console	34.445s
=== internal_console@7 (43 s)
ok  	github.com/carlosframework/platform/internal/console	40.652s
=== root@0 (32 s)
ok  	github.com/carlosframework/platform	30.000s
=== root@1 (44 s)
ok  	github.com/carlosframework/platform	41.895s
=== root@2 (41 s)
ok  	github.com/carlosframework/platform	38.945s

FAILED: rest
make: *** [Makefile:19: test] Error 1

amadan: step gate failed: exit status 2

amadan: job failed at step gate — its output is above, not at the tail of this log