amadan

carlos / platform Public

← Commits · feat/claim-root-trusted-state-e

CI failed 6m 37s

Branch
feat/claim-root-trusted-state-e
Commit
62e5b924 Paul Campbell
Runner
ci-aws~paul@keymail.dev/ip-172-31-25-182
Reported by
ci-aws~paul@keymail.dev/ip-172-31-25-182
Queued
Started
Waited for a runner
4s
Last change
Attempt
1 of 2

Steps run in order, one after another.

  1. gate failed 6m 36s 
    cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	351.767s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.327s
ok  	github.com/carlosframework/platform/cmd/moveprobe	92.891s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.384s
2026/10/13 18:43:56 activator: badseed.example.com: seed for badseed.example.com: activator: seed unproven; processes remain: processes remain after a successful seed: still running; leaving the lease to its TTL
2026/10/13 18:43:56 activator: unproven.example.com: unproven.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone; leaving the lease to its TTL
2026/10/13 18:43:56 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/13 18:43:56 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof2715425685/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof2715425685/004/next-wake.example.com.db
2026/10/13 18:43:56 activator: next-wake.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof2715425685/004/next-wake.example.com.db.seed-unproven refuses it as warm until they are
2026/10/13 18:43:56 activator: next-wake.example.com: seeding next-wake.example.com: exit status 1: activator: seed unproven; processes remain (the seed's processes remain after stopping it; its database was left in place: cgroup: strict containment unsupported); leaving the lease to its TTL
2026/10/13 18:43:56 activator: next-wake.example.com: next-wake.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup: strict containment unsupported; leaving the lease to its TTL
2026/10/13 18:43:56 activator: next-wake.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/13 18:43:56 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/13 18:43:56 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof2715425685/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof2715425685/004/next-wake.example.com.db
2026/10/13 18:43:56 activator: seeded next-wake.example.com
2026/10/13 18:43:56 activator: one.example.com idle 11s — hibernating
2026/10/13 18:43:56 activator: one.example.com hibernated
2026/10/13 18:43:56 activator: one.example.com idle 11s — hibernating
2026/10/13 18:43:56 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: two.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/13 18:43:57 activator: two.example.com hibernated
2026/10/13 18:43:57 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: LEASE LOST for one.example.com (lease renewal failed) — standing down
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: instance one.example.com exited unexpectedly — recycling
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: one.example.com idle 1m0s — hibernating
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: LEASE HELD for held.example.com by "old-incarnation" — a previous incarnation hasn't released; refusing to wake (retry after 7s)
2026/10/13 18:43:57 activator: faroff.bab.oncarlos.com idle 16m0s — hibernating
2026/10/13 18:43:57 activator: faroff.bab.oncarlos.com hibernated
2026/10/13 18:43:57 activator: untimed.bab.oncarlos.com idle 16m0s — hibernating
2026/10/13 18:43:57 activator: untimed.bab.oncarlos.com hibernated
2026/10/13 18:43:57 activator: tickheld.bab.oncarlos.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/13 18:43:57 activator: tickheld.bab.oncarlos.com hibernated
2026/10/13 18:43:57 activator: flip.example.com is no longer hibernate-managed — releasing
2026/10/13 18:43:57 activator: flip.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/13 18:43:57 activator: flip.example.com hibernated
2026/10/13 18:43:57 activator: mid.example.com has a scheduled delivery due — waking it
2026/10/13 18:43:57 activator: mid.example.com is no longer hibernate-managed — releasing (wake finished first)
2026/10/13 18:43:57 activator: mid.example.com hibernated
2026/10/13 18:43:57 activator: two.example.com hibernated
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: b.example.com hibernated
2026/10/13 18:43:57 activator: own.example.com idle 1m0s — hibernating
2026/10/13 18:43:57 activator: own.example.com hibernated
2026/10/13 18:43:57 activator: due.example.com idle 11s — hibernating
2026/10/13 18:43:57 activator: due.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/13 18:43:57 activator: due.example.com hibernated
2026/10/13 18:43:57 activator: due.example.com has a scheduled delivery due — waking it
2026/10/13 18:43:57 activator: due.example.com idle 11s — hibernating
2026/10/13 18:43:57 activator: due.example.com hibernated
2026/10/13 18:43:57 activator: late.example.com idle 11s — hibernating
2026/10/13 18:43:57 activator: late.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/13 18:43:57 activator: late.example.com hibernated
2026/10/13 18:43:57 activator: late.example.com has a scheduled delivery due — waking it
2026/10/13 18:43:57 activator: deaf.example.com idle 11s — hibernating
2026/10/13 18:43:57 activator: next-due probe deaf.example.com: dial unix: no such file — recording none (it sleeps until a request)
2026/10/13 18:43:57 activator: deaf.example.com hibernated
2026/10/13 18:43:57 activator: reborn.example.com idle 11s — hibernating
2026/10/13 18:43:57 activator: reborn.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/13 18:43:57 activator: reborn.example.com hibernated
2026/10/13 18:43:57 activator: reborn.example.com has a scheduled delivery due — waking it
2026/10/13 18:43:57 activator: SIDECAR for botless.example.com failed to start: exec: live symlink gone — serving without it (the sweep retries)
2026/10/13 18:43:57 activator: SIDECAR restart for botless.example.com failed: exec: live symlink gone (next attempt in 2s)
2026/10/13 18:43:58 activator: sidecar for botless.example.com was down — restarted (backoff now 4s)
2026/10/13 18:43:58 activator: bots.example.com idle 11s — hibernating
2026/10/13 18:43:58 activator: bots.example.com hibernated
2026/10/13 18:43:58 activator: instance bots.example.com exited unexpectedly — recycling
2026/10/13 18:43:58 activator: bots.example.com hibernated
2026/10/13 18:43:58 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/13 18:43:58 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/13 18:43:58 activator: sidecar for bots.example.com was down — restarted (backoff now 8s)
2026/10/13 18:43:58 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/13 18:43:58 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/13 18:43:58 activator: bots.example.com idle 11s — hibernating
2026/10/13 18:43:58 activator: bots.example.com hibernated
2026/10/13 18:43:58 activator: bot.x.internal idle 11s — hibernating
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: bot.x.internal idle 31s — hibernating
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: bot.x.internal hit its max_run of 1m0s — stopping
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: LEASE LOST for bot.x.internal (lease renewal failed) — standing down
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/13 18:43:58 activator: one.example.com hibernated
2026/10/13 18:43:58 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:58 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:58 activator: bot.x.internal idle 11s — hibernating
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: bot.x.internal idle 11s — hibernating
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:59 activator: bot.x.internal idle 11s — hibernating
2026/10/13 18:43:59 activator: bot.x.internal hibernated
2026/10/13 18:43:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 4s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 8s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/13 18:43:59 activator: bot.x.internal hibernated
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:43:59 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (hit its max_run), now running 
2026/10/13 18:43:59 activator: always-on sidecar bot.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot00.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot01.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot02.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot03.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot04.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot05.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot06.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot07.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot08.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot09.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot10.x.internal is not running — starting it
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:44:00 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:44:00 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:00 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/13 18:44:00 activator: bot.x.internal hibernated
2026/10/13 18:44:01 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:01 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/13 18:44:01 activator: bot.x.internal hibernated
2026/10/13 18:44:01 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:01 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/13 18:44:02 activator: bot.x.internal hibernated
2026/10/13 18:44:02 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:02 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/13 18:44:02 activator: bot.x.internal hibernated
2026/10/13 18:44:02 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:03 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/13 18:44:03 activator: bot.x.internal hibernated
2026/10/13 18:44:03 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:03 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:44:03 activator: worker bot-b.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:03 activator: worker bot-b.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:44:03 activator: worker bot-a.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:03 activator: worker bot-a.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:44:03 activator: bot.x.internal serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/13 18:44:03 activator: bot.x.internal hibernated
2026/10/13 18:44:03 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:03 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:44:03 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:03 activator: WORKER bot.x.internal failed to respawn (converging onto a new build): exec: live symlink gone — the sweep's crash restart takes it from here
2026/10/13 18:44:03 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:03 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:44:03 activator: worker bot.x.internal respawned (exited), now running v2
2026/10/13 18:44:03 activator: worker bot.x.internal reported idle but is always-on (hibernate:false) — ignoring
2026/10/13 18:44:07 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendStartMergesConfigEnvUnderOwnEnv7136075/002/merge-precedence.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/13 18:44:17 activator: purged 1 cached response(s) for deploy.example.com on spawn
2026/10/13 18:44:19 activator: restored restored.example.com from s3://test-bucket/restored.example.com
2026/10/13 18:44:19 activator: damaged.example.com restored CORRUPT from s3://test-bucket/damaged.example.com: integrity_check: file is not a database (26) — quarantined at /tmp/TestExecBackendRestoreCorruptReplicaFailsAndQuarantines3122535118/002/damaged.example.com.db.corrupt, instance NOT started
2026/10/13 18:44:19 activator: restored damaged.example.com from s3://test-bucket/damaged.example.com
2026/10/13 18:44:19 activator: no replica for brand-new.example.com in s3://test-bucket/brand-new.example.com — booting fresh
2026/10/13 18:44:19 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath3587889124/002/seeded.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/13 18:44:19 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath3587889124/001/live/seeded.example.com seed --fixtures --db /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath3587889124/002/seeded.example.com.db
2026/10/13 18:44:19 activator: seeded seeded.example.com
2026/10/13 18:44:19 activator: seeding custom-db.example.com: /tmp/TestExecBackendSeedHonoursRouteDBPath3511175839/001/live/custom-db.example.com seed --db /tmp/TestExecBackendSeedHonoursRouteDBPath3511175839/004/elsewhere.db
2026/10/13 18:44:19 activator: seeded custom-db.example.com
2026/10/13 18:44:19 activator: seeding badseed.example.com: /tmp/TestExecBackendSeedNonZeroExitFails718721801/001/live/badseed.example.com seed --db /tmp/TestExecBackendSeedNonZeroExitFails718721801/002/badseed.example.com.db
schema is corrupt
2026/10/13 18:44:19 activator: badseed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/13 18:44:19 activator: seeding hangs.example.com: /tmp/TestExecBackendSeedIsTimeBounded1082431630/001/live/hangs.example.com seed --db /tmp/TestExecBackendSeedIsTimeBounded1082431630/002/hangs.example.com.db
2026/10/13 18:44:20 activator: hangs.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/13 18:44:20 activator: seeding crashed.example.com: /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB2198959129/001/live/crashed.example.com seed --db /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB2198959129/002/crashed.example.com.db
died mid-schema
2026/10/13 18:44:20 activator: crashed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/13 18:44:20 activator: seeding existing-db.example.com: /tmp/TestExecBackendSeedFailureKeepsPreexistingDB3224982553/001/live/existing-db.example.com seed --db /tmp/TestExecBackendSeedFailureKeepsPreexistingDB3224982553/002/existing-db.example.com.db
nope
2026/10/13 18:44:20 activator: existing-db.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/13 18:44:20 activator: seeding leaky-seed.example.com: /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild260826732/001/live/leaky-seed.example.com seed --db /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild260826732/002/leaky-seed.example.com.db
2026/10/13 18:44:20 activator: seeded leaky-seed.example.com (it exited 0 but left a child holding its output pipe open for more than 200ms)
2026/10/13 18:44:21 activator: squatted.example.com: child exited during the health wait (external process serving its socket?) — not normalizing the socket mode
2026/10/13 18:44:21 activator: h.example.com idle 2m0s — hibernating
2026/10/13 18:44:21 activator: stop h.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/13 18:44:21 activator: h.example.com hibernated
2026/10/13 18:44:21 activator: stuck.example.com idle 2m0s — hibernating
2026/10/13 18:44:21 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/13 18:44:21 activator: shutdown-pending.example.com idle 2m0s — hibernating
2026/10/13 18:44:21 activator: stop shutdown-pending.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/13 18:44:22 activator: shutdown-pending.example.com hibernated
2026/10/13 18:44:22 activator: sidecar for a.example.com was down — restarted (backoff now 2s)
2026/10/13 18:44:22 activator: h.example.com: route refreshed: db "/tmp/h.example.com.db" -> "/data/instances/h.example.com/h.example.com.db" socket "/tmp/nope.sock" -> "/run/carlos/instances/h.example.com/app.sock"
2026/10/13 18:44:22 activator: stop stale.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/13 18:44:22 activator: stale.example.com hibernated
2026/10/13 18:44:22 activator: leftover.example.com: the host cgroup holds processes not proven gone: a leftover still populates the host cgroup; leaving the lease to its TTL
2026/10/13 18:44:22 activator: f6.example.com idle 2m0s — hibernating
2026/10/13 18:44:22 activator: stop f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:44:22 activator: stop f6.example.com: its host cgroup is not proven empty; keeping the lease until it is
2026/10/13 18:44:22 activator: f6.example.com hibernated
2026/10/13 18:44:22 activator: proven.example.com idle 2m0s — hibernating
2026/10/13 18:44:22 activator: proven.example.com hibernated
2026/10/13 18:44:22 activator: undelegated.example.com idle 2m0s — hibernating
2026/10/13 18:44:22 activator: undelegated.example.com: stopped with no delegated cgroup; its leader's exit is the only proof there is
2026/10/13 18:44:22 activator: undelegated.example.com hibernated
2026/10/13 18:44:22 activator: nested-stop.example.com idle 2m0s — hibernating
2026/10/13 18:44:22 activator: nested-stop.example.com hibernated
2026/10/13 18:44:22 activator: old-kernel.example.com: this kernel has no cgroup.kill, so the host cgroup cannot be killed; its leader's exit is the only proof there is
2026/10/13 18:44:22 activator: pending-f6.example.com idle 2m0s — hibernating
2026/10/13 18:44:22 activator: stop pending-f6.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/13 18:44:22 activator: stop pending-f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:44:22 activator: pending-f6.example.com hibernated
2026/10/13 18:44:28 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/13 18:44:28 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/13 18:44:28 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/13 18:44:28 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/13 18:44:28 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/13 18:44:28 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/13 18:44:28 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/13 18:44:28 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/13 18:44:30 activator: retry.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:30 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/13 18:44:30 activator: retry.example.com: strict group not closed: cgroup: strict containment refused
2026/10/13 18:44:31 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/13 18:44:31 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:31 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:31 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:31 activator: stuck.example.com: strict group still not cleaned up after 2 retries; spawns of this host stay refused until the agent restarts
2026/10/13 18:44:31 activator: owned.example.com: strict group not closed: cgroup: strict containment refused
2026/10/13 18:44:32 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/13 18:44:32 activator: owned.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:33 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/13 18:44:33 activator: race.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:33 activator: transition.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:33 activator: transition2.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:33 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/13 18:44:33 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/13 18:44:33 activator: kass.example.com idle 11s — hibernating
2026/10/13 18:44:33 activator: kass.example.com hibernated
2026/10/13 18:44:34 activator: kass.example.com hibernated
--- FAIL: TestUnitTenantWakeServesHeldRequest (0.22s)
    backend_unit_test.go:371: unit listen: listen tcp 127.0.0.1:39363: bind: address already in use
    backend_unit_test.go:421: held request = 200 "<!doctype html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n<title>Sign in · CARLOS console</title>\n<style>\n  :root {\n    --bg: #ffffff; --ink: #201d29; --muted: #6f6480;\n    --accent: #5b3e96; --accent-strong: #4c3380;\n    --accent-soft: rgba(91, 62, 150, 0.09);\n    --rule: #eae7f0; --bad: #b3403a; --warn: #a6790f; --good: #2f7a4f;\n    --mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;\n    --sans: ui-sans-serif, system-ui, -apple-system, sans-serif;\n  }\n  @media (prefers-color-scheme: dark) {\n    :root {\n      --bg: #16141c; --ink: #f0eef6; --muted: #a99ebd;\n      --accent: #b394f2; --accent-strong: #c9b2f7;\n      --accent-soft: rgba(179, 148, 242, 0.12);\n      --rule: #2c2836; --bad: #e07b76; --warn: #d9a94a; --good: #5fae7f;\n    }\n  }\n  [hidden] { display: none !important; }\n  * { margin: 0; padding: 0; box-sizing: border-box; }\n  body {\n    background: var(--bg); color: var(--ink);\n    font: 400 1rem/1.6 var(--sans);\n    -webkit-font-smoothing: antialiased; padding: 0 1.25rem;\n  }\n  a { color: var(--accent); text-decoration-thickness: 1px; text-underline-offset: 2px; }\n  code {\n    font-family: var(--mono);\n    font-size: 0.88em; background: var(--accent-soft);\n    padding: 0.1em 0.35em; border-radius: 0.25rem;\n  }\n  pre {\n    background: var(--accent-soft); border-radius: 0.5rem;\n    padding: 0.9rem 1.1rem; margin: 0 0 1rem; max-width: 40rem;\n    overflow-x: auto; font-family: var(--mono); font-size: 0.85rem; line-height: 1.6;\n  }\n  pre code { background: none; padding: 0; }\n   \n  .changelog { white-space: pre-wrap; }\n  select, input[type=text], input[type=search] {\n    font: inherit; font-family: var(--sans); font-size: 0.9rem; color: var(--ink); background: var(--bg);\n    border: 1px solid var(--rule); border-radius: 0.4rem; padding: 0.3rem 0.5rem;\n  }\n  td form { display: flex; gap: 0.4rem; align-items: center; flex-wrap: wrap; margin: 0; }\n   \n  form.inline { display: inline; margin: 0; }\n  form.inline button { vertical-align: baseline; }\n   \n  form.listctl { display: flex; gap: 0.6rem; flex-wrap: wrap; margin: 0 0 0.6rem; }\n   \n  .icon { width: 1em; height: 1em; flex: none; vertical-align: -0.125em; }\n  header {\n    max-width: 64rem; margin: 0 auto; padding: 1.4rem 0;\n    display: flex; align-items: baseline; gap: 1rem;\n    border-bottom: 1px solid var(--rule);\n  }\n  header .brand a { font-weight: 700; color: var(--ink); text-decoration: none; }\n  header nav {\n    margin-left: auto; display: flex; align-items: baseline; gap: 1rem;\n    font-size: 0.9rem;\n  }\n  header .addr { color: var(--muted); }\n   \n  header .switcher {\n    position: relative;\n  }\n  header .switcher > summary {\n    display: inline-flex; align-items: baseline; gap: 0.4rem;\n    list-style: none; cursor: pointer;\n    padding: 0.25rem 0.5rem; margin: -0.25rem -0.5rem; border-radius: 0.5rem;\n  }\n  header .switcher > summary::-webkit-details-marker { display: none; }\n  header .switcher > summary:hover { background: var(--accent-soft); }\n   \n  header .switcher .current-name {\n    font-weight: 700; min-width: 0; max-width: 14rem;\n    overflow: hidden; text-overflow: ellipsis; white-space: nowrap;\n  }\n   \n  header .switcher > summary > .icon { color: var(--muted); align-self: center; width: 0.9em; }\n  header .switcher[open] > summary > .icon { transform: rotate(180deg); }\n  header .switcher-menu {\n    position: absolute; z-index: 20; top: calc(100% + 0.6rem); left: -0.5rem;\n    min-width: 13rem; max-width: 20rem;\n    background: var(--bg); border: 1px solid var(--rule); border-radius: 0.6rem;\n    padding: 0.35rem; font-size: 0.9rem;\n    box-shadow: 0 0.4rem 1.5rem rgba(0, 0, 0, 0.14);\n  }\n  header .switcher-menu ul { list-style: none; margin: 0; padding: 0; }\n  header .switcher-menu a, header .switcher-menu .current {\n    display: flex; align-items: baseline; gap: 0.5rem;\n    padding: 0.35rem 0.55rem; border-radius: 0.4rem;\n    color: var(--ink); text-decoration: none; overflow-wrap: anywhere;\n  }\n  header .switcher-menu a:hover, header .switcher-menu a:focus { background: var(--accent-soft); }\n  header .switcher-menu .current { font-weight: 600; background: var(--accent-soft); }\n   \n  header .switcher-menu .mark {\n    color: var(--accent); width: 0.9rem; flex: none;\n    display: flex; align-self: center;\n  }\n  header .switcher-menu a .mark { visibility: hidden; }\n  header .switcher-menu .new a .mark { visibility: visible; }\n  header .switcher-menu hr { border: 0; border-top: 1px solid var(--rule); margin: 0.35rem 0.2rem; }\n  header .switcher-menu .new a { color: var(--accent); }\n  main { max-width: 64rem; margin: 0 auto; padding: 2.5rem 0 3rem; min-height: 55vh; }\n  h1 { font-size: 1.9rem; line-height: 1.15; margin: 0 0 1rem; }\n  h2 { font-size: 1.25rem; margin: 2rem 0 0.75rem; }\n  h3 { font-size: 1rem; margin: 1.5rem 0 0.5rem; }\n   \n  .settings-row { display: flex; gap: 0.6rem; flex-wrap: wrap; align-items: center; margin: 0 0 0.75rem; }\n  .settings-layout {\n    display: grid; grid-template-columns: minmax(10rem, 13rem) minmax(0, 1fr);\n    gap: 2.5rem; align-items: start;\n  }\n  .settings-nav { display: flex; flex-direction: column; gap: 0.2rem; }\n  .settings-nav a {\n    display: block; padding: 0.45rem 0.65rem; border-radius: 0.35rem;\n    color: var(--muted); text-decoration: none;\n  }\n  .settings-nav a:hover { color: var(--ink); background: var(--accent-soft); }\n  .settings-nav a[aria-current=\"page\"] {\n    color: var(--ink); background: var(--accent-soft); font-weight: 600;\n  }\n  .settings-content { min-width: 0; }\n  .settings-content > h2:first-child { margin-top: 0; }\n  .danger-head { color: var(--bad); }\n  p { max-width: 40rem; margin: 0 0 1rem; }\n  .lede { font-size: 1.15rem; color: var(--muted); }\n  .small { font-size: 0.9rem; color: var(--muted); }\n   \n  .visually-hidden {\n    position: absolute; width: 1px; height: 1px; overflow: hidden;\n    clip-path: inset(50%); white-space: nowrap;\n  }\n  .error { color: var(--bad); }\n  .banner {\n    background: var(--accent-soft); border-radius: 0.5rem;\n    padding: 0.6rem 0.9rem; margin: 0 0 1.5rem; max-width: 40rem;\n  }\n  .tablewrap { overflow-x: auto; }\n  .instances .instance-address { white-space: nowrap; }\n  .instances td:nth-child(2) { white-space: nowrap; }\n  .instance-open { display: inline-flex; padding: .25rem; }\n  .instance-group th { text-align: left; padding-top: 1rem; background: var(--accent-soft); }\n  .instance-group .small { font-weight: normal; }\n  .small a[aria-current=\"true\"] { color: var(--ink); font-weight: 700; text-decoration-thickness: 3px; }\n  .instance-facts { display: grid; grid-template-columns: minmax(8rem, auto) 1fr; gap: .5rem 1.5rem; margin: 1rem 0; }\n  .instance-facts dt { color: var(--muted); }\n  .instance-facts dd { margin: 0; overflow-wrap: anywhere; }\n  .instance-facts .cellspark { max-width: 24rem; }\n   \n  #logs-filter { display: flex; flex-wrap: wrap; align-items: end; gap: 0.65rem; margin-bottom: 1rem; }\n  #logs-filter label { display: flex; flex-direction: column; gap: 0.25rem; max-width: 100%; }\n  #logs-filter select, #logs-filter input { max-width: 100%; margin: 0; }\n  #logs-filter select[name=\"host\"] { max-width: 28rem; }\n  .term {\n    background: var(--accent-soft); border-radius: 0.5rem;\n    padding: 0.8rem 0 0.8rem 1rem; margin: 0 0 1rem;\n    min-height: 16rem; max-height: 70vh; overflow: auto;\n    font-family: var(--mono); font-size: 0.8rem; line-height: 1.55;\n  }\n   \n  .tl { white-space: pre; width: max-content; min-width: 100%; padding-right: 1rem; }\n   \n  .tl-t, .tl-h, .tl-f { color: var(--muted); }\n  .tl-s { color: var(--accent); }\n   \n  .tl-warn .tl-m { color: var(--warn, #a6790f); }\n  .tl-error .tl-m { color: var(--bad); font-weight: 600; }\n  table { border-collapse: collapse; width: 100%; margin: 0 0 1rem; font-size: 0.95rem; }\n  .tiles { display: flex; flex-wrap: wrap; gap: 1rem; margin: 0 0 1.5rem; }\n  .tile { flex: 1 1 10rem; background: var(--accent-soft); border-radius: 0.6rem; padding: 1rem 1.1rem; }\n   \n  a.tile { color: var(--ink); text-decoration: none; }\n  a.tile:hover { outline: 1px solid var(--accent); }\n   \n  .stat-tiles { display: grid; grid-template-columns: repeat(auto-fit, minmax(17rem, 1fr)); }\n  .tile h2 {\n    margin: 0 0 0.35rem; font-family: var(--mono); font-size: 0.78rem; text-transform: uppercase;\n    letter-spacing: 0.05em; color: var(--muted); font-weight: 600;\n  }\n  .tile-value { font-size: 1.6rem; font-weight: 600; margin: 0; }\n  .tile-detail { color: var(--muted); font-size: 0.8rem; margin: 0.25rem 0 0; }\n   \n  .chip {\n    display: inline-block; font-family: var(--mono); font-size: 0.78rem; font-weight: 500;\n    padding: 0.15rem 0.55rem; border-radius: 999px; margin: 0.15rem 0.3rem 0.15rem 0;\n  }\n  .chip-stale { border: 1px solid var(--warn, #a6790f); color: var(--warn, #a6790f); }\n   \n  .chip-canary { border: 1px solid var(--accent); color: var(--accent); }\n  .chip-prod { background: var(--bad); color: var(--bg); }\n   \n  .chip-expired { border: 1px solid var(--warn, #a6790f); color: var(--warn, #a6790f); }\n   \n  .chip-env { border: 1px solid var(--rule); color: var(--muted); }\n   \n  .chip-health-ok { border: 1px solid var(--good); color: var(--good); }\n  .chip-health-attention { background: var(--warn, #a6790f); color: var(--bg); }\n  .chip-health-notdeployed { border: 1px solid var(--rule); color: var(--muted); }\n  .chip-health-unknown { border: 1px solid var(--warn, #a6790f); color: var(--warn, #a6790f); }\n   \n  .chip-freetier-inside { border: 1px solid var(--warn, #a6790f); color: var(--warn, #a6790f); }\n  .chip-freetier-partial { border: 1px solid var(--accent); color: var(--accent); }\n  .chip-freetier-paying { border: 1px solid var(--good); color: var(--good); }\n  .chip-freetier-pooled { border: 1px solid var(--rule); color: var(--muted); }\n   \n  .chip-tier-unassigned { border: 1px solid var(--rule); color: var(--muted); }\n  .chip-tier-try { border: 1px solid var(--rule); color: var(--muted); }\n  .chip-tier-pool { border: 1px solid var(--rule); color: var(--muted); }\n  .chip-tier-hosted { border: 1px solid var(--good); color: var(--good); }\n  .chip-tier-fleet { border: 1px solid var(--accent); color: var(--accent); }\n  .chip-tier-dedicated { border: 1px solid var(--warn, #a6790f); color: var(--warn, #a6790f); }\n  .chip .icon { vertical-align: -0.1em; margin-right: 0.2em; }\n  .chip-up { background: var(--accent-soft); color: #2f7a4f; }\n  .chip-down { background: var(--accent-soft); color: var(--bad); }\n  .chips { display: flex; flex-wrap: wrap; margin: 0 0 1rem; }\n   \n  .bar { display: inline-block; width: 6rem; height: 0.5rem; border-radius: 999px; background: var(--accent-soft); vertical-align: middle; overflow: hidden; }\n  .bar-fill { display: block; height: 100%; background: var(--accent); }\n   \n  .knob-row { display: flex; gap: 0.6rem; align-items: center; margin: 0 0 0.5rem; flex-wrap: wrap; }\n  .knob-row label { min-width: 12rem; color: var(--muted); font-size: 0.9rem; }\n  .knob-row input[type=number], .knob-row input[type=text] { width: 8rem; }\n  .knob-row input[type=range] { width: 12rem; }\n   \n  .flow { margin: 0 0 1.75rem; }\n  .flow-line {\n    --flow-y: 0.52rem;\n    display: flex; flex-wrap: wrap; gap: 1.1rem 2.6rem;\n    list-style: none; margin: 0; padding: 0;\n  }\n  .flow-stage { position: relative; flex: 1 1 10rem; min-width: 9rem; }\n  .flow-stage + .flow-stage::before, .flow-stage + .flow-stage::after {\n    content: \"\"; position: absolute; pointer-events: none;\n  }\n  .flow-stage + .flow-stage::before {\n    left: -2.05rem; top: var(--flow-y); width: 1.3rem; border-top: 1px dashed var(--rule);\n  }\n  .flow-stage + .flow-stage::after {\n    left: -0.82rem; top: var(--flow-y); transform: translateY(-50%);\n    border: 0.28rem solid transparent; border-right: 0; border-left-color: var(--rule);\n  }\n  .flow-stage.is-carried::before { border-top-style: solid; border-top-color: var(--accent); }\n  .flow-stage.is-carried::after { border-left-color: var(--accent); }\n  .flow-label {\n    margin: 0; font-family: var(--mono); font-size: 0.74rem; text-transform: uppercase;\n    letter-spacing: 0.05em; color: var(--muted); font-weight: 600;\n  }\n  .flow-version { margin: 0.3rem 0 0; font-size: 0.95rem; }\n   \n  .flow-stage:not(.is-held) .flow-version { color: var(--muted); }\n  .flow-note { margin: 0.25rem 0 0; font-size: 0.78rem; color: var(--muted); }\n  .flow-addrs {\n    list-style: none; margin: 0.6rem 0 0; padding: 0 0 0 0.7rem;\n    border-left: 1px solid var(--rule); display: grid; gap: 0.35rem;\n  }\n  .flow-addr { font-size: 0.82rem; line-height: 1.35; }\n  .flow-addr a { display: block; overflow-wrap: anywhere; }\n  .flow-health { display: inline-flex; align-items: center; gap: 0.2rem; color: var(--muted); }\n  .flow-health .icon { width: 0.9em; height: 0.9em; flex: none; }\n  .flow-tone-ok .flow-health { color: var(--good); }\n  .flow-tone-wait .flow-health { color: var(--warn, #a6790f); }\n  .flow-tone-bad .flow-health { color: var(--bad); font-weight: 600; }\n  .flow-band { margin: 1.4rem 0 0.7rem; font-size: 0.9rem; color: var(--muted); }\n  .flow-caption { margin: 1.1rem 0 0; }\n   \n  @media (max-width: 46rem) {\n    .settings-layout { display: block; }\n    .settings-nav {\n      flex-direction: row; overflow-x: auto; margin: 0 0 1.5rem;\n      padding-bottom: 0.25rem;\n    }\n    .settings-nav a { white-space: nowrap; }\n    .flow-line { display: block; }\n    .flow-stage + .flow-stage { margin-top: 1.6rem; }\n    .flow-stage + .flow-stage::before {\n      left: 0.55rem; top: -1.2rem; width: 0; height: 0.85rem;\n      border-top: 0; border-left: 1px dashed var(--rule);\n    }\n    .flow-stage + .flow-stage::after {\n      left: 0.55rem; top: -0.34rem; transform: translateX(-50%);\n      border: 0.28rem solid transparent; border-bottom: 0; border-top-color: var(--rule);\n      border-left-color: transparent;\n    }\n    .flow-stage.is-carried::before { border-left-color: var(--accent); }\n    .flow-stage.is-carried::after { border-top-color: var(--accent); border-left-color: transparent; }\n  }\n   \n  .ladder { list-style: none; margin: 0 0 1rem; padding: 0; max-width: 40rem; }\n  .rung {\n    border: 1px solid var(--rule); border-radius: 0.6rem; padding: 0.8rem 1rem;\n  }\n  .rung + .rung { margin-top: 0.4rem; }\n  .rung-name {\n    font-family: var(--mono);\n    font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.05em;\n    font-weight: 600; color: var(--muted); margin: 0 0 0.3rem;\n  }\n  .rung p { margin: 0 0 0.2rem; }\n  .rung .gate { margin: 0.5rem 0 0; }\n  .rung .routes { list-style: none; margin: 0.3rem 0 0.6rem; padding: 0; font-size: 0.95rem; }\n  .rung .routes li { padding: 0.1rem 0; }\n   \n  .step {\n    display: flex; align-items: baseline; gap: 0.5rem; flex-wrap: wrap;\n    margin: 0.7rem -1rem -0.8rem; padding: 0.5rem 1rem;\n    border-top: 1px solid var(--rule);\n  }\n  .step form { display: flex; align-items: baseline; gap: 0.5rem; margin: 0; }\n  .step > .icon { color: var(--muted); align-self: center; }\n  th {\n    font-family: var(--mono);\n    font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.05em;\n    color: var(--muted); text-align: left; font-weight: 600;\n  }\n  th, td { padding: 0.5rem 1rem 0.5rem 0; border-bottom: 1px solid var(--rule); }\n  form.signin { display: flex; gap: 0.6rem; flex-wrap: wrap; margin: 0 0 1rem; }\n  form.signin input[type=text] { padding: 0.55rem 0.8rem; min-width: 16rem; }\n  button, .btn {\n    font-size: 0.95rem; font-weight: 600; background: var(--accent);\n    color: var(--bg); border: 0; border-radius: 0.4rem;\n    padding: 0.6rem 1.2rem; cursor: pointer;\n    text-decoration: none; display: inline-block;\n  }\n  button.linkish {\n    background: none; color: var(--accent); font-weight: 400; padding: 0;\n    border-radius: 0; text-decoration: underline; font-size: 0.9rem; cursor: pointer;\n  }\n  button:not(.linkish):hover, .btn:hover { background: var(--accent-strong); }\n  button.linkish.danger { color: var(--bad); }\n   \n  .btn-sm, button.btn-sm {\n    font-size: 0.85rem; padding: 0.35rem 0.8rem; border-radius: 0.4rem;\n  }\n  button.btn-danger, .btn-danger { background: var(--bad); color: var(--bg); }\n  button.btn-danger:hover, .btn-danger:hover { background: var(--bad); filter: brightness(0.9); }\n   \n  td .cell-value { display: inline-block; min-width: 3.5rem; font-weight: 600; }\n  .cellspark { width: 6rem; height: 1.1rem; vertical-align: -0.2rem; color: var(--muted); }\n  .cellspark polyline { stroke-width: 2.5; }\n   \n  .feed { list-style: none; margin: 0 0 0.5rem; padding: 0; max-width: 40rem; }\n  .feed li { padding: 0.25rem 0; border-bottom: 1px solid var(--rule); }\n  .feed li:last-child { border-bottom: 0; }\n   \n  .confirm-card {\n    border: 1px solid var(--rule); border-radius: 0.6rem;\n    padding: 1.25rem 1.4rem; max-width: 40rem; margin: 0 0 1rem;\n  }\n  .confirm-card .actions { display: flex; gap: 0.9rem; align-items: center; margin-top: 1rem; }\n  .confirm-card .confirm-note { margin: 0.8rem 0 0; }\n   \n   \n  .archwrap { overflow-x: auto; margin: 0.2rem 0 1.6rem; }\n   \n  h4.archband {\n    border-top: 1px solid var(--rule); color: var(--muted);\n    font: 600 0.8rem var(--mono); letter-spacing: 0.02em;\n    margin: 1.5rem 0 0; padding-top: 0.5rem;\n  }\n   \n  .archsvg { display: block; }\n  .archsvg rect { fill: var(--accent-soft); stroke: var(--rule); }\n  .archsvg .archnode-box rect { stroke: var(--accent); }\n  .archsvg .archtitle { fill: var(--ink); font: 600 13px var(--sans); }\n  .archsvg .archsub { fill: var(--muted); font: 400 11px var(--sans); }\n   \n  .archsvg .archedge { fill: none; stroke: var(--muted); stroke-width: 1.5; }\n   \n  .archsvg .archalarm { fill: var(--good); }\n  .archsvg .archalarm-firing { fill: var(--bad); }\n   \n  .archbox {\n    border: 1px solid var(--rule); border-radius: 0.6rem;\n    padding: 0.9rem 1.1rem; margin: 0 0 0.6rem; max-width: 64rem;\n  }\n  .archbox h3 { margin: 0 0 0.3rem; }\n  .archbox p { margin: 0 0 0.2rem; }\n   \n  .archbox .fact + .fact::before { content: \" · \"; color: var(--muted); }\n  footer {\n    max-width: 64rem; margin: 0 auto; padding: 1.25rem 0 2rem;\n    border-top: 1px solid var(--rule);\n    font-size: 0.85rem; color: var(--muted);\n  }\n   \n  .pagehead { margin: 0 0 1.75rem; }\n  .eyebrow {\n    font-family: var(--mono); font-size: 0.75rem; letter-spacing: 0.08em;\n    text-transform: uppercase; color: var(--muted); margin: 0 0 0.3rem;\n  }\n  .pagehead h1 { margin: 0 0 0.35rem; }\n  .pagehead .lede { margin: 0; font-size: 1.05rem; }\n   \n  .reveal > summary { font-size: 1.1rem; font-weight: 600; cursor: pointer; }\n   \n  .tabs {\n    display: flex; flex-wrap: wrap; gap: 1.25rem; margin: 0 0 2rem;\n    border-bottom: 1px solid var(--rule);\n    font-family: var(--mono); font-size: 0.85rem;\n  }\n  .tabs a {\n    color: var(--muted); text-decoration: none;\n    padding: 0 0 0.55rem; border-bottom: 2px solid transparent; margin-bottom: -1px;\n  }\n  .tabs a:hover { color: var(--ink); }\n  .tabs a[aria-current] { color: var(--ink); border-bottom-color: var(--accent); }\n  header nav a[aria-current] { color: var(--ink); font-weight: 600; text-decoration: none; }\n  header nav a.addr { color: var(--muted); text-decoration: none; }\n  header nav a.addr:hover { text-decoration: underline; }\n</style>\n</head>\n<body>\n<header>\n  \n  <div class=\"brand\"><a href=\"/\">CARLOS console</a></div>\n  \n  \n</header>\n<main>\n\n<div class=\"pagehead\">\n  <p class=\"eyebrow\">🤖 CARLOS console</p>\n  <h1>Ship it. Promote it. Sleep<span style=\"color: var(--accent)\">.</span></h1>\n  <p class=\"lede\">Your apps, their live routes, and their release channels —\n  signed releases in, immutable bytes out.</p>\n</div>\n\n<form class=\"signin\" method=\"get\" action=\"/auth/signin\">\n  <button type=\"submit\">Sign in with Keymail</button>\n</form>\n<p class=\"small\">One button, whether it's your first time or your five-hundredth.\nFirst visits approve the console once in your Keymail — that single approval\nis the whole sign-up. After that, the same button is a one-click sign-in.\nNo passwords, ever; revoke the console from Keymail's connected apps at any\ntime.</p>\n\n</main>\n<footer>🤖 Console text written by an LLM, per <a href=\"https://11factor.org/#x\">factor X</a>.</footer>\n<script>\n\n\n\n\n(function () {\n  var d = document.getElementById(\"switcher\");\n  if (!d) return;\n  document.addEventListener(\"keydown\", function (e) {\n    if (e.key === \"Escape\" && d.open) { d.open = false; d.firstElementChild.focus(); }\n  });\n  document.addEventListener(\"click\", function (e) {\n    if (d.open && !d.contains(e.target)) d.open = false;\n  });\n})();\n</script>\n</body>\n</html>\n", want 200 kass up
2026/10/13 18:44:34 activator: one.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/13 18:44:34 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: swap.example.com hibernated
2026/10/13 18:44:34 activator: swap.example.com hibernated
2026/10/13 18:44:34 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/13 18:44:34 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: one.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:34 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:44:34 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:34 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: worker bot.x.internal NOT respawned (exited): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/13 18:44:34 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:44:34 activator: bot.x.internal: v2 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: worker bot.x.internal NOT respawned (attempt (release)): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/13 18:44:34 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:44:34 activator: link.example.com hibernated
2026/10/13 18:44:51 activator: tenant.test: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:51 activator: tenant.test not started: the executable it hashed (/tmp/rl3681561758/app) is build "v1", but the start names build "v2" (the live link moved between the two reads); the next wake reads it again
2026/10/13 18:44:51 activator: sidecar staged for awake tenant.test — adopting
2026/10/13 18:44:51 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/13 18:44:51 activator: sidecar staged for awake tenant.test — adopting
2026/10/13 18:44:51 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/13 18:44:52 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:52 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:44:52 activator: worker bot.x.internal NOT respawned (exited): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/13 18:44:52 activator: bot.x.internal hibernated
2026/10/13 18:44:52 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:44:52 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:44:52 activator: worker bot.x.internal NOT respawned (hit its max_run): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/13 18:44:52 activator: bot.x.internal hibernated
2026/10/13 18:44:52 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:52 activator: worker bot.x.internal NOT respawned (converging onto a new build): instance config unavailable on this box: cannot open sealed config — standing down and releasing its lease
2026/10/13 18:44:52 activator: bot.x.internal hibernated
2026/10/13 18:44:52 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:52 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:44:52 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:44:52 activator: seeding unproven-seed.example.com: /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3246105719/002/live/unproven-seed.example.com seed --db /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3246105719/003/unproven-seed.example.com.db
2026/10/13 18:44:52 activator: unproven-seed.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3246105719/003/unproven-seed.example.com.db.seed-unproven refuses it as warm until they are
2026/10/13 18:44:52 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm457139795/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm457139795/003/obligation.example.com.db
2026/10/13 18:44:52 activator: obligation.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm457139795/003/obligation.example.com.db.seed-unproven refuses it as warm until they are
2026/10/13 18:44:52 activator: obligation.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/13 18:44:52 activator: no replica for obligation.example.com in s3://test-bucket/obligation.example.com — booting fresh
2026/10/13 18:44:52 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm457139795/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm457139795/003/obligation.example.com.db
2026/10/13 18:44:52 activator: seeded obligation.example.com
2026/10/13 18:44:52 activator: seeding helper-seed.example.com: /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt158049736/002/live/helper-seed.example.com seed --db /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt158049736/003/helper-seed.example.com.db
2026/10/13 18:44:52 activator: seeded helper-seed.example.com
2026/10/13 18:44:52 activator: seeding helper-unproven.example.com: /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven3749322535/002/live/helper-unproven.example.com seed --db /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven3749322535/003/helper-unproven.example.com.db
2026/10/13 18:44:52 activator: helper-unproven.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven3749322535/003/helper-unproven.example.com.db.seed-unproven refuses it as warm until they are
2026/10/13 18:44:52 activator: seeding write-ahead.example.com: /tmp/TestASeedRecordsItsObligationBeforeItRuns2705516293/002/live/write-ahead.example.com seed --db /tmp/TestASeedRecordsItsObligationBeforeItRuns2705516293/003/write-ahead.example.com.db
2026/10/13 18:44:52 activator: seeded write-ahead.example.com
2026/10/13 18:44:52 activator: write-ahead.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/13 18:44:52 activator: no replica for write-ahead.example.com in s3://test-bucket/write-ahead.example.com — booting fresh
2026/10/13 18:44:52 activator: seeding no-room.example.com: /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing2761295286/002/live/no-room.example.com seed --db /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing2761295286/003/no-room.example.com.db
2026/10/13 18:44:52 activator: seeding stuck-wal.example.com: /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation2303491881/002/live/stuck-wal.example.com seed --db /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation2303491881/003/stuck-wal.example.com.db
2026/10/13 18:44:52 activator: seeding preexisting.example.com: /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2173460410/002/live/preexisting.example.com seed --db /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2173460410/003/preexisting.example.com.db
2026/10/13 18:44:52 activator: preexisting.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2173460410/003/preexisting.example.com.db.seed-unproven refuses it as warm until they are
2026/10/13 18:44:52 activator: preexisting.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/13 18:44:52 activator: old-kernel-seed.example.com: this kernel has no cgroup.kill; the seed's leader's exit is the only proof there is
2026/10/13 18:44:52 activator: seeding failed-role-seed.example.com: /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB2434439967/002/live/failed-role-seed.example.com seed --db /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB2434439967/003/failed-role-seed.example.com.db
2026/10/13 18:44:52 activator: seeding discharge-fails.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai1329491394/002/live/discharge-fails.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai1329491394/003/discharge-fails.example.com.db
2026/10/13 18:44:52 activator: discharge-fails.example.com: the seed's obligation could not be discharged and its processes are not proven gone; /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai1329491394/003/discharge-fails.example.com.db.seed-unproven stays
2026/10/13 18:44:52 activator: seeding discharge-fails-proven.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur160388195/002/live/discharge-fails-proven.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur160388195/003/discharge-fails-proven.example.com.db
2026/10/13 18:44:52 activator: one.example.com serving v1 but v2 is adopted — cycling onto the new build
2026/10/13 18:44:52 activator: one.example.com hibernated
2026/10/13 18:44:52 activator: one.example.com hibernated
2026/10/13 18:44:52 activator: one.example.com hibernated
2026/10/13 18:44:53 activator: one.example.com hibernated
2026/10/13 18:44:53 activator: one.example.com hibernated
2026/10/13 18:44:53 activator: one.example.com: sidecar starting on the instance's own build v1 (/opt/carlos/versions/v1/app): this box will not start v2, the build its live link names
2026/10/13 18:44:53 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/13 18:44:53 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, nor the instance's own build v1: a failure record holds this build: v1
2026/10/13 18:44:53 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/13 18:44:53 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, and the instance's own build v1 cannot be named: staged copy unreadable
2026/10/13 18:44:53 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/13 18:44:53 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_B,_l3009872255/003/g/dca298090dc0.135.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:53 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall2096994230/003/g/dca298090dc0.136.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:53 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall2096994230/003/g/dca298090dc0.137.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:53 activator: swap.example.com hibernated
2026/10/13 18:44:53 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatholds_pinne578726437/007/g/dca298090dc0.139.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:53 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/13 18:44:53 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatoverlap_succ2906478269/007/g/dca298090dc0.141.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:53 activator: h.example.com: started carlos-instance-gen@h.example.com:143.service memory.max=536870912
2026/10/13 18:44:53 activator: swap.example.com hibernated
2026/10/13 18:44:53 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/13 18:44:54 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_super3193779310/007/g/dca298090dc0.150.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:54 activator: swap.example.com hibernated
2026/10/13 18:44:54 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartholds_pinned_build,_lea257993324/007/g/dca298090dc0.152.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:54 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/13 18:44:54 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_a_fai2667501907/007/g/dca298090dc0.154.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:54 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawngated_build_is_the_ex2533132336/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:54 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawna_held_build_starts_n3492199604/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:54 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawnno_gate_is_todays_sp1772711115/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:54 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/13 18:44:54 activator: worker bot.x.internal serving v1 but live symlink is now v3 — respawning onto the new build
2026/10/13 18:44:54 activator: worker bot.x.internal respawned (converging onto a new build), now running v3
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it recycles its captured build v1
2026/10/13 18:44:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it stays down until a release attempt or a wake starts it
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:44:55 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: the running child keeps running
2026/10/13 18:44:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:44:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:44:55 activator: one.example.com: live/one.example.com names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/13 18:44:55 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/13 18:44:55 activator: one.example.com hibernated
2026/10/13 18:44:55 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/13 18:44:55 activator: fence.example hibernated
2026/10/13 18:44:55 activator: fence.example hibernated
2026/10/13 18:44:55 activator: fence.example hibernated
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 156 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 156 is not proven stopped; keeping the lease until it is
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 157 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 157 is not proven stopped; keeping the lease until it is
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 158 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 158 is not proven stopped; keeping the lease until it is
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 159 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 159 is not proven stopped; keeping the lease until it is
2026/10/13 18:44:56 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink2648298750/003/g/c.x.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink2648298750/003/g/c.x.sock: no such file or directory)
2026/10/13 18:44:56 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink2648298750/003/g/c.xx.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink2648298750/003/g/c.xx.sock: no such file or directory)
2026/10/13 18:44:58 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:44:58 activator: one.example.com hibernated
2026/10/13 18:44:58 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished3967646874/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished3967646874/001/zdd: not a directory; the sweep retries
2026/10/13 18:44:58 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished3967646874/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished3967646874/001/zdd: not a directory; the sweep retries
2026/10/13 18:44:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake550362615/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake550362615/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake550362615/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake550362615/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor1607866586/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor1607866586/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor1607866586/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor1607866586/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:59 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake2804443547/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake2804443547/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:59 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake2804443547/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake2804443547/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:59 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake3565993057/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake3565993057/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:59 activator: x.example: hold's pinned wake refused or abandoned: x.example: refusing to wake: staged build not published before its exec
2026/10/13 18:44:59 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake3565993057/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake3565993057/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:59 activator: one.example.com hibernated
2026/10/13 18:44:59 activator: publish generations to /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile34716002/002/not-a-dir/zdd/generations.json: open directory /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile34716002/002/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:59 activator: ident.example.com hibernated
2026/10/13 18:44:59 activator: instance ident.example.com exited unexpectedly — recycling
2026/10/13 18:44:59 activator: ident.example.com hibernated
2026/10/13 18:44:59 activator: due.example.com hibernated
2026/10/13 18:44:59 activator: due.example.com hibernated
2026/10/13 18:44:59 activator: ret.example.com hibernated
2026/10/13 18:44:59 activator: ret.example.com hibernated
2026/10/13 18:44:59 activator: one.example.com hibernated
2026/10/13 18:44:59 activator: purged 1 cached response(s) for flat.example.com on spawn
2026/10/13 18:45:00 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile2338405322/001/versions/vA/app seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile2338405322/002/seedexe.example.com.db
2026/10/13 18:45:00 activator: seeded seedexe.example.com
2026/10/13 18:45:00 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile2338405322/001/live/seedexe.example.com seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile2338405322/002/seedexe.example.com.db
2026/10/13 18:45:00 activator: seeded seedexe.example.com
2026/10/13 18:45:00 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:45:00 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912
2026/10/13 18:45:00 activator: seeding h.example.com: /tmp/TestNestedSeedExeNamesTheRecordedFile1137666709/001/live/h.example.com seed --db /tmp/nb3638113001/data/instances/h.example.com/h.example.com.db
2026/10/13 18:45:00 activator: seeded h.example.com
2026/10/13 18:45:00 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:45:00 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912
2026/10/13 18:45:00 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3664468460/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/13 18:45:00 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3664468460/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/13 18:45:00 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3664468460/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/13 18:45:00 activator: seeding seedpin.example.com: /tmp/TestADeclinedFlatSeedRunsThePinnedFile718680415/001/versions/v1/app seed --db /tmp/TestADeclinedFlatSeedRunsThePinnedFile718680415/002/seedpin.example.com.db
2026/10/13 18:45:00 activator: seeded seedpin.example.com
2026/10/13 18:45:00 activator: seeding h.example.com: /tmp/TestADeclinedNestedSeedRunsThePinnedFile3608268985/001/live/h.example.com seed --db /tmp/nb1544772827/data/instances/h.example.com/h.example.com.db
2026/10/13 18:45:00 activator: seeded h.example.com
2026/10/13 18:45:00 activator: seeding tier.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed314984592/002/versions/vA/app seed --db /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed314984592/003/tier.example.com.db
2026/10/13 18:45:00 activator: tier.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed314984592/003/tier.example.com.db.seed-unproven refuses it as warm until they are
2026/10/13 18:45:00 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=1073741824
2026/10/13 18:45:01 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=1073741824
2026/10/13 18:45:01 activator: seeding h.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTiernested_seed4133211711/001/live/h.example.com seed --db /tmp/nb3275715575/data/instances/h.example.com/h.example.com.db
2026/10/13 18:45:01 activator: seeded h.example.com
2026/10/13 18:45:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsthe_configured_k666648039/001/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:01 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsan_own_key_pres1966314208/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/13 18:45:03 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestFirstStartWaitsForTheConfiguredKey3484660587/002/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:03 activator: control listener for shop.example.com: listen unix /tmp/TestFirstStartWaitsForTheConfiguredKey3484660587/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:05 activator: CARLOS_INSTANCE_KEY in /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptinto_the_own_file3533075239/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/13 18:45:05 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptdelivered_in_the_co3075965403/001/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:05 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration3511198075/001/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:05 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration3511198075/002/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:05 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyLeavesAFileWithTwoKeysAlone3639789251/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/13 18:45:19 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistse1033783924/002/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:21 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistsn3168195250/002/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:25 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance1085562006/001/live/seeded.example.com seed --db /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance1085562006/002/seeded.example.com.db
2026/10/13 18:45:25 activator: seeded seeded.example.com
2026/10/13 18:45:29 activator: restored sleepy.example.com from s3://test-bucket/sleepy.example.com
hello
world
before
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: stop fence.example: private backend detail
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: stop sidecar fence.example: private backend detail
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: final checkpoint fence.example: private backend detail
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: stop fence.example: activator: stop evidence unavailable
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: final checkpoint fence.example: activator: stop evidence unavailable
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: stop fence.example: stop refused
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: stop fence.example: earlier process may still run
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:32 activator: waking.example hibernated
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: SIDECAR for partial-sidecar.example failed to start: sidecar started then failed — retaining returned process
2026/10/13 18:45:32 activator: partial-sidecar.example hibernated
2026/10/13 18:45:32 activator: SIDECAR restart for partial-sidecar.example failed: sidecar started then failed — retaining returned process
2026/10/13 18:45:32 activator: partial-sidecar.example hibernated
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: stop fence.example: stop refused
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/13 18:45:32 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/13 18:45:32 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/13 18:45:32 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/13 18:45:32 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/13 18:45:33 activator: barrier.example.com hibernated
2026/10/13 18:45:33 activator: barrier.example.com: generation 12's child is not proven empty: g12 still holds a process
2026/10/13 18:45:33 activator: generation 12 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/13 18:45:33 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/13 18:45:33 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/13 18:45:33 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/13 18:45:33 activator: barrier.example.com hibernated
2026/10/13 18:45:33 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/13 18:45:33 activator: barrier.example.com hibernated
2026/10/13 18:45:33 activator: stop generation 11: cgroup.kill: the child is not empty
2026/10/13 18:45:33 activator: generation 11 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/13 18:45:33 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/13 18:45:33 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/13 18:45:33 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/13 18:45:33 activator: barrier.example.com hibernated
2026/10/13 18:45:33 activator: waking.example.com hibernated
2026/10/13 18:45:33 activator: placed.example.com: kill generation 1225 after a failed health wait: g still holds a process
2026/10/13 18:45:33 activator: placed.example.com: generation 1225's child is not proven empty: g still holds a process
2026/10/13 18:45:33 activator: placed.example.com: failed wake's generation 1225 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:33 activator: placed.example.com: wake failed and its generation is not proven stopped: start placed.example.com: instance not healthy within 300ms: Get "http://carlos/": dial unix /tmp/g1171128819/g/74a2d464a30d.1225.sock: connect: no such file or directory; activator: generation start failed and its termination is unproven: kill generation 1225 after a failed health wait: g still holds a process
2026/10/13 18:45:33 activator: placed.example.com: failed wake's generation 1225 is proven stopped; lease given back
2026/10/13 18:45:34 activator: unproven.example.com: generation 1228's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1228 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:34 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/13 18:45:34 activator: unproven.example.com: generation 1228's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1228 is proven stopped; lease given back
2026/10/13 18:45:34 activator: unproven.example.com hibernated
2026/10/13 18:45:34 activator: unproven.example.com: generation 1230's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1230 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:34 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: h: start unit: instance control: generation termination unproven: the unit exited before it was observed
2026/10/13 18:45:34 activator: unproven.example.com: generation 1230's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1230 is proven stopped; lease given back
2026/10/13 18:45:34 activator: unproven.example.com hibernated
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1232 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:34 activator: unproven.example.com: retry stop of failed wake's generation 1232: cgroup.kill: the child is not empty
2026/10/13 18:45:34 activator: unproven.example.com: generation 1232's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1232 is proven stopped; lease given back
2026/10/13 18:45:34 activator: unproven.example.com hibernated
2026/10/13 18:45:34 activator: unproven.example.com: generation 1234's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1234 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:34 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/13 18:45:34 activator: unproven.example.com: generation 1234's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1234 is proven stopped; lease given back
2026/10/13 18:45:34 activator: unproven.example.com hibernated
2026/10/13 18:45:34 activator: unproven.example.com: generation 1236's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1236 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:34 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/13 18:45:34 activator: unproven.example.com: generation 1236's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: generation 1236's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:35 activator: unproven.example.com: failed wake's generation 1236 is proven stopped; lease given back
2026/10/13 18:45:35 activator: unproven.example.com: generation 1237's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:35 activator: unproven.example.com: failed wake's generation 1237 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:35 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/13 18:45:35 activator: unproven.example.com: generation 1237's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:35 activator: unproven.example.com: generation 1237's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:35 activator: unproven.example.com: failed wake's generation 1237 is proven stopped; lease given back
2026/10/13 18:45:35 activator: unproven.example.com: generation 1238's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:35 activator: unproven.example.com: failed wake's generation 1238 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:35 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/13 18:45:35 activator: fenced.example.com: failed wake's generation 1239 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:35 activator: fenced.example.com: failed wake's generation 1239 is proven stopped; lease given back
2026/10/13 18:45:35 activator: fenced.example.com: failed wake's generation 1240 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:35 activator: fenced.example.com: failed wake's generation 1240 is proven stopped; lease given back
2026/10/13 18:45:35 activator: forked.example.com: failed wake's generation 1241 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:35 activator: forked.example.com: failed wake's generation 1241 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:35 activator: forked.example.com: failed wake's generation 1241 is proven stopped; lease given back
2026/10/13 18:45:35 activator: f6.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:36 activator: f6.example.com: failed wake's generation 0 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:36 activator: f6.example.com: failed wake's generation 4 ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/13 18:45:36 activator: unhealthy-f6.example.com: failed wake's generation 1242 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:36 activator: unhealthy-f6.example.com: failed wake's generation 1242 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:36 activator: unhealthy-f6.example.com: failed wake's generation 1242 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:36 activator: unhealthy-f6.example.com: failed wake's generation 1242 is proven stopped; lease given back
2026/10/13 18:45:36 activator: exited-flat.example.com: failed wake's generation 1243 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:36 activator: exited-flat.example.com: failed wake's generation 1243 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:36 activator: exited-foreign.example.com: failed wake's generation 1244 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:36 activator: exited-foreign.example.com: failed wake's generation 1244 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:37 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/13 18:45:37 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/13 18:45:37 activator: instance one.example.com exited unexpectedly — recycling
2026/10/13 18:45:37 activator: one.example.com hibernated
2026/10/13 18:45:38 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/13 18:45:38 activator: one.example.com hibernated
2026/10/13 18:45:38 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/13 18:45:38 activator: one.example.com hibernated
2026/10/13 18:45:38 activator: one.example.com idle 1h0m0s — hibernating
2026/10/13 18:45:38 activator: one.example.com hibernated
2026/10/13 18:45:38 activator: one.example.com idle 1h0m0s — hibernating
2026/10/13 18:45:38 activator: one.example.com hibernated
2026/10/13 18:45:38 activator: moved.example.com: promoting 2 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example hibernated
2026/10/13 18:45:39 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/13 18:45:39 activator: x.example hibernated
2026/10/13 18:45:39 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/13 18:45:39 activator: x.example hibernated
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/13 18:45:39 activator: x.example hibernated
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example hibernated
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example hibernated
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:41 activator: replica h.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/13 18:45:41 activator: replica h.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/13 18:45:41 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/13 18:45:43 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_keyless_replica_is_refu1889998174/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:43 activator: replica shop-au.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/13 18:45:45 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_is_1843344608/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:47 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_pre-existing_env_key_th1087830159/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:49 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_dif2410165071/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:51 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_with_no_337094479/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:51 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY cannot be checked against its home's key (no entry for it in /etc/carlos/replica-home-keys; O8)
2026/10/13 18:45:53 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_home_updated,_this_re1393216237/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:53 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY is not the one the bucket's configuration now holds: a key change is in flight (ruling 27)
2026/10/13 18:45:55 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_bucket_cannot_be_read71846389/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:55 activator: replica shop-au.example.com not started: the bucket's configuration could not be read: bucket unreachable
2026/10/13 18:45:57 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketno_fresh_reader_refused2516239466/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:57 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY cannot be checked against the bucket's configuration
2026/10/13 18:45:59 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketresidual_a_change_only_o1082814879/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:46:01 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_config-only_homes_keyl4043284078/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:46:03 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_home_row_mints_and_its_4178288933/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:46:07 activator: seeding shop-au.example.com: /tmp/TestReplicaSeedIsGatedthe_homes_key3654129565/001/live/shop-au.example.com seed --db /tmp/TestReplicaSeedIsGatedthe_homes_key3654129565/002/shop-au.example.com.db
2026/10/13 18:46:07 activator: seeded shop-au.example.com
2026/10/13 18:46:11 activator: replica shop-au.example.com not started: where its CARLOS_INSTANCE_KEY came from is unknown
2026/10/13 18:46:17 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaGateDecidesOnTheLaunchesOwnRead2484252515/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:46:20 activator: swap.example.com hibernated
2026/10/13 18:46:20 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/13 18:46:20 activator: swap.example.com hibernated
2026/10/13 18:46:20 activator: LEASE LOST for swap.example.com (renewal failed) — standing down
2026/10/13 18:46:20 activator: swap.example.com hibernated
2026/10/13 18:46:20 activator: swap.example.com hibernated
2026/10/13 18:46:20 activator: swap.example.com is no longer hibernate-managed — releasing
2026/10/13 18:46:20 activator: swap.example.com hibernated
2026/10/13 18:46:20 activator: swap.example.com hibernated
2026/10/13 18:46:20 activator: stop missed-join.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/13 18:46:20 activator: missed-join.example.com hibernated
2026/10/13 18:46:20 activator: stop nested-stuck.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/13 18:46:20 activator: nested-stuck.example.com: the sidecar a restart brought back for a stopping entry is not proven gone (activator: stop unproven; processes remain); keeping the lease until it has exited
2026/10/13 18:46:21 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/13 18:46:21 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/13 18:46:21 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/13 18:46:21 activator: nested-stuck.example.com hibernated
2026/10/13 18:46:21 activator: stop sidecar nested-costop.example.com: activator: stop unproven; processes remain
2026/10/13 18:46:21 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/13 18:46:21 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/13 18:46:21 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/13 18:46:21 activator: nested-costop.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: cgroup: the host's parent cgroup holds a process directly: holds 1 process(es)
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a cgroup that took cgroup.kill could not be made fresh before a spawn: carlos-exec@h.example.com: device or resource busy
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a generation's limits could not be applied: h.example.com: the placement shim is not available
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:46:21 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:46:21 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:46:21 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/13 18:46:21 activator: bot.x.internal hibernated
2026/10/13 18:46:21 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:46:21 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory
2026/10/13 18:46:21 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:46:21 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/13 18:46:21 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:46:21 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:46:21 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory
2026/10/13 18:46:21 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:46:21 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/13 18:46:21 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:46:21 activator: h.example.com: kill carlos-instance-gen@h.example.com:1348.service after a failed start: stop timed out
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:1356.service exited before it was observed
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: h.example.com: kill carlos-instance-gen@h.example.com:1358.service after a failed start: stop timed out
2026/10/13 18:46:21 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:1358.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: h.example.com: kill carlos-instance-gen@h.example.com:1362.service after a failed start: stop timed out
2026/10/13 18:46:21 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: the unit failed to start; activator: generation start failed and its termination is unproven: stop timed out
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/13 18:46:21 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:46:22 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:46:22 activator: worker bot.x.internal: the attempt's respawn did not finish within 150ms of its caller leaving; reporting its build as possibly executed
2026/10/13 18:46:22 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: a stop-evidence operation holds the host
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: socket dir: permission denied
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: start g: activator: generation start failed and its termination is unproven
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: LEASE HELD for swap.example.com by "box-b" — a previous incarnation hasn't released; refusing to wake (retry after 1m0s)
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: instance lease held by "box-b" (retry in 1m0s)
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: restore swap.example.com: replica unreadable
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: staging: staged digest mismatch: have be0041e1a437dbea5b67dfe58bad04a0fc30d5036d469429f3534f39ec53d8b5, want c3bae8fb3a8bf395783d7f7f6ffcd4aadc797c95ce24b7e8e2a98af6b63d8f6d
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: the backend ran a different generation from the hold's choice: generation 1391 of v9 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_3014797268/004/versions/v2/app"; the hold chose v2 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_3014797268/004/versions/v2/app"
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com: generation 1398's child is not proven empty: the child still holds a process
2026/10/13 18:46:23 activator: swap.example.com: failed wake's generation 1398 is not proven stopped; keeping the lease until it is
2026/10/13 18:46:23 activator: swap.example.com: failed wake's generation 1398 is proven stopped; lease given back
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com: failed wake's generation 1400 is not proven stopped; keeping the lease until it is
2026/10/13 18:46:23 activator: swap.example.com: failed wake's generation 1400 is proven stopped; lease given back
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3047469485/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3047469485/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:46:23 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3047469485/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3047469485/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:46:23 activator: h.example.com: started carlos-instance-gen@h.example.com:7.service memory.max=1073741824
2026/10/13 18:46:23 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: processes remain
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/13 18:46:23 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:24 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/13 18:46:24 activator: swap.example.com hibernated
2026/10/13 18:46:24 activator: swap.example.com hibernated
2026/10/13 18:46:24 activator: swap.example.com hibernated
2026/10/13 18:46:25 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/13 18:46:25 activator: swap.example.com hibernated
2026/10/13 18:46:25 activator: swap.example.com hibernated
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 4s)
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: swap.example.com hibernated
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: swap.example.com hibernated
2026/10/13 18:46:25 activator: swap.example.com hibernated
2026/10/13 18:46:25 activator: swap.example.com: a discarded SIDECAR is not proven gone (activator: stop unproven; processes remain); keeping it tracked
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:25 activator: SIDECAR for swap.example.com is not proven gone for its replacement; tracking it (the sweep re-proves it)
2026/10/13 18:46:25 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/13 18:46:25 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:25 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/13 18:46:25 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:25 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:25 activator: swap.example.com hibernated
2026/10/13 18:46:26 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:26 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:26 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:26 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/13 18:46:26 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:26 activator: swap.example.com hibernated
2026/10/13 18:46:26 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:26 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:26 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:26 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/13 18:46:26 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 8s)
2026/10/13 18:46:26 activator: sidecar for swap.example.com was down — restarted (backoff now 16s)
2026/10/13 18:46:26 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:26 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:32 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:32 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: instance control /stop: 502: stop: processes remain
2026/10/13 18:46:32 activator: h.example.com: unit restarted outside the activator (invocation someone-else, ours inv7); stopping it
2026/10/13 18:46:32 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:32 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:32 activator: h.example.com: processes remain after failed; not releasing
2026/10/13 18:46:33 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:33 activator: h.example.com: unit restarted outside the activator (invocation inv2, ours inv1); stopping it
2026/10/13 18:46:33 activator: seeding h.example.com: /tmp/TestSeedUnitCleansUpOnlyItsOwnDBOnFailure1245886/001/live/h.example.com seed --fast --db /tmp/nb2485948109/data/instances/h.example.com/h.example.com.db
2026/10/13 18:46:33 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:33 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:33 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:33 activator: seeding h.example.com: /tmp/TestSeedUnitFailureStopsBeforeCleanupAndKeepsTheDBWhenProcessesR3201482150/001/live/h.example.com seed --fast --db /tmp/nb2670581510/data/instances/h.example.com/h.example.com.db
2026/10/13 18:46:53 activator: seeding h.example.com: /tmp/TestANestedSeedRecordsItsObligationBeforeItRuns2951185448/001/live/h.example.com seed --db /tmp/nb38749598/data/instances/h.example.com/h.example.com.db
2026/10/13 18:46:53 activator: seeded h.example.com
2026/10/13 18:46:53 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI3785547654/001/live/h.example.com seed --db /tmp/nb1423004784/data/instances/h.example.com/h.example.com.db
2026/10/13 18:46:53 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/13 18:46:53 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/13 18:46:53 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI187158737/001/live/h.example.com seed --db /tmp/nb3326703105/data/instances/h.example.com/h.example.com.db
2026/10/13 18:46:53 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/13 18:46:53 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/13 18:46:53 activator: fifo-obligation.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/13 18:46:53 activator: one.example.com: first wake of v2 had 30s of its health window left at the exec, under the 1h0m0s floor; booting the good build v1 without trying it
2026/10/13 18:46:53 activator: one.example.com hibernated
2026/10/13 18:46:54 activator: tenant.test not started: it is held on v1, and the launch hashed "v2" (the live link moved after the check); the next wake checks again
2026/10/13 18:46:54 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/13 18:46:54 activator: one.example.com hibernated
2026/10/13 18:46:54 activator: one.example.com not woken: it is held on v1, and this wake would run "v2" (the live link moved after the check); the next wake checks again
2026/10/13 18:46:54 activator: one.example.com not woken: it is held on v1, and this wake would run "" (the live link moved after the check); the next wake checks again
2026/10/13 18:46:54 activator: swap.example.com hibernated
2026/10/13 18:47:10 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: two.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/13 18:47:10 activator: seeding h.example.com: /tmp/TestANestedUnversionedLinkWithNoHookWakesAsToday2475305643/001/live/h.example.com seed --db /tmp/nb1545143618/data/instances/h.example.com/h.example.com.db
2026/10/13 18:47:10 activator: seeded h.example.com
2026/10/13 18:47:10 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:47:10 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: swap.example.com hibernated
2026/10/13 18:47:10 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/13 18:47:10 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: SIDECAR restart for one.example.com failed: the build state could not be written, so this build was not started: bucket refused the build state (next attempt in 2s)
2026/10/13 18:47:10 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: SIDECAR for one.example.com failed to start: the build state could not be written, so this build was not started: bucket refused the build state — serving without it (the sweep retries)
2026/10/13 18:47:10 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/13 18:47:10 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:47:10 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:47:10 activator: SIDECAR restart for one.example.com failed: swap: lease lost (next attempt in 2s)
2026/10/13 18:47:10 activator: one.example.com: generation 2309's child is not proven empty: activator: stop unproven; processes remain
2026/10/13 18:47:10 activator: one.example.com: failed wake's generation 2309 is not proven stopped; keeping the lease until it is
2026/10/13 18:47:10 activator: one.example.com hibernated
2026/10/13 18:47:11 activator: one.example.com: first wake of v2 had 400ms of its health window left at the exec, under the 20s floor; booting the good build v1 without trying it
2026/10/13 18:47:27 activator: x.example: hold's pinned wake refused or abandoned: the claims admission refused before the exec: hostbox: unresolved lifecycle contract: isolated root launcher cannot carry claim reservations
2026/10/13 18:47:27 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866339050/003/g/f4705dce873e.2316.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:47:29 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866339050/003/g/f4705dce873e.2317.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:47:29 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866339050/003/g/f4705dce873e.2317.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866339050/003/g/f4705dce873e.2317.sock: no such file or directory)
2026/10/13 18:47:29 activator: real.example.com hibernated
2026/10/13 18:47:31 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign3766061762/004/hand-set.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign3766061762/004/hand-set.sock: no such file or directory)
2026/10/13 18:47:31 activator: real.example.com hibernated
2026/10/13 18:47:31 activator: bot.x.internal: worker generation 0's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:31 activator: bot.x.internal: worker generation 2323's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:31 activator: bot.x.internal: failed wake's generation 2323 is not proven stopped; keeping the lease until it is
2026/10/13 18:47:31 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd: not a directory; the sweep retries
2026/10/13 18:47:31 activator: worker bot.x.internal not spawned: the build it would exec (v2 /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/003/v2) could not be published first: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd: not a directory
2026/10/13 18:47:31 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd: not a directory; the sweep retries
2026/10/13 18:47:31 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd: not a directory; the sweep retries
2026/10/13 18:47:31 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn1271946159/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn1271946159/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:31 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn1271946159/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn1271946159/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:31 activator: instance one.example.com exited unexpectedly — recycling
2026/10/13 18:47:31 activator: one.example.com hibernated
2026/10/13 18:47:32 activator: one.example.com hibernated
2026/10/13 18:47:34 activator: one.example.com: first wake of v2 had 499ms of its health window left at the exec, under the 1s floor; booting the good build v1 without trying it
2026/10/13 18:47:35 activator: h.example.com: started carlos-instance-gen@h.example.com:2341.service memory.max=536870912
2026/10/13 18:47:35 activator: one.example.com hibernated
2026/10/13 18:47:35 activator: h.example.com: kill carlos-instance-gen@h.example.com:2342.service after a failed start: stop timed out
2026/10/13 18:47:35 activator: one.example.com: wake failed and its generation is not proven stopped: start one.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2342.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/13 18:47:36 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/13 18:47:36 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/13 18:47:36 activator: worker bot.example.com respawned (exited), now running 
2026/10/13 18:47:36 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/13 18:47:36 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/13 18:47:36 activator: worker bot.example.com respawned (exited), now running 
2026/10/13 18:47:36 activator: bot.example.com: worker generation 2346's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:36 activator: stop bot.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/13 18:47:36 activator: bot.example.com hibernated
2026/10/13 18:47:36 activator: bot3.example.com: worker generation 2347's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:36 activator: worker bot3.example.com NOT respawned (test: a restart with no old handle): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/13 18:47:36 activator: bot4.example.com: worker generation 2349's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:36 activator: bot4.example.com: failed wake's generation 2349 is not proven stopped; keeping the lease until it is
2026/10/13 18:47:36 activator: bot4.example.com: failed wake's generation 2349 is proven stopped; lease given back
2026/10/13 18:47:36 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/13 18:47:36 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/13 18:47:36 activator: bot2.example.com: worker generation 2350's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:36 activator: worker bot2.example.com NOT respawned (exited): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/13 18:47:36 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/13 18:47:36 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/13 18:47:36 activator: worker bot2.example.com respawned (exited), now running 
2026/10/13 18:47:36 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/13 18:47:36 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/13 18:47:36 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:36 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:36 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:47:36 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:36 activator: bot.x.internal hibernated
2026/10/13 18:47:37 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: worker bot.example.com respawned (exited), now running v1
2026/10/13 18:47:37 activator: bot.example.com hibernated
2026/10/13 18:47:37 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:37 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory; the sweep retries
2026/10/13 18:47:37 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory; the sweep retries
2026/10/13 18:47:37 activator: worker first.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory
2026/10/13 18:47:37 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory; the sweep retries
2026/10/13 18:47:37 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory; the sweep retries
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory; the sweep retries
2026/10/13 18:47:37 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory
2026/10/13 18:47:37 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory; the sweep retries
2026/10/13 18:47:37 activator: WORKER bot.x.internal failed to respawn (exited): worker build not published before its exec: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory — the sweep's crash restart takes it from here
2026/10/13 18:47:37 activator: bot.example.com hibernated
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:37 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/13 18:47:37 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:37 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: WORKER bot.x.internal failed to respawn (exited): fork/exec: resource temporarily unavailable — the sweep's crash restart takes it from here
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:37 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/13 18:47:37 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/13 18:47:37 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/13 18:47:37 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:47:37 activator: worker bot.x.internal hit its max_run; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:37 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (hit its max_run), now running v2
2026/10/13 18:47:38 activator: worker bot1.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot1.x.internal exited; it serves v1 but live/bot1.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/13 18:47:38 activator: worker bot2.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot2.x.internal exited; live/bot2.x.internal names v2, but this sweep's converge budget is spent, so it recycles its captured build v1 and converges on a later sweep
2026/10/13 18:47:38 activator: worker bot2.x.internal respawned (exited), now running v1
2026/10/13 18:47:38 activator: worker bot1.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:38 activator: worker bot2.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:47:38 activator: worker bot2.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot.x.internal exited; live/bot.x.internal names v2, but the release scheduler owns build changes, so it recycles its captured build v1
2026/10/13 18:47:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot.x.internal exited; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/13 18:47:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:38 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot10.example.com: the shed respawn child's exit is not proven (activator: stop unproven; processes remain); the entry keeps it and its build until the sweep proves it
2026/10/13 18:47:38 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/13 18:47:38 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/13 18:47:39 activator: bot10.example.com: the shed respawn child has exited
2026/10/13 18:47:39 activator: bot10.example.com hibernated
2026/10/13 18:47:39 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/13 18:47:39 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/13 18:47:39 activator: bot10.example.com respawn did not finish before its stop; stopping the rest, the lease is kept until it has
2026/10/13 18:47:39 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/13 18:47:39 activator: bot10.example.com hibernated
2026/10/13 18:47:39 activator: bot8.example.com: worker generation 2363's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:39 activator: bot8.example.com: failed wake's generation 2363 is not proven stopped; keeping the lease until it is
2026/10/13 18:47:39 activator: bot8.example.com: failed wake's generation 2363 is proven stopped; lease given back
2026/10/13 18:47:39 activator: worker bot9.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/13 18:47:39 activator: bot9.example.com: worker generation 2365's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:39 activator: generation 2365 of bot9.example.com did not prove its exit; keeping it in the operation set
2026/10/13 18:47:39 activator: LEASE LOST for bot9.example.com (renewal failed) — standing down
2026/10/13 18:47:39 activator: bot9.example.com hibernated
2026/10/13 18:47:39 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:39 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:39 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/13 18:47:39 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:39 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:39 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/13 18:47:40 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/13 18:47:40 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:40 activator: bot.x.internal hibernated
2026/10/13 18:47:40 activator: worker bot.x.internal respawned (attempt (restart)), now running v1
2026/10/13 18:47:40 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:47:40 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:40 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:41 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:41 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:47:41 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:47:41 activator: bot.x.internal: worker generation 2368's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:41 activator: worker bot.x.internal NOT respawned (attempt (release)): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/13 18:47:41 activator: bot.x.internal: worker generation 2368's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:41 activator: worker bot.x.internal NOT respawned (restoring v1): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/13 18:47:41 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: lease lost; abandoning it
2026/10/13 18:47:41 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:41 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory
2026/10/13 18:47:41 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:41 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/13 18:47:41 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:41 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:41 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory
2026/10/13 18:47:41 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:41 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:41 activator: bot.x.internal hibernated
2026/10/13 18:47:41 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:41 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:42 activator: worker bot.x.internal NOT respawned (falling back to v1): swap: lease lost; abandoning it
2026/10/13 18:47:42 activator: WORKER bot.x.internal failed to respawn (attempt (restart)): exec: text file busy — the sweep's crash restart takes it from here
2026/10/13 18:47:42 activator: WORKER bot.x.internal failed to respawn (restoring v1): exec: text file busy — the sweep's crash restart takes it from here
2026/10/13 18:47:42 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:42 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/13 18:47:42 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:47:42 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/13 18:47:42 activator: bot.x.internal hibernated
2026/10/13 18:47:42 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:47:42 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:47:42 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:47:42 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:47:42 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:42 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/13 18:47:42 activator: bot.x.internal hibernated
2026/10/13 18:47:43 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:43 activator: LEASE LOST for bot.x.internal (renewal failed) — standing down
2026/10/13 18:47:43 activator: bot.x.internal hibernated
2026/10/13 18:47:43 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:43 activator: bot.x.internal hibernated
2026/10/13 18:47:43 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:43 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/13 18:47:43 activator: bot.x.internal hibernated
FAIL
FAIL	github.com/carlosframework/platform/internal/activator	227.597s
ok  	github.com/carlosframework/platform/internal/agentsock	0.016s
ok  	github.com/carlosframework/platform/internal/alerting	0.006s
ok  	github.com/carlosframework/platform/internal/analytics	0.022s
ok  	github.com/carlosframework/platform/internal/appconfig	2.388s
ok  	github.com/carlosframework/platform/internal/apperr	0.074s
ok  	github.com/carlosframework/platform/internal/applog	0.141s
ok  	github.com/carlosframework/platform/internal/awsinv	0.004s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.006s
ok  	github.com/carlosframework/platform/internal/boxid	0.534s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.011s
ok  	github.com/carlosframework/platform/internal/caps	0.013s
ok  	github.com/carlosframework/platform/internal/certwarm	0.002s
ok  	github.com/carlosframework/platform/internal/cgroup	0.695s
ok  	github.com/carlosframework/platform/internal/changelog	0.321s
ok  	github.com/carlosframework/platform/internal/clamd	0.055s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	368.520s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.283s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.009s
ok  	github.com/carlosframework/platform/internal/dbverify	0.004s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.003s
ok  	github.com/carlosframework/platform/internal/docsite	0.005s
ok  	github.com/carlosframework/platform/internal/domains	2.185s
ok  	github.com/carlosframework/platform/internal/doorkey	0.006s
ok  	github.com/carlosframework/platform/internal/economics	0.004s
ok  	github.com/carlosframework/platform/internal/edge	27.031s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.008s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.161s
ok  	github.com/carlosframework/platform/internal/gate	1.656s
ok  	github.com/carlosframework/platform/internal/genselect	0.015s
ok  	github.com/carlosframework/platform/internal/geo	0.017s
ok  	github.com/carlosframework/platform/internal/geoip	0.029s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.093s
ok  	github.com/carlosframework/platform/internal/host	108.550s
ok  	github.com/carlosframework/platform/internal/hostactors	0.550s
ok  	github.com/carlosframework/platform/internal/hostbox	2.187s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.504s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.012s
ok  	github.com/carlosframework/platform/internal/hostexec	6.178s
ok  	github.com/carlosframework/platform/internal/hostfence	0.041s
ok  	github.com/carlosframework/platform/internal/hostfiles	11.690s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.012s
ok  	github.com/carlosframework/platform/internal/hostguard	1.016s
ok  	github.com/carlosframework/platform/internal/hostops	1.489s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.013s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.013s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.463s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.136s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.012s
ok  	github.com/carlosframework/platform/internal/infraview	0.014s
ok  	github.com/carlosframework/platform/internal/instancectl	1.829s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.709s
ok  	github.com/carlosframework/platform/internal/instancefs	0.018s
ok  	github.com/carlosframework/platform/internal/instancekey	0.015s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.018s
ok  	github.com/carlosframework/platform/internal/layout	1.437s
ok  	github.com/carlosframework/platform/internal/lease	9.603s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.006s
ok  	github.com/carlosframework/platform/internal/ledger	0.016s
ok  	github.com/carlosframework/platform/internal/mail	0.163s
ok  	github.com/carlosframework/platform/internal/maintenance	0.082s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.634s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.067s
ok  	github.com/carlosframework/platform/internal/metrics	0.006s
ok  	github.com/carlosframework/platform/internal/moves	22.306s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.795s
ok  	github.com/carlosframework/platform/internal/objstore	0.170s
ok  	github.com/carlosframework/platform/internal/ops	0.180s
ok  	github.com/carlosframework/platform/internal/park	0.222s
ok  	github.com/carlosframework/platform/internal/privatefile	0.004s
ok  	github.com/carlosframework/platform/internal/provision	18.436s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.007s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	4.354s
ok  	github.com/carlosframework/platform/internal/relay	7.523s
ok  	github.com/carlosframework/platform/internal/release	24.713s
ok  	github.com/carlosframework/platform/internal/rollout	4.088s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.003s
ok  	github.com/carlosframework/platform/internal/rootaws	0.009s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.007s
ok  	github.com/carlosframework/platform/internal/rootstate	0.025s
ok  	github.com/carlosframework/platform/internal/s3compat	0.004s
ok  	github.com/carlosframework/platform/internal/safefile	0.003s
ok  	github.com/carlosframework/platform/internal/schedule	2.365s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	19.306s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	4.482s
ok  	github.com/carlosframework/platform/internal/status	0.037s
ok  	github.com/carlosframework/platform/internal/steering	0.006s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.196s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.007s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.002s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.004s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.559s
FAIL
make: *** [Makefile:18: test] Error 1

amadan: step gate failed: exit status 2
  2. cross not run

    No output.

  3. minio-contract not run

    No output.

Whole run log 
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	351.767s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.327s
ok  	github.com/carlosframework/platform/cmd/moveprobe	92.891s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.384s
2026/10/13 18:43:56 activator: badseed.example.com: seed for badseed.example.com: activator: seed unproven; processes remain: processes remain after a successful seed: still running; leaving the lease to its TTL
2026/10/13 18:43:56 activator: unproven.example.com: unproven.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone; leaving the lease to its TTL
2026/10/13 18:43:56 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/13 18:43:56 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof2715425685/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof2715425685/004/next-wake.example.com.db
2026/10/13 18:43:56 activator: next-wake.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof2715425685/004/next-wake.example.com.db.seed-unproven refuses it as warm until they are
2026/10/13 18:43:56 activator: next-wake.example.com: seeding next-wake.example.com: exit status 1: activator: seed unproven; processes remain (the seed's processes remain after stopping it; its database was left in place: cgroup: strict containment unsupported); leaving the lease to its TTL
2026/10/13 18:43:56 activator: next-wake.example.com: next-wake.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup: strict containment unsupported; leaving the lease to its TTL
2026/10/13 18:43:56 activator: next-wake.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/13 18:43:56 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/13 18:43:56 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof2715425685/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof2715425685/004/next-wake.example.com.db
2026/10/13 18:43:56 activator: seeded next-wake.example.com
2026/10/13 18:43:56 activator: one.example.com idle 11s — hibernating
2026/10/13 18:43:56 activator: one.example.com hibernated
2026/10/13 18:43:56 activator: one.example.com idle 11s — hibernating
2026/10/13 18:43:56 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: two.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/13 18:43:57 activator: two.example.com hibernated
2026/10/13 18:43:57 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: LEASE LOST for one.example.com (lease renewal failed) — standing down
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: instance one.example.com exited unexpectedly — recycling
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: one.example.com idle 1m0s — hibernating
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: LEASE HELD for held.example.com by "old-incarnation" — a previous incarnation hasn't released; refusing to wake (retry after 7s)
2026/10/13 18:43:57 activator: faroff.bab.oncarlos.com idle 16m0s — hibernating
2026/10/13 18:43:57 activator: faroff.bab.oncarlos.com hibernated
2026/10/13 18:43:57 activator: untimed.bab.oncarlos.com idle 16m0s — hibernating
2026/10/13 18:43:57 activator: untimed.bab.oncarlos.com hibernated
2026/10/13 18:43:57 activator: tickheld.bab.oncarlos.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/13 18:43:57 activator: tickheld.bab.oncarlos.com hibernated
2026/10/13 18:43:57 activator: flip.example.com is no longer hibernate-managed — releasing
2026/10/13 18:43:57 activator: flip.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/13 18:43:57 activator: flip.example.com hibernated
2026/10/13 18:43:57 activator: mid.example.com has a scheduled delivery due — waking it
2026/10/13 18:43:57 activator: mid.example.com is no longer hibernate-managed — releasing (wake finished first)
2026/10/13 18:43:57 activator: mid.example.com hibernated
2026/10/13 18:43:57 activator: two.example.com hibernated
2026/10/13 18:43:57 activator: one.example.com hibernated
2026/10/13 18:43:57 activator: b.example.com hibernated
2026/10/13 18:43:57 activator: own.example.com idle 1m0s — hibernating
2026/10/13 18:43:57 activator: own.example.com hibernated
2026/10/13 18:43:57 activator: due.example.com idle 11s — hibernating
2026/10/13 18:43:57 activator: due.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/13 18:43:57 activator: due.example.com hibernated
2026/10/13 18:43:57 activator: due.example.com has a scheduled delivery due — waking it
2026/10/13 18:43:57 activator: due.example.com idle 11s — hibernating
2026/10/13 18:43:57 activator: due.example.com hibernated
2026/10/13 18:43:57 activator: late.example.com idle 11s — hibernating
2026/10/13 18:43:57 activator: late.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/13 18:43:57 activator: late.example.com hibernated
2026/10/13 18:43:57 activator: late.example.com has a scheduled delivery due — waking it
2026/10/13 18:43:57 activator: deaf.example.com idle 11s — hibernating
2026/10/13 18:43:57 activator: next-due probe deaf.example.com: dial unix: no such file — recording none (it sleeps until a request)
2026/10/13 18:43:57 activator: deaf.example.com hibernated
2026/10/13 18:43:57 activator: reborn.example.com idle 11s — hibernating
2026/10/13 18:43:57 activator: reborn.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/13 18:43:57 activator: reborn.example.com hibernated
2026/10/13 18:43:57 activator: reborn.example.com has a scheduled delivery due — waking it
2026/10/13 18:43:57 activator: SIDECAR for botless.example.com failed to start: exec: live symlink gone — serving without it (the sweep retries)
2026/10/13 18:43:57 activator: SIDECAR restart for botless.example.com failed: exec: live symlink gone (next attempt in 2s)
2026/10/13 18:43:58 activator: sidecar for botless.example.com was down — restarted (backoff now 4s)
2026/10/13 18:43:58 activator: bots.example.com idle 11s — hibernating
2026/10/13 18:43:58 activator: bots.example.com hibernated
2026/10/13 18:43:58 activator: instance bots.example.com exited unexpectedly — recycling
2026/10/13 18:43:58 activator: bots.example.com hibernated
2026/10/13 18:43:58 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/13 18:43:58 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/13 18:43:58 activator: sidecar for bots.example.com was down — restarted (backoff now 8s)
2026/10/13 18:43:58 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/13 18:43:58 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/13 18:43:58 activator: bots.example.com idle 11s — hibernating
2026/10/13 18:43:58 activator: bots.example.com hibernated
2026/10/13 18:43:58 activator: bot.x.internal idle 11s — hibernating
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: bot.x.internal idle 31s — hibernating
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: bot.x.internal hit its max_run of 1m0s — stopping
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: LEASE LOST for bot.x.internal (lease renewal failed) — standing down
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/13 18:43:58 activator: one.example.com hibernated
2026/10/13 18:43:58 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:58 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:58 activator: bot.x.internal idle 11s — hibernating
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: bot.x.internal idle 11s — hibernating
2026/10/13 18:43:58 activator: bot.x.internal hibernated
2026/10/13 18:43:58 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:59 activator: bot.x.internal idle 11s — hibernating
2026/10/13 18:43:59 activator: bot.x.internal hibernated
2026/10/13 18:43:59 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 4s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 8s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/13 18:43:59 activator: bot.x.internal hibernated
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:43:59 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:43:59 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:43:59 activator: worker bot.x.internal respawned (hit its max_run), now running 
2026/10/13 18:43:59 activator: always-on sidecar bot.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot00.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot01.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot02.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot03.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot04.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot05.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot06.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot07.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot08.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot09.x.internal is not running — starting it
2026/10/13 18:43:59 activator: always-on sidecar bot10.x.internal is not running — starting it
2026/10/13 18:43:59 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:43:59 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:44:00 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:44:00 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:00 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/13 18:44:00 activator: bot.x.internal hibernated
2026/10/13 18:44:01 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:01 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/13 18:44:01 activator: bot.x.internal hibernated
2026/10/13 18:44:01 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:01 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/13 18:44:02 activator: bot.x.internal hibernated
2026/10/13 18:44:02 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:02 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/13 18:44:02 activator: bot.x.internal hibernated
2026/10/13 18:44:02 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:03 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/13 18:44:03 activator: bot.x.internal hibernated
2026/10/13 18:44:03 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:03 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:44:03 activator: worker bot-b.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:03 activator: worker bot-b.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:44:03 activator: worker bot-a.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:03 activator: worker bot-a.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:44:03 activator: bot.x.internal serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/13 18:44:03 activator: bot.x.internal hibernated
2026/10/13 18:44:03 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:03 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:44:03 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:03 activator: WORKER bot.x.internal failed to respawn (converging onto a new build): exec: live symlink gone — the sweep's crash restart takes it from here
2026/10/13 18:44:03 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:03 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:44:03 activator: worker bot.x.internal respawned (exited), now running v2
2026/10/13 18:44:03 activator: worker bot.x.internal reported idle but is always-on (hibernate:false) — ignoring
2026/10/13 18:44:07 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendStartMergesConfigEnvUnderOwnEnv7136075/002/merge-precedence.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/13 18:44:17 activator: purged 1 cached response(s) for deploy.example.com on spawn
2026/10/13 18:44:19 activator: restored restored.example.com from s3://test-bucket/restored.example.com
2026/10/13 18:44:19 activator: damaged.example.com restored CORRUPT from s3://test-bucket/damaged.example.com: integrity_check: file is not a database (26) — quarantined at /tmp/TestExecBackendRestoreCorruptReplicaFailsAndQuarantines3122535118/002/damaged.example.com.db.corrupt, instance NOT started
2026/10/13 18:44:19 activator: restored damaged.example.com from s3://test-bucket/damaged.example.com
2026/10/13 18:44:19 activator: no replica for brand-new.example.com in s3://test-bucket/brand-new.example.com — booting fresh
2026/10/13 18:44:19 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath3587889124/002/seeded.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/13 18:44:19 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath3587889124/001/live/seeded.example.com seed --fixtures --db /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath3587889124/002/seeded.example.com.db
2026/10/13 18:44:19 activator: seeded seeded.example.com
2026/10/13 18:44:19 activator: seeding custom-db.example.com: /tmp/TestExecBackendSeedHonoursRouteDBPath3511175839/001/live/custom-db.example.com seed --db /tmp/TestExecBackendSeedHonoursRouteDBPath3511175839/004/elsewhere.db
2026/10/13 18:44:19 activator: seeded custom-db.example.com
2026/10/13 18:44:19 activator: seeding badseed.example.com: /tmp/TestExecBackendSeedNonZeroExitFails718721801/001/live/badseed.example.com seed --db /tmp/TestExecBackendSeedNonZeroExitFails718721801/002/badseed.example.com.db
schema is corrupt
2026/10/13 18:44:19 activator: badseed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/13 18:44:19 activator: seeding hangs.example.com: /tmp/TestExecBackendSeedIsTimeBounded1082431630/001/live/hangs.example.com seed --db /tmp/TestExecBackendSeedIsTimeBounded1082431630/002/hangs.example.com.db
2026/10/13 18:44:20 activator: hangs.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/13 18:44:20 activator: seeding crashed.example.com: /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB2198959129/001/live/crashed.example.com seed --db /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB2198959129/002/crashed.example.com.db
died mid-schema
2026/10/13 18:44:20 activator: crashed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/13 18:44:20 activator: seeding existing-db.example.com: /tmp/TestExecBackendSeedFailureKeepsPreexistingDB3224982553/001/live/existing-db.example.com seed --db /tmp/TestExecBackendSeedFailureKeepsPreexistingDB3224982553/002/existing-db.example.com.db
nope
2026/10/13 18:44:20 activator: existing-db.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/13 18:44:20 activator: seeding leaky-seed.example.com: /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild260826732/001/live/leaky-seed.example.com seed --db /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild260826732/002/leaky-seed.example.com.db
2026/10/13 18:44:20 activator: seeded leaky-seed.example.com (it exited 0 but left a child holding its output pipe open for more than 200ms)
2026/10/13 18:44:21 activator: squatted.example.com: child exited during the health wait (external process serving its socket?) — not normalizing the socket mode
2026/10/13 18:44:21 activator: h.example.com idle 2m0s — hibernating
2026/10/13 18:44:21 activator: stop h.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/13 18:44:21 activator: h.example.com hibernated
2026/10/13 18:44:21 activator: stuck.example.com idle 2m0s — hibernating
2026/10/13 18:44:21 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/13 18:44:21 activator: shutdown-pending.example.com idle 2m0s — hibernating
2026/10/13 18:44:21 activator: stop shutdown-pending.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/13 18:44:22 activator: shutdown-pending.example.com hibernated
2026/10/13 18:44:22 activator: sidecar for a.example.com was down — restarted (backoff now 2s)
2026/10/13 18:44:22 activator: h.example.com: route refreshed: db "/tmp/h.example.com.db" -> "/data/instances/h.example.com/h.example.com.db" socket "/tmp/nope.sock" -> "/run/carlos/instances/h.example.com/app.sock"
2026/10/13 18:44:22 activator: stop stale.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/13 18:44:22 activator: stale.example.com hibernated
2026/10/13 18:44:22 activator: leftover.example.com: the host cgroup holds processes not proven gone: a leftover still populates the host cgroup; leaving the lease to its TTL
2026/10/13 18:44:22 activator: f6.example.com idle 2m0s — hibernating
2026/10/13 18:44:22 activator: stop f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:44:22 activator: stop f6.example.com: its host cgroup is not proven empty; keeping the lease until it is
2026/10/13 18:44:22 activator: f6.example.com hibernated
2026/10/13 18:44:22 activator: proven.example.com idle 2m0s — hibernating
2026/10/13 18:44:22 activator: proven.example.com hibernated
2026/10/13 18:44:22 activator: undelegated.example.com idle 2m0s — hibernating
2026/10/13 18:44:22 activator: undelegated.example.com: stopped with no delegated cgroup; its leader's exit is the only proof there is
2026/10/13 18:44:22 activator: undelegated.example.com hibernated
2026/10/13 18:44:22 activator: nested-stop.example.com idle 2m0s — hibernating
2026/10/13 18:44:22 activator: nested-stop.example.com hibernated
2026/10/13 18:44:22 activator: old-kernel.example.com: this kernel has no cgroup.kill, so the host cgroup cannot be killed; its leader's exit is the only proof there is
2026/10/13 18:44:22 activator: pending-f6.example.com idle 2m0s — hibernating
2026/10/13 18:44:22 activator: stop pending-f6.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/13 18:44:22 activator: stop pending-f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:44:22 activator: pending-f6.example.com hibernated
2026/10/13 18:44:28 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/13 18:44:28 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/13 18:44:28 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/13 18:44:28 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/13 18:44:28 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/13 18:44:28 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/13 18:44:28 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/13 18:44:28 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/13 18:44:30 activator: retry.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:30 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/13 18:44:30 activator: retry.example.com: strict group not closed: cgroup: strict containment refused
2026/10/13 18:44:31 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/13 18:44:31 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:31 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:31 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:31 activator: stuck.example.com: strict group still not cleaned up after 2 retries; spawns of this host stay refused until the agent restarts
2026/10/13 18:44:31 activator: owned.example.com: strict group not closed: cgroup: strict containment refused
2026/10/13 18:44:32 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/13 18:44:32 activator: owned.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:33 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/13 18:44:33 activator: race.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:33 activator: transition.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:33 activator: transition2.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/13 18:44:33 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/13 18:44:33 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/13 18:44:33 activator: kass.example.com idle 11s — hibernating
2026/10/13 18:44:33 activator: kass.example.com hibernated
2026/10/13 18:44:34 activator: kass.example.com hibernated
--- FAIL: TestUnitTenantWakeServesHeldRequest (0.22s)
    backend_unit_test.go:371: unit listen: listen tcp 127.0.0.1:39363: bind: address already in use
    backend_unit_test.go:421: held request = 200 "<!doctype html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n<title>Sign in · CARLOS console</title>\n<style>\n  :root {\n    --bg: #ffffff; --ink: #201d29; --muted: #6f6480;\n    --accent: #5b3e96; --accent-strong: #4c3380;\n    --accent-soft: rgba(91, 62, 150, 0.09);\n    --rule: #eae7f0; --bad: #b3403a; --warn: #a6790f; --good: #2f7a4f;\n    --mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;\n    --sans: ui-sans-serif, system-ui, -apple-system, sans-serif;\n  }\n  @media (prefers-color-scheme: dark) {\n    :root {\n      --bg: #16141c; --ink: #f0eef6; --muted: #a99ebd;\n      --accent: #b394f2; --accent-strong: #c9b2f7;\n      --accent-soft: rgba(179, 148, 242, 0.12);\n      --rule: #2c2836; --bad: #e07b76; --warn: #d9a94a; --good: #5fae7f;\n    }\n  }\n  [hidden] { display: none !important; }\n  * { margin: 0; padding: 0; box-sizing: border-box; }\n  body {\n    background: var(--bg); color: var(--ink);\n    font: 400 1rem/1.6 var(--sans);\n    -webkit-font-smoothing: antialiased; padding: 0 1.25rem;\n  }\n  a { color: var(--accent); text-decoration-thickness: 1px; text-underline-offset: 2px; }\n  code {\n    font-family: var(--mono);\n    font-size: 0.88em; background: var(--accent-soft);\n    padding: 0.1em 0.35em; border-radius: 0.25rem;\n  }\n  pre {\n    background: var(--accent-soft); border-radius: 0.5rem;\n    padding: 0.9rem 1.1rem; margin: 0 0 1rem; max-width: 40rem;\n    overflow-x: auto; font-family: var(--mono); font-size: 0.85rem; line-height: 1.6;\n  }\n  pre code { background: none; padding: 0; }\n   \n  .changelog { white-space: pre-wrap; }\n  select, input[type=text], input[type=search] {\n    font: inherit; font-family: var(--sans); font-size: 0.9rem; color: var(--ink); background: var(--bg);\n    border: 1px solid var(--rule); border-radius: 0.4rem; padding: 0.3rem 0.5rem;\n  }\n  td form { display: flex; gap: 0.4rem; align-items: center; flex-wrap: wrap; margin: 0; }\n   \n  form.inline { display: inline; margin: 0; }\n  form.inline button { vertical-align: baseline; }\n   \n  form.listctl { display: flex; gap: 0.6rem; flex-wrap: wrap; margin: 0 0 0.6rem; }\n   \n  .icon { width: 1em; height: 1em; flex: none; vertical-align: -0.125em; }\n  header {\n    max-width: 64rem; margin: 0 auto; padding: 1.4rem 0;\n    display: flex; align-items: baseline; gap: 1rem;\n    border-bottom: 1px solid var(--rule);\n  }\n  header .brand a { font-weight: 700; color: var(--ink); text-decoration: none; }\n  header nav {\n    margin-left: auto; display: flex; align-items: baseline; gap: 1rem;\n    font-size: 0.9rem;\n  }\n  header .addr { color: var(--muted); }\n   \n  header .switcher {\n    position: relative;\n  }\n  header .switcher > summary {\n    display: inline-flex; align-items: baseline; gap: 0.4rem;\n    list-style: none; cursor: pointer;\n    padding: 0.25rem 0.5rem; margin: -0.25rem -0.5rem; border-radius: 0.5rem;\n  }\n  header .switcher > summary::-webkit-details-marker { display: none; }\n  header .switcher > summary:hover { background: var(--accent-soft); }\n   \n  header .switcher .current-name {\n    font-weight: 700; min-width: 0; max-width: 14rem;\n    overflow: hidden; text-overflow: ellipsis; white-space: nowrap;\n  }\n   \n  header .switcher > summary > .icon { color: var(--muted); align-self: center; width: 0.9em; }\n  header .switcher[open] > summary > .icon { transform: rotate(180deg); }\n  header .switcher-menu {\n    position: absolute; z-index: 20; top: calc(100% + 0.6rem); left: -0.5rem;\n    min-width: 13rem; max-width: 20rem;\n    background: var(--bg); border: 1px solid var(--rule); border-radius: 0.6rem;\n    padding: 0.35rem; font-size: 0.9rem;\n    box-shadow: 0 0.4rem 1.5rem rgba(0, 0, 0, 0.14);\n  }\n  header .switcher-menu ul { list-style: none; margin: 0; padding: 0; }\n  header .switcher-menu a, header .switcher-menu .current {\n    display: flex; align-items: baseline; gap: 0.5rem;\n    padding: 0.35rem 0.55rem; border-radius: 0.4rem;\n    color: var(--ink); text-decoration: none; overflow-wrap: anywhere;\n  }\n  header .switcher-menu a:hover, header .switcher-menu a:focus { background: var(--accent-soft); }\n  header .switcher-menu .current { font-weight: 600; background: var(--accent-soft); }\n   \n  header .switcher-menu .mark {\n    color: var(--accent); width: 0.9rem; flex: none;\n    display: flex; align-self: center;\n  }\n  header .switcher-menu a .mark { visibility: hidden; }\n  header .switcher-menu .new a .mark { visibility: visible; }\n  header .switcher-menu hr { border: 0; border-top: 1px solid var(--rule); margin: 0.35rem 0.2rem; }\n  header .switcher-menu .new a { color: var(--accent); }\n  main { max-width: 64rem; margin: 0 auto; padding: 2.5rem 0 3rem; min-height: 55vh; }\n  h1 { font-size: 1.9rem; line-height: 1.15; margin: 0 0 1rem; }\n  h2 { font-size: 1.25rem; margin: 2rem 0 0.75rem; }\n  h3 { font-size: 1rem; margin: 1.5rem 0 0.5rem; }\n   \n  .settings-row { display: flex; gap: 0.6rem; flex-wrap: wrap; align-items: center; margin: 0 0 0.75rem; }\n  .settings-layout {\n    display: grid; grid-template-columns: minmax(10rem, 13rem) minmax(0, 1fr);\n    gap: 2.5rem; align-items: start;\n  }\n  .settings-nav { display: flex; flex-direction: column; gap: 0.2rem; }\n  .settings-nav a {\n    display: block; padding: 0.45rem 0.65rem; border-radius: 0.35rem;\n    color: var(--muted); text-decoration: none;\n  }\n  .settings-nav a:hover { color: var(--ink); background: var(--accent-soft); }\n  .settings-nav a[aria-current=\"page\"] {\n    color: var(--ink); background: var(--accent-soft); font-weight: 600;\n  }\n  .settings-content { min-width: 0; }\n  .settings-content > h2:first-child { margin-top: 0; }\n  .danger-head { color: var(--bad); }\n  p { max-width: 40rem; margin: 0 0 1rem; }\n  .lede { font-size: 1.15rem; color: var(--muted); }\n  .small { font-size: 0.9rem; color: var(--muted); }\n   \n  .visually-hidden {\n    position: absolute; width: 1px; height: 1px; overflow: hidden;\n    clip-path: inset(50%); white-space: nowrap;\n  }\n  .error { color: var(--bad); }\n  .banner {\n    background: var(--accent-soft); border-radius: 0.5rem;\n    padding: 0.6rem 0.9rem; margin: 0 0 1.5rem; max-width: 40rem;\n  }\n  .tablewrap { overflow-x: auto; }\n  .instances .instance-address { white-space: nowrap; }\n  .instances td:nth-child(2) { white-space: nowrap; }\n  .instance-open { display: inline-flex; padding: .25rem; }\n  .instance-group th { text-align: left; padding-top: 1rem; background: var(--accent-soft); }\n  .instance-group .small { font-weight: normal; }\n  .small a[aria-current=\"true\"] { color: var(--ink); font-weight: 700; text-decoration-thickness: 3px; }\n  .instance-facts { display: grid; grid-template-columns: minmax(8rem, auto) 1fr; gap: .5rem 1.5rem; margin: 1rem 0; }\n  .instance-facts dt { color: var(--muted); }\n  .instance-facts dd { margin: 0; overflow-wrap: anywhere; }\n  .instance-facts .cellspark { max-width: 24rem; }\n   \n  #logs-filter { display: flex; flex-wrap: wrap; align-items: end; gap: 0.65rem; margin-bottom: 1rem; }\n  #logs-filter label { display: flex; flex-direction: column; gap: 0.25rem; max-width: 100%; }\n  #logs-filter select, #logs-filter input { max-width: 100%; margin: 0; }\n  #logs-filter select[name=\"host\"] { max-width: 28rem; }\n  .term {\n    background: var(--accent-soft); border-radius: 0.5rem;\n    padding: 0.8rem 0 0.8rem 1rem; margin: 0 0 1rem;\n    min-height: 16rem; max-height: 70vh; overflow: auto;\n    font-family: var(--mono); font-size: 0.8rem; line-height: 1.55;\n  }\n   \n  .tl { white-space: pre; width: max-content; min-width: 100%; padding-right: 1rem; }\n   \n  .tl-t, .tl-h, .tl-f { color: var(--muted); }\n  .tl-s { color: var(--accent); }\n   \n  .tl-warn .tl-m { color: var(--warn, #a6790f); }\n  .tl-error .tl-m { color: var(--bad); font-weight: 600; }\n  table { border-collapse: collapse; width: 100%; margin: 0 0 1rem; font-size: 0.95rem; }\n  .tiles { display: flex; flex-wrap: wrap; gap: 1rem; margin: 0 0 1.5rem; }\n  .tile { flex: 1 1 10rem; background: var(--accent-soft); border-radius: 0.6rem; padding: 1rem 1.1rem; }\n   \n  a.tile { color: var(--ink); text-decoration: none; }\n  a.tile:hover { outline: 1px solid var(--accent); }\n   \n  .stat-tiles { display: grid; grid-template-columns: repeat(auto-fit, minmax(17rem, 1fr)); }\n  .tile h2 {\n    margin: 0 0 0.35rem; font-family: var(--mono); font-size: 0.78rem; text-transform: uppercase;\n    letter-spacing: 0.05em; color: var(--muted); font-weight: 600;\n  }\n  .tile-value { font-size: 1.6rem; font-weight: 600; margin: 0; }\n  .tile-detail { color: var(--muted); font-size: 0.8rem; margin: 0.25rem 0 0; }\n   \n  .chip {\n    display: inline-block; font-family: var(--mono); font-size: 0.78rem; font-weight: 500;\n    padding: 0.15rem 0.55rem; border-radius: 999px; margin: 0.15rem 0.3rem 0.15rem 0;\n  }\n  .chip-stale { border: 1px solid var(--warn, #a6790f); color: var(--warn, #a6790f); }\n   \n  .chip-canary { border: 1px solid var(--accent); color: var(--accent); }\n  .chip-prod { background: var(--bad); color: var(--bg); }\n   \n  .chip-expired { border: 1px solid var(--warn, #a6790f); color: var(--warn, #a6790f); }\n   \n  .chip-env { border: 1px solid var(--rule); color: var(--muted); }\n   \n  .chip-health-ok { border: 1px solid var(--good); color: var(--good); }\n  .chip-health-attention { background: var(--warn, #a6790f); color: var(--bg); }\n  .chip-health-notdeployed { border: 1px solid var(--rule); color: var(--muted); }\n  .chip-health-unknown { border: 1px solid var(--warn, #a6790f); color: var(--warn, #a6790f); }\n   \n  .chip-freetier-inside { border: 1px solid var(--warn, #a6790f); color: var(--warn, #a6790f); }\n  .chip-freetier-partial { border: 1px solid var(--accent); color: var(--accent); }\n  .chip-freetier-paying { border: 1px solid var(--good); color: var(--good); }\n  .chip-freetier-pooled { border: 1px solid var(--rule); color: var(--muted); }\n   \n  .chip-tier-unassigned { border: 1px solid var(--rule); color: var(--muted); }\n  .chip-tier-try { border: 1px solid var(--rule); color: var(--muted); }\n  .chip-tier-pool { border: 1px solid var(--rule); color: var(--muted); }\n  .chip-tier-hosted { border: 1px solid var(--good); color: var(--good); }\n  .chip-tier-fleet { border: 1px solid var(--accent); color: var(--accent); }\n  .chip-tier-dedicated { border: 1px solid var(--warn, #a6790f); color: var(--warn, #a6790f); }\n  .chip .icon { vertical-align: -0.1em; margin-right: 0.2em; }\n  .chip-up { background: var(--accent-soft); color: #2f7a4f; }\n  .chip-down { background: var(--accent-soft); color: var(--bad); }\n  .chips { display: flex; flex-wrap: wrap; margin: 0 0 1rem; }\n   \n  .bar { display: inline-block; width: 6rem; height: 0.5rem; border-radius: 999px; background: var(--accent-soft); vertical-align: middle; overflow: hidden; }\n  .bar-fill { display: block; height: 100%; background: var(--accent); }\n   \n  .knob-row { display: flex; gap: 0.6rem; align-items: center; margin: 0 0 0.5rem; flex-wrap: wrap; }\n  .knob-row label { min-width: 12rem; color: var(--muted); font-size: 0.9rem; }\n  .knob-row input[type=number], .knob-row input[type=text] { width: 8rem; }\n  .knob-row input[type=range] { width: 12rem; }\n   \n  .flow { margin: 0 0 1.75rem; }\n  .flow-line {\n    --flow-y: 0.52rem;\n    display: flex; flex-wrap: wrap; gap: 1.1rem 2.6rem;\n    list-style: none; margin: 0; padding: 0;\n  }\n  .flow-stage { position: relative; flex: 1 1 10rem; min-width: 9rem; }\n  .flow-stage + .flow-stage::before, .flow-stage + .flow-stage::after {\n    content: \"\"; position: absolute; pointer-events: none;\n  }\n  .flow-stage + .flow-stage::before {\n    left: -2.05rem; top: var(--flow-y); width: 1.3rem; border-top: 1px dashed var(--rule);\n  }\n  .flow-stage + .flow-stage::after {\n    left: -0.82rem; top: var(--flow-y); transform: translateY(-50%);\n    border: 0.28rem solid transparent; border-right: 0; border-left-color: var(--rule);\n  }\n  .flow-stage.is-carried::before { border-top-style: solid; border-top-color: var(--accent); }\n  .flow-stage.is-carried::after { border-left-color: var(--accent); }\n  .flow-label {\n    margin: 0; font-family: var(--mono); font-size: 0.74rem; text-transform: uppercase;\n    letter-spacing: 0.05em; color: var(--muted); font-weight: 600;\n  }\n  .flow-version { margin: 0.3rem 0 0; font-size: 0.95rem; }\n   \n  .flow-stage:not(.is-held) .flow-version { color: var(--muted); }\n  .flow-note { margin: 0.25rem 0 0; font-size: 0.78rem; color: var(--muted); }\n  .flow-addrs {\n    list-style: none; margin: 0.6rem 0 0; padding: 0 0 0 0.7rem;\n    border-left: 1px solid var(--rule); display: grid; gap: 0.35rem;\n  }\n  .flow-addr { font-size: 0.82rem; line-height: 1.35; }\n  .flow-addr a { display: block; overflow-wrap: anywhere; }\n  .flow-health { display: inline-flex; align-items: center; gap: 0.2rem; color: var(--muted); }\n  .flow-health .icon { width: 0.9em; height: 0.9em; flex: none; }\n  .flow-tone-ok .flow-health { color: var(--good); }\n  .flow-tone-wait .flow-health { color: var(--warn, #a6790f); }\n  .flow-tone-bad .flow-health { color: var(--bad); font-weight: 600; }\n  .flow-band { margin: 1.4rem 0 0.7rem; font-size: 0.9rem; color: var(--muted); }\n  .flow-caption { margin: 1.1rem 0 0; }\n   \n  @media (max-width: 46rem) {\n    .settings-layout { display: block; }\n    .settings-nav {\n      flex-direction: row; overflow-x: auto; margin: 0 0 1.5rem;\n      padding-bottom: 0.25rem;\n    }\n    .settings-nav a { white-space: nowrap; }\n    .flow-line { display: block; }\n    .flow-stage + .flow-stage { margin-top: 1.6rem; }\n    .flow-stage + .flow-stage::before {\n      left: 0.55rem; top: -1.2rem; width: 0; height: 0.85rem;\n      border-top: 0; border-left: 1px dashed var(--rule);\n    }\n    .flow-stage + .flow-stage::after {\n      left: 0.55rem; top: -0.34rem; transform: translateX(-50%);\n      border: 0.28rem solid transparent; border-bottom: 0; border-top-color: var(--rule);\n      border-left-color: transparent;\n    }\n    .flow-stage.is-carried::before { border-left-color: var(--accent); }\n    .flow-stage.is-carried::after { border-top-color: var(--accent); border-left-color: transparent; }\n  }\n   \n  .ladder { list-style: none; margin: 0 0 1rem; padding: 0; max-width: 40rem; }\n  .rung {\n    border: 1px solid var(--rule); border-radius: 0.6rem; padding: 0.8rem 1rem;\n  }\n  .rung + .rung { margin-top: 0.4rem; }\n  .rung-name {\n    font-family: var(--mono);\n    font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.05em;\n    font-weight: 600; color: var(--muted); margin: 0 0 0.3rem;\n  }\n  .rung p { margin: 0 0 0.2rem; }\n  .rung .gate { margin: 0.5rem 0 0; }\n  .rung .routes { list-style: none; margin: 0.3rem 0 0.6rem; padding: 0; font-size: 0.95rem; }\n  .rung .routes li { padding: 0.1rem 0; }\n   \n  .step {\n    display: flex; align-items: baseline; gap: 0.5rem; flex-wrap: wrap;\n    margin: 0.7rem -1rem -0.8rem; padding: 0.5rem 1rem;\n    border-top: 1px solid var(--rule);\n  }\n  .step form { display: flex; align-items: baseline; gap: 0.5rem; margin: 0; }\n  .step > .icon { color: var(--muted); align-self: center; }\n  th {\n    font-family: var(--mono);\n    font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.05em;\n    color: var(--muted); text-align: left; font-weight: 600;\n  }\n  th, td { padding: 0.5rem 1rem 0.5rem 0; border-bottom: 1px solid var(--rule); }\n  form.signin { display: flex; gap: 0.6rem; flex-wrap: wrap; margin: 0 0 1rem; }\n  form.signin input[type=text] { padding: 0.55rem 0.8rem; min-width: 16rem; }\n  button, .btn {\n    font-size: 0.95rem; font-weight: 600; background: var(--accent);\n    color: var(--bg); border: 0; border-radius: 0.4rem;\n    padding: 0.6rem 1.2rem; cursor: pointer;\n    text-decoration: none; display: inline-block;\n  }\n  button.linkish {\n    background: none; color: var(--accent); font-weight: 400; padding: 0;\n    border-radius: 0; text-decoration: underline; font-size: 0.9rem; cursor: pointer;\n  }\n  button:not(.linkish):hover, .btn:hover { background: var(--accent-strong); }\n  button.linkish.danger { color: var(--bad); }\n   \n  .btn-sm, button.btn-sm {\n    font-size: 0.85rem; padding: 0.35rem 0.8rem; border-radius: 0.4rem;\n  }\n  button.btn-danger, .btn-danger { background: var(--bad); color: var(--bg); }\n  button.btn-danger:hover, .btn-danger:hover { background: var(--bad); filter: brightness(0.9); }\n   \n  td .cell-value { display: inline-block; min-width: 3.5rem; font-weight: 600; }\n  .cellspark { width: 6rem; height: 1.1rem; vertical-align: -0.2rem; color: var(--muted); }\n  .cellspark polyline { stroke-width: 2.5; }\n   \n  .feed { list-style: none; margin: 0 0 0.5rem; padding: 0; max-width: 40rem; }\n  .feed li { padding: 0.25rem 0; border-bottom: 1px solid var(--rule); }\n  .feed li:last-child { border-bottom: 0; }\n   \n  .confirm-card {\n    border: 1px solid var(--rule); border-radius: 0.6rem;\n    padding: 1.25rem 1.4rem; max-width: 40rem; margin: 0 0 1rem;\n  }\n  .confirm-card .actions { display: flex; gap: 0.9rem; align-items: center; margin-top: 1rem; }\n  .confirm-card .confirm-note { margin: 0.8rem 0 0; }\n   \n   \n  .archwrap { overflow-x: auto; margin: 0.2rem 0 1.6rem; }\n   \n  h4.archband {\n    border-top: 1px solid var(--rule); color: var(--muted);\n    font: 600 0.8rem var(--mono); letter-spacing: 0.02em;\n    margin: 1.5rem 0 0; padding-top: 0.5rem;\n  }\n   \n  .archsvg { display: block; }\n  .archsvg rect { fill: var(--accent-soft); stroke: var(--rule); }\n  .archsvg .archnode-box rect { stroke: var(--accent); }\n  .archsvg .archtitle { fill: var(--ink); font: 600 13px var(--sans); }\n  .archsvg .archsub { fill: var(--muted); font: 400 11px var(--sans); }\n   \n  .archsvg .archedge { fill: none; stroke: var(--muted); stroke-width: 1.5; }\n   \n  .archsvg .archalarm { fill: var(--good); }\n  .archsvg .archalarm-firing { fill: var(--bad); }\n   \n  .archbox {\n    border: 1px solid var(--rule); border-radius: 0.6rem;\n    padding: 0.9rem 1.1rem; margin: 0 0 0.6rem; max-width: 64rem;\n  }\n  .archbox h3 { margin: 0 0 0.3rem; }\n  .archbox p { margin: 0 0 0.2rem; }\n   \n  .archbox .fact + .fact::before { content: \" · \"; color: var(--muted); }\n  footer {\n    max-width: 64rem; margin: 0 auto; padding: 1.25rem 0 2rem;\n    border-top: 1px solid var(--rule);\n    font-size: 0.85rem; color: var(--muted);\n  }\n   \n  .pagehead { margin: 0 0 1.75rem; }\n  .eyebrow {\n    font-family: var(--mono); font-size: 0.75rem; letter-spacing: 0.08em;\n    text-transform: uppercase; color: var(--muted); margin: 0 0 0.3rem;\n  }\n  .pagehead h1 { margin: 0 0 0.35rem; }\n  .pagehead .lede { margin: 0; font-size: 1.05rem; }\n   \n  .reveal > summary { font-size: 1.1rem; font-weight: 600; cursor: pointer; }\n   \n  .tabs {\n    display: flex; flex-wrap: wrap; gap: 1.25rem; margin: 0 0 2rem;\n    border-bottom: 1px solid var(--rule);\n    font-family: var(--mono); font-size: 0.85rem;\n  }\n  .tabs a {\n    color: var(--muted); text-decoration: none;\n    padding: 0 0 0.55rem; border-bottom: 2px solid transparent; margin-bottom: -1px;\n  }\n  .tabs a:hover { color: var(--ink); }\n  .tabs a[aria-current] { color: var(--ink); border-bottom-color: var(--accent); }\n  header nav a[aria-current] { color: var(--ink); font-weight: 600; text-decoration: none; }\n  header nav a.addr { color: var(--muted); text-decoration: none; }\n  header nav a.addr:hover { text-decoration: underline; }\n</style>\n</head>\n<body>\n<header>\n  \n  <div class=\"brand\"><a href=\"/\">CARLOS console</a></div>\n  \n  \n</header>\n<main>\n\n<div class=\"pagehead\">\n  <p class=\"eyebrow\">🤖 CARLOS console</p>\n  <h1>Ship it. Promote it. Sleep<span style=\"color: var(--accent)\">.</span></h1>\n  <p class=\"lede\">Your apps, their live routes, and their release channels —\n  signed releases in, immutable bytes out.</p>\n</div>\n\n<form class=\"signin\" method=\"get\" action=\"/auth/signin\">\n  <button type=\"submit\">Sign in with Keymail</button>\n</form>\n<p class=\"small\">One button, whether it's your first time or your five-hundredth.\nFirst visits approve the console once in your Keymail — that single approval\nis the whole sign-up. After that, the same button is a one-click sign-in.\nNo passwords, ever; revoke the console from Keymail's connected apps at any\ntime.</p>\n\n</main>\n<footer>🤖 Console text written by an LLM, per <a href=\"https://11factor.org/#x\">factor X</a>.</footer>\n<script>\n\n\n\n\n(function () {\n  var d = document.getElementById(\"switcher\");\n  if (!d) return;\n  document.addEventListener(\"keydown\", function (e) {\n    if (e.key === \"Escape\" && d.open) { d.open = false; d.firstElementChild.focus(); }\n  });\n  document.addEventListener(\"click\", function (e) {\n    if (d.open && !d.contains(e.target)) d.open = false;\n  });\n})();\n</script>\n</body>\n</html>\n", want 200 kass up
2026/10/13 18:44:34 activator: one.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/13 18:44:34 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: swap.example.com hibernated
2026/10/13 18:44:34 activator: swap.example.com hibernated
2026/10/13 18:44:34 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/13 18:44:34 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: one.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:34 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:44:34 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:34 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: worker bot.x.internal NOT respawned (exited): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/13 18:44:34 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:44:34 activator: bot.x.internal: v2 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:34 activator: worker bot.x.internal NOT respawned (attempt (release)): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/13 18:44:34 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:44:34 activator: link.example.com hibernated
2026/10/13 18:44:51 activator: tenant.test: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:44:51 activator: tenant.test not started: the executable it hashed (/tmp/rl3681561758/app) is build "v1", but the start names build "v2" (the live link moved between the two reads); the next wake reads it again
2026/10/13 18:44:51 activator: sidecar staged for awake tenant.test — adopting
2026/10/13 18:44:51 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/13 18:44:51 activator: sidecar staged for awake tenant.test — adopting
2026/10/13 18:44:51 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/13 18:44:52 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:52 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:44:52 activator: worker bot.x.internal NOT respawned (exited): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/13 18:44:52 activator: bot.x.internal hibernated
2026/10/13 18:44:52 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:44:52 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:44:52 activator: worker bot.x.internal NOT respawned (hit its max_run): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/13 18:44:52 activator: bot.x.internal hibernated
2026/10/13 18:44:52 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:44:52 activator: worker bot.x.internal NOT respawned (converging onto a new build): instance config unavailable on this box: cannot open sealed config — standing down and releasing its lease
2026/10/13 18:44:52 activator: bot.x.internal hibernated
2026/10/13 18:44:52 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:52 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:44:52 activator: worker bot.x.internal respawned (exited), now running 
2026/10/13 18:44:52 activator: seeding unproven-seed.example.com: /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3246105719/002/live/unproven-seed.example.com seed --db /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3246105719/003/unproven-seed.example.com.db
2026/10/13 18:44:52 activator: unproven-seed.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3246105719/003/unproven-seed.example.com.db.seed-unproven refuses it as warm until they are
2026/10/13 18:44:52 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm457139795/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm457139795/003/obligation.example.com.db
2026/10/13 18:44:52 activator: obligation.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm457139795/003/obligation.example.com.db.seed-unproven refuses it as warm until they are
2026/10/13 18:44:52 activator: obligation.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/13 18:44:52 activator: no replica for obligation.example.com in s3://test-bucket/obligation.example.com — booting fresh
2026/10/13 18:44:52 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm457139795/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm457139795/003/obligation.example.com.db
2026/10/13 18:44:52 activator: seeded obligation.example.com
2026/10/13 18:44:52 activator: seeding helper-seed.example.com: /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt158049736/002/live/helper-seed.example.com seed --db /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt158049736/003/helper-seed.example.com.db
2026/10/13 18:44:52 activator: seeded helper-seed.example.com
2026/10/13 18:44:52 activator: seeding helper-unproven.example.com: /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven3749322535/002/live/helper-unproven.example.com seed --db /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven3749322535/003/helper-unproven.example.com.db
2026/10/13 18:44:52 activator: helper-unproven.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven3749322535/003/helper-unproven.example.com.db.seed-unproven refuses it as warm until they are
2026/10/13 18:44:52 activator: seeding write-ahead.example.com: /tmp/TestASeedRecordsItsObligationBeforeItRuns2705516293/002/live/write-ahead.example.com seed --db /tmp/TestASeedRecordsItsObligationBeforeItRuns2705516293/003/write-ahead.example.com.db
2026/10/13 18:44:52 activator: seeded write-ahead.example.com
2026/10/13 18:44:52 activator: write-ahead.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/13 18:44:52 activator: no replica for write-ahead.example.com in s3://test-bucket/write-ahead.example.com — booting fresh
2026/10/13 18:44:52 activator: seeding no-room.example.com: /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing2761295286/002/live/no-room.example.com seed --db /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing2761295286/003/no-room.example.com.db
2026/10/13 18:44:52 activator: seeding stuck-wal.example.com: /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation2303491881/002/live/stuck-wal.example.com seed --db /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation2303491881/003/stuck-wal.example.com.db
2026/10/13 18:44:52 activator: seeding preexisting.example.com: /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2173460410/002/live/preexisting.example.com seed --db /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2173460410/003/preexisting.example.com.db
2026/10/13 18:44:52 activator: preexisting.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept2173460410/003/preexisting.example.com.db.seed-unproven refuses it as warm until they are
2026/10/13 18:44:52 activator: preexisting.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/13 18:44:52 activator: old-kernel-seed.example.com: this kernel has no cgroup.kill; the seed's leader's exit is the only proof there is
2026/10/13 18:44:52 activator: seeding failed-role-seed.example.com: /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB2434439967/002/live/failed-role-seed.example.com seed --db /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB2434439967/003/failed-role-seed.example.com.db
2026/10/13 18:44:52 activator: seeding discharge-fails.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai1329491394/002/live/discharge-fails.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai1329491394/003/discharge-fails.example.com.db
2026/10/13 18:44:52 activator: discharge-fails.example.com: the seed's obligation could not be discharged and its processes are not proven gone; /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai1329491394/003/discharge-fails.example.com.db.seed-unproven stays
2026/10/13 18:44:52 activator: seeding discharge-fails-proven.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur160388195/002/live/discharge-fails-proven.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur160388195/003/discharge-fails-proven.example.com.db
2026/10/13 18:44:52 activator: one.example.com serving v1 but v2 is adopted — cycling onto the new build
2026/10/13 18:44:52 activator: one.example.com hibernated
2026/10/13 18:44:52 activator: one.example.com hibernated
2026/10/13 18:44:52 activator: one.example.com hibernated
2026/10/13 18:44:53 activator: one.example.com hibernated
2026/10/13 18:44:53 activator: one.example.com hibernated
2026/10/13 18:44:53 activator: one.example.com: sidecar starting on the instance's own build v1 (/opt/carlos/versions/v1/app): this box will not start v2, the build its live link names
2026/10/13 18:44:53 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/13 18:44:53 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, nor the instance's own build v1: a failure record holds this build: v1
2026/10/13 18:44:53 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/13 18:44:53 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, and the instance's own build v1 cannot be named: staged copy unreadable
2026/10/13 18:44:53 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/13 18:44:53 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_B,_l3009872255/003/g/dca298090dc0.135.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:53 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall2096994230/003/g/dca298090dc0.136.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:53 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall2096994230/003/g/dca298090dc0.137.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:53 activator: swap.example.com hibernated
2026/10/13 18:44:53 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatholds_pinne578726437/007/g/dca298090dc0.139.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:53 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/13 18:44:53 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatoverlap_succ2906478269/007/g/dca298090dc0.141.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:53 activator: h.example.com: started carlos-instance-gen@h.example.com:143.service memory.max=536870912
2026/10/13 18:44:53 activator: swap.example.com hibernated
2026/10/13 18:44:53 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/13 18:44:54 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_super3193779310/007/g/dca298090dc0.150.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:54 activator: swap.example.com hibernated
2026/10/13 18:44:54 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartholds_pinned_build,_lea257993324/007/g/dca298090dc0.152.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:54 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/13 18:44:54 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_a_fai2667501907/007/g/dca298090dc0.154.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:54 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawngated_build_is_the_ex2533132336/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:54 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawna_held_build_starts_n3492199604/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:54 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawnno_gate_is_todays_sp1772711115/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:44:54 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/13 18:44:54 activator: worker bot.x.internal serving v1 but live symlink is now v3 — respawning onto the new build
2026/10/13 18:44:54 activator: worker bot.x.internal respawned (converging onto a new build), now running v3
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it recycles its captured build v1
2026/10/13 18:44:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it stays down until a release attempt or a wake starts it
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:44:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:44:55 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: the running child keeps running
2026/10/13 18:44:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:44:55 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:44:55 activator: one.example.com: live/one.example.com names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/13 18:44:55 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/13 18:44:55 activator: one.example.com hibernated
2026/10/13 18:44:55 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/13 18:44:55 activator: fence.example hibernated
2026/10/13 18:44:55 activator: fence.example hibernated
2026/10/13 18:44:55 activator: fence.example hibernated
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 156 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 156 is not proven stopped; keeping the lease until it is
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 157 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 157 is not proven stopped; keeping the lease until it is
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 158 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 158 is not proven stopped; keeping the lease until it is
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 159 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:44:56 activator: f6-layout.example.com: failed wake's generation 159 is not proven stopped; keeping the lease until it is
2026/10/13 18:44:56 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink2648298750/003/g/c.x.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink2648298750/003/g/c.x.sock: no such file or directory)
2026/10/13 18:44:56 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink2648298750/003/g/c.xx.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink2648298750/003/g/c.xx.sock: no such file or directory)
2026/10/13 18:44:58 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:44:58 activator: one.example.com hibernated
2026/10/13 18:44:58 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished3967646874/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished3967646874/001/zdd: not a directory; the sweep retries
2026/10/13 18:44:58 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished3967646874/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished3967646874/001/zdd: not a directory; the sweep retries
2026/10/13 18:44:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake550362615/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake550362615/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake550362615/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake550362615/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor1607866586/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor1607866586/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:58 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor1607866586/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor1607866586/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:59 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake2804443547/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake2804443547/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:59 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake2804443547/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake2804443547/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:59 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake3565993057/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake3565993057/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:59 activator: x.example: hold's pinned wake refused or abandoned: x.example: refusing to wake: staged build not published before its exec
2026/10/13 18:44:59 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake3565993057/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake3565993057/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:59 activator: one.example.com hibernated
2026/10/13 18:44:59 activator: publish generations to /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile34716002/002/not-a-dir/zdd/generations.json: open directory /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile34716002/002/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:44:59 activator: ident.example.com hibernated
2026/10/13 18:44:59 activator: instance ident.example.com exited unexpectedly — recycling
2026/10/13 18:44:59 activator: ident.example.com hibernated
2026/10/13 18:44:59 activator: due.example.com hibernated
2026/10/13 18:44:59 activator: due.example.com hibernated
2026/10/13 18:44:59 activator: ret.example.com hibernated
2026/10/13 18:44:59 activator: ret.example.com hibernated
2026/10/13 18:44:59 activator: one.example.com hibernated
2026/10/13 18:44:59 activator: purged 1 cached response(s) for flat.example.com on spawn
2026/10/13 18:45:00 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile2338405322/001/versions/vA/app seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile2338405322/002/seedexe.example.com.db
2026/10/13 18:45:00 activator: seeded seedexe.example.com
2026/10/13 18:45:00 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile2338405322/001/live/seedexe.example.com seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile2338405322/002/seedexe.example.com.db
2026/10/13 18:45:00 activator: seeded seedexe.example.com
2026/10/13 18:45:00 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:45:00 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912
2026/10/13 18:45:00 activator: seeding h.example.com: /tmp/TestNestedSeedExeNamesTheRecordedFile1137666709/001/live/h.example.com seed --db /tmp/nb3638113001/data/instances/h.example.com/h.example.com.db
2026/10/13 18:45:00 activator: seeded h.example.com
2026/10/13 18:45:00 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:45:00 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912
2026/10/13 18:45:00 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3664468460/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/13 18:45:00 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3664468460/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/13 18:45:00 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3664468460/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/13 18:45:00 activator: seeding seedpin.example.com: /tmp/TestADeclinedFlatSeedRunsThePinnedFile718680415/001/versions/v1/app seed --db /tmp/TestADeclinedFlatSeedRunsThePinnedFile718680415/002/seedpin.example.com.db
2026/10/13 18:45:00 activator: seeded seedpin.example.com
2026/10/13 18:45:00 activator: seeding h.example.com: /tmp/TestADeclinedNestedSeedRunsThePinnedFile3608268985/001/live/h.example.com seed --db /tmp/nb1544772827/data/instances/h.example.com/h.example.com.db
2026/10/13 18:45:00 activator: seeded h.example.com
2026/10/13 18:45:00 activator: seeding tier.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed314984592/002/versions/vA/app seed --db /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed314984592/003/tier.example.com.db
2026/10/13 18:45:00 activator: tier.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed314984592/003/tier.example.com.db.seed-unproven refuses it as warm until they are
2026/10/13 18:45:00 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=1073741824
2026/10/13 18:45:01 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=1073741824
2026/10/13 18:45:01 activator: seeding h.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTiernested_seed4133211711/001/live/h.example.com seed --db /tmp/nb3275715575/data/instances/h.example.com/h.example.com.db
2026/10/13 18:45:01 activator: seeded h.example.com
2026/10/13 18:45:01 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsthe_configured_k666648039/001/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:01 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsan_own_key_pres1966314208/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/13 18:45:03 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestFirstStartWaitsForTheConfiguredKey3484660587/002/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:03 activator: control listener for shop.example.com: listen unix /tmp/TestFirstStartWaitsForTheConfiguredKey3484660587/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:05 activator: CARLOS_INSTANCE_KEY in /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptinto_the_own_file3533075239/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/13 18:45:05 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptdelivered_in_the_co3075965403/001/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:05 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration3511198075/001/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:05 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration3511198075/002/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:05 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyLeavesAFileWithTwoKeysAlone3639789251/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/13 18:45:19 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistse1033783924/002/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:21 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistsn3168195250/002/shop.example.com.env: the platform mints none while it is
2026/10/13 18:45:25 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance1085562006/001/live/seeded.example.com seed --db /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance1085562006/002/seeded.example.com.db
2026/10/13 18:45:25 activator: seeded seeded.example.com
2026/10/13 18:45:29 activator: restored sleepy.example.com from s3://test-bucket/sleepy.example.com
hello
world
before
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: stop fence.example: private backend detail
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: stop sidecar fence.example: private backend detail
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: final checkpoint fence.example: private backend detail
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: stop fence.example: activator: stop evidence unavailable
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: final checkpoint fence.example: activator: stop evidence unavailable
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: stop fence.example: stop refused
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: stop fence.example: earlier process may still run
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:31 activator: fence.example hibernated
2026/10/13 18:45:32 activator: waking.example hibernated
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: SIDECAR for partial-sidecar.example failed to start: sidecar started then failed — retaining returned process
2026/10/13 18:45:32 activator: partial-sidecar.example hibernated
2026/10/13 18:45:32 activator: SIDECAR restart for partial-sidecar.example failed: sidecar started then failed — retaining returned process
2026/10/13 18:45:32 activator: partial-sidecar.example hibernated
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: stop fence.example: stop refused
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: fence.example hibernated
2026/10/13 18:45:32 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/13 18:45:32 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/13 18:45:32 activator: barrier.example.com hibernated
2026/10/13 18:45:32 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/13 18:45:32 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/13 18:45:32 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/13 18:45:32 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/13 18:45:33 activator: barrier.example.com hibernated
2026/10/13 18:45:33 activator: barrier.example.com: generation 12's child is not proven empty: g12 still holds a process
2026/10/13 18:45:33 activator: generation 12 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/13 18:45:33 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/13 18:45:33 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/13 18:45:33 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/13 18:45:33 activator: barrier.example.com hibernated
2026/10/13 18:45:33 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/13 18:45:33 activator: barrier.example.com hibernated
2026/10/13 18:45:33 activator: stop generation 11: cgroup.kill: the child is not empty
2026/10/13 18:45:33 activator: generation 11 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/13 18:45:33 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/13 18:45:33 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/13 18:45:33 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/13 18:45:33 activator: barrier.example.com hibernated
2026/10/13 18:45:33 activator: waking.example.com hibernated
2026/10/13 18:45:33 activator: placed.example.com: kill generation 1225 after a failed health wait: g still holds a process
2026/10/13 18:45:33 activator: placed.example.com: generation 1225's child is not proven empty: g still holds a process
2026/10/13 18:45:33 activator: placed.example.com: failed wake's generation 1225 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:33 activator: placed.example.com: wake failed and its generation is not proven stopped: start placed.example.com: instance not healthy within 300ms: Get "http://carlos/": dial unix /tmp/g1171128819/g/74a2d464a30d.1225.sock: connect: no such file or directory; activator: generation start failed and its termination is unproven: kill generation 1225 after a failed health wait: g still holds a process
2026/10/13 18:45:33 activator: placed.example.com: failed wake's generation 1225 is proven stopped; lease given back
2026/10/13 18:45:34 activator: unproven.example.com: generation 1228's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1228 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:34 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/13 18:45:34 activator: unproven.example.com: generation 1228's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1228 is proven stopped; lease given back
2026/10/13 18:45:34 activator: unproven.example.com hibernated
2026/10/13 18:45:34 activator: unproven.example.com: generation 1230's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1230 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:34 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: h: start unit: instance control: generation termination unproven: the unit exited before it was observed
2026/10/13 18:45:34 activator: unproven.example.com: generation 1230's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1230 is proven stopped; lease given back
2026/10/13 18:45:34 activator: unproven.example.com hibernated
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1232 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:34 activator: unproven.example.com: retry stop of failed wake's generation 1232: cgroup.kill: the child is not empty
2026/10/13 18:45:34 activator: unproven.example.com: generation 1232's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1232 is proven stopped; lease given back
2026/10/13 18:45:34 activator: unproven.example.com hibernated
2026/10/13 18:45:34 activator: unproven.example.com: generation 1234's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1234 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:34 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/13 18:45:34 activator: unproven.example.com: generation 1234's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1234 is proven stopped; lease given back
2026/10/13 18:45:34 activator: unproven.example.com hibernated
2026/10/13 18:45:34 activator: unproven.example.com: generation 1236's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: failed wake's generation 1236 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:34 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/13 18:45:34 activator: unproven.example.com: generation 1236's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:34 activator: unproven.example.com: generation 1236's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:35 activator: unproven.example.com: failed wake's generation 1236 is proven stopped; lease given back
2026/10/13 18:45:35 activator: unproven.example.com: generation 1237's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:35 activator: unproven.example.com: failed wake's generation 1237 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:35 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/13 18:45:35 activator: unproven.example.com: generation 1237's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:35 activator: unproven.example.com: generation 1237's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:35 activator: unproven.example.com: failed wake's generation 1237 is proven stopped; lease given back
2026/10/13 18:45:35 activator: unproven.example.com: generation 1238's child is not proven empty: the generation's child still holds a process
2026/10/13 18:45:35 activator: unproven.example.com: failed wake's generation 1238 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:35 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/13 18:45:35 activator: fenced.example.com: failed wake's generation 1239 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:35 activator: fenced.example.com: failed wake's generation 1239 is proven stopped; lease given back
2026/10/13 18:45:35 activator: fenced.example.com: failed wake's generation 1240 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:35 activator: fenced.example.com: failed wake's generation 1240 is proven stopped; lease given back
2026/10/13 18:45:35 activator: forked.example.com: failed wake's generation 1241 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:35 activator: forked.example.com: failed wake's generation 1241 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:35 activator: forked.example.com: failed wake's generation 1241 is proven stopped; lease given back
2026/10/13 18:45:35 activator: f6.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:36 activator: f6.example.com: failed wake's generation 0 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:36 activator: f6.example.com: failed wake's generation 4 ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/13 18:45:36 activator: unhealthy-f6.example.com: failed wake's generation 1242 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:36 activator: unhealthy-f6.example.com: failed wake's generation 1242 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:36 activator: unhealthy-f6.example.com: failed wake's generation 1242 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:36 activator: unhealthy-f6.example.com: failed wake's generation 1242 is proven stopped; lease given back
2026/10/13 18:45:36 activator: exited-flat.example.com: failed wake's generation 1243 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:36 activator: exited-flat.example.com: failed wake's generation 1243 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:36 activator: exited-foreign.example.com: failed wake's generation 1244 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/13 18:45:36 activator: exited-foreign.example.com: failed wake's generation 1244 is not proven stopped; keeping the lease until it is
2026/10/13 18:45:37 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/13 18:45:37 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/13 18:45:37 activator: instance one.example.com exited unexpectedly — recycling
2026/10/13 18:45:37 activator: one.example.com hibernated
2026/10/13 18:45:38 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/13 18:45:38 activator: one.example.com hibernated
2026/10/13 18:45:38 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/13 18:45:38 activator: one.example.com hibernated
2026/10/13 18:45:38 activator: one.example.com idle 1h0m0s — hibernating
2026/10/13 18:45:38 activator: one.example.com hibernated
2026/10/13 18:45:38 activator: one.example.com idle 1h0m0s — hibernating
2026/10/13 18:45:38 activator: one.example.com hibernated
2026/10/13 18:45:38 activator: moved.example.com: promoting 2 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example hibernated
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:38 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:38 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example hibernated
2026/10/13 18:45:39 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/13 18:45:39 activator: x.example hibernated
2026/10/13 18:45:39 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/13 18:45:39 activator: x.example hibernated
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/13 18:45:39 activator: x.example hibernated
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example hibernated
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example hibernated
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/13 18:45:41 activator: replica h.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/13 18:45:41 activator: replica h.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/13 18:45:41 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/13 18:45:43 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_keyless_replica_is_refu1889998174/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:43 activator: replica shop-au.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/13 18:45:45 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_is_1843344608/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:47 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_pre-existing_env_key_th1087830159/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:49 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_dif2410165071/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:51 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_with_no_337094479/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:51 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY cannot be checked against its home's key (no entry for it in /etc/carlos/replica-home-keys; O8)
2026/10/13 18:45:53 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_home_updated,_this_re1393216237/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:53 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY is not the one the bucket's configuration now holds: a key change is in flight (ruling 27)
2026/10/13 18:45:55 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_bucket_cannot_be_read71846389/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:55 activator: replica shop-au.example.com not started: the bucket's configuration could not be read: bucket unreachable
2026/10/13 18:45:57 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketno_fresh_reader_refused2516239466/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:45:57 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY cannot be checked against the bucket's configuration
2026/10/13 18:45:59 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketresidual_a_change_only_o1082814879/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:46:01 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_config-only_homes_keyl4043284078/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:46:03 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_home_row_mints_and_its_4178288933/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:46:07 activator: seeding shop-au.example.com: /tmp/TestReplicaSeedIsGatedthe_homes_key3654129565/001/live/shop-au.example.com seed --db /tmp/TestReplicaSeedIsGatedthe_homes_key3654129565/002/shop-au.example.com.db
2026/10/13 18:46:07 activator: seeded shop-au.example.com
2026/10/13 18:46:11 activator: replica shop-au.example.com not started: where its CARLOS_INSTANCE_KEY came from is unknown
2026/10/13 18:46:17 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaGateDecidesOnTheLaunchesOwnRead2484252515/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:46:20 activator: swap.example.com hibernated
2026/10/13 18:46:20 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/13 18:46:20 activator: swap.example.com hibernated
2026/10/13 18:46:20 activator: LEASE LOST for swap.example.com (renewal failed) — standing down
2026/10/13 18:46:20 activator: swap.example.com hibernated
2026/10/13 18:46:20 activator: swap.example.com hibernated
2026/10/13 18:46:20 activator: swap.example.com is no longer hibernate-managed — releasing
2026/10/13 18:46:20 activator: swap.example.com hibernated
2026/10/13 18:46:20 activator: swap.example.com hibernated
2026/10/13 18:46:20 activator: stop missed-join.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/13 18:46:20 activator: missed-join.example.com hibernated
2026/10/13 18:46:20 activator: stop nested-stuck.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/13 18:46:20 activator: nested-stuck.example.com: the sidecar a restart brought back for a stopping entry is not proven gone (activator: stop unproven; processes remain); keeping the lease until it has exited
2026/10/13 18:46:21 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/13 18:46:21 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/13 18:46:21 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/13 18:46:21 activator: nested-stuck.example.com hibernated
2026/10/13 18:46:21 activator: stop sidecar nested-costop.example.com: activator: stop unproven; processes remain
2026/10/13 18:46:21 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/13 18:46:21 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/13 18:46:21 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/13 18:46:21 activator: nested-costop.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: cgroup: the host's parent cgroup holds a process directly: holds 1 process(es)
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a cgroup that took cgroup.kill could not be made fresh before a spawn: carlos-exec@h.example.com: device or resource busy
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a generation's limits could not be applied: h.example.com: the placement shim is not available
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:46:21 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:46:21 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:46:21 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/13 18:46:21 activator: bot.x.internal hibernated
2026/10/13 18:46:21 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:46:21 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory
2026/10/13 18:46:21 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:46:21 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/13 18:46:21 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:46:21 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:46:21 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory
2026/10/13 18:46:21 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:46:21 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_3043273682/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/13 18:46:21 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:46:21 activator: h.example.com: kill carlos-instance-gen@h.example.com:1348.service after a failed start: stop timed out
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:1356.service exited before it was observed
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: h.example.com: kill carlos-instance-gen@h.example.com:1358.service after a failed start: stop timed out
2026/10/13 18:46:21 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:1358.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: h.example.com: kill carlos-instance-gen@h.example.com:1362.service after a failed start: stop timed out
2026/10/13 18:46:21 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: the unit failed to start; activator: generation start failed and its termination is unproven: stop timed out
2026/10/13 18:46:21 activator: swap.example.com hibernated
2026/10/13 18:46:21 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/13 18:46:21 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:46:22 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:46:22 activator: worker bot.x.internal: the attempt's respawn did not finish within 150ms of its caller leaving; reporting its build as possibly executed
2026/10/13 18:46:22 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: a stop-evidence operation holds the host
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: socket dir: permission denied
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: start g: activator: generation start failed and its termination is unproven
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: LEASE HELD for swap.example.com by "box-b" — a previous incarnation hasn't released; refusing to wake (retry after 1m0s)
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: instance lease held by "box-b" (retry in 1m0s)
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: restore swap.example.com: replica unreadable
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: staging: staged digest mismatch: have be0041e1a437dbea5b67dfe58bad04a0fc30d5036d469429f3534f39ec53d8b5, want c3bae8fb3a8bf395783d7f7f6ffcd4aadc797c95ce24b7e8e2a98af6b63d8f6d
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/13 18:46:22 activator: swap.example.com hibernated
2026/10/13 18:46:22 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: the backend ran a different generation from the hold's choice: generation 1391 of v9 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_3014797268/004/versions/v2/app"; the hold chose v2 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_3014797268/004/versions/v2/app"
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com: generation 1398's child is not proven empty: the child still holds a process
2026/10/13 18:46:23 activator: swap.example.com: failed wake's generation 1398 is not proven stopped; keeping the lease until it is
2026/10/13 18:46:23 activator: swap.example.com: failed wake's generation 1398 is proven stopped; lease given back
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com: failed wake's generation 1400 is not proven stopped; keeping the lease until it is
2026/10/13 18:46:23 activator: swap.example.com: failed wake's generation 1400 is proven stopped; lease given back
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3047469485/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3047469485/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:46:23 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3047469485/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing3047469485/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/13 18:46:23 activator: h.example.com: started carlos-instance-gen@h.example.com:7.service memory.max=1073741824
2026/10/13 18:46:23 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: processes remain
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/13 18:46:23 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:23 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/13 18:46:23 activator: swap.example.com hibernated
2026/10/13 18:46:24 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/13 18:46:24 activator: swap.example.com hibernated
2026/10/13 18:46:24 activator: swap.example.com hibernated
2026/10/13 18:46:24 activator: swap.example.com hibernated
2026/10/13 18:46:25 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/13 18:46:25 activator: swap.example.com hibernated
2026/10/13 18:46:25 activator: swap.example.com hibernated
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 4s)
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: swap.example.com hibernated
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: swap.example.com hibernated
2026/10/13 18:46:25 activator: swap.example.com hibernated
2026/10/13 18:46:25 activator: swap.example.com: a discarded SIDECAR is not proven gone (activator: stop unproven; processes remain); keeping it tracked
2026/10/13 18:46:25 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:25 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:25 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:25 activator: SIDECAR for swap.example.com is not proven gone for its replacement; tracking it (the sweep re-proves it)
2026/10/13 18:46:25 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/13 18:46:25 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:25 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/13 18:46:25 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:25 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:25 activator: swap.example.com hibernated
2026/10/13 18:46:26 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:26 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:26 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:26 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/13 18:46:26 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:26 activator: swap.example.com hibernated
2026/10/13 18:46:26 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:26 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:26 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/13 18:46:26 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/13 18:46:26 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 8s)
2026/10/13 18:46:26 activator: sidecar for swap.example.com was down — restarted (backoff now 16s)
2026/10/13 18:46:26 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:46:26 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:46:32 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:32 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: instance control /stop: 502: stop: processes remain
2026/10/13 18:46:32 activator: h.example.com: unit restarted outside the activator (invocation someone-else, ours inv7); stopping it
2026/10/13 18:46:32 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:32 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:32 activator: h.example.com: processes remain after failed; not releasing
2026/10/13 18:46:33 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:33 activator: h.example.com: unit restarted outside the activator (invocation inv2, ours inv1); stopping it
2026/10/13 18:46:33 activator: seeding h.example.com: /tmp/TestSeedUnitCleansUpOnlyItsOwnDBOnFailure1245886/001/live/h.example.com seed --fast --db /tmp/nb2485948109/data/instances/h.example.com/h.example.com.db
2026/10/13 18:46:33 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:33 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:33 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:46:33 activator: seeding h.example.com: /tmp/TestSeedUnitFailureStopsBeforeCleanupAndKeepsTheDBWhenProcessesR3201482150/001/live/h.example.com seed --fast --db /tmp/nb2670581510/data/instances/h.example.com/h.example.com.db
2026/10/13 18:46:53 activator: seeding h.example.com: /tmp/TestANestedSeedRecordsItsObligationBeforeItRuns2951185448/001/live/h.example.com seed --db /tmp/nb38749598/data/instances/h.example.com/h.example.com.db
2026/10/13 18:46:53 activator: seeded h.example.com
2026/10/13 18:46:53 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI3785547654/001/live/h.example.com seed --db /tmp/nb1423004784/data/instances/h.example.com/h.example.com.db
2026/10/13 18:46:53 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/13 18:46:53 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/13 18:46:53 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI187158737/001/live/h.example.com seed --db /tmp/nb3326703105/data/instances/h.example.com/h.example.com.db
2026/10/13 18:46:53 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/13 18:46:53 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/13 18:46:53 activator: fifo-obligation.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/13 18:46:53 activator: one.example.com: first wake of v2 had 30s of its health window left at the exec, under the 1h0m0s floor; booting the good build v1 without trying it
2026/10/13 18:46:53 activator: one.example.com hibernated
2026/10/13 18:46:54 activator: tenant.test not started: it is held on v1, and the launch hashed "v2" (the live link moved after the check); the next wake checks again
2026/10/13 18:46:54 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/13 18:46:54 activator: one.example.com hibernated
2026/10/13 18:46:54 activator: one.example.com not woken: it is held on v1, and this wake would run "v2" (the live link moved after the check); the next wake checks again
2026/10/13 18:46:54 activator: one.example.com not woken: it is held on v1, and this wake would run "" (the live link moved after the check); the next wake checks again
2026/10/13 18:46:54 activator: swap.example.com hibernated
2026/10/13 18:47:10 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: two.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/13 18:47:10 activator: seeding h.example.com: /tmp/TestANestedUnversionedLinkWithNoHookWakesAsToday2475305643/001/live/h.example.com seed --db /tmp/nb1545143618/data/instances/h.example.com/h.example.com.db
2026/10/13 18:47:10 activator: seeded h.example.com
2026/10/13 18:47:10 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/13 18:47:10 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: swap.example.com hibernated
2026/10/13 18:47:10 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/13 18:47:10 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: SIDECAR restart for one.example.com failed: the build state could not be written, so this build was not started: bucket refused the build state (next attempt in 2s)
2026/10/13 18:47:10 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/13 18:47:10 activator: SIDECAR for one.example.com failed to start: the build state could not be written, so this build was not started: bucket refused the build state — serving without it (the sweep retries)
2026/10/13 18:47:10 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/13 18:47:10 activator: sidecar staged for awake swap.example.com — adopting
2026/10/13 18:47:10 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/13 18:47:10 activator: SIDECAR restart for one.example.com failed: swap: lease lost (next attempt in 2s)
2026/10/13 18:47:10 activator: one.example.com: generation 2309's child is not proven empty: activator: stop unproven; processes remain
2026/10/13 18:47:10 activator: one.example.com: failed wake's generation 2309 is not proven stopped; keeping the lease until it is
2026/10/13 18:47:10 activator: one.example.com hibernated
2026/10/13 18:47:11 activator: one.example.com: first wake of v2 had 400ms of its health window left at the exec, under the 20s floor; booting the good build v1 without trying it
2026/10/13 18:47:27 activator: x.example: hold's pinned wake refused or abandoned: the claims admission refused before the exec: hostbox: unresolved lifecycle contract: isolated root launcher cannot carry claim reservations
2026/10/13 18:47:27 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866339050/003/g/f4705dce873e.2316.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:47:29 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866339050/003/g/f4705dce873e.2317.sock.control: bind: invalid argument (continuing without it)
2026/10/13 18:47:29 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866339050/003/g/f4705dce873e.2317.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat1866339050/003/g/f4705dce873e.2317.sock: no such file or directory)
2026/10/13 18:47:29 activator: real.example.com hibernated
2026/10/13 18:47:31 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign3766061762/004/hand-set.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign3766061762/004/hand-set.sock: no such file or directory)
2026/10/13 18:47:31 activator: real.example.com hibernated
2026/10/13 18:47:31 activator: bot.x.internal: worker generation 0's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:31 activator: bot.x.internal: worker generation 2323's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:31 activator: bot.x.internal: failed wake's generation 2323 is not proven stopped; keeping the lease until it is
2026/10/13 18:47:31 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd: not a directory; the sweep retries
2026/10/13 18:47:31 activator: worker bot.x.internal not spawned: the build it would exec (v2 /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/003/v2) could not be published first: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd: not a directory
2026/10/13 18:47:31 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd: not a directory; the sweep retries
2026/10/13 18:47:31 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing698180230/002/zdd: not a directory; the sweep retries
2026/10/13 18:47:31 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn1271946159/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn1271946159/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:31 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn1271946159/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn1271946159/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:31 activator: instance one.example.com exited unexpectedly — recycling
2026/10/13 18:47:31 activator: one.example.com hibernated
2026/10/13 18:47:32 activator: one.example.com hibernated
2026/10/13 18:47:34 activator: one.example.com: first wake of v2 had 499ms of its health window left at the exec, under the 1s floor; booting the good build v1 without trying it
2026/10/13 18:47:35 activator: h.example.com: started carlos-instance-gen@h.example.com:2341.service memory.max=536870912
2026/10/13 18:47:35 activator: one.example.com hibernated
2026/10/13 18:47:35 activator: h.example.com: kill carlos-instance-gen@h.example.com:2342.service after a failed start: stop timed out
2026/10/13 18:47:35 activator: one.example.com: wake failed and its generation is not proven stopped: start one.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2342.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/13 18:47:36 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/13 18:47:36 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/13 18:47:36 activator: worker bot.example.com respawned (exited), now running 
2026/10/13 18:47:36 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/13 18:47:36 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/13 18:47:36 activator: worker bot.example.com respawned (exited), now running 
2026/10/13 18:47:36 activator: bot.example.com: worker generation 2346's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:36 activator: stop bot.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/13 18:47:36 activator: bot.example.com hibernated
2026/10/13 18:47:36 activator: bot3.example.com: worker generation 2347's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:36 activator: worker bot3.example.com NOT respawned (test: a restart with no old handle): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/13 18:47:36 activator: bot4.example.com: worker generation 2349's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:36 activator: bot4.example.com: failed wake's generation 2349 is not proven stopped; keeping the lease until it is
2026/10/13 18:47:36 activator: bot4.example.com: failed wake's generation 2349 is proven stopped; lease given back
2026/10/13 18:47:36 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/13 18:47:36 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/13 18:47:36 activator: bot2.example.com: worker generation 2350's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:36 activator: worker bot2.example.com NOT respawned (exited): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/13 18:47:36 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/13 18:47:36 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/13 18:47:36 activator: worker bot2.example.com respawned (exited), now running 
2026/10/13 18:47:36 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/13 18:47:36 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/13 18:47:36 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:36 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:36 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:47:36 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:36 activator: bot.x.internal hibernated
2026/10/13 18:47:37 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: worker bot.example.com respawned (exited), now running v1
2026/10/13 18:47:37 activator: bot.example.com hibernated
2026/10/13 18:47:37 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:37 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory; the sweep retries
2026/10/13 18:47:37 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory; the sweep retries
2026/10/13 18:47:37 activator: worker first.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory
2026/10/13 18:47:37 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory; the sweep retries
2026/10/13 18:47:37 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory; the sweep retries
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory; the sweep retries
2026/10/13 18:47:37 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory
2026/10/13 18:47:37 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory; the sweep retries
2026/10/13 18:47:37 activator: WORKER bot.x.internal failed to respawn (exited): worker build not published before its exec: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished713347073/002/data/zdd: not a directory — the sweep's crash restart takes it from here
2026/10/13 18:47:37 activator: bot.example.com hibernated
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:37 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/13 18:47:37 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:37 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: WORKER bot.x.internal failed to respawn (exited): fork/exec: resource temporarily unavailable — the sweep's crash restart takes it from here
2026/10/13 18:47:37 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:37 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/13 18:47:37 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/13 18:47:37 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/13 18:47:37 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:47:37 activator: worker bot.x.internal hit its max_run; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:37 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/13 18:47:37 activator: worker bot.x.internal respawned (hit its max_run), now running v2
2026/10/13 18:47:38 activator: worker bot1.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot1.x.internal exited; it serves v1 but live/bot1.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/13 18:47:38 activator: worker bot2.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot2.x.internal exited; live/bot2.x.internal names v2, but this sweep's converge budget is spent, so it recycles its captured build v1 and converges on a later sweep
2026/10/13 18:47:38 activator: worker bot2.x.internal respawned (exited), now running v1
2026/10/13 18:47:38 activator: worker bot1.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:38 activator: worker bot2.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/13 18:47:38 activator: worker bot2.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot.x.internal exited; live/bot.x.internal names v2, but the release scheduler owns build changes, so it recycles its captured build v1
2026/10/13 18:47:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:38 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:38 activator: worker bot.x.internal exited; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/13 18:47:38 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/13 18:47:38 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/13 18:47:38 activator: worker bot10.example.com: the shed respawn child's exit is not proven (activator: stop unproven; processes remain); the entry keeps it and its build until the sweep proves it
2026/10/13 18:47:38 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/13 18:47:38 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/13 18:47:39 activator: bot10.example.com: the shed respawn child has exited
2026/10/13 18:47:39 activator: bot10.example.com hibernated
2026/10/13 18:47:39 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/13 18:47:39 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/13 18:47:39 activator: bot10.example.com respawn did not finish before its stop; stopping the rest, the lease is kept until it has
2026/10/13 18:47:39 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/13 18:47:39 activator: bot10.example.com hibernated
2026/10/13 18:47:39 activator: bot8.example.com: worker generation 2363's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:39 activator: bot8.example.com: failed wake's generation 2363 is not proven stopped; keeping the lease until it is
2026/10/13 18:47:39 activator: bot8.example.com: failed wake's generation 2363 is proven stopped; lease given back
2026/10/13 18:47:39 activator: worker bot9.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/13 18:47:39 activator: bot9.example.com: worker generation 2365's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:39 activator: generation 2365 of bot9.example.com did not prove its exit; keeping it in the operation set
2026/10/13 18:47:39 activator: LEASE LOST for bot9.example.com (renewal failed) — standing down
2026/10/13 18:47:39 activator: bot9.example.com hibernated
2026/10/13 18:47:39 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:39 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:39 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/13 18:47:39 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:39 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:39 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/13 18:47:40 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/13 18:47:40 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:40 activator: bot.x.internal hibernated
2026/10/13 18:47:40 activator: worker bot.x.internal respawned (attempt (restart)), now running v1
2026/10/13 18:47:40 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:47:40 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:40 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:41 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:41 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:47:41 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:47:41 activator: bot.x.internal: worker generation 2368's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:41 activator: worker bot.x.internal NOT respawned (attempt (release)): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/13 18:47:41 activator: bot.x.internal: worker generation 2368's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/13 18:47:41 activator: worker bot.x.internal NOT respawned (restoring v1): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/13 18:47:41 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: lease lost; abandoning it
2026/10/13 18:47:41 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:41 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory
2026/10/13 18:47:41 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:41 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/13 18:47:41 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:41 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:41 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory
2026/10/13 18:47:41 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory; the sweep retries
2026/10/13 18:47:41 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned3161122531/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:41 activator: bot.x.internal hibernated
2026/10/13 18:47:41 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:41 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/13 18:47:41 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:42 activator: worker bot.x.internal NOT respawned (falling back to v1): swap: lease lost; abandoning it
2026/10/13 18:47:42 activator: WORKER bot.x.internal failed to respawn (attempt (restart)): exec: text file busy — the sweep's crash restart takes it from here
2026/10/13 18:47:42 activator: WORKER bot.x.internal failed to respawn (restoring v1): exec: text file busy — the sweep's crash restart takes it from here
2026/10/13 18:47:42 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:42 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/13 18:47:42 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:47:42 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/13 18:47:42 activator: bot.x.internal hibernated
2026/10/13 18:47:42 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:47:42 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:47:42 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/13 18:47:42 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/13 18:47:42 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:42 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/13 18:47:42 activator: bot.x.internal hibernated
2026/10/13 18:47:43 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:43 activator: LEASE LOST for bot.x.internal (renewal failed) — standing down
2026/10/13 18:47:43 activator: bot.x.internal hibernated
2026/10/13 18:47:43 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:43 activator: bot.x.internal hibernated
2026/10/13 18:47:43 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/13 18:47:43 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/13 18:47:43 activator: bot.x.internal hibernated
FAIL
FAIL	github.com/carlosframework/platform/internal/activator	227.597s
ok  	github.com/carlosframework/platform/internal/agentsock	0.016s
ok  	github.com/carlosframework/platform/internal/alerting	0.006s
ok  	github.com/carlosframework/platform/internal/analytics	0.022s
ok  	github.com/carlosframework/platform/internal/appconfig	2.388s
ok  	github.com/carlosframework/platform/internal/apperr	0.074s
ok  	github.com/carlosframework/platform/internal/applog	0.141s
ok  	github.com/carlosframework/platform/internal/awsinv	0.004s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.006s
ok  	github.com/carlosframework/platform/internal/boxid	0.534s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.011s
ok  	github.com/carlosframework/platform/internal/caps	0.013s
ok  	github.com/carlosframework/platform/internal/certwarm	0.002s
ok  	github.com/carlosframework/platform/internal/cgroup	0.695s
ok  	github.com/carlosframework/platform/internal/changelog	0.321s
ok  	github.com/carlosframework/platform/internal/clamd	0.055s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	368.520s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.283s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.009s
ok  	github.com/carlosframework/platform/internal/dbverify	0.004s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.003s
ok  	github.com/carlosframework/platform/internal/docsite	0.005s
ok  	github.com/carlosframework/platform/internal/domains	2.185s
ok  	github.com/carlosframework/platform/internal/doorkey	0.006s
ok  	github.com/carlosframework/platform/internal/economics	0.004s
ok  	github.com/carlosframework/platform/internal/edge	27.031s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.008s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.161s
ok  	github.com/carlosframework/platform/internal/gate	1.656s
ok  	github.com/carlosframework/platform/internal/genselect	0.015s
ok  	github.com/carlosframework/platform/internal/geo	0.017s
ok  	github.com/carlosframework/platform/internal/geoip	0.029s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.093s
ok  	github.com/carlosframework/platform/internal/host	108.550s
ok  	github.com/carlosframework/platform/internal/hostactors	0.550s
ok  	github.com/carlosframework/platform/internal/hostbox	2.187s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.504s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.012s
ok  	github.com/carlosframework/platform/internal/hostexec	6.178s
ok  	github.com/carlosframework/platform/internal/hostfence	0.041s
ok  	github.com/carlosframework/platform/internal/hostfiles	11.690s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.012s
ok  	github.com/carlosframework/platform/internal/hostguard	1.016s
ok  	github.com/carlosframework/platform/internal/hostops	1.489s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.013s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.013s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.463s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.136s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.012s
ok  	github.com/carlosframework/platform/internal/infraview	0.014s
ok  	github.com/carlosframework/platform/internal/instancectl	1.829s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.709s
ok  	github.com/carlosframework/platform/internal/instancefs	0.018s
ok  	github.com/carlosframework/platform/internal/instancekey	0.015s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.018s
ok  	github.com/carlosframework/platform/internal/layout	1.437s
ok  	github.com/carlosframework/platform/internal/lease	9.603s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.006s
ok  	github.com/carlosframework/platform/internal/ledger	0.016s
ok  	github.com/carlosframework/platform/internal/mail	0.163s
ok  	github.com/carlosframework/platform/internal/maintenance	0.082s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.634s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.067s
ok  	github.com/carlosframework/platform/internal/metrics	0.006s
ok  	github.com/carlosframework/platform/internal/moves	22.306s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.795s
ok  	github.com/carlosframework/platform/internal/objstore	0.170s
ok  	github.com/carlosframework/platform/internal/ops	0.180s
ok  	github.com/carlosframework/platform/internal/park	0.222s
ok  	github.com/carlosframework/platform/internal/privatefile	0.004s
ok  	github.com/carlosframework/platform/internal/provision	18.436s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.007s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	4.354s
ok  	github.com/carlosframework/platform/internal/relay	7.523s
ok  	github.com/carlosframework/platform/internal/release	24.713s
ok  	github.com/carlosframework/platform/internal/rollout	4.088s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.003s
ok  	github.com/carlosframework/platform/internal/rootaws	0.009s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.007s
ok  	github.com/carlosframework/platform/internal/rootstate	0.025s
ok  	github.com/carlosframework/platform/internal/s3compat	0.004s
ok  	github.com/carlosframework/platform/internal/safefile	0.003s
ok  	github.com/carlosframework/platform/internal/schedule	2.365s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	19.306s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	4.482s
ok  	github.com/carlosframework/platform/internal/status	0.037s
ok  	github.com/carlosframework/platform/internal/steering	0.006s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.196s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.007s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.002s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.004s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.559s
FAIL
make: *** [Makefile:18: test] Error 1

amadan: step gate failed: exit status 2

amadan: job failed at step gate — its output is above, not at the tail of this log