carlos / platform Public

Claim authority census: the mail provisioner call in serve

serve now calls mail.NewAWSProvisioner(ctx, region, mailBoundary) inside
a closure instead of handing the function value to mailBackends, so the
census counts a new call site (151 to 152). There was no old entry to
replace: a function value is not a counted call. Noted in
census-proposal-sec-mail-user-boundary.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 20 hours ago 1a0f725436ea087c5e2c472de52891566cb58402 parent 626ed6b

2 files changed, +15 −0

docs/superpowers/notes/census-proposal-sec-mail-user-boundary.md +14 −0
internal/console/testdata/claim-authority-sites.txt +1 −0

CI failed — run details

CI log

=== gate ===
.amadan/test-gate.sh
=== launchctx (0 s)
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
=== vet (18 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (169 s)
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.757s
ok  	github.com/carlosframework/platform/cmd/moveprobe	11.959s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.788s
2026/10/24 17:23:19 activator: badseed.example.com: seed for badseed.example.com: activator: seed unproven; processes remain: processes remain after a successful seed: still running; leaving the lease to its TTL
2026/10/24 17:23:19 activator: unproven.example.com: unproven.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone; leaving the lease to its TTL
2026/10/24 17:23:19 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/24 17:23:19 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3028411059/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3028411059/004/next-wake.example.com.db
2026/10/24 17:23:19 activator: next-wake.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3028411059/004/next-wake.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 17:23:19 activator: next-wake.example.com: seeding next-wake.example.com: exit status 1: activator: seed unproven; processes remain (the seed's processes remain after stopping it; its database was left in place: cgroup: strict containment unsupported); leaving the lease to its TTL
2026/10/24 17:23:19 activator: next-wake.example.com: next-wake.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup: strict containment unsupported; leaving the lease to its TTL
2026/10/24 17:23:19 activator: next-wake.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 17:23:19 activator: no replica for next-wake.example.com in s3://test-bucket/next-wake.example.com — booting fresh
2026/10/24 17:23:19 activator: seeding next-wake.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3028411059/003/live/next-wake.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedWaitsForItsProof3028411059/004/next-wake.example.com.db
2026/10/24 17:23:19 activator: seeded next-wake.example.com
2026/10/24 17:23:19 activator: instance one.example.com exited unexpectedly — recycling
2026/10/24 17:23:19 activator: one.example.com hibernated
2026/10/24 17:23:19 activator: LEASE HELD for held.example.com by "old-incarnation" — a previous incarnation hasn't released; refusing to wake (retry after 7s)
2026/10/24 17:23:19 activator: one.example.com hibernated
2026/10/24 17:23:19 activator: two.example.com hibernated
2026/10/24 17:23:19 activator: b.example.com hibernated
2026/10/24 17:23:19 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:23:19 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:23:20 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 17:23:20 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:23:20 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 17:23:20 activator: bot.x.internal hibernated
2026/10/24 17:23:20 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendStartMergesConfigEnvUnderOwnEnv4212583397/002/merge-precedence.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/24 17:23:20 activator: purged 1 cached response(s) for deploy.example.com on spawn
2026/10/24 17:23:20 activator: restored restored.example.com from s3://test-bucket/restored.example.com
2026/10/24 17:23:20 activator: damaged.example.com restored CORRUPT from s3://test-bucket/damaged.example.com: integrity_check: file is not a database (26) — quarantined at /tmp/TestExecBackendRestoreCorruptReplicaFailsAndQuarantines2921590190/002/damaged.example.com.db.corrupt, instance NOT started
2026/10/24 17:23:20 activator: restored damaged.example.com from s3://test-bucket/damaged.example.com
2026/10/24 17:23:20 activator: no replica for brand-new.example.com in s3://test-bucket/brand-new.example.com — booting fresh
2026/10/24 17:23:20 activator: CARLOS_ADMIN_TOKEN in /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath3272322924/002/seeded.example.com.env is not the shape the platform mints (64 hex characters) — left in place, because the running instance holds this same value and rotating it would lock the platform out of a live instance; if it is corrupt, stop the instance, delete the line and restart it to have one minted
2026/10/24 17:23:20 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath3272322924/001/live/seeded.example.com seed --fixtures --db /tmp/TestExecBackendSeedRunsDeclaredArgvWithDBPath3272322924/002/seeded.example.com.db
2026/10/24 17:23:20 activator: seeded seeded.example.com
2026/10/24 17:23:20 activator: seeding custom-db.example.com: /tmp/TestExecBackendSeedHonoursRouteDBPath663619259/001/live/custom-db.example.com seed --db /tmp/TestExecBackendSeedHonoursRouteDBPath663619259/004/elsewhere.db
2026/10/24 17:23:20 activator: seeded custom-db.example.com
2026/10/24 17:23:20 activator: seeding badseed.example.com: /tmp/TestExecBackendSeedNonZeroExitFails274781271/001/live/badseed.example.com seed --db /tmp/TestExecBackendSeedNonZeroExitFails274781271/002/badseed.example.com.db
schema is corrupt
2026/10/24 17:23:20 activator: badseed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 17:23:20 activator: seeding hangs.example.com: /tmp/TestExecBackendSeedIsTimeBounded3070741715/001/live/hangs.example.com seed --db /tmp/TestExecBackendSeedIsTimeBounded3070741715/002/hangs.example.com.db
2026/10/24 17:23:21 activator: hangs.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 17:23:21 activator: seeding crashed.example.com: /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB566630446/001/live/crashed.example.com seed --db /tmp/TestExecBackendSeedFailureRemovesHalfBuiltDB566630446/002/crashed.example.com.db
died mid-schema
2026/10/24 17:23:21 activator: crashed.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 17:23:21 activator: seeding existing-db.example.com: /tmp/TestExecBackendSeedFailureKeepsPreexistingDB949226359/001/live/existing-db.example.com seed --db /tmp/TestExecBackendSeedFailureKeepsPreexistingDB949226359/002/existing-db.example.com.db
nope
2026/10/24 17:23:21 activator: existing-db.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 17:23:21 activator: seeding leaky-seed.example.com: /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild3367502845/001/live/leaky-seed.example.com seed --db /tmp/TestExecBackendSeedSucceedsDespiteLingeringChild3367502845/002/leaky-seed.example.com.db
2026/10/24 17:23:21 activator: seeded leaky-seed.example.com (it exited 0 but left a child holding its output pipe open for more than 200ms)
2026/10/24 17:23:22 activator: sidecar for a.example.com was down — restarted (backoff now 2s)
2026/10/24 17:23:22 activator: h.example.com: route refreshed: db "/tmp/h.example.com.db" -> "/data/instances/h.example.com/h.example.com.db" socket "/tmp/nope.sock" -> "/run/carlos/instances/h.example.com/app.sock"
2026/10/24 17:23:22 activator: stop stale.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/24 17:23:22 activator: stale.example.com hibernated
2026/10/24 17:23:22 activator: leftover.example.com: the host cgroup holds processes not proven gone: a leftover still populates the host cgroup; leaving the lease to its TTL
2026/10/24 17:23:22 activator: old-kernel.example.com: this kernel has no cgroup.kill, so the host cgroup cannot be killed; its leader's exit is the only proof there is
2026/10/24 17:23:22 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/24 17:23:22 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/24 17:23:22 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/24 17:23:22 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/24 17:23:22 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/24 17:23:22 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/24 17:23:22 activator: moved.example.com: promoting 1 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/24 17:23:22 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/24 17:23:22 activator: retry.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 17:23:22 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/24 17:23:22 activator: retry.example.com: strict group not closed: cgroup: strict containment refused
2026/10/24 17:23:23 activator: retry.example.com: strict group cleaned up on retry 1
2026/10/24 17:23:23 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 17:23:23 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 17:23:23 activator: stuck.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 17:23:23 activator: stuck.example.com: strict group still not cleaned up after 2 retries; spawns of this host stay refused until the agent restarts
2026/10/24 17:23:23 activator: owned.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 17:23:24 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/24 17:23:24 activator: owned.example.com: strict group not closed: cgroup: strict containment refused
2026/10/24 17:23:24 activator: owned.example.com: strict group cleaned up on retry 1
2026/10/24 17:23:24 activator: race.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 17:23:24 activator: transition.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 17:23:24 activator: transition2.example.com: strict group did not drain: cgroup: strict containment requires a live deadline
2026/10/24 17:23:25 activator: no replica for moved.example.com in s3://test-bucket/moved.example.com — booting fresh
2026/10/24 17:23:25 activator: restored moved.example.com from s3://test-bucket/moved.example.com
2026/10/24 17:23:25 activator: kass.example.com hibernated
2026/10/24 17:23:25 activator: kass.example.com idle 11s — hibernating
2026/10/24 17:23:25 activator: kass.example.com hibernated
2026/10/24 17:23:25 activator: one.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/24 17:23:25 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:23:25 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:23:25 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:23:25 activator: bot.x.internal: v2 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:23:25 activator: worker bot.x.internal NOT respawned (attempt (release)): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/24 17:23:25 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 17:23:25 activator: link.example.com hibernated
2026/10/24 17:23:30 activator: tenant.test: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:23:30 activator: tenant.test not started: the executable it hashed (/tmp/rl3205600765/app) is build "v1", but the start names build "v2" (the live link moved between the two reads); the next wake reads it again
2026/10/24 17:23:30 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/24 17:23:30 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 17:23:30 activator: replica h.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/24 17:23:30 activator: replica h.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/24 17:23:30 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 17:23:30 activator: why.example.com hibernated
2026/10/24 17:23:30 activator: c.example.com hibernated
2026/10/24 17:23:30 activator: h.example.com hibernated
2026/10/24 17:23:30 activator: h.example.com: run run_a is not proven stopped: cgroup.events: permission denied
2026/10/24 17:23:30 activator: stop h.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 17:23:30 activator: h.example.com hibernated
2026/10/24 17:23:30 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 17:23:30 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 17:23:30 activator: seeding unproven-seed.example.com: /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3427349992/002/live/unproven-seed.example.com seed --db /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3427349992/003/unproven-seed.example.com.db
2026/10/24 17:23:30 activator: unproven-seed.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFailedSeedWhoseCleanupIsUnprovenKeepsTheDBAndLeavesAnObliga3427349992/003/unproven-seed.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 17:23:30 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm7827074/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm7827074/003/obligation.example.com.db
2026/10/24 17:23:30 activator: obligation.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm7827074/003/obligation.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 17:23:30 activator: obligation.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 17:23:30 activator: no replica for obligation.example.com in s3://test-bucket/obligation.example.com — booting fresh
2026/10/24 17:23:30 activator: seeding obligation.example.com: /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm7827074/002/live/obligation.example.com seed --db /tmp/TestTheNextWakeAfterAnUnprovenSeedProvesItBeforeTheDBIsWarm7827074/003/obligation.example.com.db
2026/10/24 17:23:30 activator: seeded obligation.example.com
2026/10/24 17:23:30 activator: seeding helper-seed.example.com: /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt4084093521/002/live/helper-seed.example.com seed --db /tmp/TestAFlagOnSeedKillsAHelperThatOutlivesIt4084093521/003/helper-seed.example.com.db
2026/10/24 17:23:30 activator: seeded helper-seed.example.com
2026/10/24 17:23:30 activator: seeding helper-unproven.example.com: /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven1552986348/002/live/helper-unproven.example.com seed --db /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven1552986348/003/helper-unproven.example.com.db
2026/10/24 17:23:30 activator: helper-unproven.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAFlagOnSeedWhoseHelperIsNotProvenGoneIsUnproven1552986348/003/helper-unproven.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 17:23:30 activator: seeding write-ahead.example.com: /tmp/TestASeedRecordsItsObligationBeforeItRuns540140995/002/live/write-ahead.example.com seed --db /tmp/TestASeedRecordsItsObligationBeforeItRuns540140995/003/write-ahead.example.com.db
2026/10/24 17:23:30 activator: seeded write-ahead.example.com
2026/10/24 17:23:30 activator: write-ahead.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 17:23:30 activator: no replica for write-ahead.example.com in s3://test-bucket/write-ahead.example.com — booting fresh
2026/10/24 17:23:30 activator: seeding no-room.example.com: /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing1354276319/002/live/no-room.example.com seed --db /tmp/TestASeedWhoseObligationCannotBeWrittenRunsNothing1354276319/003/no-room.example.com.db
2026/10/24 17:23:30 activator: seeding stuck-wal.example.com: /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation70865681/002/live/stuck-wal.example.com seed --db /tmp/TestAProvenFailedSeedWhoseDBCannotBeRemovedKeepsItsObligation70865681/003/stuck-wal.example.com.db
2026/10/24 17:23:30 activator: seeding preexisting.example.com: /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept3027441979/002/live/preexisting.example.com seed --db /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept3027441979/003/preexisting.example.com.db
2026/10/24 17:23:30 activator: preexisting.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestAPreexistingDBBesideAnUnprovenSeedIsRefusedThenKept3027441979/003/preexisting.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 17:23:30 activator: preexisting.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/24 17:23:30 activator: old-kernel-seed.example.com: this kernel has no cgroup.kill; the seed's leader's exit is the only proof there is
2026/10/24 17:23:30 activator: seeding failed-role-seed.example.com: /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB1642304912/002/live/failed-role-seed.example.com seed --db /tmp/TestAFlagOnFailedSeedProvesItsRoleChildBeforeRemovingTheDB1642304912/003/failed-role-seed.example.com.db
2026/10/24 17:23:30 activator: seeding discharge-fails.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai4099031696/002/live/discharge-fails.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai4099031696/003/discharge-fails.example.com.db
2026/10/24 17:23:30 activator: discharge-fails.example.com: the seed's obligation could not be discharged and its processes are not proven gone; /tmp/TestAFlagOffSeedWhoseDischargeFailsIsUnprovenWhileItsHelperRemai4099031696/003/discharge-fails.example.com.db.seed-unproven stays
2026/10/24 17:23:30 activator: seeding discharge-fails-proven.example.com: /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur3071925607/002/live/discharge-fails-proven.example.com seed --db /tmp/TestAFlagOffSeedWhoseDischargeFailsAfterAProofIsAnOrdinaryFailur3071925607/003/discharge-fails-proven.example.com.db
2026/10/24 17:23:30 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_B,_l3665305397/003/g/56/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:30 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall3615864366/003/g/57/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:30 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatplanned_fall3615864366/003/g/58/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:31 activator: swap.example.com hibernated
2026/10/24 17:23:31 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatholds_pinne284058664/007/g/60/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:31 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 17:23:31 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtTheBackendsLaunchPointflatoverlap_succ4261933838/007/g/62/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:31 activator: h.example.com: started carlos-instance-gen@h.example.com:64.service memory.max=536870912
2026/10/24 17:23:31 activator: swap.example.com hibernated
2026/10/24 17:23:31 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 17:23:31 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_super1954170855/007/g/71/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:31 activator: swap.example.com hibernated
2026/10/24 17:23:31 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartholds_pinned_build,_lea4124355237/007/g/73/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:31 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 17:23:31 activator: control listener for launch.example.com: listen unix /tmp/TestTheExecGateIsAskedAtEachCgroupStartoverlap_successor,_a_fai1822918502/007/g/75/launch.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:31 activator: fence.example hibernated
2026/10/24 17:23:31 activator: fence.example hibernated
2026/10/24 17:23:31 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink2563057623/003/g/c.x.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink2563057623/003/g/c.x.sock: no such file or directory)
2026/10/24 17:23:31 activator: c.example.com: /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink2563057623/003/g/c.xx.sock is not a socket we can normalize (stat: lstat /tmp/TestExecBackendGenSwitchedRepointsTheCanonicalLink2563057623/003/g/c.xx.sock: no such file or directory)
2026/10/24 17:23:32 activator: publish generations to /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile2414612285/002/not-a-dir/zdd/generations.json: open directory /tmp/TestStartDoesNotFailOnAnUnwritableGenerationsFile2414612285/002/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 17:23:32 activator: ident.example.com hibernated
2026/10/24 17:23:32 activator: instance ident.example.com exited unexpectedly — recycling
2026/10/24 17:23:32 activator: ident.example.com hibernated
2026/10/24 17:23:32 activator: due.example.com hibernated
2026/10/24 17:23:32 activator: due.example.com hibernated
2026/10/24 17:23:32 activator: ret.example.com hibernated
2026/10/24 17:23:32 activator: ret.example.com hibernated
2026/10/24 17:23:32 activator: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.example.com: a generation socket path would exceed 107 bytes; generation 9 keeps the canonical socket
2026/10/24 17:23:32 activator: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.example.com: a generation socket path would exceed 107 bytes; generation 9 keeps the canonical socket
2026/10/24 17:23:32 activator: one.example.com hibernated
2026/10/24 17:23:32 activator: purged 1 cached response(s) for flat.example.com on spawn
2026/10/24 17:23:33 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile1518108112/001/versions/vA/app seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile1518108112/002/seedexe.example.com.db
2026/10/24 17:23:33 activator: seeded seedexe.example.com
2026/10/24 17:23:33 activator: seeding seedexe.example.com: /tmp/TestSeedExeFlatRunsTheRecordedFile1518108112/001/live/seedexe.example.com seed --db /tmp/TestSeedExeFlatRunsTheRecordedFile1518108112/002/seedexe.example.com.db
2026/10/24 17:23:33 activator: seeded seedexe.example.com
2026/10/24 17:23:33 activator: seeding h.example.com: /tmp/TestNestedSeedExeNamesTheRecordedFile2312399478/001/live/h.example.com seed --db /tmp/nb3092817210/data/instances/h.example.com/h.example.com.db
2026/10/24 17:23:33 activator: seeded h.example.com
2026/10/24 17:23:33 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3126685611/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/24 17:23:33 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3126685611/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/24 17:23:33 activator: h.example.com not started: the executable its live link names (/tmp/TestADeclinedLinkOutsideTheStoreAsksRootForNothing3126685611/003/app) is outside the version store, so this box cannot check it against its holds; move the instance onto a promoted build
2026/10/24 17:23:33 activator: seeding h.example.com: /tmp/TestADeclinedNestedSeedRunsThePinnedFile3588346985/001/live/h.example.com seed --db /tmp/nb863107020/data/instances/h.example.com/h.example.com.db
2026/10/24 17:23:33 activator: seeded h.example.com
2026/10/24 17:23:33 activator: seeding tier.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed1668128833/002/versions/vA/app seed --db /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed1668128833/003/tier.example.com.db
2026/10/24 17:23:33 activator: tier.example.com: the seed's processes are not proven gone; its database is kept and /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTierflat_seed1668128833/003/tier.example.com.db.seed-unproven refuses it as warm until they are
2026/10/24 17:23:33 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=1073741824
2026/10/24 17:23:33 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=1073741824
2026/10/24 17:23:33 activator: seeding h.example.com: /tmp/TestStartGenExeAndSeedExeUseTheRecordedBuildsTiernested_seed2262867151/001/live/h.example.com seed --db /tmp/nb334492436/data/instances/h.example.com/h.example.com.db
2026/10/24 17:23:33 activator: seeded h.example.com
2026/10/24 17:23:33 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsthe_configured_k2279517995/001/shop.example.com.env: the platform mints none while it is
2026/10/24 17:23:33 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyAsksTheBucketBeforeItMintsan_own_key_pres2697425904/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/24 17:23:34 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestFirstStartWaitsForTheConfiguredKey433991760/002/shop.example.com.env: the platform mints none while it is
2026/10/24 17:23:34 activator: control listener for shop.example.com: listen unix /tmp/TestFirstStartWaitsForTheConfiguredKey433991760/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:34 activator: CARLOS_INSTANCE_KEY in /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptinto_the_own_file812217755/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/24 17:23:34 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestAKeyThatAppearsDuringTheBucketReadIsKeptdelivered_in_the_co1244734376/001/shop.example.com.env: the platform mints none while it is
2026/10/24 17:23:34 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration2527889395/001/shop.example.com.env: the platform mints none while it is
2026/10/24 17:23:34 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestEnsureInstanceKeyDefersToTheRenderedConfiguration2527889395/002/shop.example.com.env: the platform mints none while it is
2026/10/24 17:23:34 activator: CARLOS_INSTANCE_KEY in /tmp/TestEnsureInstanceKeyLeavesAFileWithTwoKeysAlone2274897395/001/shop.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/24 17:23:34 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistse101027565/002/shop.example.com.env: the platform mints none while it is
2026/10/24 17:23:34 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/TestExecBackendStartNeverHandsTheChildAnEmptyKeyWhileOneExistsn1643514266/002/shop.example.com.env: the platform mints none while it is
2026/10/24 17:23:34 activator: seeding seeded.example.com: /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance2773879450/001/live/seeded.example.com seed --db /tmp/TestExecBackendSeedMintsTheKeyForAFreshFlatInstance2773879450/002/seeded.example.com.db
2026/10/24 17:23:34 activator: seeded seeded.example.com
2026/10/24 17:23:34 activator: restored sleepy.example.com from s3://test-bucket/sleepy.example.com
hello
world
2026/10/24 17:23:34 activator: fence.example hibernated
2026/10/24 17:23:34 activator: fence.example hibernated
2026/10/24 17:23:34 activator: fence.example hibernated
2026/10/24 17:23:34 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/24 17:23:34 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/24 17:23:34 activator: barrier.example.com hibernated
2026/10/24 17:23:34 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/24 17:23:34 activator: generation 9 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/24 17:23:35 activator: stop barrier.example.com: a generation's termination is unproven; keeping the lease until it is
2026/10/24 17:23:35 activator: barrier.example.com: generation 9's child is not proven empty: g9 still holds a process
2026/10/24 17:23:35 activator: barrier.example.com hibernated
2026/10/24 17:23:35 activator: barrier.example.com: generation 12's child is not proven empty: g12 still holds a process
2026/10/24 17:23:35 activator: generation 12 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/24 17:23:35 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/24 17:23:35 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 17:23:35 activator: stop barrier.example.com: cgroup.kill: the child is not empty
2026/10/24 17:23:35 activator: barrier.example.com hibernated
2026/10/24 17:23:35 activator: stop generation 11: cgroup.kill: the child is not empty
2026/10/24 17:23:35 activator: generation 11 of barrier.example.com did not prove its exit; keeping it in the operation set
2026/10/24 17:23:35 activator: placed.example.com: kill generation 1097 after a failed health wait: g still holds a process
2026/10/24 17:23:35 activator: placed.example.com: generation 1097's child is not proven empty: g still holds a process
2026/10/24 17:23:35 activator: placed.example.com: failed wake's generation 1097 is not proven stopped; keeping the lease until it is
2026/10/24 17:23:35 activator: placed.example.com: wake failed and its generation is not proven stopped: start placed.example.com: instance not healthy within 300ms: Get "http://carlos/": dial unix /tmp/g454411062/g/1097/placed.example.com.sock: connect: no such file or directory; activator: generation start failed and its termination is unproven: kill generation 1097 after a failed health wait: g still holds a process
2026/10/24 17:23:35 activator: placed.example.com: failed wake's generation 1097 is proven stopped; lease given back
2026/10/24 17:23:36 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/24 17:23:36 activator: unproven.example.com: failed wake's generation 1100 is not proven stopped; keeping the lease until it is
2026/10/24 17:23:36 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 17:23:36 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/24 17:23:36 activator: unproven.example.com: generation 1100's child is not proven empty: the generation's child still holds a process
2026/10/24 17:23:36 activator: unproven.example.com: failed wake's generation 1100 is proven stopped; lease given back
2026/10/24 17:23:36 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/24 17:23:36 activator: unproven.example.com: failed wake's generation 1101 is not proven stopped; keeping the lease until it is
2026/10/24 17:23:36 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 17:23:36 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/24 17:23:36 activator: unproven.example.com: generation 1101's child is not proven empty: the generation's child still holds a process
2026/10/24 17:23:36 activator: unproven.example.com: failed wake's generation 1101 is proven stopped; lease given back
2026/10/24 17:23:36 activator: unproven.example.com: generation 1102's child is not proven empty: the generation's child still holds a process
2026/10/24 17:23:36 activator: unproven.example.com: failed wake's generation 1102 is not proven stopped; keeping the lease until it is
2026/10/24 17:23:36 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: health wait failed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 17:23:37 activator: unhealthy-f6.example.com: failed wake's generation 1103 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 17:23:37 activator: unhealthy-f6.example.com: failed wake's generation 1103 is not proven stopped; keeping the lease until it is
2026/10/24 17:23:37 activator: unhealthy-f6.example.com: failed wake's generation 1103 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 17:23:37 activator: unhealthy-f6.example.com: failed wake's generation 1103 is proven stopped; lease given back
2026/10/24 17:23:38 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/24 17:23:38 activator: one.example.com: an earlier planned wake's outcome is still being delivered after 200ms; refusing this wake until it is
2026/10/24 17:23:38 activator: instance one.example.com exited unexpectedly — recycling
2026/10/24 17:23:38 activator: one.example.com hibernated
2026/10/24 17:23:38 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:23:38 activator: bot.x.internal hibernated
2026/10/24 17:23:38 activator: worker bot.x.internal respawned (attempt (restart)), now running v2
2026/10/24 17:23:38 activator: bot.x.internal hibernated
2026/10/24 17:23:39 activator: worker bot.x.internal respawned (attempt (config)), now running v2
2026/10/24 17:23:39 activator: bot.x.internal hibernated
2026/10/24 17:23:39 activator: bot.x.internal hibernated
2026/10/24 17:23:39 activator: bot.x.internal hibernated
2026/10/24 17:23:39 activator: bot.x.internal hibernated
2026/10/24 17:23:39 activator: moved.example.com: promoting 2 staged object(s) from moves/moved.example.com/ (residence move landing)
2026/10/24 17:23:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:23:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:23:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:23:39 activator: x.example hibernated
2026/10/24 17:23:39 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/24 17:23:39 activator: x.example hibernated
2026/10/24 17:23:39 activator: x.example recovery probe healthy, but its fence changed; stopping it
2026/10/24 17:23:39 activator: x.example hibernated
2026/10/24 17:23:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:23:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:23:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:23:39 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/24 17:23:39 activator: x.example hibernated
2026/10/24 17:23:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:23:39 activator: x.example hibernated
2026/10/24 17:23:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:23:39 activator: x.example hibernated
2026/10/24 17:23:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:23:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:23:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:23:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:23:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:23:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:23:39 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:23:39 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:23:39 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY is not its home's key
2026/10/24 17:23:39 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_keyless_replica_is_refu3508242024/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:39 activator: replica shop-au.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/24 17:23:39 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_is_2885697394/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:39 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_pre-existing_env_key_th3076942557/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:39 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_that_dif3546284636/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:39 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_configured_key_with_no_2278302123/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:39 activator: replica shop-au.example.com not started: CARLOS_INSTANCE_KEY cannot be checked against its home's key (no entry for it in /etc/carlos/replica-home-keys; O8)
2026/10/24 17:23:39 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_home_updated,_this_re2058859639/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:39 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY is not the one the bucket's configuration now holds: a key change is in flight (ruling 27)
2026/10/24 17:23:39 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketthe_bucket_cannot_be_read4054620665/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:39 activator: replica shop-au.example.com not started: the bucket's configuration could not be read: bucket unreachable
2026/10/24 17:23:39 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketno_fresh_reader_refused3877456544/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:39 activator: replica shop-au.example.com not started: its configured CARLOS_INSTANCE_KEY cannot be checked against the bucket's configuration
2026/10/24 17:23:39 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketresidual_a_change_only_o596349339/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:39 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_config-only_homes_keyl2176708068/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:39 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaLaunchWithoutAControlSocketa_home_row_mints_and_its_3012744311/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:39 activator: seeding shop-au.example.com: /tmp/TestReplicaSeedIsGatedthe_homes_key899424443/001/live/shop-au.example.com seed --db /tmp/TestReplicaSeedIsGatedthe_homes_key899424443/002/shop-au.example.com.db
2026/10/24 17:23:39 activator: seeded shop-au.example.com
2026/10/24 17:23:39 activator: replica shop-au.example.com not started: where its CARLOS_INSTANCE_KEY came from is unknown
2026/10/24 17:23:39 activator: control listener for shop-au.example.com: listen unix /tmp/TestReplicaGateDecidesOnTheLaunchesOwnRead561465033/003/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:39 provision: a.shop.example.com: swapped for import sd_5ik3ia4bz5koj2t23zfd6anhqy (the original route is journaled)
2026/10/24 17:23:39 provision: a.shop.example.com: import sd_5ik3ia4bz5koj2t23zfd6anhqy rolled back to its original route
2026/10/24 17:23:39 provision: a.shop.example.com: import sd_5ik3ia4bz5koj2t23zfd6anhqy is cleaned up and its journal row retired
2026/10/24 17:23:39 activator: h.example.com: placed seed sd_t4b6fft4r65vgdacp5gqpsrx7y (8192 bytes)
2026/10/24 17:23:39 activator: h.example.com: placed seed sd_6s63ybizo5ojlfshdtfnt6lale (8192 bytes)
2026/10/24 17:23:39 activator: CARLOS_INSTANCE_KEY is set by the app's configuration for /tmp/nb563516256/data/instances/h.example.com/h.example.com.env: the platform mints none while it is
2026/10/24 17:23:39 activator: h.example.com: placed seed sd_uebj4uyhumimhpebyoltengbiu (8192 bytes)
2026/10/24 17:23:39 activator: CARLOS_INSTANCE_KEY in /tmp/nb3149525546/data/instances/h.example.com/h.example.com.env is not the shape the platform mints (44 base64 characters, 32 bytes): left in place, because the app may already have sealed data with it
2026/10/24 17:23:39 activator: h.example.com: placed seed sd_3xjwyddvlcsdw76ggtdlnp4haa (8192 bytes)
2026/10/24 17:23:39 activator: h.example.com: placed seed sd_mu3waemsxsi6xz42eoarhm7bku (8192 bytes)
2026/10/24 17:23:39 activator: h.example.com: placed seed sd_qzfna5rvzsq37zt5v3xdsc3pea (8192 bytes)
2026/10/24 17:23:39 activator: h.example.com: keeping seed sd_qzfna5rvzsq37zt5v3xdsc3pea placed by an earlier wake
2026/10/24 17:23:39 activator: h.example.com: placed seed sd_xaygyyekwr4mslsnbetnj5szpy (8192 bytes)
2026/10/24 17:23:39 activator: h.example.com: placed seed sd_ax6rtb54zjgzgfap6vd7gd2v2m (8192 bytes)
2026/10/24 17:23:39 activator: h.example.com: placed seed sd_gvcjaiekpw53omlmjtluloahve (8192 bytes)
2026/10/24 17:23:39 activator: h.example.com: seed sd_ibtnzdyro2ny25derkg6v23qam: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/24 17:23:39 activator: h.example.com: seed sd_upcqcr2mc3cv6eyk2mtpscsn5e: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/24 17:23:39 activator: h.example.com: seed sd_igkqeh5w3h3tpq43tdkqiofusq: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/24 17:23:39 activator: h.example.com: seed sd_4ntcou4wrq22ddti4ozd66r7hi: h.example.com could not start from its seed: the snapshot could not be read. The next wake tries again.: bucket down
2026/10/24 17:23:39 activator: h.example.com: seed sd_7vvk36dw7d3ykuhwmz77hkdf7e: h.example.com could not start from its seed: the snapshot's length or sha256 does not match its record.
2026/10/24 17:23:39 activator: h.example.com: seed sd_aeafgzlbppoywxo7ema54mlq7u: h.example.com could not start from its seed: the snapshot is not a database that passes an integrity check.: integrity_check: file is not a database (26)
2026/10/24 17:23:39 activator: h.example.com: seed sd_rcauciil2pllkyidqy3aqn66ey: h.example.com could not start from its seed: claim control is on for this box.
2026/10/24 17:23:39 activator: h.example.com: seed sd_rcauciil2pllkyidqy3aqn66ey: h.example.com could not start from its seed: this box does not run instances nested.
2026/10/24 17:23:39 activator: h.example.com: seed sd_nxq44inrjwv7yfpw7frv7en4ui: seed fallback for h.example.com: cannot tell whether it has a replica of its own: listing failed
2026/10/24 17:23:39 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 17:23:39 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 17:23:39 activator: h.example.com: seed sd_2pk33jcohrjg7q6wgxdgee2dxu: seed fallback for h.example.com: found no replica of its own to restore
2026/10/24 17:23:39 activator: h.example.com: promoting 1 staged object(s) from moves/h.example.com/ (residence move landing)
2026/10/24 17:23:39 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 17:23:39 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 17:23:39 activator: h.example.com: seed sd_pawu5i4zqq5mmorr7pfxwl4shy: seed fallback for h.example.com: found no replica of its own to restore
2026/10/24 17:23:39 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 17:23:39 activator: h.example.com: seed sd_wlaww56umhushbtm2s2y3ehit4: seed fallback for h.example.com: found no replica of its own to restore
2026/10/24 17:23:39 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 17:23:39 activator: h.example.com: placed seed sd_6ausb4xggc7qp4ku6antq24xyu (8192 bytes)
2026/10/24 17:23:39 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 17:23:39 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 17:23:39 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 17:23:39 activator: h.example.com: seed sd_xctgd4q6qrrstwppvatyv6vrru: h.example.com could not start from its seed: the snapshot could not be read. The next wake tries again.: connection reset
2026/10/24 17:23:39 activator: h.example.com: seed sd_bof6klelqcxcbbkaby3ot4uml4: h.example.com could not start from its seed: it could not be put in place. The next wake tries again.: openat2 /tmp/nb1466067167/data/instances/h.example.com/.seed-sd_bof6klelqcxcbbkaby3ot4uml4.tmp: permission denied
2026/10/24 17:23:39 activator: h.example.com: seed sd_3xcu3lgp6phqgdvetm72afz7wu: h.example.com could not start from its seed: the snapshot is not a database that passes an integrity check.: open: unable to open database file (14)
2026/10/24 17:23:39 activator: h.example.com: seed sd_rrpdwcgn6mzjh7ygybjzrmxyty: h.example.com could not start from its seed: it could not be put in place. The next wake tries again.: instancefs: path contains a symlink: /tmp/nb3938763158/data/instances/h.example.com/.seed-sd_rrpdwcgn6mzjh7ygybjzrmxyty.tmp
2026/10/24 17:23:39 activator: h.example.com: placed seed sd_kjqrwdm53liwjcg4pxg3ira37m (8192 bytes)
2026/10/24 17:23:39 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 17:23:39 activator: h.example.com: seed sd_vb4rdochlox6mm66enrbrwf2ti not recorded as placed yet: record unreachable
2026/10/24 17:23:39 activator: h.example.com: /tmp/nb2923871701/data/instances/h.example.com/h.example.com.db held seed sd_v3kaqwkxfjzkeqojyksc5cw22m's marker, not this route's (seed "sd_bigdknv2kqc4xgt5xxa32pnlou"): purged before the restore
2026/10/24 17:23:39 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 17:23:39 activator: h.example.com: /tmp/nb1517356728/data/instances/h.example.com/h.example.com.db held seed sd_tesxahrsucnukpikgp7stp345e's marker, not this route's (seed "sd_kksi5e2ksyhl3lxd64374ofqv4"): purged before the restore
2026/10/24 17:23:39 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 17:23:39 activator: h.example.com: /tmp/nb1034424976/data/instances/h.example.com/h.example.com.db held seed sd_lbbdlrghqhmta4badvhmjxw6ye's marker, not this route's (seed ""): purged before the restore
2026/10/24 17:23:39 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 17:23:39 activator: h.example.com: /tmp/nb2392441651/data/instances/h.example.com/h.example.com.db held seed sd_zok3vgwkmxxanbwl4hk4a2bqfu's marker, not this route's (seed "sd_zim2zybnzzl6i3qlkd5qiv4aue"): purged before the restore
2026/10/24 17:23:39 activator: restored h.example.com from s3://test-bucket/h.example.com
2026/10/24 17:23:39 activator: h.example.com: /tmp/nb3511881511/data/instances/h.example.com/h.example.com.db held seed sd_rd3cf7nvhngiqrq4n4sspcdvmy's marker, not this route's (seed ""): purged before the restore
2026/10/24 17:23:39 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 17:23:39 activator: h.example.com: seed sd_5oydzqyf3qg7g3uwgdakz2sffa: h.example.com: activator: seed unproven; processes remain: an earlier seed's processes are not proven gone, so its database is not used: cgroup.events: unreadable
2026/10/24 17:23:39 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 17:23:39 activator: h.example.com: placed seed sd_5oydzqyf3qg7g3uwgdakz2sffa (8192 bytes)
2026/10/24 17:23:39 activator: h.example.com: placed seed sd_6dwerpshturcjd4fhlit2gtltu (8192 bytes)
2026/10/24 17:23:39 activator: h.example.com: /tmp/nb1802135665/data/instances/h.example.com/h.example.com.db held seed an aborted import's data's marker, not this route's (seed ""): purged before the restore
2026/10/24 17:23:39 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 17:23:39 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 17:23:40 activator: swap.example.com hibernated
2026/10/24 17:23:40 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/24 17:23:40 activator: swap.example.com hibernated
2026/10/24 17:23:40 activator: LEASE LOST for swap.example.com (renewal failed) — standing down
2026/10/24 17:23:40 activator: swap.example.com hibernated
2026/10/24 17:23:40 activator: swap.example.com hibernated
2026/10/24 17:23:40 activator: swap.example.com is no longer hibernate-managed — releasing
2026/10/24 17:23:40 activator: swap.example.com hibernated
2026/10/24 17:23:40 activator: swap.example.com hibernated
2026/10/24 17:23:40 activator: exit.example.com hibernated
2026/10/24 17:23:40 activator: wake shop.example.com: activator stopped during startup
2026/10/24 17:23:40 activator: wake shop.example.com: activator stopped during startup
2026/10/24 17:23:40 activator: shop.example.com: route refreshed: db "" -> "" socket "/run/carlos/shop.example.com.sock" -> "/run/carlos/instances/shop.example.com/app.sock"
2026/10/24 17:23:40 activator: stop missed-join.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/24 17:23:41 activator: missed-join.example.com hibernated
2026/10/24 17:23:41 activator: stop nested-stuck.example.com: a sidecar start is still in flight; keeping the lease until it returns
2026/10/24 17:23:41 activator: nested-stuck.example.com: the sidecar a restart brought back for a stopping entry is not proven gone (activator: stop unproven; processes remain); keeping the lease until it has exited
2026/10/24 17:23:41 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 17:23:41 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 17:23:41 activator: stop nested-stuck.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 17:23:41 activator: nested-stuck.example.com hibernated
2026/10/24 17:23:41 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 17:23:41 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 17:23:41 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 17:23:41 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/24 17:23:41 activator: bot.x.internal hibernated
2026/10/24 17:23:41 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir: not a directory; the sweep retries
2026/10/24 17:23:41 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir: not a directory
2026/10/24 17:23:41 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir: not a directory; the sweep retries
2026/10/24 17:23:41 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/24 17:23:41 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir: not a directory; the sweep retries
2026/10/24 17:23:41 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir: not a directory; the sweep retries
2026/10/24 17:23:41 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir: not a directory
2026/10/24 17:23:41 activator: publish generations to /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir: not a directory; the sweep retries
2026/10/24 17:23:41 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerReportsWhetherTheAttemptsBuildExecutedrefused_729123786/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/24 17:23:41 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:23:41 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:23:42 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 17:23:42 activator: worker bot.x.internal: the attempt's respawn did not finish within 150ms of its caller leaving; reporting its build as possibly executed
2026/10/24 17:23:42 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:23:42 activator: swap.example.com hibernated
2026/10/24 17:23:42 activator: swap.example.com hibernated
2026/10/24 17:23:42 activator: swap.example.com hibernated
2026/10/24 17:23:42 activator: swap.example.com hibernated
2026/10/24 17:23:42 activator: swap.example.com hibernated
2026/10/24 17:23:42 activator: swap.example.com hibernated
2026/10/24 17:23:42 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/24 17:23:42 activator: swap.example.com hibernated
2026/10/24 17:23:42 activator: swap.example.com hibernated
2026/10/24 17:23:43 activator: swap.example.com hibernated
2026/10/24 17:23:43 activator: swap.example.com hibernated
2026/10/24 17:23:43 activator: swap.example.com hibernated
2026/10/24 17:23:43 activator: swap.example.com hibernated
2026/10/24 17:23:43 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing76249333/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing76249333/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 17:23:43 activator: publish generations to /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing76249333/005/not-a-dir/zdd/generations.json: open directory /tmp/TestSwapHoldUnpublishedLeavesTheIncumbentServing76249333/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 17:23:43 activator: h.example.com: started carlos-instance-gen@h.example.com:7.service memory.max=1073741824
2026/10/24 17:23:43 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: processes remain
2026/10/24 17:23:43 activator: swap.example.com hibernated
2026/10/24 17:23:51 activator: h.example.com: kill carlos-instance-gen@h.example.com:7.service after a failed start: instance control /stop: 502: stop: processes remain
2026/10/24 17:23:51 activator: seeding h.example.com: /tmp/TestSeedUnitCleansUpOnlyItsOwnDBOnFailure3365098981/001/live/h.example.com seed --fast --db /tmp/nb2847355007/data/instances/h.example.com/h.example.com.db
2026/10/24 17:23:51 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/24 17:23:51 activator: seeding h.example.com: /tmp/TestSeedUnitFailureStopsBeforeCleanupAndKeepsTheDBWhenProcessesR2564126974/001/live/h.example.com seed --fast --db /tmp/nb3442560292/data/instances/h.example.com/h.example.com.db
2026/10/24 17:23:51 activator: seeding h.example.com: /tmp/TestANestedSeedRecordsItsObligationBeforeItRuns456691631/001/live/h.example.com seed --db /tmp/nb2166332612/data/instances/h.example.com/h.example.com.db
2026/10/24 17:23:51 activator: seeded h.example.com
2026/10/24 17:23:51 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI1538708530/001/live/h.example.com seed --db /tmp/nb3031497366/data/instances/h.example.com/h.example.com.db
2026/10/24 17:23:51 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 17:23:51 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 17:23:51 activator: seeding h.example.com: /tmp/TestAnUnprovenNestedSeedLeavesItsObligationAndTheNextWakeProvesI292806663/001/live/h.example.com seed --db /tmp/nb3651583197/data/instances/h.example.com/h.example.com.db
2026/10/24 17:23:51 activator: h.example.com: the earlier seed's processes are proven gone; its database is removed and the seed runs again
2026/10/24 17:23:51 activator: no replica for h.example.com in s3://test-bucket/h.example.com — booting fresh
2026/10/24 17:23:51 activator: fifo-obligation.example.com: the earlier seed's processes are proven gone; the database that was there before it is kept
2026/10/24 17:23:51 activator: one.example.com: first wake of v2 had 30s of its health window left at the exec, under the 1h0m0s floor; booting the good build v1 without trying it
2026/10/24 17:23:52 activator: one.example.com hibernated
2026/10/24 17:23:52 activator: tenant.test not started: it is held on v1, and the launch hashed "v2" (the live link moved after the check); the next wake checks again
2026/10/24 17:23:52 activator: one.example.com not woken: it is held on v1, and this wake would run "v2" (the live link moved after the check); the next wake checks again
2026/10/24 17:23:52 activator: one.example.com not woken: it is held on v1, and this wake would run "" (the live link moved after the check); the next wake checks again
2026/10/24 17:23:52 activator: swap.example.com hibernated
2026/10/24 17:23:55 activator: seeding h.example.com: /tmp/TestANestedUnversionedLinkWithNoHookWakesAsToday1007200654/001/live/h.example.com seed --db /tmp/nb2674165882/data/instances/h.example.com/h.example.com.db
2026/10/24 17:23:55 activator: seeded h.example.com
2026/10/24 17:23:55 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/24 17:23:55 activator: swap.example.com hibernated
2026/10/24 17:23:55 activator: one.example.com: generation 1959's child is not proven empty: activator: stop unproven; processes remain
2026/10/24 17:23:55 activator: one.example.com: failed wake's generation 1959 is not proven stopped; keeping the lease until it is
2026/10/24 17:23:55 activator: one.example.com hibernated
2026/10/24 17:23:56 activator: one.example.com: first wake of v2 had 400ms of its health window left at the exec, under the 20s floor; booting the good build v1 without trying it
2026/10/24 17:23:57 activator: x.example: hold's pinned wake refused or abandoned: the claims admission refused before the exec: hostbox: unresolved lifecycle contract: isolated root launcher cannot carry claim reservations
2026/10/24 17:23:57 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat319187821/003/g/1966/real.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:59 activator: control listener for real.example.com: listen unix /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat319187821/003/g/1967/real.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:23:59 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat319187821/003/g/1967/real.example.com.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeflat319187821/003/g/1967/real.example.com.sock: no such file or directory)
2026/10/24 17:23:59 activator: real.example.com hibernated
2026/10/24 17:24:01 activator: real.example.com: /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign1821785048/004/hand-set.sock is not a socket we can normalize (stat: lstat /tmp/TestAPlannedFirstWakeOnTheRealExecBackendFallsBackInTimeforeign1821785048/004/hand-set.sock: no such file or directory)
2026/10/24 17:24:01 activator: real.example.com hibernated
2026/10/24 17:24:01 activator: bot.x.internal: worker generation 0's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 17:24:01 activator: bot.x.internal: worker generation 1971's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 17:24:01 activator: bot.x.internal: failed wake's generation 1971 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:01 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3985028151/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3985028151/002/zdd: not a directory; the sweep retries
2026/10/24 17:24:01 activator: worker bot.x.internal not spawned: the build it would exec (v2 /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3985028151/003/v2) could not be published first: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3985028151/002/zdd: not a directory
2026/10/24 17:24:01 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3985028151/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3985028151/002/zdd: not a directory; the sweep retries
2026/10/24 17:24:01 activator: publish generations to /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3985028151/002/zdd/generations.json: open directory /tmp/TestAPlannedWorkerRefusedForPublicationRecordsNothing3985028151/002/zdd: not a directory; the sweep retries
2026/10/24 17:24:01 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn4115272274/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn4115272274/002/not-a-dir: not a directory; the sweep retries
2026/10/24 17:24:01 activator: publish generations to /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn4115272274/002/not-a-dir/generations.json: open directory /tmp/TestAPlannedWorkerRefusedAtItsPublicationAttemptsNoSpawn4115272274/002/not-a-dir: not a directory; the sweep retries
2026/10/24 17:24:01 activator: instance one.example.com exited unexpectedly — recycling
2026/10/24 17:24:01 activator: one.example.com hibernated
2026/10/24 17:24:02 activator: one.example.com hibernated
2026/10/24 17:24:03 activator: one.example.com: first wake of v2 had 500ms of its health window left at the exec, under the 1s floor; booting the good build v1 without trying it
2026/10/24 17:24:05 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/24 17:24:05 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/24 17:24:05 activator: worker bot.example.com respawned (exited), now running 
2026/10/24 17:24:05 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/24 17:24:05 activator: worker bot.example.com exited; no captured build to recycle onto, so it respawns from live/bot.example.com
2026/10/24 17:24:06 activator: worker bot.example.com respawned (exited), now running 
2026/10/24 17:24:06 activator: bot.example.com: worker generation 1986's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 17:24:06 activator: stop bot.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 17:24:06 activator: bot.example.com hibernated
2026/10/24 17:24:06 activator: bot3.example.com: worker generation 1987's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 17:24:06 activator: worker bot3.example.com NOT respawned (test: a restart with no old handle): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/24 17:24:06 activator: bot4.example.com: worker generation 1989's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 17:24:06 activator: bot4.example.com: failed wake's generation 1989 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:06 activator: bot4.example.com: failed wake's generation 1989 is proven stopped; lease given back
2026/10/24 17:24:06 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/24 17:24:06 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/24 17:24:06 activator: bot2.example.com: worker generation 1990's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 17:24:06 activator: worker bot2.example.com NOT respawned (exited): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/24 17:24:06 activator: worker bot2.example.com exited — restarting (backoff now 2s)
2026/10/24 17:24:06 activator: worker bot2.example.com exited; no captured build to recycle onto, so it respawns from live/bot2.example.com
2026/10/24 17:24:06 activator: worker bot2.example.com respawned (exited), now running 
2026/10/24 17:24:06 activator: worker bot.example.com exited — restarting (backoff now 2s)
2026/10/24 17:24:06 activator: worker bot.example.com respawned (exited), now running v1
2026/10/24 17:24:06 activator: bot.example.com hibernated
2026/10/24 17:24:06 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/24 17:24:06 activator: worker bot10.example.com: the shed respawn child's exit is not proven (activator: stop unproven; processes remain); the entry keeps it and its build until the sweep proves it
2026/10/24 17:24:06 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/24 17:24:06 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/24 17:24:06 activator: bot10.example.com: the shed respawn child has exited
2026/10/24 17:24:06 activator: bot10.example.com hibernated
2026/10/24 17:24:06 activator: worker bot10.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/24 17:24:06 activator: LEASE LOST for bot10.example.com (renewal failed) — standing down
2026/10/24 17:24:06 activator: bot10.example.com respawn did not finish before its stop; stopping the rest, the lease is kept until it has
2026/10/24 17:24:06 activator: stop bot10.example.com: a respawn or the child it shed is not proven finished; keeping the lease until it is
2026/10/24 17:24:06 activator: bot10.example.com hibernated
2026/10/24 17:24:07 activator: bot8.example.com: worker generation 1993's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 17:24:07 activator: bot8.example.com: failed wake's generation 1993 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:07 activator: bot8.example.com: failed wake's generation 1993 is proven stopped; lease given back
2026/10/24 17:24:07 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:07 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:07 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/24 17:24:07 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:07 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:07 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/24 17:24:07 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/24 17:24:07 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:07 activator: bot.x.internal hibernated
2026/10/24 17:24:07 activator: worker bot.x.internal respawned (attempt (restart)), now running v1
2026/10/24 17:24:07 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 17:24:07 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:07 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:08 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 17:24:08 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 17:24:08 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:08 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 17:24:08 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 17:24:08 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 17:24:08 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 17:24:09 activator: bot.x.internal: worker generation 1996's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 17:24:09 activator: worker bot.x.internal NOT respawned (attempt (release)): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/24 17:24:09 activator: bot.x.internal: worker generation 1996's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 17:24:09 activator: worker bot.x.internal NOT respawned (restoring v1): an earlier child's termination is not proven (its host cgroup or a shed respawn child may still hold a process, or the proof could not run), so nothing execs beside it; the sweep retries
2026/10/24 17:24:09 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: lease lost; abandoning it
2026/10/24 17:24:09 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir: not a directory; the sweep retries
2026/10/24 17:24:09 activator: worker bot.x.internal not spawned: the build it would exec (v2 /opt/carlos/versions/v2/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir: not a directory
2026/10/24 17:24:09 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir: not a directory; the sweep retries
2026/10/24 17:24:09 activator: WORKER bot.x.internal failed to respawn (attempt (release)): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/24 17:24:09 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir: not a directory; the sweep retries
2026/10/24 17:24:09 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir: not a directory; the sweep retries
2026/10/24 17:24:09 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir: not a directory
2026/10/24 17:24:09 activator: publish generations to /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir/generations.json: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir: not a directory; the sweep retries
2026/10/24 17:24:09 activator: WORKER bot.x.internal failed to respawn (restoring v1): worker build not published before its exec: open directory /tmp/TestReplaceWorkerRefusedForPublicationIsAbandoned2148873124/002/not-a-dir: not a directory — the sweep's crash restart takes it from here
2026/10/24 17:24:09 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:09 activator: bot.x.internal hibernated
2026/10/24 17:24:09 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:09 activator: worker bot.x.internal NOT respawned (falling back to v1): swap: lease lost; abandoning it
2026/10/24 17:24:09 activator: WORKER bot.x.internal failed to respawn (attempt (restart)): exec: text file busy — the sweep's crash restart takes it from here
2026/10/24 17:24:09 activator: WORKER bot.x.internal failed to respawn (restoring v1): exec: text file busy — the sweep's crash restart takes it from here
2026/10/24 17:24:09 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:09 activator: worker bot.x.internal respawned (falling back to v1), now running v1
2026/10/24 17:24:09 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 17:24:09 activator: worker bot.x.internal: hold attempt onto v2 was superseded after its exec; the build it replaced (v1) is not restored, and the worker stops
2026/10/24 17:24:09 activator: bot.x.internal hibernated
2026/10/24 17:24:10 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 17:24:10 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 17:24:10 activator: worker bot.x.internal NOT respawned (attempt (release)): swap: superseded by a newer attempt; abandoning it
2026/10/24 17:24:10 activator: worker bot.x.internal respawned (restoring v1), now running v1
2026/10/24 17:24:10 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:10 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/24 17:24:10 activator: bot.x.internal hibernated
2026/10/24 17:24:10 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:10 activator: LEASE LOST for bot.x.internal (renewal failed) — standing down
2026/10/24 17:24:10 activator: bot.x.internal hibernated
2026/10/24 17:24:10 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:10 activator: bot.x.internal hibernated
2026/10/24 17:24:11 activator: worker bot.x.internal respawned (attempt (release)), now running v2
2026/10/24 17:24:11 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/24 17:24:11 activator: bot.x.internal hibernated
2026/10/24 17:24:11 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
--- FAIL: TestExecBackendWorkerRecycleExecsTheCapturedBuildAfterARepoint (0.00s)
    worker_staged_test.go:155: fork/exec /tmp/TestExecBackendWorkerRecycleExecsTheCapturedBuildAfterARepoint105511492/001/versions/v1/app: text file busy
2026/10/24 17:24:11 activator: h.example.com: processes remain after failed; not releasing
2026/10/24 17:24:11 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/24 17:24:11 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 17:24:11 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:11 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 17:24:11 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:11 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 17:24:11 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:11 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 17:24:11 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:11 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 17:24:11 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:11 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/24 17:24:11 activator: swap.example.com hibernated
2026/10/24 17:24:11 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 17:24:11 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:11 activator: swap.example.com hibernated
2026/10/24 17:24:11 activator: swap.example.com: failed wake's generation 2030 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:11 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 17:24:11 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:11 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:11 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:11 activator: swap.example.com: failed wake's generation 2030 is proven stopped; lease given back
2026/10/24 17:24:11 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 17:24:11 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:11 activator: one.example.com idle 1m31s — hibernating
2026/10/24 17:24:11 activator: one.example.com hibernated
2026/10/24 17:24:11 activator: h.example.com: unit restarted outside the activator (invocation someone-else, ours inv7); stopping it
2026/10/24 17:24:11 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/24 17:24:11 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:11 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:11 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:11 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:11 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 17:24:11 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 17:24:11 activator: swap.example.com idle 2h0m0s — hibernating
2026/10/24 17:24:11 activator: swap.example.com hibernated
2026/10/24 17:24:11 activator: swap.example.com: a discarded SIDECAR is not proven gone (activator: stop unproven; processes remain); keeping it tracked
2026/10/24 17:24:11 activator: sidecar for swap.example.com was down — restarted (backoff now 4s)
2026/10/24 17:24:11 activator: swap.example.com hibernated
2026/10/24 17:24:11 activator: worker bot9.example.com NOT respawned (test: shed after the exec): swap: lease lost; abandoning it
2026/10/24 17:24:11 activator: bot9.example.com: worker generation 2035's host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 17:24:11 activator: generation 2035 of bot9.example.com did not prove its exit; keeping it in the operation set
2026/10/24 17:24:11 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 17:24:11 activator: SIDECAR for swap.example.com is not proven gone for its replacement; tracking it (the sweep re-proves it)
2026/10/24 17:24:11 activator: swap.example.com hibernated
2026/10/24 17:24:11 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/24 17:24:11 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 17:24:11 activator: LEASE LOST for bot9.example.com (renewal failed) — standing down
2026/10/24 17:24:11 activator: bot9.example.com hibernated
2026/10/24 17:24:11 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/24 17:24:11 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 17:24:11 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 4s)
2026/10/24 17:24:11 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:11 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:11 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 17:24:11 activator: stop swap.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 17:24:11 activator: c.example.com hibernated
2026/10/24 17:24:11 activator: SIDECAR restart for swap.example.com failed: a legacy sidecar of this host is not proven gone (next attempt in 8s)
2026/10/24 17:24:11 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:11 activator: worker bot.x.internal exited; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/24 17:24:11 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 17:24:11 activator: swap.example.com hibernated
2026/10/24 17:24:11 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 17:24:11 activator: sidecar for swap.example.com was down — restarted (backoff now 16s)
2026/10/24 17:24:12 activator: swap.example.com hibernated
2026/10/24 17:24:12 activator: swap.example.com: the sidecar's leader exited but its cgroup is not proven empty: role-sidecar: still populated
2026/10/24 17:24:12 activator: instance slowproof.example.com exited unexpectedly — recycling
2026/10/24 17:24:12 activator: slowproof.example.com hibernated
2026/10/24 17:24:12 activator: swap.example.com hibernated
2026/10/24 17:24:12 activator: swap.example.com hibernated
2026/10/24 17:24:12 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/24 17:24:12 activator: early.example.com: the aborted start's unit is not proven stopped (context canceled); keeping the lease until it is
2026/10/24 17:24:12 activator: queued.example.com hibernated
2026/10/24 17:24:12 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 17:24:12 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:12 activator: refused.example.com: giving the lease back failed: delete lease: context deadline exceeded; retrying it
2026/10/24 17:24:12 activator: refused.example.com: lease given back on retry
2026/10/24 17:24:12 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/24 17:24:12 activator: one.example.com hibernated
2026/10/24 17:24:12 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 17:24:12 activator: instance unproven.example.com exited unexpectedly — recycling
2026/10/24 17:24:12 activator: unproven.example.com hibernated
2026/10/24 17:24:12 activator: instance restart.example.com exited unexpectedly — recycling
2026/10/24 17:24:12 activator: restart.example.com hibernated
2026/10/24 17:24:12 activator: SIDECAR for late-sidecar.example failed to start: activator: maintenance fence active — retaining returned process
2026/10/24 17:24:12 activator: late-sidecar.example hibernated
2026/10/24 17:24:12 activator: unproven.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@unproven: instance control socket: connection refused); keeping the lease until it is
2026/10/24 17:24:12 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/24 17:24:12 activator: swap.example.com hibernated
2026/10/24 17:24:12 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/24 17:24:12 activator: own.example.com: failed wake's generation 0 is proven stopped; lease given back
2026/10/24 17:24:12 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/24 17:24:12 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/24 17:24:12 provision: a.shop.example.com: swapped for import sd_2klkjsc54uoqffjhcju7v7ua2y (the original route is journaled)
2026/10/24 17:24:12 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 17:24:12 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:12 activator: slowproof.example.com: the aborted start's unit is not proven stopped (context deadline exceeded); keeping the lease until it is
2026/10/24 17:24:12 activator: swap.example.com hibernated
2026/10/24 17:24:12 provision: a.shop.example.com: swapped for import sd_mlioc6itcrpgcrk2sornqygenu (the original route is journaled)
2026/10/24 17:24:12 activator: stuck.example.com hibernated
2026/10/24 17:24:12 provision: a.shop.example.com: import sd_2klkjsc54uoqffjhcju7v7ua2y rolled back to its original route
2026/10/24 17:24:12 provision: a.shop.example.com: import sd_2klkjsc54uoqffjhcju7v7ua2y is cleaned up and its journal row retired
2026/10/24 17:24:12 activator: fence.example hibernated
2026/10/24 17:24:12 provision: a.shop.example.com: swapped for import sd_5ax4nk574rukjfnndkth44fdly (the original route is journaled)
2026/10/24 17:24:12 provision: a.shop.example.com: swapped for import sd_bilmtjn2z3rpmbnhvy2iwaarii (the original route is journaled)
2026/10/24 17:24:12 activator: a.shop.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@a.shop.example.com: unit still active); keeping the lease until it is
2026/10/24 17:24:12 provision: a.shop.example.com: import sd_mlioc6itcrpgcrk2sornqygenu rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 17:24:12 provision: a.shop.example.com: import sd_mlioc6itcrpgcrk2sornqygenu rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 17:24:12 activator: a.shop.example.com: the aborted start's unit is not proven stopped (stop carlos-instance@a.shop.example.com: instance control socket: connection refused); keeping the lease until it is
2026/10/24 17:24:12 provision: a.shop.example.com: import sd_5ax4nk574rukjfnndkth44fdly rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 17:24:12 provision: a.shop.example.com: import sd_bilmtjn2z3rpmbnhvy2iwaarii rolled back to its original route
2026/10/24 17:24:12 provision: a.shop.example.com: import sd_bilmtjn2z3rpmbnhvy2iwaarii is cleaned up and its journal row retired
2026/10/24 17:24:12 activator: a.shop.example.com: failed wake's generation 2074 is proven stopped; lease given back
2026/10/24 17:24:12 provision: a.shop.example.com: import sd_mlioc6itcrpgcrk2sornqygenu rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 17:24:12 provision: a.shop.example.com: import sd_5ax4nk574rukjfnndkth44fdly rolled back to its original route
2026/10/24 17:24:12 provision: a.shop.example.com: import sd_5ax4nk574rukjfnndkth44fdly is cleaned up and its journal row retired
2026/10/24 17:24:12 provision: a.shop.example.com: import sd_mlioc6itcrpgcrk2sornqygenu rollback deferred: the host is not fenced and stopped yet (activator: stop evidence unavailable)
2026/10/24 17:24:12 activator: own.example.com: failed wake's generation 0 is proven stopped; lease given back
2026/10/24 17:24:12 activator: swap.example.com hibernated
2026/10/24 17:24:12 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 17:24:12 activator: exited-flat.example.com: failed wake's generation 2092 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 17:24:12 activator: exited-flat.example.com: failed wake's generation 2092 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:12 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:24:12 activator: swap.example.com hibernated
2026/10/24 17:24:12 activator: swap.example.com hibernated
2026/10/24 17:24:12 activator: restore parent limit for swap.example.com: cgroup: parent memory.max: device or resource busy (kept; the sweep retries)
2026/10/24 17:24:12 activator: swap.example.com hibernated
2026/10/24 17:24:12 activator: h.example.com: started carlos-instance-gen@h.example.com:2103.service memory.max=536870912
2026/10/24 17:24:12 activator: f6.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 17:24:12 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 17:24:12 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:24:12 activator: worker bot.x.internal respawned (hit its max_run), now running 
2026/10/24 17:24:12 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/24 17:24:12 activator: c.example.com hibernated
2026/10/24 17:24:12 activator: forked.example.com: failed wake's generation 2106 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:12 activator: swap.example.com hibernated
2026/10/24 17:24:12 activator: swap.example.com: generation 2108's child is not proven empty: the child still holds a process
2026/10/24 17:24:12 activator: swap.example.com: failed wake's generation 2108 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:12 activator: swap.example.com hibernated
2026/10/24 17:24:12 activator: shutdown: out of time with 1 failed wake(s) not proven stopped, their leases still held: unproven.example.com (generation 2051)
2026/10/24 17:24:12 activator: forked.example.com: failed wake's generation 2106 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 17:24:12 activator: swap.example.com: failed wake's generation 2108 is proven stopped; lease given back
2026/10/24 17:24:12 activator: one.example.com hibernated
2026/10/24 17:24:12 activator: forked.example.com: failed wake's generation 2106 is proven stopped; lease given back
2026/10/24 17:24:12 provision: a.shop.example.com: swapped for import sd_exw3jqfh2yqcno7hgxkkowutze (the original route is journaled)
2026/10/24 17:24:12 activator: c.example.com hibernated
2026/10/24 17:24:12 activator: a.shop.example.com hibernated
2026/10/24 17:24:12 activator: swap.example.com hibernated
2026/10/24 17:24:12 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/24 17:24:12 provision: a.shop.example.com: import sd_exw3jqfh2yqcno7hgxkkowutze rolled back to its original route
2026/10/24 17:24:12 provision: a.shop.example.com: import sd_exw3jqfh2yqcno7hgxkkowutze is cleaned up and its journal row retired
2026/10/24 17:24:13 activator: fenced.example.com: failed wake's generation 2116 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:13 activator: swap.example.com hibernated
2026/10/24 17:24:13 activator: LEASE HELD for swap.example.com by "box-b" — a previous incarnation hasn't released; refusing to wake (retry after 1m0s)
2026/10/24 17:24:13 activator: swap.example.com: hold's pinned wake refused or abandoned: instance lease held by "box-b" (retry in 1m0s)
2026/10/24 17:24:13 activator: fenced.example.com: failed wake's generation 2116 is proven stopped; lease given back
2026/10/24 17:24:13 activator: unproven.example.com: generation 2118's child is not proven empty: the generation's child still holds a process
2026/10/24 17:24:13 activator: unproven.example.com: failed wake's generation 2118 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:13 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 17:24:13 activator: unproven.example.com: generation 2118's child is not proven empty: the generation's child still holds a process
2026/10/24 17:24:13 activator: unproven.example.com: failed wake's generation 2118 is proven stopped; lease given back
2026/10/24 17:24:13 activator: unproven.example.com hibernated
2026/10/24 17:24:13 activator: swap.example.com hibernated
2026/10/24 17:24:13 activator: SIDECAR restart for one.example.com failed: swap: lease lost (next attempt in 2s)
2026/10/24 17:24:13 activator: sidecar staged for awake swap.example.com — adopting
2026/10/24 17:24:13 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:13 activator: exited-foreign.example.com: failed wake's generation 2132 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 17:24:13 activator: exited-foreign.example.com: failed wake's generation 2132 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:13 activator: swap.example.com hibernated
2026/10/24 17:24:13 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: lease lost: lease renewal failed
2026/10/24 17:24:13 activator: swap.example.com hibernated
2026/10/24 17:24:13 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: socket dir: permission denied
2026/10/24 17:24:13 activator: c.example.com idle 1h0m0s — hibernating
2026/10/24 17:24:13 activator: c.example.com hibernated
2026/10/24 17:24:13 activator: swap.example.com: v1 not started: its build state could not be written: bucket: 503
2026/10/24 17:24:13 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:13 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:13 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: instance control: 502 mid-roll; the sweep tries again
2026/10/24 17:24:13 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/24 17:24:13 activator: h.example.com: kill carlos-instance-gen@h.example.com:2153.service after a failed start: stop timed out
2026/10/24 17:24:13 activator: one.example.com: wake failed and its generation is not proven stopped: start one.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2153.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/24 17:24:13 activator: instance c.example.com exited unexpectedly — recycling
2026/10/24 17:24:13 activator: c.example.com hibernated
2026/10/24 17:24:13 activator: swap.example.com hibernated
2026/10/24 17:24:13 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: the backend ran a different generation from the hold's choice: generation 2156 of v9 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_2738363679/004/versions/v2/app"; the hold chose v2 from "/tmp/TestSwapHoldWakeEventsCarryNoSwapTexta_mismatched_spawns_text_2738363679/004/versions/v2/app"
2026/10/24 17:24:13 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: processes remained; the sweep tries again
2026/10/24 17:24:13 activator: swap.example.com hibernated
2026/10/24 17:24:13 activator: swap.example.com: hold's pinned wake refused or abandoned: restore swap.example.com: replica unreadable
2026/10/24 17:24:13 activator: fenced.example.com: failed wake's generation 2160 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:13 activator: unproven.example.com: generation 2163's child is not proven empty: the generation's child still holds a process
2026/10/24 17:24:13 activator: unproven.example.com: failed wake's generation 2163 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:13 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: h: start unit: instance control: generation termination unproven: the unit exited before it was observed
2026/10/24 17:24:13 activator: unproven.example.com: generation 2163's child is not proven empty: the generation's child still holds a process
2026/10/24 17:24:13 activator: fenced.example.com: failed wake's generation 2160 is proven stopped; lease given back
2026/10/24 17:24:13 activator: unproven.example.com: failed wake's generation 2163 is proven stopped; lease given back
2026/10/24 17:24:13 activator: unproven.example.com hibernated
2026/10/24 17:24:13 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:13 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 17:24:13 activator: swap.example.com hibernated
2026/10/24 17:24:13 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: a stop-evidence operation holds the host
2026/10/24 17:24:13 activator: swap.example.com hibernated
2026/10/24 17:24:13 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: start g: activator: generation start failed and its termination is unproven
2026/10/24 17:24:13 activator: f6.example.com: failed wake's generation 0 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 17:24:13 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:13 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:24:13 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 17:24:13 activator: swap.example.com hibernated
2026/10/24 17:24:13 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:13 activator: swap.example.com hibernated
2026/10/24 17:24:13 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:24:13 activator: SIDECAR restart for one.example.com failed: the build state could not be written, so this build was not started: bucket refused the build state (next attempt in 2s)
2026/10/24 17:24:13 activator: sidecar for swap.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:13 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/24 17:24:13 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:13 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:13 activator: swap.example.com hibernated
2026/10/24 17:24:13 activator: swap.example.com: run run_a on retired generation 1099511627776 is not proven stopped: instance control: 502 mid-roll; the sweep tries again
2026/10/24 17:24:13 activator: LEASE LOST for c.example.com (lost) — standing down
2026/10/24 17:24:13 activator: c.example.com hibernated
2026/10/24 17:24:13 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:13 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:13 activator: x.example hibernated
2026/10/24 17:24:13 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:24:13 activator: swap.example.com hibernated
2026/10/24 17:24:13 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: staging: staged digest mismatch: have be0041e1a437dbea5b67dfe58bad04a0fc30d5036d469429f3534f39ec53d8b5, want c3bae8fb3a8bf395783d7f7f6ffcd4aadc797c95ce24b7e8e2a98af6b63d8f6d
2026/10/24 17:24:13 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:13 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:13 activator: x.example hibernated
2026/10/24 17:24:14 activator: unproven.example.com: failed wake's generation 2196 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:14 activator: unproven.example.com: retry stop of failed wake's generation 2196: cgroup.kill: the child is not empty
2026/10/24 17:24:14 activator: stop fence.example: stop refused
2026/10/24 17:24:14 activator: fence.example hibernated
2026/10/24 17:24:14 activator: one.example.com idle 1h0m0s — hibernating
2026/10/24 17:24:14 activator: one.example.com hibernated
2026/10/24 17:24:14 activator: unproven.example.com: generation 2196's child is not proven empty: the generation's child still holds a process
2026/10/24 17:24:14 activator: swap.example.com hibernated
2026/10/24 17:24:14 activator: unproven.example.com: failed wake's generation 2196 is proven stopped; lease given back
2026/10/24 17:24:14 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:14 activator: unproven.example.com hibernated
2026/10/24 17:24:14 activator: one.example.com idle 1h0m0s — hibernating
2026/10/24 17:24:14 activator: one.example.com hibernated
2026/10/24 17:24:14 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:24:14 activator: SIDECAR for one.example.com failed to start: the build state could not be written, so this build was not started: bucket refused the build state — serving without it (the sweep retries)
2026/10/24 17:24:14 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/24 17:24:14 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 17:24:14 activator: stop barrier.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 17:24:14 activator: barrier.example.com: generation 2199023255552's child is not proven empty: the child still holds a process
2026/10/24 17:24:14 activator: barrier.example.com hibernated
2026/10/24 17:24:14 activator: barrier.example.com hibernated
2026/10/24 17:24:14 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/24 17:24:14 activator: c.example.com serving v1 but v2 is adopted — cycling onto the new build
2026/10/24 17:24:14 activator: c.example.com hibernated
2026/10/24 17:24:14 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:14 activator: bot.example.com hibernated
2026/10/24 17:24:14 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:24:14 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:14 activator: x.example hibernated
2026/10/24 17:24:14 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/24 17:24:14 activator: barrier.example.com hibernated
2026/10/24 17:24:14 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd: not a directory; the sweep retries
2026/10/24 17:24:14 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd: not a directory; the sweep retries
2026/10/24 17:24:14 activator: worker first.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd: not a directory
2026/10/24 17:24:14 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd: not a directory; the sweep retries
2026/10/24 17:24:14 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd: not a directory; the sweep retries
2026/10/24 17:24:14 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/24 17:24:14 activator: one.example.com hibernated
2026/10/24 17:24:14 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:14 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd: not a directory; the sweep retries
2026/10/24 17:24:14 activator: worker bot.x.internal not spawned: the build it would exec (v1 /opt/carlos/versions/v1/app) could not be published first: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd: not a directory
2026/10/24 17:24:14 activator: publish generations to /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd/generations.json: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd: not a directory; the sweep retries
2026/10/24 17:24:14 activator: WORKER bot.x.internal failed to respawn (exited): worker build not published before its exec: open directory /tmp/TestWorkerSpawnRefusedWhenItsBuildCannotBePublished494000629/002/data/zdd: not a directory — the sweep's crash restart takes it from here
2026/10/24 17:24:14 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:14 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:14 activator: x.example hibernated
2026/10/24 17:24:14 activator: fence.example hibernated
2026/10/24 17:24:14 activator: fence.example hibernated
2026/10/24 17:24:14 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 17:24:14 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 17:24:14 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:14 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:14 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 17:24:14 activator: unproven.example.com: generation 2226's child is not proven empty: the generation's child still holds a process
2026/10/24 17:24:14 activator: unproven.example.com: failed wake's generation 2226 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:14 activator: unproven.example.com: wake failed and its generation is not proven stopped: start unproven.example.com: start unit: the unit exited before it was observed; activator: generation start failed and its termination is unproven: kill timed out
2026/10/24 17:24:14 activator: unproven.example.com: generation 2226's child is not proven empty: the generation's child still holds a process
2026/10/24 17:24:14 activator: swap.example.com hibernated
2026/10/24 17:24:14 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2228.service exited before it was observed
2026/10/24 17:24:14 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:14 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:14 activator: bot.x.internal hibernated
2026/10/24 17:24:14 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:14 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:14 activator: unproven.example.com: failed wake's generation 2226 is proven stopped; lease given back
2026/10/24 17:24:14 activator: unproven.example.com hibernated
2026/10/24 17:24:14 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:14 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 17:24:14 activator: bot.x.internal hibernated
2026/10/24 17:24:14 activator: worker bot6.example.com NOT respawned (test: the lease fails mid-preparation): swap: lease lost; abandoning it
2026/10/24 17:24:14 activator: stop sidecar nested-costop.example.com: activator: stop unproven; processes remain
2026/10/24 17:24:14 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 17:24:14 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/24 17:24:14 activator: one.example.com hibernated
2026/10/24 17:24:14 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/24 17:24:14 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:14 activator: c.example.com hibernated
2026/10/24 17:24:14 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:24:14 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 17:24:14 activator: undelegated.example.com idle 2m0s — hibernating
2026/10/24 17:24:14 activator: undelegated.example.com: stopped with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 17:24:14 activator: undelegated.example.com hibernated
2026/10/24 17:24:14 activator: stop nested-costop.example.com: 1 sidecar(s) it stopped have not exited; keeping the lease until they have
2026/10/24 17:24:14 activator: nested-costop.example.com hibernated
2026/10/24 17:24:14 activator: barrier.example.com idle 2h0m0s — hibernating
2026/10/24 17:24:14 activator: barrier.example.com hibernated
2026/10/24 17:24:14 activator: replica shop.example.com not started: it has no key of its home's and no control socket, so an app on the instance-key helper would mint its own
2026/10/24 17:24:14 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:14 activator: x.example hibernated
2026/10/24 17:24:14 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:14 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:24:14 activator: sidecar staged for awake tenant.test — adopting
2026/10/24 17:24:14 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/24 17:24:15 activator: sidecar staged for awake tenant.test — adopting
2026/10/24 17:24:15 activator: sidecar for tenant.test was down — restarted (backoff now 2s)
2026/10/24 17:24:15 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:15 activator: swap.example.com hibernated
2026/10/24 17:24:15 activator: h.example.com: kill carlos-instance-gen@h.example.com:2253.service after a failed start: stop timed out
2026/10/24 17:24:15 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: carlos-instance-gen@h.example.com:2253.service exited before it was observed; activator: generation start failed and its termination is unproven: stop timed out
2026/10/24 17:24:15 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 17:24:15 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 17:24:15 activator: bot.x.internal hibernated
2026/10/24 17:24:15 activator: stop fence.example: earlier process may still run
2026/10/24 17:24:15 activator: fence.example hibernated
2026/10/24 17:24:15 activator: fence.example hibernated
2026/10/24 17:24:15 activator: h.example.com: kill carlos-instance-gen@h.example.com:2255.service after a failed start: stop timed out
2026/10/24 17:24:15 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:15 activator: seeding h.example.com: /tmp/TestPlan14GenerationBackendAuxiliaryBoundaryclaim=falseseedpi2394304651/001/live/h.example.com seed --db /tmp/nb2108513153/data/instances/h.example.com/h.example.com.db
2026/10/24 17:24:15 activator: seeded h.example.com
2026/10/24 17:24:15 activator: seeding h.example.com: /tmp/TestPlan14GenerationBackendAuxiliaryBoundaryclaim=falseseedpi3738434050/001/live/h.example.com seed --db /tmp/nb3366360914/data/instances/h.example.com/h.example.com.db
2026/10/24 17:24:15 activator: seeded h.example.com
2026/10/24 17:24:15 activator: worker bot7.example.com NOT respawned (test: the lease fails at the exec): swap: lease lost; abandoning it
2026/10/24 17:24:15 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:15 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:24:15 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:24:15 activator: c.example.com is no longer hibernate-managed — releasing
2026/10/24 17:24:15 activator: c.example.com hibernated
before
2026/10/24 17:24:15 activator: worker bot5.example.com NOT respawned (test: the lease is lost): swap: lease lost; abandoning it
2026/10/24 17:24:15 activator: stop fence.example: stop refused
2026/10/24 17:24:15 activator: fence.example hibernated
2026/10/24 17:24:15 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/24 17:24:15 activator: barrier.example.com hibernated
2026/10/24 17:24:15 activator: control listener for shop.example.com: listen unix /tmp/TestARefusedLaunchLeavesNoBuildStateRecord1536053669/002/ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:24:15 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:15 activator: x.example hibernated
2026/10/24 17:24:15 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:15 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:24:15 activator: fence.example hibernated
2026/10/24 17:24:15 activator: worker bot.x.internal reported idle but is always-on (hibernate:false) — ignoring
2026/10/24 17:24:15 activator: f6.example.com: failed wake's generation 4 ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 17:24:15 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:15 activator: swap.example.com hibernated
2026/10/24 17:24:15 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/24 17:24:15 activator: waking.example hibernated
2026/10/24 17:24:15 activator: proven.example.com idle 2m0s — hibernating
2026/10/24 17:24:15 activator: proven.example.com hibernated
2026/10/24 17:24:15 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:15 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 17:24:15 activator: bot.x.internal hibernated
2026/10/24 17:24:15 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:24:15 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:15 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:24:15 activator: c.example.com hibernated
2026/10/24 17:24:15 activator: seeding seedpin.example.com: /tmp/TestADeclinedFlatSeedRunsThePinnedFile3293769473/001/versions/v1/app seed --db /tmp/TestADeclinedFlatSeedRunsThePinnedFile3293769473/002/seedpin.example.com.db
2026/10/24 17:24:15 activator: seeded seedpin.example.com
2026/10/24 17:24:15 activator: sidecar bot.x.internal exited cleanly — its work is done; parking it
2026/10/24 17:24:15 activator: bot.x.internal hibernated
2026/10/24 17:24:15 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/24 17:24:15 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912
2026/10/24 17:24:15 activator: LEASE LOST for barrier.example.com (renewal failed) — standing down
2026/10/24 17:24:15 activator: barrier.example.com hibernated
2026/10/24 17:24:15 activator: seeding shop.example.com: /tmp/TestARefusedLaunchLeavesNoBuildStateRecordseed_refused_beside_a4270183582/002/versions/v1/app seed --db /tmp/TestARefusedLaunchLeavesNoBuildStateRecordseed_refused_beside_a4270183582/003/shop.example.com.db
2026/10/24 17:24:15 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/24 17:24:15 activator: h.example.com: started carlos-instance-gen@h.example.com:6.service memory.max=536870912
2026/10/24 17:24:15 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:15 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/24 17:24:15 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:15 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/24 17:24:15 activator: x.example hibernated
2026/10/24 17:24:15 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:15 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:24:15 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 17:24:15 activator: stop fence.example: private backend detail
2026/10/24 17:24:15 activator: fence.example hibernated
2026/10/24 17:24:15 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:15 activator: WORKER bot.x.internal failed to respawn (exited): worker bot.x.internal: staged copy no longer matches the digest it was verified with — the sweep's crash restart takes it from here
2026/10/24 17:24:15 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/24 17:24:15 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/24 17:24:15 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:15 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:24:15 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 17:24:15 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:15 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:15 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:24:15 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 17:24:15 activator: swap.example.com hibernated
2026/10/24 17:24:15 activator: h.example.com: kill carlos-instance-gen@h.example.com:2292.service after a failed start: stop timed out
2026/10/24 17:24:15 activator: swap.example.com: wake failed and its generation is not proven stopped: start swap.example.com: h.example.com: start unit: the unit failed to start; activator: generation start failed and its termination is unproven: stop timed out
2026/10/24 17:24:15 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:15 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:24:15 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 17:24:15 activator: worker bot.x.internal exited — restarting (backoff now 4s)
2026/10/24 17:24:15 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:24:15 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 17:24:15 activator: f6.example.com idle 2m0s — hibernating
2026/10/24 17:24:15 activator: stop f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 17:24:15 activator: stop f6.example.com: its host cgroup is not proven empty; keeping the lease until it is
2026/10/24 17:24:16 activator: worker bot.x.internal exited — restarting (backoff now 8s)
2026/10/24 17:24:16 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:24:16 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 17:24:16 activator: f6.example.com hibernated
2026/10/24 17:24:16 activator: swap.example.com hibernated
2026/10/24 17:24:16 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: the unit failed to start
2026/10/24 17:24:16 activator: two.example.com not started: the executable it would run is outside the version store, so no build state can name it (settings spec 4.2); move the instance onto a promoted build
2026/10/24 17:24:16 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:24:16 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:16 activator: bot.x.internal hibernated
2026/10/24 17:24:16 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:16 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 17:24:16 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 17:24:16 activator: x.example source opened on the recovery probe's session; serving
2026/10/24 17:24:16 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:16 activator: WORKER bot.x.internal failed to respawn (converging onto a new build): exec: live symlink gone — the sweep's crash restart takes it from here
2026/10/24 17:24:16 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 17:24:16 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 17:24:16 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:16 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:24:16 activator: worker bot.x.internal respawned (exited), now running v2
2026/10/24 17:24:16 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 17:24:16 activator: worker bot.x.internal respawned (hit its max_run), now running v1
2026/10/24 17:24:16 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:16 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 17:24:16 activator: worker bot.x.internal hit its max_run; it serves v1 but live/bot.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/24 17:24:16 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 17:24:16 activator: seeding shop.example.com: /tmp/TestARefusedLaunchLeavesNoBuildStateRecorda_seed_that_ran_is_re1141851193/001/versions/v1/app seed --db /tmp/TestARefusedLaunchLeavesNoBuildStateRecorda_seed_that_ran_is_re1141851193/002/shop.example.com.db
2026/10/24 17:24:16 activator: swap.example.com hibernated
2026/10/24 17:24:16 activator: shop.example.com: the seed ran with no delegated cgroup; its leader's exit is the only proof there is
2026/10/24 17:24:16 activator: barrier.example.com hibernated
2026/10/24 17:24:16 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 17:24:16 activator: worker bot.x.internal respawned (hit its max_run), now running v2
2026/10/24 17:24:16 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:16 activator: x.example recovery probe's session is over (hostbox: selected authority changed); stopping it
2026/10/24 17:24:16 activator: x.example hibernated
2026/10/24 17:24:16 activator: stop sidecar fence.example: private backend detail
2026/10/24 17:24:16 activator: fence.example hibernated
2026/10/24 17:24:16 activator: kass.example.com idle 11s — hibernating
2026/10/24 17:24:16 activator: kass.example.com hibernated
2026/10/24 17:24:16 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:16 activator: swap.example.com hibernated
2026/10/24 17:24:16 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: cgroup: the host's parent cgroup holds a process directly: holds 1 process(es)
2026/10/24 17:24:16 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:16 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:16 activator: swap.example.com hibernated
2026/10/24 17:24:16 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/24 17:24:16 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 17:24:16 activator: bot.x.internal serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 17:24:16 activator: bot.x.internal hibernated
2026/10/24 17:24:16 activator: bot.x.internal idle 11s — hibernating
2026/10/24 17:24:16 activator: bot.x.internal hibernated
2026/10/24 17:24:16 activator: late.example.com idle 11s — hibernating
2026/10/24 17:24:16 activator: late.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/24 17:24:16 activator: late.example.com hibernated
2026/10/24 17:24:16 activator: swap.example.com hibernated
2026/10/24 17:24:16 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: h.example.com: start unit: instancectl: staged executable refused: digest mismatch
2026/10/24 17:24:16 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:16 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 17:24:16 activator: bot.x.internal hibernated
2026/10/24 17:24:16 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 17:24:16 activator: deaf.example.com idle 11s — hibernating
2026/10/24 17:24:16 activator: next-due probe deaf.example.com: dial unix: no such file — recording none (it sleeps until a request)
2026/10/24 17:24:16 activator: deaf.example.com hibernated
2026/10/24 17:24:16 activator: late.example.com has a scheduled delivery due — waking it
2026/10/24 17:24:16 activator: one.example.com: v3 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:24:16 activator: due.example.com idle 11s — hibernating
2026/10/24 17:24:16 activator: due.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/24 17:24:16 activator: due.example.com hibernated
2026/10/24 17:24:16 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 17:24:16 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 17:24:16 activator: bot.x.internal hibernated
2026/10/24 17:24:16 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 17:24:16 activator: due.example.com has a scheduled delivery due — waking it
2026/10/24 17:24:16 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 17:24:16 activator: bot.x.internal idle 11s — hibernating
2026/10/24 17:24:16 activator: bot.x.internal hibernated
2026/10/24 17:24:16 activator: due.example.com idle 11s — hibernating
2026/10/24 17:24:16 activator: due.example.com hibernated
2026/10/24 17:24:16 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 17:24:16 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 17:24:16 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 17:24:16 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 17:24:16 activator: swap.example.com hibernated
2026/10/24 17:24:16 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:24:16 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/24 17:24:16 activator: one.example.com is no longer hibernate-managed — releasing
2026/10/24 17:24:16 activator: one.example.com hibernated
2026/10/24 17:24:16 activator: barrier.example.com hibernated
2026/10/24 17:24:16 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 17:24:16 activator: bot.x.internal has a scheduled delivery due — waking it
2026/10/24 17:24:16 activator: shop.example.com: generation 4's child is not proven empty: kill gen: timeout
2026/10/24 17:24:16 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:16 activator: bot.x.internal idle 11s — hibernating
2026/10/24 17:24:16 activator: bot.x.internal hibernated
2026/10/24 17:24:16 activator: final checkpoint fence.example: private backend detail
2026/10/24 17:24:16 activator: fence.example hibernated
2026/10/24 17:24:16 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:16 activator: own.example.com idle 1m0s — hibernating
2026/10/24 17:24:16 activator: own.example.com hibernated
2026/10/24 17:24:16 activator: always-on sidecar bot.x.internal is not running — starting it
2026/10/24 17:24:16 activator: pending-f6.example.com idle 2m0s — hibernating
2026/10/24 17:24:16 activator: stop pending-f6.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 17:24:16 activator: stop pending-f6.example.com: its host cgroup is not proven empty: the host cgroup still holds a process
2026/10/24 17:24:16 activator: bot.x.internal hibernated
2026/10/24 17:24:16 activator: pending-f6.example.com hibernated
2026/10/24 17:24:16 activator: mid.example.com has a scheduled delivery due — waking it
2026/10/24 17:24:16 activator: always-on sidecar bot00.x.internal is not running — starting it
2026/10/24 17:24:16 activator: always-on sidecar bot01.x.internal is not running — starting it
2026/10/24 17:24:16 activator: always-on sidecar bot02.x.internal is not running — starting it
2026/10/24 17:24:16 activator: always-on sidecar bot03.x.internal is not running — starting it
2026/10/24 17:24:16 activator: always-on sidecar bot04.x.internal is not running — starting it
2026/10/24 17:24:16 activator: always-on sidecar bot05.x.internal is not running — starting it
2026/10/24 17:24:16 activator: always-on sidecar bot06.x.internal is not running — starting it
2026/10/24 17:24:16 activator: always-on sidecar bot07.x.internal is not running — starting it
2026/10/24 17:24:16 activator: always-on sidecar bot08.x.internal is not running — starting it
2026/10/24 17:24:16 activator: always-on sidecar bot09.x.internal is not running — starting it
2026/10/24 17:24:16 activator: always-on sidecar bot10.x.internal is not running — starting it
2026/10/24 17:24:16 activator: LEASE LOST for bot.x.internal (lease renewal failed) — standing down
2026/10/24 17:24:16 activator: bot.x.internal hibernated
2026/10/24 17:24:16 activator: mid.example.com is no longer hibernate-managed — releasing (wake finished first)
2026/10/24 17:24:16 activator: mid.example.com hibernated
2026/10/24 17:24:16 activator: one.example.com hibernated
2026/10/24 17:24:16 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:16 activator: worker bot.x.internal NOT respawned (converging onto a new build): instance config unavailable on this box: cannot open sealed config — standing down and releasing its lease
2026/10/24 17:24:16 activator: bot.x.internal hibernated
2026/10/24 17:24:16 activator: bot.x.internal hibernated
2026/10/24 17:24:16 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:16 activator: bot.x.internal hibernated
2026/10/24 17:24:16 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2444797949/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2444797949/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 17:24:16 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2444797949/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedplanned_wake2444797949/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 17:24:16 activator: one.example.com: sidecar starting on the instance's own build v1 (/opt/carlos/versions/v1/app): this box will not start v2, the build its live link names
2026/10/24 17:24:16 activator: sidecar for one.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:17 activator: shop.example.com: failed wake's generation 4 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 17:24:17 activator: flip.example.com is no longer hibernate-managed — releasing
2026/10/24 17:24:17 activator: flip.example.com next scheduled delivery at 2023-11-14T22:13:15Z — will wake for it
2026/10/24 17:24:17 activator: flip.example.com hibernated
2026/10/24 17:24:17 activator: SIDECAR for partial-sidecar.example failed to start: sidecar started then failed — retaining returned process
2026/10/24 17:24:17 activator: barrier.example.com hibernated
2026/10/24 17:24:17 activator: partial-sidecar.example hibernated
2026/10/24 17:24:17 activator: fence.example hibernated
2026/10/24 17:24:17 activator: one.example.com hibernated
2026/10/24 17:24:17 activator: one.example.com hibernated
2026/10/24 17:24:17 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/24 17:24:17 activator: tickheld.bab.oncarlos.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 17:24:17 activator: tickheld.bab.oncarlos.com hibernated
2026/10/24 17:24:17 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:17 activator: swap.example.com hibernated
2026/10/24 17:24:17 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a cgroup that took cgroup.kill could not be made fresh before a spawn: carlos-exec@h.example.com: device or resource busy
2026/10/24 17:24:17 activator: untimed.bab.oncarlos.com idle 16m0s — hibernating
2026/10/24 17:24:17 activator: untimed.bab.oncarlos.com hibernated
2026/10/24 17:24:17 activator: fence.example hibernated
2026/10/24 17:24:17 activator: shutdown-pending.example.com idle 2m0s — hibernating
2026/10/24 17:24:17 activator: stop shutdown-pending.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 17:24:17 activator: faroff.bab.oncarlos.com idle 16m0s — hibernating
2026/10/24 17:24:17 activator: faroff.bab.oncarlos.com hibernated
2026/10/24 17:24:17 activator: one.example.com hibernated
2026/10/24 17:24:17 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor2696917379/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor2696917379/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 17:24:17 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor2696917379/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedoverlap_successor2696917379/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 17:24:17 activator: h.example.com: started carlos-instance-cmd@h.example.com.service for run run_a
2026/10/24 17:24:17 activator: bot.x.internal hibernated
2026/10/24 17:24:17 activator: h.example.com idle 2m0s — hibernating
2026/10/24 17:24:17 activator: stop h.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 17:24:17 activator: h.example.com hibernated
2026/10/24 17:24:17 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, nor the instance's own build v1: a failure record holds this build: v1
2026/10/24 17:24:17 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/24 17:24:17 activator: barrier.example.com hibernated
2026/10/24 17:24:17 activator: SIDECAR restart for partial-sidecar.example failed: sidecar started then failed — retaining returned process
2026/10/24 17:24:17 activator: fence.example hibernated
2026/10/24 17:24:17 activator: shutdown-pending.example.com hibernated
2026/10/24 17:24:17 activator: partial-sidecar.example hibernated
2026/10/24 17:24:17 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:17 activator: f6-layout.example.com: failed wake's generation 2359 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 17:24:17 activator: f6-layout.example.com: failed wake's generation 2359 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:17 activator: nested-stop.example.com idle 2m0s — hibernating
2026/10/24 17:24:17 activator: nested-stop.example.com hibernated
2026/10/24 17:24:17 activator: stuck.example.com idle 2m0s — hibernating
2026/10/24 17:24:17 activator: stop stuck.example.com: activator: stop unproven; processes remain; keeping the lease until termination is proven
2026/10/24 17:24:17 activator: c.example.com hibernated
2026/10/24 17:24:17 activator: fence.example hibernated
2026/10/24 17:24:17 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:17 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:24:17 activator: worker bot.x.internal NOT respawned (exited): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/24 17:24:17 activator: bot.x.internal hibernated
2026/10/24 17:24:17 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:17 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 17:24:17 activator: worker bot-a.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:17 activator: worker bot-a.x.internal respawned (converging onto a new build), now running v2
2026/10/24 17:24:17 activator: bot.x.internal hit its max_run of 1m0s — stopping
2026/10/24 17:24:17 activator: bot.x.internal hibernated
2026/10/24 17:24:17 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawngated_build_is_the_ex2562182448/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:24:17 activator: one.example.com hibernated
2026/10/24 17:24:17 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawna_held_build_starts_n1138361125/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:24:17 activator: worker bot-b.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:17 activator: worker bot-b.x.internal respawned (converging onto a new build), now running v2
2026/10/24 17:24:17 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished1207252737/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished1207252737/001/zdd: not a directory; the sweep retries
2026/10/24 17:24:17 activator: publish generations to /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished1207252737/001/zdd/generations.json: open directory /tmp/TestAReplannedBuildIsPublishedBeforeItExecsunpublished1207252737/001/zdd: not a directory; the sweep retries
2026/10/24 17:24:17 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake254888207/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake254888207/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 17:24:17 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake254888207/005/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedholds_pinned_wake254888207/005/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 17:24:17 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:17 activator: instance bots.example.com exited unexpectedly — recycling
2026/10/24 17:24:17 activator: bots.example.com hibernated
2026/10/24 17:24:17 activator: shop.example.com: seed placement not recorded yet: record unreachable
2026/10/24 17:24:17 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:17 activator: WORKER bot.x.internal failed to respawn (exited): fork/exec: resource temporarily unavailable — the sweep's crash restart takes it from here
2026/10/24 17:24:17 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/24 17:24:17 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:17 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:17 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:17 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:17 activator: one.example.com: live/one.example.com names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/24 17:24:17 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:17 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:24:17 activator: worker bot.x.internal NOT respawned (exited): refused at its exec: the build state could not be written, so this build was not started: bucket refused the build state
2026/10/24 17:24:17 activator: sidecar for bots.example.com was down — restarted (backoff now 8s)
2026/10/24 17:24:17 activator: one.example.com: sidecar not started: this box will not start v2, the build its live link names, and the instance's own build v1 cannot be named: staged copy unreadable
2026/10/24 17:24:17 activator: SIDECAR restart for one.example.com failed: a failure record holds this build: v2 (next attempt in 2s)
2026/10/24 17:24:17 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/24 17:24:17 activator: barrier.example.com hibernated
2026/10/24 17:24:17 activator: stop fence.example: activator: stop evidence unavailable
2026/10/24 17:24:17 activator: fence.example hibernated
2026/10/24 17:24:17 activator: x.example recovery probe healthy; held until the source opens
2026/10/24 17:24:17 activator: sidecar for bots.example.com was down — restarted (backoff now 2s)
2026/10/24 17:24:17 activator: control listener for live.example.com: listen unix /tmp/TestALiveExecGateGatesTheFlatLiveLinkSpawnno_gate_is_todays_sp4294537832/003/live.example.com.sock.control: bind: invalid argument (continuing without it)
2026/10/24 17:24:17 activator: swap.example.com hibernated
2026/10/24 17:24:17 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: cgroup: a generation's limits could not be applied: h.example.com: the placement shim is not available
2026/10/24 17:24:17 activator: f6-layout.example.com: failed wake's generation 2380 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 17:24:17 activator: f6-layout.example.com: failed wake's generation 2380 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:17 activator: sidecar for bots.example.com was down — restarted (backoff now 4s)
2026/10/24 17:24:17 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 17:24:17 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 17:24:17 activator: one.example.com serving v1 but live symlink is now v3 — cycling onto the new build
2026/10/24 17:24:17 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:17 activator: one.example.com hibernated
2026/10/24 17:24:17 activator: c.example.com hibernated
2026/10/24 17:24:18 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/24 17:24:18 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:18 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 17:24:18 activator: worker bot.x.internal hit its max_run; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:24:18 activator: worker bot.x.internal NOT respawned (hit its max_run): instance config unavailable on this box: no key — standing down and releasing its lease
2026/10/24 17:24:18 activator: bot.x.internal hibernated
2026/10/24 17:24:18 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:18 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it recycles its captured build v1
2026/10/24 17:24:18 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:18 activator: worker bot.x.internal serving v1 but live symlink is now v3 — respawning onto the new build
2026/10/24 17:24:18 activator: worker bot.x.internal respawned (converging onto a new build), now running v3
2026/10/24 17:24:18 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:18 activator: bots.example.com idle 11s — hibernating
2026/10/24 17:24:18 activator: bots.example.com hibernated
2026/10/24 17:24:18 activator: bot.x.internal idle 31s — hibernating
2026/10/24 17:24:18 activator: bot.x.internal hibernated
2026/10/24 17:24:18 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:18 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 17:24:18 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 17:24:18 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1782935923/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1782935923/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 17:24:18 activator: x.example: hold's pinned wake refused or abandoned: x.example: refusing to wake: staged build not published before its exec
2026/10/24 17:24:18 activator: publish generations to /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1782935923/003/not-a-dir/zdd/generations.json: open directory /tmp/TestAnUnpublishedBuildIsNeverSpawnedclaimed_planned_wake1782935923/003/not-a-dir/zdd: not a directory; the sweep retries
2026/10/24 17:24:18 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 17:24:18 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 17:24:18 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 17:24:18 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 17:24:18 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 17:24:18 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 17:24:18 activator: LEASE LOST for one.example.com (lease renewal failed) — standing down
2026/10/24 17:24:18 activator: one.example.com hibernated
2026/10/24 17:24:18 activator: c.example.com hibernated
2026/10/24 17:24:18 activator: final checkpoint fence.example: activator: stop evidence unavailable
2026/10/24 17:24:18 activator: fence.example hibernated
2026/10/24 17:24:18 activator: bot.x.internal idle 11s — hibernating
2026/10/24 17:24:18 activator: bot.x.internal hibernated
2026/10/24 17:24:18 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 17:24:18 activator: one.example.com hibernated
2026/10/24 17:24:18 activator: old.shop.example.com: route refreshed: db "/data/instances/old/old.shop.example.com.db" -> "" socket "/run/carlos/instances/old/app.sock" -> ""
2026/10/24 17:24:18 activator: wake old.shop.example.com: old.shop.example.com: the route changed before the wake: the activator no longer manages it
2026/10/24 17:24:18 activator: f6-layout.example.com: failed wake's generation 2394 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 17:24:18 activator: f6-layout.example.com: failed wake's generation 2394 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:18 activator: two.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 17:24:18 activator: two.example.com hibernated
2026/10/24 17:24:18 activator: worker bot.x.internal NOT respawned (converging onto a new build): swap: instance not awake; abandoning it
2026/10/24 17:24:18 activator: bot.x.internal hibernated
2026/10/24 17:24:18 activator: barrier.example.com is no longer hibernate-managed — releasing
2026/10/24 17:24:18 activator: barrier.example.com hibernated
2026/10/24 17:24:18 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 17:24:18 activator: one.example.com serving v1 but live symlink is now v2 — cycling onto the new build
2026/10/24 17:24:18 activator: one.example.com hibernated
2026/10/24 17:24:18 activator: one.example.com hibernated
2026/10/24 17:24:18 activator: one.example.com idle 11s — hibernating
2026/10/24 17:24:18 activator: one.example.com hibernated
2026/10/24 17:24:18 activator: one.example.com idle 11s — hibernating
2026/10/24 17:24:18 activator: one.example.com hibernated
2026/10/24 17:24:18 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it keeps running v1
2026/10/24 17:24:18 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: it stays down until a release attempt or a wake starts it
2026/10/24 17:24:18 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: SIDECAR for botless.example.com failed to start: exec: live symlink gone — serving without it (the sweep retries)
2026/10/24 17:24:18 activator: SIDECAR restart for botless.example.com failed: exec: live symlink gone (next attempt in 2s)
2026/10/24 17:24:18 activator: bots.example.com idle 11s — hibernating
2026/10/24 17:24:18 activator: bots.example.com hibernated
2026/10/24 17:24:18 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:24:18 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:18 activator: sidecar for botless.example.com was down — restarted (backoff now 4s)
2026/10/24 17:24:18 activator: worker bot.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:18 activator: worker bot.x.internal respawned (converging onto a new build), now running v2
2026/10/24 17:24:18 activator: one.example.com serving v1 but v2 is adopted — cycling onto the new build
2026/10/24 17:24:18 activator: one.example.com hibernated
2026/10/24 17:24:18 activator: one.example.com idle 11s — hibernating
2026/10/24 17:24:18 activator: one.example.com hibernated
2026/10/24 17:24:18 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:18 activator: squatted.example.com: child exited during the health wait (external process serving its socket?) — not normalizing the socket mode
2026/10/24 17:24:18 activator: fence.example hibernated
2026/10/24 17:24:18 activator: f6-layout.example.com: failed wake's generation 2413 had no cgroup of its own, and its host cgroup is not proven empty: carlos-exec@<host> still holds a process
2026/10/24 17:24:18 activator: f6-layout.example.com: failed wake's generation 2413 is not proven stopped; keeping the lease until it is
2026/10/24 17:24:18 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 17:24:18 activator: stop c.example.com: not every process of the entry is proven gone; keeping the lease until it is
2026/10/24 17:24:18 activator: h.example.com: started carlos-instance@h.example.com.service memory.max=536870912
2026/10/24 17:24:18 activator: h.example.com: unit restarted outside the activator (invocation inv2, ours inv1); stopping it
2026/10/24 17:24:18 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 17:24:18 activator: barrier.example.com hibernated
2026/10/24 17:24:18 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: worker bot.x.internal exited; a queued release attempt makes the spawn instead of a recycle
2026/10/24 17:24:18 activator: reborn.example.com idle 11s — hibernating
2026/10/24 17:24:18 activator: reborn.example.com next scheduled delivery at 2023-11-14T22:13:50Z — will wake for it
2026/10/24 17:24:18 activator: reborn.example.com hibernated
2026/10/24 17:24:18 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: worker bot.x.internal exited; no captured build to recycle onto, so it respawns from live/bot.x.internal
2026/10/24 17:24:18 activator: worker bot.x.internal respawned (exited), now running 
2026/10/24 17:24:18 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 17:24:18 activator: reborn.example.com has a scheduled delivery due — waking it
2026/10/24 17:24:18 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 17:24:18 activator: c.example.com hibernated
2026/10/24 17:24:18 activator: worker bot1.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: worker bot1.x.internal exited; it serves v1 but live/bot1.x.internal has named v2 for at least 1m30s, so it converges onto the new build instead of recycling
2026/10/24 17:24:18 activator: worker bot2.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:18 activator: worker bot2.x.internal exited; live/bot2.x.internal names v2, but this sweep's converge budget is spent, so it recycles its captured build v1 and converges on a later sweep
2026/10/24 17:24:18 activator: worker bot2.x.internal respawned (exited), now running v1
2026/10/24 17:24:18 activator: worker bot1.x.internal respawned (converging onto a new build), now running v2
2026/10/24 17:24:18 activator: one.example.com idle 5m1s — hibernating
2026/10/24 17:24:18 activator: one.example.com hibernated
2026/10/24 17:24:18 activator: worker bot2.x.internal serving v1 but live symlink is now v2 — respawning onto the new build
2026/10/24 17:24:18 activator: worker bot2.x.internal respawned (converging onto a new build), now running v2
2026/10/24 17:24:18 activator: bot.x.internal idle 1m31s — hibernating
2026/10/24 17:24:18 activator: bot.x.internal hibernated
2026/10/24 17:24:19 activator: c.example.com hibernated
2026/10/24 17:24:19 activator: one.example.com hibernated
2026/10/24 17:24:19 activator: one.example.com idle 1m0s — hibernating
2026/10/24 17:24:19 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 17:24:19 activator: bot.x.internal: live/bot.x.internal names v2, which this box will not start (a failure record holds this build: v2), so it does not move onto it: the running child keeps running
2026/10/24 17:24:19 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 17:24:19 activator: c.example.com idle 1h0m0s — hibernating
2026/10/24 17:24:19 activator: always-on sidecar bot.x.internal hit its max_run of 30s — recycling its child
2026/10/24 17:24:19 activator: c.example.com hibernated
2026/10/24 17:24:19 activator: one.example.com idle 11s — hibernating
2026/10/24 17:24:19 activator: one.example.com hibernated
2026/10/24 17:24:19 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:19 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:19 activator: swap.example.com hibernated
2026/10/24 17:24:19 activator: swap.example.com: hold's pinned wake refused or abandoned: start swap.example.com: swap: superseded by a newer attempt
2026/10/24 17:24:19 activator: one.example.com hibernated
2026/10/24 17:24:19 activator: barrier.example.com hibernated
2026/10/24 17:24:19 activator: worker bot.x.internal exited — restarting (backoff now 2s)
2026/10/24 17:24:19 activator: worker bot.x.internal exited; live/bot.x.internal names v2, but the release scheduler owns build changes, so it recycles its captured build v1
2026/10/24 17:24:19 activator: worker bot.x.internal respawned (exited), now running v1
2026/10/24 17:24:19 activator: one.example.com: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:24:19 activator: one.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:24:19 activator: bot.x.internal: v1 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:24:19 activator: bot.x.internal idle 1m1s — hibernating
2026/10/24 17:24:19 activator: bot.x.internal hibernated
2026/10/24 17:24:19 activator: swap.example.com: v2 not started: its build state could not be written: bucket refused the build state
2026/10/24 17:24:19 activator: swap.example.com hibernated
2026/10/24 17:24:19 activator: c.example.com idle 1h0m0s — hibernating
2026/10/24 17:24:19 activator: c.example.com hibernated
2026/10/24 17:24:19 activator: swap.example.com hibernated
2026/10/24 17:24:19 activator: c.example.com idle 1h0m0s — hibernating
2026/10/24 17:24:19 activator: c.example.com: run run_a is not proven stopped: processes remained
2026/10/24 17:24:19 activator: stop c.example.com: a run is not proven stopped; keeping replication and the lease until it is
2026/10/24 17:24:19 activator: c.example.com hibernated
2026/10/24 17:24:19 activator: swap.example.com hibernated
FAIL
FAIL	github.com/carlosframework/platform/internal/activator	60.490s
ok  	github.com/carlosframework/platform/internal/agentsock	0.060s
ok  	github.com/carlosframework/platform/internal/alerting	0.014s
ok  	github.com/carlosframework/platform/internal/analytics	0.045s
ok  	github.com/carlosframework/platform/internal/appconfig	2.891s
ok  	github.com/carlosframework/platform/internal/apperr	0.176s
ok  	github.com/carlosframework/platform/internal/applog	0.233s
ok  	github.com/carlosframework/platform/internal/awsinv	0.010s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.013s
ok  	github.com/carlosframework/platform/internal/boxid	1.353s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.011s
ok  	github.com/carlosframework/platform/internal/caps	0.027s
ok  	github.com/carlosframework/platform/internal/certwarm	0.011s
ok  	github.com/carlosframework/platform/internal/cgroup	0.855s
ok  	github.com/carlosframework/platform/internal/changelog	0.566s
ok  	github.com/carlosframework/platform/internal/claimactivation	1.401s
ok  	github.com/carlosframework/platform/internal/clamd	0.068s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	8.549s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.029s
ok  	github.com/carlosframework/platform/internal/dbquery	0.012s
ok  	github.com/carlosframework/platform/internal/dbreads	2.152s
?   	github.com/carlosframework/platform/internal/dbreads/dbreadstest	[no test files]
ok  	github.com/carlosframework/platform/internal/dbverify	0.017s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.016s
ok  	github.com/carlosframework/platform/internal/docsite	0.016s
ok  	github.com/carlosframework/platform/internal/domains	3.059s
ok  	github.com/carlosframework/platform/internal/doorkey	0.016s
ok  	github.com/carlosframework/platform/internal/economics	0.015s
ok  	github.com/carlosframework/platform/internal/edge	45.462s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.010s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.139s
ok  	github.com/carlosframework/platform/internal/gate	1.661s
ok  	github.com/carlosframework/platform/internal/genselect	0.035s
ok  	github.com/carlosframework/platform/internal/geo	0.013s
ok  	github.com/carlosframework/platform/internal/geoip	0.052s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.194s
ok  	github.com/carlosframework/platform/internal/host	141.352s
ok  	github.com/carlosframework/platform/internal/hostactors	6.367s
ok  	github.com/carlosframework/platform/internal/hostactors/awsmembership	0.005s
ok  	github.com/carlosframework/platform/internal/hostbox	2.925s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.831s
ok  	github.com/carlosframework/platform/internal/hostenv	0.007s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.013s
ok  	github.com/carlosframework/platform/internal/hostexec	5.243s
ok  	github.com/carlosframework/platform/internal/hostfence	0.068s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.289s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.020s
ok  	github.com/carlosframework/platform/internal/hostguard	1.044s
ok  	github.com/carlosframework/platform/internal/hostops	1.900s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.004s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.004s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.619s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.186s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.003s
ok  	github.com/carlosframework/platform/internal/infraview	0.004s
ok  	github.com/carlosframework/platform/internal/instancectl	5.240s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.821s
ok  	github.com/carlosframework/platform/internal/instancefs	0.005s
ok  	github.com/carlosframework/platform/internal/instancekey	0.006s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.012s
ok  	github.com/carlosframework/platform/internal/layout	1.820s
ok  	github.com/carlosframework/platform/internal/lbhealth	0.006s
ok  	github.com/carlosframework/platform/internal/lease	10.269s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.468s
ok  	github.com/carlosframework/platform/internal/ledger	0.016s
ok  	github.com/carlosframework/platform/internal/mail	0.221s
ok  	github.com/carlosframework/platform/internal/maintenance	0.098s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.519s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.131s
ok  	github.com/carlosframework/platform/internal/metrics	0.004s
ok  	github.com/carlosframework/platform/internal/moves	22.467s
ok  	github.com/carlosframework/platform/internal/moves/archive	1.006s
ok  	github.com/carlosframework/platform/internal/objstore	0.330s
ok  	github.com/carlosframework/platform/internal/ops	0.196s
ok  	github.com/carlosframework/platform/internal/park	0.293s
ok  	github.com/carlosframework/platform/internal/plans	0.009s
ok  	github.com/carlosframework/platform/internal/privatefile	0.003s
ok  	github.com/carlosframework/platform/internal/provision	23.356s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.010s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.581s
ok  	github.com/carlosframework/platform/internal/relay	7.190s
ok  	github.com/carlosframework/platform/internal/release	25.554s
ok  	github.com/carlosframework/platform/internal/rollout	3.768s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.006s
ok  	github.com/carlosframework/platform/internal/rootaws	0.007s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.061s
ok  	github.com/carlosframework/platform/internal/rootstate	0.028s
ok  	github.com/carlosframework/platform/internal/runs	0.596s
ok  	github.com/carlosframework/platform/internal/s3compat	0.004s
ok  	github.com/carlosframework/platform/internal/safefile	0.004s
ok  	github.com/carlosframework/platform/internal/schedule	2.646s
ok  	github.com/carlosframework/platform/internal/seeds	0.020s
ok  	github.com/carlosframework/platform/internal/sim/engines	2.636s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	10.500s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.657s
ok  	github.com/carlosframework/platform/internal/status	0.072s
ok  	github.com/carlosframework/platform/internal/steering	0.005s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.299s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.005s
ok  	github.com/carlosframework/platform/internal/unixdial	0.003s
ok  	github.com/carlosframework/platform/internal/updatewords	0.008s
ok  	github.com/carlosframework/platform/internal/usage	0.006s
ok  	github.com/carlosframework/platform/internal/version	0.003s
ok  	github.com/carlosframework/platform/internal/vet	1.648s
FAIL
=== internal_console@0 (79 s)
ok  	github.com/carlosframework/platform/internal/console	77.911s
=== internal_console@1 (41 s)
ok  	github.com/carlosframework/platform/internal/console	39.677s
=== internal_console@2 (45 s)
ok  	github.com/carlosframework/platform/internal/console	43.773s
=== internal_console@3 (42 s)
ok  	github.com/carlosframework/platform/internal/console	41.150s
=== internal_console@4 (45 s)
ok  	github.com/carlosframework/platform/internal/console	43.906s
=== internal_console@5 (42 s)
ok  	github.com/carlosframework/platform/internal/console	40.440s
=== internal_console@6 (40 s)
ok  	github.com/carlosframework/platform/internal/console	38.153s
=== internal_console@7 (36 s)
ok  	github.com/carlosframework/platform/internal/console	34.671s
=== root@0 (108 s)
ok  	github.com/carlosframework/platform	106.815s
=== root@1 (89 s)
ok  	github.com/carlosframework/platform	87.325s
=== root@2 (81 s)
ok  	github.com/carlosframework/platform	79.910s

FAILED: rest
make: *** [Makefile:19: test] Error 1

amadan: step gate failed: exit status 2

amadan: job failed at step gate — its output is above, not at the tail of this log

diff --git a/docs/superpowers/notes/census-proposal-sec-mail-user-boundary.md b/docs/superpowers/notes/census-proposal-sec-mail-user-boundary.md

index 3ff18e578..2594bfb12 100644

--- a/docs/superpowers/notes/census-proposal-sec-mail-user-boundary.md

+++ b/docs/superpowers/notes/census-proposal-sec-mail-user-boundary.md

@@ -60,5 +60,19 @@ Seal: `65857c29501dc7676f52921bea79d08217542ecb1d775c67d20a3460c851d831`

(static-redirects, checked first) to

`d82a9191c6be745d34211a8add6ac0e63dc35b28c3f6bcde99cdbc3f67154f62`.

+## Claim authority census (TestClaimAuthoritySourceCensus)

+`internal/console/testdata/claim-authority-sites.txt` gains one line, 151 to

+152 calls:

+ ../../cmd/console/main.go|serve|mail.NewAWSProvisioner(ctx, region, mailBoundary)

+There was no old entry to replace. On main, `serve` handed

+`mail.NewAWSProvisioner` to `mailBackends` as a function value, which the

+census does not count as a call. The boundary has to reach the constructor, so

+`serve` now wraps it in a closure, and the closure's call is the site. It is

+the same constructor `mailBackends` already called through its parameter,

+with one more argument, the boundary ARN.

Record-write census (9B8): unchanged (the provision package's census tests

pass without an overlay). Claim routes: unchanged. go.mod: unchanged.

diff --git a/internal/console/testdata/claim-authority-sites.txt b/internal/console/testdata/claim-authority-sites.txt

index e4c8047d5..f3d26689e 100644

--- a/internal/console/testdata/claim-authority-sites.txt

+++ b/internal/console/testdata/claim-authority-sites.txt

@@ -3,6 +3,7 @@

../../cmd/console/main.go|openStore|release.NewDeploymentS3Store(context.Background(), b)

../../cmd/console/main.go|serve|lease.NewVersionedS3(context.Background(), buckets)

../../cmd/console/main.go|serve|release.NewS3StoreIn(context.Background(), dbBucket, dbRegion)

+../../cmd/console/main.go|serve|mail.NewAWSProvisioner(ctx, region, mailBoundary)

../../cmd/console/main.go|storeBackend|objstore.NewAWSProvisioner(context.Background(), region, endpoint, boundary)

../../cmd/console/main.go|storeScannerFor|objstore.NewClamAVScanner(context.Background(), d.Region, d.Endpoint, addr, maxBytes, signer)

../hostops/app_mutation_runner.go|RunAppMutation|b.control.BeginExclusive(ctx, account, app, a.Value.AppID, AppOperation{ID: id, Kind: OpAppMutation, AppID: a.Value.AppID, Phase: "issued", Intent: intent}, owner)