carlos / platform Public

Sixth dead run: 4997a30d, "step gate failed: signal: terminated" after 53 seconds of the gate step (updated 20:09:35Z). That is too early for any test load: at 53 seconds the gate has built and is listing packages, and the e2e witnesses now start only after every other job has finished. main's run on 0d9e81dc passed at 19:36Z with the gate taking 178 seconds, so whatever sends the TERM reaches this branch's jobs and not main's; two things I would look at on the runner: whether more than one runner claims carlos/platform jobs (the repo requires nothing of its runners) and whether anything restarts the runner or its sandbox during a job. An agent account cannot list the runner's requests or offers from here.

Two findings from a local run of this branch's own gate (a clean export of a1ac66fc, nice 19, on a DOCS-T1 at load 40-60, so the numbers are inflated but the shape holds):

1. The 30-minute timeouts have a cause on the branch. The e2e witnesses ran as one test per name, and one of them, TestEveryBoundaryRecovers, held the whole boundary x fault x second-crash matrix (about 255 subtests): 1492 of the suite's 2491 seconds under load, and 60% of the suite in any shard the name deal put it in. With the console shards ahead of it (710-850 s each under this load; main's whole gate takes 178 s on the runner) the job could not fit 30 minutes. Fix pushed as 7fbeecbc: one top-level test per boundary (TestBoundaryRecovers<Name>, 17 of them), TestEveryBoundaryRecovers kept as the completeness check, and the e2e stage dealt across four shards instead of two. Its CI run is the test of this.

1. The "signal: terminated" runs are still unexplained and look like a second thing: twice in a row (4997a30d, a1ac66fc) the gate died 48-53 seconds in, while every other branch's gate that hour ran 160-214 s to completion. At 50 s the gate is listing packages and starting the test shards; nothing in the branch's diff against main adds a kill, a signal or a process-group call. If 7fbeecbc's run dies the same way, the runner's journal around the kill is the only place left to look.

The gate now narrates (283ffb29), and the first narrated run says where the kill lands:

gate: 2026-10-23T21:49:54Z on aarch64, 32 cpus, 57279 MB available, go1.26.8 linux/arm64 gate: +0s start launchctx / vet / gofmt gate: +8s start rest gate: +49s start internal_console@0 .. @7 gate: +51s start root@0 .. @2 amadan: step gate failed: signal: terminated (step elapsed 90 s)

So the runner is the arm64 32-cpu box (amadan's cloud runner, by the shape), the module cache was cold (the log before that is all go: downloading), and the TERM arrives within about 40 seconds of the eleven test shards starting their compiles, before any of them prints. The gate's own TERM trap did not get to print, which means the signal did not reach the gate shell first: whatever sent it killed the step from outside or at the top of the tree. Nothing in this branch's diff against main adds a kill, a signal, a Setpgid or a Setsid, and the kill-using test files are unchanged. The same gate on main ran in 178 s today.

Three places I would look on that box, in order: the runner's journal at 21:50:30Z-21:51:30Z today (and 20:09:35Z, 20:23:46Z); any pids/memory limit on the runner's unit (eleven concurrent go test compiles on 32 cpus is the moment the task count jumps); and whether a second runner also claims carlos/platform jobs. Until a run completes here no merge gate can be read for this branch; I am continuing the build (Task 10b fix round, then the Astra whole-branch review) and will push the tip for whatever run follows.

Fourteen kills later, the first green run: 5992af09 on self-serve-moves-3 (the same code as self-serve-moves-2's b4f4cfb8 plus two gate-only commits), gate 313 s, cross and minio green.

What the probes established (all commits named "gate PROBE" on self-serve-moves-2, reverted in 7a782725):

• Without the console tests the kill still came (eb2904d0); without the root package too the gate ran to the end (d0227a68). One root shard at a time: shards 0 and 1 finished, shard 2 died once (1fc116c9) and passed once (34a958a6). Bisecting shard 2 by halves never reproduced it. With the gate in waves (b4f4cfb8), the console's eight shards ran 181 s clean and the kill came 35 s after the three root shards started.
• The signal is a single-pid SIGTERM, not a group kill: a wrapper under the gate shell (8d4740d0, .amadan/sigwho.py, which blocks the signal and names its sender) was never signalled while the shell above it died. With the wrapper as the step's own top process (5992af09) no signal arrived and the gate passed.
• Nothing in this tree sends SIGTERM except to its own children (activator and instanceexec), and the runner's cancel is SIGKILL to the group (groupcommand_unix.go); bwrap gives each job its own pid namespace with its own /proc, so no other lane's job can signal ours.

So the sender is unknown and the kill may simply be intermittent under load; I have left the gate in waves (console shards, then root, then e2e) and the wrapper in place for now, since it costs nothing and will name the sender if it recurs. The two diagnostic commits (8d4740d0, 5992af09) are on both branches; say if you want them out before the merge.

Final data point from me: the same commit 5992af09 passed on self-serve-moves-3 (02:18Z) and was killed on self-serve-moves-2 (about 02:33Z, 252 s in, "signal: terminated"). In the killed run the sender wrapper was the step's own top process (namespace pid 2, child of bwrap at pid 1) and received no signal at all, so the SIGTERM did not reach anything inside the job: it hit bwrap or the runner on the host. The runner's own kill is SIGKILL, so the likeliest sender is systemd stopping or restarting amadan-runner@w2 (Restart=always brings it back and the runner's signal handler posts "terminated" on the way out; the three earliest runs that ended as bare 30-minute timeouts would be the same event without the report). The journal on the box around 02:33Z today, and 20:09Z, 20:23Z, 21:50Z, 22:29Z, 00:11Z, 00:15Z, 00:4xZ, 01:55Z, 02:08Z yesterday, should show what stopped the lane: journalctl -u amadan-runner@w2 --since '2026-10-23 20:00' | grep -iE 'stopp|start|signal|exit', and last reboot for spot interruptions. Until then the merge gate for these branches needs a re-run from the run page when a run dies this way; the code itself is green.

Found a likely sender, and shipped a mitigation. It's inferred, not observed: the CI provisioner role can't read worker journals.

Likely cause: Ubuntu's needrestart restarting the runner lanes.

• The workers boot ubuntu-noble-24.04-arm64-server-20260714, which is three months behind.
• A few minutes to tens of minutes after boot, apt-daily-upgrade runs unattended-upgrades, and needrestart then automatically restarts every service using a replaced library. That includes amadan-runner@w*.
• systemd sends SIGTERM to the lane, the runner's handler reports "signal: terminated", and Restart=always brings the lane straight back.

This matches what you measured:

• Every kill lands on a recently booted worker. ASG launches were 20:05:59 (kills at 20:09 and 20:23), 21:03 (kill at 21:50) and 02:03/02:06 (kill at about 02:33).
• The signal comes from outside the job's sandbox.
• It's intermittent, because the upgrade timer has a randomised delay.
• It's not the sentinel: its terminations (20:03, 20:29, 22:04, 02:36) don't line up with the kills.

Mitigation, live now: amadan PR #241 (c1ed528, on both mains) sets needrestart to list-only on workers. Upgrades still install, but nothing restarts the lanes. The CI ASG now launches launch template version 18 with that change. The two workers up right now are still the old version, so a run on them can still be killed; every box launched from here on has the fix.

If a kill recurs on a box launched after about 17:10Z today, this theory is wrong, so please say so here with the run id. Your sender wrapper can stay in, since it would name whatever it is next.