amadan
gate: narrate job starts and name the in-flight jobs when a signal stops the gate
Seven CI runs on 2026-10-23 ended with an empty gate log: three at the 30-minute job limit and four by "signal: terminated", three of those 48-53 s in. The gate printed nothing until every job had finished, so none of them said how far it had got or what was running. Now it prints the runner's shape first (arch, cpus, available memory, go version), one line per job start with the elapsed second, a line at each stage boundary, and on INT, TERM or HUP the elapsed time and the jobs still in flight before stopping them.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+24
−1
.amadan/test-gate.sh+24 −1
CI failed — run details
CI log
=== gate === .amadan/test-gate.sh gate: 2026-10-23T21:49:54Z on aarch64, 32 cpus, 57279 MB available, go1.26.8 linux/arm64 gate: +0s start launchctx gate: +0s start vet gate: +0s start gofmt go: downloading github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.68.0 go: downloading github.com/aws/aws-sdk-go-v2/service/costexplorer v1.67.5 go: downloading github.com/aws/aws-sdk-go-v2 v1.44.0 go: downloading github.com/aws/smithy-go v1.28.1 go: downloading github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0 go: downloading github.com/superfly/ltx v0.5.1 go: downloading golang.org/x/net v0.56.0 go: downloading modernc.org/sqlite v1.55.0 go: downloading github.com/aws/aws-sdk-go-v2/config v1.32.6 go: downloading github.com/aws/aws-sdk-go-v2/service/sns v1.43.0 go: downloading github.com/aws/aws-sdk-go-v2/service/route53 v1.64.1 go: downloading github.com/benbjohnson/litestream v0.5.14 go: downloading golang.org/x/crypto v0.53.0 go: downloading github.com/aws/aws-sdk-go-v2/service/autoscaling v1.74.0 go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.324.0 go: downloading golang.org/x/sys v0.46.0 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.19.6 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.45.6 go: downloading github.com/aws/aws-sdk-go-v2/service/acm v1.45.0 go: downloading github.com/aws/aws-sdk-go-v2/service/iam v1.58.2 go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.324.0 go: downloading github.com/aws/aws-sdk-go-v2/service/acm v1.45.0 go: downloading github.com/superfly/ltx v0.5.1 go: downloading github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0 go: downloading modernc.org/sqlite v1.55.0 go: downloading github.com/oschwald/maxminddb-golang v1.13.1 go: downloading github.com/aws/aws-sdk-go-v2/service/iam v1.58.2 go: downloading github.com/aws/aws-sdk-go-v2/service/ssm v1.74.0 go: downloading github.com/aws/aws-sdk-go-v2/service/resourcegroupstaggingapi v1.37.0 go: downloading github.com/aws/aws-sdk-go-v2/service/autoscaling v1.74.0 go: downloading github.com/aws/aws-sdk-go-v2/service/resourcegroupstaggingapi v1.37.0 go: downloading github.com/oschwald/maxminddb-golang v1.13.1 go: downloading github.com/pion/logging v0.2.4 go: downloading github.com/pion/stun/v3 v3.0.1 go: downloading github.com/pion/turn/v4 v4.1.4 go: downloading github.com/aws/aws-sdk-go-v2/service/sesv2 v1.66.6 go: downloading github.com/aws/aws-sdk-go-v2/service/sesv2 v1.66.6 go: downloading github.com/pierrec/lz4/v4 v4.1.22 go: downloading github.com/prometheus/client_golang v1.17.0 go: downloading golang.org/x/sync v0.21.0 go: downloading github.com/pion/stun/v3 v3.0.1 go: downloading github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.18 go: downloading github.com/pion/turn/v4 v4.1.4 go: downloading github.com/klauspost/compress v1.18.0 go: downloading github.com/pierrec/lz4/v4 v4.1.22 go: downloading github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.18 go: downloading github.com/lmittmann/tint v1.1.3 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 go: downloading github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 go: downloading github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.40 go: downloading github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.38 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.40 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.40 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.40 go: downloading github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 go: downloading github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.38 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.40 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.40 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 go: downloading github.com/prometheus/common v0.45.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/procfs v0.12.0 go: downloading github.com/prometheus/client_model v0.5.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading google.golang.org/protobuf v1.36.10 go: downloading github.com/prometheus/client_model v0.5.0 go: downloading github.com/prometheus/common v0.45.0 go: downloading github.com/prometheus/procfs v0.12.0 go: downloading google.golang.org/protobuf v1.36.10 go: downloading github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 go: downloading github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 go: downloading github.com/pion/transport/v3 v3.0.8 go: downloading github.com/pion/dtls/v3 v3.0.7 go: downloading golang.org/x/text v0.38.0 go: downloading github.com/pion/dtls/v3 v3.0.7 go: downloading github.com/pion/transport/v3 v3.0.8 go: downloading golang.org/x/text v0.38.0 go: downloading github.com/pion/randutil v0.1.0 go: downloading github.com/pion/transport/v4 v4.0.1 go: downloading github.com/pion/randutil v0.1.0 go: downloading github.com/pion/transport/v4 v4.0.1 go: downloading github.com/wlynxg/anet v0.0.5 go: downloading github.com/wlynxg/anet v0.0.5 go: downloading modernc.org/libc v1.74.1 go: downloading modernc.org/libc v1.74.1 go: downloading modernc.org/memory v1.11.0 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading modernc.org/mathutil v1.7.1 go: downloading modernc.org/memory v1.11.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading modernc.org/mathutil v1.7.1 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec gate: +8s start rest gate: +49s start internal_console@0 gate: +49s start internal_console@1 gate: +49s start internal_console@2 gate: +49s start internal_console@3 gate: +49s start internal_console@4 gate: +49s start internal_console@5 gate: +49s start internal_console@6 gate: +49s start internal_console@7 gate: +51s start root@0 gate: +51s start root@1 gate: +51s start root@2 amadan: step gate failed: signal: terminated amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/.amadan/test-gate.sh b/.amadan/test-gate.sh| index 328a2ee84..c1fe71009 100755 |
| --- a/.amadan/test-gate.sh |
| +++ b/.amadan/test-gate.sh |
| @@ -97,8 +97,28 @@ stop_jobs() { |
| wait 2>/dev/null || true |
| rm -rf "$out" |
| } |
| +# The gate narrates as it goes (one line per job started, with the second it |
| +# started at) and says what was in flight when a signal stops it: CI shows |
| +# the step's output only at the end, so a gate killed from outside would |
| +# otherwise leave an empty log (seven such runs on 2026-10-23). |
| +gate_t0=$(date +%s) |
| +gate_elapsed() { echo $(($(date +%s) - gate_t0)); } |
| +echo "gate: $(date -u +%FT%TZ) on $(uname -m), $(nproc) cpus, $(awk '/MemAvailable/ { printf "%d", $2 / 1024 }' /proc/meminfo 2>/dev/null || echo '?') MB available, $(go version | sed 's/^go version //')" |
| +in_flight() { |
| + for job in $started; do |
| + [ -e "$out/${job%:*}.secs" ] || printf '%s ' "${job%:*}" |
| + done |
| +} |
| +on_signal() { |
| + trap - EXIT |
| + echo "gate: stopped by $1 at $(gate_elapsed)s; in flight: $(in_flight)" |
| + stop_jobs |
| + exit 130 |
| +} |
| trap stop_jobs EXIT |
| -trap 'trap - EXIT; stop_jobs; exit 130' INT TERM |
| +trap 'on_signal INT' INT |
| +trap 'on_signal TERM' TERM |
| +trap 'on_signal HUP' HUP |
| names="" |
| # A failure found before any test runs (a package that will not list, a |
| @@ -122,6 +142,7 @@ start() { |
| echo $(($(date +%s) - t0)) >"$out/$start_name.secs" |
| ) & |
| started="$started $start_name:$!" |
| + echo "gate: +$(gate_elapsed)s start $start_name" |
| } |
| # Runs one discovery command in the background and waits for it, so an |
| @@ -229,6 +250,7 @@ done |
| wait |
| +echo "gate: +$(gate_elapsed)s every split-package job finished; e2e stage next" |
| # The end-to-end witnesses (internal/e2e/..., build tag e2e) are outside |
| # `go list ./...`. They start only once every other job has finished: |
| @@ -273,6 +295,7 @@ fi |
| start e2e-vet go vet -tags e2e ./internal/e2e/... |
| wait |
| +echo "gate: +$(gate_elapsed)s e2e stage finished" |
| started="" |
| failed="" |