carlos / platform Public

Test suite speed: where the CI gate's 375 seconds go, and a ranked plan open

Measured 2026-10-13/14 on main 6dfbff10: each of the five slowest packages run once with -json, niced, one at a time. No code was changed. The gate's wall clock is the slowest package, because Go runs packages side by side.

Where the time is

| Package | CI | What it is spending it on | |---|---|---| | internal/console | 370 s | One test, TestClaimConsoleMutationInvariant, is 36% of the package: 233 subtests, each building a full console fixture (about 1.2 s) and polling for quiet in 100 ms steps. Tests named Claim* are 53% in all. | | root package | 359 s | About 135 separate go builds of the carlos binary, 2.0 s each warm. buildCarlos has 48 callers and e2e_test.go repeats the same build inline at 20 sites. Nothing is shared. Roughly 270 s of the package. | | internal/activator | 227 s | Real waiting. About 30 tests call Start with a 2 s context against a fake child that never binds its socket, and wait the 2 s out. Two tests wait out real 1+2+4+8 s backoff, one a 20 s stop timeout. | | internal/host | 95 s | Off the critical path. One test makes 30,000 fsynced SQLite commits: seconds on CI's tmpfs, 16 minutes on a disk. | | cmd/moveprobe | 93 s | Off the critical path. One test waits out a real 60 s grace period. |

No test in these packages calls t.Parallel() (one in moveprobe does, to no effect). Nothing uses fixed ports; the blockers are t.Setenv, t.Chdir and saved package variables.

The local figures for console and host were inflated several times over by disk fsync (8.8 ms each on /var/tmp) and machine load, so the shares above are reliable and the CI-second estimates below are rough.

Ranked plan

| # | Change | Saves in its package | Gate effect | Risk | |---|---|---|---|---| | 1 | Root: build the carlos binary once and share it | about 220 s of 359 | none alone (console stays at 370); it is what makes #2 or #3 pay | low | | 2 | Gate: split console's tests by name across amadan shards (@N), with a fence that the union equals go test -list | console 370 to about 3 x 125 | large; the only cut to console that edits no test | medium: make test and CI stop being one definition | | 3 | Console: run the invariant test's subtests in bounded parallel, or share its fixture | about 130 s | about 130 s once #1 is in | highest: it is a safety-invariant test that measures a quiet window, and contention could hide a late effect | | 4 | Activator: stop waiting out 2 s once the fake child has recorded; give the refusal tests a real "nothing ran" signal | 55 to 65 s of 227 | only once console and root are under 227 | medium | | 5 | Activator: make the stop timeout and backoff base injectable | about 50 s | as #4 | medium; touches production constants |

Not worth doing for CI time: moveprobe, the host journal test, synchronous=OFF in fixtures.

Biggest unknown: how much of console's 370 s is its own work and how much is CPU contention from root linking 120 MB binaries beside it. #1 answers that as a side effect.

Doing now

#1, on branch test-suite-speed, stacked on test-suite-debt (both touch the root package's TestMain).

For Paul

#2 against #3 is a ruling: #2 gives up "CI is make test" (the principle written at the top of .amadan/ci.d/10-gate); #3 keeps it and touches a safety test. #4 and #5 can follow either.

#1 is built on test-suite-speed (43175cbd, stacked on test-suite-debt), CI green, Astra review running, not merged.

| | main | this branch | |---|---|---| | root package | 358.9 s | 90.0 s | | internal/console | 369.7 s | 365.9 s | | internal/activator | 227 s | 227.8 s | | gate step | 375 s | 392 s |

buildCarlos keeps its name and now returns one binary built lazily per test process, read-only (0555), in a directory TestMain makes under the isolated temp root. Eighteen inline builds in e2e_test.go use the same file. The echoapp builds and the cross-compiled tiny program are untouched. No caller needed a private copy.

The unknown is answered. Console did not move when root stopped linking 135 binaries beside it, so its 370 s is its own work, not contention. The gate is now exactly console (the 392 against 375 is run-to-run noise on one sample each). Nothing short of #2 or #3 above will bring the gate down; after either, activator at 227 s is the next floor.

Review update on #1 (test-suite-speed). Astra's first pass: no P1, three P2s and a P3, all fixed in 783d4fba (one build owner for the whole test process through a test-only export, the build environment captured after the temp-root guard, the binary's directory under GOTMPDIR when set, the directory created only when a build happens). Second pass: those four fixed, one new P2 (the re-exec location test failed instead of skipping with no Go toolchain), fixed in the tip. The second pass's other checks were sound: export_test.go stays out of ordinary builds and the external import initialises nothing twice.

CI on 783d4fba: root 90.2 s, console 360.5 s, gate 370 s. CI on the new tip is running.

Waiting on Paul to merge, after test-suite-debt, which it is stacked on.

#1 landed. Paul gave the go-ahead on 2026-10-21. test-suite-debt is on main as 512baa5 and test-suite-speed as aaa6a94, each with main merged in again and green CI on the tip. On CI the root package dropped from 326 s to 113 s. Notes for anyone writing root-package tests:

  • buildCarlos(t) returns one shared binary per test process. It is read-only (0555): copy it if a test needs a private one, and call buildCarlos rather than running go build inline.
  • Root TestMain moves the temp root to /var/tmp/carlos-test-* when an ancestor holds .carlos/config, and unsets CARLOS_CONFIG and CARLOS_DESTINATION.

#2 is built on ci-speed, done another way. Paul asked to keep going. The ruling between #2 and #3 is avoided by splitting inside make test (scripts/test-gate.sh), so CI is still make test:

  • launchctx, vet, gofmt and the tests run side by side.
  • internal/console is split 8 ways and root 3 ways across concurrent go test -run processes, dealt by measured weight (.amadan/test-weights.tsv).
  • A fence checks that every go test -list name runs exactly once, and a shard that matches no tests fails the gate.
  • go build runs beside the tests.

The new floor is TestClaimConsoleMutationInvariant: 200 s of console's 398 s locally, now in a shard of its own. It cannot be split across processes. Its unreached-writer check needs every door subtest in one process, and its 100 ms quiet windows are deliberate. Parallelising its subtests is still #3, and still needs Paul's ruling. Not done.

#4 and #5 are in progress on ci-speed-activator (activator, moveprobe) and ci-speed-host (host, provision).

On main's CI today, the gate step takes 503 s and the cross step 58 s. The steps run one after another.

ci-speed landed as d9149a0. It went through three Astra rounds and the last found nothing new at P1 or P2. On CI the gate step dropped from 503 s to 265 s, and a whole run from about 566 s to 283 s.

What make test does now (.amadan/test-gate.sh):

  • Runs everything side by side: launchctx, vet, gofmt and the tests.
  • Splits the slowest packages: internal/console 8 ways and root 3 ways, dealt across processes by .amadan/test-weights.tsv.
  • Checks coverage: every name go test -list reports must run, and a group that matches no tests fails the gate.
  • Refuses bad input: a missing directory, a failed go list, a bad shard count, or a missing pgrep fails the gate before anything runs.
  • Stops cleanly: stopping the gate stops every go test process and test binary under it.

The scripts sit in .amadan/ because the root-read census counts every .sh file outside dot-directories.

What holds the gate up now:

  • activator, 250 s: ci-speed-activator brings it to about 58 s locally. Its four production timings became package vars with the same values, which needs a census ruling: asked on #280 as PROPOSED-ci-speed-activator.
  • console's TestClaimConsoleMutationInvariant, about 205 s: still #3, still Paul's ruling.
  • host, about 195 s: ci-speed-host (test-only) brings it to about 102 s locally. Astra is reviewing it. Most of the rest needs production changes: settings read straight from the environment, the umask in #337, and a 30 s wake bound in one test.

State at end of 2026-10-21. Eight branches landed today. On CI, the gate step went from 503 s to about 145-160 s, and a whole run from about 566 s to about 180 s.

| Branch | On main | Effect | |---|---|---| | test-suite-debt | 512baa5 | root tests isolated from operator config | | test-suite-speed | aaa6a94 | root package 326 s to 113 s (one shared binary) | | ci-speed | d9149a0 | make test splits console 8 ways and root 3 ways, side by side | | ci-speed-host | 6da1cd0 | host 192 s to 126 s, provision 39 s to 22 s | | ci-speed-activator | 4f55bfb | activator 250 s to 62 s, moveprobe 93 s to 12 s (census-ruling-ci-speed-activator) | | securelisten-337 | 4a1e4b2 | #337: no process umask flip (census-ruling-securelisten-337) | | console-invariant-parallel | 521c6ee | option 3, Paul's go: invariant 205 s to about 60 s, 8 at a time, two quiet windows, claimMustRunAlone guard | | umask-parallel | 4d6f0e6 | 26 activator and 95 host tests made parallel after #337 |

What holds the gate up now:

  • host, about 115-140 s. Its sequential phase is mostly tests whose helpers set environment variables (CARLOSHOSTCLAIM, CARLOSOPTDIR and the like). Moving those settings onto config structs would let them run in parallel; that is a production change.
  • console's largest shard, about 80 s. It holds the invariant test.
  • The vet step (about 18 s) and the compile run alongside the tests.

ci-split-host landed as ab4e907 (2026-10-25). On CI the gate step dropped from 206 s (main e3fd2e02) to 112 s.

  • TEST_SPLIT now deals internal/host 4 ways and root 5 ways, beside console's 8. Host's groups take 29-65 s and root's 49-70 s.
  • test-weights.sh now rewrites only the packages it is given and keeps the other rows. It records tests of 0.25 s or more, and divides a t.Parallel test's time by 8. Unweighted tests count as 0.05 s.

What holds the gate up now, each about 65-77 s:

  • console's invariant shard (77 s).
  • the rest of the packages (71 s), mostly activator (59 s).
  • root's DBForward group (70 s). TestDBForwardPrintsNothingForABadAnswer's "no result" case drives the binary through the console's real 1-minute grace. The other two DBForward tests (55 s, 43 s) likely wait the same way. Shortening them needs a test hook in console production code and a census ruling.
  • host's group 0 (65 s), which holds TestClaimAckRefreshProductionClient (20 s).

The cross (20 s) and MinIO (32 s) steps still run after the gate, one after another.