amadan

CI

CI on amadan is one command and no configuration language. amadan.net runs no runners for you — you run one, and it takes jobs for the repos you can already write to.

If you would rather not run a machine, somebody else can lend you theirs: a namespace owner asks, the machine's operator accepts, and neither can arrange it alone. That is a person lending you a computer, not a hosted service — see lending a runner.

amadan runner

That machine now takes CI jobs. There is no registration step, no token to paste, and no new binary — the identity ceremony already happened when you ran amadan keygen and amadan auth login.

How a job happens#

  1. Somebody pushes a commit to a repo you're a member of.
  2. The hub queues a job.
  3. A runner owned by a member claims it, over an outbound long-poll.
  4. The runner checks out the commit and runs the repo's CI script.
  5. It posts a status and a log back.
  6. The commit shows a tick, a cross, or a pending dot in the web UI.

Runners talk outbound only. A laptop behind NAT needs no port forwarding, no tunnel, and nothing inbound. If the machine can pull from the hub, it can be a runner.

A push is the only trigger — with one deliberately narrow exception: a writer can re-run the exact job that already ran for a commit, from that commit's CI run page, without pushing anything at all. There is still no schedule and no cron, and a re-run is not a new kind of trigger — it re-queues the same commit at the same manifest, nothing more (see "Re-running a job" below).

Start the runner before your first push#

Step 2 has a precondition, and it is the one thing here you can trip over silently:

> A repo with no runners never queues and never nags.

The hub queues a job only if a runner has polled that repo within the last seven days. So the order most people try first — create the repo, push, then start the runner — produces nothing at all. No job, no error, no pending dot. Nothing is broken; there was simply no runner to queue for, and the hub does not nag you about a repo you may never have wanted CI on.

It does say so, in the places you would look. amadan ci status on a commit in a repo no runner has polled answers "No runner serves this repo — CI will not run until one polls it" (and -json carries no_runner: true), rather than the "No CI status recorded" that would otherwise stand forever. The branch page's merge card and its CI tab say the same thing.

Pushing again does not fix it either: only a changed ref enqueues. Push a new commit, or delete the branch and push it again.

A runner that is already up covers a new repo on its own, as long as it was started without -repo: it re-reads its grants once a minute, so a repo you create now is polled within a minute, and a push after that queues. A runner started with a fixed -repo list serves that list and nothing else, and needs a restart to learn a new name. After the first poll the window is forgiving — the runner does not need to be up at the moment of a push, only to have been seen in the last week.

The job definition#

There is no CI DSL. Your CI is a script, and the runner looks for one in this order:

  1. .amadan/ci.d/ — a directory of executables, run as ordered steps. See below.
  2. .amadan/ci — a single executable file.
  3. make ci — used if the Makefile declares a ci target.
  4. Otherwise the job is reported skipped. Not failed: there is simply nothing to run.

Exit 0 is success. Anything else is failure. stdout and stderr are merged, so the log reads in the order things actually happened.

Debugging CI is running the script yourself, because it is the same script.

An empty ci.d does not shadow a working .amadan/ci — the single-script path stays reachable.

Not executable is an error, not a skip#

A committed CI script without its executable bit fails the whole job with a message telling you to chmod +x it. That is deliberate: a silently-skipped gate is how a test suite quietly stops running.

Inside ci.d, a file with an extension (README.md, notes.txt) is treated as clutter and ignored. A file with no extension and no executable bit is treated as a step you forgot to chmod, and fails.

Steps#

.amadan/ci.d/ turns one job into a readable sequence. Entries run in filename order, so the conventional shape is a numeric prefix:

.amadan/ci.d/10-vet
.amadan/ci.d/20-test
.amadan/ci.d/30-browser

The numeric prefix is stripped from the name a person reads — those are steps vet, test and browser.

Execution stops at the first failure, and the steps after it are reported skipped rather than dropped: "we never got there" is a different fact from "it did not exist", and the report keeps them apart.

All steps share one timeout for the whole job, not one each.

Sharding a slow step#

A step can declare that it should run as several concurrent copies of itself, by ending its filename with @N:

.amadan/ci.d/30-browser@20

That runs twenty copies of the same script at once, each told which one it is. The step is still one entry in the report, and its start and end times are the wall clock across all shards, not the sum of them — reporting the sum would misreport the one thing sharding exists to improve.

Shards of a step run concurrently with each other and with nothing else, so the job stays a sequence you can read top to bottom.

N must be a positive integer up to 64. A malformed or oversized @N fails the job at discovery, before anything runs — every shard is a process, so a typo'd @1000 must not be fork-bombed onto the box.

A step with no @N reports exactly as it did before sharding existed.

Reading a failed shard#

Every shard keeps its own log, and the run page gives each one its own row — a red twenty-way step tells you which of the twenty went red before you open anything.

The job's whole log is a different thing: it is every step's output concatenated, and a sharded step's is every shard's, in index order. So a failing shard sits in the middle of it, and the end of it is whichever shard happened to be last. The last line of that log therefore names the failing step and shard, for readers — a tail in a terminal, a merge gate somewhere else — that only ever see the one blob.

What the job is told#

  • AMADAN_CI=1
  • AMADAN_COMMIT — the commit SHA being built
  • AMADAN_BRANCH — the branch name
  • AMADAN_STEP — the step's name, on the ci.d path
  • AMADAN_SHARD — which shard this is, 0-based
  • AMADAN_SHARD_COUNT — how many shards the step has

AMADAN_SHARD=0 and AMADAN_SHARD_COUNT=1 are set even for a single .amadan/ci, so a repo migrating a script into ci.d never sees its environment change underneath it.

*Every `AMADAN_ variable in the runner's own environment is stripped before your script runs**, and only the job's own set is added back. That is a credential boundary: AMADAN_*` is exactly the surface amadan owns, so dropping it cannot break a third-party build, and it is the variable set that would otherwise leak the runner's credentials into every job.

Everything else — PATH, HOME, GOPROXY, SSL_CERT_FILE — is inherited, deliberately. An allowlist that misses one breaks builds slowly, confusingly, and silently.

Limits#

  • 30 minutes per job, wall clock, across all steps.
  • 1 MB of log, keeping the tail — where a failure explains itself.
  • 14 days of log history, or 250 MB per repo, whichever comes first — oldest dropped first. Run and stats are kept.

Runner flags#

  • No -repo at all — the runner follows its grants: every repo your token can write, plus every repo in a namespace that has accepted this identity as a CI runner (see lending a runner). Those two sets rarely overlap: a CI identity holds the runner role, which is below writer, so it discovers nothing the first way and everything the second. It re-reads the grants once a minute, so a repo created or granted after it started joins its rotation without a restart, and one whose grant is revoked leaves it. Each change is logged (now serving, no longer serving). Starting with nothing to serve is fine in this mode: it is a box waiting for its first grant.
  • -ns <name> — repeatable; follow grants in these namespaces only. A filter on what the hub already grants, never a way to reach a namespace that did not.
  • -repo <ns>/<name> — repeatable; serve exactly these repos, fixed for the life of the process. Cannot be combined with -ns.
  • -workdir <dir> — keep clones and per-branch worktrees between jobs, so a repeat push to a branch skips re-cloning and, on the encrypted tier, re-decrypting. A bare amadan runner uses a fresh temp directory and removes it on exit.
  • -clean — run git clean -xdff before every job. Hermetic, and slower: it deletes the cached build artifacts (node_modules, target, .venv) that a warm workdir exists to keep. Off by default for that reason.
  • -workdir-max <bytes> — evict least-recently-used repo workspaces once the workdir passes this size. 0, the default, means no cap.
  • -isolate none|bwrap — what a job is confined by. See below.
  • -caps <tag> — repeatable; declare something this machine has, like -caps browser. See the next section.

When a job needs something not every runner has#

Two runners polled one repo whose test suite needs a headless browser; one of them had one. Whichever polled first took the job, so the same push went red or green by coin flip. That is what this section ends.

Both sides declare, once each:

  • The repo's owner says what its CI needs: amadan ci requires -set "browser" (bare amadan ci requires shows it, -clear removes it).
  • Whoever starts a runner says what the machine has: amadan runner -caps browser -caps node.

A runner missing any required tag is answered "idle" when it polls — exactly as if the queue were empty — and the job stays queued until a runner declaring every tag polls. Nothing fails, nothing is claimed and given back: the incapable runner simply never sees the job. The idle answer names the missing tags (unmet_caps), so a runner's operator can see why a repo's jobs pass it by.

Tags are free-form lowercase words ([a-z0-9-]+). browser, node, gpu — they mean whatever you and your runner operators agree they mean; amadan only matches them. The requirement is repo-wide, on purpose: a per-step requirement would mean growing the job-definition language, and one tag on the repo is what would have kept the day green. Two costs to know about:

  • A repo that requires caps no polling runner declares queues forever — the same behaviour as having no runner at all, and the run page's queue-wait line is where it shows.
  • The declaration is trusted, not probed: a runner started with -caps browser on a box without a browser will claim the job and fail it honestly, which is at least a red build you can read rather than a coin flip.

Running it as a service#

amadan runner install

Installs a systemd user unit so the runner survives a reboot. It also picks a stable -workdir under $XDG_STATE_HOME, so an installed runner actually benefits from warm workspaces — which a bare amadan runner does not, by design.

*It is a user unit*, which is right for a laptop and wrong for a headless box. A user unit stops when your login session ends, so the obvious sequence — ssh in, amadan runner install, log out — stops CI at logout, and looks like the runner crashed. On a box you log out of, either enable lingering:

loginctl enable-linger "$USER"

or install a system unit instead. amadan's own cloud runner does the latter, for exactly this reason — see hack/cloud-runner-user-data.sh if you want a worked example.

Isolation, and what it does not buy you#

-isolate none is the default: jobs run as the runner's user, on the runner's machine, with no sandbox at all.

That is a real decision, not an oversight. In exchange, jobs are only ever created for commits pushed by members holding the writer role — people who can already rewrite the repository you share with them. The trust you extend is the trust you already extend by pulling a colleague's branch and running make test on it.

-isolate bwrap runs each job under bubblewrap: the host filesystem read-only, a private /tmp, per-repo Go caches, and the runner's own credentials directory masked. That is what a box serving several repos wants, and it is the honest recommendation for one.

But be clear about what it is: bubblewrap shares the host's kernel. It fixes cross-repo bleed and stops a job reading the runner's keys. It is not a sandbox for hostile code, and it does not make CI safe for untrusted contributors.

A runner asked for bwrap on a machine without bubblewrap installed refuses to start, rather than quietly running unsandboxed.

It refuses just as loudly when bubblewrap is installed but cannot actually build a sandbox — being on PATH is not the same as working, so the startup check creates a real sandbox and throws it away. The common case is Ubuntu 24.04, where AppArmor declines the unprivileged user namespace and every job would otherwise die with setting up uid map: Permission denied. The refusal names the fix:

sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

Refusing at startup is deliberate. A runner that fell back to no sandbox would report itself as isolated while running every job bare, which is the failure this flag exists to prevent.

Do not point a runner at a repo whose writers you would not hand a shell.

Re-running a job#

Before this existed, the only way to retry a flake was to push an empty commit — which permanently writes a lie into the branch's history to work around something that had nothing to do with the code. The run page for any finished job (/{ns}/{repo}/-/ci/{sha}) now has a Re-run this job button instead.

It is deliberately the smallest thing that could work: same commit, same manifest, one new job — not a partial re-run of only the failed steps, and not a re-run with different inputs. Those need a job model that can express partial results; this is an endpoint that enqueues exactly what already ran.

A few things worth knowing about how it behaves:

  • Who can: a writer or above — the same role a push already requires. Re-running spends real CI compute on somebody's own runner, so this is not a reader's button.
  • It never cancels other work. A push always supersedes whatever unclaimed job was queued for that branch (newest-push-wins); a re-run does not. It queues alongside anything else waiting, so retrying an old, suspected-flaky commit can never cancel a real, newer push that simply hasn't been claimed yet.
  • It's a new job, not a replay of the old one. Re-running writes a fresh queue entry with its own id, claim, and attempt history — the original job's record (who claimed it, when, and what it reported) stays exactly as it was. The commit's dot and log update to whatever the new run finds, the same way a genuinely re-pushed commit already updates its verdict.
  • You can't pile them up. A commit that is already queued or running refuses another re-run request — there's nothing yet to distrust.

The button lives on the web run page, which only exists for public/ private repos — the hub holds ciphertext for private_e2ee and has no run page to put a button on. The re-run endpoint itself is tier-switched like every other CI call, so an E2EE repo's own client-side tooling can reach it directly; the web UI just doesn't wire it up yet.

CI on encrypted repos#

On the private_e2ee tier the status and the log are sealed under the repo key before they leave the runner. The hub schedules work it cannot read and stores results it cannot open: not the log text, not even pass or fail.

The trigger rule is deliberately identical on every tier. The hub never inspects a tree to decide whether a CI script exists — it can't on the encrypted tier — so whether there is a job is decided runner-side everywhere. Both worlds behave the same way, which is the only way to keep the encrypted tier from being a second-class one.

One practical consequence worth knowing: a job log on a private repo is readable only through an authenticated browser session. There is no credential a machine can hold that will fetch it.

What a lent runner actually has#

Every self-hosted runner has whatever its owner installed — that's true by construction. But if you're accepting an invitation onto somebody else's machine (see lending a runner), you're trusting its shape sight-unseen, and one concrete example is worth documenting rather than assuming.

amadan's own shared cloud runner — the box described in ci-cloud-runner.md, whose whole definition is hack/cloud-runner-user-data.sh — has:

  • A Go toolchain, Node, git, make, and bubblewrap. Chromium is there too, for browser suites, installed from Playwright's arm64 build rather than a system package (see that script's comments for why apt-get install chromium doesn't work here).
  • No container runtime of any kind. No docker, no podman, nothing OCI-adjacent. A job that shells out to docker or docker compose fails on this runner today, full stop — this is not an oversight to route around, see the container-access decision memo for why that's a deliberate, live decision and not a gap waiting to be filled.
  • arm64 (Graviton), not amd64. A CI recipe carried over from somewhere else that hardwires linux-amd64 — a pinned binary download is the usual shape — needs a rewrite before it runs here. This has already bitten one real job: carlos/platform's minio-contract pins its mc download to linux-amd64 and would need an arm64 build to run on this box even once (or if) it gets a container runtime.

None of this is a promise about every runner — a self-hosted one can have whatever you put on it. It's a statement about the one shared box amadan itself operates, documented because an invited namespace has no other way to find out before accepting.

If you're onboarding a repo to a bwrap-isolated runner: check that a test actually ran#

-isolate bwrap changes what a job's environment looks like (its own /tmp, its own view of the filesystem, AMADAN_* stripped — see "What the job is told" above). A test suite that silently no-ops when a tool it expects is missing from PATH will keep exiting 0 under isolation, just for a different reason than before — and a green step and a step that ran nothing look identical from the CI tab.

This is not hypothetical: carlos/platform has 23 call sites that do exec.LookPath("go") to find a Go toolchain for a subprocess test, and 22 of them t.Skip when it isn't found rather than failing. A job whose PATH doesn't happen to resolve go — plausible on any runner, isolated or not, but easier to get wrong once a job's environment is something other than "whatever's on this laptop" — exits 0 having silently skipped most of that suite. The fix belongs to each repo, not to amadan: when onboarding a repo to any runner, especially a shared or isolated one, assert that a specific, known test ran (check its name appears in the log, not just that the step's exit code was 0) before trusting a green result.

What doesn't exist#

No runners operated by amadan.net. No matrix builds. No caching service. No artifact storage. No scheduled jobs. There is a manual re-run now (see above) — but only that: same commit, same manifest, one job. No re-run of individual failed steps, and no re-run with different inputs. No container runtime on amadan's own shared runner — see above.

There are also no quotas. Somebody who lends you their machine is paying for it with nothing metering what you use, which is exactly why lending is by invitation and not self-serve.