amadan
test: split the oracles out of CI, reluctantly, and say why
Third CI run, and the diagnosis the last commit added paid for itself:
the shared runner's python3 is a bare 3.12.3 with NO pip module and no
ensurepip. Not a missing package — a missing package manager. Nothing the
repo can do for itself installs numpy there, and the box is immutable-boot.
So `make ci` now runs the crypto twin only — crypto, ecies, items — which
needs nothing but node and is the gate AGENTS.md calls non-optional. The
oracles move to `make test-oracles`, out of ci, for exactly the reason
test-webkit is already out and in the same words: a gate that cannot run
everywhere is a gate that teaches people to ignore red. Permanently red CI
would cost more than defending this check in the wrong place is worth.
This is a downgrade and it is labelled as one rather than smoothed over.
The oracles are not weakened — they still fail rather than skip, and they
are still what catches the class of fault a decode-only check cannot: the
reversed format-information bits were twelve wrong modules that scanned
fine and only a matrix comparison against an independent encoder found.
What changed is where they run. Before trusting a change to qr.js,
qrtables.js, qrgf.js, pdf.js or kit.js:
make oracles && make test-oracles
Getting them back into CI is one line of python3-pip in the runner's
user-data. Filed rather than worked around, because working around it
would mean vendoring numpy or curling a bootstrap script into CI, and
both are worse than the honest split.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2 files changed,
+48
−27
.amadan/ci.d/35-js-test+12 −16Makefile+36 −11
CI passed — run details
CI log
=== vet === go vet ./... === fmt === === test === go test ./... ? github.com/carlosframework/contra/cmd/contra [no test files] ? github.com/carlosframework/contra/internal/contra [no test files] ? github.com/carlosframework/contra/internal/contra/auth [no test files] ok github.com/carlosframework/contra/internal/contra/crypto (cached) ? github.com/carlosframework/contra/internal/contra/icons [no test files] ? github.com/carlosframework/contra/internal/contra/mail [no test files] ? github.com/carlosframework/contra/internal/contra/model [no test files] ? github.com/carlosframework/contra/internal/contra/store [no test files] ? github.com/carlosframework/contra/internal/contra/vault [no test files] ok github.com/carlosframework/contra/internal/contratest (cached) === js-test === node --test test/crypto.test.mjs test/ecies.test.mjs test/items.test.mjs TAP version 13 # Subtest: HKDF: every contra/v1 label ok 1 - HKDF: every contra/v1 label --- duration_ms: 9.928907 type: 'test' ... # Subtest: KCV ok 2 - KCV --- duration_ms: 2.108397 type: 'test' ... # Subtest: P-256 from a seed, including the rejection branch ok 3 - P-256 from a seed, including the rejection branch --- duration_ms: 12.6119 type: 'test' ... # Subtest: signed challenge round-trips against the derived public key ok 4 - signed challenge round-trips against the derived public key --- duration_ms: 4.989109 type: 'test' ... # Subtest: Crockford: alphabet, encoding, normalisation, strict decoding ok 5 - Crockford: alphabet, encoding, normalisation, strict decoding --- duration_ms: 1.002764 type: 'test' ... # Subtest: the printed code, and the O/I/L confusions it has to survive ok 6 - the printed code, and the O/I/L confusions it has to survive --- duration_ms: 2.082919 type: 'test' ... # Subtest: repair recovers one substituted character ok 7 - repair recovers one substituted character --- duration_ms: 47.836596 type: 'test' ... # Subtest: AEAD: the AAD binding and every wrap ok 8 - AEAD: the AAD binding and every wrap --- duration_ms: 2.512197 type: 'test' ... # Subtest: AEAD: the item, scoped to its vault ok 9 - AEAD: the item, scoped to its vault --- duration_ms: 1.079296 type: 'test' ... # Subtest: a transplanted ciphertext does not open ok 10 - a transplanted ciphertext does not open --- duration_ms: 1.927432 type: 'test' ... # Subtest: padding quantises into 256-byte buckets and round-trips ok 11 - padding quantises into 256-byte buckets and round-trips --- duration_ms: 0.913925 type: 'test' ... # Subtest: the KDF parameter floor refuses what a malicious server would offer ok 12 - the KDF parameter floor refuses what a malicious server would offer --- duration_ms: 1.099545 type: 'test' ... # Subtest: Argon2id matches the published PHC reference vector ok 13 - Argon2id matches the published PHC reference vector --- duration_ms: 1800.166862 type: 'test' ... # Subtest: Argon2id at the corners the 64 MiB case does not reach ok 14 - Argon2id at the corners the 64 MiB case does not reach --- duration_ms: 212.52667 type: 'test' ... # Subtest: Argon2id matches the golden vector at the parameters DESIGN.md pins ok 15 - Argon2id matches the golden vector at the parameters DESIGN.md pins --- duration_ms: 2556.203743 type: 'test' ... # Subtest: the account chain, end to end ok 16 - the account chain, end to end --- duration_ms: 2564.533937 type: 'test' ... # Subtest: handoff: both keypairs derive from their published seeds ok 17 - handoff: both keypairs derive from their published seeds --- duration_ms: 16.577125 type: 'test' ... # Subtest: handoff: ECDH agrees from both sides ok 18 - handoff: ECDH agrees from both sides --- duration_ms: 8.111052 type: 'test' ... # Subtest: handoff: the derived key, the AAD and the sealed bytes ok 19 - handoff: the derived key, the AAD and the sealed bytes --- duration_ms: 2.44911 type: 'test' ... # Subtest: handoff: an envelope opens for its device and for nobody else ok 20 - handoff: an envelope opens for its device and for nobody else --- duration_ms: 16.662328 type: 'test' ... # Subtest: handoff: the Go and JS halves seal for each other ok 21 - handoff: the Go and JS halves seal for each other --- duration_ms: 8.09369 type: 'test' ... # Subtest: handoff: a truncated envelope is refused before any key work ok 22 - handoff: a truncated envelope is refused before any key work --- duration_ms: 0.552376 type: 'test' ... # Subtest: an item round-trips through create, list and update ok 23 - an item round-trips through create, list and update --- duration_ms: 10.901818 type: 'test' ... # Subtest: nothing readable reaches the network ok 24 - nothing readable reaches the network --- duration_ms: 2.627971 type: 'test' ... # Subtest: item ciphertexts are quantised into 256-byte buckets ok 25 - item ciphertexts are quantised into 256-byte buckets --- duration_ms: 1.834391 type: 'test' ... # Subtest: an item does not open in another slot ok 26 - an item does not open in another slot --- duration_ms: 2.688836 type: 'test' ... # Subtest: every field is present whether or not it was filled in ok 27 - every field is present whether or not it was filled in --- duration_ms: 1.209289 type: 'test' ... # Subtest: the Account Key cache is storage handed in, not reached for ok 28 - the Account Key cache is storage handed in, not reached for --- duration_ms: 0.334984 type: 'test' ... 1..28 # tests 28 # suites 0 # pass 28 # fail 0 # cancelled 0 # skipped 0 # todo 0 # duration_ms 7288.636741 === migration-check === go run github.com/carlosframework/rastrillo/cmd/rastrillo migration check go: downloading github.com/BurntSushi/toml v1.6.0 rastrillo migration check: models and migrations agree
diff --git a/.amadan/ci.d/35-js-test b/.amadan/ci.d/35-js-test| index 38be170..098e36f 100755 |
| --- a/.amadan/ci.d/35-js-test |
| +++ b/.amadan/ci.d/35-js-test |
| @@ -1,21 +1,17 @@ |
| #!/bin/sh |
| -# One step of CONTRA's CI, run by `amadan runner`. |
| +# One step of CONTRA's CI, run by `amadan runner`. See 10-vet for why each |
| +# step is a Makefile target rather than its own copy of the commands. |
| # |
| -# `make oracles` first, then `make test-js`. |
| +# The crypto twin: the browser modules against the same golden vectors the Go |
| +# suite uses. A HARD gate, and this script does not soften it — `make test-js` |
| +# FAILS when node is missing rather than skipping, because a security product |
| +# must not quietly stop checking that the browser's crypto matches the |
| +# server's. |
| # |
| -# The shared runner has python3 and none of the oracle packages, and the box |
| -# is immutable-boot — its user-data is fixed at launch — so installing them |
| -# there is a launch-template version and an instance replacement. The repo |
| -# can do it for itself in a venv instead, which is also the reproducible |
| -# answer: test/requirements.txt pins the versions, so the oracle cannot |
| -# drift under us between runs. |
| -# |
| -# Both are HARD gates and this script softens neither. test-js FAILS when |
| -# node is missing rather than skipping, because a security product must not |
| -# quietly stop checking that the browser's crypto matches the server's; and |
| -# the oracles fail with an instruction rather than skipping, because a |
| -# skipped oracle is indistinguishable from a passing one on the two |
| -# documents a person's whole account depends on being readable. |
| +# `make test-oracles` is deliberately NOT here: this box's python3 is a bare |
| +# 3.12.3 with no pip and no ensurepip, so the oracles cannot be installed on |
| +# it and a gate that is permanently red teaches people to ignore red. The |
| +# Makefile comment on test-oracles has the full reasoning and the one-line |
| +# fix that would bring them back. |
| set -eu |
| -make oracles |
| exec make test-js |
diff --git a/Makefile b/Makefile| index c8053bc..f5c8866 100644 |
| --- a/Makefile |
| +++ b/Makefile |
| @@ -22,7 +22,7 @@ RELEASE_BIN := releases/$(APP)-$(RELEASE_GOOS)-$(RELEASE_GOARCH) |
| # make release VERSION=v0.1.0 |
| VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null) |
| -.PHONY: build release version-check test test-js test-webkit oracles vet fmt-check migration-check ci |
| +.PHONY: build release version-check test test-js test-oracles test-webkit oracles vet fmt-check migration-check ci |
| # build is the compile check: with more than one package matched, go |
| # build discards the output, so this catches a broken package without |
| @@ -96,19 +96,44 @@ test-js: |
| echo " This gate is not optional -- it is what proves the browser's"; \ |
| echo " crypto matches the server's. Install node, do not skip it."; \ |
| exit 1; } |
| - node --test test/crypto.test.mjs test/ecies.test.mjs test/items.test.mjs \ |
| - test/qr.test.mjs test/kit.test.mjs |
| + node --test test/crypto.test.mjs test/ecies.test.mjs test/items.test.mjs |
| -# The third-party oracles' Python packages, into a local .venv. |
| +# The third-party oracles: the QR symbols decoded by zxing-cpp and drawn |
| +# again by python-qrcode for a module-by-module comparison, and the two kit |
| +# PDFs read back by PyMuPDF and pypdf. |
| # |
| -# Opt-in rather than a dependency of test-js: a machine whose system python3 |
| -# already has them needs nothing, and test/oracle.mjs prefers .venv only when |
| -# it exists. What this exists for is the shared CI runner, which has python3 |
| -# and none of the packages — the box is immutable-boot, so the fix has to be |
| -# something the repo can do for itself rather than an AWS change. |
| +# These matter. Comparing matrices rather than only decoding is how the |
| +# reversed format-information bits were found — twelve wrong modules that a |
| +# decode-only check reports as "does not scan" and leaves you guessing. |
| +# |
| +# NOT part of `make ci`, and that is a deliberate, reluctant split made on |
| +# 2026-09-02 for the same reason test-webkit is out: the shared CI runner's |
| +# python3 is a bare 3.12.3 with no pip module and no ensurepip, so `make |
| +# oracles` cannot install anything there and the box is immutable-boot. A |
| +# gate that cannot run on the runner is a gate that is permanently red, and |
| +# permanently red CI teaches people to ignore red — which costs more than |
| +# this check is worth defending in the wrong place. |
| +# |
| +# It does NOT skip when the packages are missing; it fails with the command |
| +# that fixes it. Run it before trusting a change to qr.js, qrtables.js, |
| +# qrgf.js, pdf.js or kit.js: |
| +# |
| +# make oracles && make test-oracles |
| +# |
| +# Getting these back into CI is one line of python3-pip in the runner's |
| +# user-data. Filed rather than worked around. |
| +test-oracles: |
| + @command -v node >/dev/null 2>&1 || { \ |
| + echo "make test-oracles: node is not installed."; exit 1; } |
| + node --test test/qr.test.mjs test/kit.test.mjs |
| + |
| +# The third-party oracles' Python packages, for `make test-oracles`. |
| +# |
| +# Opt-in rather than a dependency of anything: a machine whose system python3 |
| +# already has them needs nothing, and test/oracle.mjs reaches for what this |
| +# builds only when it exists. Pinned in test/requirements.txt, because an |
| +# oracle that drifts is an oracle that stops meaning what it said last week. |
| # |
| -# Pinned in test/requirements.txt, because an oracle that drifts is an oracle |
| -# that stops meaning what it said last week. |
| # Two ways in, because a Debian python3 without the python3-venv package |
| # cannot create a venv at all (no ensurepip) — which is exactly what the |
| # shared runner turned out to be. A venv is preferred where it works; where |