carlos / contra Public

test: run the ECIES vectors in CI, and give the oracles their packages

The first real CI run found two things, which is the argument for having
turned it on.

**The ECIES vector tests were never in the gate.** test/ecies.test.mjs was
added with the handoff and never listed in the test-js target, so the six
tests pinning the one place a secret crosses the server — Go against JS
against the C generator — had never run anywhere but by hand. Now in:
`make test-js` goes from 29 tests to 35.

**The oracles had no packages on the runner**, so three tests failed with
a bare `ModuleNotFoundError: No module named 'numpy'`. The shared box is
immutable-boot, so installing them there means a launch-template version
and an instance replacement. The repo can do it for itself instead, and
that is also the better answer: test/requirements.txt pins the versions,
so the oracle cannot drift under us between runs.

`make oracles` builds a .venv; test/oracle.mjs resolves the interpreter as
$CONTRA_PYTHON, then .venv, then python3 — so a machine whose system
python3 already has the packages needs no venv and notices nothing.

Neither gate is softened. A missing oracle does NOT skip: a skipped oracle
is indistinguishable from a passing one, and these are what stop the QR
and PDF writers being checked only against themselves — which is how the
reversed format-information bits were caught, a fault a decode-only check
would have called "does not scan" and left us guessing. What changed is
that a missing package now fails with the command that fixes it instead of
a traceback that says nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Paul Campbell pushed by paul@keymail.dev 1 month ago 2b0646a3f616bd5f76d48a85f1a0555eb0c02164 parent d1623b2

7 files changed, +123 −20

.amadan/ci.d/35-js-test +17 −6
.gitignore +4 −0
Makefile +21 −2
test/kit.test.mjs +2 −5
test/oracle.mjs +60 −0
test/qr.test.mjs +2 −7
test/requirements.txt +17 −0

CI failed — run details

CI log

=== vet ===
go vet ./...
=== fmt ===
=== test ===
go test ./...
?   	github.com/carlosframework/contra/cmd/contra	[no test files]
?   	github.com/carlosframework/contra/internal/contra	[no test files]
?   	github.com/carlosframework/contra/internal/contra/auth	[no test files]
ok  	github.com/carlosframework/contra/internal/contra/crypto	(cached)
?   	github.com/carlosframework/contra/internal/contra/icons	[no test files]
?   	github.com/carlosframework/contra/internal/contra/mail	[no test files]
?   	github.com/carlosframework/contra/internal/contra/model	[no test files]
?   	github.com/carlosframework/contra/internal/contra/store	[no test files]
?   	github.com/carlosframework/contra/internal/contra/vault	[no test files]
ok  	github.com/carlosframework/contra/internal/contratest	(cached)
=== js-test ===
python3 -m venv .venv
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt install python3.12-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: /var/lib/amadan-runner/w2/repos/carlos__contra/wt/web__shared-vaults/.venv/bin/python3

make: *** [Makefile:114: oracles] Error 1

amadan: step js-test failed: exit status 2

amadan: job failed at step js-test — its output is above, not at the tail of this log

diff --git a/.amadan/ci.d/35-js-test b/.amadan/ci.d/35-js-test

index adfd98c..38be170 100755

--- a/.amadan/ci.d/35-js-test

+++ b/.amadan/ci.d/35-js-test

@@ -1,10 +1,21 @@

#!/bin/sh

-# One step of CONTRA's CI. It runs a Makefile target rather than its own copy

-# of the commands, so CI and the local gate stay ONE definition: `make ci`

-# runs these same targets in this same order, and `make test-js` is exactly

-# what CI runs here. Debugging CI is running it.

+# One step of CONTRA's CI, run by `amadan runner`.

-# Numbered 35 so it sits where `make ci` puts it, after the Go suite: the

-# browser crypto only matters once the server's compiles.

+# `make oracles` first, then `make test-js`.

+# The shared runner has python3 and none of the oracle packages, and the box

+# is immutable-boot — its user-data is fixed at launch — so installing them

+# there is a launch-template version and an instance replacement. The repo

+# can do it for itself in a venv instead, which is also the reproducible

+# answer: test/requirements.txt pins the versions, so the oracle cannot

+# drift under us between runs.

+# Both are HARD gates and this script softens neither. test-js FAILS when

+# node is missing rather than skipping, because a security product must not

+# quietly stop checking that the browser's crypto matches the server's; and

+# the oracles fail with an instruction rather than skipping, because a

+# skipped oracle is indistinguishable from a passing one on the two

+# documents a person's whole account depends on being readable.

set -eu

+make oracles

exec make test-js

diff --git a/.gitignore b/.gitignore

index 8314bab..3868e97 100644

--- a/.gitignore

+++ b/.gitignore

@@ -10,3 +10,7 @@

/contra.db-wal

/contra.db-shm

/node_modules/

+# The oracle venv, built by `make oracles`. Pinned by test/requirements.txt,

+# so it is reproducible rather than worth committing.

+/.venv/

diff --git a/Makefile b/Makefile

index 0133681..ead136b 100644

--- a/Makefile

+++ b/Makefile

@@ -22,7 +22,7 @@ RELEASE_BIN := releases/$(APP)-$(RELEASE_GOOS)-$(RELEASE_GOARCH)

# make release VERSION=v0.1.0

VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null)

-.PHONY: build release version-check test test-js test-webkit vet fmt-check migration-check ci

+.PHONY: build release version-check test test-js test-webkit oracles vet fmt-check migration-check ci

# build is the compile check: with more than one package matched, go

# build discards the output, so this catches a broken package without

@@ -96,7 +96,26 @@ test-js:

echo " This gate is not optional -- it is what proves the browser's"; \

echo " crypto matches the server's. Install node, do not skip it."; \

exit 1; }

- node --test test/crypto.test.mjs test/items.test.mjs test/qr.test.mjs test/kit.test.mjs

+ node --test test/crypto.test.mjs test/ecies.test.mjs test/items.test.mjs \

+ test/qr.test.mjs test/kit.test.mjs

+# The third-party oracles' Python packages, into a local .venv.

+# Opt-in rather than a dependency of test-js: a machine whose system python3

+# already has them needs nothing, and test/oracle.mjs prefers .venv only when

+# it exists. What this exists for is the shared CI runner, which has python3

+# and none of the packages — the box is immutable-boot, so the fix has to be

+# something the repo can do for itself rather than an AWS change.

+# Pinned in test/requirements.txt, because an oracle that drifts is an oracle

+# that stops meaning what it said last week.

+oracles:

+ @command -v python3 >/dev/null 2>&1 || { \

+ echo "make oracles: python3 is not installed."; exit 1; }

+ python3 -m venv .venv

+ .venv/bin/pip install --quiet --upgrade pip

+ .venv/bin/pip install --quiet -r test/requirements.txt

+ @echo "oracles ready: .venv/bin/python3"

# The browser crypto in WebKit, which is the engine this product can least

# afford to be wrong about and the one the Chromium rig never touches.

diff --git a/test/kit.test.mjs b/test/kit.test.mjs

index 24d8cf4..90550a9 100644

--- a/test/kit.test.mjs

+++ b/test/kit.test.mjs

@@ -12,9 +12,9 @@

import { test } from "node:test";

import assert from "node:assert/strict";

-import { execFileSync } from "node:child_process";

import { fileURLToPath } from "node:url";

import { dirname, join } from "node:path";

+import { run as runOracle } from "./oracle.mjs";

const here = dirname(fileURLToPath(import.meta.url));

const mod = (name) => join(here, "..", "internal", "contra", "static", "crypto", name);

@@ -30,10 +30,7 @@ const RECOVERY = fromHex("70da5ffc983510bba4672030946b9eed");

const WHEN = new Date(Date.UTC(2026, 8, 2));

function ask(cases) {

- const out = execFileSync("python3", [join(here, "pdforacle.py")], {

- input: JSON.stringify(cases), encoding: "utf8", maxBuffer: 1 << 28,

- });

- return JSON.parse(out);

+ return JSON.parse(runOracle("pdforacle.py", cases));

}

const b64 = (bytes) => Buffer.from(bytes).toString("base64");

diff --git a/test/oracle.mjs b/test/oracle.mjs

new file mode 100644

index 0000000..6f7426e

--- /dev/null

+++ b/test/oracle.mjs

@@ -0,0 +1,60 @@

+// Finding the Python the third-party oracles run under.

+//

+// The oracles are what stop the QR writer and the PDF writer being checked

+// only against themselves: zxing-cpp decodes what we drew, python-qrcode

+// draws what we drew and we compare matrices, PyMuPDF and pypdf read the

+// documents back. An encoder tested against its own decoder proves the two

+// agree and nothing else.

+//

+// They need python3 with numpy, zxing-cpp, qrcode, pillow, pymupdf and

+// pypdf. A machine — or a shared CI runner — may not have those, and the

+// failure was a raw ModuleNotFoundError traceback that said nothing about

+// what to do. This resolves the interpreter and turns a missing dependency

+// into an instruction.

+//

+// Order: $CONTRA_PYTHON, then .venv/bin/python3 if `make oracles` has been

+// run, then python3. So a developer whose system already has the packages

+// needs no venv, and CI gets a reproducible one without changing the box.

+import { execFileSync } from "node:child_process";

+import { existsSync } from "node:fs";

+import { fileURLToPath } from "node:url";

+import { dirname, join } from "node:path";

+const here = dirname(fileURLToPath(import.meta.url));

+const repo = join(here, "..");

+export function oraclePython() {

+ if (process.env.CONTRA_PYTHON) return process.env.CONTRA_PYTHON;

+ const venv = join(repo, ".venv", "bin", "python3");

+ if (existsSync(venv)) return venv;

+ return "python3";

+// run executes one oracle and returns its stdout.

+//

+// It does NOT skip when the packages are missing. A skipped oracle is

+// indistinguishable from a passing one, and these guard the two documents a

+// person's whole account depends on being able to read back — that is

+// exactly the class of check this repo refuses to let degrade silently.

+// What it does is fail with the command that fixes it.

+export function run(script, input) {

+ const python = oraclePython();

+ try {

+ return execFileSync(python, [join(here, script)], {

+ input: JSON.stringify(input),

+ maxBuffer: 64 << 20,

+ }).toString();

+ } catch (err) {

+ const text = String(err.stderr || err.message || "");

+ if (/ModuleNotFoundError|No module named|ENOENT/.test(text)) {

+ throw new Error(

+ `the ${script} oracle cannot run: its Python packages are missing.\n` +

+ ` Interpreter tried: ${python}\n` +

+ ` Fix it with: make oracles\n` +

+ ` (or set CONTRA_PYTHON to an interpreter that has numpy, zxing-cpp,\n` +

+ ` qrcode, pillow, pymupdf and pypdf)\n\n` + text);

+ }

+ throw err;

+ }

diff --git a/test/qr.test.mjs b/test/qr.test.mjs

index 1c777b6..af333e6 100644

--- a/test/qr.test.mjs

+++ b/test/qr.test.mjs

@@ -24,9 +24,9 @@

import { test } from "node:test";

import assert from "node:assert/strict";

-import { execFileSync } from "node:child_process";

import { fileURLToPath } from "node:url";

import { dirname, join } from "node:path";

+import { run as runOracle } from "./oracle.mjs";

const here = dirname(fileURLToPath(import.meta.url));

const { encode, smallestVersion } = await import(

@@ -46,12 +46,7 @@ const PAYLOADS = [

];

function ask(cases) {

- const out = execFileSync("python3", [join(here, "qroracle.py")], {

- input: JSON.stringify(cases),

- encoding: "utf8",

- maxBuffer: 1 << 28,

- });

- return JSON.parse(out);

+ return JSON.parse(runOracle("qroracle.py", cases));

}

test("every symbol scans, and is the symbol another encoder would have made", () => {

diff --git a/test/requirements.txt b/test/requirements.txt

new file mode 100644

index 0000000..adbeb6c

--- /dev/null

+++ b/test/requirements.txt

@@ -0,0 +1,17 @@

+# The third-party oracles' Python packages.

+# These are what stop the QR writer and the PDF writer being checked only

+# against themselves. zxing-cpp decodes the symbols we drew; python-qrcode

+# draws the same payload independently so the two matrices can be compared

+# module by module — which is how the reversed format-information bits were

+# found, a fault a decode-only check would have reported as "does not scan"

+# and left us guessing. PyMuPDF and pypdf read the two documents back.

+# Pinned, because an oracle that drifts is an oracle that stops meaning what

+# it said last week. Install with `make oracles`.

+numpy==2.1.3

+pillow==11.0.0

+pypdf==5.1.0

+pymupdf==1.24.14

+qrcode==8.0

+zxing-cpp==2.2.0