amadan

carlos / todomvc-bakeoff Public

chore: all builds, reviews and fix rounds; measurement scripts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paul Campbell pushed by claude 9bb16b2a2e7213e02043d808b1ce281bcfeb30f1 parent 7569dda
11 files changed, +294 −39
  • builders/reviews/haskell.1.md +3 −0
  • builders/reviews/haskell.2.md +1 −0
  • builders/reviews/ocaml.2.md +1 −0
  • builders/reviews/spinel.1.md +7 −0
  • builders/reviews/spinel.2.md +5 −0
  • results/haskell.json +19 −19
  • results/mojo.json +102 −0
  • results/ocaml.json +22 −20
  • results/spinel.json +106 −0
  • scripts/measure-build.sh +12 −0
  • scripts/measure-live.sh +16 −0
diff --git a/builders/reviews/haskell.1.md b/builders/reviews/haskell.1.md
new file mode 100644
index 0000000..14c0539
--- /dev/null
+++ b/builders/reviews/haskell.1.md
@@ -0,0 +1,3 @@
+1. [correctness] `apps/haskell/static/app.js:22` — Every action response replaces the entire list, discarding any edit started while that request was pending. Toggle one item, then double-click another and type before the response arrives: the editor disappears and the typed changes are lost. Preserve pending edits when applying responses.
+
+2. [spec] `apps/haskell/app/Main.hs:103` — `parseListen` accepts any `127.x.x.x` address, although the brief requires binding only to `127.0.0.1`. For example, `--listen 127.0.0.2:8080` passes validation. Require the exact permitted address.
\ No newline at end of file
diff --git a/builders/reviews/haskell.2.md b/builders/reviews/haskell.2.md
new file mode 100644
index 0000000..f6e8b4b
--- /dev/null
+++ b/builders/reviews/haskell.2.md
@@ -0,0 +1 @@
+No findings.
\ No newline at end of file
diff --git a/builders/reviews/ocaml.2.md b/builders/reviews/ocaml.2.md
new file mode 100644
index 0000000..c2ed70e
--- /dev/null
+++ b/builders/reviews/ocaml.2.md
@@ -0,0 +1 @@
+1. [correctness] `apps/ocaml/static/app.js:14` — Filter navigation can discard queued edits. With a toggle request pending, edit another item and click “Active”: blur queues the save, but the ordinary filter link immediately unloads the page before that save runs. The changed title is lost. Wait for pending mutations before navigating between filters.
\ No newline at end of file
diff --git a/builders/reviews/spinel.1.md b/builders/reviews/spinel.1.md
new file mode 100644
index 0000000..a9e9051
--- /dev/null
+++ b/builders/reviews/spinel.1.md
@@ -0,0 +1,7 @@
+1. [security] `apps/spinel/patches/2-tep-bakeoff.patch:152` — The five-second timeout applies separately to each socket read. A client sending another body byte every four seconds can occupy the sole worker for hours, blocking every visitor. Enforce a deadline for the entire request.
+
+2. [correctness] `apps/spinel/.cache/x86/tep/tep/sphttp.c:254` — The dependency’s body reader silently caps each drain at 65,535 bytes, and `consume_body` calls it only once. A 10,000-emoji title encodes to about 120,000 bytes; sending headers before the body causes the title to be truncated and saved with a success response. Patch the dependency to read complete bodies, support the required title length, and reject incomplete requests before mutation.
+
+3. [spec] `apps/spinel/app.rb:203` — Any syntactically valid 22-character cookie is accepted without checking whether the server issued it. An unknown `tid=AAAAAAAAAAAAAAAAAAAAAA` remains the list identifier, and separate clients supplying it share subsequent todos. Track issued tokens and replace unrecognised values with fresh random cookies.
+
+4. [spec] `apps/spinel/patches/2-tep-bakeoff.patch:21` — `--listen` accepts arbitrary addresses, including `0.0.0.0`; omitting the listener also retains Tep’s wildcard-binding fallback. This violates the loopback-only requirement. Reject non-loopback addresses and require a valid socket or loopback listener.
\ No newline at end of file
diff --git a/builders/reviews/spinel.2.md b/builders/reviews/spinel.2.md
new file mode 100644
index 0000000..947ea42
--- /dev/null
+++ b/builders/reviews/spinel.2.md
@@ -0,0 +1,5 @@
+1. [correctness] `apps/spinel/app.rb:475` and `:527` — `strip` only removes ASCII whitespace. A title containing only U+00A0 or U+2003 creates a blank-looking todo, and editing to that value does not delete it. Apply Unicode whitespace trimming before the empty-title check.
+
+2. [correctness] `apps/spinel/patches/2-tep-bakeoff.patch:330` — URL decoding turns `%00` into a NUL inside a C string, which is then truncated when copied into Ruby. Posting `title=before%00after` silently stores `before`; editing with `title=%00after` deletes the todo. Preserve decoded string lengths so input cannot silently become a different title or action.
+
+3. [spec] `apps/spinel/Makefile:130` — `clean-deps` deletes only the local SQLite and tep artifacts. The arm64 build still links precompiled OpenSSL, zlib and zstd libraries from the shared toolchain root (`scripts/arm64-cc:21`), so the next build does not compile every third-party library from scratch as required. Build those dependencies into the app’s own cache and include them in `clean-deps`.
\ No newline at end of file
diff --git a/results/haskell.json b/results/haskell.json
index 215e823..e145d3a 100644
--- a/results/haskell.json
+++ b/results/haskell.json
@@ -3,7 +3,7 @@
"Language": "Haskell",
"Status": "LIVE",
"URL": "https://todo-haskell.bfj.oncarlos.com",
- "Version": "7881a7e",
+ "Version": "7d40789",
"Stack": "",
"Source": "",
"BehaviourJSOn": {
@@ -62,13 +62,13 @@
"CostUSD": 4.4539882
},
"TokensFix": {
- "Input": 0,
- "Output": 0,
- "CacheRead": 0,
- "CacheWrite": 0,
- "Turns": 0,
- "Seconds": 0,
- "CostUSD": 0
+ "Input": 20,
+ "Output": 6470,
+ "CacheRead": 1823381,
+ "CacheWrite": 13744,
+ "Turns": 10,
+ "Seconds": 238.426,
+ "CostUSD": 0.6041082000000007
},
"TokensSecurity": {
"Input": 0,
@@ -80,22 +80,22 @@
"CostUSD": 0
},
"TokensReview": {
- "Input": 0,
- "Output": 0,
- "CacheRead": 0,
+ "Input": 664393,
+ "Output": 3522,
+ "CacheRead": 573184,
"CacheWrite": 0,
- "Turns": 0,
+ "Turns": 2,
"Seconds": 0,
"CostUSD": 0
},
"Readability": {
- "Naming": 0,
- "Structure": 0,
- "Traceability": 0,
- "Errors": 0,
- "Comments": 0,
- "Overall": 0,
- "Notes": ""
+ "Naming": 4,
+ "Structure": 4,
+ "Traceability": 5,
+ "Errors": 3,
+ "Comments": 5,
+ "Overall": 8,
+ "Notes": "Strongest: tracing a request is effortless. The toggle route goes from the pathInfo case through the `post` helper to Db.toggleTodo and back to Html.fragment, with the Bool-means-found contract stated at each end. Comments consistently explain why (WAL via query_, getrandom over /dev, single connection, the `swapping` flag) without restating code. Weakest: error handling is mostly implicit. Database exceptions reach Warp's default 500 with no logging; migrate silently assumes version 0 on an odd PRAGMA result; the getrandom loop retries on any negative result, not only EINTR; app.js hides every failure behind location.reload(). Minor: `app`'s where-block is long and the multiway-if in `post` is dense; `common`, `with`, `f`, `v` are terse names."
},
"ReviewFindings": null,
"Notes": null
diff --git a/results/mojo.json b/results/mojo.json
new file mode 100644
index 0000000..dc46f14
--- /dev/null
+++ b/results/mojo.json
@@ -0,0 +1,102 @@
+{
+ "Slug": "mojo",
+ "Language": "Mojo",
+ "Status": "BLOCKED platform: single-file binaries only (Mojo needs libKGENCompilerRTShared.so)",
+ "URL": "",
+ "Version": "",
+ "Stack": "",
+ "Source": "",
+ "BehaviourJSOn": {
+ "Passed": 0,
+ "Failed": 0,
+ "Skipped": 0
+ },
+ "BehaviourJSOff": {
+ "Passed": 0,
+ "Failed": 0,
+ "Skipped": 0
+ },
+ "TTFBEmpty": {
+ "N": 0,
+ "P50": 0,
+ "P95": 0,
+ "P99": 0,
+ "Min": 0,
+ "Max": 0
+ },
+ "TTFB50": {
+ "N": 0,
+ "P50": 0,
+ "P95": 0,
+ "P99": 0,
+ "Min": 0,
+ "Max": 0
+ },
+ "Wake": null,
+ "Load": null,
+ "Weight": {
+ "HTMLBytes": 0,
+ "JSBytes": 0
+ },
+ "Code": {
+ "Server": 1007,
+ "Templates": 0,
+ "JS": 62,
+ "Tests": 251,
+ "Deps": 1
+ },
+ "Build": {
+ "ColdSeconds": 0,
+ "WarmSeconds": 0,
+ "EditSeconds": 0,
+ "Bytes": 0,
+ "Host": ""
+ },
+ "TokensBuild": {
+ "Input": 170,
+ "Output": 83318,
+ "CacheRead": 9923207,
+ "CacheWrite": 167410,
+ "Turns": 86,
+ "Seconds": 1242.5520000000001,
+ "CostUSD": 4.9909614000000015
+ },
+ "TokensFix": {
+ "Input": 0,
+ "Output": 0,
+ "CacheRead": 0,
+ "CacheWrite": 0,
+ "Turns": 0,
+ "Seconds": 0,
+ "CostUSD": 0
+ },
+ "TokensSecurity": {
+ "Input": 0,
+ "Output": 0,
+ "CacheRead": 0,
+ "CacheWrite": 0,
+ "Turns": 0,
+ "Seconds": 0,
+ "CostUSD": 0
+ },
+ "TokensReview": {
+ "Input": 0,
+ "Output": 0,
+ "CacheRead": 0,
+ "CacheWrite": 0,
+ "Turns": 0,
+ "Seconds": 0,
+ "CostUSD": 0
+ },
+ "Readability": {
+ "Naming": 4,
+ "Structure": 4,
+ "Traceability": 4,
+ "Errors": 4,
+ "Comments": 4,
+ "Overall": 8,
+ "Notes": "Strongest: clean module split (routes, store, HTTP parsing, libc shims, render) with module docstrings; why-comments where it matters (fixed TERM_PIPE_FD and signal-handler constraints, SQLITE_TRANSIENT, migrations, the fetch same-round-trip page). SQLite errors surface with sqlite3_errmsg and handle() logs and returns 500. Weakest: _action dispatches on len(parts) of a split path, so the URL shape is implicit rather than stated; _list returns a token but reads like it returns todos; Stmt.int/text shadow builtins; several external_call results discarded with `_ =` and the JS swallows every failure into location.reload(); page() is one long string-concatenation function that is hard to scan against the template."
+ },
+ "ReviewFindings": null,
+ "Notes": null
+}
diff --git a/results/ocaml.json b/results/ocaml.json
index 72f20a4..8f34b51 100644
--- a/results/ocaml.json
+++ b/results/ocaml.json
@@ -3,7 +3,7 @@
"Language": "OCaml",
"Status": "LIVE",
"URL": "https://todo-ocaml.bfj.oncarlos.com",
- "Version": "b927326",
+ "Version": "a24ca85",
"Stack": "",
"Source": "",
"BehaviourJSOn": {
@@ -62,13 +62,13 @@
"CostUSD": 5.5922082
},
"TokensFix": {
- "Input": 0,
- "Output": 0,
- "CacheRead": 0,
- "CacheWrite": 0,
- "Turns": 0,
- "Seconds": 0,
- "CostUSD": 0
+ "Input": 62,
+ "Output": 23981,
+ "CacheRead": 6448874,
+ "CacheWrite": 35642,
+ "Turns": 31,
+ "Seconds": 364.49199999999996,
+ "CostUSD": 2.0547788000000002
},
"TokensSecurity": {
"Input": 0,
@@ -80,23 +80,25 @@
"CostUSD": 0
},
"TokensReview": {
- "Input": 0,
- "Output": 0,
- "CacheRead": 0,
+ "Input": 794733,
+ "Output": 4130,
+ "CacheRead": 689664,
"CacheWrite": 0,
- "Turns": 0,
+ "Turns": 2,
"Seconds": 0,
"CostUSD": 0
},
"Readability": {
- "Naming": 0,
- "Structure": 0,
- "Traceability": 0,
- "Errors": 0,
- "Comments": 0,
- "Overall": 0,
- "Notes": ""
+ "Naming": 4,
+ "Structure": 4,
+ "Traceability": 5,
+ "Errors": 4,
+ "Comments": 5,
+ "Overall": 8,
+ "Notes": "Strongest: a toggle is trivially traceable (handle → action_of_path → post → perform → Store.toggle → View.fragment or 303), helped by App being server-independent and each module having one clear job. Comments are nearly all 'why' (Budget rationale, cookie age, migration rule, CSP hash, JS render invariants) with no noise. Weakest: handle's `filter p \u003c\u003e None ... Option.get (filter p)` double evaluation is clunky and hides intent; Store errors surface only as failwith exceptions caught by run_server's generic logger, and user_version's exec result is ignored; main.ml's Budget/connection plumbing and nested try blocks take effort to follow; a few terse names (change, shows, done_, go) need a moment."
},
- "ReviewFindings": null,
+ "ReviewFindings": [
+ "[correctness] `apps/ocaml/static/app.js:14` — Filter navigation can discard queued edits. With a toggle request pending, edit another item and click “Active”: blur queues the save, but the ordinary filter link immediately unloads the page before that save runs. The changed title is lost. Wait for pending mutations before navigating between filters."
+ ],
"Notes": null
}
diff --git a/results/spinel.json b/results/spinel.json
new file mode 100644
index 0000000..0a1f6a3
--- /dev/null
+++ b/results/spinel.json
@@ -0,0 +1,106 @@
+{
+ "Slug": "spinel",
+ "Language": "Spinel",
+ "Status": "LIVE",
+ "URL": "https://todo-spinel.bfj.oncarlos.com",
+ "Version": "6692def",
+ "Stack": "",
+ "Source": "",
+ "BehaviourJSOn": {
+ "Passed": 35,
+ "Failed": 0,
+ "Skipped": 0
+ },
+ "BehaviourJSOff": {
+ "Passed": 32,
+ "Failed": 0,
+ "Skipped": 3
+ },
+ "TTFBEmpty": {
+ "N": 0,
+ "P50": 0,
+ "P95": 0,
+ "P99": 0,
+ "Min": 0,
+ "Max": 0
+ },
+ "TTFB50": {
+ "N": 0,
+ "P50": 0,
+ "P95": 0,
+ "P99": 0,
+ "Min": 0,
+ "Max": 0
+ },
+ "Wake": null,
+ "Load": null,
+ "Weight": {
+ "HTMLBytes": 0,
+ "JSBytes": 0
+ },
+ "Code": {
+ "Server": 0,
+ "Templates": 0,
+ "JS": 0,
+ "Tests": 0,
+ "Deps": 0
+ },
+ "Build": {
+ "ColdSeconds": 0,
+ "WarmSeconds": 0,
+ "EditSeconds": 0,
+ "Bytes": 0,
+ "Host": ""
+ },
+ "TokensBuild": {
+ "Input": 258,
+ "Output": 99309,
+ "CacheRead": 22408692,
+ "CacheWrite": 276287,
+ "Turns": 132,
+ "Seconds": 4631.851,
+ "CostUSD": 8.679246399999998
+ },
+ "TokensFix": {
+ "Input": 44,
+ "Output": 20701,
+ "CacheRead": 6777148,
+ "CacheWrite": 31920,
+ "Turns": 22,
+ "Seconds": 565.253,
+ "CostUSD": 2.0249855999999955
+ },
+ "TokensSecurity": {
+ "Input": 0,
+ "Output": 0,
+ "CacheRead": 0,
+ "CacheWrite": 0,
+ "Turns": 0,
+ "Seconds": 0,
+ "CostUSD": 0
+ },
+ "TokensReview": {
+ "Input": 916819,
+ "Output": 4711,
+ "CacheRead": 797824,
+ "CacheWrite": 0,
+ "Turns": 2,
+ "Seconds": 0,
+ "CostUSD": 0
+ },
+ "Readability": {
+ "Naming": 4,
+ "Structure": 4,
+ "Traceability": 5,
+ "Errors": 3,
+ "Comments": 4,
+ "Overall": 7,
+ "Notes": "Strongest: traceability. Each route reads top to bottom: origin check, visitor, path_id, todo_exists, one inline SQL statement, finish_post, render_lists. Comments explain the non-obvious (Spinel subset, single-cursor shim, token-issuance check, BUILD_NOTE). Weakest: errors. Write paths (run_sql, add_todo, rename_todo, record_token) discard DB.step's result, so a failed INSERT or UPDATE silently returns a stale fragment; the JS catch-all reload hides failures too. Lesser points: parallel arrays with a `done` field of ints, generic run_sql with a magic `id \u003e 0` bind rule, layout/page/path_id filed under the Actions divider, and unexplained constants (18 and 19 character limits)."
+ },
+ "ReviewFindings": [
+ "[correctness] `apps/spinel/app.rb:475` and `:527` — `strip` only removes ASCII whitespace. A title containing only U+00A0 or U+2003 creates a blank-looking todo, and editing to that value does not delete it. Apply Unicode whitespace trimming before the empty-title check.",
+ "[correctness] `apps/spinel/patches/2-tep-bakeoff.patch:330` — URL decoding turns `%00` into a NUL inside a C string, which is then truncated when copied into Ruby. Posting `title=before%00after` silently stores `before`; editing with `title=%00after` deletes the todo. Preserve decoded string lengths so input cannot silently become a different title or action.",
+ "[spec] `apps/spinel/Makefile:130` — `clean-deps` deletes only the local SQLite and tep artifacts. The arm64 build still links precompiled OpenSSL, zlib and zstd libraries from the shared toolchain root (`scripts/arm64-cc:21`), so the next build does not compile every third-party library from scratch as required. Build those dependencies into the app’s own cache and include them in `clean-deps`."
+ ],
+ "Notes": null
+}
diff --git a/scripts/measure-build.sh b/scripts/measure-build.sh
new file mode 100755
index 0000000..ad8b6ca
--- /dev/null
+++ b/scripts/measure-build.sh
@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+# Build measurements for one app, in its worktree: cold, warm and
+# one-line-change builds of the arm64 binary (median of 3), its size, and
+# the code counts. Run apps one at a time, with nothing else heavy running.
+# scripts/measure-build.sh <slug> [runs]
+set -euo pipefail
+slug=$1 runs=${2:-3}
+cd "$(dirname "$0")/.."
+source scripts/env.sh
+dir=.worktrees/$slug/apps/$slug
+bakeoff code --slug "$slug" --dir "$dir"
+bakeoff build --slug "$slug" --dir "$dir" --runs "$runs"
diff --git a/scripts/measure-live.sh b/scripts/measure-live.sh
new file mode 100755
index 0000000..a684c04
--- /dev/null
+++ b/scripts/measure-live.sh
@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+# Network measurements for one live app, from docs-t1: the shared suite,
+# time to first byte (empty and 50 items), page weight, then the six load
+# runs. Run apps one at a time, never in parallel.
+# scripts/measure-live.sh <slug>
+set -euo pipefail
+slug=$1
+cd "$(dirname "$0")/.."
+url=https://todo-$slug.bfj.oncarlos.com
+out=$(mktemp -d -p "${TMPDIR:-$HOME/.cache}")
+trap 'rm -rf "$out"' EXIT
+scripts/test-app.sh "$url" "$out/suite.json" >/dev/null 2>&1 || true
+bakeoff behaviour --slug "$slug" --file "$out/suite.json"
+bakeoff ttfb --slug "$slug"
+bakeoff weight --slug "$slug"
+bakeoff load --slug "$slug"