| new file mode 100644 |
| index 0000000..f95940b |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-23-todomvc-bakeoff.md |
| @@ -0,0 +1,250 @@ |
| +# π€ TodoMVC Bakeoff Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Nine TodoMVC apps (server-rendered, progressively enhanced) built by Opus 5.5 in nine language setups, deployed to Carloku account `bfj`, measured by one shared harness, and compared on a static report page on the same account. |
| + |
| +**Architecture:** This session (the orchestrator) first builds everything that must be identical for all contestants: the brief, the TodoMVC assets, a pinned HTTP and markup contract, a Playwright behaviour suite, and a Go measurement harness. Both are proven against a small reference fixture app that builders never see. Then headless `claude` builder sessions (one per language, each in its own sparse git worktree) build and deploy. Astra reviews through `codex exec`, the same builder session fixes, and Fable scores readability blind. Finally the harness measures the live apps and renders the report. |
| + |
| +**Tech Stack:** Go 1.26 (harness, reference fixture, report generator; stdlib only), Node 24 + `@playwright/test` (behaviour suite), `oha` (load generator), `qemu-aarch64-static` (local arm64 smoke runs), `carlos` CLI v0.28, `claude` CLI (Opus 5.5 builders, Fable 5.1 scorer), `codex` CLI (Astra reviewer), `amadan` CLI (platform requests). |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-23-todomvc-bakeoff-design.md` |
| + |
| +## Global Constraints |
| + |
| +- Contestant slugs, in report order: `go`, `go-rastrillo`, `rust`, `zig`, `ocaml`, `haskell`, `swift`, `mojo`, `spinel`. |
| +- Carloku account `bfj`. App names `todo-<slug>`. Hosts `todo-<slug>.bfj.oncarlos.com`. Report app `bakeoff` at `bakeoff.bfj.oncarlos.com`. |
| +- Process contract: `<bin> --socket <path> --db <path>`; `GET /healthz` β 200; `GET /api/version` β body is the build version; reads `CARLOS_HOST`, never parses the socket path; SQLite in WAL mode; additive migrations; clean exit on SIGTERM; built for `linux-arm64`, static or libc-only; `carlos vet --boot` passes before the first deploy; the app is created with `--build-probe /api/version`. |
| +- For local tests every binary also accepts `--listen 127.0.0.1:<port>` instead of `--socket`. |
| +- Builders set no resource, tier or memory fields; every app runs at the platform default. |
| +- Markup and CSS: TodoMVC `todomvc-app-template` markup with `todomvc-common/base.css` and `todomvc-app-css/index.css`, vendored once in `shared/`. Each app serves those two files byte-identical from `/static/base.css` and `/static/index.css`. |
| +- No client framework and no JS build step. Plain ES modules; vendored VanJS allowed. |
| +- One builder model: `claude-opus-5-5`. Reviewer: Astra through `codex exec`. Readability scorer: `claude-fable-5-1`. |
| +- Effort limit: 4 hours of wall-clock time per builder (`timeout 4h`), then `BLOCKED`. |
| +- One review-and-fix round per app; a second round only for security findings, counted separately. |
| +- Machine rules: servers bind `127.0.0.1` only; scratch in `$TMPDIR`; don't override `GOCACHE`; no `sudo`; stop every process and delete scratch at the end. |
| +- Never work around Carloku. A missing platform capability becomes an `amadan discuss new carlos/platform` Feature Request, and the report names it. |
| +- Commits end with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. |
| + |
| +## The pinned contract (`CONTRACT.md`, written in Task 1) |
| + |
| +The suite and harness drive every app the same way, so these are fixed for every builder: |
| + |
| +| Request | Without JS (no special headers) | |
| +|---|---| |
| +| `GET /`, `/active`, `/completed` | Full page, filtered; `a.selected` on the matching filter link (`href="/"`, `/active`, `/completed`) | |
| +| `GET /<filter>?edit=<id>` | Same page with that `li` given class `editing` and its `input.edit` inside a form posting to `/todos/<id>/edit` | |
| +| `POST /todos` (`title`) | Trimmed; empty β no item. 303 β current filter | |
| +| `POST /todos/<id>/toggle` | 303 | |
| +| `POST /todos/<id>/edit` (`title`) | Trimmed; empty β delete. 303 | |
| +| `POST /todos/<id>/destroy` | 303 | |
| +| `POST /todos/toggle-all` | All complete, or all active if all were complete. 303 | |
| +| `POST /todos/clear-completed` | 303 | |
| + |
| +- Every POST form carries `<input type="hidden" name="filter" value="all|active|completed">`, and the 303 goes to `/`, `/active` or `/completed` to match. Anything else falls back to `/`. |
| +- Every `li` in `ul.todo-list` carries `data-id="<id>"`. |
| +- Controls for use without JS: the toggle is a submit control `[data-action="toggle"]` inside the item's toggle form. Editing is a link `a[data-action="edit"]` to `?edit=<id>`. `button.destroy`, `button.clear-completed` and `[data-action="toggle-all"]` are submit buttons in their forms. `input.new-todo` sits in a form that submits on Enter. With JS on, the page must match the reference visually. With JS off, extra controls may show if they're styled to fit. |
| +- Lists: an anonymous random cookie, `HttpOnly; SameSite=Lax; Path=/`, at least 128 bits of randomness. An unknown or garbled cookie value gets a fresh empty list, never an error. |
| +- Cross-site protection: SameSite plus an Origin check. A POST whose `Origin` is present and isn't `https://$CARLOS_HOST` (or the `--listen` origin locally) gets a 403. There's no CSRF token, so the harness can POST. |
| +- A POST naming an id that isn't in the caller's list gets a 404, never a 500 and never a change to someone else's list. |
| +- The JS may call the same endpoints with any request header it likes, to get fragments or JSON back. The no-header behaviour above must not change. |
| +- The footer reads `<p>Created by <a href="https://bakeoff.bfj.oncarlos.com">the CARLOS bakeoff</a> Β· <LANGUAGE></p>` in place of the template's "Created by you" line. |
| + |
| +## Review Focus |
| + |
| +1. **Titles with HTML in them** (`<img src=x onerror=alert(1)> & "q"`) must show as text and run nothing. Test: `escapes HTML in titles` (Task 3). |
| +2. **Non-ASCII titles** (`naΓ―ve π ζ₯ζ¬θͺ`) round-trip intact, and `Content-Length` equals the bytes sent. Spinel/tep truncated multi-byte responses before. Test: `round-trips UTF-8 and sends correct Content-Length` (Task 3). |
| +3. **Whitespace-only and 10,000-character titles:** the first creates nothing, and the second is stored and shown without a 500. Test: `rejects blank and accepts very long titles` (Task 3). |
| +4. **Separate visitors are separate:** two browser contexts see separate lists, and a garbled cookie gets an empty list. Test: `isolates lists per visitor and survives a garbled cookie` (Task 3). |
| +5. **Hostile POSTs:** a foreign `Origin` gets a 403, and an id from another list gets a 404 with that list unchanged. Test: `rejects cross-site and cross-list posts` (Task 3). |
| + |
| +--- |
| + |
| +### Task 1: Repo, brief, contract and shared assets |
| + |
| +**Files:** |
| +- Create: `.gitignore` (`.worktrees/`, `node_modules/`, `dist/`, `tests/behaviour/test-results/`, `*.db*`) |
| +- Create: `AGENTS.md`: the bakeoff repo's rules (fairness rules from the spec's anti-goals, machine rules, layout below) |
| +- Create: `BRIEF.md`: the builder brief. It's the spec's "What every app must do" section, the Global Constraints that apply to apps, the per-app `Makefile` targets below, the `STATUS.md` protocol, and a pointer to `CONTRACT.md`. |
| +- Create: `CONTRACT.md`: the pinned contract above, verbatim |
| +- Create: `shared/base.css`, `shared/index.css` (from npm `todomvc-common` and `todomvc-app-css`, latest) |
| +- Create: `shared/reference.html`: the template markup with real-path filters and the contract's no-JS controls, showing three items in the visual fixture state: "Buy milk" completed, "Walk the dog", "Write the report", filter All, `2 items left`, Clear completed visible |
| +- Create: `vendor/carlos-skills/`: copy of `carlosframework/skills/skills/{getting-started,building-carlos-apps}` at the current commit, with the commit SHA in `vendor/carlos-skills/SOURCE` |
| +- Create: `apps/.keep`, `results/.keep` |
| + |
| +**Interfaces:** |
| +- Produces: the per-app `Makefile` targets every later task relies on. `make build` builds the host binary to `dist/app`. `make build-arm64` builds `dist/app-linux-arm64`. `make test` runs the app's own tests. `make clean` removes build output only, never the dependency caches. `make clean-deps` also removes every compiled third-party artifact but keeps downloaded sources, so the next `make build-arm64` is a cold build. Where a toolchain keeps compiled libraries in a shared per-user store (cabal, opam, zig, swiftpm), the Makefile points that store inside `apps/<slug>/.cache/` so `clean-deps` can delete it without touching anything else. Go is the exception: never change `GOCACHE`; instead `make build-arm64 COLD=1` passes `-a`, and the report notes that this also recompiles Go's standard library. `make bump` changes the text of one string constant in the file holding the `GET /` handler (so the next build must recompile it), and `make unbump` reverts it. `make deploy` runs `carlos deploy --account bfj --app todo-<slug> --target linux-arm64 ...` and needs no prompts. |
| +- Produces: `apps/<slug>/STATUS.md`, whose first line is exactly `LIVE https://todo-<slug>.bfj.oncarlos.com <version>` or `BLOCKED <one-line reason>`. A blocked app adds a section headed `## Where it got stuck`. The reason `needs-arm64-host` is reserved for "builds and passes on x86-64 but can't produce linux-arm64 on docs-t1". |
| +- Produces: `apps/<slug>/DEPS`, one direct third-party dependency per line (empty file if none). |
| + |
| +- [ ] **Step 1:** `git init -b main`, then write the files above. |
| +- [ ] **Step 2:** Check that `shared/reference.html` opened in Chromium looks like the TodoMVC template screenshot (`todomvc-app-css/screenshot.png`), apart from the item text. Run: `npx playwright screenshot --browser chromium file://$PWD/shared/reference.html $TMPDIR/ref.png` and look at it. |
| +- [ ] **Step 3:** Commit: `chore: bakeoff repo, brief, contract and shared TodoMVC assets`. |
| + |
| +### Task 2: Reference fixture app (never shown to builders) |
| + |
| +A minimal Go implementation of `CONTRACT.md`, used only to prove the suite and harness work before any builder starts. It keeps data in memory, has no SQLite and isn't a contestant. |
| + |
| +**Files:** |
| +- Create: `harness/go.mod` (module `bakeoff/harness`, go 1.26, stdlib only) |
| +- Create: `harness/refapp/main.go`, `harness/refapp/app.js`, `harness/refapp/page.html.tmpl` |
| +- Test: `harness/refapp/refapp_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `go run ./refapp --listen 127.0.0.1:<port>`, a server that fully meets `CONTRACT.md` with JS on and off, serving `shared/*.css` from `/static/`. |
| + |
| +- [ ] **Step 1: Write the failing tests** in `refapp_test.go` with `httptest`. `TestAddRedirectsToFilter`: POST `/todos` with `title=" a "` and `filter=active` β 303 `Location: /active`, and the page then has one `li[data-id]` labelled `a`. `TestForeignOriginForbidden`: `Origin: https://evil.example` β 403. `TestOtherListIDNotFound`: B toggles A's id β 404, and A's item is unchanged. `TestGarbledCookieGetsEmptyList`: cookie `todo=%%%` β 200 with no `li`. |
| +- [ ] **Step 2:** Run `cd harness && go test ./refapp/ -run . -v`. Expected: FAIL (package doesn't build yet). |
| +- [ ] **Step 3:** Implement the handlers and the page template from `shared/reference.html`. `app.js` (β€300 lines) adds double-click to edit, Enter/Escape/blur, and fetch-based toggle/add/destroy that swaps the `section.todoapp` HTML it gets back. |
| +- [ ] **Step 4:** Run `go test ./refapp/ -v`. Expected: PASS. |
| +- [ ] **Step 5:** Commit: `test: reference fixture app implementing the contract`. |
| + |
| +### Task 3: Behaviour suite (Playwright, JS on and JS off) |
| + |
| +**Files:** |
| +- Create: `tests/behaviour/package.json` (pin `@playwright/test` to the version matching the cached `chromium-1234`, otherwise latest plus `npx playwright install chromium`) |
| +- Create: `tests/behaviour/playwright.config.ts`: two projects, `js-on` and `js-off` (`javaScriptEnabled: false`); `baseURL` from `BASE_URL`; `workers: 4`; JSON reporter to `$RESULTS_FILE` when set |
| +- Create: `tests/behaviour/todomvc.spec.ts`: ported from `tastejs/todomvc/cypress/e2e/spec.cy.js` |
| +- Create: `tests/behaviour/robustness.spec.ts`: the five Review Focus tests |
| +- Create: `tests/behaviour/visual.spec.ts`, plus its golden `visual.spec.ts-snapshots/` generated from `shared/reference.html` |
| +- Create: `tests/behaviour/actions.ts`: `addTodo(page, title)`, `toggle(page, i)`, `edit(page, i, title)`, `destroy(page, i)`, `toggleAll(page)`, `clearCompleted(page)`, `gotoFilter(page, f)`. Each checks `testInfo.project.name` and performs the action the JS way (dblclick, checkbox click) or the no-JS way (the contract's submit controls and edit link). |
| +- Create: `scripts/test-app.sh <base-url> <results-file>` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 2's refapp, used as the target during development. |
| +- Produces: `scripts/test-app.sh URL OUT.json`, which runs both projects and writes Playwright JSON. It exits 0 only when everything passes. |
| + |
| +Porting rules. Every Cypress `it` becomes a test in both projects, with these exceptions in `js-off`: `should save edits on blur` and `should cancel edits on escape` are skipped (they're JS-only). `should hide other controls when editing` uses `?edit=<id>`. Persistence checks reload the page instead of reading `localStorage`. Routing uses the real paths. Each test runs in a fresh browser context, which means a fresh cookie and an empty list, so there's no cleanup. |
| + |
| +- [ ] **Step 1: Write `robustness.spec.ts`** (both projects): |
| + - `escapes HTML in titles`: add `<img src=x onerror=alert(1)> & "q"`; `li label` text equals it exactly; `page.on('dialog')` never fires. |
| + - `round-trips UTF-8 and sends correct Content-Length`: add `naΓ―ve π ζ₯ζ¬θͺ`; the label equals it after a reload; `request.get('/')` has `content-length` (when present) equal to `body().length`. |
| + - `rejects blank and accepts very long titles`: `' '` β zero items; `'x'.repeat(10000)` β one item whose label length is 10000. |
| + - `isolates lists per visitor and survives a garbled cookie`: context A adds `a` and context B sees zero items; a third context holding every cookie name A received, each set to the value `%%%`, gets 200 and zero items. |
| + - `rejects cross-site and cross-list posts`: `request.post('/todos', {headers:{Origin:'https://evil.example'}})` β 403; B posts `/todos/<A's id>/toggle` β 404, and A's item is still active. |
| +- [ ] **Step 2:** Write `todomvc.spec.ts` (the 28 ported tests) and `visual.spec.ts`. The visual test seeds the fixture state in the `js-on` project only, then runs `expect(page).toHaveScreenshot({ mask: [page.locator('footer.info')], maxDiffPixelRatio: 0.02 })` against the golden made from `reference.html`. |
| +- [ ] **Step 3:** Run `scripts/test-app.sh http://127.0.0.1:<port> $TMPDIR/ref.json` against the refapp. Expected: all pass; record the counts (passed/skipped per project) in `tests/behaviour/README.md`. |
| +- [ ] **Step 4:** Prove it can fail. Break one rule at a time in the refapp (send a 500 on a foreign id; drop HTML escaping; render `#/` filter links), run the suite, see the matching test fail, and revert. |
| +- [ ] **Step 5:** Commit: `test: TodoMVC behaviour suite, JS on and off, plus robustness and visual checks`. |
| + |
| +### Task 4: Measurement harness |
| + |
| +**Files (all in `harness/`, package `bakeoff`, plus `cmd/bakeoff/main.go` dispatch):** |
| +- `session.go`: `type Session`; `NewSession(base string) (*Session, error)` (cookie jar, doesn't follow redirects, sets `Origin` to base); `(*Session) Add(title string) error`; `(*Session) IDs(path string) ([]string, error)` (parses `li[data-id]`); `(*Session) Toggle(id string) error`; `(*Session) CookieHeader() string`; `Seed(s *Session, n int) ([]string, error)` adds `item 1..n` and toggles every second one |
| +- `ttfb.go`: `type Stats struct{ N int; P50, P95, P99, Min, Max float64 }` (milliseconds); `MeasureTTFB(s *Session, path string, warmup, n int) (Stats, error)`, measured to the first response byte, with a fresh connection for each request |
| +- `wake.go`: `IsAsleep(app, host string) (bool, error)` (runs `carlos instances list --account bfj --app <app> --health asleep` and reports whether `host` is listed); `MeasureWake(s *Session, path string) (float64, error)`, the TTFB of one request |
| +- `load.go`: `type LoadResult struct{ Scenario string; Concurrency int; RPS, P50Ms, P95Ms, P99Ms, ErrorRate float64 }`; `ParseOha(b []byte) (LoadResult, error)`; `RunLoad(s *Session, scenario string, concurrency int, d time.Duration) (LoadResult, error)`. Scenario `read` = `GET /`. Scenario `toggle` = `POST /todos/<first id>/toggle` with `filter=all`, sent with `--redirect 0`, where a 303 counts as success. Both send the session cookie and `Origin`. For `ErrorRate`, anything other than 200 (read) or 303 (toggle), and any transport error, counts as an error. |
| +- `pageweight.go`: `type Weight struct{ HTMLBytes, JSBytes int }`; `PageWeight(s *Session, path string) (Weight, error)` sums the HTML plus every same-origin `<script src>` |
| +- `codestats.go`: `type CodeStats struct{ Server, Templates, JS, Tests, Deps int }`; `CountCode(dir string) (CodeStats, error)` counts non-blank lines. JS is `*.js`/`*.mjs` except `van*.js`. Templates are `*.html`, `*.tmpl`, `*.erb`, `*.mustache`, plus anything under `templates/` or `views/`. Tests are files under `test/` or `tests/`, or named `*_test.*`, `*.test.*`, `*_spec.*`. Server is every other source file. `dist/`, `vendor/`, lockfiles, `*.css` and `*.md` are skipped. Deps is the line count of `DEPS`. |
| +- `buildstats.go`: `type Build struct{ ColdSeconds, WarmSeconds, EditSeconds float64; Bytes int64; Host string }`; `BuildStats(dir string, runs int) (Build, error)`. It runs each mode `runs` times and takes the median. Cold is `make clean clean-deps` then a timed `make build-arm64` (with `COLD=1` for the Go slugs). Warm is `make clean` then a timed `make build-arm64`. Edit is a build, then `make bump`, a timed `make build-arm64`, and `make unbump`. It also records the size of `dist/app-linux-arm64`. |
| +- `tokens.go`: `type Tokens struct{ Input, Output, CacheRead, CacheWrite int64; Turns int; Seconds float64; CostUSD float64 }`; `ParseClaudeLog(r io.Reader) (Tokens, error)` (the final `"type":"result"` event of `--output-format stream-json`: `usage.*`, `num_turns`, `duration_ms`, `total_cost_usd`); `ParseCodexLog(r io.Reader) (Tokens, error)` (sums the usage on `turn.completed` events from `codex exec --json`) |
| +- `results.go`: `type AppResult` holding the slug, language, status, URL, version, stack summary, behaviour pass/fail/skip per project, `TTFBEmpty`/`TTFB50 Stats`, `Wake []float64`, `Load []LoadResult`, `Weight`, `Code CodeStats`, `Build`, `TokensBuild`, `TokensFix`, `TokensSecurity`, `TokensReview Tokens`, `Readability` (scores and notes), `ReviewFindings []string`, `Notes []string`; `Update(slug string, f func(*AppResult)) error` reads and writes `results/<slug>.json` with indented JSON |
| +- `cmd/bakeoff`: subcommands `ttfb|wake|load|weight|code|build|tokens|report`, each taking `--slug` and writing into `results/<slug>.json` through `Update`. `tokens` also takes `--phase build|fix|security|review`, reads `builders/logs/<slug>.<phase>*.jsonl` (with `ParseCodexLog` for `review`, `ParseClaudeLog` otherwise) and sums them into `TokensBuild`, `TokensFix`, `TokensSecurity` or `TokensReview`. |
| +- Tests: `*_test.go` next to each file. Fixtures go in `testdata/`: a real `oha` JSON output, a real `claude` stream-json log and a real `codex --json` log, captured in Step 1. |
| + |
| +**Interfaces:** |
| +- Consumes: CONTRACT routes; Task 2's refapp for testing. |
| +- Produces: `bakeoff <subcommand> --slug <s> [--url U]` β `results/<s>.json`; `type AppResult` (Task 10 renders it). |
| + |
| +- [ ] **Step 1:** Install `oha` from its GitHub release binary into `~/.local/bin`. Capture fixtures: run `oha --output-format json -z 2s` against the refapp; run `claude -p --model claude-opus-5-5 --output-format stream-json --verbose "say ok"`; run `codex exec --json "say ok"`. Save them under `testdata/`. |
| +- [ ] **Step 2: Write the failing tests.** `TestSeed` (refapp: 50 ids, 25 completed from `IDs("/completed")`); `TestParseOha` (fixture β RPS > 0, P99 β₯ P50); `TestParseClaudeLog` / `TestParseCodexLog` (fixtures β Output > 0); `TestCountCode` (a `testdata/codetree` with one file per category and a `vendor/` file that must be ignored β exact counts); `TestPageWeight` (refapp β JSBytes equals the size of `app.js`); `TestBuildStats` (a `testdata/buildtree` Makefile whose targets `sleep` known amounts β Cold > Warm > Edit > 0, and `unbump` was called); `TestMeasureTTFB` (refapp, n=5 β `N==5`, `Min β€ P50 β€ Max`). |
| +- [ ] **Step 3:** Run `go test ./... -short`. Expected: FAIL. |
| +- [ ] **Step 4:** Implement the files listed above. |
| +- [ ] **Step 5:** Run `go test ./... -short`. Expected: PASS. Then run `bakeoff load --slug refapp --url http://127.0.0.1:<port>` and check that `results/refapp.json` has 6 load rows (2 scenarios Γ concurrency 1, 16, 64, 30 s each with a 60 s rest), then delete `results/refapp.json`. |
| +- [ ] **Step 6:** Commit: `feat: measurement harness`. |
| + |
| +### Task 5: Toolchains and arm64 smoke builds |
| + |
| +**Files:** |
| +- Create: `scripts/toolchains.sh`: idempotent; installs under `~` only; prints each tool's version |
| +- Create: `scripts/smoke/<slug>/`: hello-world HTTP-over-unix-socket in each language, plus `scripts/smoke.sh <slug>` |
| +- Create: `docs/toolchains.md`: versions installed and the arm64 route for each language |
| + |
| +**Interfaces:** |
| +- Produces: on `PATH` (through `~/.local/bin` or `source scripts/env.sh`): Go 1.26, `rustup` stable with `aarch64-unknown-linux-musl`, `zig` (latest stable), `opam` + OCaml 5.x, `ghcup` + GHC + cabal, Swift 6.2 + the Static Linux SDK, Mojo through `pixi`, Spinel + tep + spinelkit at the pins in `paulca/lpay/spinel`, `qemu-aarch64-static` (extracted with `apt-get download qemu-user-static && dpkg-deb -x`), and `oha`. Builders don't install compilers; they only add libraries. |
| + |
| +- [ ] **Step 1:** Write `toolchains.sh` and run it. GHC and Mojo downloads are approved (spec). Check `df -h ~` first. |
| +- [ ] **Step 2:** For each slug, `scripts/smoke.sh <slug>` builds a linux-arm64 hello binary that serves `GET /healthz` on `--socket`, checks `file` says `ARM aarch64` and `statically linked` (or libc-only), then runs it under `qemu-aarch64-static` and curls `/healthz` over the socket. Expected: `ok` for each slug. |
| +- [ ] **Step 3:** Any slug that fails gets the reason and the tito-studio fallback written in `docs/toolchains.md`. That slug's builder is told up front (Task 6 prompt `{ARM64_NOTE}`). |
| +- [ ] **Step 4:** Commit: `chore: toolchains and arm64 smoke builds`. |
| + |
| +### Task 6: Builder pipeline and the Go pilot |
| + |
| +**Files:** |
| +- Create: `builders/prompts/build.md`: template with `{SLUG}`, `{LANGUAGE}`, `{EXTRA}`, `{ARM64_NOTE}` |
| +- Create: `builders/extra/<slug>.md`, one per slug: `go` "Don't use Rastrillo"; `go-rastrillo` "Use Rastrillo v0.28 (`amadan.net/rastrillo/rastrillo`); read `~/amadan.net/rastrillo/rastrillo/SKILL.md`; use the TodoMVC CSS, not Rastrillo's UI"; `spinel` "Start from `paulca/lpay` `spinel/` (README, BUILD-arm64.md, patches/tep-carlos.patch)"; `mojo` "A single file is required; if Mojo can't produce one, stop with `BLOCKED single-binary`"; the others are empty |
| +- Create: `builders/worktree.sh <slug>`: `git worktree add .worktrees/<slug> -b app/<slug>` with sparse checkout `/*`, `!/harness/refapp/`, `!/apps/`, `/apps/<slug>/` |
| +- Create: `builders/run-build.sh <slug>`: `cd .worktrees/<slug> && timeout 4h claude -p --model claude-opus-5-5 --permission-mode bypassPermissions --output-format stream-json --verbose "$(render prompt)" > ../../builders/logs/<slug>.build.jsonl`. It records `session_id` from the first event into `builders/logs/<slug>.session`. |
| + |
| +The prompt tells the builder to: read `BRIEF.md`, `CONTRACT.md`, `vendor/carlos-skills/*/SKILL.md` and `references/platform.md`; work only in `apps/{SLUG}/`; not read other apps or the harness source; choose its stack and say why in `README.md`; write `AGENTS.md` first; get `make test` and `scripts/test-app.sh` passing locally against `--listen`; `carlos apps create --account bfj --app todo-{SLUG} --build-probe /api/version`, `carlos vet --boot`, `make deploy`; run `scripts/test-app.sh https://todo-{SLUG}.bfj.oncarlos.com`; commit on its branch; and finish by writing `STATUS.md`. If it hits a Carloku gap, it stops and writes `BLOCKED platform: <what's missing>` instead of working around it. |
| + |
| +**Interfaces:** |
| +- Consumes: Tasks 1, 3 and 5. |
| +- Produces: `builders/logs/<slug>.{build.jsonl,session}`; branch `app/<slug>` with `apps/<slug>/` and `STATUS.md`. |
| + |
| +- [ ] **Step 1:** Write the files. Run `builders/run-build.sh go` in the background. |
| +- [ ] **Step 2:** When it exits, check: `head -1 .worktrees/go/apps/go/STATUS.md` starts with `LIVE`; `curl -sI https://todo-go.bfj.oncarlos.com | grep -i x-carlos-version` matches the version; `scripts/test-app.sh https://todo-go.bfj.oncarlos.com $TMPDIR/go.json` passes; `bakeoff tokens --slug go --phase build` writes nonzero tokens. |
| +- [ ] **Step 3:** Fix any pipeline problem the pilot exposed, in the scripts or the brief only. Never fix the app by hand. If the brief changed, delete `todo-go` (`carlos apps delete`) and the branch, and rerun the pilot so all nine builders get the same brief. |
| +- [ ] **Step 4:** Commit: `feat: builder pipeline; Go pilot`. |
| +- [ ] **Step 5: Checkpoint with Paul.** Show him the pilot's live URL, its tokens and wall-clock time, and the estimated total for eight more. Wait for his go-ahead. |
| + |
| +### Task 7: The other eight builders |
| + |
| +- [ ] **Step 1:** For each of `go-rastrillo rust zig ocaml haskell swift mojo spinel`: `builders/worktree.sh <slug>`, then `builders/run-build.sh <slug>` in the background. All eight run in parallel. |
| +- [ ] **Step 2:** As each exits, read `STATUS.md`. For `BLOCKED needs-arm64-host`, ask Paul for tito-studio access, then continue in the same session (`claude -p --resume <session> "An arm64 host is available: <how>. Continue."`, appending to a separate `<slug>.build2.jsonl` that counts toward build tokens). Count that time against the same 4-hour limit. Any other `BLOCKED` stays blocked. |
| +- [ ] **Step 3:** For every `LIVE` app, run the same checks as Task 6 Step 2. |
| +- [ ] **Step 4:** Merge each `app/<slug>` branch into `main` (`git merge --no-ff`) and commit: `feat: <slug> builder output`. |
| + |
| +### Task 8: Astra review, one fix round, security round |
| + |
| +**Files:** |
| +- Create: `builders/prompts/review.md`: review `apps/{SLUG}/` against `BRIEF.md`, `CONTRACT.md` and the TodoMVC spec. Correctness, spec compliance and security only, with no performance tuning. Output is a numbered list, each item tagged `[security]`, `[correctness]` or `[spec]`, written to `builders/reviews/<slug>.md`. |
| +- Create: `builders/run-review.sh <slug> <n>`: `codex exec -s read-only --json -o builders/reviews/<slug>.<n>.md "$(render)" > builders/logs/<slug>.review<n>.jsonl` (`n` is 1 for the first review, 2 for the re-review) |
| +- Create: `builders/run-fix.sh <slug> <phase>`: `claude -p --resume $(cat <slug>.session) --output-format stream-json --verbose "Fix these review findings, redeploy, re-run the suite, update STATUS.md: $(cat review)" > builders/logs/<slug>.<phase>.jsonl` |
| + |
| +- [ ] **Step 1:** For each LIVE app: `run-review.sh <slug> 1`, then `run-fix.sh <slug> fix` with `builders/reviews/<slug>.1.md`. |
| +- [ ] **Step 2:** Run `run-review.sh <slug> 2`. If it has any `[security]` items, run the fix with phase `security` using only those items. There's no third round. |
| +- [ ] **Step 3:** Store the `[security]`/`[correctness]`/`[spec]` items that remain in `results/<slug>.json` `ReviewFindings`, and the tokens from every phase. |
| +- [ ] **Step 4:** Commit: `chore: review and fix round`. |
| + |
| +### Task 9: Blind readability scoring |
| + |
| +**Files:** |
| +- Create: `builders/prompts/readability.md`: rubric. Score 1β5 each on naming, structure, following one request end to end, error handling, and comments. Give an overall score of 1β10 and a justification of at most 120 words. Output only JSON `{"naming":n,"structure":n,"traceability":n,"errors":n,"comments":n,"overall":n,"notes":"..."}`. |
| +- Create: `builders/run-readability.sh <slug>`: concatenates `apps/<slug>/` source as `CountCode` defines it (server, templates, JS; no tests, vendor or docs) with relative paths, removes the strings `todo-<slug>` and `<slug>` from paths, names only the language, then runs `claude -p --model claude-fable-5-1 --output-format json` three times and stores the per-field median |
| + |
| +- [ ] **Step 1:** Run it for each LIVE app and write the scores to `results/<slug>.json`. |
| +- [ ] **Step 2:** Commit: `chore: readability scores`. |
| + |
| +### Task 10: Measurement on live apps |
| + |
| +Run one app at a time, in slug order, from docs-t1. |
| + |
| +- [ ] **Step 1:** Behaviour: `scripts/test-app.sh <live url> results/<slug>.behaviour.json` β pass/fail/skip into `AppResult`. |
| +- [ ] **Step 2:** `bakeoff ttfb` (empty list and 50 items; warmup 3, n 30), `bakeoff weight`, `bakeoff code`, `bakeoff build --runs 3` (on the host recorded in `docs/toolchains.md`), `bakeoff load` (6 runs). |
| +- [ ] **Step 3:** Wake: once all load runs are done, leave every app idle. Every 5 minutes, check `IsAsleep` for each app. As soon as an app is asleep, `bakeoff wake` records one sample. Repeat until each app has 3 samples (each sample needs another idle period of at least 15 minutes). |
| +- [ ] **Step 4:** Memory: check `carlos instances list --account bfj --app todo-go` and the console's instance view for memory figures. If there are none, search `amadan discuss list carlos/platform` for an existing request. Then file or reply: `amadan discuss new carlos/platform -title "Member-visible memory use per instance" -body -` (type `Feature Request`), explaining the bakeoff use. Record the discussion id in every `AppResult.Notes`. If wake measurement needed anything the platform lacks (for example, a member-side "sleep now"), file that the same way. |
| +- [ ] **Step 5:** Commit: `chore: live measurements`. |
| + |
| +### Task 11: Report page |
| + |
| +**Files:** |
| +- Create: `harness/report/report.go`: `Render(results []AppResult, w io.Writer) error` (html/template plus inline SVG charts, no JS, no external assets) |
| +- Create: `harness/report/report.html.tmpl`, `harness/report/copy.md` (all prose on the page, as one file for copy review) |
| +- Test: `harness/report/report_test.go`: `TestRenderAllApps` (a fixture with one LIVE and one BLOCKED result β output contains both slugs, the BLOCKED reason, and no `NaN` or `<no value>`) |
| +- Output: `report/site/index.html` |
| + |
| +Page sections: summary table (one row per slug, a column per measure, blocked apps shown with their reason), one chart per performance measure, a section per app (stack, notable choices, Astra's remaining findings, Fable's notes, live link, source link), "How we measured", and "What this does and doesn't show" (network included; Spinel's concurrency model; build hosts; blocked apps; memory status). |
| + |
| +- [ ] **Step 1:** Load the `dataviz` skill before writing any chart code. Write the failing test, run `go test ./report/`, and see it fail. |
| +- [ ] **Step 2:** Implement `Render`, then run the test and see it pass. |
| +- [ ] **Step 3:** Draft `copy.md`, then run the `copy-review` skill on it and apply Paul's edits verbatim. |
| +- [ ] **Step 4:** Source links: ask Paul where the repo should be published (amadan or GitHub, public). Publish, then put the links in the results. |
| +- [ ] **Step 5:** `bakeoff report > report/site/index.html`, then `carlos deploy --account bfj --app bakeoff --kind static --host bakeoff.bfj.oncarlos.com report/site`. Verify: `curl -sI https://bakeoff.bfj.oncarlos.com | grep -i x-carlos-version`, and open it in Chromium. |
| +- [ ] **Step 6:** Commit: `feat: bakeoff report page`. |
| + |
| +### Task 12: Clean up and hand off |
| + |
| +- [ ] **Step 1:** Stop every background process this work started (builders, refapp, local app servers, oha). Remove `.worktrees/` (`git worktree remove`) and `$TMPDIR/recon` and other scratch. Keep `builders/logs/`, because they're the token evidence. |
| +- [ ] **Step 2:** Confirm that `ss -ltnp` shows no listener from this work, and that `carlos apps list --account bfj` shows the nine `todo-*` apps (or fewer, matching the BLOCKED list) plus `bakeoff`. |
| +- [ ] **Step 3:** Run the `handoff` skill. |
amadan