amadan
carlos / todomvc-bakeoff Public
feat: write-up page — appraisal of every metric and conclusions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
12 files changed,
+357
−47
.gitignore+1 −0copy-review/strings.json+14 −0docs/writeup.md+83 −0harness/cmd/bakeoff/main.go+23 −1harness/report/markdown.go+28 −2harness/report/report.go+20 −1harness/report/report.html.tmpl+2 −43harness/report/report_test.go+26 −0harness/report/style.tmpl+45 −0harness/report/writeup.html.tmpl+15 −0report/site/index.html+4 −0report/site/writeup.html+96 −0
diff --git a/.gitignore b/.gitignore| index 55d70fe..3322136 100644 |
| --- a/.gitignore |
| +++ b/.gitignore |
| @@ -26,3 +26,4 @@ copy-review/preview/ |
| copy-review/*.log |
| copy-review/draft*.json |
| copy-review/result.json |
| +writing/ |
diff --git a/copy-review/strings.json b/copy-review/strings.json| index b441569..2f6ee12 100644 |
| --- a/copy-review/strings.json |
| +++ b/copy-review/strings.json |
| @@ -15,6 +15,20 @@ |
| "context": "The opening paragraphs under the heading. They say what was built, by whom, where it runs, and that the page compares rather than ranks.", |
| "notes": "Markdown: paragraphs, links, **bold**." |
| }, |
| + { |
| + "id": "page.writeup_link", |
| + "section": "Top", |
| + "label": "Link to the write-up", |
| + "text": "Read the write-up: what the numbers mean and what we conclude", |
| + "context": "A link under the introduction to the longer write-up page." |
| + }, |
| + { |
| + "id": "writeup.back", |
| + "section": "Write-up", |
| + "label": "Back link", |
| + "text": "Back to the report", |
| + "context": "The link at the top of the write-up page, back to the report with the charts." |
| + }, |
| { |
| "id": "findings.heading", |
| "section": "Findings", |
diff --git a/docs/writeup.md b/docs/writeup.md| new file mode 100644 |
| index 0000000..aa066fc |
| --- /dev/null |
| +++ b/docs/writeup.md |
| @@ -0,0 +1,83 @@ |
| +# 🤖 What the TodoMVC bakeoff tells us |
| + |
| +CARLOS and Carloku aren't tied to Go. We built the same small app in nine language setups, and eight of them shipped as a single static ARM executable that runs on Carloku with no changes to the platform. The ninth, Mojo, couldn't make a self-contained executable, so Nim took its place. |
| + |
| +The app is TodoMVC, a small to-do list. The server builds every page, every action works with JavaScript turned off, and a little JavaScript on top makes it feel like the familiar TodoMVC. One AI model, Opus 5.5, built each version from the same brief. A second, Astra, reviewed each one, and the builder got one round to fix what it found. A third, Fable, scored how readable the code is. It was told the language but not which app it was reading. Every app ran on Carloku's default instance size. The charts are on the [report page](https://bakeoff.bfj.oncarlos.com), and the apps, tests and measurement tools are in the [code repository](https://amadan.net/carlos/todomvc-bakeoff). |
| + |
| +## The numbers that matter |
| + |
| +| Setup | Page loads a second | Toggles a second | Memory at rest / peak | Executable | Cold build / one-line rebuild | Cost to build | |
| +|---|---|---|---|---|---|---| |
| +| Go | 349 | 810 | 17 / 71 MB | 12.3 MB | 10 s / 0.1 s | $3.69 | |
| +| Go + Rastrillo | 336 | 309 | 19 / 63 MB | 14.1 MB | 10 s / 0.1 s | $4.67 | |
| +| Rust | 904 (1,398 uncapped) | 1,022 | 14 / 53 MB | 3.2 MB | 50 s / 5.8 s | $2.92 | |
| +| Zig | 1,382 | 3,113 | 2.5 / 25 MB | 1.9 MB | 152 s / 5.4 s | $5.71 | |
| +| OCaml | 992 | 1,058 | 11 / 17 MB | 8.2 MB | emulated | $7.65 | |
| +| Haskell | 400 | 862 | 11 / 77 MB | 28.7 MB | emulated | $5.06 | |
| +| Swift | 537 | 1,453 | 24 / 33 MB | 22.0 MB | 81 s / 3.9 s | $4.60 | |
| +| Nim | 1,491 | 3,561 | 3.5 / 11.5 MB | 1.8 MB | 34 s / 0.9 s | $4.84 | |
| +| Spinel | 722 | 2,451 | 8 / 13 MB | 8.7 MB | emulated | $10.70 | |
| + |
| +No single language came out best. Go is the practical default. Nim and Zig used the least memory and served the most requests. Rust kept pace on speed where the test let it, has the most mature server stack of the three, and was the cheapest to build with AI. How hard it was to get a static ARM executable mattered more to cost and effort than the language did. |
| + |
| +## Correctness |
| + |
| +Every app passed all 67 shared tests: TodoMVC's own 29 tests run with JavaScript on and again with it off, plus checks for unsafe input, very long titles, one visitor reaching another's list, and a screenshot comparison with TodoMVC's page. After the fix round, the review left between zero and three minor findings per app. |
| + |
| +The two security problems were the same bug. A title containing an invalid UTF-8 byte crashed the whole server, in Zig and in Nim, and both were fixed in a second round. Our shared tests only send valid UTF-8, so they missed it and the reviewer caught it. The next bakeoff's tests should send broken input too. |
| + |
| +## Building with AI |
| + |
| +Building cost between $2.92 (Rust) and $10.70 (Spinel), and took the builder between 15 minutes (Go) and 87 minutes (Spinel). The expensive builds were the ones where a static ARM executable was hard to get. Spinel's builder spent most of its time on cross-compiling, and OCaml and Haskell had to be built under emulation, with software on our x86 machine pretending to be an ARM machine. None of this says a language is harder for an AI to write. |
| + |
| +The code itself came out similar everywhere. The apps run from 566 lines (Spinel) to 1,058 (Zig), and readability scores ranged from 7 to 9 out of 10 with most at 8, so they barely separate the languages. |
| + |
| +## Speed under load |
| + |
| +The load test loaded a 50-item page while doubling the number of simultaneous connections every 10 seconds, then did the same for writes, ticking a to-do on and off. Reads fell into three bands: Nim and Zig at about 1,400 to 1,500 page loads a second, Rust, OCaml and Spinel at about 700 to 1,000, and Go, Go with Rastrillo, Haskell and Swift at about 340 to 540. Writes spread wider, from Nim's 3,561 toggles a second down to Go with Rastrillo's 309. |
| + |
| +Put that in proportion before reading much into it. The slowest, Go at 349 page loads a second, is still about 29 million a day. Go's read rate is probably down to its SQLite library, a pure-Go translation chosen so the build needs no C compiler, though we didn't measure that directly. |
| + |
| +Rust's numbers need a footnote. In its first read ramp and its write ramp, Rust was still speeding up at 64 connections when Carloku's edge started turning requests away, so the test never found its limit. In the one ramp where the edge didn't step in, the reads after the writes, Rust reached 1,398 page loads a second, close to Nim's 1,491. |
| + |
| +That edge limit shapes everything above 64 connections. The edge is Carloku's set of front servers that receive traffic from the internet. When a request arrives at an edge server that isn't the one running the app, the edge allows at most 64 such requests in flight at once and answers the rest with `503`, "try again shortly". Whether it applied depended on how our traffic was routed, so results above 64 measure the platform as much as the app. |
| + |
| +Some apps failed outright under very heavy load, which the edge reports as `502`: Swift at 512 connections and Nim at 256. Spinel's server process exited 16 times during its test and Carloku restarted it each time. In one 10-second Spinel step, no request completed at all. |
| + |
| +## Waking up and the first byte |
| + |
| +Carloku puts an app to sleep after 15 quiet minutes. Every app answered its first request after sleeping in 1.2 to 1.45 seconds, and Carloku's own logs show about one second of that is the platform starting the process. The language barely matters. |
| + |
| +The first byte of a page arrived after about 50 to 100 milliseconds. Our test machine reached Carloku over the internet, and the network's base delay drifted between about 17 and 37 milliseconds during the runs, so the two clusters, near 50 and near 95 milliseconds, say more about when each app was measured than about the apps. |
| + |
| +## Memory and size |
| + |
| +Carloku doesn't show members how much memory an instance uses, so we ran each app on our own x86 machine and read it from the operating system. We've asked for the figures in [carlos/platform discussion 365](https://amadan.net/carlos/platform/-/discussion/365). At rest, the apps used 2.5 MB (Zig) to 24 MB (Swift). Under 64 simultaneous connections, Go and Haskell, both garbage-collected, peaked highest at 71 and 77 MB, with Rust next at 53 MB, while Nim, Spinel and OCaml stayed under 17 MB. Every app used a small fraction of an instance's memory limit. |
| + |
| +Executables ranged from 1.8 MB (Nim) to 28.7 MB (Haskell). |
| + |
| +## Builds |
| + |
| +These times are for the ARM executable that ships. A cold build compiles every library from already-downloaded source. A one-line rebuild adds one line to the code that serves the page, with everything else already compiled. Go takes about 10 seconds cold and 0.1 seconds to rebuild, with or without Rastrillo, which makes its edit-and-rebuild loop its clearest advantage. Nim follows at 34 seconds and 0.9 seconds. |
| + |
| +OCaml, Haskell and Spinel were built under emulation, so their times don't compare with the others: 18 to 19 minutes cold for OCaml and Haskell, and 104 seconds for every Spinel rebuild, because Spinel compiles the whole program each time. |
| + |
| +## Choosing a language for a CARLOS app |
| + |
| +Start with Go. It builds a static ARM executable in about 10 seconds, rebuilds after a small edit in a tenth of a second, and the platform's own tools and Rastrillo, the CARLOS web framework, are written for it. Rastrillo matched plain Go on page loads and code size but managed 309 toggles a second against plain Go's 810. In exchange you get its routing, sessions and migrations, which save design work as an app grows. |
| + |
| +Use Nim or Zig when memory, executable size and request rate matter. Both made executables under 2 MB, used a few megabytes at rest, and served about four times as many page loads as Go. The catch is maturity: Nim's builder had to patch a copy of the standard library's basic HTTP server to listen on a Unix socket, and that server has no request timeouts. |
| + |
| +Use Rust when you want a mature server stack without giving much away. It runs axum on tokio, reached 1,398 page loads a second when the edge didn't intervene, used 14 MB at rest, and was the cheapest to build with AI. |
| + |
| +Whatever you pick, check the route to a static ARM executable first. A hard route costs more time and money even when the app's code is straightforward. |
| + |
| +## What it means for Carloku |
| + |
| +Mojo couldn't ship because its runtime comes only as shared libraries, which led to [Idea 360](https://amadan.net/carlos/platform/-/discussion/360): binary releases that carry their own shared libraries. Members can't empty or reset an instance's database, and deleting an instance parks its hostname until an operator purges it, which is [Feature Request 364](https://amadan.net/carlos/platform/-/discussion/364). Members also can't see an instance's memory use ([discussion 365](https://amadan.net/carlos/platform/-/discussion/365)). |
| + |
| +Three smaller things came up. `carlos vet` is listed as a member command but needs operator access. The health column in `carlos instances list` lags by minutes, while the platform's own log events are accurate. And the edge's 64-request limit shapes any load test run from one machine. None of this stopped the apps from running, but it limits what members can measure and diagnose. |
| + |
| +## Caveats |
| + |
| +Each language got one build, and another run could come out differently. One test machine measured the live apps over the internet, on the default instance size only. Three apps were built under emulation, and memory was measured on x86 rather than on Carloku. |
diff --git a/harness/cmd/bakeoff/main.go b/harness/cmd/bakeoff/main.go| index 3b510cf..30ea59f 100644 |
| --- a/harness/cmd/bakeoff/main.go |
| +++ b/harness/cmd/bakeoff/main.go |
| @@ -53,7 +53,7 @@ func main() { |
| preview := fs.Bool("preview", false, "report: tag strings for copy review") |
| replies := fs.String("replies", "", "readability: comma-separated files, one scorer reply each") |
| fs.Parse(os.Args[2:]) |
| - if *slug == "" && cmd != "report" { |
| + if *slug == "" && cmd != "report" && cmd != "writeup" { |
| usage() |
| } |
| b.ResultsDir = *results |
| @@ -85,6 +85,8 @@ func main() { |
| err = tokens(*slug, *logs, *phase, *since) |
| case "behaviour": |
| err = behaviour(*slug, *file) |
| + case "writeup": |
| + err = renderWriteup(*file) |
| case "report": |
| err = renderReport(*file, *preview) |
| case "findings": |
| @@ -467,3 +469,23 @@ func memory(slug, dir string, d time.Duration) error { |
| fmt.Printf("%s memory: idle %d KB, peak %d KB, after %d KB\n", slug, m.IdleKB, m.PeakKB, m.AfterKB) |
| return b.Update(slug, func(r *b.AppResult) { r.Memory = m }) |
| } |
| + |
| +// renderWriteup writes docs/writeup.md (or --file) as the write-up page. |
| +func renderWriteup(file string) error { |
| + if file == "" { |
| + file = "docs/writeup.md" |
| + } |
| + src, err := os.ReadFile(file) |
| + if err != nil { |
| + return err |
| + } |
| + text, err := os.ReadFile("copy-review/strings.json") |
| + if err != nil { |
| + return err |
| + } |
| + words, err := report.LoadStrings(text) |
| + if err != nil { |
| + return err |
| + } |
| + return report.RenderWriteup(string(src), words, os.Stdout) |
| +} |
diff --git a/harness/report/markdown.go b/harness/report/markdown.go| index 7da3087..b53bdf5 100644 |
| --- a/harness/report/markdown.go |
| +++ b/harness/report/markdown.go |
| @@ -2,6 +2,7 @@ package report |
| import ( |
| "encoding/json" |
| + "fmt" |
| "html" |
| "html/template" |
| "regexp" |
| @@ -73,12 +74,37 @@ func inline(s string) string { |
| return mdLink.ReplaceAllString(s, `<a href="$2">$1</a>`) |
| } |
| -// markdown renders the small subset the copy uses: paragraphs, "- " lists, |
| -// links, inline code and bold. |
| +// markdown renders the small subset the copy and the write-up use: |
| +// headings, paragraphs, "- " lists, pipe tables, links, inline code and |
| +// bold. |
| func markdown(src string) template.HTML { |
| var out strings.Builder |
| for _, block := range strings.Split(strings.TrimSpace(src), "\n\n") { |
| lines := strings.Split(strings.TrimSpace(block), "\n") |
| + if h := strings.IndexFunc(lines[0], func(r rune) bool { return r != '#' }); h > 0 && h <= 3 && len(lines) == 1 && lines[0][h] == ' ' { |
| + fmt.Fprintf(&out, "<h%d>%s</h%d>\n", h, inline(strings.TrimSpace(lines[0][h:])), h) |
| + continue |
| + } |
| + if strings.HasPrefix(lines[0], "|") { |
| + out.WriteString(`<div class="scroll"><table>`) |
| + for i, l := range lines { |
| + cells := strings.Split(strings.Trim(strings.TrimSpace(l), "|"), "|") |
| + if i == 1 && strings.Trim(l, "|-: ") == "" { |
| + continue |
| + } |
| + tag := "td" |
| + if i == 0 { |
| + tag = "th" |
| + } |
| + out.WriteString("<tr>") |
| + for _, c := range cells { |
| + fmt.Fprintf(&out, "<%s>%s</%s>", tag, inline(strings.TrimSpace(c)), tag) |
| + } |
| + out.WriteString("</tr>") |
| + } |
| + out.WriteString("</table></div>\n") |
| + continue |
| + } |
| if strings.HasPrefix(lines[0], "- ") { |
| out.WriteString("<ul>") |
| for _, l := range lines { |
diff --git a/harness/report/report.go b/harness/report/report.go| index 3e8bd9c..37ec6ed 100644 |
| --- a/harness/report/report.go |
| +++ b/harness/report/report.go |
| @@ -15,6 +15,12 @@ import ( |
| //go:embed report.html.tmpl |
| var pageSrc string |
| +//go:embed style.tmpl |
| +var styleSrc string |
| + |
| +//go:embed writeup.html.tmpl |
| +var writeupSrc string |
| + |
| var page = template.Must(template.New("page").Funcs(template.FuncMap{ |
| "md": func(c Copy, key string) template.HTML { return markdown(c.T(key)) }, |
| "t": func(c Copy, key string) string { return c.T(key) }, |
| @@ -41,7 +47,20 @@ var page = template.Must(template.New("page").Funcs(template.FuncMap{ |
| "ran": func(x b.Behaviour) int { return x.Passed + x.Failed }, |
| "float": func(n int64) float64 { return float64(n) }, |
| "inc": func(i int) int { return i + 1 }, |
| -}).Parse(pageSrc)) |
| +}).Parse(pageSrc + styleSrc)) |
| + |
| +var writeupPage = template.Must(template.Must(page.Clone()).New("writeup").Parse(writeupSrc)) |
| + |
| +// RenderWriteup writes the write-up page from its Markdown, with the |
| +// report's styles; the title is the Markdown's first heading. |
| +func RenderWriteup(src string, copy Copy, w io.Writer) error { |
| + title := strings.TrimSpace(strings.TrimLeft(strings.SplitN(strings.TrimSpace(src), "\n", 2)[0], "# ")) |
| + return writeupPage.ExecuteTemplate(w, "writeup", struct { |
| + Title string |
| + Body template.HTML |
| + Copy Copy |
| + }{title, markdown(src), copy}) |
| +} |
| func live(r b.AppResult) bool { return strings.HasPrefix(r.Status, "LIVE") } |
diff --git a/harness/report/report.html.tmpl b/harness/report/report.html.tmpl| index 1b54924..a0b27c1 100644 |
| --- a/harness/report/report.html.tmpl |
| +++ b/harness/report/report.html.tmpl |
| @@ -4,54 +4,13 @@ |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>{{t .Copy "page.title"}}</title> |
| -<style> |
| -:root { |
| - color-scheme: light; |
| - --plane: #f9f9f7; --surface: #fcfcfb; --ink: #0b0b0b; --ink-2: #52514e; --muted: #898781; |
| - --grid: #e1e0d9; --axis: #c3c2b7; --ring: rgba(11,11,11,0.10); |
| - --s1: #2a78d6; --s2: #eb6834; --s3: #1baf7a; |
| -} |
| -@media (prefers-color-scheme: dark) { |
| - :root { |
| - color-scheme: dark; |
| - --plane: #0d0d0d; --surface: #1a1a19; --ink: #ffffff; --ink-2: #c3c2b7; --muted: #898781; |
| - --grid: #2c2c2a; --axis: #383835; --ring: rgba(255,255,255,0.10); |
| - --s1: #3987e5; --s2: #d95926; --s3: #199e70; |
| - } |
| -} |
| -* { box-sizing: border-box; } |
| -body { margin: 0; background: var(--plane); color: var(--ink); font: 16px/1.55 system-ui, -apple-system, "Segoe UI", sans-serif; } |
| -main { max-width: 760px; margin: 0 auto; padding: 48px 20px 80px; } |
| -h1 { font-size: 2rem; line-height: 1.2; margin: 0 0 16px; } |
| -h2 { font-size: 1.35rem; margin: 56px 0 12px; } |
| -h3 { font-size: 1.05rem; margin: 0 0 4px; } |
| -p, li { color: var(--ink-2); } |
| -a { color: inherit; } |
| -code { font-size: .9em; } |
| -.card { background: var(--surface); border: 1px solid var(--ring); border-radius: 10px; padding: 18px 20px; margin: 16px 0; } |
| -.note { color: var(--muted); font-size: .9rem; margin: 2px 0 8px; } |
| -.chart { width: 100%; height: auto; display: block; } |
| -.chart .label { fill: var(--ink-2); font-size: 13px; text-anchor: end; } |
| -.chart .value { fill: var(--ink); font-size: 13px; font-variant-numeric: tabular-nums; } |
| -.chart .missing { fill: var(--muted); font-size: 13px; font-style: italic; } |
| -.chart .axis { stroke: var(--axis); stroke-width: 1; } |
| -.chart .s1 { fill: var(--s1); } .chart .s2 { fill: var(--s2); } .chart .s3 { fill: var(--s3); } |
| -.legend { display: flex; gap: 16px; font-size: .85rem; color: var(--ink-2); margin: 4px 0 0; padding: 0; list-style: none; } |
| -.legend li::before { content: ""; display: inline-block; width: 10px; height: 10px; border-radius: 2px; margin-right: 6px; background: var(--k); } |
| -.scroll { overflow-x: auto; } |
| -table { border-collapse: collapse; width: 100%; font-size: .85rem; font-variant-numeric: tabular-nums; } |
| -th, td { padding: 6px 8px; border-bottom: 1px solid var(--grid); text-align: right; white-space: nowrap; } |
| -th:first-child, td:first-child { text-align: left; } |
| -th { color: var(--ink-2); font-weight: 600; } |
| -.blocked td:not(:first-child) { color: var(--muted); text-align: left; white-space: normal; } |
| -dl { display: grid; grid-template-columns: max-content 1fr; gap: 4px 16px; margin: 8px 0 0; font-size: .92rem; } |
| -dt { color: var(--muted); } dd { margin: 0; color: var(--ink-2); } |
| -</style> |
| +{{template "css"}} |
| </head> |
| <body> |
| <main> |
| <h1>{{cp $.Copy "page.title" $.Preview}}</h1> |
| {{md .Copy "page.intro"}} |
| +<p><a href="writeup.html">{{cp $.Copy "page.writeup_link" $.Preview}}</a></p> |
| <h2>{{cp $.Copy "findings.heading" $.Preview}}</h2> |
| {{md .Copy "findings.body"}} |
diff --git a/harness/report/report_test.go b/harness/report/report_test.go| index 38a9f28..9c7fa3f 100644 |
| --- a/harness/report/report_test.go |
| +++ b/harness/report/report_test.go |
| @@ -93,3 +93,29 @@ func TestValueLabelsFitTheChart(t *testing.T) { |
| } |
| } |
| } |
| + |
| +func TestMarkdownHeadingsAndTables(t *testing.T) { |
| + got := string(markdown("# Title\n\nIntro with **bold**.\n\n## Section\n\n| A | B |\n|---|---|\n| 1 | x & y |\n| 2 | `z` |")) |
| + for _, want := range []string{"<h1>Title</h1>", "<p>Intro with <strong>bold</strong>.</p>", "<h2>Section</h2>", |
| + "<table>", "<th>A</th>", "<td>x & y</td>", "<td><code>z</code></td>", "</table>"} { |
| + if !strings.Contains(got, want) { |
| + t.Errorf("markdown lacks %q in:\n%s", want, got) |
| + } |
| + } |
| + if strings.Contains(got, "|---") { |
| + t.Error("the table's separator row leaked into the output") |
| + } |
| +} |
| + |
| +func TestRenderWriteup(t *testing.T) { |
| + var buf bytes.Buffer |
| + if err := RenderWriteup("# 🤖 What it tells us\n\nPara.\n\n| A |\n|---|\n| 1 |", Copy{"writeup.back": "Back to the report"}, &buf); err != nil { |
| + t.Fatal(err) |
| + } |
| + out := buf.String() |
| + for _, want := range []string{"<title>🤖 What it tells us</title>", "<h1>🤖 What it tells us</h1>", "<td>1</td>", `<a href="./">Back to the report</a>`, "--plane"} { |
| + if !strings.Contains(out, want) { |
| + t.Errorf("write-up page lacks %q", want) |
| + } |
| + } |
| +} |
diff --git a/harness/report/style.tmpl b/harness/report/style.tmpl| new file mode 100644 |
| index 0000000..f670bc6 |
| --- /dev/null |
| +++ b/harness/report/style.tmpl |
| @@ -0,0 +1,45 @@ |
| +{{define "css"}}<style> |
| +:root { |
| + color-scheme: light; |
| + --plane: #f9f9f7; --surface: #fcfcfb; --ink: #0b0b0b; --ink-2: #52514e; --muted: #898781; |
| + --grid: #e1e0d9; --axis: #c3c2b7; --ring: rgba(11,11,11,0.10); |
| + --s1: #2a78d6; --s2: #eb6834; --s3: #1baf7a; |
| +} |
| +@media (prefers-color-scheme: dark) { |
| + :root { |
| + color-scheme: dark; |
| + --plane: #0d0d0d; --surface: #1a1a19; --ink: #ffffff; --ink-2: #c3c2b7; --muted: #898781; |
| + --grid: #2c2c2a; --axis: #383835; --ring: rgba(255,255,255,0.10); |
| + --s1: #3987e5; --s2: #d95926; --s3: #199e70; |
| + } |
| +} |
| +* { box-sizing: border-box; } |
| +body { margin: 0; background: var(--plane); color: var(--ink); font: 16px/1.55 system-ui, -apple-system, "Segoe UI", sans-serif; } |
| +main { max-width: 760px; margin: 0 auto; padding: 48px 20px 80px; } |
| +h1 { font-size: 2rem; line-height: 1.2; margin: 0 0 16px; } |
| +h2 { font-size: 1.35rem; margin: 56px 0 12px; } |
| +h3 { font-size: 1.05rem; margin: 0 0 4px; } |
| +p, li { color: var(--ink-2); } |
| +a { color: inherit; } |
| +code { font-size: .9em; } |
| +.card { background: var(--surface); border: 1px solid var(--ring); border-radius: 10px; padding: 18px 20px; margin: 16px 0; } |
| +.note { color: var(--muted); font-size: .9rem; margin: 2px 0 8px; } |
| +.chart { width: 100%; height: auto; display: block; } |
| +.chart .label { fill: var(--ink-2); font-size: 13px; text-anchor: end; } |
| +.chart .value { fill: var(--ink); font-size: 13px; font-variant-numeric: tabular-nums; } |
| +.chart .missing { fill: var(--muted); font-size: 13px; font-style: italic; } |
| +.chart .axis { stroke: var(--axis); stroke-width: 1; } |
| +.chart .s1 { fill: var(--s1); } .chart .s2 { fill: var(--s2); } .chart .s3 { fill: var(--s3); } |
| +.legend { display: flex; gap: 16px; font-size: .85rem; color: var(--ink-2); margin: 4px 0 0; padding: 0; list-style: none; } |
| +.legend li::before { content: ""; display: inline-block; width: 10px; height: 10px; border-radius: 2px; margin-right: 6px; background: var(--k); } |
| +.scroll { overflow-x: auto; } |
| +table { border-collapse: collapse; width: 100%; font-size: .85rem; font-variant-numeric: tabular-nums; } |
| +th, td { padding: 6px 8px; border-bottom: 1px solid var(--grid); text-align: right; white-space: nowrap; } |
| +th:first-child, td:first-child { text-align: left; } |
| +th { color: var(--ink-2); font-weight: 600; } |
| +.blocked td:not(:first-child) { color: var(--muted); text-align: left; white-space: normal; } |
| +.prose table { margin: 16px 0; } |
| +.prose h2 { margin-top: 40px; } |
| +dl { display: grid; grid-template-columns: max-content 1fr; gap: 4px 16px; margin: 8px 0 0; font-size: .92rem; } |
| +dt { color: var(--muted); } dd { margin: 0; color: var(--ink-2); } |
| +</style>{{end}} |
diff --git a/harness/report/writeup.html.tmpl b/harness/report/writeup.html.tmpl| new file mode 100644 |
| index 0000000..5b79f55 |
| --- /dev/null |
| +++ b/harness/report/writeup.html.tmpl |
| @@ -0,0 +1,15 @@ |
| +<!doctype html> |
| +<html lang="en"> |
| +<head> |
| +<meta charset="utf-8"> |
| +<meta name="viewport" content="width=device-width, initial-scale=1"> |
| +<title>{{.Title}}</title> |
| +{{template "css"}} |
| +</head> |
| +<body> |
| +<main class="prose"> |
| +<p><a href="./">{{t .Copy "writeup.back"}}</a></p> |
| +{{.Body}} |
| +</main> |
| +</body> |
| +</html> |
diff --git a/report/site/index.html b/report/site/index.html| index 6454e83..c4d8fe4 100644 |
| --- a/report/site/index.html |
| +++ b/report/site/index.html |
| @@ -44,6 +44,8 @@ th, td { padding: 6px 8px; border-bottom: 1px solid var(--grid); text-align: rig |
| th:first-child, td:first-child { text-align: left; } |
| th { color: var(--ink-2); font-weight: 600; } |
| .blocked td:not(:first-child) { color: var(--muted); text-align: left; white-space: normal; } |
| +.prose table { margin: 16px 0; } |
| +.prose h2 { margin-top: 40px; } |
| dl { display: grid; grid-template-columns: max-content 1fr; gap: 4px 16px; margin: 8px 0 0; font-size: .92rem; } |
| dt { color: var(--muted); } dd { margin: 0; color: var(--ink-2); } |
| </style> |
| @@ -55,6 +57,7 @@ dt { color: var(--muted); } dd { margin: 0; color: var(--ink-2); } |
| <p>The point is to show that CARLOS apps, and the platform they run on, aren't tied to Go. So this page compares the nine; it doesn't rank them.</p> |
| <p>One AI model, Opus 5.5, built every app from the same brief. A second, Astra, reviewed each one. A third, Fable, scored how readable the code is without being told which app it was reading. All nine apps are live, and their code is public at <a href="https://amadan.net/carlos/todomvc-bakeoff">amadan.net/carlos/todomvc-bakeoff</a>.</p> |
| +<p><a href="writeup.html">Read the write-up: what the numbers mean and what we conclude</a></p> |
| <h2>What stood out</h2> |
| <ul><li><strong>All nine pass every shared test</strong> on Carloku: 67 of 67, with JavaScript on and with it off.<li><strong>Nim and Zig handled the most traffic</strong> on Carloku's default size: about 1,400 to 1,500 page loads a second, and 3,100 to 3,600 toggles. They also made the smallest programs (under 2 MB) and used the least memory at rest (2.5 to 3.5 MB).<li><strong>Go builds fastest.</strong> About 10 seconds from scratch and under a second to rebuild, with or without Rastrillo.<li><strong>Waking up costs about the same everywhere.</strong> Every app answered its first request after sleeping in 1.2 to 1.5 seconds; about one second of that is Carloku starting the program.<li><strong>Build costs ranged from $2.92 to $10.70.</strong> Rust was the cheapest. Spinel was the most expensive, mostly because getting it to compile for ARM took the builder 77 minutes.<li><strong>Spinel restarted under heavy load.</strong> Its server process exited 16 times during the load test, and Carloku started it again each time. In one 10-second step, no request completed at all.</ul> |
| @@ -293,3 +296,4 @@ dt { color: var(--muted); } dd { margin: 0; color: var(--ink-2); } |
| </main> |
| </body> |
| </html> |
| + |
diff --git a/report/site/writeup.html b/report/site/writeup.html| new file mode 100644 |
| index 0000000..665cd41 |
| --- /dev/null |
| +++ b/report/site/writeup.html |
| @@ -0,0 +1,96 @@ |
| +<!doctype html> |
| +<html lang="en"> |
| +<head> |
| +<meta charset="utf-8"> |
| +<meta name="viewport" content="width=device-width, initial-scale=1"> |
| +<title>🤖 What the TodoMVC bakeoff tells us</title> |
| +<style> |
| +:root { |
| + color-scheme: light; |
| + --plane: #f9f9f7; --surface: #fcfcfb; --ink: #0b0b0b; --ink-2: #52514e; --muted: #898781; |
| + --grid: #e1e0d9; --axis: #c3c2b7; --ring: rgba(11,11,11,0.10); |
| + --s1: #2a78d6; --s2: #eb6834; --s3: #1baf7a; |
| +} |
| +@media (prefers-color-scheme: dark) { |
| + :root { |
| + color-scheme: dark; |
| + --plane: #0d0d0d; --surface: #1a1a19; --ink: #ffffff; --ink-2: #c3c2b7; --muted: #898781; |
| + --grid: #2c2c2a; --axis: #383835; --ring: rgba(255,255,255,0.10); |
| + --s1: #3987e5; --s2: #d95926; --s3: #199e70; |
| + } |
| +} |
| +* { box-sizing: border-box; } |
| +body { margin: 0; background: var(--plane); color: var(--ink); font: 16px/1.55 system-ui, -apple-system, "Segoe UI", sans-serif; } |
| +main { max-width: 760px; margin: 0 auto; padding: 48px 20px 80px; } |
| +h1 { font-size: 2rem; line-height: 1.2; margin: 0 0 16px; } |
| +h2 { font-size: 1.35rem; margin: 56px 0 12px; } |
| +h3 { font-size: 1.05rem; margin: 0 0 4px; } |
| +p, li { color: var(--ink-2); } |
| +a { color: inherit; } |
| +code { font-size: .9em; } |
| +.card { background: var(--surface); border: 1px solid var(--ring); border-radius: 10px; padding: 18px 20px; margin: 16px 0; } |
| +.note { color: var(--muted); font-size: .9rem; margin: 2px 0 8px; } |
| +.chart { width: 100%; height: auto; display: block; } |
| +.chart .label { fill: var(--ink-2); font-size: 13px; text-anchor: end; } |
| +.chart .value { fill: var(--ink); font-size: 13px; font-variant-numeric: tabular-nums; } |
| +.chart .missing { fill: var(--muted); font-size: 13px; font-style: italic; } |
| +.chart .axis { stroke: var(--axis); stroke-width: 1; } |
| +.chart .s1 { fill: var(--s1); } .chart .s2 { fill: var(--s2); } .chart .s3 { fill: var(--s3); } |
| +.legend { display: flex; gap: 16px; font-size: .85rem; color: var(--ink-2); margin: 4px 0 0; padding: 0; list-style: none; } |
| +.legend li::before { content: ""; display: inline-block; width: 10px; height: 10px; border-radius: 2px; margin-right: 6px; background: var(--k); } |
| +.scroll { overflow-x: auto; } |
| +table { border-collapse: collapse; width: 100%; font-size: .85rem; font-variant-numeric: tabular-nums; } |
| +th, td { padding: 6px 8px; border-bottom: 1px solid var(--grid); text-align: right; white-space: nowrap; } |
| +th:first-child, td:first-child { text-align: left; } |
| +th { color: var(--ink-2); font-weight: 600; } |
| +.blocked td:not(:first-child) { color: var(--muted); text-align: left; white-space: normal; } |
| +.prose table { margin: 16px 0; } |
| +.prose h2 { margin-top: 40px; } |
| +dl { display: grid; grid-template-columns: max-content 1fr; gap: 4px 16px; margin: 8px 0 0; font-size: .92rem; } |
| +dt { color: var(--muted); } dd { margin: 0; color: var(--ink-2); } |
| +</style> |
| +</head> |
| +<body> |
| +<main class="prose"> |
| +<p><a href="./">Back to the report</a></p> |
| +<h1>🤖 What the TodoMVC bakeoff tells us</h1> |
| +<p>CARLOS and Carloku aren't tied to Go. We built the same small app in nine language setups, and eight of them shipped as a single static ARM executable that runs on Carloku with no changes to the platform. The ninth, Mojo, couldn't make a self-contained executable, so Nim took its place.</p> |
| +<p>The app is TodoMVC, a small to-do list. The server builds every page, every action works with JavaScript turned off, and a little JavaScript on top makes it feel like the familiar TodoMVC. One AI model, Opus 5.5, built each version from the same brief. A second, Astra, reviewed each one, and the builder got one round to fix what it found. A third, Fable, scored how readable the code is. It was told the language but not which app it was reading. Every app ran on Carloku's default instance size. The charts are on the <a href="https://bakeoff.bfj.oncarlos.com">report page</a>, and the apps, tests and measurement tools are in the <a href="https://amadan.net/carlos/todomvc-bakeoff">code repository</a>.</p> |
| +<h2>The numbers that matter</h2> |
| +<div class="scroll"><table><tr><th>Setup</th><th>Page loads a second</th><th>Toggles a second</th><th>Memory at rest / peak</th><th>Executable</th><th>Cold build / one-line rebuild</th><th>Cost to build</th></tr><tr><td>Go</td><td>349</td><td>810</td><td>17 / 71 MB</td><td>12.3 MB</td><td>10 s / 0.1 s</td><td>$3.69</td></tr><tr><td>Go + Rastrillo</td><td>336</td><td>309</td><td>19 / 63 MB</td><td>14.1 MB</td><td>10 s / 0.1 s</td><td>$4.67</td></tr><tr><td>Rust</td><td>904 (1,398 uncapped)</td><td>1,022</td><td>14 / 53 MB</td><td>3.2 MB</td><td>50 s / 5.8 s</td><td>$2.92</td></tr><tr><td>Zig</td><td>1,382</td><td>3,113</td><td>2.5 / 25 MB</td><td>1.9 MB</td><td>152 s / 5.4 s</td><td>$5.71</td></tr><tr><td>OCaml</td><td>992</td><td>1,058</td><td>11 / 17 MB</td><td>8.2 MB</td><td>emulated</td><td>$7.65</td></tr><tr><td>Haskell</td><td>400</td><td>862</td><td>11 / 77 MB</td><td>28.7 MB</td><td>emulated</td><td>$5.06</td></tr><tr><td>Swift</td><td>537</td><td>1,453</td><td>24 / 33 MB</td><td>22.0 MB</td><td>81 s / 3.9 s</td><td>$4.60</td></tr><tr><td>Nim</td><td>1,491</td><td>3,561</td><td>3.5 / 11.5 MB</td><td>1.8 MB</td><td>34 s / 0.9 s</td><td>$4.84</td></tr><tr><td>Spinel</td><td>722</td><td>2,451</td><td>8 / 13 MB</td><td>8.7 MB</td><td>emulated</td><td>$10.70</td></tr></table></div> |
| +<p>No single language came out best. Go is the practical default. Nim and Zig used the least memory and served the most requests. Rust kept pace on speed where the test let it, has the most mature server stack of the three, and was the cheapest to build with AI. How hard it was to get a static ARM executable mattered more to cost and effort than the language did.</p> |
| +<h2>Correctness</h2> |
| +<p>Every app passed all 67 shared tests: TodoMVC's own 29 tests run with JavaScript on and again with it off, plus checks for unsafe input, very long titles, one visitor reaching another's list, and a screenshot comparison with TodoMVC's page. After the fix round, the review left between zero and three minor findings per app.</p> |
| +<p>The two security problems were the same bug. A title containing an invalid UTF-8 byte crashed the whole server, in Zig and in Nim, and both were fixed in a second round. Our shared tests only send valid UTF-8, so they missed it and the reviewer caught it. The next bakeoff's tests should send broken input too.</p> |
| +<h2>Building with AI</h2> |
| +<p>Building cost between $2.92 (Rust) and $10.70 (Spinel), and took the builder between 15 minutes (Go) and 87 minutes (Spinel). The expensive builds were the ones where a static ARM executable was hard to get. Spinel's builder spent most of its time on cross-compiling, and OCaml and Haskell had to be built under emulation, with software on our x86 machine pretending to be an ARM machine. None of this says a language is harder for an AI to write.</p> |
| +<p>The code itself came out similar everywhere. The apps run from 566 lines (Spinel) to 1,058 (Zig), and readability scores ranged from 7 to 9 out of 10 with most at 8, so they barely separate the languages.</p> |
| +<h2>Speed under load</h2> |
| +<p>The load test loaded a 50-item page while doubling the number of simultaneous connections every 10 seconds, then did the same for writes, ticking a to-do on and off. Reads fell into three bands: Nim and Zig at about 1,400 to 1,500 page loads a second, Rust, OCaml and Spinel at about 700 to 1,000, and Go, Go with Rastrillo, Haskell and Swift at about 340 to 540. Writes spread wider, from Nim's 3,561 toggles a second down to Go with Rastrillo's 309.</p> |
| +<p>Put that in proportion before reading much into it. The slowest, Go at 349 page loads a second, is still about 29 million a day. Go's read rate is probably down to its SQLite library, a pure-Go translation chosen so the build needs no C compiler, though we didn't measure that directly.</p> |
| +<p>Rust's numbers need a footnote. In its first read ramp and its write ramp, Rust was still speeding up at 64 connections when Carloku's edge started turning requests away, so the test never found its limit. In the one ramp where the edge didn't step in, the reads after the writes, Rust reached 1,398 page loads a second, close to Nim's 1,491.</p> |
| +<p>That edge limit shapes everything above 64 connections. The edge is Carloku's set of front servers that receive traffic from the internet. When a request arrives at an edge server that isn't the one running the app, the edge allows at most 64 such requests in flight at once and answers the rest with <code>503</code>, "try again shortly". Whether it applied depended on how our traffic was routed, so results above 64 measure the platform as much as the app.</p> |
| +<p>Some apps failed outright under very heavy load, which the edge reports as <code>502</code>: Swift at 512 connections and Nim at 256. Spinel's server process exited 16 times during its test and Carloku restarted it each time. In one 10-second Spinel step, no request completed at all.</p> |
| +<h2>Waking up and the first byte</h2> |
| +<p>Carloku puts an app to sleep after 15 quiet minutes. Every app answered its first request after sleeping in 1.2 to 1.45 seconds, and Carloku's own logs show about one second of that is the platform starting the process. The language barely matters.</p> |
| +<p>The first byte of a page arrived after about 50 to 100 milliseconds. Our test machine reached Carloku over the internet, and the network's base delay drifted between about 17 and 37 milliseconds during the runs, so the two clusters, near 50 and near 95 milliseconds, say more about when each app was measured than about the apps.</p> |
| +<h2>Memory and size</h2> |
| +<p>Carloku doesn't show members how much memory an instance uses, so we ran each app on our own x86 machine and read it from the operating system. We've asked for the figures in <a href="https://amadan.net/carlos/platform/-/discussion/365">carlos/platform discussion 365</a>. At rest, the apps used 2.5 MB (Zig) to 24 MB (Swift). Under 64 simultaneous connections, Go and Haskell, both garbage-collected, peaked highest at 71 and 77 MB, with Rust next at 53 MB, while Nim, Spinel and OCaml stayed under 17 MB. Every app used a small fraction of an instance's memory limit.</p> |
| +<p>Executables ranged from 1.8 MB (Nim) to 28.7 MB (Haskell).</p> |
| +<h2>Builds</h2> |
| +<p>These times are for the ARM executable that ships. A cold build compiles every library from already-downloaded source. A one-line rebuild adds one line to the code that serves the page, with everything else already compiled. Go takes about 10 seconds cold and 0.1 seconds to rebuild, with or without Rastrillo, which makes its edit-and-rebuild loop its clearest advantage. Nim follows at 34 seconds and 0.9 seconds.</p> |
| +<p>OCaml, Haskell and Spinel were built under emulation, so their times don't compare with the others: 18 to 19 minutes cold for OCaml and Haskell, and 104 seconds for every Spinel rebuild, because Spinel compiles the whole program each time.</p> |
| +<h2>Choosing a language for a CARLOS app</h2> |
| +<p>Start with Go. It builds a static ARM executable in about 10 seconds, rebuilds after a small edit in a tenth of a second, and the platform's own tools and Rastrillo, the CARLOS web framework, are written for it. Rastrillo matched plain Go on page loads and code size but managed 309 toggles a second against plain Go's 810. In exchange you get its routing, sessions and migrations, which save design work as an app grows.</p> |
| +<p>Use Nim or Zig when memory, executable size and request rate matter. Both made executables under 2 MB, used a few megabytes at rest, and served about four times as many page loads as Go. The catch is maturity: Nim's builder had to patch a copy of the standard library's basic HTTP server to listen on a Unix socket, and that server has no request timeouts.</p> |
| +<p>Use Rust when you want a mature server stack without giving much away. It runs axum on tokio, reached 1,398 page loads a second when the edge didn't intervene, used 14 MB at rest, and was the cheapest to build with AI.</p> |
| +<p>Whatever you pick, check the route to a static ARM executable first. A hard route costs more time and money even when the app's code is straightforward.</p> |
| +<h2>What it means for Carloku</h2> |
| +<p>Mojo couldn't ship because its runtime comes only as shared libraries, which led to <a href="https://amadan.net/carlos/platform/-/discussion/360">Idea 360</a>: binary releases that carry their own shared libraries. Members can't empty or reset an instance's database, and deleting an instance parks its hostname until an operator purges it, which is <a href="https://amadan.net/carlos/platform/-/discussion/364">Feature Request 364</a>. Members also can't see an instance's memory use (<a href="https://amadan.net/carlos/platform/-/discussion/365">discussion 365</a>).</p> |
| +<p>Three smaller things came up. <code>carlos vet</code> is listed as a member command but needs operator access. The health column in <code>carlos instances list</code> lags by minutes, while the platform's own log events are accurate. And the edge's 64-request limit shapes any load test run from one machine. None of this stopped the apps from running, but it limits what members can measure and diagnose.</p> |
| +<h2>Caveats</h2> |
| +<p>Each language got one build, and another run could come out differently. One test machine measured the live apps over the internet, on the default instance size only. Three apps were built under emulation, and memory was measured on x86 rather than on Carloku.</p> |
| + |
| +</main> |
| +</body> |
| +</html> |