| 1 | [ |
| 2 | { |
| 3 | "id": "page.title", |
| 4 | "section": "Top", |
| 5 | "label": "Page title", |
| 6 | "text": "One TodoMVC, nine languages, one platform", |
| 7 | "context": "The page heading and the browser tab title. It names what the page compares.", |
| 8 | "notes": "Keep short." |
| 9 | }, |
| 10 | { |
| 11 | "id": "page.intro", |
| 12 | "section": "Top", |
| 13 | "label": "Introduction", |
| 14 | "text": "We built the same small app nine times, once in each of nine language setups, and put every one on Carloku, the hosted CARLOS Platform. The app is [TodoMVC](https://todomvc.com): a to-do list. Here it's a page the server builds, which works with JavaScript switched off, plus a little JavaScript on top so it feels like the familiar TodoMVC.\n\nThe point is to show that CARLOS apps, and the platform they run on, aren't tied to Go. So this page compares the nine; it doesn't rank them.\n\nOne AI model, Opus 5.5, built every app from the same brief. A second, Astra, reviewed each one. A third, Fable, scored how readable the code is without being told which app it was reading. All nine apps are live, and their code is public at [amadan.net/carlos/todomvc-bakeoff](https://amadan.net/carlos/todomvc-bakeoff).", |
| 15 | "context": "The opening paragraphs under the heading. They say what was built, by whom, where it runs, and that the page compares rather than ranks.", |
| 16 | "notes": "Markdown: paragraphs, links, **bold**." |
| 17 | }, |
| 18 | { |
| 19 | "id": "page.writeup_link", |
| 20 | "section": "Top", |
| 21 | "label": "Link to the write-up", |
| 22 | "text": "Read the write-up: what the numbers mean and what we conclude", |
| 23 | "context": "A link under the introduction to the longer write-up page." |
| 24 | }, |
| 25 | { |
| 26 | "id": "writeup.back", |
| 27 | "section": "Write-up", |
| 28 | "label": "Back link", |
| 29 | "text": "Back to the report", |
| 30 | "context": "The link at the top of the write-up page, back to the report with the charts." |
| 31 | }, |
| 32 | { |
| 33 | "id": "findings.heading", |
| 34 | "section": "Findings", |
| 35 | "label": "Heading", |
| 36 | "text": "What stood out", |
| 37 | "context": "Heading over a short list of the results worth knowing before reading the charts." |
| 38 | }, |
| 39 | { |
| 40 | "id": "findings.body", |
| 41 | "section": "Findings", |
| 42 | "label": "Body", |
| 43 | "text": "- **All nine pass every shared test** on Carloku: 67 of 67, with JavaScript on and with it off.\n- **Nim and Zig handled the most traffic** on Carloku's default size: about 1,400 to 1,500 page loads a second, and 3,100 to 3,600 toggles. They also made the smallest programs (under 2 MB) and used the least memory at rest (2.5 to 3.5 MB).\n- **Go builds fastest.** About 10 seconds from scratch and under a second to rebuild, with or without Rastrillo.\n- **Waking up costs about the same everywhere.** Every app answered its first request after sleeping in 1.2 to 1.5 seconds; about one second of that is Carloku starting the program.\n- **Build costs ranged from $2.92 to $10.70.** Rust was the cheapest. Spinel was the most expensive, mostly because getting it to compile for ARM took the builder 77 minutes.\n- **Spinel restarted under heavy load.** Its server process exited 16 times during the load test, and Carloku started it again each time. In one 10-second step, no request completed at all.", |
| 44 | "context": "A handful of plain bullet points: the notable results, each with its number.", |
| 45 | "notes": "Markdown list." |
| 46 | }, |
| 47 | { |
| 48 | "id": "glance.heading", |
| 49 | "section": "At a glance", |
| 50 | "label": "Heading", |
| 51 | "text": "At a glance", |
| 52 | "context": "Heading over the summary table, one row per language." |
| 53 | }, |
| 54 | { |
| 55 | "id": "glance.col.setup", |
| 56 | "section": "At a glance", |
| 57 | "label": "Column: Setup", |
| 58 | "text": "Setup", |
| 59 | "context": "Column heading in the summary table. The language (and framework, for Go with Rastrillo) each row is about.", |
| 60 | "notes": "Short: a table header." |
| 61 | }, |
| 62 | { |
| 63 | "id": "glance.col.tests", |
| 64 | "section": "At a glance", |
| 65 | "label": "Column: Tests (JS on · off)", |
| 66 | "text": "Tests (JS on · off)", |
| 67 | "context": "Column heading in the summary table. How many of the shared behaviour tests passed, with JavaScript on and with it off.", |
| 68 | "notes": "Short: a table header." |
| 69 | }, |
| 70 | { |
| 71 | "id": "glance.col.first_byte", |
| 72 | "section": "At a glance", |
| 73 | "label": "Column: First byte", |
| 74 | "text": "First byte", |
| 75 | "context": "Column heading in the summary table. Median time to the first byte of the page, with an empty list.", |
| 76 | "notes": "Short: a table header." |
| 77 | }, |
| 78 | { |
| 79 | "id": "glance.col.wake", |
| 80 | "section": "At a glance", |
| 81 | "label": "Column: Wake", |
| 82 | "text": "Wake", |
| 83 | "context": "Column heading in the summary table. Median time to the first byte when the app had gone to sleep.", |
| 84 | "notes": "Short: a table header." |
| 85 | }, |
| 86 | { |
| 87 | "id": "glance.col.reads", |
| 88 | "section": "At a glance", |
| 89 | "label": "Column: Peak reads/s", |
| 90 | "text": "Peak reads/s", |
| 91 | "context": "Column heading in the summary table. The most page loads per second the app served under the load ramp.", |
| 92 | "notes": "Short: a table header." |
| 93 | }, |
| 94 | { |
| 95 | "id": "glance.col.binary", |
| 96 | "section": "At a glance", |
| 97 | "label": "Column: Binary", |
| 98 | "text": "Binary", |
| 99 | "context": "Column heading in the summary table. Size of the arm64 file that ships.", |
| 100 | "notes": "Short: a table header." |
| 101 | }, |
| 102 | { |
| 103 | "id": "glance.col.build", |
| 104 | "section": "At a glance", |
| 105 | "label": "Column: Cold / warm build", |
| 106 | "text": "Cold build / one-line change", |
| 107 | "context": "Column heading in the summary table: build time with every library compiled from scratch, and rebuild time after a one-line change with everything else already compiled.", |
| 108 | "notes": "Short: a table header." |
| 109 | }, |
| 110 | { |
| 111 | "id": "glance.col.lines", |
| 112 | "section": "At a glance", |
| 113 | "label": "Column: Lines", |
| 114 | "text": "Lines", |
| 115 | "context": "Column heading in the summary table. Lines of the app's own code, tests excluded.", |
| 116 | "notes": "Short: a table header." |
| 117 | }, |
| 118 | { |
| 119 | "id": "glance.col.deps", |
| 120 | "section": "At a glance", |
| 121 | "label": "Column: Deps", |
| 122 | "text": "Deps", |
| 123 | "context": "Column heading in the summary table. Number of third-party libraries the app uses directly.", |
| 124 | "notes": "Short: a table header." |
| 125 | }, |
| 126 | { |
| 127 | "id": "glance.col.cost", |
| 128 | "section": "At a glance", |
| 129 | "label": "Column: Cost", |
| 130 | "text": "Cost", |
| 131 | "context": "Column heading in the summary table. What the AI build and fix rounds cost at list price.", |
| 132 | "notes": "Short: a table header." |
| 133 | }, |
| 134 | { |
| 135 | "id": "glance.col.readability", |
| 136 | "section": "At a glance", |
| 137 | "label": "Column: Readability", |
| 138 | "text": "Readability", |
| 139 | "context": "Column heading in the summary table. The blind readability score out of 10.", |
| 140 | "notes": "Short: a table header." |
| 141 | }, |
| 142 | { |
| 143 | "id": "chart.ttfb_empty.title", |
| 144 | "section": "Charts", |
| 145 | "label": "Time to first byte, empty list: title", |
| 146 | "text": "Time to first byte, empty list", |
| 147 | "context": "Title of one chart. How quickly a first page arrives when a visitor has no todos.", |
| 148 | "notes": "Short." |
| 149 | }, |
| 150 | { |
| 151 | "id": "chart.ttfb_empty.note", |
| 152 | "section": "Charts", |
| 153 | "label": "Time to first byte, empty list: note", |
| 154 | "text": "Median of 30 page loads from our test machine, each on a new connection, so it includes the network.", |
| 155 | "context": "The line under the chart title that says exactly what was measured.", |
| 156 | "notes": "One sentence or two." |
| 157 | }, |
| 158 | { |
| 159 | "id": "chart.ttfb_50.title", |
| 160 | "section": "Charts", |
| 161 | "label": "Time to first byte, 50 items: title", |
| 162 | "text": "Time to first byte, 50 items", |
| 163 | "context": "Title of one chart. The same measure with a realistic list.", |
| 164 | "notes": "Short." |
| 165 | }, |
| 166 | { |
| 167 | "id": "chart.ttfb_50.note", |
| 168 | "section": "Charts", |
| 169 | "label": "Time to first byte, 50 items: note", |
| 170 | "text": "The same, with 50 todos on the list.", |
| 171 | "context": "The line under the chart title that says exactly what was measured.", |
| 172 | "notes": "One sentence or two." |
| 173 | }, |
| 174 | { |
| 175 | "id": "chart.wake.title", |
| 176 | "section": "Charts", |
| 177 | "label": "Wake from sleep: title", |
| 178 | "text": "Wake from sleep", |
| 179 | "context": "Title of one chart. Carloku puts idle apps to sleep; this is what the first visitor after a quiet spell waits.", |
| 180 | "notes": "Short." |
| 181 | }, |
| 182 | { |
| 183 | "id": "chart.wake.note", |
| 184 | "section": "Charts", |
| 185 | "label": "Wake from sleep: note", |
| 186 | "text": "Time to the first byte when the app had gone to sleep after 15 idle minutes. Median of three.", |
| 187 | "context": "The line under the chart title that says exactly what was measured.", |
| 188 | "notes": "One sentence or two." |
| 189 | }, |
| 190 | { |
| 191 | "id": "chart.reads_peak.title", |
| 192 | "section": "Charts", |
| 193 | "label": "Peak page loads per second: title", |
| 194 | "text": "Peak page loads per second", |
| 195 | "context": "Title of one chart. Throughput for reads.", |
| 196 | "notes": "Short." |
| 197 | }, |
| 198 | { |
| 199 | "id": "chart.reads_peak.note", |
| 200 | "section": "Charts", |
| 201 | "label": "Peak page loads per second: note", |
| 202 | "text": "Loading a 50-item list, starting with one connection and doubling the number of simultaneous connections every 10 seconds.", |
| 203 | "context": "The line under the chart title that says exactly what was measured.", |
| 204 | "notes": "One sentence or two." |
| 205 | }, |
| 206 | { |
| 207 | "id": "chart.reads_knee.title", |
| 208 | "section": "Charts", |
| 209 | "label": "Page loads: where it slows down: title", |
| 210 | "text": "Page loads: where it slows down", |
| 211 | "context": "Title of one chart. Where extra load starts adding waiting rather than throughput.", |
| 212 | "notes": "Short." |
| 213 | }, |
| 214 | { |
| 215 | "id": "chart.reads_knee.note", |
| 216 | "section": "Charts", |
| 217 | "label": "Page loads: where it slows down: note", |
| 218 | "text": "The fewest simultaneous connections from which adding more never raised throughput by 10% or more.", |
| 219 | "context": "The line under the chart title that says exactly what was measured.", |
| 220 | "notes": "One sentence or two." |
| 221 | }, |
| 222 | { |
| 223 | "id": "chart.toggles_peak.title", |
| 224 | "section": "Charts", |
| 225 | "label": "Peak toggles per second: title", |
| 226 | "text": "Peak toggles per second", |
| 227 | "context": "Title of one chart. Throughput for writes.", |
| 228 | "notes": "Short." |
| 229 | }, |
| 230 | { |
| 231 | "id": "chart.toggles_peak.note", |
| 232 | "section": "Charts", |
| 233 | "label": "Peak toggles per second: note", |
| 234 | "text": "Ticking one todo on and off, ramping the same way.", |
| 235 | "context": "The line under the chart title that says exactly what was measured.", |
| 236 | "notes": "One sentence or two." |
| 237 | }, |
| 238 | { |
| 239 | "id": "chart.toggles_knee.title", |
| 240 | "section": "Charts", |
| 241 | "label": "Toggles: where it slows down: title", |
| 242 | "text": "Toggles: where it slows down", |
| 243 | "context": "Title of one chart. Where writes stop scaling.", |
| 244 | "notes": "Short." |
| 245 | }, |
| 246 | { |
| 247 | "id": "chart.toggles_knee.note", |
| 248 | "section": "Charts", |
| 249 | "label": "Toggles: where it slows down: note", |
| 250 | "text": "The same rule as for page loads.", |
| 251 | "context": "The line under the chart title that says exactly what was measured.", |
| 252 | "notes": "One sentence or two." |
| 253 | }, |
| 254 | { |
| 255 | "id": "chart.reads_after.title", |
| 256 | "section": "Charts", |
| 257 | "label": "Peak page loads per second, after the writes: title", |
| 258 | "text": "Peak page loads per second, after the writes", |
| 259 | "context": "Title of one chart. Whether heavy writing leaves the app slower to read.", |
| 260 | "notes": "Short." |
| 261 | }, |
| 262 | { |
| 263 | "id": "chart.reads_after.note", |
| 264 | "section": "Charts", |
| 265 | "label": "Peak page loads per second, after the writes: note", |
| 266 | "text": "The read ramp again, straight after the toggle ramp, on the same database.", |
| 267 | "context": "The line under the chart title that says exactly what was measured.", |
| 268 | "notes": "One sentence or two." |
| 269 | }, |
| 270 | { |
| 271 | "id": "chart.memory_idle.title", |
| 272 | "section": "Charts", |
| 273 | "label": "Memory at rest: title", |
| 274 | "text": "Memory at rest", |
| 275 | "context": "Title of the chart showing how much memory each app uses when it has just started and nobody is using it.", |
| 276 | "notes": "Short." |
| 277 | }, |
| 278 | { |
| 279 | "id": "chart.memory_idle.note", |
| 280 | "section": "Charts", |
| 281 | "label": "Memory at rest: note", |
| 282 | "text": "Resident memory just after start, measured on our own machine because Carloku doesn't show members memory figures.", |
| 283 | "context": "The line under that chart saying exactly what was measured, and why it was measured locally.", |
| 284 | "notes": "One sentence or two." |
| 285 | }, |
| 286 | { |
| 287 | "id": "chart.memory_peak.title", |
| 288 | "section": "Charts", |
| 289 | "label": "Memory under load: title", |
| 290 | "text": "Memory under load", |
| 291 | "context": "Title of the chart showing the most memory each app used while being load-tested.", |
| 292 | "notes": "Short." |
| 293 | }, |
| 294 | { |
| 295 | "id": "chart.memory_peak.note", |
| 296 | "section": "Charts", |
| 297 | "label": "Memory under load: note", |
| 298 | "text": "The most resident memory each app used during 10 seconds of 64 simultaneous page loads and then 10 seconds of 64 simultaneous toggles, on our own machine.", |
| 299 | "context": "The line under that chart saying exactly what was measured.", |
| 300 | "notes": "One sentence or two." |
| 301 | }, |
| 302 | { |
| 303 | "id": "chart.reads_after_knee.title", |
| 304 | "section": "Charts", |
| 305 | "label": "Page loads after writes: slow-down title", |
| 306 | "text": "Page loads after the writes: where it slows down", |
| 307 | "context": "Title of the slow-down chart for the second read ramp.", |
| 308 | "notes": "Short." |
| 309 | }, |
| 310 | { |
| 311 | "id": "chart.reads_after_knee.note", |
| 312 | "section": "Charts", |
| 313 | "label": "Page loads after writes: slow-down note", |
| 314 | "text": "The same rule, for the read ramp run straight after the writes.", |
| 315 | "context": "Note under that chart.", |
| 316 | "notes": "" |
| 317 | }, |
| 318 | { |
| 319 | "id": "chart.binary.title", |
| 320 | "section": "Charts", |
| 321 | "label": "Binary size: title", |
| 322 | "text": "Binary size", |
| 323 | "context": "Title of one chart. How big the deployable is.", |
| 324 | "notes": "Short." |
| 325 | }, |
| 326 | { |
| 327 | "id": "chart.binary.note", |
| 328 | "section": "Charts", |
| 329 | "label": "Binary size: note", |
| 330 | "text": "The linux-arm64 file that ships.", |
| 331 | "context": "The line under the chart title that says exactly what was measured.", |
| 332 | "notes": "One sentence or two." |
| 333 | }, |
| 334 | { |
| 335 | "id": "chart.build_cold.title", |
| 336 | "section": "Charts", |
| 337 | "label": "Cold build: title", |
| 338 | "text": "Cold build", |
| 339 | "context": "Title of one chart. Build time from scratch.", |
| 340 | "notes": "Short." |
| 341 | }, |
| 342 | { |
| 343 | "id": "chart.build_cold.note", |
| 344 | "section": "Charts", |
| 345 | "label": "Cold build: note", |
| 346 | "text": "Every library compiled from already-downloaded sources. One run each, because some take close to an hour under emulation.", |
| 347 | "context": "The line under the chart title that says exactly what was measured.", |
| 348 | "notes": "One sentence or two." |
| 349 | }, |
| 350 | { |
| 351 | "id": "chart.build_edit.title", |
| 352 | "section": "Charts", |
| 353 | "label": "One-line change: title", |
| 354 | "text": "Rebuild after a one-line change", |
| 355 | "context": "Title of one chart. The edit-and-rebuild wait a developer feels.", |
| 356 | "notes": "Short." |
| 357 | }, |
| 358 | { |
| 359 | "id": "chart.build_edit.note", |
| 360 | "section": "Charts", |
| 361 | "label": "One-line change: note", |
| 362 | "text": "Everything already compiled, then one line added to the code that serves the page (a unique comment, so no cache can skip the work). Median of three.", |
| 363 | "context": "The line under the chart title that says exactly what was measured.", |
| 364 | "notes": "One sentence or two." |
| 365 | }, |
| 366 | { |
| 367 | "id": "chart.loc.title", |
| 368 | "section": "Charts", |
| 369 | "label": "Lines of code: title", |
| 370 | "text": "Lines of code", |
| 371 | "context": "Title of one chart. How much code each app needed.", |
| 372 | "notes": "Short." |
| 373 | }, |
| 374 | { |
| 375 | "id": "chart.loc.note", |
| 376 | "section": "Charts", |
| 377 | "label": "Lines of code: note", |
| 378 | "text": "Non-blank lines of each app's own code. Tests are not counted.", |
| 379 | "context": "The line under the chart title that says exactly what was measured.", |
| 380 | "notes": "One sentence or two." |
| 381 | }, |
| 382 | { |
| 383 | "id": "chart.cost.title", |
| 384 | "section": "Charts", |
| 385 | "label": "Cost to build: title", |
| 386 | "text": "Cost to build", |
| 387 | "context": "Title of one chart. AI spend per app.", |
| 388 | "notes": "Short." |
| 389 | }, |
| 390 | { |
| 391 | "id": "chart.cost.note", |
| 392 | "section": "Charts", |
| 393 | "label": "Cost to build: note", |
| 394 | "text": "What Opus 5.5 cost at list price for the build and its fix rounds.", |
| 395 | "context": "The line under the chart title that says exactly what was measured.", |
| 396 | "notes": "One sentence or two." |
| 397 | }, |
| 398 | { |
| 399 | "id": "chart.tokens.title", |
| 400 | "section": "Charts", |
| 401 | "label": "Tokens written to build: title", |
| 402 | "text": "Tokens written to build", |
| 403 | "context": "Title of one chart. Another view of the same effort.", |
| 404 | "notes": "Short." |
| 405 | }, |
| 406 | { |
| 407 | "id": "chart.tokens.note", |
| 408 | "section": "Charts", |
| 409 | "label": "Tokens written to build: note", |
| 410 | "text": "Output tokens Opus 5.5 produced for the build and its fix rounds.", |
| 411 | "context": "The line under the chart title that says exactly what was measured.", |
| 412 | "notes": "One sentence or two." |
| 413 | }, |
| 414 | { |
| 415 | "id": "chart.readability.title", |
| 416 | "section": "Charts", |
| 417 | "label": "Readability: title", |
| 418 | "text": "Readability", |
| 419 | "context": "Title of one chart. How easy the code is to read.", |
| 420 | "notes": "Short." |
| 421 | }, |
| 422 | { |
| 423 | "id": "chart.readability.note", |
| 424 | "section": "Charts", |
| 425 | "label": "Readability: note", |
| 426 | "text": "A blind score out of 10 from Fable, a different model from the builder. Median of three readings.", |
| 427 | "context": "The line under the chart title that says exactly what was measured.", |
| 428 | "notes": "One sentence or two." |
| 429 | }, |
| 430 | { |
| 431 | "id": "chart.loc.server", |
| 432 | "section": "Charts", |
| 433 | "label": "Lines of code legend: server", |
| 434 | "text": "server", |
| 435 | "context": "A legend entry on the lines-of-code chart, naming one part of the bar.", |
| 436 | "notes": "One or two words." |
| 437 | }, |
| 438 | { |
| 439 | "id": "chart.loc.templates", |
| 440 | "section": "Charts", |
| 441 | "label": "Lines of code legend: templates", |
| 442 | "text": "templates", |
| 443 | "context": "A legend entry on the lines-of-code chart, naming one part of the bar.", |
| 444 | "notes": "One or two words." |
| 445 | }, |
| 446 | { |
| 447 | "id": "chart.loc.js", |
| 448 | "section": "Charts", |
| 449 | "label": "Lines of code legend: JavaScript", |
| 450 | "text": "JavaScript", |
| 451 | "context": "A legend entry on the lines-of-code chart, naming one part of the bar.", |
| 452 | "notes": "One or two words." |
| 453 | }, |
| 454 | { |
| 455 | "id": "unit.connections", |
| 456 | "section": "Labels", |
| 457 | "label": "Connections value", |
| 458 | "text": "{n} connections", |
| 459 | "context": "Shown at the end of a bar in the slow-down charts, e.g. \"32 connections\".", |
| 460 | "notes": "Keep {n}." |
| 461 | }, |
| 462 | { |
| 463 | "id": "unit.out_of_ten", |
| 464 | "section": "Labels", |
| 465 | "label": "Readability value", |
| 466 | "text": "{n} / 10", |
| 467 | "context": "Shown at the end of a bar in the readability chart.", |
| 468 | "notes": "Keep {n}." |
| 469 | }, |
| 470 | { |
| 471 | "id": "label.errors_from", |
| 472 | "section": "Labels", |
| 473 | "label": "Errors marker", |
| 474 | "text": " · Carloku turned some requests away from {n}", |
| 475 | "context": "Appended after a slow-down value when, at the next step, Carloku's edge started answering some requests with 503 (its limit of 64 requests in flight per home box). E.g. \"64 connections · Carloku turned some requests away from 128\".", |
| 476 | "notes": "Starts with a space and a dot. Keep {n}." |
| 477 | }, |
| 478 | { |
| 479 | "id": "label.failed_from", |
| 480 | "section": "Labels", |
| 481 | "label": "Failures marker", |
| 482 | "text": " · requests started failing from {n}", |
| 483 | "context": "Appended after a slow-down value when, at the next step, the app itself stopped answering some requests (a 502 from Carloku's edge), as opposed to the edge turning them away.", |
| 484 | "notes": "Starts with a space and a dot. Keep {n}." |
| 485 | }, |
| 486 | { |
| 487 | "id": "label.still_scaling", |
| 488 | "section": "Labels", |
| 489 | "label": "Still scaling", |
| 490 | "text": "still scaling at {n}", |
| 491 | "context": "Shown in place of a bar when an app never slowed down within the test.", |
| 492 | "notes": "Keep {n}." |
| 493 | }, |
| 494 | { |
| 495 | "id": "label.blocked", |
| 496 | "section": "Labels", |
| 497 | "label": "Blocked", |
| 498 | "text": "blocked", |
| 499 | "context": "Shown in place of a bar for an app that couldn't run on Carloku. None remain, but the label exists." |
| 500 | }, |
| 501 | { |
| 502 | "id": "label.not_measured", |
| 503 | "section": "Labels", |
| 504 | "label": "Not measured", |
| 505 | "text": "not measured", |
| 506 | "context": "Shown in place of a bar when a number is missing." |
| 507 | }, |
| 508 | { |
| 509 | "id": "apps.heading", |
| 510 | "section": "The apps", |
| 511 | "label": "Heading", |
| 512 | "text": "The apps", |
| 513 | "context": "Heading over one short card per language with its links and notes." |
| 514 | }, |
| 515 | { |
| 516 | "id": "apps.live", |
| 517 | "section": "The apps", |
| 518 | "label": "Label: Live app", |
| 519 | "text": "Live app", |
| 520 | "context": "A label in each app's card. Link to the running app.", |
| 521 | "notes": "One or two words." |
| 522 | }, |
| 523 | { |
| 524 | "id": "apps.source", |
| 525 | "section": "The apps", |
| 526 | "label": "Label: Source", |
| 527 | "text": "Source", |
| 528 | "context": "A label in each app's card. Link to the app's code.", |
| 529 | "notes": "One or two words." |
| 530 | }, |
| 531 | { |
| 532 | "id": "apps.stack", |
| 533 | "section": "The apps", |
| 534 | "label": "Label: Stack", |
| 535 | "text": "Stack", |
| 536 | "context": "A label in each app's card. What the app is built with.", |
| 537 | "notes": "One or two words." |
| 538 | }, |
| 539 | { |
| 540 | "id": "apps.readability", |
| 541 | "section": "The apps", |
| 542 | "label": "Label: Readability", |
| 543 | "text": "Readability", |
| 544 | "context": "A label in each app's card. Fable's short note on the code.", |
| 545 | "notes": "One or two words." |
| 546 | }, |
| 547 | { |
| 548 | "id": "apps.unfixed", |
| 549 | "section": "The apps", |
| 550 | "label": "Label: Left after review", |
| 551 | "text": "Left after review", |
| 552 | "context": "A label in each app's card. Review findings that weren't fixed in the one fix round, quoted from the reviewer.", |
| 553 | "notes": "One or two words." |
| 554 | }, |
| 555 | { |
| 556 | "id": "apps.note", |
| 557 | "section": "The apps", |
| 558 | "label": "Label: Note", |
| 559 | "text": "Note", |
| 560 | "context": "A label in each app's card. Anything else worth knowing about this app.", |
| 561 | "notes": "One or two words." |
| 562 | }, |
| 563 | { |
| 564 | "id": "stack.go", |
| 565 | "section": "The apps", |
| 566 | "label": "Go: stack", |
| 567 | "text": "Go's standard web server and templates, with SQLite through modernc.org/sqlite (pure Go).", |
| 568 | "context": "The 'Stack' line in the Go card: what the builder chose to build the app with.", |
| 569 | "notes": "One sentence." |
| 570 | }, |
| 571 | { |
| 572 | "id": "stack.go-rastrillo", |
| 573 | "section": "The apps", |
| 574 | "label": "Go + Rastrillo: stack", |
| 575 | "text": "Rastrillo 0.28, the CARLOS web framework: chi for routing and GORM over SQLite.", |
| 576 | "context": "The 'Stack' line in the Go + Rastrillo card: what the builder chose to build the app with.", |
| 577 | "notes": "One sentence." |
| 578 | }, |
| 579 | { |
| 580 | "id": "stack.rust", |
| 581 | "section": "The apps", |
| 582 | "label": "Rust: stack", |
| 583 | "text": "axum on tokio, Maud for HTML, and rusqlite with SQLite built in.", |
| 584 | "context": "The 'Stack' line in the Rust card: what the builder chose to build the app with.", |
| 585 | "notes": "One sentence." |
| 586 | }, |
| 587 | { |
| 588 | "id": "stack.zig", |
| 589 | "section": "The apps", |
| 590 | "label": "Zig: stack", |
| 591 | "text": "Zig's standard-library HTTP server, with SQLite 3.54 compiled in.", |
| 592 | "context": "The 'Stack' line in the Zig card: what the builder chose to build the app with.", |
| 593 | "notes": "One sentence." |
| 594 | }, |
| 595 | { |
| 596 | "id": "stack.ocaml", |
| 597 | "section": "The apps", |
| 598 | "label": "OCaml: stack", |
| 599 | "text": "OCaml 5 with Eio and cohttp-eio, and sqlite3-ocaml over SQLite 3.54.", |
| 600 | "context": "The 'Stack' line in the OCaml card: what the builder chose to build the app with.", |
| 601 | "notes": "One sentence." |
| 602 | }, |
| 603 | { |
| 604 | "id": "stack.haskell", |
| 605 | "section": "The apps", |
| 606 | "label": "Haskell: stack", |
| 607 | "text": "warp and WAI, Lucid for HTML, and sqlite-simple.", |
| 608 | "context": "The 'Stack' line in the Haskell card: what the builder chose to build the app with.", |
| 609 | "notes": "One sentence." |
| 610 | }, |
| 611 | { |
| 612 | "id": "stack.swift", |
| 613 | "section": "The apps", |
| 614 | "label": "Swift: stack", |
| 615 | "text": "Hummingbird 2 on SwiftNIO, with SQLiteNIO.", |
| 616 | "context": "The 'Stack' line in the Swift card: what the builder chose to build the app with.", |
| 617 | "notes": "One sentence." |
| 618 | }, |
| 619 | { |
| 620 | "id": "stack.nim", |
| 621 | "section": "The apps", |
| 622 | "label": "Nim: stack", |
| 623 | "text": "Nim's standard-library async HTTP server (a patched copy, to serve a Unix socket and cap request bodies), with SQLite through db_connector.", |
| 624 | "context": "The 'Stack' line in the Nim card: what the builder chose to build the app with.", |
| 625 | "notes": "One sentence." |
| 626 | }, |
| 627 | { |
| 628 | "id": "stack.spinel", |
| 629 | "section": "The apps", |
| 630 | "label": "Spinel: stack", |
| 631 | "text": "tep, a Sinatra-style layer, compiled to a native program by Spinel, with SQLite 3.50 compiled in.", |
| 632 | "context": "The 'Stack' line in the Spinel card: what the builder chose to build the app with.", |
| 633 | "notes": "One sentence." |
| 634 | }, |
| 635 | { |
| 636 | "id": "note.spinel", |
| 637 | "section": "The apps", |
| 638 | "label": "Spinel: note", |
| 639 | "text": "Its server process exited 16 times during the load test, and Carloku started it again each time. Requests that arrived meanwhile got an error.", |
| 640 | "context": "A note in the Spinel card about what happened under heavy load.", |
| 641 | "notes": "" |
| 642 | }, |
| 643 | { |
| 644 | "id": "method.heading", |
| 645 | "section": "How we measured", |
| 646 | "label": "Heading", |
| 647 | "text": "How we measured", |
| 648 | "context": "Heading over the method section." |
| 649 | }, |
| 650 | { |
| 651 | "id": "method.body", |
| 652 | "section": "How we measured", |
| 653 | "label": "Body", |
| 654 | "text": "- **The same starting point.** Every builder got the same brief, the same rules for routes and forms, TodoMVC's own page and styles, and the same tests. It chose its own libraries.\n- **The same tests.** TodoMVC's 29 standard tests, run with JavaScript on and again with it off, plus checks for unsafe input, foreign characters, very long titles, one visitor reaching another's list, and a screenshot comparison with TodoMVC's own page.\n- **One build each.** Opus 5.5 built each app in a single unattended session, with a four-hour limit.\n- **One round of review.** Astra reviewed each app for bugs, missed requirements and security problems, and the same builder fixed what it found. A second round was allowed only for security problems; Zig and Nim needed one.\n- **Readability.** Fable scored each app's code out of 10 three times, without knowing which app it was. We show the median.\n- **Speed.** Measured from one test machine over the internet, one app at a time, each on a fresh copy of the app with an empty database. Time to first byte is the median of 30 page loads. The load test doubles the number of simultaneous connections from 1 upwards, 10 seconds per step, first loading a 50-item list, then ticking a to-do on and off, then loading the list again.\n- **Waking up.** Carloku puts an app to sleep after 15 quiet minutes. We timed the first request after each app fell asleep, three times.\n- **Builds.** Every app is built for the 64-bit ARM servers Carloku runs on. A cold build compiles every library from scratch, once per app. A one-line change rebuilds after adding one line to the code that serves the page, with everything else already compiled; it's the median of three.\n- **Cost.** What each build and its fixes cost at list price, from the builders' own session records.", |
| 655 | "context": "Explains the process: same brief, same model, same tests, review and fix rounds, readability scoring, and how the numbers were taken.", |
| 656 | "notes": "Markdown." |
| 657 | }, |
| 658 | { |
| 659 | "id": "caveats.heading", |
| 660 | "section": "Caveats", |
| 661 | "label": "Heading", |
| 662 | "text": "What this does and doesn't show", |
| 663 | "context": "Heading over the limits of the comparison." |
| 664 | }, |
| 665 | { |
| 666 | "id": "caveats.body", |
| 667 | "section": "Caveats", |
| 668 | "label": "Body", |
| 669 | "text": "- **The network is in every speed number.** Requests travelled from our test machine to Carloku over the internet, and the base delay swung between about 17 and 37 milliseconds during the runs. Small differences in first-byte times mean little.\n- **Carloku caps traffic at the edge.** When a request reaches an edge server that isn't the one running the app, Carloku lets at most 64 such requests be in flight at once and turns the rest away with a \"try again shortly\" response. Above 64 connections, the load test partly measures that limit, and whether it applied depended on how our traffic was routed at the time.\n- **Some apps failed outright at very high load.** At 256 or 512 connections, Swift, Nim and Spinel stopped answering some requests, which Carloku's edge reported as errors (502) rather than turning them away. The charts mark where that started.\n- **Every app ran on Carloku's default size.** Throughput limits reflect that size, not what each language could do on a bigger server.\n- **One build per language.** Another session with the same model could produce a different app, faster or slower.\n- **Readability scores cluster.** Most apps scored 8 out of 10. Treat the score as a rough signal.\n- **Three apps were built under emulation.** OCaml, Haskell and Spinel had no straightforward way to compile for ARM from our x86 machine, so they were built inside an emulated ARM system. Their build times are much longer for that reason, and don't compare with the others.\n- **Spinel restarted under heavy load.** Its server process exited during the load test and Carloku started it again each time.\n- **Mojo was dropped.** Mojo can't produce a single self-contained program file, which is what Carloku runs, so we replaced it with Nim. The platform question is filed as [an idea](https://amadan.net/carlos/platform/-/discussion/360).\n- **Fresh copies for measuring.** To give every app an empty database, each was measured on a new copy of itself, which is the one the live links point to.", |
| 670 | "context": "Plain list of what to be careful about when reading the numbers.", |
| 671 | "notes": "Markdown list." |
| 672 | } |
| 673 | ] |
amadan