Same task, same model, same agent, fresh context every run. Only the language changes.
Each language attempted three refactors, 24 times each. Python appears twice because one version adds type annotations and a type checker, holding everything else fixed. An agent step is one model action: read a file, run a command, apply an edit.
JavaScript 18/24 · TypeScript 15/23 · Python 13/24 · Python typed 16/24 · Go 15/24
JavaScript 7.00 · TypeScript 8.52 · Python 6.62 · Python typed 7.62 · Go 9.58
JavaScript $0.0065 · TypeScript $0.0074 · Python $0.0064 · Python typed $0.0070 · Go $0.0083
| Setup | Type checking | Passed | Agent steps | Output tokens | Mean cost |
|---|---|---|---|---|---|
| JavaScript | None | 18/24 | 7.00 | 3,196 | $0.006531 |
| TypeScript | Strict | 15/23 | 8.52 | 3,386 | $0.007440 |
| Python | None | 13/24 | 6.62 | 3,071 | $0.006395 |
| Python (typed) | mypy strict | 16/24 | 7.62 | 3,310 | $0.006987 |
| Go | Compiler | 15/24 | 9.58 | 3,961 | $0.008338 |
Pass rate measures errors. Steps, tokens, and cost measure the work required to reach a correct result. Elapsed time is not shown: this cohort ran four attempts at once, so wall clock includes contention and is not comparable across studies. Total measured spend was $0.85.
Setups within one run of each other are shown as tied (≈), because eight attempts cannot separate them.
expr-eval JavaScript 7/8 > Python 4/8 ≈ Python (typed) 4/8 ≈ TypeScript 3/8 > Go 0/8circuit-breaker Go 8/8 > TypeScript 5/8 ≈ JavaScript 4/8 ≈ Python (typed) 4/8 > Python 2/8money-rollup Python (typed) 8/8 ≈ TypeScript 7/7 ≈ Go 7/8 ≈ JavaScript 7/8 ≈ Python 7/8The state machine task fails when a branch of a union is dropped, which go build, tsc, and mypy all name at the point of the mistake: Go passes 8 of 8, Python 2 of 8. The money task fails on rounding mode and number formatting, which type check cleanly while being wrong, so a checker is silent and a library that does exact arithmetic is what pays. Both are ordinary bugs. Which kind a ticket contains is not a property of the language.
The evaluator task specifies 64-bit two's complement integers. JavaScript has no such type, so the model makes a visible decision and reaches for BigInt: 7 of 8. Go has exactly that arithmetic in int64 and passes 0 of 8, missing the parts that are not free, like reading a literal as unsigned before reinterpreting it and checking a shift count, while spending the most steps of any setup.
The cleanest comparison holds the language fixed: identical Python, differing only by annotations and a mypy --strict step that must pass before the tests run. The gap in pass rate was +0.125 (95% CI -0.125 to 0.375), an interval spanning zero, the same answer this pair gave last round. At this sample size no effect on correctness is detectable, which is a legitimate answer rather than a failed experiment.
JavaScript needed fewer agent steps than TypeScript, typed Python, and Go; plain Python needed fewer than Go. Those intervals clear zero. Cost follows effort directly: Go averaged $0.008338 a run against Python's $0.006395, 1.30 times as much. The price in effort shows up reliably even where a benefit to correctness does not.
Rerunning a fixed task and language with nothing changed but the random seed has moved a cell by 4 runs out of 8 in this project. Which task beats which holds up; the exact order inside one task does not, and no reading that leans on it is safe. A language here is a bundle including its toolchain and ecosystem, so none of this is a claim about syntax.
Not sure what a score like 2 out of 8 means? How a run is scored walks through one real failure step by step. Every paired contrast and the per-case failure breakdown are in the written report.
A command-line reporting tool has to stop doing money math in floating point. It switches to exact arithmetic, converts currencies along the shortest rate path and rejects ambiguous ones, totals every parent account, rounds half-to-even per entry, and rejects about thirty kinds of malformed input.
A simulator has to track one breaker per target through closed, open, and recovering, opening after enough consecutive failures, recovering only once a cooldown has elapsed, limiting trial calls while recovering, and treating an unlisted status code as neither success nor failure. Its starting code fails nine of twelve hidden checks in every language.
An evaluator for a small expression language has to handle a full precedence ladder, let bindings, and signed 64-bit two's complement arithmetic: wrapping, division that truncates toward zero, an arithmetic right shift, a checked shift count, and literals read as unsigned then reinterpreted. Its ticket is eight lines and points at a spec file in the workspace; its reference is a tokenizer, a parser, and an evaluator. The starting code fails 18 of 20 hidden checks in every language.
A task everything passes pins every difference at zero and answers nothing, so each of these has to fail often enough to compare, and each was chosen to stress something different. Before a task joins a round it has to clear a paid ten-run probe with its admission rule written down first: too easy and it ranks nothing, too hard and it ranks nothing either.
Before any paid run: reference solutions pass every check, the five untouched starters fail the identical checks, and four seeded sabotages are caught with identical failure sets in every language. Instructions are byte-identical across languages and the hidden expectations are not readable from the workspace. A green gate proves the scoring is fair, not that the task is hard enough to tell the languages apart.
Intervals are 95% percentile intervals from a fixed-seed 20,000-resample paired bootstrap over matched attempt blocks. The number of runs was fixed at 120 before the first paid call and no interim look influenced whether to continue.