Run / attempt
One fresh agent context working on one language variant. Attempts do not share memory.
Prospective v0.7
Technical details and paired uncertainty
This page keeps replication data available without asking every reader to understand tokens, cache behavior, or agent-loop internals before they understand the experiment.
Terms used in the results
Pass
The final program satisfied all twelve checks in the shared behavioral verifier, including exact arithmetic, chained conversion, rounding ties, and the rejection surface.
10/12 passed
Twelve separate agent attempts were run in that condition and ten passed. This is not a count of test cases.
Agent step
One model action in the linear mini-swe-agent loop. It can contain shell commands, inspection, edits, or verification work.
Model rung
A fixed model and effort setting. The study repeats the same task at more than one rung, because whether the language matters may depend on how capable the agent is.
Output tokens
Tokens generated by the model. This is a rough measure of response volume, not code quality.
Fixed study configuration
ModelGPT-5.6 Luna
ReasoningLow
Agentmini-swe-agent 2.4.6
RunnerPier 0.3.1
The model, effort, agent scaffold, instruction bytes, language runtime versions, verifier bytes, and per-run budget were held fixed across languages. Each run began with a fresh context, and the randomized schedule was fixed before launch and executed without reordering.
Why the task changed for v0.7
Through v0.6 the benchmark had a ceiling: all 36 prospective attempts passed, so every measured language difference in correctness was exactly zero by construction and no interval could be estimated. v0.7 introduces money-rollup, a four-file brownfield refactor from floating-point money handling to exact rational arithmetic, with shortest-path currency conversion and ambiguity rejection, ancestor rollups, half-to-even per-entry rounding, and a large rejection surface. Calibration is green: the reference solutions pass every check, all four starters fail the identical eight checks, and four seeded sabotages are caught with identical failure sets in every language. Unlike the earlier families, this one ships a developer-only test harness so hidden expectations are not readable from the workspace.
Prospective five-arm results
| Arm | Passed | Mean cost | Input tokens | Output tokens | Agent steps |
|---|---|---|---|---|---|
| JavaScript | 18/24 | $0.006531 | 39,662 | 3,196 | 7.00 |
| TypeScript | 15/23 | $0.007440 | 60,795 | 3,386 | 8.52 |
| Python | 13/24 | $0.006395 | 36,683 | 3,071 | 6.62 |
| Python (typed) | 16/24 | $0.006987 | 47,593 | 3,310 | 7.62 |
| Go | 15/24 | $0.008338 | 72,503 | 3,961 | 9.58 |
Interpretation: one pooled pass-rate contrast excludes zero, JavaScript over Python, while the primary contrast between Python and typed Python does not. Every agent-step contrast against Go excludes zero, and so does JavaScript against TypeScript and typed Python. These pooled figures conceal the per-task breakdown on the overview, which is the more useful result: Go passes 8 of 8 on the state machine task and 0 of 8 on the evaluator. Paired contrasts use matched attempt blocks. Per-case failure breakdowns and patch metrics are in the full report.
Spend and confidence
Attempts119
Passed77
API spend$0.849144
Infrastructure failures1
Eight attempts per cell is enough to expose the reversal between families but not to resolve small differences within one. The stopping rule was fixed at 120 rollouts before the first paid call and no interim analysis informed continuation, unlike v0.7 where the second batch was run because the first left an interval touching zero. Elapsed time is not reported: this cohort ran four attempts concurrently, so wall clock includes contention. Rollouts ending in a runner exception are recorded in the ledger and excluded from scoring as infrastructure rather than counted as failures; this cohort had one, at $0.0005.
Known limitations
This is one model snapshot, one bash-only agent scaffold, three task families, and eight attempts per cell. It does not test long-term maintenance, browser applications, dependency migrations, or human review. Language is a treatment bundle that includes file topology, toolchain, and ecosystem differences, so a step difference is not evidence about syntax. A different hard task could rank the languages differently, and in this cohort one demonstrably did.