How a run is scored

What one run is

A run is one model, with an empty memory, given one ticket and a shell. It reads the code, edits files, runs whatever commands it likes, and stops when it believes the ticket is done. Nothing carries over between runs. Eight runs of the same task in the same language are eight independent attempts by the same model, differing only in sampling.

So 2 out of 8 means the task was attempted eight times from scratch and the finished program was correct twice. It is not two test cases out of eight, and it is not a partially working program.

The model can check its work, but only partly

Each project ships a developer test command. Running it is the model's own choice, and most runs do. That command covers the obvious path: the basic behaviour described in the ticket, and the empty input. It is deliberately a small subset.

Grading uses a larger hidden set that the model never sees and that only runs after it has stopped. For the circuit breaker there are twelve checks in total; two are visible to the model and ten are not. The hidden ones cover the parts of the ticket that are easy to read past: recovery after the cooldown expires, the limit on trial calls while recovering, an unlisted status code that should count as neither success nor failure, and the full list of malformed inputs that must be rejected.

This split is why a run can end confidently and still be wrong. It mirrors the ordinary situation of a change that passes the tests someone wrote and breaks something nobody tested.

A failure, step by step

This is a real failing attempt at the circuit breaker in untyped Python, in six steps.

1. Listed the project and read the ticket.
2. Read the three source files and identified the real defects out loud: one breaker shared across all targets, no cooldown, almost no validation. That diagnosis was correct.
3. Rewrote the code so each target has its own breaker, and added the validation the ticket asks for.
4. Ran the developer test command. It reported "passed": true on both visible checks.
5. Invented one extra scenario by hand and ran it. The output included a call recorded as "state":"open","admitted":true: the breaker was open and the call was let through anyway, which is precisely the bug. The model did not notice.
6. Declared the work finished.

The hidden checks then failed it on cooldown behaviour, the recovery limit, and the transitions out of recovery. Nothing crashed, nothing timed out, and the model did not run out of turns.

Failing runs are not short runs

A reasonable worry about this setup is that failures are really the model giving up early or hitting a limit. They are not.

Failing runs did fractionally more work than passing ones, and every one of them ended because the model decided it was done. The step limit and the wall-clock budget are far above anything reached here, so neither is binding. Runs that end in a runner error are recorded separately as infrastructure and excluded from scoring rather than counted as failures; this cohort had none in 120 runs.

Why the verifier is trusted

Before any run costs money, the same checks are pointed at three known inputs. A correct reference implementation must pass everything in every language. The untouched starting code must fail the identical checks in every language. Then several deliberately broken versions, each with one specific fault, must be caught by the identical checks in every language.

If a check were stricter in one language than another, that gate would catch it, and no result is published without it passing. This matters more than it sounds: a verifier that is quietly harsher on one language would produce a confident language ranking out of nothing but its own bug.

← Return to the results