Benchmarking a Coding Agent Honestly

Fixed model, pinned provider, both harnesses run identically at single attempt. Atlarix resolved 42 of 88 scored tasks (47.7%); opencode resolved 39 of 88 (44.3%). Three tasks apart.

That is a preliminary, self-run result, not an official leaderboard entry. The official Terminal-Bench leaderboard requires five attempts per task; we ran one. A three-task gap at single attempt sits well inside run-to-run noise. It does not prove Atlarix is ahead, and we ask you not to quote it as though it does.

Our benchmark page has been reporting these as 42 / 89 and 39 / 89. The raw result files say otherwise: the run attempted 89 trials, but the scored evaluation contains 88, and the pass and fail counts confirm it — 42 passed plus 46 failed is 88, and 39 plus 49 is 88 as well.

So one trial ran and never produced a reward record, and we divided by the attempted count instead of the scored one. The summary file published alongside the page said 88 the whole time. Two artefacts, same directory, disagreeing.

Correcting it moves Atlarix from 47.2% to 47.7%, which is half a percentage point and changes no conclusion. We are fixing it because a benchmark page that cannot keep its own denominator straight has no business asking anyone to trust the rest — not because it mattered to the result.

What should worry a reader far more is the next paragraph, which is about a quarter of the run.

Atlarix recorded 22 errored trials, 21 of them agent timeouts. opencode recorded 21, 20 of them timeouts. On a scored set of 88, that is roughly a quarter of the run on each side ending without a real verdict.

A timeout is not a wrong answer. It is the absence of one, and it is counted as a failure. So both harnesses are being scored substantially on how much work they finished inside a time limit rather than on whether they could solve the problem. The three-task gap is a small number sitting on top of a much larger pile of unfinished trials.

There is no Atlarix SWE-bench number. We have never run it, never published one, and any figure attributed to us is fabricated. We say that explicitly because SWE-bench is the benchmark people assume a coding agent has run, and silence reads as modesty rather than absence.

The Terminal-Bench figure above is also superseded — it was run in the v14.36 era against Terminal-Bench 2.0, and the benchmark itself has moved on since. We have not re-run it. Until we do, that is a historical measurement, not a current claim.

Raw result files, the exact command line, the model and provider pinned, and the failures left in. Our benchmark page publishes all of it, including the JSON these corrections came out of — which is how the denominator error was findable at all.

The rule we hold is that a number you cannot check is worth less than no number. That cuts against us as often as for us, and this post is what it looks like when it does.