Benchmarks

    One controlled, self-run comparison of Atlarix against opencode on Terminal-Bench 2.0, holding the model, provider and infrastructure constant. Raw result files and full reproduction steps are below.

    42 / 89
    Tasks resolved by Atlarix

    Terminal-Bench 2.0, all 89 tasks

    39 / 89
    Tasks resolved by opencode

    Same model, provider and infrastructure

    k=1
    Attempts per task

    The official leaderboard requires k=5. We have not run it.

    Atlarix

    Resolved
    42 / 89
    Score
    47%

    opencode

    Resolved
    39 / 89
    Score
    44%

    MiniMax-M3 (fp8) · single attempt · native function-calling · matched timeouts · Modal.

    This does not show Atlarix is ahead. A 3-task difference at single-attempt is within run-to-run noise, and the official leaderboard requires five attempts. The most it supports: on this model, Atlarix is competitive with a leading harness. The sections below say exactly why, and how to check the numbers yourself.

    The full picture

    Nothing here is hidden — every section is part of the prerendered page, collapsed so the result reads first.

    These numbers are a historical data point, not a current claim

    This run is superseded in two ways, and we would rather say so than let the table read as current.

    Terminal-Bench has moved to 4.0. Scores on 2.x are not comparable to it — 4.0 recalibrated task resources, dropped saturated tasks and standardised an 8-hour agent timeout.

    Our headless runner was handicapping itself. It dispatched reasoning one notch below each model's declared ceiling, while every published leaderboard entry runs at the ceiling. That is fixed, but this run has not been repeated.

    What a single-attempt, self-run comparison can and cannot show

    This is a preliminary, single-attempt (k=1) run we did ourselves. The official Terminal-Bench leaderboard requires 5 attempts (k=5), which we have not yet done. So it does not prove Atlarix is "ahead" of anyone — a 3-task difference at single-attempt is within run-to-run noise.

    The honest takeaway is that an open model expresses its ability roughly as well under Atlarix as under a leading harness — the harness is not holding the model back. Absolute scores sit below this model's ~66% tuned-scaffold ceiling, as expected for a general harness at fp8 and single attempt.

    We are also still working toward a verified place on the Terminal-Bench leaderboard. We are not there yet.

    Raw results — the unedited job files, verify it yourself

    The unedited output files from the run — Harbor job results with per-task pass/fail for both harnesses. Nothing here is hand-typed; download and check them.

    summary.csvsummary.mdatlarix — full job result.jsonopencode — full job result.json
    Reproduce it yourself — the exact bundle, dataset and commands

    Setup we used

    The exact bundle we ran is public — the same Electron-free Atlarix headless build, downloadable as a release tarball. The benchmark runs on the open-source Harbor framework.

    • Dataset: terminal-bench/terminal-bench-2 (all 89 tasks)
    • Model: minimax/minimax-m3, routed through OpenRouter and pinned to one provider at fp8 — identical for both harnesses
    • Infrastructure: Harbor on Modal (-e modal), one isolated container per task
    • Settings (both harnesses, identical): single attempt (-k 1), native timeout (--timeout-multiplier 1), native function-calling forced (no text-tool shim)
    • Atlarix bundle (public): atlarix-headless-linux-amd64.tar.gz
    # Atlarix harness
harbor run -d terminal-bench/terminal-bench-2 \
  -m openai/minimax/minimax-m3 \
  -n 24 -k 1 -y --timeout-multiplier 1 --max-retries 3 \
  -e modal --agent-import-path atlarix_tb:AtlarixAgent

# opencode harness (same model + provider + infra)
harbor run -d terminal-bench/terminal-bench-2 \
  -m bench/minimax/minimax-m3 \
  -n 24 -k 1 -y --timeout-multiplier 1 --max-retries 3 \
  -e modal --agent-import-path atlarix_tb.opencode_proxy:BenchOpenCodeAgent

    A note on routing, going forward. The run above went through OpenRouter, pinned to a single provider, because that was the cleanest way to guarantee both harnesses saw byte-identical weights and precision. Atlarix Core now calls the provider APIs directly, so future runs will too — which also makes them easier for you to reproduce: bring your own key from the provider itself and point the headless build at it. We are leaving the setup above exactly as recorded rather than restating it in current terms; the numbers belong to the run that produced them.

    Full disclosure: the one change we made to run unattended

    In the Atlarix desktop app, the agent asks your approval before every file write and any dangerous command, and file edits land through a diff you review — that is a core safety feature. A benchmark runs unattended, with no human to approve anything. So to participate at all, we grant that approval once, up front, via an explicit operator flag (ATLARIX_AUTONOMOUS_DANGER=1). Without it, every task that needs a flagged install, a cleanup, or a privileged command would simply be blocked and fail.

    This is the only deviation from the shipping app's default behavior — and it is not an advantage over the other harness: every agent auto-approves to run an automated benchmark (it is inherent to running unattended). We state it plainly so the setup is fully transparent. The flag is off by default; the interactive app always asks before file writes and dangerous commands.

    Where we are, and what's next

    • Official leaderboard: a full 5-attempt (k=5) run for a verified Terminal-Bench submission — the goal we're working toward.
    • More open models: the same head-to-head on GPT 5.6, Kimi and Qwen, so any claim holds across more than one model.
    • More benchmarks: SWE-bench and beyond, so the picture never rests on a single test.

    More engineering write-ups on the blog.