crux

Crux takes pi, an open-source coding agent, and re-engineers its agent layer for long-horizon terminal work. On Terminal-Bench 2.1 — 89 real tasks spanning compilers, emulators, cryptanalysis, ML training and systems administration, each graded by a hidden test suite inside a container — it lifts pi from 0.539 to 0.773 pass@1, to the level of Claude Code running the same model (0.730).

The model weights never change. Crux's work is in the agent layer: the loop and its recovery paths, context management and compaction, tool contracts, and the runtime that keeps multi-hour runs alive. Every change ships with the measurement that justified it.

Highlights

• pi → Claude Code-level, same weights. pass@1 from 0.539 to 0.773 (+23.4 points) on Terminal-Bench 2.1 with a self-hosted 27B model; Claude Code on the same model scores 0.730. Reproduced at 0.793 on an independent run.
• A context engine that holds up under pressure. Compaction success taken from 14% to 100%; output truncation cut from 18.0% to 0.7%; single-task sessions summarized in full, with the task's exact wording carried through every compaction.
• Hang-free long runs. Timed-out trials had been idle for 94% of their wall clock; three root causes found and fixed, taking idle time to 3%.
• Statistically rigorous evaluation. Paired sign tests on concurrently run arms, a measured noise floor, and a failure-attribution pipeline that separates agent failures from infrastructure failures automatically.
• Designs proven against data, not adopted on reputation. Mechanisms from Claude Code, Codex, opencode, hermes-agent and grok-build were evaluated against 356 trajectories; 34 candidate designs were ruled out by the numbers before any code was written.

Results

Self-hosted Qwen3.8-27B · the 89 Terminal-Bench 2.1 tasks that run without a GPU inside the container · 8× agent budget · pass@1 over whole runs.

Reproducibility. Two independent runs of the final configuration scored 0.773 and 0.793. The benchmark's run-to-run variance was measured directly, and every comparison in this repository is paired task by task to account for it.

Architecture

flowchart TB
    subgraph agent["Agent · packages/"]
        direction LR
        ctx["Context engine<br/>window-fitted compaction<br/>verbatim task"] <--> loop["Agent loop<br/>state machine · recovery<br/>budget pacing"] <--> tools["Tools<br/>bash · read · edit · write<br/>stream watchdog"]
    end
    subgraph evals["Evaluation system · benchmark/"]
        direction LR
        pre["Preflight"] --> run["Containerized runs"] --> attr["Failure attribution"] --> stats["Paired sign tests<br/>→ next change"]
    end
    agent ==> evals

Core engineering

Each capability below is tied to the trajectory evidence that motivated it.

Context engine

Liveness and robustness

Agent loop

Tools and prompting

Evaluation methodology

• Paired, concurrent comparisons. Arms are compared task by task with a sign test, and only when run in the same time window — run-to-run variance on this benchmark is large enough that totals alone mislead.
• Mechanism health before tuning. A mechanism's success rate is counted before any of its parameters are touched; this is how the compaction failure rate was found and fixed.
• Failure attribution. A seven-category taxonomy classifies every trial automatically and separates the agent's failures from the environment's, including verifiers whose own test runners failed to install.
• Within-task analysis. For tasks solved in some runs and not others, the solved and failed runs are compared directly, which controls for task difficulty.
• Preflight gating. Every launch is checked for endpoint, launch settings and harness checksum before it spends GPU hours.

Repository layout

packages/          the agent: core loop, model layer, tools, CLI (built on pi)
benchmark/         the evaluation system
  src/crux/        harbor agent, prompt sections, failure analysis
  scripts/         launchers, preflight, paired comparison, health checks
  tests/           486 tests

Quick start

cd benchmark && uv sync
harbor run --dataset terminal-bench/terminal-bench-2-1 \
  --agent crux.pi_agent:CruxPiAgent --model openai/<model>

benchmark/scripts/preflight.sh <launcher> validates the endpoint, the launch settings and the harness checksum before a run. See benchmark/README.md for the evaluation system.

Acknowledgements

Crux is built on pi by Earendil Works. Upstream issues and pull requests belong at earendil-works/pi.