Public results and task definitions for FrontierHarness Eval
# Add to your Claude Code skills
git clone https://github.com/frontier-harness-eval/evaleval is an open-source ai agents skill for AI coding assistants such as Claude Code, Codex CLI, and ChatGPT, built by frontier-harness-eval. Public results and task definitions for FrontierHarness Eval. It has 94 GitHub stars.
eval's catalog security scan is still queued. You can run an instant dependency and prompt-injection check now with the "Scan for vulnerabilities" button above.
Clone the repository with "git clone https://github.com/frontier-harness-eval/eval" and add it to your Claude Code skills directory (see the Installation section above).
eval is primarily written in JavaScript. It is open-source under frontier-harness-eval on GitHub, so you can review or fork the full source.
Yes. SkillsLLM lists many other AI Agents skills you can browse and compare side by side. Open the AI Agents category from the badge at the top of this page, or use the Related Skills and comparison links further down to weigh eval against similar tools.
No comments yet. Be the first to share your thoughts!
Unlocks once the catalog security scan passes (runs nightly).
⚠️ Third-Party Software Notice
This skill is third-party open-source software developed and hosted independently on GitHub. SkillsLLM is an informational directory and does not control or maintain the underlying repository.
Any security checks, ratings, or warnings displayed by SkillsLLM are automated and limited in scope. They do not constitute a security certification or guarantee that the software is safe, error-free, or free from malicious code, vulnerabilities, compromised dependencies, or prompt-injection risks.
Review the source code, permissions, dependencies, and configuration before installing or running any third-party skill. Use is at your own risk. To the maximum extent permitted by applicable law, SkillsLLM is not liable for losses arising from third-party software.
The deep catalog scan for this skill is still queued. Run an instant dependency check now instead.
We ran the same Kimi K3 model through nine coding-agent harnesses—12 configurations in total—on the same 30 software-engineering tasks. With the model, tasks, and runtime held constant, changing the harness changed pass rate, cost, cache behavior, and speed.
| Harness (configuration) | Pass rate | Median cost per pass | Cache, median cell | Median time |
|---|---|---|---|---|
| Codex | 66.7% | $3.47 | 88.0% | 6m 43s |
| DSH Creator | 63.3% | $3.28 | 84.3% | 6m 44s |
| Claude Code | 63.3% | $18.34 | 67.8% | 9m 38s |
| Pi | 60.0% | $2.43 | 79.4% | 7m 33s |
| DSH PTC | 60.0% | $4.58 | 87.2% | 7m 44s |
| DSH Standard | 60.0% | $3.46 | 86.5% | 6m 17s |
| Oh My Pi | 56.7% | $4.75 | 82.2% | 6m 46s |
| Kimi Code | 56.7% | $3.65 | 88.0% | 7m 56s |
| DSH Minimal | 56.7% | $4.72 | 84.6% | 5m 41s |
| Exo Harness | 53.3% | $1.05 | 70.3% | 6m 17s |
| OpenCode | 50.0% | $3.24 | 78.4% | 6m 27s |
| Hermes | 50.0% | $2.90 | 85.9% | 6m 58s |
The interactive report includes failed runs, total cost per task, cache behavior, speed, and task-level results. For the evaluation design and analysis, read the launch article.
.
├── benchmark.json # Public benchmark definition
├── metadata/
│ ├── difficulty.json # Difficulty assignments and source methodology
│ └── harness-versions.json # Harness versions used for the run
├── results/
│ └── eval-data.json # Normalized aggregate and task-level results
└── tasks/<task>/
├── instruction.md # Prompt shown to every harness
└── task.toml # Public task metadata and environment definition
The repository intentionally contains results and task definitions only. Internal infrastructure, credentials, runtime identifiers, private evidence, solutions, and deployment configuration are not included.
| Configuration | Version | Configuration | Version |
|---|---|---|---|
| Codex | 0.148.0 |
DSH Creator | 0.1.0-rc.8 |
| Claude Code | 2.1.237 |
DSH Minimal | 0.1.0-rc.8 |
| Pi | 0.84.2 |
DSH PTC | 0.1.0-rc.8 |
| DSH Standard | 0.1.0-rc.8 |
Oh My Pi | 17.4.0 |
| Kimi Code | 0.37.2 |
Exo Harness | 0.1.0 |
| OpenCode | 1.18.19 |
Hermes | 0.20.4 |
task.toml are prepared once as a golden checkpoint. Every run is a fresh restore with identical vCPU, memory, disk size, disk contents, and memory state.See benchmark.json for the public benchmark definition and results/eval-data.json for the complete normalized result set.
jq '.harnesses[] | {name, successful, effective_cost_per_pass}' results/eval-data.json
Every task directory contains the exact public instruction and task metadata used by the benchmark.
Runta provided the isolated runtimes and Golden Checkpoint restores used across all 360 evaluations.