Public results and task definitions for FrontierHarness Eval
# Add to your Claude Code skills
git clone https://github.com/frontier-harness-eval/evalLast scanned: 9/4/2026
{
"issues": [],
"status": "PASSED",
"scannedAt": "2026-09-04T08:38:23.011Z",
"npmAuditRan": true,
"pipAuditRan": true,
"promptInjectionRan": true
}eval is an open-source ai agents skill for AI coding assistants such as Claude Code, Codex CLI, and ChatGPT, built by frontier-harness-eval. Public results and task definitions for FrontierHarness Eval. It has 144 GitHub stars.
Yes. eval passed SkillsLLM's automated security scan — a dependency vulnerability audit plus prompt-injection heuristics — with no high-severity issues. You can read the full report in the Security Report section on this page.
Clone the repository with "git clone https://github.com/frontier-harness-eval/eval" and add it to your Claude Code skills directory (see the Installation section above).
eval is primarily written in JavaScript. It is open-source under frontier-harness-eval on GitHub, so you can review or fork the full source.
Yes. SkillsLLM lists many other AI Agents skills you can browse and compare side by side. Open the AI Agents category from the badge at the top of this page, or use the Related Skills and comparison links further down to weigh eval against similar tools.
No comments yet. Be the first to share your thoughts!
⚠️ Third-Party Software Notice
This skill is third-party open-source software developed and hosted independently on GitHub. SkillsLLM is an informational directory and does not control or maintain the underlying repository.
Any security checks, ratings, or warnings displayed by SkillsLLM are automated and limited in scope. They do not constitute a security certification or guarantee that the software is safe, error-free, or free from malicious code, vulnerabilities, compromised dependencies, or prompt-injection risks.
Review the source code, permissions, dependencies, and configuration before installing or running any third-party skill. Use is at your own risk. To the maximum extent permitted by applicable law, SkillsLLM is not liable for losses arising from third-party software.
We ran the same Kimi K3 model through nine coding-agent harnesses—12 configurations in total—on the same 30 software-engineering tasks. With the model, tasks, and runtime held constant, changing the harness changed pass rate, cost, cache behavior, and speed.
| Harness (configuration) | Pass rate | Median cost per pass | Cache, median cell | Median time |
|---|---|---|---|---|
| Codex | 66.7% | $3.47 | 88.0% | 6m 43s |
| DSH Creator | 63.3% | $3.28 | 84.3% | 6m 44s |
| Claude Code | 63.3% | $18.34 | 67.8% | 9m 38s |
| Pi | 60.0% | $2.43 | 79.4% | 7m 33s |
| DSH PTC | 60.0% | $4.58 | 87.2% | 7m 44s |
| DSH Standard | 60.0% | $3.46 | 86.5% | 6m 17s |
| Oh My Pi | 56.7% | $4.75 | 82.2% | 6m 46s |
| Kimi Code | 56.7% | $3.65 | 88.0% | 7m 56s |
| DSH Minimal | 56.7% | $4.72 | 84.6% | 5m 41s |
| Exo Harness | 53.3% | $1.05 | 70.3% | 6m 17s |
| OpenCode | 50.0% | $3.24 | 78.4% | 6m 27s |
| Hermes | 50.0% | $2.90 | 85.9% | 6m 58s |
The interactive report includes failed runs, total cost per task, cache behavior, speed, and task-level results. For the evaluation design and analysis, read the launch article.
.
├── benchmark.json # Public benchmark definition
├── cli/index.mjs # `npx @frontierharness/eval`: workspace + skill installer
├── metadata/
│ ├── difficulty.json # Difficulty assignments and source methodology
│ └── harness-versions.json # Harness versions used for the run
├── results/
│ └── eval-data.json # Normalized aggregate and task-level results
├── tasks/<task>/
│ ├── instruction.md # Prompt shown to every harness
│ └── task.toml # Public task metadata and environment definition
└── skills/frontierharness-eval/ # Agent-neutral skill, usable by hand
├── SKILL.md # Evaluation workflow for a third-party harness
├── PROMPT.md # Copy-paste prompt that points an agent at the skill
├── reference.md # Command reference, runner templates, troubleshooting
└── scripts/ # Provisioning, trial runner, scoring, chart, report
The repository intentionally contains results, task definitions, and the evaluation workflow. Internal infrastructure, credentials, runtime identifiers, private evidence, solutions, and deployment configuration are not included.
The workflow that produced the table above ships with this repository, so a harness that is not in it can be scored on the same tasks, runtime, and cost accounting, then placed directly next to the twelve baseline configurations.
skills/frontierharness-eval/ is an agent-neutral skill: point any coding agent that reads SKILL.md at it and it will drive the whole run — freezing the golden checkpoint, running every task from an identical fresh restore, scoring the trials, and building the report.
Clone this repository and open it in any coding agent:
git clone https://github.com/frontier-harness-eval/eval.git
Ask the agent to evaluate your harness:
use the skill located in the repo to evaluate [your harness github link]
The skill's scripts are the same ones an agent would call, so the run works without an agent at all. Every path is relative to the workspace root, and this repo has its own scripts/ directory, so address the skill's scripts through a variable:
FH=skills/frontierharness-eval/scripts
1. Prerequisites. The runta CLI (brew install runta-dev/tap/runta or npm i -g @runta/runta-cli) authenticated with runta login, plus jq and node >= 18.
2. Install script. Write a script that builds your harness on a clean Linux box. If it is not a built-in agent for Harbor or Pier, register it as a custom agent in both runner registries there, and use the registered name as --harness.
3. Provider key. Store it once as a Runta secret, named after the env var for your provider (FIREWORKS_API_KEY, MOONSHOT_API_KEY, OPENROUTER_API_KEY, or TOGETHER_API_KEY). The interactive prompt keeps the value out of your shell history:
runta secret set FIREWORKS_API_KEY --prompt
The API never hands the value back, so provisioning reuses the stored secret instead of asking for plaintext again. A --value-env or --value-stdin route works too if you already have the key in the environment.
4. Golden checkpoint. One command creates the clean runtime, clones the harness at a pinned commit, installs the Harbor and Pier stacks, pre-pulls the task images, and freezes the checkpoint:
$FH/provision-golden-checkpoint.sh \
--runtime fh-build --checkpoint fh-golden-myharness-v1 \
--harness my-harness --provider fireworks \
--repo https://github.com/acme/my-harness --commit 9f2c1ab \
--cpus 4 --memory 8192 --disk-size-gib 100 \
--prepull-tasks tasks --install-script ./install-my-harness.sh
The real key stays in the egress proxy, so confirm the runtime only ever sees a stub:
runta exec fh-build -- sh -lc 'test "$FIREWORKS_API_KEY" = runta-secret-stub'
5. Trials. Each task gets its own fresh restore, which is then deleted. With no --tasks, this runs the published 30-task set read from tasks/:
$FH/run-trials.sh \
--checkpoint fh-golden-myharness-v1 --harness my-harness \
--provider fireworks --run-id 2026-09-02-myharness --out runs
Pass a file of suite-prefixed ids to --tasks to run a subset — worth doing first with one Terminal-Bench and one DeepSWE task to prove the plumbing before spending the full budget. Re-running the same --run-id replaces only the tasks you list. If a trial dies on infrastructure twice, mark it rather than scoring it as a failure:
trial=runs/2026-09-02-myharness/trials/terminal-bench-<task>/trial.json
jq '.status = "infra_invalid" | .success = false' "$trial" > "$trial.tmp" && mv "$trial.tmp" "$trial"
6. Score, chart, and report.
node $FH/normalize-results.mjs --run runs/2026-09-02-myharness --label "My Harness"
node $FH/generate-chart.mjs --run runs/2026-09-02-myharness
node $FH/build-report.mjs --run runs/2026-09-02-myharness
Per-step reasoning, runner templates, and troubleshooting are in SKILL.md and reference.md.
A score only belongs next to the published numbers if the run holds these invariants. The report states any that were relaxed.
reference.md.terminal-bench-sample.infra_invalid, not scored as task failures.Cost is compared on effective_cost_per_pass, which is total cost across all tasks divided by passes and is reproducible from raw per-task cost. The *_normalized fields in results/eval-data.json reprice first-turn cache reads using data that is not public, so the scoring script leaves them empty rather than inventing values.
Metric definitions and the trial record contract are in SKILL.md. Runner templates, the alternative