Skill Eval Harness

Skill Eval Harness is a Python CLI that measures the causal lift of an Agent Skill: it runs the same case with and without the skill, then reports what changed, what passed, and whether the eval leaked its own answer. It reads evals/shared-benchmark.json, emits answer-key-safe task rows, grades files under eval-runs/ locally and deterministically — no model call in the grade path — and writes benchmark reports you can diff across variants.

General eval frameworks (openai/evals, vitest-evals, viteval) score one output against a rubric. This one measures the difference the skill makes, and spends its surface area on keeping that difference honest: paired with/without comparison, tune/holdout/holdback split discipline, leakage lint, materialized ablations with provenance gates, and per-model lift. None of those frameworks have them, and they are what make a reported number trustworthy rather than merely green.

Questions this helps answer

Core loop

1. Describe cases in evals/shared-benchmark.json: prompt, split, fixture files, variants, assertions, and ablations.
2. Prepare tasks with skill-benchmark prepare; generation rows omit expected_behavior and judge rubrics unless you explicitly request them.
3. Run tasks with Claude, Codex, Jetty, or any runner that writes the run-output contract; Pi support is currently trigger-focused plus workspace-specific smoke tooling.
4. Grade outputs with deterministic assertions: string, regex, file, JSON field, and opt-in script oracles.
5. Inspect the report for pass rates, flaky repeated runs, no-lift cases, saturated assertions, judge tasks, and trigger/no-trigger results.

What the CLI owns

• Causal lift: with_skill vs without_skill (plus optional old_skill and ablation:<id>), with paired significance and per-model lift.
• Split discipline: tune, holdout, and holdback stay separate, so you can't tune on your test set.
• Local grading: deterministic assertions run without model calls.
• Eval hygiene: leakage lint, manifest audit, trigger checks, repeated-run stats, and fixture recommendations.
• Activation: does the skill load on its own? skill-trigger-matrix reports autonomous trigger rates per (agent × model), split by should-fire / should-not-fire.
• Cost as a signal: normalized token/dollar telemetry per run, a suite cost ledger, and lift-per-dollar (cost-summary, token-overhead).
• Interop: Anthropic-style exports, static/served HTML review pages, and Jetty runbook-mode import/export.
• Judge plumbing: judge/rubric assertions can be exported or run through native Claude/Codex backends (--judge-backend) or a user-supplied --judge-cmd; the harness does not choose a model for you.

Contents

• Questions this helps answer
• Quick start
• Installation
• Manifest format
• Assertions
• Run output contract
• Ablations
• Commands (full detail in docs/commands.md)
• Jetty adapter
• Contributing

Quick start

Requires Python 3.10+ and uv. Install from PyPI:

uv tool install skill-eval-harness

For exact reproducibility, pin the current release: uv tool install skill-eval-harness==0.5.1.

Run these from a skill repo that has evals/shared-benchmark.json:

# 1. Check manifest shape and fixture paths.
skill-benchmark validate evals/shared-benchmark.json

# 2. Emit answer-key-safe task rows for a runner.
skill-benchmark prepare evals/shared-benchmark.json \
  --split tune \
  --runs-per-variant 3 \
  --out /tmp/tasks.jsonl

# 3. Run each task with your agent runner and save:
# eval-runs/latest/<case_id>/<variant>/run-<n>/output.md
# eval-runs/latest/<case_id>/<variant>/run-<n>/metadata.json

# 4. Grade saved outputs. Add --allow-scripts only if you trust repo-owned oracles.
skill-benchmark benchmark evals/shared-benchmark.json \
  --runs eval-runs/latest \
  --split tune \
  --allow-scripts \
  --out benchmark.json

# 5. Open a static review page.
skill-benchmark render-viewer \
  --benchmark benchmark.json \
  --runs eval-runs/latest \
  --out review.html

Expected landmarks:

validate  -> OK: <skill-name> — <case-count> cases, <ablation-count> ablations
prepare   -> /tmp/tasks.jsonl, one JSON object per case/variant/run
benchmark -> benchmark.json with summary, results, and case_flags
viewer    -> review.html with assertion evidence and output previews

benchmark.json records one row per case/variant/run, plus aggregate pass rates, timing/token summaries, and flags for saturated, no-lift, flaky, or with-skill-failed cases. It also carries a reliability block — unbiased pass@k and pass^k per (case, variant) from the repeated runs — beside the paired lift's sign-flip significance.

Installation

From PyPI

uv tool install skill-eval-harness
skill-benchmark --help
skill-pi-trigger-eval --help

# Pin a release exactly:
uv tool install skill-eval-harness==0.5.1

# One-shot without installing globally:
uvx --from skill-eval-harness skill-benchmark --help

From GitHub

Use this for development snapshots before the next PyPI release:

uv tool install git+https://github.com/adewale/skill-eval-harness.git@main
uvx --from git+https://github.com/adewale/skill-eval-harness.git@main skill-benchmark --help

The installed commands are:

Local development

git clone https://github.com/adewale/skill-eval-harness.git
cd skill-eval-harness
uv tool install --editable .
skill-benchmark --help

Documentation map

docs/README.md groups these by kind (user journeys, concepts, specs, audits) and holds the convention for adding a new user-journey walkthrough.