Driftproof

Is the gap your skill makes real, or noise? Did it hold on the last model release?

A SKILL.md teaches an AI coding agent how you like things done. A skill's own tests passing is one answer. Driftproof asks two more: whether the scores with the skill and without it separate beyond their spread, and whether that held when the model changed. Each answer is a dated, hash-verified receipt, and when there were too few draws to tell, the receipt says so.

 — live badge for the bundled commit-message-conventions example, generated from its own receipt.

Quickstart · Latest report · driftproofhq.com

Driftproof consumes the agentskills.io/evals eval format; it does not invent its own.

📊 Nine published reports (each re-derived from committed receipts, nothing hand-entered), spanning seven published report types, the newest being instrument comparison. All nine read Driftproof's own arms by one band-based, floor-gated verdict rule and differ in what moves underneath the skill — or, in the value report, in which axes are measured; or, in the instrument re-measurement, in the instrument itself; or, in the instrument comparison, in which instrument measures:

• Report #001 — release drift: ten public agent skills across a current-vs-previous Sonnet release; 9 of 10 showed a separation detected under the rule. (markdown)
• Report #002 — substrate durability: the same suites across two vendors' CLIs (Claude vs Codex); 3 durable, 4 substrate-dependent, 2 regressed, 1 no effect.
• Report #003 — release drift: claude-opus-4-8 → claude-opus-5; 4 improved, 2 regressed, 4 with no separation detected.
• Report #004 — capability gap: claude-opus-5 (flagship) vs claude-fable-5 (frontier tier); 3 durable, 5 tier-dependent, 0 regressions, 2 no effect — encoded expertise survives the frontier tier.
• Report #005 — value: what a skill costs to run, on three axes (accuracy, cost, latency); the same ten suites on three substrates (claude-sonnet-5, claude-fable-5, gpt-5.6-sol) — 14 of 30 cells cleared the floor on aggregate: 10 carry a price and 4 report a saving instead, having improved quality while reducing cost. Three of those cells carry an amendment (v1.1, applied when Report #006 published): their lifts rest on single-draw baselines since shown unstable. Cause-agnostic, no corrected figures offered, and the cost-driver and substrate-disagreement findings below are unaffected.
• Report #006 — revision drift: the pinned skill revision against the one upstream ships today, on a held substrate. The reuse premise was tested and refused: 3 of 3 cells returned no verdict, each blocked by its own baseline control. A 120-call probe found generation-level sampling noise 3.2× and 7.5× larger than the judge-level noise this instrument actually samples — enough to account for every gap the controls saw without any other cause being established — and the receipt spec gains generation sampling as a result. No cause is asserted; the control proves non-reproduction and cannot say why. The tally a refusal carries: 3 cells, 0 measured, 3 refused.
• Report #007 — instrument re-measurement: the three cells Report #005 published for these skills, run again with the generation sampled adaptively instead of once and with the call timeout the surface policy declares. No cell separates: every lift is smaller than its own band, and the 21 cases read 3 improved, 0 regressed, 18 no effect, 0 not measured. Five of six comparisons against the archive were refused on their baseline control. The instrument defect it reports is its own: a declared 300 s timeout had been shadowed by a 120000 literal since 2026-07-27, and the truncated run it caused measured less variance than the clean re-run, which is the direction that flatters an instrument. Both runs are published, the broken one as evidence. Amends #005 to v1.2 and #006 to v1.1.
• Report #008 — release drift: two of Report #007's cells re-measured on claude-fable-5-1 against claude-fable-5, with the skill content_hash and suite_hash asserted identical before the first call. Neither cell showed a separation detected under the rule: the 14 cases read 0 improved, 0 regressed, 14 with no separation detected, 0 not measured, which is not evidence that nothing changed. The first release pair in this project where both sides are generation-sampled receipts, which is what makes the delta attributable to the model rather than to the instrument. One case sits inside the verdict on the effect floor alone and the report names it.
• Report #009 — instrument comparison: three skills from one plugin, one case each, measured by Claude Code's native plugin eval and by Driftproof on the same SKILL.md bytes, task prompts and rubrics. The two tools apply different treatments and grade differently, so the report reads them side by side, ranks neither, and states what each can and cannot establish. Both tools judged with claude-opus-5, which departs from the judge policy, so its figures are not comparable with Reports #001 to #008.

✍️ The launch essay, Three model releases later: what actually happens to agent skills, reads all nine reports together: what moves underneath a skill, what the skill costs to run, and what a corrected instrument did to three published results. Revised 2026-09-01; every figure in it is gate-checked against the report page it cites.

Why

A skill is usually tested once, against one model, and the verdict is treated as permanent. But the substrate moves: models get updated, retired, and replaced. A skill that measurably helped last quarter can quietly become a no-op — or a net negative — the next time the model underneath it changes, and nobody re-checks.

Verdicts age because the substrate moves. Driftproof exists to keep the verdict current: cheap, repeatable, hash-stamped measurements bound to a specific model version, so "does this skill still help?" has a dated, verifiable answer instead of a stale one.

The hard part isn't running an eval once — it's making the number credible enough to act on. An LLM judge is noisy, so a naive score can swing run to run by more than the drift you're trying to detect. Driftproof's answer is to sample and report bands, each a descriptive spread (the mean plus or minus one sample standard deviation), and to only claim a regression when the bands don't overlap: a separation detected under that rule, not proof. A tool that cries wolf is worse than no tool.

A verdict without a price is half an answer. The same receipts price the marginal cost of a skill firing, and Report #005 found the dominant cost driver is not the skill's own text but the input it causes the model to pull in: across those 30 cells the input delta tracks cost at r = +0.92 while the skill's own length tracks it at only r = +0.33, and one 738-token skill drew 34× its own size in extra input. Identical token deltas also price very differently across substrates — the same skill at near-identical deltas costs 3.3× more on claude-fable-5 than on claude-sonnet-5, which is exactly their input-rate ratio in the frozen snapshot.

Quickstart — receipt for your own skill in ~10 minutes

You need Node ≥ 22 and an ANTHROPIC_API_KEY.

# 1. Scaffold a skill skeleton: SKILL.md + evals/evals.json (3 example cases) + .driftproofrc
npx driftproof init my-skill

# 2. Edit the 3 example cases in my-skill/evals/evals.json so each one is grounded
#    in a claim your SKILL.md makes. Every rubric is anchored at 0.80 = "fully correct".

# 3. Point the runner at the metered API (scanner-safe: never commit a key)
export CLAUDE_PROVIDER=api
read -rsp "Anthropic API key: " ANTHROPIC_API_KEY && export ANTHROPIC_API_KEY

# 4. Run the suite. The shipped defaults (DEV_MAX_USD / DEV_MAX_CALLS in config.js)
#    refuse the run up front if the projection exceeds either; --max-usd and
#    --max-calls override them.
npx driftproof run my-skill --models claude-haiku-4-5

# 5. Read the receipt + human summary written to ./receipts/
cat receipts/*.summary.md

Reading the result. The verdict answers "does the skill still help on this model?" — PASSED when the with-skill score beats the baseline by at least the effect floor, NO_EFFECT when it doesn't, REGRESSED when the skill hurts. Each case is judged several times, so it carries a band (mean ± stddev) instead of one fragile number, and a case is only ever called regressed/improved when its two bands don't overlap. The effect floor (0.05, one judge quantization step) is the minimum move required before a separation is called a change: band separation plus a floor-sized delta, never either alone.

What the band covers. Since receipt spec v0.5 a run samples the generation as well as the judge: each arm is drawn at least 3 and at most 10 times (GENERATION_SAMPLES_MIN and GENERATION_SAMPLES_MAX in config.js, applied by lib/sampling.js), and each draw is judged several times. A run stops at 3 dra