How to Test an AI Skill: A Practical Guide to Evals

A skill can produce a convincing answer and still miss the task. A documentation reviewer might write polished feedback while overlooking the one setup command that prevents a new contributor from running the project.

The useful question is whether the skill improves a result you can inspect. This guide presents an original evaluation exercise for a documentation-review skill, with checks you can adapt to your own workflow. It is a proposed test plan, not a benchmark we have run.

What the current guidance recommends

The official Agent Skills evaluation guide recommends realistic prompts, expected outputs, and optional input files. It suggests starting with a few cases and comparing runs with the skill against runs without it or with an earlier revision. Fresh context helps keep previous attempts from influencing the comparison.

That gives maintainers a practical starting point. Our example below turns those principles into a small review exercise, rather than attempting to rank models or skills across unrelated tasks.

If you are still building your first package, start with how to create a SKILL.md file. The format specification defines the package and its metadata; a valid format alone does not establish that its instructions improve task performance.

Build a fixture with a known answer

Imagine a skill that checks whether a project's README matches its setup scripts. Create a small fictional repository specifically for the evaluation:

• Its package manifest defines a dev command and a lint command.
• Its README tells contributors to run a nonexistent serve command.
• Its README links to a missing local configuration guide.
• Its contribution guide correctly documents the lint command.

Write the expected findings before running the agent. For this fixture, success means identifying both real problems, pointing to their locations, and leaving the correct lint instructions alone.

Use synthetic files with no credentials or customer data. Keep the fixture in version control so another maintainer can reproduce the same starting point.

Test more than the obvious request

Here is a suggested starter set. These cases and criteria are editorial examples, not measured results.

Case	User request	What to inspect
Broken setup	Review this README against the project scripts.	Finds the missing command and broken local link.
Clean documentation	Check this corrected copy for setup errors.	Does not invent defects or demand unnecessary rewrites.
Missing manifest	Review setup instructions; the manifest is unavailable.	Explains which claims cannot be verified.
Narrow scope	Check only local links; leave wording alone.	Reports the broken link without expanding into a full rewrite.

The clean copy matters: a reviewer that always produces a list of problems can look productive while creating extra work. The missing-file case checks whether uncertainty appears in the final report instead of being hidden behind confident suggestions.

Compare the artifacts, not the success message

Run the same fixture and request with the skill and with your chosen baseline. Keep the model, available tools, permissions, and starting files consistent. Save the actual report and any proposed patch from each attempt.

For our example, use a short scorecard:

1. Were both planted defects found?
2. Did each finding identify the relevant file and evidence?
3. Were the valid lint instructions preserved?
4. Did the output respect the requested scope?
5. Were unavailable inputs acknowledged?

Some checks can be automated, such as verifying that a referenced local file exists. Others need judgment: a correct finding can still recommend a confusing fix. Review the proposed changes as well as the prose explaining them.

If you repeat a case, retain every attempt. Report counts with their denominator and describe the fixture set. A result on four small documentation exercises cannot establish reliability across arbitrary repositories.

Use failures to make smaller edits

Suppose the skill finds the broken command but invents a replacement. A useful revision might tell it to consult the manifest before recommending a command and to state when the manifest is missing.

Change that instruction, rerun the affected cases, and check the clean fixture again. Avoid adding a paragraph for every awkward sentence in an output: excessive instructions can make a narrow workflow harder to maintain.

Keep a few extra fixtures outside the examples used to revise the skill. Otherwise, improvements may reflect familiarity with one repository rather than better review behavior.

Record what you actually tested

Alongside a skill revision, save the fixture revision, model and client versions, enabled tools, run date, outputs, and unresolved failures. Record time or token usage only when the environment provides those measurements; do not infer savings from a shorter answer.

Quality evaluation also has limits. Finding README defects does not establish that bundled scripts are safe. Use the separate skill security checklist when reviewing unfamiliar packages.

When exploring skills by category, look for examples and reproducible checks that match your intended task. A useful evaluation tells you where a skill helped, where it failed, and what remains untested.

Sources checked November 1, 2026. The fixture, scorecard, and workflow suggestions are editorial examples; no performance results or product-wide guarantees are claimed.