Maintainer of this skill?
Automated security scan — not a certification.
by NVIDIA
Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.
# Add to your Claude Code skills
git clone https://github.com/NVIDIA/SkillEvaluatorGuides for using ai agents skills like SkillEvaluator.
Last scanned: 9/2/2026
{
"issues": [],
"status": "PASSED",
"scannedAt": "2026-09-02T08:25:47.963Z",
"npmAuditRan": true,
"pipAuditRan": true,
"promptInjectionRan": true
}See how SkillEvaluator compares with popular alternatives.
SkillEvaluator is an open-source ai agents skill for AI coding assistants such as Claude Code, Codex CLI, and ChatGPT, built by NVIDIA. Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. It has 572 GitHub stars.
Yes. SkillEvaluator passed SkillsLLM's automated security scan — a dependency vulnerability audit plus prompt-injection heuristics — with no high-severity issues. You can read the full report in the Security Report section on this page.
Clone the repository with "git clone https://github.com/NVIDIA/SkillEvaluator" and add it to your Claude Code skills directory (see the Installation section above).
SkillEvaluator is primarily written in Python. It is open-source under NVIDIA on GitHub, so you can review or fork the full source.
Yes. SkillsLLM lists many other AI Agents skills you can browse and compare side by side. Open the AI Agents category from the badge at the top of this page, or use the Related Skills and comparison links further down to weigh SkillEvaluator against similar tools.
No comments yet. Be the first to share your thoughts!
30 days in the Featured rail · terms & refunds
⚠️ Third-Party Software Notice
This skill is third-party open-source software developed and hosted independently on GitHub. SkillsLLM is an informational directory and does not control or maintain the underlying repository.
Any security checks, ratings, or warnings displayed by SkillsLLM are automated and limited in scope. They do not constitute a security certification or guarantee that the software is safe, error-free, or free from malicious code, vulnerabilities, compromised dependencies, or prompt-injection risks.
Review the source code, permissions, dependencies, and configuration before installing or running any third-party skill. Use is at your own risk. To the maximum extent permitted by applicable law, SkillsLLM is not liable for losses arising from third-party software.
Automated security scan — not a certification.
📺 Livestream: Ask the Experts: Evaluating Agent Skills
SkillEvaluator is an open-source, multi-tier framework for evaluating AI agent artifacts, starting with agent skills: deterministic quality gates, semantic overlap detection, synthetic eval dataset generation, and live agent evaluation.
Agent skills extend AI agents with instructions and supporting files, as defined by the Agent Skills specification. SkillEvaluator is part of the NVIDIA Verified Skills pipeline. Skills that pass are published to the NVIDIA skills catalog.
Research foundation. Tier 3 implements methods from Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills: paired with/without-skill trials and Skill Lift; SkillEvaluator provides validation, deduplication, dataset authoring, and reporting.
Run each tier independently.
| Tier | Purpose | Run one tier | Requires |
|---|---|---|---|
| Tier 1: Validation | Safe & well-formed? | skillevaluator tier1 ./my-skill |
External scanners for full coverage; a provider enables rubric and LLM security checks |
| Tier 2: Deduplication | Repeated or overlapping guidance? | skillevaluator tier2 ./my-skill |
Embeddings and chat; add --catalog ./skill-catalog.json for inter-skill comparison |
| Tier 3: Live Evaluation | Does it help the agent? | skillevaluator tier3 ./my-skill |
Provider access, a supported agent, and a running sandbox (Docker by default) |
SkillSpector provides Tier 1 security scanning; Harbor, the open-source agent evaluation framework, powers Tier 3 sandboxed agent runs. See the tier guides.
Install with uv:
uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
NVIDIA Build: set just two variables. API key:
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
This key covers chat, Tier 2 embeddings, and supported Tier 3 agents in Docker or local mode.
For gateways, set SKILL_EVAL_LLM_PROVIDER=openai-compatible,
SKILL_EVAL_LLM_BASE_URL, and SKILL_EVAL_LLM_API_KEY.
Model defaults are overridable;
the gateway must support the selected models and APIs.
Run all tiers on ./my-skill, a directory containing SKILL.md:
skillevaluator validate ./my-skill
Run individual tiers:
skillevaluator tier1 ./my-skill
skillevaluator tier2 ./my-skill
skillevaluator tier3 ./my-skill
--no-llm to opt out of LLM checks.--catalog ./skill-catalog.json to compare against other skills too; without
a catalog, the output explicitly says inter-skill comparison did not run.A tier without a path shows help.
Expert commands such as tier1 security-scan, tier2 similarity-check, and
tier3 create-eval-dataset remain available.
[all] installs Python extras; full Tier 1 needs Semgrep, SkillSpector, and Gitleaks,
and Tier 3 needs an agent runtime and sandbox. Follow the
installation guide,
then check readiness:
skillevaluator doctor --env-mode docker
For a keyless check without external scanners, run
skillevaluator quality-check ./my-skill. If the command is unavailable,
run uv tool update-shell and reopen your terminal.
OpenAI uses two variables and defaults to Codex for Tier 3:
export SKILL_EVAL_LLM_PROVIDER=openai
export OPENAI_API_KEY='sk-...'
NVIDIA Build defaults to nvidia/nemotron-3-super-120b-a12b for evaluator chat,
Tier 3 agent execution, and judging. Its embedding default is
nvidia/nemotron-3-embed-1b; OpenAI defaults to gpt-6.1-sol and
text-embedding-3-small. These are configured defaults, not automatically
updated selections of the latest models. To select a different evaluator model:
export SKILL_EVAL_LLM_MODEL='your-provider-model-id'
The standard judge inherits that model unless overridden. Tier 3 defaults to
OpenCode for NVIDIA Build, Codex for OpenAI, and Claude Code for Anthropic.
Use --agents and --agent-model to override the agent and its model separately.
Anthropic and Bedrock need a separate embedding provider for Tier 2. Custom OpenAI-compatible endpoints also require explicit endpoint and model settings. See Providers & Credentials for these setups and advanced overrides.
validate runs Tier 1, Tier 2, and Tier 3 with autopilot by default for skills:
skillevaluator validate ./my-skill
--full remains supported but is unnecessary.
Use --tiers 1,2 or --no-tier3 to skip live evaluation; --tiers 1 runs
only the static suite. --no-autopilot requires an existing evaluation source.
Rules, workflows, and plugins retain their applicable validation stages without
automatically enabling skill evaluation.
Generate and review broader coverage; templates omit negatives unless authored in EVAL.md:
skillevaluator tier3 create-eval-dataset ./my-skill --full
Tier 1 always gates validate. Tier 2 gates by default;
--no-block-on-dedup makes its findings advisory. Tier 3 is advisory by default;
--block-on-agent-eval makes its findings gate too. Live model calls and managed
sandboxes can incur charges; start with one agent and a small dataset. See the
Tier 3 guide
for runtime and cost details.
git clone https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator
uv sync --python 3.13 --extra all
uv run --extra all skillevaluator tier1 ./my-skill
Use the provider exports above. Prefix other commands with
uv run --extra all to select this checkout.
Resources: paper, PDF, DOI, BibTeX, and the NVIDIA Developer blog. Cite the paper for methodology and results. For reproducibility, record the SkillEvaluator release or commit. See CITATION.cff.
See the documentation for setup, tier guides, reports, CI integration, and the CLI reference.
Follow the installation guide to choose the full installation or a smaller per-tier setup.
This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.
Contributions are welcome. Read CONTRIBUTING.md, include tests for behavior changes, and run the checks before opening a pull request:
make lint && make test && make build
Project governance is described in GOVERNANCE.md. Participation is governed by the Code of Conduct.
Support level: Experimental. SkillEvaluator is community-supported on a best-effort basis with no SLA or NVIDIA enterprise support entitlement. Report reproducible bugs and feature requests through GitHub Issues; see SUPPORT.md for details.
Report suspected vulnerabilities using the private process in SECURITY.md. Do not disclose security issues in a public GitHub issue.
Release changes are recorded in CHANGELOG.md and GitHub Releases.
Apache License 2.0 — see LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md.
Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.