by relai-ai
Build evals from traces, code, and coding sessions, then measure and improve AI agent behavior, including your coding agents
# Add to your Claude Code skills
git clone https://github.com/relai-ai/agent-eval-loopGuides for using ai agents skills like agent-eval-loop.
See how agent-eval-loop compares with popular alternatives.
agent-eval-loop is an open-source ai agents skill for AI coding assistants such as Claude Code, Codex CLI, and ChatGPT, built by relai-ai. Build evals from traces, code, and coding sessions, then measure and improve AI agent behavior, including your coding agents. It has 50 GitHub stars.
agent-eval-loop's catalog security scan is still queued. You can run an instant dependency and prompt-injection check now with the "Scan for vulnerabilities" button above.
Clone the repository with "git clone https://github.com/relai-ai/agent-eval-loop" and add it to your Claude Code skills directory (see the Installation section above).
agent-eval-loop is primarily written in Python. It is open-source under relai-ai on GitHub, so you can review or fork the full source.
Yes. SkillsLLM lists many other AI Agents skills you can browse and compare side by side. Open the AI Agents category from the badge at the top of this page, or use the Related Skills and comparison links further down to weigh agent-eval-loop against similar tools.
No comments yet. Be the first to share your thoughts!
Available after the catalog security scan passes.
⚠️ Third-Party Software Notice
This skill is third-party open-source software developed and hosted independently on GitHub. SkillsLLM lists this project for discovery and does not control or maintain the underlying repository.
Any security checks, ratings, or warnings displayed by SkillsLLM are automated and limited in scope. They do not constitute a security certification or guarantee that the software is safe, error-free, or free from malicious code, vulnerabilities, compromised dependencies, or prompt-injection risks.
Review the source code, permissions, dependencies, and configuration before installing or running any third-party skill. Use is at your own risk. To the maximum extent permitted by applicable law, SkillsLLM is not liable for losses arising from third-party software.
The deep catalog scan for this skill is still queued. Run an instant dependency check now instead.
RELAI helps you build evals, measure agent behavior and compare improvements. It runs through skills in your coding agent, with no separate RELAI service. You keep the tests, review tools and results in your own project. Each skill works independently; start where your question needs it.
https://github.com/user-attachments/assets/c8002907-e339-4679-9328-246a66a2bea1
For Claude Code, Codex, OpenHands or your own setup. Build tests from your team's work to measure whether a model, instruction file or skill improves results on your repositories:
1. Turn past work into evals
Use RELAI's coding-evals skill to build 20 evals from the last six months of
merged pull requests and reverts in {repository}
This creates tasks that start before the original fix, include the repository and its toolchain, and score the evaluated agent using hidden tests and reviewed checks. Any coding agent can run them. You can measure solved rate and cost per solved task, then compare changes to your setup.
Continue from the task-building request above:
2. Evaluate your coding agent
Use RELAI's coding-evals skill to evaluate Claude Code on these evals and report
the solved rate and cost per solved task
This reports the agent's results on your tasks, with uncertainty and cost per solved task.
3. Improve your setup before you roll it out
Use RELAI's improve skill to improve our CLAUDE.md and skills against these evals
You get a branch with the improved setup, the solved rate with its noise before and after, cost per solved task, and the evals where the two setups differ.
You can start at any step without completing the full workflow.
Task sources, grading safeguards and eval-project setup.
https://github.com/user-attachments/assets/53ce7659-21f8-4f42-897e-a5e3cc4d1ffa
For agents you build and deploy, on any framework. Open your agent's repository and ask:
1. Discover where it can fail
Use RELAI's discover skill to read this agent's code, prompts and the traces
in ./traces, and tell me where it is likely to fail
This builds an EvalsWiki in relai/wiki/: requirements, ranked risks with supporting evidence,
and questions about intended behavior. Traces are optional. From there, RELAI can build tests,
record a baseline and compare changes on a branch, with a record of what it tried and rejected.
Continue from the discovery request above:
2. Turn the risks into evals
Use RELAI's eval skill to build tests for the top five risks and get a baseline
You get tests that run the real agent end to end, with graders you preview before they are frozen, and a baseline score for each risk.
3. Improve without regressions
Use RELAI's improve skill to fix the failing tests without breaking the passing ones.
You may change prompts and tools, not the model
You get a branch with the change, before-and-after scores on the same frozen tests, and a record of what was tried and rejected.
Continue as new traces arrive:
Use RELAI's loop skill to keep improving this agent from ./traces;
optimize task success first, then cost
Explore discovery and runner options, review and annotate traces, or bring an existing benchmark.
Paste this into any coding agent. It picks the install method that fits that agent:
Install the RELAI plugin from https://github.com/relai-ai/agent-eval-loop for this
coding agent, following docs/installation.md in that repository. Tell me when to
start a new session
Or run the commands yourself:
claude plugin marketplace add relai-ai/agent-eval-loop
claude plugin install relai@agent-eval-loop
codex plugin marketplace add relai-ai/agent-eval-loop
codex plugin add relai@agent-eval-loop
Start a new session after installing. Codex asks you to review and trust the optional hooks before they run. The hooks surface frozen-test edits and checker results in RELAI projects.
All installation methods and requirements, including other agents, Skills CLI, manual installation, hooks and troubleshooting. Bundled checkers need Python 3.11+; Harbor execution needs uv and a supported container engine. Native runners use their own prerequisites. RELAI checks what the current task needs and reuses suitable project tooling.
Update RELAI or uninstall it using the method you installed with. Plugin updates do not replace project-owned files or frozen tasks; preserve local adaptations using the linked project-copy policy.
relai/STATUS.md records progress, pending decisions and spend when
managed state is useful.Results describe the evaluated scenarios, with no guarantee against every regression. Understand evaluation, review and the limits of the results.
Before an eval scores your agent, RELAI checks that it tests the right thing and grades fairly. You approve the plan and grading, or delegate that to an independent reviewer.
Not sure where to begin? Ask: "Use RELAI's start skill to look at this agent and suggest a first step." It reads your repository and recommends one, using your coding agent's usage without running the agent under test or making additional model calls.
| Skill | When to use it |
|---|---|
start |
Find the right starting point; initialize relai/ only when needed. |
discover |
Understand requirements, risks and open questions from code, docs or traces. |
trace-review |
Build or adapt a project-owned app for reading traces and saving source-linked feedback. |
eval |
Build and review tests, import a benchmark, or establish a baseline. |
improve |
Invest |