labrat is a local-first runtime that puts Claude Code or Codex on a real research problem with a scoreboard and enough structure to run for hours. Population search, not single-thread: families of ideas compete for compute budget, and the ones that produce real signal earn more room to keep going.
Live example run: the baseline still leads on the main selection metric, while classifier_search has already won two decisive held-out challenges and earned extra funding.
labrat treats Claude Code and Codex as peer operator interfaces. Stronger reasoning models still help most on synthesis, audit, and consolidation, but the runtime contract and file layout stay the same across both.
The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
Supervisor + worker model: the agent supervises the runtime, while probe / mutation / crossover / audit workers execute bounded tasks.
File-as-Bus workspace: durable files and append-only logs carry state forward so the supervisor can keep thin control over thick project state.
Decisive challenges: a family earns extra status when it wins a held-out challenge that was not already baked into the local search metric.
This means a family can become strategically important even before it becomes the global selection champion. The dashboard now shows both the current champion and the current decisive-challenge leaders.
Good first problems
labrat works best when the problem has:
a clear baseline
a bounded experiment runner
a metric you can score consistently
at least one harder held-out challenge beyond the main hill-climb metric
Good examples:
tune a small classifier or ranking model
iterate on prompt + rubric combinations with fixed evaluation
search over retrieval / reranking strategies
compare workflow variants where one family should win a specific hard slice, not just the average score
python -m venv .venv
source .venv/bin/activate
pip install -e '.[nlp-sentiment]'
labrat doctor --lab-dir examples/nlp-sentiment/research_lab
labrat bootstrap --lab-dir examples/nlp-sentiment/research_lab
python -m http.server 8787 --directory examples/nlp-sentiment/research_lab
labrat status --lab-dir examples/nlp-sentiment/research_lab
labrat next-prompt --lab-dir examples/nlp-sentiment/research_lab --runner claude --phase auto
Use --runner codex for Codex.
The editable install is intentional: the labrat CLI keeps using the templates, profiles, and scripts from this checkout. If you prefer the original in-lab workflow, the copied scripts/*.py entrypoints still work unchanged inside each lab.
What you get from the example:
a running dashboard
a fully scaffolded lab
held-out decisive challenges on top of search / selection metrics
a reference supervisor + worker flow that you can copy into a new lab
Agent interfaces
labrat does not depend on hidden local skills, private prompts, or machine-specific setup. The operator contract ships in the repo and in every generated lab:
repo root: AGENTS.md for Codex, CLAUDE.md for Claude Code
each lab: AGENTS.md, CLAUDE.md, .claude/commands/, and agent_prompts/
shared runtime surface: labrat ... from the repo root or python scripts/... inside a lab
That means a user can clone the repo, open either Codex or Claude Code, and operate the lab from files that are already present in version control. There is no required SKILLS.md convention to make the repo work.
Start from a profile
If you already know the shape of your research problem, a profile scaffolds a runnable lab in one command. No Phase 0 hand-editing, no LABRAT_PLACEHOLDER stubs.
labrat new ~/labs/my_search --profile=transformer-arch
cd ~/labs/my_search
python -m pip install -r requirements.txt
labrat doctor --lab-dir .
labrat check-readiness --lab-dir .
labrat bootstrap --lab-dir .
Operator surfaces
Every lab, whether profile-scaffolded or hand-built, ships both primary operator surfaces:
AGENTS.md for Codex
CLAUDE.md plus .claude/commands/ for Claude Code
agent_prompts/ for the shared phase prompts consumed by either interface
The Claude Code slash commands are short markdown files that wrap common operator actions so you do not have to remember the CLI invocations:
/next — print the prompt for the current phase and execute it.
/why-stuck — diagnose a stalled frontier from state/frontier.json and recent evaluations.
/synthesize — summarise the last ~10 evaluations before dispatching more work.
/audit-candidate — walk the highest-signal suspicious candidate through the audit worker.
/frame-break — propose a structural pivot once cheap probes and audits are exhausted.
/consolidate — write a compact checkpoint note to logs/checkpoints/.
Open Claude Code in the lab directory and type /next, or hand-run python scripts/operator_helper.py next-prompt --runner claude --phase auto. In Codex, read AGENTS.md and run python scripts/operator_helper.py next-prompt --runner codex --phase auto.
Available profiles
transformer-arch — tiny character-level transformer architecture search with held-out-distribution decisive challenges. Ships a synthetic runner so you can exercise the full runtime loop without a training framework; replace scripts/run_experiment.py with your own trainer when you want real training.
More profiles (world-model, multi-dataset) land in follow-up PRs. See docs/PROFILES.md for the profile contract and docs/LONG_HORIZON.md for interim-checkpoint and long-running-job conventions.
Create your own lab from scratch
If no profile fits your problem, scaffold an empty lab and finish Phase 0 by hand. The default path is deep research first.
labrat new my_lab
cd my_lab
labrat doctor --lab-dir .
labrat next-prompt --lab-dir . --runner claude --phase design
labrat check-readiness --lab-dir .
labrat bootstrap --lab-dir .
python -m http.server 8787
labrat next-prompt --lab-dir . --runner claude --phase auto
Use --runner codex if you are operating from Codex instead of Claude Code.
Phase 0 must produce:
branches.yaml
dead_ends.md
research_brief.md
research_sources.md
evaluation.yaml
runtime.yaml
evaluation.yaml now includes at least one held-out prediction_tests challenge. That is how the runtime distinguishes “fit the known metric a bit better” from “this family actually predicted something hard.”
labrat comes out of DXRG. We first used variants of this runtime internally to explore different financial world-model architectures and adjacent research workflows, then published the parts that generalized cleanly beyond that domain.
References
labrat is its own system, but the current shape is informed by a few clear predecessors and adjacent designs:
karpathy/autoresearch: the minimal agent-driven autonomous experiment loop that helped establish the basic pattern of fixed evaluation plus overnight iteration.
AIRA_2: population search, stronger evaluation discipline, and stateful operator quality as first-class system levers.
A Claude Code plugin that automatically captures everything Claude does during your coding sessions, compresses it with AI (using Claude's agent-sdk), and injects relevant context back into future sessions.
