name: codex-autoresearch description: "Run autonomous, measurable experiments in a Git repository: change one hypothesis, verify a numeric metric, keep improvements, and revert failures. Use when the user wants Codex to keep iterating toward a numeric target in the foreground or as a detached background run. Do not use for ordinary one-shot coding, open-ended work without a mechanical metric, or non-Git directories." metadata: short-description: "Run measurable autonomous experiments"

Codex Autoresearch

Turn a repo-level goal into a controlled loop:

inspect -> change one thing -> verify -> keep or revert -> repeat

Codex supplies the engineering judgment. The bundled control script supplies strict Git boundaries, measurement, rollback, state, and logs.

Load

• Read references/workflow.md for every invocation, including status, history, report, stop, and resume.
• Read references/experiment.md before starting or continuing an active run.
• Read references/background.md only for a background run.

Resolve commands from this skill's own directory as <skill-root>/scripts/autoresearch.py. Never assume the target repository contains the script.

Before Starting

1. Require one Git repository root. If the task spans repositories, ask the user to choose one run per repository.

2. Check for autoresearch-results/run.json with:

   python3 <skill-root>/scripts/autoresearch.py status --repo <repo>
   

   not_initialized is fresh. Any other status or schema error must be surfaced; do not infer state from other files.

3. For a fresh run, inspect the repo and propose:

   • one plain-language goal,
   • repository-relative file or directory scopes (no globs),
   • one numeric metric and whether lower or higher is better,
   • a command whose final non-empty stdout line is that number, or a JSON object plus one explicit key,
   • a numeric target,
   • an optional baseline-passing guard command,
   • foreground or background,
   • an optional iteration limit.

4. Run candidate measurement commands read-only if needed, then show one concise confirmation. Include the baseline, target, scope, commands, mode, and the fact that each trial is committed and failed trials are reverted.

5. Do not write project files, initialize artifacts, create a Goal, or launch a controller before clear user approval such as go.

Start

After approval, use the exact confirmed values.

Foreground

Initialize once:

python3 <skill-root>/scripts/autoresearch.py init \
  --repo <repo> --goal <goal> --scope <path> \
  --metric-name <name> --direction <lower|higher> \
  --verify <command> [--metric-key <key>] --target <number> \
  [--guard <command>] [--max-iterations <n>]

Then call get_goal. Reuse a matching unfinished Goal, otherwise call create_goal. The Goal objective must identify this as codex-autoresearch, include the returned run id, metric and target, and say to continue the validated experiment loop until terminal status. If a different unfinished Goal exists, stop and explain the conflict. Official Codex Goal continuation owns foreground persistence; this skill does not install hooks or modify Codex configuration.

If Goal tools are unavailable, do not claim the foreground run can continue autonomously across turns. Explain that the installed Codex does not expose the required Goal capability.

Background

Launch once with the same configuration:

python3 <skill-root>/scripts/autoresearch.py launch \
  --repo <repo> --goal <goal> --scope <path> \
  --metric-name <name> --direction <lower|higher> \
  --verify <command> [--metric-key <key>] --target <number> \
  [--guard <command>] [--max-iterations <n>] \
  --execution-policy <danger-full-access|workspace-write>

Background defaults to danger-full-access; show this in the confirmation. Use workspace-write only when the user explicitly prefers the sandbox and accepts that Git operations may be restricted. Do not create a Codex Goal for background runs.

After a successful launch, report the run id, baseline, controller PID, results path, and status command. Do not poll unless asked.

Experiment Loop

For each foreground iteration:

1. Read validated status and recent events.

2. Inspect evidence and choose one focused hypothesis that differs from discarded attempts.

3. Modify only confirmed scopes. Do not manually commit, revert, or edit autoresearch-results/.

4. Finalize exactly once:

   python3 <skill-root>/scripts/autoresearch.py finish \
     --repo <repo> --description <short-description>
   

finish checks scope and Git provenance, creates the trial commit, runs the metric and guard, keeps an improvement, reverts a failed trial, appends the audit event, and marks the run complete when the target is reached.

Continue immediately while status is active. On complete, verify status, call update_goal(status="complete"), and summarize the baseline, final metric, iterations, and retained commits.

Use block only when progress truly requires external input or an environment change, and only after the same blocker has prevented progress on three consecutive Goal turns:

python3 <skill-root>/scripts/autoresearch.py block --repo <repo> --reason <reason>

Then call update_goal(status="blocked"). A failed hypothesis, difficult bug, or lack of immediate improvement is not a blocker.

Existing Runs

• History request: run history --repo <repo>; use --format tsv only for tabular export.
• HTML report request: run report --repo <repo> and return its generated path. Both views validate the complete event history; neither is runtime state.
• Same foreground goal: validate status, resume the matching official Goal, and continue.
• Background status, stop, or resume: use the corresponding script command. Resume requires a user note or new direction.
• Different goal: show the current run. Stop a live background run first; for an active foreground run, ask the user to clear its official Goal with /goal clear. Then ask before archive and initialize the fresh run.
• complete: never resume it. Archive before a new goal.
• Invalid JSON, unknown schema, event gap, Git mismatch, stale controller, out-of-scope change, or malformed metric output: stop and report the exact error and log path. Never reconstruct, guess, or silently repair state.
• A failed initialization may leave init-error.json and command logs but no run.json. Report the diagnostic and use explicit archive before retrying; do not treat it as a fresh run.

Invariants

1. Ask before the first write or launch.
2. Require a clean named Git branch at initialization.
3. Keep one authoritative configuration in run.json and one append-only state history in events.jsonl.
4. Use one numeric metric and one target. A guard is pass/fail and must pass at baseline.
5. One focused experiment per finish; one repository per run.
6. Never stage autoresearch artifacts or touch paths outside confirmed scope.
7. Verification commands must exit zero, emit UTF-8, and use an explicit scalar or JSON-key parser. Parsing or command errors stop the run.
8. Never hide failures with fallback parsing, old-layout recovery, or synthetic success.
9. Never ask "should I continue?" after launch. Continue until target, user stop, iteration limit, or a verified external blocker.
10. Preserve command output and controller events under autoresearch-results/ for diagnosis.

Tell Codex what measurable result you want. Codex inspects the repository, confirms the experiment with you, changes one thing, verifies it, keeps improvements, reverts failures, and repeats until the target is reached.

Autoresearch works for test failures, coverage, type errors, warnings, latency, binary size, reproducible security findings, and any other outcome a command can measure.

Quick Start

Install from Codex:

$skill-installer install https://github.com/leo-lilinxiao/codex-autoresearch

Open a clean Git repository with Full Access:

codex --dangerously-bypass-approvals-and-sandbox

Then invoke the skill:

You:   $codex-autoresearch
       Reduce `python3 scripts/score.py` error_count to 0.

Codex: Baseline: 5
       Target: 0 (lower is better)
       Scope: src/
       Verify: python3 scripts/score.py, JSON key error_count
       Guard: python3 -m pytest -q
       Run in foreground or background?

You:   Background. Go.

Codex launches the confirmed run. No Codex configuration changes or special prompt syntax are required.

See Installation for manual and development installs.

The Loop

inspect evidence
      |
change one focused thing
      |
commit and measure
      |
      +-- improved + guard passes --> keep
      |
      +-- otherwise ---------------> revert
      |
append an audit event
      |
repeat until target

The control script owns commits, verification, rollback, and state. Codex owns the hypotheses and code changes.

Foreground And Background

Foreground and background use the same experiment rules. A run uses one mode at a time. Foreground continuation uses a Codex Goal; background continuation belongs to the detached controller.

What Gets Confirmed

Before the first write, Codex shows:

• the goal and numeric target;
• repository-relative paths it may change;
• the metric command and explicit parser;
• an optional regression guard;
• foreground or background mode;
• an optional iteration limit.

Initialization requires a clean named Git branch. One run manages one repository.

Results

Run artifacts live in autoresearch-results/ and stay uncommitted:

events.jsonl is the state history. Missing, malformed, contradictory, or partial state is an error; the skill never guesses a result from old files or conversational memory.

Review Results

Ask the skill to show the validated experiment history:

$codex-autoresearch show experiment history

Codex Autoresearch
Run: 0a516883  Status: complete  Mode: foreground
Metric: error_count  2 -> 0  Target: 0 (lower is better)

SEQ  ITER  EVENT     PREVIOUS  TRIAL  RETAINED  DESCRIPTION
---  ----  --------  --------  -----  --------  ------------------------------------
  0     0  baseline         -      -         2  Initial measurement
  1     1  discard          2      3         2  Broaden parser fallback
  2     2  keep             2      1         1  Fix nested parser branch
  3     3  keep             1      0         0  Remove final parser error
  4     3  complete         -      -         0  retained metric satisfies the target

The same validated events can be exported as TSV or rendered as a self-contained static report:

$codex-autoresearch export experiment history as TSV
$codex-autoresearch generate an HTML report

The report is written to autoresearch-results/report.html. It is a replaceable snapshot, not runtime state.

Safety Model

• Every trial is a Git commit.
• A non-improving trial or failed guard is reverted with git revert.
• Out-of-scope edits, branch changes, HEAD drift, malformed metrics, command failures, timeouts, and generated byproducts stop the run with an exact error and log path.
• Autoresearch artifacts are never staged.
• A run reports complete only when the retained metric reaches the confirmed target.
• A genuine external blocker is reported explicitly; a difficult or unsuccessful hypothesis is not treated as blocked.

This strictness is intentional. Silent recovery makes long autonomous runs impossible to trust.

Good Metrics

The verify command must exit successfully and place one finite number on its final non-empty stdout line. It may instead print a JSON object on that line when Codex names one numeric key explicitly.

7

{"error_count": 7, "passed": 12}

Use a guard for behavior the metric does not protect, such as a test suite around a latency benchmark. The guard must pass at baseline.

Documentation

FAQ

Does installation change my Codex settings?

No. Installation copies the skill files. Use a current Codex release so foreground runs can use the built-in Goal capability.

Why Full Access?

Each iteration creates or reverts a Git commit. Restricted sandboxes may block writes under .git. Background runs therefore default to Full Access; workspace-write remains an explicit option when its limitations are acceptable.

Can I stop and resume?

Yes. Interrupt or pause a foreground Goal. For background, invoke $codex-autoresearch and ask for status, stop, or resume with a new direction.

Can it run without Git or across several repos?

No. Git is the experiment memory and rollback boundary. Use one run per repository so commit ownership and metrics remain unambiguous.

Is this only for small changes?

No. One experiment should test one coherent hypothesis. Its size should match the hypothesis, while still being independently measurable and reversible.

Acknowledgments

Inspired by Karpathy's autoresearch, generalized for Codex and software repositories.

Citation

@misc{codex-autoresearch,
  author = {Li, Linxiao},
  title = {Codex Autoresearch: Autonomous Goal-Driven Experimentation for Codex},
  year = {2026},
  publisher = {GitHub},
  url = {https://github.com/leo-lilinxiao/codex-autoresearch}
}

GitHub also reads CITATION.cff for its Cite this repository menu.

Star History

License

MIT, see LICENSE.