ResearchClawBench

Overview

✨ Highlights

🎬 Demo

https://github.com/user-attachments/assets/94829265-80a8-4d61-a744-3800603de6d9

💡 Why ResearchClawBench?

Most AI benchmarks evaluate what models know. We evaluate what agents can do.

• Real science, not toy problems. 40 tasks sourced from published papers across 10 disciplines, each with curated experimental datasets.
• Two-stage pipeline. Autonomous research first, rigorous evaluation second — just like peer review.
• Fine-grained, multimodal scoring. A weighted checklist with text and image criteria, judged by an LLM acting as a strict peer reviewer.
• Agent-agnostic. Ships with built-in support for Claude Code, Codex CLI, ARIS Codex, OpenClaw, Nanobot, EvoScientist, ResearchClaw, and a lightweight ResearchHarness baseline. Bring your own agent in one line.
• From Re-Discovery to New-Discovery. Scoring above 50 means matching the original paper; above 70 means surpassing it. The frontier is wide open.

📢 News

• 2026-04-13 🧭 Added built-in ARIS Codex UI support and documentation. Imported benchmark runs are supported in the leaderboard and run browser; one-click launch is not yet supported in the public preset.
• 2026-04-10 🔬 Added built-in ResearchClaw support — an intelligent agent-powered research assistant with built-in skills for paper search, literature review, and data analysis.
• 2026-04-07 🧪 Added built-in ResearchHarness support as a lightweight baseline agent for testing different LLMs under the same ResearchClawBench workflow.
• 2026-03-30 🧬 Added built-in EvoScientist support and clarified multimodal judge prompting so the first attached image is explicitly treated as the ground-truth figure.
• 2026-03-27 🤗 Released a Hugging Face dataset mirror at InternScience/ResearchClawBench, including 10 additional tasks from ResearchClawBench-Self and a task downloader script.
• 2026-03-27 📨 Opened the ResearchClawBench submission Space for community task uploads. New tasks are validated there and reviewed through Hugging Face dataset PRs instead of being added to this GitHub repository.
• 2026-03-20 🐈 Added Nanobot as a new agent — ultra-lightweight OpenClaw alternative with reliable multi-step tool execution. Agent config moved to agents.json for easy customization.
• 2026-03-19 🚀 Initial release with Claude Code, Codex CLI, and OpenClaw support. 40 tasks across 10 scientific domains.

Understanding The Benchmark

🏗️ Data Construction

Every task in ResearchClawBench is built through a rigorous, expert-driven pipeline to ensure scientific validity and reproducibility:

flowchart TD
    A["📄 High-Quality Paper Collection\n(Target Paper)"] --> B["🧑‍🔬 Human Expert Extraction\n(Core Task Instructions)"]
    B --> C["📋 Evaluation Checklist\n(Criteria + Keywords + Weights)"]
    B --> D["📂 Data & Related Work Collection\n(Datasets + Reference Papers)"]
    C --> E["✅ Human Reproduction & Validation\n(Verify checklist is reproducible)"]
    D --> E

    style A fill:#e0f2fe,stroke:#0284c7,stroke-width:2px
    style B fill:#fef3c7,stroke:#f59e0b,stroke-width:2px
    style C fill:#fce7f3,stroke:#ec4899,stroke-width:2px
    style D fill:#f0fdf4,stroke:#22c55e,stroke-width:2px
    style E fill:#f5f3ff,stroke:#8b5cf6,stroke-width:2px

1. High-Quality Paper Collection — Domain experts select recent, high-impact publications with clear methodology and reproducible results across 10 scientific disciplines.

2. Expert Task Extraction — Human experts read each paper and distill the core research task into structured instructions, identifying the key scientific question, input data, and expected outputs.

3. Checklist Design — Experts create a fine-grained evaluation checklist with weighted criteria (text and image items), each with specific technical keywords that a judge must verify.

4. Data & Related Work Collection — The datasets and related reference materials are curated to form a research workspace for the task.

5. Human Reproduction & Validation — Human researchers independently reproduce the paper's results from the provided workspace and instructions, verifying that every checklist item is achievable. This ensures the benchmark is fair and the checklist is grounded in reality.

⚙️ How It Works

ResearchClawBench operates in two distinct stages:

flowchart LR
    subgraph Stage1["Stage 1 — Auto Research"]
        A["Raw Data\n+ Instructions"] --> B["AI Agent\n(autonomous)"]
        B --> C["Code\n+ Figures\n+ Report"]
    end

    subgraph Stage2["Stage 2 — Evaluation"]
        C --> D["LLM Judge"]
        E["Target Paper\n+ Checklist"] --> D
        D --> F["Per-Item Scores\n+ Reasoning"]
    end

    style Stage1 fill:#f0f4ff,stroke:#3b82f6,stroke-width:2px
    style Stage2 fill:#fff7ed,stroke:#f59e0b,stroke-width:2px

Stage 1: Autonomous Research

The AI agent receives a workspace containing raw datasets, reference materials, and task instructions. It must independently:

1. Explore the data and understand the research question
2. Write code to analyze, model, and visualize the data
3. Produce a research report (report/report.md) with figures, methodology, results, and discussion

No hand-holding. No chain-of-thought hints. The agent works in its own sandboxed workspace with full tool access — just like a real researcher.

Stage 2: Reference-Based Evaluation

Once the agent finishes, its report is evaluated against the original published paper using a fine-grained checklist. The judge receives the task instructions, the AI report, and the checklist criteria — then scores each item using a dual-mode rubric:

flowchart TD
    subgraph Inputs
        I["INSTRUCTIONS.md\n(task background)"]
        R["Agent Report\n(text + figures)"]
        CL["Checklist\n(from target paper)"]
    end

    I & R & CL --> J["Multimodal LLM Judge"]

    J --> DET{"Determine\nEvaluation Mode"}

    DET -->|"Quantitative\nresults"| OBJ["Mode A: Objective\n(Metric Optimization)"]
    DET -->|"Qualitative\nreasoning"| SUB["Mode B: Subjective\n(Mechanism Analysis)"]

    OBJ --> SO["Score by m