ScienceClaw

ScienceClaw is an agent system for scientific work that gets better the more it is used. It solves each task as a typed, executable workflow. When a repair is reproduced under a clean replay, it becomes a linked Skill (strategy) and Operator (a typed, executable capability), and it is kept only if it still solves its source task and improves independent validation tasks.

[!NOTE] This repository is the agent system, built on the OpenClaw gateway. Its companion benchmark, ScienceClaw-Eval, keeps its evaluation data on Hugging Face.

✨ Highlights

🚀 Quick start

git clone https://github.com/beita6969/ScienceClaw.git
cd ScienceClaw

# One-click setup: Node, Python, the engine, its tools and weights, MCP servers, skills
chmod +x setup.sh && ./setup.sh

pnpm install && npx openclaw onboard
pip install -e packages/scienceclaw          # Python >= 3.11; extras: [all], vision, audio, nlp, forecast, materials, retrieval
python -m scienceclaw.cli setup              # install and verify every tool and pretrained weight (--profile light skips assets > 1.5 GB)
python -m scienceclaw.cli doctor             # which tool modules can run on this machine, and why not

The engine refuses tasks until setup has completed (SCIENCECLAW_SKIP_SETUP_CHECK=1 is for development only), and the full profile downloads about 19 GB of weights. setup.sh runs it for you; SCIENCECLAW_SKIP_TOOLS=1 defers it to first use.

[!TIP] Bring your own model. The engine never assumes a model family and this repository names none. Point it at any OpenAI-compatible endpoint, a local program, or your own backend:

export SCIENCECLAW_API_BASE_URL=<your OpenAI-compatible endpoint>
export SCIENCECLAW_API_KEY=<your key>
export SCIENCECLAW_MODEL=<your model name>      # or SCIENCECLAW_<POLICY|EXECUTOR|PATCH>_MODEL per role

llm.backend can also be command (a local program reads the prompt and writes the completion) or package.module:factory. See configs/default.yaml.

🧠 How it works

The problem

LLM agents increasingly solve scientific tasks by connecting reasoning to data, domain tools and executable code. But a repair that works once rarely survives: it lives in a transient context, or in a single tool or prompt, so verified executions seldom become persistent improvements. Existing work evolves individual tools or Skills, and existing benchmarks treat tasks as independent episodes, so it has been hard to tell whether verified scientific executions turn into persistent, transferable improvements.

How can verified scientific executions drive persistent and transferable program-level self-evolution without updating foundation-model parameters?

The answer: replay-gated, linked Skill–Operator evolution

1. A task for program-level self-evolution. The model Θ₀ is fixed. What evolves is an editable program A_r = (Skills, Operators). A task D_t = (D_T, D_V, D_E) specifies its objective and inputs, its evaluation protocol and hard constraints (units, feasibility, convergence, reproducibility), and its data, tools and environment.

2. Typed, execution-guided workflows. A solution is a directed graph whose ports carry a schema (type, shape, unit, provenance); an edge is valid only if types and shapes match and any unit conversion is explicit and recorded. The agent edits the graph one atomic action at a time, and the executor checkpoints every node by fingerprint and reruns only the affected descendants, so a long scientific workflow is repaired rather than regenerated.

3. Evidence you can replay. Every candidate result is regenerated from a reset environment and checked against the task's evaluator and hard constraints. A replay that fails and a later replay that passes bracket the shortest reproduced repair, e⁻ → e⁺. Replay alone never authorizes persistence.

4. Linked Skill–Operator candidates, with no LLM judge. The reproduced repair is split by edit type. Control edits (topology, routing, configuration) become a Skill patch. Generated or repaired executable nodes are grouped into convex components and abstracted into typed Operators, each checked by boundary replay in isolation. Both are committed as one atomic bundle, so a strategy never arrives without the capability behind it, nor a capability without a strategy that selects it.

5. A replay-and-validation gate. A candidate persists only if source replay reproduces the repair and actually uses every new component (R_src = Pass ∧ Use), every hard integrity and scientific check holds on independent validation tasks, its cost stays within budget, and it strictly improves the validation score over the incumbent. A noise guard requires at least two improved and at most one regressed validation episode. Otherwise the incumbent program is kept.

📊 Results

Seven evolution rounds over a common stream of 23 disciplines, with 64 IID and 64 OOD instances per discipline, one fixed foundation model, and the same tools, source stream, validation data and update budget throughout. OOD means an independently sourced dataset of the same discipline; OOD results never generate or select updates.

• Consistent gains. ScienceClaw improves on the frozen agent in every discipline: +16.45 % on average out-of-distribution (+11.57 % to +23.34 %) and +12.25 % in-distribution. Its OOD score falls 11.73 % below its IID score on average, against 16.06 % for the frozen agent.
• It keeps improving. The OOD macro success rate rises from 77.72 to 91.30 over seven rounds (+13.59 pp). 70 of 161 subm