ResearchClawBench
End-to-end autonomous scientific research benchmark: 40 tasks across 10 scientific domains where agents must rediscover a real published paper's findings from raw data and literature, graded by expert multimodal rubrics.
Overview
ResearchClawBench evaluates whether AI agents can conduct genuine end-to-end scientific research — not just answer questions about papers, but independently reproduce the research process. Each of its 40 tasks (spanning 10 scientific domains) is grounded in an actual published paper: the agent is given related literature and raw data while the target paper itself is hidden, and it must work through the investigation to arrive at the paper's core findings, with room for the rubric to credit genuinely novel discoveries as well.
Built by a large multi-institution team of 51 researchers led by Wanghan Xu, the benchmark's central innovation is its evaluation methodology: expert-curated, multimodal rubrics decompose each target paper's scientific artifacts (figures, tables, claims, methods) into weighted criteria, scored via a lightweight harness called ResearchHarness. This lets the benchmark distinguish partial scientific progress from full rediscovery, and diagnose specific failure modes rather than issuing a single pass/fail score.
Results show current systems are far from reliable research agents: the strongest autonomous coding agent (Claude Code) averaged only 21.5 out of 100 on the rubric, the strongest harness-driven LLM (Claude Opus 4.7) averaged 20.7, and the frontier mean across models was just 26.5. Error analysis found failures concentrated in experimental-protocol mismatches, evidence mismatches, and missing core scientific concepts — indicating agents struggle most with the scientific reasoning chain itself, not just execution.
How It Works
Task setup, inputs, outputs, and evaluation.
Results
Model performance on this evaluation.
| # | Model | Score |
|---|---|---|
| 1 | Claude Opus 4.7 | 20.7/100 |
Key Findings
Strongest autonomous agent (Claude Code) averages only 21.5/100; frontier mean across models is just 26.5/100.
Failures concentrate in experimental-protocol mismatches, evidence mismatches, and missing core scientific concepts, not execution/tooling errors.
Current systems remain far from reliable target-paper-level rediscovery of real published research.
What Makes It Unique
- ✓
Grounds every task in a real published paper's actual data and literature, hiding the target paper to test genuine rediscovery rather than paraphrase.
- ✓
Uses weighted multimodal rubrics (figures, tables, claims, methods) rather than binary pass/fail, allowing credit for partial or novel findings.
- ✓
Explicitly diagnoses failure categories (protocol mismatch, evidence mismatch, missing concepts) rather than reporting a single aggregate score.
Controversies & Caveats
Grading relies on expert-curated rubrics that require significant domain expertise to construct and may embed subjective weighting choices.
Small sample size (40 tasks) across 10 domains means per-domain results carry high variance.