NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
NatureBench evaluates AI coding agents on 90 scientific tasks; strongest model surpasses SOTA in only 17.8%.
Key Findings
Methodology
This study develops NatureBench based on NatureGym pipeline, transforming scientific papers into standardized, containerized tasks through filtering, data verification, and packaging. The pipeline ensures reproducibility and fairness by automating environment construction and removing source methods, forcing models to discover solutions. Evaluation involves twelve models across disciplines, using a normalized SOTA gap metric and strict web-search restrictions. Results indicate models mainly rely on method translation, converting tasks into familiar supervised problems rather than genuine scientific invention. Failures are primarily due to incorrect method choices and limited compute budgets. The approach emphasizes discovery over reproduction, with automated scoring and environment encapsulation enhancing credibility.
Key Results
- Claude Opus 4.7 achieved surpassing SOTA in only 17.8% of tasks, matching in 47.8%. Success mainly stems from methodological translation (~45.5%), not innovation. Failures are due to method misselection (45.1%) and insufficient compute (24.4%).
- Analysis shows models convert scientific tasks into known prediction problems, with environment automation improving fairness. Results highlight that current models lack deep scientific creativity, emphasizing the need for cross-domain reasoning.
- Overall, the findings reveal AI agents excel at transfer learning but are limited in autonomous scientific discovery. Future work should focus on enhancing models' inventive capabilities and resource efficiency.
Significance
This benchmark provides a new standard for evaluating AI's scientific discovery potential, moving beyond mere reproduction. Automated environment construction and discovery-oriented metrics enable fairer, more meaningful assessments. The platform identifies current limitations—primarily reliance on transfer rather than innovation—guiding future research toward models capable of genuine scientific breakthroughs. It bridges the gap between AI development and real-world scientific progress, fostering tools that can autonomously generate novel knowledge, thus transforming AI from a tool into a collaborator in scientific research.
Technical Contribution
The paper introduces NatureGym, an automated pipeline that converts peer-reviewed papers into reproducible, containerized tasks, addressing environment fragmentation. It constructs a diverse set of 90 tasks across six scientific domains, with a SOTA-normalized evaluation metric. The evaluation protocol enforces environment encapsulation and automated scoring, ensuring fairness and reproducibility. The experimental setup reveals that success hinges on method transfer, not innovation, providing insights into current AI capabilities and limitations. This framework sets a new standard for scientific AI benchmarking.
Novelty
This work is the first to systematically build a cross-disciplinary, discovery-oriented benchmark derived directly from peer-reviewed scientific literature. It combines automated pipeline construction, environment encapsulation, and a novel evaluation metric focused on surpassing SOTA, emphasizing genuine scientific discovery rather than mere reproduction. Unlike prior benchmarks limited to code reproduction or engineering tasks, NatureBench promotes models that can innovate within real scientific problems, marking a significant step forward in AI for science.
Limitations
- Models primarily rely on method transfer, lacking autonomous invention, which limits true scientific discovery. The benchmark emphasizes transfer learning, not innovation.
- Computational costs remain high, constraining large-scale testing and iterative exploration. Limited compute budgets hinder full solution space coverage.
- Environment encapsulation, while improving fairness, may restrict adaptability to complex, multi-modal scientific tasks. Enhancing cross-domain reasoning remains a challenge.
Future Work
Future directions include developing models with stronger cross-disciplinary reasoning and inventive capabilities, integrating multi-modal data, and reducing computational costs. Improving environment flexibility to handle complex tasks and expanding the benchmark to more scientific fields will be key. Additionally, combining large pre-trained models with reinforcement learning could foster genuine scientific innovation, pushing AI closer to autonomous discovery.
AI Executive Summary
The pursuit of autonomous scientific discovery by AI has gained momentum, yet evaluating true innovation remains challenging. Traditional benchmarks focus on reproducing published methods, which does not fully capture a model’s ability to generate novel insights. To address this, we introduce NatureBench, a comprehensive, cross-disciplinary benchmark derived from peer-reviewed Nature articles. Using the NatureGym pipeline, each paper is transformed into a standardized, containerized task environment, ensuring reproducibility and fairness. This pipeline automates the extraction, verification, and packaging of datasets and evaluation metrics, with an emphasis on discovery rather than mere reproduction.
The benchmark spans six scientific domains—cellomics, protein biology, biomedical modeling, physical modeling, molecular design, and relational reasoning—covering 90 tasks. Evaluation employs a normalized SOTA gap metric, supplemented by an automated, environment-encapsulated scoring system that prevents environment fragmentation and bias. Experiments with twelve leading models reveal that the strongest, Claude Opus 4.7, surpasses SOTA in only 17.8% of tasks, with nearly half of successes attributable to method translation rather than genuine invention. Failures are mainly due to incorrect method choices and limited compute budgets.
These findings highlight that current AI models excel at transferring known methods but lack the capacity for autonomous scientific innovation. The study underscores the importance of developing models with deeper reasoning and creative abilities, capable of pushing the frontiers of science. The benchmark provides a vital new standard for assessing AI’s discovery potential, guiding future research toward models that can truly innovate within complex scientific problems. Ultimately, this work aims to accelerate AI-driven scientific breakthroughs, transforming AI from a mere tool into a genuine partner in discovery.
Deep Analysis
Background
近年来，AI在科学研究中的应用不断深化，从自动化数据分析到结构预测，代表性工作如AlphaFold、ProteinMPNN、AutoML等推动了模型在特定任务中的突破。然而，现有评估体系多集中于模型复制或工程优化，缺乏对模型自主创新能力的系统考察。随着AI逐步迈向自主科研，建立科学导向的评估平台成为迫切需求。此前的基准多偏重单一任务或环境碎片化，难以全面衡量模型在跨学科科学任务中的潜力。科学研究的复杂性和多样性要求模型不仅能迁移已有知识，更能自主探索未知领域。
Core Problem
核心问题在于如何设计一个跨学科、科学导向的评估体系，既保证任务的真实性，又能有效衡量模型的创新能力。传统方法多依赖人工重现，缺乏自动化和公平性。环境碎片化导致模型在不同任务间难以公平比较，且多模型偏重迁移技巧，缺乏深层次的科学创新验证。这些限制阻碍了AI在科学探索中的实际应用，亟需一个能真实反映模型创新能力的评估平台。
Innovation
本研究的创新点包括：1）开发NatureGym自动化管线，实现论文到任务的标准化转化，解决环境碎片化问题；2）构建涵盖六大科学领域的90个任务，强调科学发现而非复制；3）引入Discovery导向的评价指标，衡量模型超越已知SOTA的能力；4）采用严格的环境封装和自动评分，确保评估的公平性和可复现性。这些创新推动了科学AI评估体系的升级，为模型自主创新提供了新的激励机制。
Methodology
- �� 论文筛选：利用多级过滤确保任务的科学性和自动评估可行性。
- �� 数据验证：下载并验证数据完整性，确保与论文一致。
- �� 任务封装：构建容器化任务包，隔离原始方法，促使模型自主发现。
- �� 评估协议：采用SOTA归一化指标，结合自动评分和环境封装，确保公平。
- �� 模型评估：测试十余模型，分析成功路径和失败原因，重点关注方法迁移与创新差异。
Experiments
实验在多种科学任务上进行，包括细胞组学、蛋白质结构、医学建模等。使用公开数据集，设定不同计算预算，比较模型在超越SOTA、匹配和失败的表现。采用多轮验证确保结果的稳定性，同时进行消融分析，验证环境封装和指标的有效性。实验还评估模型在跨领域任务中的泛化能力，旨在揭示当前模型的潜力与局限。
Results
最强模型Claude Opus 4.7在90个任务中仅超越17.8%，在47.8%任务中达到SOTA水平。成功主要依赖方法转化，约占45.5%，而非自主创新。失败多因方法选择错误（45.1%）和计算资源不足（24.4%）。分析显示，模型偏重技术迁移，缺乏深层次的科学创新能力。环境封装和自动评估提升了评测的可信度，为未来改进提供方向。
Applications
该基准可用于评估未来科研AI模型的创新潜力，推动自动化科学探索。行业中，可辅助药物设计、材料发现、基因分析等领域的创新研究。学术界可借助此平台优化模型架构，提升自主发现能力，促进跨学科合作。未来，结合大模型和强化学习，有望实现更深层次的科学创新。
Limitations & Outlook
当前模型主要通过迁移实现成功，缺乏真正的自主创新。环境封装虽提升公平性，但对复杂任务的适应性仍有限。计算成本较高，限制了大规模应用。未来需增强模型的跨领域推理和创新能力，改善环境设计，降低成本，推动AI在科学中的深度应用。
Plain Language Accessible to non-experts
想象一个工厂里生产各种商品，工人们按照说明书操作，但说明书都是别人写的。现在，你希望工厂自己能想出新商品，而不是只照着说明书做。科学研究也是一样，科学家写论文，告诉别人怎么做，但真正的创新是自己想出新方法。这个研究就像让AI变成工厂的“聪明工人”，让它自己发现新方法，而不是只复制已有的。通过设计一套标准的“生产线”，让AI在没有告诉它具体怎么做的情况下，自己探索出新科学知识。结果显示，AI大部分时候只是把任务变成它熟悉的题型，真正的创新还不够。未来，我们希望让它像真正的“发明家”一样，自己想出新点子，而不是只会模仿。
Abstract
We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating ten frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: https://github.com/FrontisAI/NatureBench