Can LLMs Discover Scientific Laws in Real and Parallel Worlds?
Introduces SCILAWS-BENCH, a real-data benchmark evaluating LLMs' ability to discover scientific laws, covering 118 problems, 291 candidate laws, and 8M data points.
Key Findings
Methodology
The paper constructs SCILAWS-BENCH with 118 problems from scientific literature across six disciplines, involving 291 candidate laws and 8 million data points. It features two evaluation modes: SCILAWS-REAL, where models propose laws from fixed observations, and SCILAWS-PARALLEL, where models actively query a simulated environment to recover hidden laws. Nine state-of-the-art LLMs, integrated within a ReAct framework, perform multi-turn interactions, with metrics assessing predictive fit, scientific validity, and structural recovery, validated against source literature and expert judgment.
Key Results
- Models exhibit a significant gap between predictive accuracy and scientific validity; GPT-5.5 achieves 50.77% in fit but only 81.84% in validity. Performance varies across disciplines, with top models excelling in some areas but struggling with novel structure discovery, where full recovery rates are below 21%. Memorization plays a key role in reproducing known formulas, but generalization to new laws remains limited.
- Memory influences formula reproduction, with models recalling canonical laws in known tasks (average recall 11.9%) but performing poorly on unseen problems. Active querying improves structure recovery modestly, yet the maximum full-law recovery remains under 21%. These findings highlight the potential and current limitations of LLMs in scientific discovery, emphasizing the need for better reasoning and generalization capabilities.
- Overall, the study demonstrates that while LLMs can approximate scientific laws, their reliability and capacity for genuine discovery are still evolving. The benchmark provides a comprehensive tool to measure progress and guide future research toward more robust, generalizable AI for science.
Significance
This work establishes a rigorous, real-data benchmark for evaluating AI's scientific discovery ability, addressing the gap between synthetic benchmarks and real-world complexity. It offers a systematic framework for assessing models' reasoning, structure recovery, and scientific validity across disciplines, fostering progress toward autonomous scientific discovery. The benchmark's multi-faceted evaluation encourages development of models capable of reasoning, generalization, and innovation in noisy, heterogeneous environments, ultimately accelerating scientific breakthroughs and reducing reliance on manual labor.
Technical Contribution
The paper's key technical innovations include constructing a multi-disciplinary, real-data benchmark integrating passive and active evaluation modes, and developing a dual-metric system combining predictive fit and scientific validity. It introduces a novel active querying paradigm within a ReAct framework, enabling models to simulate scientific exploration. The approach emphasizes structure recovery and validity, moving beyond mere data fitting, and provides a comprehensive, scalable platform for future AI-driven scientific research.
Novelty
This is the first systematic attempt to evaluate LLMs' ability to discover scientific laws directly from real scientific data, moving beyond synthetic or textbook-based benchmarks. The integration of active querying with real data, along with a dual evaluation of fit and validity, represents a significant step forward in benchmarking AI for genuine scientific reasoning and discovery, setting a new standard for future research.
Limitations
- Models still struggle with discovering novel, complex structures beyond memorized formulas, especially in noisy, multi-source data environments. The current approach relies heavily on existing literature, limiting exploration of truly unknown laws.
- Evaluation metrics, while comprehensive, cannot fully capture scientific validity or creativity, requiring further refinement and expert validation. Computational costs remain high, especially for multi-turn interactions.
- Future improvements should focus on enhancing reasoning depth, reducing reliance on memorization, and scaling active exploration strategies to handle more complex, high-dimensional scientific problems.
Future Work
Future research will expand to more disciplines, incorporate domain-specific knowledge, and develop more sophisticated active querying strategies. Enhancing model reasoning, creativity, and robustness will be key, aiming toward fully autonomous scientific discovery systems. Integrating physical constraints and causal reasoning could further improve the scientific validity of discovered laws, pushing AI closer to true scientific reasoning.
AI Executive Summary
Discovering fundamental scientific laws has historically driven scientific progress, yet automating this process remains a significant challenge. Traditional approaches rely heavily on manual experimentation and human intuition, which are time-consuming and limited by cognitive biases. Recent advances in large language models (LLMs) like GPT-5.5 and Claude 4.8 have demonstrated remarkable capabilities in language understanding and reasoning, prompting researchers to explore their potential in scientific discovery.
This paper introduces SCILAWS-BENCH, a comprehensive benchmark designed to evaluate LLMs' ability to discover scientific laws from real-world data. Unlike prior benchmarks based on synthetic data or textbook formulas, SCILAWS-BENCH leverages authentic scientific papers and datasets across six disciplines, including astronomy, physics, materials science, biology, ecology, and social sciences. It comprises 118 problems, 291 candidate laws, and approximately 8 million data points, providing a realistic and challenging environment for testing models.
The benchmark features two evaluation modes: SCILAWS-REAL, where models propose laws based on fixed observational data, and SCILAWS-PARALLEL, which simulates active scientific exploration by allowing models to query a virtual environment to uncover hidden laws. This dual setup captures both passive inference and active hypothesis testing, mimicking real scientific workflows.
Experimental results reveal that current models, despite high predictive fit in some cases, often produce laws that lack scientific validity. The best model, GPT-5.5, achieves a 50.77% fit rate but only 81.84% validity, with structure recovery rates below 21%. These findings highlight the gap between data fitting and scientific reasoning, emphasizing the need for models that can generalize beyond memorized formulas and reason about unseen phenomena.
Overall, this work provides a vital step toward AI systems capable of genuine scientific discovery. It offers a rigorous, real-data benchmark to guide future research, encouraging the development of models that can reason, innovate, and validate scientific laws reliably. While challenges remain, especially in discovering complex, novel laws, the benchmark sets a foundation for advancing AI's role in automating scientific progress, ultimately accelerating innovation and understanding across disciplines.
Deep Dive
Glossary
Symbolic Regression (符号回归)
一种自动发现数学表达式的算法，旨在从数据中找到符合科学规律的公式。
用于评估模型是否能发现符合科学规律的数学表达式。
ReAct Framework (反应框架)
结合推理与行动的交互式框架，使模型在多轮对话中进行推理和操作。
用于模型在科学任务中的多轮交互和探索。
科学有效性 (Scientific Validity)
判断提出的公式是否符合科学原理和论文中的验证，确保其合理性。
作为模型输出的公式是否具有科学依据的重要评判标准。
结构恢复 (Structure Recovery)
模型从数据中正确识别潜在的数学结构或规律的能力。
评估模型是否能发现隐藏的科学规律。
主动查询 (Active Querying)
模型通过主动选择测量或数据点以探索未知规律的过程。
在SCILAWS-PARALLEL中模拟科学家主动探索未知规律。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在复杂多源、多学科环境中的推理和创新能力，尤其是在缺乏先验知识或面对高度噪声数据时的表现。
- 2 如何设计更有效的评估指标，全面衡量模型的科学合理性、创新性和泛化能力，避免偏重拟合或记忆。
- 3 未来能否结合因果推理和物理约束，增强模型的科学推理深度和可信度。
Applications
Immediate Applications
科学研究辅助
利用模型自动分析实验数据，提出潜在规律，加快新理论的发现过程。
教育和培训
作为学习工具，帮助学生理解科学规律的推导和验证过程，提升科学素养。
Long-term Vision
自动化科学探索
开发全自动的科学发现系统，减少人类科研投入，推动新材料、新药等创新。
Abstract
Scientific equation discovery has long been central to scientific progress, proceeding through iterative cycles of hypothesis generation, observational testing, and refinement under scientific constraints. As LLM capabilities advance and their role in AI for Science expands, it remains an open problem whether they can genuinely discover scientific laws and how this ability should be evaluated. Existing evaluations, however, often either simplify discovery through synthetic settings or reuse published targets that may already be familiar to LLMs. We therefore introduce SCILAWS-BENCH, a benchmark for scientific law discovery built from published research and real scientific data. It comprises 118 problems drawn from 381 scientific papers, covering 291 candidate laws and roughly 8M real data points across six scientific disciplines. Each problem is instantiated in two complementary settings: (1) SCILAWS-REAL asks models to propose laws from fixed real observations and evaluates held-out predictive fit and scientific validity derived from the source literature, and (2) SCILAWS-PARALLEL asks models to actively query residual-calibrated worlds and recover synthesized hidden laws derived from published forms. This two-setting task design preserves each problem's scientific context while separately evaluating fixed-record law discovery and active recovery of a newly synthesized hidden law. We find that predictive fit can diverge from scientific validity, memorization shapes whether models reproduce or move beyond published formulas, and our best-of-N study reveals a selection bottleneck. Our work provides a paper-grounded benchmark and new empirical perspectives for evaluating AI for scientific discovery. Project page: https://yiyihum.github.io/SciLaws-Bench