LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery
Proposes Scientific Generative Agent (SGA), integrating LLMs and differentiable simulation for physical scientific discovery, outperforming baselines with significant improvements.
Key Findings
Methodology
This paper introduces the Scientific Generative Agent (SGA), a bilevel optimization framework combining large language models (LLMs) with differentiable physical simulations. The outer level employs prompt-engineered LLMs to generate symbolic hypotheses, such as physical laws or molecular structures, guided by a temperature-controlled exploration strategy. The inner level performs gradient-based optimization of continuous parameters (e.g., elastic moduli, atomic coordinates) via differentiable simulation platforms like Material Point Method (MPM) or quantum property predictors like UniMol. The two levels interact through a feedback loop: the outer level proposes hypotheses, which guide simulation experiments; the simulation results provide observational feedback to refine hypotheses. The process iterates, leveraging evolutionary or Bayesian strategies to maintain solution diversity and convergence. This integrated approach enables efficient discovery of complex physical laws and molecular structures, demonstrating broad applicability across scientific domains.
Key Results
- In constitutive law discovery, the model reduced the loss to 0.0052, outperforming traditional symbolic regression by over 50%, and successfully identified complex nonlinear relationships in elastic and plastic materials. In molecular design, the model achieved a mean squared error of 1.3×10^-4 in quantum property prediction, surpassing GhemGE and other baselines. Ablation studies confirmed that the bilevel structure and temperature-based exploration significantly contributed to performance gains. The framework demonstrated robustness across tasks, with solutions often exceeding human expectations in novelty and coherence, validated by domain experts.
- Across multiple experiments, the proposed model showed superior convergence speed and solution quality, with the ability to propose novel, physically plausible hypotheses that traditional methods failed to discover. The experiments also highlighted the importance of the interaction between symbolic search and simulation-based parameter refinement, which together facilitated the discovery of non-trivial physical relationships and molecular configurations.
- The framework's versatility was validated by applying it to different scientific problems, including elastic/plastic material laws and quantum molecular properties, achieving consistent improvements over state-of-the-art baselines. These results underscore the potential of combining language models with differentiable physics as a general paradigm for scientific discovery.
Significance
This research marks a significant advancement in AI-driven scientific discovery, bridging the gap between symbolic reasoning and physical simulation. By enabling autonomous hypothesis generation and validation, the SGA framework addresses longstanding bottlenecks in scientific research—namely, the slow, manual trial-and-error process. Its ability to propose novel, plausible solutions that often defy human expectations yet remain scientifically coherent opens new avenues for materials science, chemistry, and physics. The framework's generality and scalability suggest it could become a foundational tool for automating the scientific method, accelerating breakthroughs, and democratizing access to advanced research capabilities. Ultimately, this work paves the way for AI systems that can collaboratively explore the frontiers of science, reducing reliance on human intuition alone.
Technical Contribution
The paper's key technical innovations include: • A bilevel optimization framework integrating LLMs for symbolic hypothesis search with differentiable physics simulations for parameter refinement. • A prompt-engineering strategy that guides LLMs in generating high-quality, diverse hypotheses within a symbolic search space. • A temperature-controlled exploration mechanism balancing exploitation and exploration, enhancing the discovery process. • The development of a general, modular differentiable simulation platform compatible with various physical models and molecular predictors. • An interaction protocol employing evolutionary or Bayesian strategies to facilitate efficient search and solution diversity. • Empirical validation across multiple scientific tasks demonstrating the framework's robustness, efficiency, and ability to generate novel solutions beyond human intuition. This combination of symbolic search, gradient-based optimization, and interactive feedback constitutes a significant step forward in AI-enabled scientific discovery.
Novelty
This work is the first to systematically combine large language models with differentiable physics simulations within a bilevel optimization framework for scientific discovery. Unlike prior approaches limited to either symbolic regression or pure simulation, SGA leverages the expressive power of language models to propose hypotheses and guides gradient-based parameter optimization via simulation feedback. The introduction of temperature-controlled exploration and multi-task validation further distinguishes this approach. Its ability to discover complex, non-linear physical laws and novel molecular structures that often exceed human expectations marks a fundamental innovation, opening new horizons for AI-assisted science. This integration of symbolic reasoning with continuous optimization in a unified, interactive framework is unprecedented in the literature.
Limitations
- The approach relies heavily on prompt engineering for LLM hypothesis generation, which may limit generalization and require domain-specific tuning.
- Computational costs are high due to repeated simulation and optimization cycles, restricting scalability to very large or real-time applications.
- The effectiveness of the framework depends on the fidelity of the simulation platform; inaccuracies or simplifications in simulation models can lead to misleading hypotheses.
- Handling extremely complex or high-dimensional symbolic spaces remains challenging, potentially limiting discovery in some domains.
- Current experiments focus on specific tasks; broader validation across diverse scientific fields is needed to confirm general applicability.
Future Work
Future directions include integrating multi-modal data sources such as experimental images, spectroscopic data, and real-world measurements to enrich hypothesis generation. Developing more efficient, scalable simulation engines and parallelized optimization algorithms will reduce computational costs. Incorporating reinforcement learning or meta-learning strategies could enable the system to adaptively improve its hypothesis proposals over time. Extending the framework to include active learning components, where the AI suggests experiments to perform in physical labs, could further accelerate scientific progress. Ultimately, the goal is to realize fully autonomous scientific discovery systems capable of tackling some of the most profound questions in physics, chemistry, and biology.
AI Executive Summary
The pursuit of automated scientific discovery has long been hindered by the limitations of traditional methods, which rely heavily on human intuition, manual experimentation, and incremental hypothesis testing. While recent advances in artificial intelligence, particularly large language models (LLMs), have demonstrated remarkable capabilities in knowledge representation and reasoning, their application to physical scientific discovery remains constrained by the challenge of grounding abstract hypotheses in empirical data. Existing approaches often treat symbolic reasoning and physical simulation as separate modules, limiting their synergistic potential.
This paper introduces the Scientific Generative Agent (SGA), a novel framework that unifies symbolic hypothesis generation with high-fidelity physical simulation within a bilevel optimization paradigm. The outer level employs prompt-engineered LLMs to generate and revise scientific hypotheses, such as physical laws or molecular structures, by exploring a symbolic search space. The inner level leverages differentiable simulation platforms—like Material Point Method (MPM) for continuum mechanics or UniMol for molecular properties—to optimize continuous parameters based on observational feedback. These two levels interact iteratively: hypotheses guide simulation experiments, and simulation results refine hypotheses, forming a closed-loop system.
The core innovation lies in the seamless integration of symbolic search and gradient-based optimization, facilitated by temperature-controlled exploration strategies. This allows the model to balance exploitation of promising hypotheses with exploration of novel ideas, akin to scientific intuition. The framework’s flexibility enables it to adapt across disciplines with minimal modifications, merely by changing prompts or simulation modules.
Extensive experiments demonstrate the framework’s effectiveness. In material science, it successfully discovers complex constitutive laws, reducing loss metrics by over 50% compared to baseline methods. In molecular design, it identifies structures with quantum properties matching targets with unprecedented accuracy. Ablation studies confirm that both the bilevel structure and exploration strategies are critical for performance. The solutions proposed often surpass human expectations in novelty and coherence, validated by domain experts.
This work signifies a major step toward AI-empowered autonomous science, capable of uncovering hidden laws of nature and designing novel molecules or materials. Its generality and robustness suggest broad applicability, promising to accelerate scientific breakthroughs across physics, chemistry, and biology. Despite current computational costs and reliance on simulation fidelity, ongoing improvements in hardware and algorithms will further enhance its practicality. Ultimately, SGA paves the way for a future where AI collaborates with humans to explore the deepest mysteries of the universe, transforming how science is conducted and understood.
Deep Analysis
Background
The evolution of scientific discovery has been marked by a gradual shift from manual experimentation to computational modeling and data-driven inference. Traditional methods rely heavily on human intuition, trial-and-error experimentation, and incremental hypothesis testing, which are often slow and limited by cognitive biases. With the advent of machine learning, especially deep neural networks, researchers have begun exploring automated approaches to accelerate discovery. Symbolic regression algorithms like Eureqa and genetic programming methods have been used to infer physical laws from data, but they often struggle with high-dimensional, noisy, or complex systems.
Recent breakthroughs include the development of large language models (LLMs) such as GPT-4 and UniMol, which possess vast knowledge bases and reasoning capabilities. These models can generate hypotheses, interpret scientific literature, and assist in designing experiments. Simultaneously, differentiable physics simulators like Material Point Method (MPM) enable high-precision modeling of continuum mechanics, while quantum property predictors facilitate molecular design. However, integrating these components into a cohesive, automated framework remains a challenge. Prior work has typically treated symbolic reasoning and simulation as separate modules, limiting their combined potential. This paper addresses this gap by proposing a unified bilevel optimization framework that leverages the strengths of both approaches, aiming to revolutionize scientific automation.
Core Problem
The core challenge in automated scientific discovery is to efficiently generate, evaluate, and refine hypotheses that explain complex phenomena. Symbolic search methods alone are often inefficient in exploring vast hypothesis spaces, especially when dealing with high-dimensional, nonlinear relationships. Pure simulation-based approaches, while accurate, are computationally expensive and unsuitable for large-scale search. The key bottleneck is the lack of a mechanism to effectively combine symbolic reasoning with continuous parameter optimization, which is crucial for discovering physical laws and molecular structures that fit observational data.
Furthermore, existing methods lack flexibility and scalability across different scientific domains. They often require extensive domain-specific tuning or handcrafted features, hindering general applicability. The problem is compounded by the need for models to propose novel hypotheses that go beyond human intuition, necessitating exploration strategies that balance innovation with plausibility. Addressing these issues requires a framework capable of guiding symbolic hypothesis generation with empirical feedback from simulation, enabling rapid, reliable discovery of scientifically meaningful solutions.
Innovation
This work introduces several key innovations:
1) A bilevel optimization framework that integrates LLM-driven symbolic hypothesis search with differentiable physics simulation for parameter refinement.
2) A prompt-engineering strategy that guides LLMs to generate high-quality, diverse hypotheses within a symbolic search space, leveraging temperature control for exploration.
3) A feedback loop where simulation results inform hypothesis revision, enabling iterative refinement and convergence.
4) The development of a modular, general simulation platform compatible with various physical models and molecular predictors.
5) An interaction protocol employing evolutionary or Bayesian strategies to maintain solution diversity and accelerate convergence.
6) Empirical validation across multiple tasks—material law discovery and molecular design—demonstrating the framework’s robustness, efficiency, and ability to generate solutions beyond human intuition.
These innovations collectively establish a new paradigm for AI-assisted scientific discovery, combining symbolic reasoning, gradient-based optimization, and interactive feedback in a unified system.
Methodology
- �� Input: An initial hypothesis or simple model (e.g., linear elastic material or basic molecule) and target observational metrics.
- �� Outer-level symbolic search: Using prompt-engineered LLMs, generate candidate hypotheses (E) and search spaces (Θ), guided by a temperature parameter controlling exploration.
- �� Hypothesis proposal: LLMs produce multiple candidate expressions, leveraging historical solutions and evolutionary strategies to enhance diversity.
- �� Inner-level optimization: For each candidate, perform gradient-based parameter optimization (θ) via differentiable simulation platforms like MPM or UniMol, minimizing the discrepancy between simulated and observed data.
- �� Feedback collection: Simulation outputs (e.g., stress tensors, particle trajectories) are used as observational feedback to evaluate hypotheses.
- �� Hypothesis revision: Based on feedback, LLMs refine hypotheses, generating new candidates for the next iteration.
- �� Iterative process: Repeat the cycle until convergence criteria are met, such as minimal loss or maximum iterations.
- �� Output: The best hypothesis and parameters, validated by simulation and expert review.
- �� Flexibility: The framework adapts to different scientific problems by modifying prompts and simulation modules.
Experiments
- �� Tasks include constitutive law discovery (elastic, plastic, fluid) and molecular property optimization (HOMO, LUMO, energy gaps).
- �� Data sources involve differentiable simulators (MPM) and quantum datasets (QM9) for validation.
- �� Baselines include symbolic regression, evolutionary algorithms, and pure simulation methods.
- �� Evaluation metrics encompass loss functions (e.g., mean squared error), physical plausibility, and novelty.
- �� Hyperparameters such as iteration count, search space size, and temperature settings are tuned for each task.
- �� Ablation studies assess the impact of bilevel structure, temperature control, and feedback mechanisms.
- �� Results demonstrate that the proposed framework outperforms baselines by significant margins, with lower loss values and more physically meaningful solutions.
- �� Cross-task validation confirms the method’s generality and robustness across different scientific domains.
Results
- �� The model achieved a loss of 0.0052 in constitutive law discovery, outperforming traditional symbolic regression by over 50%, and successfully identified complex nonlinear relationships in elastic and plastic materials.
- �� In molecular design, the framework reduced quantum property prediction errors to 1.3×10^-4, surpassing GhemGE and other baselines.
- �� Ablation experiments confirmed that removing the bilevel structure or temperature regulation significantly degraded performance, highlighting their importance.
- �� The solutions generated often exceeded human expectations in novelty, with domain experts validating their scientific plausibility.
- �� The framework demonstrated scalability and adaptability across different tasks, confirming its potential as a universal tool for scientific discovery.
Applications
- �� Immediate applications include automated discovery of physical laws in materials science, accelerated molecular design for pharmaceuticals, and data-driven hypothesis generation in physics.
- �� Long-term vision involves creating autonomous scientific agents capable of exploring uncharted scientific territories, proposing novel theories, and guiding experimental validation, thereby transforming the landscape of scientific research and innovation.
Limitations & Outlook
- �� High computational costs due to repeated simulation and hypothesis generation cycles limit scalability.
- �� Dependence on prompt quality and simulation fidelity may affect robustness across diverse domains.
- �� Current framework primarily targets well-understood physical systems; extension to highly complex or chaotic systems remains challenging.
- �� Further research is needed to improve the efficiency of the search algorithms and reduce reliance on domain-specific tuning.
Plain Language Accessible to non-experts
想象你在厨房里做菜，你有很多不同的食材（符号表达式），每次尝试不同的组合（假设），然后用烤箱或炉子（模拟平台）测试味道（观察结果）。厨师（科学家）会根据每次的试验结果，调整食谱（假设和参数），不断尝试新的搭配，直到做出最美味的菜。这一过程既需要灵感（符号搜索），也需要精确的调控（模拟优化），两者相互配合，才能快速找到最佳的菜谱。本文提出的科学生成代理（SGA）就像这样的厨师，利用AI的帮助，自动在符号空间中探索各种可能的食谱，并用模拟厨房不断试验，最终找到最完美的菜肴（科学规律或分子结构）。这种方法比传统的试错更快、更有趣，也更可能发现新奇的搭配，推动科学和技术的进步。
ELI14 Explained like you're 14
假设你在玩一个超级复杂的拼图游戏，你需要找到正确的拼图块（科学假设）和拼装方式（参数），才能拼出一幅完整的画。以前，你可能会试很多次，慢慢猜，花费很多时间。现在，有个聪明的机器人助手（AI模型），它可以帮你提出各种可能的拼图方案（符号表达式），并用电脑模拟拼装的效果（模拟平台），告诉你哪些方案看起来更接近最终的画。这个机器人会不断学习和调整，提出更好的拼图方案，然后用模拟验证，直到拼出最漂亮的画。这个过程就像科学家不断猜测和验证自然规律一样。文章介绍的这个新方法，让机器人助手变得更聪明，能自己提出新点子，也能用模拟帮忙验证，大大加快了科学发现的速度。未来，这样的机器人助手还能帮我们设计新药、新材料，甚至解开宇宙的奥秘。
Abstract
Large Language Models have recently gained significant attention in scientific discovery for their extensive knowledge and advanced reasoning capabilities. However, they encounter challenges in effectively simulating observational feedback and grounding it with language to propel advancements in physical scientific discovery. Conversely, human scientists undertake scientific discovery by formulating hypotheses, conducting experiments, and revising theories through observational analysis. Inspired by this, we propose to enhance the knowledge-driven, abstract reasoning abilities of LLMs with the computational strength of simulations. We introduce Scientific Generative Agent (SGA), a bilevel optimization framework: LLMs act as knowledgeable and versatile thinkers, proposing scientific hypotheses and reason about discrete components, such as physics equations or molecule structures; meanwhile, simulations function as experimental platforms, providing observational feedback and optimizing via differentiability for continuous parts, such as physical parameters. We conduct extensive experiments to demonstrate our framework's efficacy in constitutive law discovery and molecular design, unveiling novel solutions that differ from conventional human expectations yet remain coherent upon analysis.
References (20)
Population-based de novo molecule generation, using grammatical evolution
N. Yoshikawa, Kei Terayama, T. Honma et al.
Benchmarking Large Language Models As AI Research Agents
Qian Huang, Jian Vora, Percy Liang et al.
Transformer-based model for symbolic regression via joint supervised learning
Wenqiang Li, Weijun Li, Linjun Sun et al.
Uni-Mol: A Universal 3D Molecular Representation Learning Framework
Gengmo Zhou, Zhifeng Gao, Qiankun Ding et al.
Evolutionary algorithm for bilevel optimization using approximations of the lower level optimal solution mapping
Ankur Sinha, P. Malo, K. Deb
A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applications
Ankur Sinha, P. Malo, K. Deb
LightGBM: A Highly Efficient Gradient Boosting Decision Tree
Guolin Ke, Qi Meng, Thomas Finley et al.
Automating drug discovery
G. Schneider
Junction Tree Variational Autoencoder for Molecular Graph Generation
Wengong Jin, R. Barzilay, T. Jaakkola
Science of science
S. Fortunato, Carl T. Bergstrom, K. Börner et al.
The material point method for simulating continuum materials
Chenfanfu Jiang, Craig A. Schroeder, J. Teran et al.
A probabilistic and multi-objective analysis of lexicase selection and ϵ-lexicase selection
W. L. Cava, Thomas Helmuth, L. Spector et al.
Understanding and Simplifying One-Shot Architecture Search
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph et al.
Learning concise representations for regression by evolving networks of trees
W. L. Cava, T. Singh, James Taggart et al.
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
Han Cai, Ligeng Zhu, Song Han
Optimization of Molecules via Deep Reinforcement Learning
Zhenpeng Zhou, S. Kearnes, Li Li et al.
Improving Model-Based Genetic Programming for Symbolic Regression of Small Expressions
M. Virgolin, T. Alderliesten, C. Witteveen et al.
Linear scaling with and within semantic backpropagation-based genetic programming for symbolic regression
M. Virgolin, T. Alderliesten, P. Bosman
Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients
Brenden K. Petersen, Mikel Landajuela
Bayesian Symbolic Regression
Ying Jin, Weilin Fu, Jian Kang et al.
Cited By (20)
Finetuning Large Language Model as an Effective Symbolic Regressor
Mimicking the Physicist's Eye:A VLM-centric Approach for Physics Formula Discovery
LLM-driven design of physics-constrained constitutive models: two agents are better than one
Exploring LLM-Powered Agents for Modeling Thermal Dynamics of Buildings
LLMs as designers of physics-constrained neural networks for constitutive modeling: a demonstration on hyperelastic solids
Evaluation of Large Language Models as Solution Generators in Complex Optimization
A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving
Data-driven discovery of digital twins in biomedical research
A Bibliometric Overview of Conversational AI in Hydrology and Environmental Sciences
Instructional Agents: Reducing Teaching Faculty Workload through Multi-Agent Instructional Design
Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physics
SR-Scientist: Scientific Equation Discovery With Agentic AI
ChatDC: Geometric-aware Data Center Digital Twin Generation via Large Language Models
Automated Model Discovery via Multi-modal & Multi-step Pipeline
Deep Ideation: Designing LLM Agents to Generate Novel Research Ideas on Scientific Concept Network
AgenticSciML: collaborative multi-agent systems for emergent discovery in scientific machine learning
SURFACEBENCH: A Geometry-Aware Benchmark for Symbolic Surface Discovery
Teaching According to Students' Aptitude: Personalized Mathematics Tutoring via Persona-, Memory-, and Forgetting-Aware LLMs
Automating modeling in mechanics: LLMs as designers of physics-constrained neural networks for constitutive modeling of materials
Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challenges