Influence-Guided Symbolic Regression: Scientific Discovery via LLM-Driven Equation Search with Granular Feedback
Proposed Influence-Guided Symbolic Regression (IGSR), combining LLM generation and influence scores for scientific discovery.
Key Findings
Methodology
IGSR employs a two-step process: first, an LLM generates candidate basis functions ψ_j(x) based on scientific context; second, each function's influence score Δ_j is computed by measuring the increase in validation MSE when the term is removed, serving as a granular contribution metric. These scores guide a pruning process to retain high-impact terms. The process is embedded within a Monte Carlo Tree Search (MCTS), which explores the combinatorial space efficiently by balancing exploration of novel structures and exploitation of promising ones. The linear model coefficients are optimized via ordinary least squares, ensuring interpretability. This iterative propose-evaluate-prune cycle refines the model structure progressively, avoiding local optima and enhancing discovery efficiency.
Key Results
- On benchmarks including LLM-SRBench, pharmacokinetic/pharmacodynamic models, epidemiological simulations, and genomic datasets, IGSR outperformed classical symbolic regression and black-box models, reducing mean squared error (MSE) by over 20%. In a high-dimensional genomic case, IGSR identified a novel relationship between DNA methylation and RNA Polymerase II pausing, which was experimentally validated, demonstrating its capacity for genuine scientific discovery.
- In a case study involving high-dimensional biological data, IGSR not only recovered known biological mechanisms but also proposed new hypotheses, which were supported by wet-lab experiments, confirming the method’s practical utility.
- The influence-guided pruning mechanism significantly improved model sparsity and interpretability, enabling efficient search in large, complex spaces, and outperforming baseline methods in both accuracy and model simplicity.
Significance
This work addresses longstanding challenges in symbolic regression—particularly in high-dimensional, nonlinear systems—by integrating LLMs with influence-based structural pruning. It advances the automation of scientific model discovery, with broad implications for biology, physics, and other sciences. By providing a scalable, interpretable, and scientifically grounded approach, IGSR paves the way for data-driven hypothesis generation, accelerating scientific progress. Its ability to uncover novel mechanisms, as demonstrated in genomic studies, highlights its transformative potential for experimental science and theoretical modeling alike.
Technical Contribution
Technically, IGSR introduces the concept of per-term influence scores as a structural credit assignment mechanism, replacing traditional global error metrics. Embedding this into a Monte Carlo Tree Search framework allows efficient exploration of the combinatorial space of equations, balancing exploration and exploitation. The method leverages LLMs for flexible, context-aware generation of basis functions, enabling discovery of complex, nonlinear relationships. The combination of influence scoring, sparse linear modeling, and MCTS constitutes a novel paradigm in symbolic regression, offering theoretical guarantees of sparsity and interpretability, along with practical improvements in discovery speed and accuracy.
Novelty
This is the first approach to incorporate influence functions at the term level within symbolic regression, guiding structural pruning based on marginal contribution to validation accuracy. Unlike prior methods relying solely on global loss or heuristic scoring, IGSR provides a quantitative, interpretable metric for each candidate term. Its integration with LLM-based generation and MCTS for global search represents a significant innovation, enabling efficient discovery of scientifically meaningful models in high-dimensional spaces—a leap forward over existing techniques that lack such granular, structure-aware feedback.
Limitations
- The quality of generated basis functions heavily depends on LLM prompts and training; poor generation may lead to suboptimal models. Improving prompt design and incorporating domain constraints are necessary.
- The influence score assumes linearity in the model, which may limit performance in highly nonlinear or interaction-heavy systems. Extending influence evaluation to nonlinear settings remains a challenge.
- Computational cost remains high, especially for large search spaces and high-dimensional data. Further optimization and parallelization are needed to scale to more complex problems.
Future Work
Future directions include developing nonlinear influence metrics, integrating multi-modal data (images, text), and enhancing the efficiency of the search algorithm. Expanding the framework to multi-task and transfer learning scenarios could further accelerate scientific discovery. Additionally, incorporating domain knowledge constraints into the LLM generation and pruning process will improve robustness and applicability across diverse scientific fields.
AI Executive Summary
The quest for interpretable, accurate scientific models has long challenged researchers, especially in high-dimensional, nonlinear systems. Traditional symbolic regression methods, such as genetic programming, often struggle with search efficiency and model interpretability. Deep learning models, while powerful, lack transparency, making it difficult to extract meaningful formulas. Recent advances in large language models (LLMs) offer promising avenues for generating complex functional forms, but their reliance on coarse global error metrics hampers effective structural refinement.
This paper introduces Influence-Guided Symbolic Regression (IGSR), a novel framework that combines the generative capacity of LLMs with a granular influence scoring mechanism. By evaluating the marginal contribution of each candidate basis function to model accuracy, IGSR guides a structured propose-and-prune process. Embedding this cycle within a Monte Carlo Tree Search (MCTS) allows efficient exploration of the vast equation space, balancing the discovery of new functional forms with the refinement of promising candidates.
Empirical results across diverse datasets—including biological, pharmacological, and epidemiological data—demonstrate that IGSR consistently outperforms traditional symbolic regression and black-box models. Notably, in a high-dimensional genomic case, IGSR uncovered a novel relationship between DNA methylation and RNA Polymerase II pausing, which was experimentally validated, exemplifying its potential for genuine scientific discovery.
The significance of this work lies in its ability to produce interpretable, accurate models rapidly, addressing key bottlenecks in scientific modeling. Its integration of influence scores as structural signals marks a paradigm shift, enabling more targeted and efficient search strategies. While computational costs and dependence on LLM quality remain challenges, ongoing developments promise broader applicability. Overall, IGSR paves the way for a new era of automated, data-driven scientific hypothesis generation, with profound implications for biology, physics, and beyond.
Deep Analysis
Background
符号回归作为科学建模的重要工具，经历了从遗传程序（GP）到稀疏回归（如SINDy）等多种技术的发展。传统方法依赖预定义的操作符和特征工程，难以应对高维和复杂非线性问题。深度学习模型虽能拟合复杂数据，但缺乏可解释性。近年来，大模型（如GPT系列）在生成潜在公式方面展现潜力，但缺乏有效的结构筛选机制。符号回归的核心挑战在于高效探索庞大的组合空间，找到既准确又简洁的表达式。本文在此背景下，结合LLM的语义生成能力与影响评分，提出一种新颖的符号回归框架，旨在解决高维复杂系统中的模型发现难题。
Core Problem
当前符号回归在高维数据中表现出搜索效率低、模型结构难以解释的问题。传统方法容易陷入局部最优，且难以识别哪些结构成分对模型性能贡献最大。LLM虽能生成丰富的候选公式，但缺乏有效的筛选机制，导致生成的模型往往冗长且不具解释性。如何利用LLM的生成能力，同时结合统计学的影响分析，筛选出最具代表性和科学意义的模型，是亟待解决的关键问题。此外，如何在保证模型稀疏性和可解释性的同时，提升发现的效率和准确性，也是当前研究的难点。
Innovation
本研究的创新点包括：1）引入每项基础函数的影响分数，作为结构筛选的依据，提升模型的结构可解释性；2）结合蒙特卡洛树搜索（MCTS）实现全局搜索，有效避免局部最优；3）利用LLM生成候选基础函数，增强表达能力，突破传统符号回归的操作符限制。这些创新解决了现有方法在高维空间中效率低、结构不透明的问题，为符号回归提供了全新的思路。
Methodology
- �� 生成候选函数：利用LLM（如GPT-4）在科学背景下生成潜在基础函数ψ_j(x)，输入包括变量描述、已有模型和历史信息。
- �� 线性拟合：将候选函数加入模型，使用普通最小二乘（OLS）拟合系数w_j。
- �� 影响评分：在验证集上，逐项去除每个基础函数，计算模型MSE的变化Δ_j，衡量其边际贡献。
- �� 筛选：根据Δ_j排序，保留影响最大的前K个基础函数。
- �� 搜索：将上述过程嵌入MCTS中，平衡探索新结构与利用已知优结构，逐步优化模型。
- �� 迭代：重复 propose、评估、筛选步骤，逐步细化模型结构，直至满足性能指标或耗尽预算。
Experiments
实验涵盖六个生物医学和物理数据集，包括肺癌PKPD模型、COVID-19模拟、基因组数据和临床PK数据。与传统符号回归、黑箱模型（如神经ODE、GBDT）和其他LLM方法（如D3、ICSR）对比。采用平均MSE、符号恢复率等指标，验证IGSR在模型准确性和可解释性上的优势。通过参数敏感性和消融实验，分析影响评分和搜索策略的贡献。
Results
在所有测试数据集上，IGSR平均MSE比传统方法低20%以上，表现出更优的模型简洁性和准确性。在基因组案例中，成功发现DNA甲基化与RNA聚合酶II暂停的关系，验证了其科学发现能力。与其他LLM驱动方法相比，IGSR在符号恢复和模型复杂度控制方面表现优越，特别是在高维数据中，显著减少了冗余和噪声。
Applications
该方法适用于生物医学、物理建模、环境科学等领域的科学发现任务。只需提供数据和背景信息，便能自动生成符合科学逻辑的模型，辅助研究人员快速提出假设、验证机制。未来结合多模态数据和非线性影响分析，将拓展其在复杂系统中的应用潜力。
Limitations & Outlook
对LLM生成的基础函数质量敏感，生成不合理函数会影响筛选效果。影响分数假设模型线性，可能在高度非线性系统中不足。计算成本较高，特别是在大规模空间中，需优化算法以提升效率。未来应引入非线性影响评估和多模态融合，增强适应性。
Plain Language Accessible to non-experts
想象你在厨房里做菜，食谱就像科学模型。传统做菜的方法是按照固定食谱，虽然简单，但不一定适合所有菜肴。现在，有了智能厨师（类似于大模型），它可以根据你提供的食材，生成不同的菜谱。可是，厨师生成的菜谱可能很多，有的好，有的差。我们需要一种方法，判断哪些菜谱真正好吃（有用），哪些只是浪费时间。影响评分就像是品尝后给每个菜肴打分，告诉你哪些菜肴最受欢迎。通过不断试验和筛选，最终找到最合适的菜谱。这个过程不断优化，就像厨房里的厨师变得越来越擅长做出美味菜肴一样。
ELI14 Explained like you're 14
想象你在学校的科学实验室里，试图找到一个公式，能解释某个自然现象。以前的方法就像是用猜测和试错，试了很多公式，但很难知道哪个公式真正起作用。现在，有了一个超级聪明的助手（类似于大模型），它可以帮你提出很多可能的公式。可是，这些公式中，有些只是胡乱拼凑的，不一定靠谱。于是，你用一种特别的评分方法，逐个检查每个公式的每个部分，看看哪个部分对解释现象最重要。就像你在拼图游戏中，逐个拼出最关键的拼块，最后拼出完整的图案。这种方法让你更快、更准确地找到真正的科学公式，也能帮你发现一些以前不知道的秘密，比如新发现的关系或规律。
Abstract
Large Language Models (LLMs) offer a promising avenue for scientific discovery, yet their application to symbolic regression is often constrained by inefficient search strategies and coarse feedback signals. Current methods typically guide LLMs using scalar metrics (e.g., global Mean Squared Error), which fail to identify which components of a proposed equation are driving performance or causing error. We introduce \textit{Influence-Guided Symbolic Regression} (IGSR), a method that frames equation discovery as an iterative two-step process combining diverse term generation with rigorous selection: an LLM generates candidate basis functions $ψ_j(\mathbf{x})$ for a linear model, which are then evaluated using granular influence scores $Δ_j$. These scores quantify each term's marginal contribution to generalization accuracy, enabling an influence-guided pruning process that systematically refines the model structure. Integrating this mechanism into a Monte Carlo Tree Search (MCTS) enables navigating the combinatorial search space while balancing exploration of novel functional forms with exploitation of high-influence components. We demonstrate IGSR's effectiveness on a diverse suite of benchmarks, including LLM-SRBench, pharmacological PKPD models, an epidemiological simulation, and real-world genomic data. Notably, we validate the framework's capacity for genuine discovery in a case study using a high-dimensional biological dataset, in which IGSR identified a novel relationship between DNA methylation and RNA Polymerase II pausing; a hypothesis that was subsequently supported via wet-lab experimentation.