Benchmarking Zero-Shot LLM-Generated Parent Selection in Genetic Programming for Symbolic Regression
This study evaluates eight large language models (LLMs) in zero-shot generation of parent selection operators for symbolic regression in genetic programming, demonstrating competitive performance.
Key Findings
Methodology
Using a standard GP framework, natural language prompts were employed to generate parent selection operators from eight LLMs (e.g., Claude Sonnet 4.6, Gemini 3.1 Pro). Each model produced ten operators, which were syntax-checked and then evaluated on 12 OpenML regression datasets. The evaluation involved replacing only the parent selection component while keeping other GP components fixed, measuring performance via training and test R^2 scores. Results were compared against baseline methods like lexicase and tournament selection, with statistical analysis confirming the effectiveness of certain models in producing operators that outperform baselines in search efficiency and generalization.
Key Results
- Claude Sonnet 4.6 and Gemini 3.1 Pro consistently achieved high training and test R^2 scores, with median training R^2 exceeding 0.70 and test R^2 over 0.55 across datasets. Kimi K2.5’s zero-shot synthesized operator surpassed manual heuristics, with an average training R^2 of 0.75, demonstrating superior search effectiveness. Many generated operators utilized semantic cues, indicating that LLMs can produce non-trivial heuristics solely from task descriptions. The performance of models correlated with their leaderboard scores, though with variability, highlighting the potential of zero-shot synthesis.
- The findings suggest that zero-shot LLMs can generate competitive parent selection strategies, reducing reliance on handcrafted heuristics and multi-round optimization. This approach offers a promising avenue for automating algorithm design, with implications for accelerating research and deployment in symbolic regression and related fields.
Significance
This work advances the frontier of automated algorithm synthesis by demonstrating that large language models can produce effective parent selection operators in a zero-shot setting. It addresses the longstanding challenge of designing search heuristics without manual intervention, leveraging the semantic understanding of natural language. The results imply that future systems could autonomously generate and adapt search strategies, significantly reducing development time and expertise requirements. Such capability can transform fields like symbolic regression, neural architecture search, and evolutionary computation, enabling more flexible, scalable, and intelligent optimization workflows. Moreover, the study provides insights into the relationship between model ranking and task performance, guiding future model selection and prompt engineering efforts.
Technical Contribution
This research introduces a novel framework for zero-shot synthesis of parent selection operators in GP, utilizing natural language prompts to directly generate executable code from eight state-of-the-art LLMs. It systematically compares model performance across diverse datasets, revealing that certain models (e.g., Claude and Gemini) can produce operators that outperform traditional heuristics like lexicase and tournament selection. The study also analyzes the structural features of generated operators, showing that semantic information guides the search heuristics. The approach bridges the gap between natural language understanding and program synthesis, opening new avenues for automated algorithm design and metaheuristic development.
Novelty
This is the first comprehensive evaluation of multiple contemporary LLMs in the zero-shot synthesis of parent selection operators for symbolic regression. Unlike prior work relying on multi-round or iterative refinement, this study demonstrates that a single natural language prompt can elicit highly effective search heuristics. The emphasis on direct code generation, combined with rigorous benchmarking across multiple datasets, distinguishes this work from existing approaches that depend on reinforcement learning, reflection, or multi-stage evolution. It highlights the untapped potential of large models in automating core components of evolutionary algorithms.
Limitations
- Generated operators exhibit limited generalization on unseen data, with some overfitting to training datasets. The correctness and efficiency of code depend heavily on the prompt design and model capacity, leading to variability in quality. Computational costs are significant, especially when multiple attempts are needed to produce valid operators, which may hinder scalability. Additionally, the approach’s effectiveness diminishes on more complex or high-dimensional problems, indicating room for improvement in model robustness and prompt engineering. Future work should focus on enhancing generalization, reducing costs, and integrating feedback mechanisms.
Future Work
Future research will explore multi-round feedback and reinforcement learning to refine generated operators, improving robustness and adaptability. Combining these methods with meta-learning could accelerate the discovery of effective heuristics across diverse tasks. Extending the framework to other domains such as neural architecture search or multi-objective optimization is also promising. Additionally, optimizing prompt design and leveraging model fine-tuning may boost performance and reduce costs. Ultimately, integrating zero-shot synthesis into autonomous AI systems could revolutionize algorithm development, making it more accessible and scalable.
AI Executive Summary
This study investigates the capacity of eight large language models (LLMs) to generate parent selection operators for symbolic regression within a standard genetic programming (GP) framework, relying solely on natural language prompts. Unlike traditional approaches that depend on handcrafted heuristics or multi-stage optimization, the proposed zero-shot synthesis approach evaluates the models' ability to produce effective operators directly from task descriptions. Each model generated ten operators, which were syntactically validated and then embedded into the GP process to assess their search effectiveness across twelve diverse OpenML regression datasets. The evaluation metrics included training and test R^2 scores, providing insights into both search efficiency and generalization.
Results indicate that models like Claude Sonnet 4.6 and Gemini 3.1 Pro consistently outperform baseline methods such as lexicase and tournament selection, achieving median training R^2 above 0.70 and test R^2 above 0.55. Notably, the best operator from Kimi K2.5 surpassed manual heuristics, demonstrating the potential of LLMs to produce sophisticated, semantically guided heuristics solely from natural language descriptions. Structural analysis of generated operators revealed that many leverage semantic cues, hinting at the models’ ability to produce non-trivial search heuristics.
The findings underscore the viability of zero-shot LLM synthesis as a new paradigm for automated algorithm design, reducing reliance on manual crafting and multi-round optimization. This approach can significantly accelerate the development of adaptive, scalable search strategies, with broad implications for symbolic regression, neural architecture search, and evolutionary computation. Despite promising results, challenges remain, including generalization gaps, computational costs, and the need for improved prompt engineering. Future work will focus on integrating feedback mechanisms, expanding to other domains, and optimizing model efficiency, paving the way for fully autonomous algorithm synthesis systems.
Deep Dive
Abstract
Parent selection significantly affects exploration, exploitation, and complexity control in genetic programming (GP) for symbolic regression. It is unclear whether large language models (LLMs) can synthesize effective operators in a zero-shot setting without iterative meta-evolution. Here, zero-shot means that the model receives only the task description, with no reference operators or iterative feedback. In this work, we benchmark zero-shot synthesis of parent-selection operators across eight LLMs within a standard GP framework for symbolic regression. Each model receives the same natural-language prompt to generate a parent-selection operator, which is then evaluated in a standard GP framework with only the parent-selection operator replaced, while all other components and the evolutionary-search budget are held constant. For each LLM, ten independent zero-shot operators are evaluated on twelve OpenML regression benchmarks and compared against automatic lexicase and tournament selection baselines. Claude Sonnet~4.6 and Gemini~3.1 Pro stand out for consistently strong performance on both training and held-out test $R^2$. The strongest operator in our benchmark---a Kimi~K2.5 zero-shot synthesis---surpasses the automatic lexicase and tournament baselines in search effectiveness. These results suggest that zero-shot LLM synthesis is a viable approach to generating competitive GP selection operators. Analysis shows that many generated operators use semantics to guide selection, suggesting that LLMs can produce non-trivial search heuristics from the task description alone. We also examine the relationship between public LLM leaderboard rankings and GP performance. Widely used benchmarks, such as Humanity's Last Exam and SWE-bench Verified, strongly correlate with training $R^2$, while their relationship to held-out test $R^2$ is weaker and less clear.