GptGet

LLM4AD: A Platform for Algorithm Design with Large Language Model

TL;DR

LLM4AD unifies LLM-driven algorithm search; EoH, FunSearch, and (1+1)-EPS beat random sampling on most of nine tasks.

cs.AI 🟡 Intermediate 2024-12-23 17 views
Fei Liu Rui Zhang Zhuoliang Xie Rui Sun Kai Li Qinglong Hu Ping Guo Xi Lin Xialiang Tong Mingxuan Yuan Zhenkun Wang Zhichao Lu Qingfu Zhang
AI Reader Arxiv Page Download PDF
LLM automated algorithm design evolutionary search optimization scientific discovery

Key Findings

Methodology

LLM4AD is a modular Python platform built around Search Methods, an LLM Interface, and a Task Evaluation Interface. LLMs generate executable candidate algorithms; an iterative search maintains a population, evaluates candidates on task instances, and preserves elites. The platform integrates Sampling, Tabu Search, Simulated Annealing, EoH, FunSearch, (1+1)-EPS, and multi-objective methods including MEoH, NSGA-II, and MOEA/D. It supports remote APIs, local models, parallel sampling, and sandboxed execution.

Key Results

  • Under GPT-4o-Mini, 2,000 function evaluations, and three independent runs, EoH, FunSearch, and (1+1)-EPS outperform random sampling on most ACRO, BACT, CVRP, OBP, OSC, SET, TSP, and VRPTW tasks. Figure 3 shows that search-guided generation is generally more stable than repeated unstructured sampling.
  • The study evaluates Llama-3.1-8B, Yi-34b-Chat, GLM-3-Turbo, Claude-3-Haiku, Doubao-pro-4k, GPT-3.5-Turbo, GPT-4o-Mini, and Qwen-Turbo. GPT-4o-Mini scores 87.2 on HumanEval and 82.0 on MMLU, yet higher coding or knowledge scores do not consistently imply better algorithm-design performance.
  • Task difficulty varies substantially: methods are close on Mountain Car but diverge on OSC, SET, TSP, and VRPTW. EoH and FunSearch are usually robust because they preserve diversity, whereas greedy (1+1)-EPS varies more strongly across tasks.

Significance

The paper addresses a persistent infrastructure problem in LLM-assisted algorithm design: researchers lack a common toolkit, pipeline, and benchmark. LLM4AD makes it possible to study optimization, machine learning, and scientific discovery without independently rebuilding prompting, model access, search, execution, and logging. For academia, fixed budgets—2,000 evaluations, a 50-second timeout, and three runs—support more reproducible comparisons. For industry, the platform turns an LLM from a one-shot coding assistant into a monitored algorithm-search component.

Technical Contribution

The main contribution is systems-level abstraction rather than a new optimizer. LLM4AD represents design as a pluggable generate–evaluate–select loop. Its Evaluation interface standardizes objectives, timeouts, protected division, and optional Numba acceleration; its Sampler interface supports HTTPS APIs, transformers, and vLLM; and its search layer encapsulates populations, island models, evolutionary operators, and neighborhood procedures. Base, TensorBoard, and Weights & Biases profilers complete an engineering loop from prompt execution to convergence analysis.

Novelty

Unlike individual studies such as EoH, FunSearch, or ReEvo, LLM4AD unifies multiple search paradigms, LLM backends, and task families in one extensible platform. Its novelty is primarily methodological and infrastructural: standardized experimentation, secure evaluation, GUI support, documentation, and comparability. The paper does not claim a new convergence theorem or a universally superior search algorithm.

Limitations

  • Only nine benchmark tasks are analyzed in detail, with a maximum of 2,000 evaluations. The figures emphasize convergence trends rather than a complete numerical summary, limiting conclusions about aggregate superiority.
  • Performance depends on prompts, random seeds, model versions, API latency, and evaluation budget. The paper does not systematically report monetary cost, token usage, confidence intervals, or statistical significance.
  • The GUI currently supports only one method and one LLM configuration per run; batch experimentation still requires scripting.

Future Work

The authors plan to expand the suite beyond 160 tasks and add batch experiments to the GUI. Important community directions include cost-aware search, cross-task transfer, automated prompt optimization, stronger local-model support, and formal statistical testing. Program verification, interpretability, safety auditing, and out-of-distribution generalization are also necessary before deployment in high-stakes scientific or industrial settings.

AI Executive Summary

Algorithm design traditionally depends on expert intuition: researchers invent heuristics, write code, tune parameters, and repeatedly test alternatives. Large language models can now generate heuristics, optimizers, and scientific programs, but prior studies often use incompatible prompts, languages, evaluation scripts, and model interfaces. These differences make results difficult to reproduce and comparisons unreliable. LLM4AD is proposed as a shared Python infrastructure for this emerging field.

The platform connects three modules: search methods, an LLM interface, and task evaluation. An LLM proposes executable algorithms; sampling, neighborhood, or evolutionary search repeatedly refines them; and a secure sandbox scores candidates while enforcing timeouts and preventing invalid code from disrupting the run. The library includes EoH, FunSearch, (1+1)-EPS, MEoH, NSGA-II, and MOEA/D, together with remote and local model access, parallel sampling, profilers, a GUI, and more than 20 tasks. These span CVRP, TSP, OBP, Acrobot, Mountain Car, Bacterial Growth, Admissible Sets, and Nonlinear Oscillators, with more than 160 tasks planned or added.

The benchmark uses eight LLMs, nine tasks, 2,000 function evaluations, a 50-second per-evaluation limit, and three independent runs. With GPT-4o-Mini, EoH, FunSearch, and (1+1)-EPS outperform random sampling on most tasks. EoH and FunSearch are generally more stable because they preserve diversity, while greedy (1+1)-EPS is more task-sensitive. Model coding scores are not reliable predictors of design quality. LLM4AD therefore contributes less a single winning algorithm than a reproducible, extensible, and safer environment for studying how language models can participate in algorithm discovery.

Deep Analysis

Background

LLM-assisted design has progressed from code completion to heuristic generation, mathematical discovery, and scientific equation discovery. Representative systems include EoH, FunSearch, (1+1)-EPS, MEoH, ReEvo, and LLM-SR. However, these projects commonly use distinct prompts, evaluation code, programming conventions, and model APIs. As a result, researchers cannot easily determine whether an improvement comes from the LLM, the search strategy, or task-specific implementation. LLM4AD provides a common experimental substrate.

Core Problem

The central challenge is producing executable, high-quality algorithms reliably rather than obtaining an occasional plausible program. Random sampling fails to exploit feedback; a single search strategy may not suit routing, control, or scientific discovery; and generated code can loop indefinitely, time out, or corrupt the experiment. A useful platform must therefore coordinate search, model access, evaluation, safety, logging, and fair comparison.

Innovation

  • ��Modularity: Search Methods, LLM Interface, and Evaluation Interface can be replaced independently.
  • ��Search coverage: Sampling, Tabu Search, Simulated Annealing, EoH, FunSearch, (1+1)-EPS, NSGA-II, and MOEA/D are integrated.
  • ��Task ecosystem: Optimization, machine learning, and scientific discovery are represented by 20-plus tasks, with 160-plus planned or added.
  • ��Operational support: Timeouts, protected division, optional Numba acceleration, profilers, GUI, tutorials, and examples lower the barrier to experimentation.

Methodology

  • ��Initialization: The user selects an LLM, search method, task template, and evaluation budget. A template specifies imports, a function signature, a docstring, and an example implementation.
  • ��Generation: A Sampler queries an OpenAI-format HTTPS endpoint or a local transformers/vLLM model, with parallel sampling available.
  • ��Search: Candidates form a population. EoH uses evolutionary heuristic search, FunSearch uses an island model, and (1+1)-EPS retains improving variants; MEoH, NSGA-II, and MOEA/D support multiple objectives.
  • ��Evaluation: Each candidate runs on task instances and receives an objective score. The sandbox enforces time limits, handles invalid code, and can apply acceleration.
  • ��Feedback: Profilers record logs, curves, and best programs while the search retains elites and launches later generations.

Experiments

The benchmark covers ACRO, CAR, CVRP, OBP, TSP, VRPTW, BACT, SET, and OSC, spanning machine learning, combinatorial optimization, and scientific discovery. Random sampling, EoH, FunSearch, and (1+1)-EPS are compared across eight LLMs. The common budget is 2,000 function evaluations; EoH uses population size 10, while FunSearch uses 10 islands and four samples per prompt. Each algorithm evaluation is capped at 50 seconds, and every experiment is independently repeated three times.

Results

With GPT-4o-Mini, EoH, FunSearch, and (1+1)-EPS outperform Random Sampling on most tasks. Mountain Car shows small method differences, whereas OSC, SET, TSP, and VRPTW show clear separation, revealing heterogeneous task difficulty. EoH and FunSearch generally converge more reliably because of diversity mechanisms; greedy (1+1)-EPS is more task-dependent. Across LLMs, performance varies by task, and no model is significantly dominant everywhere. HumanEval and MMLU therefore provide weak proxies for algorithm-design quality.

Applications

Researchers can generate heuristics for CVRP, TSP, OBP, and VRPTW, control strategies for Acrobot or Cart Pole, and programs for BACT, ODE, or SRSD-Feynman scientific discovery. Industrial teams can wrap internal objectives in an Evaluation subclass and expose a safe template for scheduling, routing, or agent policies. Successful use requires clear metrics, typed input-output interfaces, representative instances, and explicit safety constraints.

Limitations & Outlook

Comparability remains sensitive to prompts, random seeds, model services, and computational budgets; 2,000 evaluations may be insufficient for difficult search spaces. The paper does not provide a complete per-task numerical table, detailed API costs, token usage, or formal significance tests, nor does it offer guarantees that generated programs converge. Future work should scale beyond 160 tasks, enable GUI batch runs, and add cost-aware optimization, transfer learning, program verification, interpretability, and out-of-distribution testing.

Plain Language Accessible to non-experts

Think of LLM4AD as a smart kitchen that invents recipes. In the old approach, one cook had to imagine a recipe, buy ingredients, cook it, judge it, and decide whether to try again. Different cooks used different judges, so it was hard to know who was genuinely better. LLM4AD standardizes the kitchen, the judge, and the record book.

The LLM proposes a new recipe: an algorithm. Search strategies are different improvement coaches. Random sampling keeps trying unrelated dishes; EoH and FunSearch preserve good dishes, mix useful ideas, and explore variations; (1+1)-EPS accepts a new dish only when it beats the current one. The evaluation sandbox is a safe kitchen: every dish gets limited cooking time and cannot damage the equipment.

The platform supplies many competitions: delivery routes, packing boxes, robot control, bacterial growth, and scientific formulas. Researchers can change the model, improvement strategy, or scoring rule while keeping the experiment format consistent. The results show that guided improvement usually beats asking the model for isolated ideas, but the model best at writing code is not always the best recipe inventor. LLM4AD is therefore a shared kitchen for discovering and comparing algorithms—not a guarantee that every generated solution is correct.

ELI14 Explained like you're 14

Imagine a game where you invent strategies for delivery trucks, packing boxes, or robots. You ask an AI for a strategy, but its first idea might be slow, silly, or even freeze the game. LLM4AD is like a special training arena: the AI keeps proposing strategies, the computer tests them, scores them, and saves the winners.

There are several coaches. Random Sampling is like trying random moves. EoH and FunSearch act like coaches who keep good moves, combine them, and search for clever variations. (1+1)-EPS is stricter: it keeps a new strategy only if it beats the current record. A safety referee stops programs that run forever or behave badly. Pretty useful, right?

The researchers tested eight AI models and nine challenges, including Acrobot, Mountain Car, CVRP, TSP, OBP, BACT, SET, and OSC. They allowed up to 2,000 tests, limited each test to 50 seconds, and repeated experiments three times. Usually, strategies with search coaches beat random attempts. But different challenges preferred different models and coaches, so there was no universal champion.

The coolest idea is that LLM4AD works like a shared science lab. Everyone can compare methods using similar rules, and anyone can add a new challenge or AI model. Still, it is not magic: AI ideas can be wrong, experiments cost time and money, and the best coding score does not guarantee the best strategy. Maybe one day, platforms like this will help people discover smarter solutions for games, engineering, and science!

Glossary

Large Language Model

A model that generates text or code from prompts. In this paper, it proposes candidate algorithms rather than serving as the entire optimization procedure.

The LLM Interface unifies GPT, Claude, Llama, Qwen, and other remote or local models.

Algorithm Design

The construction of procedures or programs that solve a specified problem. Quality depends on objective performance, feasibility, and computational efficiency.

LLM4AD treats design as generation and evaluation of executable programs.

EoH

Evolution of Heuristics, an approach that uses LLMs to generate and evolve heuristic algorithms. It maintains a population and preserves strong candidates while retaining variation.

EoH is one of the principal single-objective search methods benchmarked.

FunSearch

A program-search framework that combines LLM generation with an island model. Multiple islands explore different regions of the program space.

The benchmark uses 10 islands and four samples per prompt for FunSearch.

Evaluation Sandbox

An isolated, configurable environment for executing generated code. It limits runtime and prevents invalid programs from disrupting the search.

LLM4AD imposes a 50-second evaluation limit and supports protected division.

Function Evaluation

One execution of a candidate algorithm to obtain its objective score. The evaluation budget determines how many candidate programs the search can test.

The main benchmark uses a maximum of 2,000 function evaluations.

Open Questions Unanswered questions from this research

  • 1 It remains unclear how to predict which LLM will be reliable for a particular design task. HumanEval and MMLU do not consistently track design quality, motivating task-aware model-selection metrics.
  • 2 The relationship among API cost, token usage, search budget, and solution quality is not quantified. A cost–quality–latency benchmark is needed.
  • 3 Transfer to unseen instances and real industrial data remains under-tested. Cross-distribution evaluation and long-term deployment studies are required.

Applications

Immediate Applications

Automated logistics heuristics

A logistics team can wrap CVRP, TSP, OBP, or VRPTW objectives in the Evaluation interface and use EoH or FunSearch to generate routing and packing rules. The 50-second timeout and common instance evaluation enable rapid screening against human-designed baselines.

Scientific program discovery

Scientific-computing teams can use BACT, OSC, ODE, or SRSD-Feynman tasks to search for symbolic expressions and dynamical programs. They must provide precise input-output types, docstrings, representative data, and safe execution limits.

Long-term Vision

Domain-specific algorithm laboratories

Companies could connect scheduling, warehouse, control, or agent problems to LLM4AD and let models continuously search for improved procedures. With cost constraints, historical experiments, and transfer learning, the system could build reusable algorithm libraries.

Verifiable scientific discovery

A future platform could combine program search with formal verification, physical constraints, and experimental data to produce interpretable scientific models. Reliability, distribution shift, computation cost, and auditability remain major obstacles.

Abstract

We introduce LLM4AD, a unified Python platform for algorithm design (AD) with large language models (LLMs). LLM4AD is a generic framework with modularized blocks for search methods, algorithm design tasks, and LLM interface. The platform integrates numerous key methods and supports a wide range of algorithm design tasks across various domains including optimization, machine learning, and scientific discovery. We have also designed a unified evaluation sandbox to ensure a secure and robust assessment of algorithms. Additionally, we have compiled a comprehensive suite of support resources, including tutorials, examples, a user manual, online resources, and a dedicated graphical user interface (GUI) to enhance the usage of LLM4AD. We believe this platform will serve as a valuable tool for fostering future development in the merging research direction of LLM-assisted algorithm design.

cs.AI