Using Large Language Models for Hyperparameter Optimization
Using large language models (like GPT-4) for hyperparameter tuning outperforms Bayesian optimization in limited-query scenarios.
Key Findings
Methodology
This paper introduces a framework where large language models (LLMs) are prompted with dataset, model descriptions, and search spaces to generate hyperparameter configurations. The process involves iterative feedback: the LLM suggests parameters, the environment evaluates them, and the result is fed back for refinement. Chain-of-thought prompting enhances reasoning, while code generation allows the model to produce training scripts, removing the need for manual search space specification. Experiments with GPT-4 variants demonstrate that, within small budgets (e.g., 30 evaluations), LLMs can match or outperform traditional methods like Bayesian optimization, especially in early search phases. The approach is flexible, scalable, and applicable across tasks from toy problems to real-world deep learning models.
Key Results
- On HPOBench, GPT-4 Turbo outperformed random search in 81.25% of tasks, with a median validation error improvement of 13.7%. In CIFAR-10 experiments, LLM-based tuning achieved lower validation loss than random search within 20 epochs for Vision Transformers and ResNets. Chain-of-thought prompting further improved results, especially over longer trajectories (up to 30 calls). Code generation methods excelled at ultra-low budgets (5 evaluations), showing high flexibility. Overall, LLMs demonstrated strong exploration and exploitation capabilities, reducing tuning time significantly.
Significance
This work demonstrates that large language models can serve as effective, flexible hyperparameter tuning assistants, especially in resource-constrained settings. By leveraging natural language understanding and reasoning, the approach reduces reliance on manual search space design, democratizing AutoML. The integration of code generation offers a novel avenue for automating model configuration, potentially transforming industry practices. The method bridges NLP and AutoML, opening new research directions and practical applications for scalable, autonomous model optimization.
Technical Contribution
The paper pioneers the use of LLM prompting and code generation for hyperparameter search, combining chain-of-thought reasoning with iterative feedback. It introduces a hybrid approach that integrates natural language interaction with automated code output, enabling adaptive, context-aware tuning without predefined search spaces. The framework is validated across multiple benchmarks, showing superior early-stage performance compared to Bayesian methods. This work extends the capabilities of LLMs into the AutoML domain, providing a new paradigm for flexible, data-driven hyperparameter optimization.
Novelty
This is the first comprehensive attempt to leverage large pre-trained language models for hyperparameter tuning, integrating reasoning and code generation. Unlike traditional black-box methods, this approach uses natural language prompts to interact with models, enabling dynamic, context-sensitive configuration. The use of chain-of-thought prompting and code output for search space exploration is novel, offering a flexible, scalable alternative to existing AutoML techniques. It effectively combines NLP advances with hyperparameter search, marking a significant step forward in automated model tuning.
Limitations
- Dependence on the quality and robustness of large language models; in some cases, the generated configurations may lack diversity or fail to improve performance. The computational cost of running large models, especially with multiple iterations, remains high, limiting real-time applications. The approach's effectiveness may vary across different tasks and datasets, requiring further validation for broader deployment.
Future Work
Future research will focus on integrating reinforcement learning to improve exploration efficiency, expanding multi-modal inputs (e.g., combining text and images), and optimizing inference speed. Developing methods for automatic correction of generated code and extending the framework to multi-task and multi-objective optimization are promising directions. Additionally, exploring smaller, more efficient models tailored for HPO could make the approach more accessible for industry-scale deployment.
AI Executive Summary
This study explores a novel approach to hyperparameter optimization (HPO) by leveraging the capabilities of large language models (LLMs) such as GPT-4. Traditional HPO methods, including Bayesian optimization and random search, often struggle with limited budgets and require manual specification of search spaces, which can be time-consuming and error-prone. The proposed framework prompts LLMs with detailed descriptions of datasets, models, and search spaces, enabling the models to generate candidate hyperparameters iteratively. Chain-of-thought prompting enhances the reasoning process, allowing the models to understand complex interactions among hyperparameters, while code generation extends flexibility by producing training scripts directly. Experimental results on benchmarks like HPOBench and real-world tasks such as CIFAR-10 demonstrate that LLMs can outperform classical methods in early search phases, achieving comparable or better validation accuracy with fewer evaluations. Notably, the approach is highly adaptable, capable of handling unknown search spaces by generating code that defines models and optimizers automatically. These innovations suggest a paradigm shift in AutoML, where natural language interfaces and AI-driven code synthesis reduce the need for expert intervention, democratizing model tuning. Future work aims to improve exploration strategies, reduce computational costs, and extend multi-modal capabilities, promising a new era of autonomous machine learning systems that are both efficient and accessible.
Deep Analysis
Background
超参数调优在机器学习中具有核心地位，影响模型的泛化能力和性能。早期方法如随机搜索（Bergstra & Bengio, 2012）和贝叶斯优化（Mockus, 1994; Hutter et al., 2011）已被广泛采用，但在高维空间或预算有限时效果受限。近年来，AutoML技术如Hyperband（Li et al., 2017）和多任务、多模糊策略不断提升调优效率。大模型（如GPT系列）展现出强大的自然语言理解和推理能力，为自动调参提供新思路，但在HPO中的应用尚未系统化。本文结合大模型的能力，旨在突破传统调优的瓶颈，探索其在自动化、智能化调参中的潜力。
Core Problem
传统HPO方法依赖于手工定义搜索空间和启发式策略，难以在有限评估次数内找到最优配置。贝叶斯优化虽能有效，但受模型参数和高维空间限制，扩展性不足。手动调参费时费力，且对非专业用户不友好。如何利用大模型的推理和生成能力，动态调整超参数，突破预定义空间的限制，成为亟待解决的问题。
Innovation
核心创新包括：1）将超参数调优转化为自然语言交互，通过提示引导模型生成配置；2）利用模型生成训练代码，自动定义模型和优化器，跳过手工空间设计；3）引入链式推理（CoT）增强模型推理能力，提升长轨迹调优表现；4）结合多策略（提示、代码、推理）实现高效探索。这些创新极大提升调优的灵活性和效率，超越传统黑盒优化方法。
Methodology
- �� 输入：数据集描述、模型结构、搜索空间（如学习率、正则化参数）
- �� 过程：
- �� 通过提示引导大模型生成超参数配置（JSON格式）
- �� 执行训练，评估验证指标（如损失、准确率）
- �� 将评估结果反馈给模型，模型根据反馈调整下一配置
- �� 采用链式推理（CoT）增强推理能力，提升配置质量
- �� 若搜索空间未知，模型生成训练代码（PyTorch），自动定义模型和优化器
- �� 迭代直到预算耗尽
- �� 输出：最优超参数配置或训练代码，提升调优效率。
Experiments
在HPOBench上，调优支持多模型（逻辑回归、SVM、随机森林、神经网络），共8个数据集，比较随机搜索、贝叶斯优化（SMAC）、以及大模型（GPT-4 Turbo）的方法。深度学习方面，调优CIFAR-10上的Vision Transformer和ResNet-9，限制训练轮数（20轮），仅用简洁提示。评估指标为验证损失，重点观察少量评估（如30次）内的性能提升。还进行了链式推理和代码生成的对比实验，验证其效果。
Results
在HPOBench上，GPT-4 Turbo在81.25%的任务中优于随机搜索，平均验证误差改善13.7%。在深度模型调优中，模型在20轮内实现更低验证损失，优于随机搜索。链式推理增强效果明显，长轨迹（30次调用）中表现稳定。代码生成在极低预算（5次）时优于随机，显示出极强的灵活性。整体来看，LLMs在少量评估下展现出优越的探索能力，显著提升调优效率。
Applications
该方法适用于自动调参、模型部署、AutoML平台，尤其在资源有限或缺乏专业知识的场景。通过自然语言交互，非专家用户也能实现高效调优。未来结合自动代码修正和多模态输入，有望实现全自动化的模型调优流程，推动工业界智能化水平提升。
Limitations & Outlook
模型性能依赖于预训练模型的推理能力，可能在特定任务中表现不稳定。高维复杂空间可能导致生成配置缺乏多样性。训练和推理成本较高，限制大规模应用。未来需优化模型效率，降低成本，增强鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房里做饭，调味料的用量和烹饪时间就像机器学习中的超参数。传统做法是你自己试错，调整每个调料的量，花费很多时间。现在，有个聪明的厨师（大模型）可以听你描述菜的类型和你喜欢的口味，然后给出调料的建议。你试一试，味道不满意，再告诉厨师结果，他会根据你的反馈继续调整建议。这个厨师还能自己写出调味的步骤和配方，帮你省去繁琐的试验过程。通过不断沟通和调整，最终你能做出美味佳肴，效率大大提升。这就像用大模型调超参数，快速找到最佳配置，节省时间和精力。
ELI14 Explained like you're 14
想象你在玩游戏，想让角色变得更厉害。你可以自己试很多装备和技能组合，但很费时间。现在，有个超级聪明的朋友（大模型），你告诉他你想让角色变强，他会给你一些建议。你试试效果，如果不满意，再告诉他，他会根据你的反馈调整建议。这个朋友还能帮你写出详细的升级攻略，让你不用自己试错。通过不断沟通，角色变得越来越强。这就像用大模型调超参数，快速找到最适合的设置，让模型表现更好，省时省力。
Abstract
This paper explores the use of foundational large language models (LLMs) in hyperparameter optimization (HPO). Hyperparameters are critical in determining the effectiveness of machine learning models, yet their optimization often relies on manual approaches in limited-budget settings. By prompting LLMs with dataset and model descriptions, we develop a methodology where LLMs suggest hyperparameter configurations, which are iteratively refined based on model performance. Our empirical evaluations on standard benchmarks reveal that within constrained search budgets, LLMs can match or outperform traditional HPO methods like Bayesian optimization across different models on standard benchmarks. Furthermore, we propose to treat the code specifying our model as a hyperparameter, which the LLM outputs and affords greater flexibility than existing HPO approaches.